[gentoo-soc] Project IDFetch - Weekly report #3 ("Strawberry issues")

[gentoo-soc] Project IDFetch - Weekly report #3 ("Strawberry issues")

[Kostyantyn Ovechko]

-      ========================================================       -
-      Project IDFetch - Weekly report #3 ("Strawberry issues")       - 
-      ========================================================       -

Switching development environment is usually not an easy thing, and it
was not so easy this time when i decided to move to village. Thought
you get clear air, lungs full of oxygen, and mouthful of strawberries [1]
there're some drawbacks - Internet connection choices are limited to
GPRS. Without going into details of my experiments on attaching my cell
phone to the 3rd store of the 2-stored house let's just say that taking
things for granted when you have them is simple, but fighting for
things you used to have can be quite a deal. In conclusion results are:

I) No 3G yet, but DTR is up to 15 Kb/s (not so bad); 

II) 2 broken battery-chargers for my cellphone (one got broken connector
when the phone was falling from the 3rd floor, i still can't
understand how i managed to brake the 2nd one);

III) My cat "MOOrcello Mastroianni" AKA "Moorchik" gone missing ;-((((

IV) Few more results on idfetch project itself (read further).



IDFetch project progress 
========================

1. [ Exporting duplicate distfiles ]

Sometimes exporting pkg.list by #emerge -f pkg command produces
duplicate distfiles. Some fixes for fetch.py have been added to
control/skip duplicate distfiles.


2. [ Skipping downloaded distfiles ]

Skipping distfiles is fun, so skipping for already downloaded distfiles
has been added to the segget.


3. [ Skipping downloaded segments ]

More skipping. If settings.resume_on option is set then fully downloaded
segments are also skipped.


4. [ Interface improvements ]
Introduction of Ncurses [2] into segget for more intuitive view on the
downloading processes [3]. 


5. [ Gethering stats ]

5.1. Runtime visualization of progress for each connection has been
implemented [3].

5.2. Each connection provided with speed measurement [3].
For now it's only current speed, though it's actually avg over 1 second.
TO-DO: avg speed measurement for the whole connection time period.

5.3. Total speed is also measured [3].
TO-DO: Avg speed measurement from the moment segget started.
Note: Later in daemon mode - times when daemon is idle won't be counted.

5.4. In case of unsuccessful segment fetch, segget increases
segment->try_num and restarts fetching.


6. [ Combining segments into distfiles ]
There're few improvements/fixes to the process of combining segments
into distfiles. Segments get deleted after being joined.


7. [ RMD160, SHA1, SHA256, SHA512, WHIRLPOOL, MD5, CRC32 ]

7.1. libcrypto++[4] employed to check RMD160, SHA1 and SHA256 hashes of
downloaded distfiles. Errors are logged.

7.2. Support for SHA512, WHIRLPOOL, MD5, CRC32 have also been added
after remark from Robin H.Johnson.

7.3. Later libcrypto++ might be replaced by one of: libmhash, NSS,
openssl.


8. [ Logs ]
Ncurses is a real jam but there's no life without logs: segget.log,
error.log and debug.log have been added.


9. [ Strawberries in virtual boxes ]
Cost of going for strawberries is more expensive and slow Internet
connection. Hence, emulating mirrors became quite useful:
- no need to consume Internet traffic;
- no cable in my laptop (no wifi limitations either);
- more control on experimental environment (changing speed, connection
limits on the server side).
For this purpose VirtualBox with Host-only Ethernet adapters [5] is used
and some modifications to pkg.list file has been made (see json fragment
with pkg dict):

    {
        "CATEGORY": "dev-libs", 
        "distfile_list": [
            {
                "RMD160": "82e5061ec76f23643ba5477ab28bbee6eebd393a", 
                "SHA1": "b836783ebd72d5bc6a916620ab2b1ecec316fef1",
"SHA256":"b522f0b5f850b50e9917823ea986f855295407380fafbe30f358875c41998bc5", 
                "name": "cryptopp560.zip", 
                "size": 1049029, 
                "url_list": [
                    "ftp://192.168.6.11/cryptopp560.zip",
                    "ftp://192.168.6.12/cryptopp560.zip", 
                    "ftp://192.168.6.13/cryptopp560.zip", 
                    "ftp://192.168.6.14/cryptopp560.zip"
                ]
            }
        ], 
        "pkg_name": "crypto++-5.6.0-r1"
    }

[1] http://soc.dev.gentoo.org/~simka/wr3/strawberry.jpg
[2] http://en.wikipedia.org/wiki/Ncurses
[3] http://soc.dev.gentoo.org/~simka/wr3/ui.jpg
[4] http://www.cryptopp.com/
[5] http://www.virtualbox.org/manual/ch06.html#network_hostonly

Best regards,
Kostyantyn Ovechko AKA simka