[gentoo-soc] Project IDFetch - Weekly report #2

[gentoo-soc] Project IDFetch - Weekly report #2

[Kostyantyn Ovechko]

-                   Project IDFetch - Weekly report #2                 -
                    ==================================

        Though, twrapper.py is a good thing to play with: it takes list
        of files from pkg.list and starts downloading them
        simultaneously. Nevertheless it uses wget tool for this and
        communication between twrapper.py has some serious limitations
        especially when it comes to controling simultaneous downloads
        (speed, segments, etc). So next step for idfetch project was to
        make its own tool for segmented file fetching.
        
        
     1. [Replacing stuff]
        Started from replacing chocolate module by dates [1] (i still
        use coffee though), python by cpp, wget by libcurl [2].
        
     2. [JSON struggles]. Going cpp grows my love to python.

        2.1 Using python
        
                2.1.1 Saving json data with python
                
                        To save data to json-formated pkg.list fetch.py
                        uses following lines:
                
                                import json
                                
                                idfetch_pkg_list_file.write(json.dumps(idfetch_pkg_list, sort_keys=1, indent=4))
                        
                2.1.2 Loading json data with python
                
                        To load data from json-formated pkg.list
                        twrapper.py uses these 2 lines:
                        
                                import json
                                
                                idfetch_pkg_list=json.loads(idfetch_pkg_list_file.read())
                        
        2.2 Using cpp
                        
        
                2.2.1 Loading json data with cpp
                        Reading data in cpp resulted in:
                                a) #emerge json-c
                                b) g++ -o task task.c -ljson (where
                                task.c has 146 lines of code = 5037
                                bytes);
                                c++) minus one day spend on these 146
                                lines of code.
                      
     3. [Discovering libcurl].
        It's the first time I use libcurl for my project (Thanks to
        Robin H. Johnson). Libcurl happened to be pretty interesting lib
        with lots of opportunities to discover and implement into
        idfetch. Therefore I only hope not to bump into some grueling
        bug or limitation.
        
     4. [Downloading with libcurl].
        
                Experiments at stage 3 resulted in segget.cpp. Segget downloads        
                files, splitting them into segments. It's possible to set number of
                parallel connections for segget. After fetching segments, segget
                combines them together to provide original file (no checksum check
                implemented yet).                  
                
[1] http://en.wikipedia.org/wiki/Phoenix_dactylifera
[2] http://curl.haxx.se/libcurl/

Best regards,
Kostyantyn Ovechko AKA simka