[gentoo-dev] Repoman rewrite stage3. Migrate check data to the tree

[gentoo-dev] Repoman rewrite stage3. Migrate check data to the tree

[Brian Dolbec]

With the repoman rewrite stage2 code about to be merged into master.
And a release including it to be made soon.  I am going to start on
stage3 of the rewrite. So...

It is time to start painting a bikeshed about moving all possible
check data to the tree.  In that way small changes to things like
the deprecated eclasses to scan for can be added to/or removed and the
updates will propagate to everyone as they git pull the gentoo repo.

So, here is my initial thoughts so far.

We (portage team) discussed the possibility of making the data available
for download via api.gentoo.org.  But we decided it would be much
better if that data was included in the repository.  In that way it
makes off-line work easier without having to deal with having a live
connection to fetch the latest version of the data.  It also then
matches your current tree checkout.  It could also make it easier for
alternate repositories to establish their own QA data files.

I'm thinking that we should establish a directory for the files
containing the data for the checks being run.  With the module systems
now in place with the stage2 rewrite code. I was thinking that we
should break up the data into logical files to go with the different
scan modules present in repoman.  If a new module is created, then a
new data file may be created for it's dynamically changeable QA data.

This also has the advantage that if a new app comes along that performs
these same types of checks.  It should be able to make use of this same
data.


 So, where do we place this directory and what rules do we
establish about it's modifications?

   location? : in the metadata dir alongside the install-qa-check.d
               directory?

   name of the directory? : repoman, qa-rules, qa-data,
                            repo-qa-data, ... ideas?

   data format? : json (my favorite) 
                        compatible with many lanquages/interfaces
                        is flexible to match various data types
                          ie: dictionaries, lists, strings...
                        is human readable/editable
                        can be validated

                   xml (PLEASE NO!)

                   native python file  (too language dependant)

                   ini style (python configparser compatible) meh :/

                   other ideas?

   editing rules? :  Similar to eclass modifications... email to the
                     gentoo-dev list for review, possible objections.



The current releases of portage/repoman would continue to use internal
data.  But this new data would need to precede a repoman release that
uses it.  Initially this stage3 code will continue to be developed in
the repoman branch of the portage repo.  It is also possible to test
this git checkout code by establishing a symlink to the checkouts
repoman command somewhere in your users PATH.  Personally I named mine
repom, and lives in my ~/bin directory.  In that way I can easily
switch between an installed repoman and the developing code to compare
results to aid in debugging.

-- 
Brian Dolbec <dolsen>

Re: [gentoo-dev] Repoman rewrite stage3. Migrate check data to the tree

[Patrick McLean]

On Thu, 10 Mar 2016 18:30:07 -0800
Brian Dolbec <dolsen@gentoo.org> wrote:

>  So, where do we place this directory and what rules do we
> establish about it's modifications?
> 
>    location? : in the metadata dir alongside the install-qa-check.d
>                directory?

That sounds reasonable to me, it is certainly metadata.

> 
>    name of the directory? : repoman, qa-rules, qa-data,
>                             repo-qa-data, ... ideas?

Something not project name specific, so nothing about repoman. Perhaps
something like "repo-checks", my personal vote would be make it a
directory with the contents being merged (so repo-checks.d maybe?)

> 
>    data format? : json (my favorite) 
>                         compatible with many lanquages/interfaces
>                         is flexible to match various data types
>                           ie: dictionaries, lists, strings...
>                         is human readable/editable
>                         can be validated
> 
>                    xml (PLEASE NO!)
> 
>                    native python file  (too language dependant)
> 
>                    ini style (python configparser compatible) meh :/
> 
>                    other ideas?

YAML - like JSON but made to be edited/read by humans (comment support
is a big feature). Also valid JSON is valid YAML. Also can be validated
just like JSON can.

Re: [gentoo-dev] Repoman rewrite stage3. Migrate check data to the tree

[Brian Dolbec]

On Thu, 10 Mar 2016 18:40:31 -0800
Patrick McLean <chutzpah@gentoo.org> wrote:

> On Thu, 10 Mar 2016 18:30:07 -0800
> Brian Dolbec <dolsen@gentoo.org> wrote:
> 
> >  So, where do we place this directory and what rules do we
> > establish about it's modifications?
> > 
> >    location? : in the metadata dir alongside the install-qa-check.d
> >                directory?  
> 
> That sounds reasonable to me, it is certainly metadata.
> 
> > 
> >    name of the directory? : repoman, qa-rules, qa-data,
> >                             repo-qa-data, ... ideas?  
> 
> Something not project name specific, so nothing about repoman. Perhaps
> something like "repo-checks", my personal vote would be make it a
> directory with the contents being merged (so repo-checks.d maybe?)
> 
> > 
> >    data format? : json (my favorite) 
> >                         compatible with many lanquages/interfaces
> >                         is flexible to match various data types
> >                           ie: dictionaries, lists, strings...
> >                         is human readable/editable
> >                         can be validated
> > 
> >                    xml (PLEASE NO!)
> > 
> >                    native python file  (too language dependant)
> > 
> >                    ini style (python configparser compatible) meh :/
> > 
> >                    other ideas?  
> 
> YAML - like JSON but made to be edited/read by humans (comment support
> is a big feature). Also valid JSON is valid YAML. Also can be
> validated just like JSON can.

OK, I just had a closer look at yaml.  It does look easier for humans
to edit and read.  And seems to have the same data type flexibility.
Maybe not quite as many languages have libs for it.  But I don't think
that is an issue for this data.


I also want to separate the repoman release from the main portage
release.  This will have several advantages.

    1) smaller portage install for installs that don't need repoman
    capabilities.

    2) Repoman can add extra dependencies as needed for things like
    pyyaml and xmlschema.  These deps would not be required for the
    base portage/emerge code.  Keeping a base install stage3 lighter
    and not complicate bootstrapping a stage1, stage2.

    3) They can be released indendant of each other as the need for a
    new release is desired/required.  While repoman will still be tied
    to the portage codebase.  Most of it's code use is via fairly
    stable API's.  So only when those API's change will there need to
    be a simultaneos release.

-- 
Brian Dolbec <dolsen>

Re: [gentoo-dev] Repoman rewrite stage3. Migrate check data to the tree

[Kristian Fiskerstrand]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 03/11/2016 07:35 PM, Brian Dolbec wrote:
> On Thu, 10 Mar 2016 18:40:31 -0800 Patrick McLean
> <chutzpah@gentoo.org> wrote:
> 
>> On Thu, 10 Mar 2016 18:30:07 -0800 Brian Dolbec
>> <dolsen@gentoo.org> wrote:


>> YAML - like JSON but made to be edited/read by humans (comment
>> support is a big feature). Also valid JSON is valid YAML. Also
>> can be validated just like JSON can.
> 
> OK, I just had a closer look at yaml.  It does look easier for
> humans to edit and read.  And seems to have the same data type
> flexibility. Maybe not quite as many languages have libs for it.
> But I don't think that is an issue for this data.
> 

Yaml should be well supported so indeed, wouldn't expect any issues.

> 
> I also want to separate the repoman release from the main portage 
> release.  This will have several advantages.
> 

Sounds reasonable


- -- 
Kristian Fiskerstrand
OpenPGP certificate reachable at hkp://pool.sks-keyservers.net
fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3
-----BEGIN PGP SIGNATURE-----

iQEcBAEBCgAGBQJW5Fl4AAoJECULev7WN52FDlQH/16B/DG09LOJnJiVeMhsCVYe
MpvsPy4EJfzzAmqgYJhKY4NxD3/X/Y0CbMlO2o1C65WtaO6B85DdzCYuV44lq4Wd
EujeJT5dveRU/uihEIPU9P45yLiHd/3iUJ6p0mu2f+OxcVXbEubuf16o4J9j2nmJ
4p3xhOjPoYUlFKalDN2dNEy+JpAB/0gFjnmO5HMyhgnwnxj6o7SLHBLANBm4bReA
m6CpmKxoxVpQ88c+xEJz9/X/FjtMVEFixVDphA9zrv1MEMh1K5RuDat0Y2U5BuYT
71/M8AvG+2B64hSmM22EwK/Wk/mzddFQrwYahFfDofd42jYYIQw18gd5t76j6es=
=FNfI
-----END PGP SIGNATURE-----