[gentoo-dev] [rfc] layman-global.txt, repositories.xml, layman, overlays.gentoo.org

[gentoo-dev] [rfc] layman-global.txt, repositories.xml, layman, overlays.gentoo.org

[Sebastian Pipping]

Hello there!


This may look like like a lot of text but it actually isn't.
Please read on.  Thanks.


During the last few days rbu and I have been teaming up on a proposal
implementation improving on some of the issues that persist around
layman-global.txt.  (previously summarized in [1])

Both layman and the feed aggregator (planet) behind overlays.gentoo.org
work with a database of information on repositories.  These databases
are not shared so changes do not propagate.  That's not cool.


What we have done for now:

 - Extended layman-global.txt format to repositories.xml format adding
   - A list of feed URIs
         .. for the planet side
   - Several source URIs
         .. for mirroring and several protocols
   - Owner name
         i.e. the name of the person to contact
   - quality classification
         (core, stable, testing, experimental, graveyard)
         .. to better guide users
   - project/person distinction
       .. for the planet side mainly

 - Created DTDs and Relax NG schemas for both the old and new format

 - Wrote a converter script from layman-global.txt format to
     repositories.xml format and back.  It also adds
     "generated file" style warnings.  The scripts have no dependencies
     other than plain Python.

 - Prepared a patch against layman trunk at r38 that adds support for
   - repositories.xml format
   - display of owner names (in addition to e-mail addresses)


We have more or less the following future process in mind:

 - Collect feedback from the list (right now)
     and adjust the format and process as needed

 - Give birth to repositories.xml online
   - Decide about the final web location of repositories.xml
   - Put an initial repositories.xml on gentoo infra
   - Set up the repositories.xml to layman-global.txt converter
     script on the same machine to propagate all changes over
     to layman-global.txt in an automated fashion so we can keep
     it alive and up-to-date while still only editing its replacement

 - Revise layman patch until wrobel wants it upstream
     and put our a new layman release after

 - Setup script to create planet config from repositories.xml
     (Script creation is in progress.)

 - Take layman-global.txt offline in a year or so


Here is a sample entry for each XML formats:

layman-global.txt
=====================================================================
  <overlay
      contact="sebastian@pipping.org"
      name="sping" src="git://git.goodpoint.de/overlay-sping.git"
      status="unofficial"
      type="git">
    <link>http://git.goodpoint.de/?p=overlay-sping.git</link>
    <description>Gentoo overlay of Sebastian Pipping</description>
  </overlay>
=====================================================================

repositories.xml
=====================================================================
  <repo
      name="sping"
      quality="experimental"
      status="unofficial">
    <description>Gentoo overlay of Sebastian Pipping</description>
    <homepage>http://git.goodpoint.de/?p=overlay-sping.git</homepage>
    <owner type="person">
      <email>sebastian@pipping.org</email>
      <name>Sebastian Pipping</name>
    </owner>
    <source type="git">git://git.goodpoint.de/overlay-sping.git</source>
    <feed>http://git.goodpoint.de/?p=overlay-sping.git.git;a=atom</feed>
  </repo>
=====================================================================


The code and schemas can be found at [2].

Now please ask questions and let us know what you think.



Sebastian


[1] http://www.mail-archive.com/gentoo-dev@lists.gentoo.org/msg35095.html
[2] http://git.goodpoint.de/?p=overlays-xml-specification.git;a=summary

Re: [gentoo-dev] [rfc] layman-global.txt, repositories.xml, layman, overlays.gentoo.org

[volkmar]

On Mon, Sep 28, 2009 at 08:23:34PM +0200, Sebastian Pipping wrote:
> Hello there!
> 
> 
> This may look like like a lot of text but it actually isn't.
> Please read on.  Thanks.

Thanks for your good work.
I agree with your changes it was indeed needed.

Actually, during my GSOC, I've noted layman-global.txt should be changed
to have a cleanier description information and it's probably the moment
to introduce this.

At the moment, <description> is used for very short descriptions:
<description>Arcon Overlay!</description>
or quite longer:
<description>A collection of Bazaar-related ebuilds, including release candidates of Bazaar.</description>
even ery long:
<description>Berkano overlay provides an overlay for Gentoo Linux with
ebuilds of bleeding edge software (such as live subversion builds) and other handy tools that are missing from the official portage tree. Areas of interest include multimedia and system administration.</description>

layman is a cli so it doesn't really care of the description lenght but
it's a pain to use in a GUI like PackageKit repository list. I think we
should introduce a <longdescription> field even with a lang parameter
like we already have in metadata.xml. So, description field will have to
be used for short descriptions (a few words) to complete the name of the
overlay like:
<description>experimental gnome ebuilds</description>

I don't think DTD have abilities to restrict field value lenght but even
if we will have to change a few <description> by hands, having a
<longdescription> field could help.

Sebastian, do you think you can add this to your set of changes ? (only
the new field in the DTD at the moment).

Thanks,
Mounir

Re: [gentoo-dev] [rfc] layman-global.txt, repositories.xml, layman, overlays.gentoo.org

[Sebastian Pipping]

volkmar@gentoo.org wrote:
> I think we
> should introduce a <longdescription> field even with a lang parameter
> like we already have in metadata.xml.

Done.
http://git.goodpoint.de/?p=overlays-xml-specification.git;a=commit;h=c13a394fe1a868012548b2be5fb58359b3bc2891



Sebastian

Re: [gentoo-dev] [rfc] layman-global.txt, repositories.xml, layman, overlays.gentoo.org

[Tiziano Müller]

[-- Attachment #1.1: Type: text/plain, Size: 1311 bytes --]

Am Montag, den 28.09.2009, 20:23 +0200 schrieb Sebastian Pipping:
> repositories.xml
> =====================================================================
>   <repo
>       name="sping"
>       quality="experimental"
>       status="unofficial">
>     <description>Gentoo overlay of Sebastian Pipping</description>
>     <homepage>http://git.goodpoint.de/?p=overlay-sping.git</homepage>
>     <owner type="person">
>       <email>sebastian@pipping.org</email>
>       <name>Sebastian Pipping</name>
>     </owner>
>     <source type="git">git://git.goodpoint.de/overlay-sping.git</source>
>     <feed>http://git.goodpoint.de/?p=overlay-sping.git.git;a=atom</feed>
>   </repo>
> =====================================================================

What is the reason that "name" is an attribute? While quality, status
and type have a distinct set of allowed values, name doesn't and I'd
therefore set it as an element instead.

How about adding an attribute "lang" to <description> to be able to give
descriptions in different languages?

Cheers,
Tiziano

-- 
Tiziano Müller
Gentoo Linux Developer
Areas of responsibility:
  Samba, PostgreSQL, CPP, Python, sysadmin, GLEP Editor
E-Mail   : dev-zero@gentoo.org
GnuPG FP : F327 283A E769 2E36 18D5  4DE2 1B05 6A63 AE9C 1E30

[-- Attachment #1.2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 3551 bytes --]

Re: [gentoo-dev] [rfc] layman-global.txt, repositories.xml, layman, overlays.gentoo.org

[Sebastian Pipping]

Tiziano Müller wrote:
> Am Montag, den 28.09.2009, 20:23 +0200 schrieb Sebastian Pipping:
>> repositories.xml
>> =====================================================================
>>   <repo
>>       name="sping"
>>       quality="experimental"
>>       status="unofficial">
>>     <description>Gentoo overlay of Sebastian Pipping</description>
>>     <homepage>http://git.goodpoint.de/?p=overlay-sping.git</homepage>
>>     <owner type="person">
>>       <email>sebastian@pipping.org</email>
>>       <name>Sebastian Pipping</name>
>>     </owner>
>>     <source type="git">git://git.goodpoint.de/overlay-sping.git</source>
>>     <feed>http://git.goodpoint.de/?p=overlay-sping.git.git;a=atom</feed>
>>   </repo>
>> =====================================================================
> 
> What is the reason that "name" is an attribute? While quality, status
> and type have a distinct set of allowed values, name doesn't and I'd
> therefore set it as an element instead.

I don't see value in that change and I like name as it is.


> How about adding an attribute "lang" to <description> to be able to give
> descriptions in different languages?

Done.
http://git.goodpoint.de/?p=overlays-xml-specification.git;a=commitdiff;h=f7d068d772aa9961338f78d4b91b4783f43ff40f



Sebastian

Re: [gentoo-dev] [rfc] layman-global.txt, repositories.xml, layman, overlays.gentoo.org

[Fabian Groffen]

On 30-09-2009 17:36:47 +0200, Sebastian Pipping wrote:
> Tiziano Müller wrote:
> > Am Montag, den 28.09.2009, 20:23 +0200 schrieb Sebastian Pipping:
> >> repositories.xml
> >> =====================================================================
> >>   <repo
> >>       name="sping"
> >>       quality="experimental"
> >>       status="unofficial">
> >>     <description>Gentoo overlay of Sebastian Pipping</description>
> >>     <homepage>http://git.goodpoint.de/?p=overlay-sping.git</homepage>
> >>     <owner type="person">
> >>       <email>sebastian@pipping.org</email>
> >>       <name>Sebastian Pipping</name>
> >>     </owner>
> >>     <source type="git">git://git.goodpoint.de/overlay-sping.git</source>
> >>     <feed>http://git.goodpoint.de/?p=overlay-sping.git.git;a=atom</feed>
> >>   </repo>
> >> =====================================================================
> > 
> > What is the reason that "name" is an attribute? While quality, status
> > and type have a distinct set of allowed values, name doesn't and I'd
> > therefore set it as an element instead.
> 
> I don't see value in that change and I like name as it is.

Point remains that it looks in-consistant, for repo, name is an
attribute, while for owner it is a sub-element.  Why having attributes
in the first place anyway?
It's XML, so it may be extremely annoying to parse/use.


-- 
Fabian Groffen
Gentoo on a different level

Re: [gentoo-dev] [rfc] layman-global.txt, repositories.xml, layman, overlays.gentoo.org

[Sebastian Pipping]

Fabian Groffen wrote:
> Point remains that it looks in-consistant, for repo, name is an
> attribute, while for owner it is a sub-element.  Why having attributes
> in the first place anyway?

It's closer to the original layman-global.txt which also makes the
converter scripts simpler (and faster) than the element approach.  I
think it's just right.


> It's XML, so it may be extremely annoying to parse/use.

Please elaborate on that.



Sebastian

Re: [gentoo-dev] [rfc] layman-global.txt, repositories.xml, layman, overlays.gentoo.org

[Tiziano Müller]

[-- Attachment #1.1: Type: text/plain, Size: 909 bytes --]

Am Mittwoch, den 30.09.2009, 17:57 +0200 schrieb Sebastian Pipping:
> Fabian Groffen wrote:
> > Point remains that it looks in-consistant, for repo, name is an
> > attribute, while for owner it is a sub-element.  Why having attributes
> > in the first place anyway?
> 
> It's closer to the original layman-global.txt which also makes the
> converter scripts simpler (and faster) than the element approach.  I
> think it's just right.
Then you're doing something wrong in your converter script because that
would be a simple change.
A simple rule of thumb is: use attributes for values with predefined
contents, use elements otherwise. So, make "name" an element please.

-- 
Tiziano Müller
Gentoo Linux Developer
Areas of responsibility:
  Samba, PostgreSQL, CPP, Python, sysadmin, GLEP Editor
E-Mail   : dev-zero@gentoo.org
GnuPG FP : F327 283A E769 2E36 18D5  4DE2 1B05 6A63 AE9C 1E30

[-- Attachment #1.2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 3551 bytes --]

Re: [gentoo-dev] [rfc] layman-global.txt, repositories.xml, layman, overlays.gentoo.org

[Sebastian Pipping]

Tiziano Müller wrote:
> A simple rule of thumb is: use attributes for values with predefined
> contents, use elements otherwise. So, make "name" an element please.

Okay, let's give it a try.

http://git.goodpoint.de/?p=overlays-xml-specification.git;a=commitdiff;h=8b10c5c7fd5b42bb5540fd686e0d97f5d11ad841



Sebastian

Re: [gentoo-dev] [rfc] layman-global.txt, repositories.xml, layman, overlays.gentoo.org

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 883 bytes --]

On Mon, 28 Sep 2009 20:23:34 +0200
Sebastian Pipping <webmaster@hartwork.org> wrote:
> Now please ask questions and let us know what you think.

Here's an alternative idea:

* Move the repository information into the overlays themselves. Require
  overlays to provide a file containing the description, homepage,
  owner information etc.

* After a certain amount of time, switch new versions of layman to use a
  new file that's generated from these. Any overlay that doesn't
  contain such a file is clearly unmaintained and should no longer be
  supported.

* Also use this opportunity to clean up the repo_name vs layman name
  mess: any repository that gets it wrong should not be included in the
  new layman file.

Sounds like a perfect opportunity to make sure that everything in
layman's still actively maintained and developed.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] [rfc] layman-global.txt, repositories.xml, layman, overlays.gentoo.org

[Sebastian Pipping]

Ciaran McCreesh wrote:
> On Mon, 28 Sep 2009 20:23:34 +0200
> Sebastian Pipping <webmaster@hartwork.org> wrote:
>> Now please ask questions and let us know what you think.
> 
> Here's an alternative idea:
> 
> * Move the repository information into the overlays themselves. Require
>   overlays to provide a file containing the description, homepage,
>   owner information etc.

I have created another script yesterday that can auto-merge information
from gitosis.conf into repositories.xml.  With that script in a Git hook
setting up new Git-based Gentoo-hosted overlays requires changes at only
an single place and propagates to repositories.xml and
layman-global.txt.  (The only extension needed to make it work is adding
a line "gentoo-is-overlay = True" to Git repos that are overlays in
gitosis.conf.  The "gentoo-" prefix is meant as a namespace and gitosis
upstream confirmed that this works without interfering with gitosis itself.)

At the moment moving overlay meta info into the overlays does not seem
like a good idea to.  It would mean that any script working with
overlays needs to check the repo out to get to that data.  That doesn't
sound like fun to me.


> Sounds like a perfect opportunity to make sure that everything in
> layman's still actively maintained and developed.

Let me second that.



Sebastian

Re: [gentoo-dev] [rfc] layman-global.txt, repositories.xml, layman, overlays.gentoo.org

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 529 bytes --]

On Wed, 30 Sep 2009 17:51:02 +0200
Sebastian Pipping <webmaster@hartwork.org> wrote:
> At the moment moving overlay meta info into the overlays does not seem
> like a good idea to.  It would mean that any script working with
> overlays needs to check the repo out to get to that data.  That
> doesn't sound like fun to me.

No, you can copy the data into your master file. It's just that the
original source of the data would be the repository, not the person
adding the repository to the list.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] [rfc] layman-global.txt, repositories.xml, layman, overlays.gentoo.org

[Sebastian Pipping]

Ciaran McCreesh wrote:
> On Wed, 30 Sep 2009 17:51:02 +0200
> Sebastian Pipping <webmaster@hartwork.org> wrote:
>> At the moment moving overlay meta info into the overlays does not seem
>> like a good idea to.  It would mean that any script working with
>> overlays needs to check the repo out to get to that data.  That
>> doesn't sound like fun to me.
> 
> No, you can copy the data into your master file. It's just that the
> original source of the data would be the repository, not the person
> adding the repository to the list.

If it can change, it needs to be kept in sync.  I really don't think it
belongs in there.



Sebastian

Re: [gentoo-dev] [rfc] layman-global.txt, repositories.xml, layman, overlays.gentoo.org

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1050 bytes --]

On Wed, 30 Sep 2009 18:00:13 +0200
Sebastian Pipping <webmaster@hartwork.org> wrote:
> Ciaran McCreesh wrote:
> > On Wed, 30 Sep 2009 17:51:02 +0200
> > Sebastian Pipping <webmaster@hartwork.org> wrote:
> >> At the moment moving overlay meta info into the overlays does not
> >> seem like a good idea to.  It would mean that any script working
> >> with overlays needs to check the repo out to get to that data.
> >> That doesn't sound like fun to me.
> > 
> > No, you can copy the data into your master file. It's just that the
> > original source of the data would be the repository, not the person
> > adding the repository to the list.
> 
> If it can change, it needs to be kept in sync.  I really don't think
> it belongs in there.

Sure. Just periodically fetch the repository centrally. Have a master
list of sync URLs with expected repository names, and use that to
generate the full master list that includes metadata.

Added bonus: you can quickly remove any repository that no longer
exists.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] [rfc] layman-global.txt, repositories.xml, layman, overlays.gentoo.org

[Sebastian Pipping]

Ciaran McCreesh wrote:
> Sure. Just periodically fetch the repository centrally. Have a master
> list of sync URLs with expected repository names, and use that to
> generate the full master list that includes metadata.
> 
> Added bonus: you can quickly remove any repository that no longer
> exists.

How long do you want the time frame for add-meta-data-or-get-kicked
to be?  If half the repos don't not make it will kicking them help
anybody?  What if that metadata format changes later or is extended by
additional required entries?



Sebastian

Re: [gentoo-dev] [rfc] layman-global.txt, repositories.xml, layman, overlays.gentoo.org

[Tiziano Müller]

[-- Attachment #1.1: Type: text/plain, Size: 1348 bytes --]

Am Mittwoch, den 30.09.2009, 18:17 +0200 schrieb Sebastian Pipping:
> Ciaran McCreesh wrote:
> > Sure. Just periodically fetch the repository centrally. Have a master
> > list of sync URLs with expected repository names, and use that to
> > generate the full master list that includes metadata.
> > 
> > Added bonus: you can quickly remove any repository that no longer
> > exists.
> 
> How long do you want the time frame for add-meta-data-or-get-kicked
> to be?  If half the repos don't not make it will kicking them help
> anybody?
Yes, then they're not maintained. And unmaintained overlays tend to
contain even more broken ebuilds than others.

>   What if that metadata format changes later or is extended by
> additional required entries?
How about using the power of xml and version the schemas? As long as the
metadata-file specifies the respective dtd/xsd/relaxng you know exactly
how to validate (and parse) it and may apply a xsl-trafo if necessary to
convert it to a new format on the fly. As long as you keep using xml you
can then change the complete format in a new schema version.


-- 
Tiziano Müller
Gentoo Linux Developer
Areas of responsibility:
  Samba, PostgreSQL, CPP, Python, sysadmin, GLEP Editor
E-Mail   : dev-zero@gentoo.org
GnuPG FP : F327 283A E769 2E36 18D5  4DE2 1B05 6A63 AE9C 1E30

[-- Attachment #1.2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 3551 bytes --]

Re: [gentoo-dev] [rfc] layman-global.txt, repositories.xml, layman, overlays.gentoo.org

[Robert Buchholz]

[-- Attachment #1: Type: text/plain, Size: 1146 bytes --]

On Thursday 01 October 2009, Tiziano Müller wrote:
> Am Mittwoch, den 30.09.2009, 18:17 +0200 schrieb Sebastian Pipping:
> > Ciaran McCreesh wrote:
> > > Sure. Just periodically fetch the repository centrally. Have a
> > > master list of sync URLs with expected repository names, and use
> > > that to generate the full master list that includes metadata.
> > >
> > > Added bonus: you can quickly remove any repository that no longer
> > > exists.
> >
> > How long do you want the time frame for add-meta-data-or-get-kicked
> > to be?  If half the repos don't not make it will kicking them help
> > anybody?
>
> Yes, then they're not maintained. And unmaintained overlays tend to
> contain even more broken ebuilds than others.

There are plenty of unmaintained overlays. Keeping them in our index 
allows everyone to use what is still usable, and develoers to pick up 
maintenance. Removing them from the index makes it harder to do this.

I guess what you want to do is keep unsuspecting users safe from these 
overlays, and the quality levels (i.e. graveyard in this case) are 
there to serve this purpose.


Robert

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] [rfc] layman-global.txt, repositories.xml, layman, overlays.gentoo.org

[Sebastian Pipping]

Tiziano Müller wrote:
>>   What if that metadata format changes later or is extended by
>> additional required entries?
> How about using the power of xml and version the schemas? As long as the
> metadata-file specifies the respective dtd/xsd/relaxng you know exactly
> how to validate (and parse) it and may apply a xsl-trafo if necessary to
> convert it to a new format on the fly. As long as you keep using xml you
> can then change the complete format in a new schema version.

We are version the schemas already.  My point is that we would need to
wait for the overlay maintainers once again to update their metadata.
Transforming does not save us as we are missing new data.


Sebastian

Re: [gentoo-dev] [rfc] layman-global.txt, repositories.xml, layman, overlays.gentoo.org

[Sebastian Pipping]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Sebastian Pipping wrote:
> I have created another script yesterday that can auto-merge information
> from gitosis.conf into repositories.xml.  With that script in a Git hook
> setting up new Git-based Gentoo-hosted overlays requires changes at only
> an single place and propagates to repositories.xml and
> layman-global.txt.

A post-update Git hook seems to be working well, code here:

http://git.goodpoint.de/?p=overlays-xml-specification.git;a=blob;f=post-update.sample.gitosis-conf-merge


I have sent a patch for layman to Gunnar already.  Let's see if he likes it.



Sebastian

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkrGmz8ACgkQsAvGakAaFgDmNQCgq6IkOXgRC3SAMpUkleBwPYer
xboAn2gijL3U0fbsylFIfoe280vpO48i
=uM1k
-----END PGP SIGNATURE-----