[gentoo-dev] GLEP 67 is in, please update your metadata.dtd!

[gentoo-dev] GLEP 67 is in, please update your metadata.dtd!

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1821 bytes --]

Hello, everyone.

I've finished the GLEP 67 transition last night, and it officially
applies to all metadata.xml files now.

In order to have repoman apply it correct (and not throw errors on new
metadata.xml files), it needs to refetch metadata.dtd. Sadly, this
currently happens once a week, so it's better to remove the file
manually to force refetch:

  rm "$(portageq envvar DISTDIR)"/metadata.dtd

Please note that the metadata.dtd is unable to correctly enforce
obligatory type="" attribute on maintainers. Please make sure to add
it. I will be adding pkgcheck/repoman checks for that soon.

If you'd like to convert your own repository, you can reuse the herdfix
tool [1]. It's pretty dumb, and can work with any repository:

  git clone https://bitbucket.org/mgorny/herdfix/
  cd herdfix
  # note: this will fetch all data, including ssh-ing to d.g.o
  # to get developer list from /etc/passwd
  make
  ./01-pre-clean.py /repo/path
  ./03-clean-maint-needed.py /repo/path
  ./02-herdfix.py /repo/path herd-mapping.json
  ./04-set-type.py /repo/path projects.xml dev.names

01-pre-clean.py is optional and useful only if you want to separately
commit quoting changes to avoid mixing them in with other changes.

03-clean-maint-needed.py is probably unnecessary since it just updates
maintainer-needed@g.o packages. I doubt you find them often in custom
repositories.

02-herdfix.py is the meat grinder replacing herds with projects. Useful
only if you used <herd/> in your repository (which you shouldn't have
been).

04-set-type.py sets type="" on all <maintainer/>s. It uses
type="project" for anything found in projects.xml, type="person"
otherwise.

[1]:https://bitbucket.org/mgorny/herdfix/

-- 
Best regards,
Michał Górny
<http://dev.gentoo.org/~mgorny/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] GLEP 67 is in, please update your metadata.dtd!

[Andrew Savchenko]

[-- Attachment #1: Type: text/plain, Size: 621 bytes --]

On Mon, 25 Jan 2016 10:37:15 +0100 Michał Górny wrote:
> Hello, everyone.
> 
> I've finished the GLEP 67 transition last night, and it officially
> applies to all metadata.xml files now.
> 
> In order to have repoman apply it correct (and not throw errors on new
> metadata.xml files), it needs to refetch metadata.dtd. Sadly, this
> currently happens once a week, so it's better to remove the file
> manually to force refetch:

skel.metadata.xml should be updated as well, otherwise developers
will have broken example.

That said, skel.ChangeLog should be removed :)

Best regards,
Andrew Savchenko

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] GLEP 67 is in, please update your metadata.dtd!

[Luis Ressel]

On Mon, 25 Jan 2016 10:37:15 +0100
Michał Górny <mgorny@gentoo.org> wrote:

> Hello, everyone.
> 
> I've finished the GLEP 67 transition last night, and it officially
> applies to all metadata.xml files now.
> 

Great!

> In order to have repoman apply it correct (and not throw errors on new
> metadata.xml files), it needs to refetch metadata.dtd. Sadly, this
> currently happens once a week, so it's better to remove the file
> manually to force refetch:
> 
>   rm "$(portageq envvar DISTDIR)"/metadata.dtd
> 

I might be asking this for a second time, but why does repoman download
the metadata.dtd at all? If one fetches from
git://../gentoo-mirror/gentoo (or via rsync, afaik) it is included
in /usr/portage/metadata/dtd/.

For me, it's in fact very annoying when repoman wants to download
metatdata.dtd, as my "normal" unix user isn't a member of the portage
group.

By the way, the herds.xml file is still available at
https://api.gentoo.org/packages/herds.xml and can probably be removed
from there as well.


-- 
Regards,
Luis Ressel

Re: [gentoo-dev] GLEP 67 is in, please update your metadata.dtd!

[Mike Gilbert]

On Mon, Jan 25, 2016 at 11:31 AM, Luis Ressel <aranea@aixah.de> wrote:
> On Mon, 25 Jan 2016 10:37:15 +0100
> Michał Górny <mgorny@gentoo.org> wrote:
>
>> Hello, everyone.
>>
>> I've finished the GLEP 67 transition last night, and it officially
>> applies to all metadata.xml files now.
>>
>
> Great!
>
>> In order to have repoman apply it correct (and not throw errors on new
>> metadata.xml files), it needs to refetch metadata.dtd. Sadly, this
>> currently happens once a week, so it's better to remove the file
>> manually to force refetch:
>>
>>   rm "$(portageq envvar DISTDIR)"/metadata.dtd
>>
>
> I might be asking this for a second time, but why does repoman download
> the metadata.dtd at all? If one fetches from
> git://../gentoo-mirror/gentoo (or via rsync, afaik) it is included
> in /usr/portage/metadata/dtd/.

repoman is a developer tool. The git repo used by developers does not
include metadata.dtd.

[gentoo-dev] Re: GLEP 67 is in, please update your metadata.dtd!

[Martin Vaeth]

Mike Gilbert <floppym@gentoo.org> wrote:
> On Mon, Jan 25, 2016 at 11:31 AM, Luis Ressel <aranea@aixah.de> wrote:
>>
>> I might be asking this for a second time, but why does repoman download
>> the metadata.dtd at all? If one fetches from
>> git://../gentoo-mirror/gentoo (or via rsync, afaik) it is included
>> in /usr/portage/metadata/dtd/.
>
> repoman is a developer tool. The git repo used by developers does not
> include metadata.dtd.

repoman is useful for everybody who manages a local
or public repository. I guess only a small number
of these people use the developers' git repo.
(For instance, I am one of these...)

It would be nice if repoman would first check whether
the file is in some ${repo}/metadata/dtd/
and download to DISTDIR only as a fallback.
Or if it can be at least configured (e.g. with an option)
to check for this file (maybe the path can be configured, too)
before downloading metadata.dtd.

In fact, I was so annoyed by the forced downloading
(which as mentioned is a permission/security issue)
that I regularly apply a patch to portage which sets
the download interval to (almost) infinity.
Perhaps I am not the only one who did such a hack.

Re: [gentoo-dev] Re: GLEP 67 is in, please update your metadata.dtd!

[Alec Warner]

[-- Attachment #1: Type: text/plain, Size: 1343 bytes --]

On Mon, Jan 25, 2016 at 10:45 AM, Martin Vaeth <martin@mvath.de> wrote:

> Mike Gilbert <floppym@gentoo.org> wrote:
> > On Mon, Jan 25, 2016 at 11:31 AM, Luis Ressel <aranea@aixah.de> wrote:
> >>
> >> I might be asking this for a second time, but why does repoman download
> >> the metadata.dtd at all? If one fetches from
> >> git://../gentoo-mirror/gentoo (or via rsync, afaik) it is included
> >> in /usr/portage/metadata/dtd/.
> >
> > repoman is a developer tool. The git repo used by developers does not
> > include metadata.dtd.
>
> repoman is useful for everybody who manages a local
> or public repository. I guess only a small number
> of these people use the developers' git repo.
> (For instance, I am one of these...)
>
> It would be nice if repoman would first check whether
> the file is in some ${repo}/metadata/dtd/
> and download to DISTDIR only as a fallback.
> Or if it can be at least configured (e.g. with an option)
> to check for this file (maybe the path can be configured, too)
> before downloading metadata.dtd.
>
> In fact, I was so annoyed by the forced downloading
> (which as mentioned is a permission/security issue)
> that I regularly apply a patch to portage which sets
> the download interval to (almost) infinity.
> Perhaps I am not the only one who did such a hack.
>
>
>

So submit a patch to portage?

-A

[-- Attachment #2: Type: text/html, Size: 2066 bytes --]

Re: [gentoo-dev] Re: GLEP 67 is in, please update your metadata.dtd!

[Mike Gilbert]

On Mon, Jan 25, 2016 at 1:45 PM, Martin Vaeth <martin@mvath.de> wrote:
> Mike Gilbert <floppym@gentoo.org> wrote:
>> On Mon, Jan 25, 2016 at 11:31 AM, Luis Ressel <aranea@aixah.de> wrote:
>>>
>>> I might be asking this for a second time, but why does repoman download
>>> the metadata.dtd at all? If one fetches from
>>> git://../gentoo-mirror/gentoo (or via rsync, afaik) it is included
>>> in /usr/portage/metadata/dtd/.
>>
>> repoman is a developer tool. The git repo used by developers does not
>> include metadata.dtd.
>
> repoman is useful for everybody who manages a local
> or public repository. I guess only a small number
> of these people use the developers' git repo.
> (For instance, I am one of these...)
>
> It would be nice if repoman would first check whether
> the file is in some ${repo}/metadata/dtd/
> and download to DISTDIR only as a fallback.
> Or if it can be at least configured (e.g. with an option)
> to check for this file (maybe the path can be configured, too)
> before downloading metadata.dtd.

I agree that this would be a nice enhancement.

[gentoo-dev] Re: [gentoo-dev-announce] GLEP 67 is in, please update your metadata.dtd!

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1097 bytes --]

Hello again,

On Mon, 25 Jan 2016 10:37:15 +0100
Michał Górny <mgorny@gentoo.org> wrote:

> Please note that the metadata.dtd is unable to correctly enforce
> obligatory type="" attribute on maintainers. Please make sure to add
> it. I will be adding pkgcheck/repoman checks for that soon.

As requested by a number of developers, I have reverted the DTD change
blocking <herd/> elements in order to provide additional transition
period.

The support for <herd/>, as well as herds.xml and herds.dtd, will be
removed no earlier than 2 weeks from now. Please use that time to
update metadata.xml in your repositories.

While at it, I've added three GLEP 67-related checks to gentoo-ci.
Therefore, the metadata.xml files in Gentoo repository and pull
requests will be periodically checked for the following problems:

1. <herd/> elements,

2. <maintainer/> elements with no type="" (invalid is already covered
by the DTD),

3. <maintainer type="project"/> referencing projects not
in projects.xml.

-- 
Best regards,
Michał Górny
<http://dev.gentoo.org/~mgorny/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]