[gentoo-dev] metadata.xml un<herd/>-ization, v2

[gentoo-dev] metadata.xml un<herd/>-ization, v2

[Michał Górny]

[-- Attachment #1.1: Type: text/plain, Size: 1439 bytes --]

Hello, all.

So considering the previous thread, the Council and QA discussions, I
have prepared a new version of the metadata.xml update. To hopefully
make everyone happy, I come with this three-step process:

1. Add type="" attribute to <maintainer/> tag (see attached patch),

2. Convert <herd/> to <maintainer type="herd"/>,

3. Eventually drop <herd/> from DTD.

If you like the idea, I'll prepare a smart conversion script soon.

As for the exact details, I've pretty much decided to go for featurism
here, IOW making everyone happy. It also proves how absurd typing
maintainers is but if you really feel like having it, sure. The default
is 'developer', <herd/> tags would be converted into 'herd' and there
are other options including 'proxy-maintainer', 'project', 'team' meant
to fit all our wannabies. The diff explains the particular options.

The main benefit of this project over other ideas is that it preserves
backwards compatibility. We're adding a new attribute which should
simply be ignored by old tools. Since we still require <email/> to be
something valid, the output will change a bit but will still be
meaningful (or even more meaningful in some cases).

And since I removed <herd>no-herd</herd> some time ago, we can drop
<herd/> tags without worrying.

Note: this is just about metadata.xml, I'm not touching herds.xml.

What do you think?

-- 
Best regards,
Michał Górny

[-- Attachment #1.2: metadata-maint-type.patch --]
[-- Type: text/plain, Size: 1087 bytes --]

? .git
Index: metadata.dtd
===================================================================
RCS file: /var/cvsroot/gentoo/xml/htdocs/dtd/metadata.dtd,v
retrieving revision 1.13
diff -u -B -r1.13 metadata.dtd
--- metadata.dtd	9 May 2013 06:58:55 -0000	1.13
+++ metadata.dtd	8 Dec 2014 23:38:09 -0000
@@ -13,6 +13,14 @@
 
   <!-- One tag for each maintainer of a package, multiple allowed-->
   <!ELEMENT maintainer ( email, (description| name)* )>
+    <!-- maintainer organizational type -->
+    <!-- developer: regular Gentoo developer (direct e-mail) -->
+    <!-- herd: herd (defined in herds.xml) -->
+    <!-- project: project (having Wiki/g.o project page) -->
+    <!-- proxy-maintainer: maintainer that is not a Gentoo developer
+      and commits via a proxy -->
+    <!-- team: team of people that is not a herd nor a project -->
+    <!ATTLIST maintainer type (developer|herd|project|proxy-maintainer|team) "developer" >
 
   <!-- Natural name for package, example: LibreOffice (for app-office/libreoffice) -->
   <!ELEMENT natural-name (#PCDATA) >

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] metadata.xml un<herd/>-ization, v2

[Michał Górny]

[-- Attachment #1.1: Type: text/plain, Size: 740 bytes --]

Dnia 2014-12-09, o godz. 00:46:28
Michał Górny <mgorny@gentoo.org> napisał(a):

> Hello, all.
> 
> So considering the previous thread, the Council and QA discussions, I
> have prepared a new version of the metadata.xml update. To hopefully
> make everyone happy, I come with this three-step process:
> 
> 1. Add type="" attribute to <maintainer/> tag (see attached patch),
> 
> 2. Convert <herd/> to <maintainer type="herd"/>,

If anyone cares, attaching my script doing the conversion. It tries
hard to preserve the indentation used in metadata.xml. However, it does
not preserve ' vs " in attribute names.

The diff can be seen @
http://dev.gentoo.org/~mgorny/tmp/herds.diff.xz

-- 
Best regards,
Michał Górny

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.2: herdfix.py --]
[-- Type: text/x-python, Size: 3020 bytes --]

#!/usr/bin/env python

from collections import namedtuple
import errno
import glob
from lxml.builder import E
import lxml.etree
import os
import os.path

def main():
	herdtuple = namedtuple('herdtuple', ('email', 'name'))
	herddb = {}
	portdir = '/var/db/repos/gentoo'
	herdsfile = os.path.join(portdir, 'metadata/herds.xml')
	herdsxml = lxml.etree.parse(herdsfile)
	for h in herdsxml.getroot():
		k = h.find('name').text
		e = h.find('email').text
		d = h.find('description').text
		herddb[k] = herdtuple(e, d)

	intree = portdir
	outtree = '/tmp/1'

	# LAZINESS!
	for f in glob.glob(os.path.join(intree, '*/*/metadata.xml')):
		subpath = os.path.relpath(f, intree)
		print(subpath)
		outf = os.path.join(outtree, subpath)

		xml = lxml.etree.parse(f)
		herds = xml.getroot().findall('herd')
		if not herds: # yay, one file less to care about
			continue
		r = xml.getroot()
		maints = r.findall('maintainer')
		if maints:
			insertpoint = maints[-1]
		else:
			insertpoint = herds[-1]

		# try to guess indentation
		def all_texts(node):
			first = True
			for e in node:
				if first:
					yield node.text
					first = False
				yield e.tail
		def all_indents(node):
			for t in all_texts(node):
				if t is None:
					yield ''
					return
				spl = t.split('\n')
				# go to last line without text
				for l in spl:
					if l.lstrip(' \t') != '':
						break
				# go to the last line
				t = l[:len(l) - len(l.lstrip(' \t'))]
				yield t
		def sub_indents(node):
			for e in node:
				for x in all_indents(e):
					yield x


		# some random defaults
		indent = '\t'
		try:
			indent = max(all_indents(r), key=len)
		except ValueError:
			pass

		inner_indent = indent*2 if indent else '\t'
		try:
			inner_indent = max(sub_indents(r), key=len)
		except ValueError:
			pass

		# start adding new herds after maintainers
		for h in herds:
			he = herddb[h.text.strip()]
			attrs = dict(h.items())
			attrs['type'] = 'herd'
			nm = E.maintainer('\n',
				inner_indent, E.email(he.email), '\n',
				inner_indent, E.name(he.name), '\n',
				indent,
				**attrs
			)
			nextinsert = insertpoint.getnext()
			nm.tail = insertpoint.tail
			if nextinsert is not None:
				r.insert(r.index(nextinsert), nm)
			else:
				# avoid extra indent
				nm.tail = '\n'
				r.append(nm)
			insertpoint = nm

			# now we can remove it safely
			r.remove(h)

			# now fix pre-indent
			prev = nm.getprevious()
			if prev is not None:
				prev.tail = '\n' + indent
			else:
				nm.getparent().text = '\n' + indent

		try:
			os.makedirs(os.path.dirname(outf))
		except OSError as e:
			if e.errno != errno.EEXIST:
				raise
		try:
			os.unlink(outf)
		except OSError as e:
			if e.errno != errno.ENOENT:
				raise
		xml.write(outf, encoding='UTF-8', xml_declaration='1.0')
		# yay, add trailing newline because lxml is dumb
		with open(outf, 'ab') as f:
			f.write(b'\n')

if __name__ == '__main__':
	main()

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] metadata.xml un<herd/>-ization, v2

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 1582 bytes --]

>>>>> On Tue, 9 Dec 2014, Michał Górny wrote:

> As for the exact details, I've pretty much decided to go for
> featurism here, IOW making everyone happy. It also proves how absurd
> typing maintainers is but if you really feel like having it, sure.
> The default is 'developer', <herd/> tags would be converted into
> 'herd' and there are other options including 'proxy-maintainer',
> 'project', 'team' meant to fit all our wannabies. The diff explains
> the particular options.

>    <!ELEMENT maintainer ( email, (description| name)* )>
> +    <!-- maintainer organizational type -->
> +    <!-- developer: regular Gentoo developer (direct e-mail) -->
> +    <!-- herd: herd (defined in herds.xml) -->

As the previously stated goal was to get rid of herds, I don't
understand why you want to reintroduce them as a value of the
type attribute. The existing herd elements should become either
type="project" or type="team" (everything that is not a project,
I suppose).

> +    <!-- project: project (having Wiki/g.o project page) -->
> +    <!-- proxy-maintainer: maintainer that is not a Gentoo developer
> +      and commits via a proxy -->

"proxy-maintainer" is very confusing because you won't put the proxy
maintainer there, but the user who is being proxied. Please rename
to something like "proxied" (assuming that this exists as a word in
English) or "by-proxy".

> +    <!-- team: team of people that is not a herd nor a project -->
> +    <!ATTLIST maintainer type (developer|herd|project|proxy-maintainer|team) "developer" >

Ulrich

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] metadata.xml un<herd/>-ization, v2

[Jeroen Roovers]

On Tue, 9 Dec 2014 12:59:26 +0100
Ulrich Mueller <ulm@gentoo.org> wrote:

> >>>>> On Tue, 9 Dec 2014, Michał Górny wrote:
> 
> > As for the exact details, I've pretty much decided to go for
> > featurism here, IOW making everyone happy. It also proves how absurd
> > typing maintainers is but if you really feel like having it, sure.
> > The default is 'developer', <herd/> tags would be converted into
> > 'herd' and there are other options including 'proxy-maintainer',
> > 'project', 'team' meant to fit all our wannabies. The diff explains
> > the particular options.
> 
> >    <!ELEMENT maintainer ( email, (description| name)* )>
> > +    <!-- maintainer organizational type -->
> > +    <!-- developer: regular Gentoo developer (direct e-mail) -->
> > +    <!-- herd: herd (defined in herds.xml) -->
> 
> As the previously stated goal was to get rid of herds, I don't
> understand why you want to reintroduce them as a value of the
> type attribute. The existing herd elements should become either
> type="project" or type="team" (everything that is not a project,
> I suppose).

Probably because of the rather verbose criticism brought forward by a
single developer.


     jer

Re: [gentoo-dev] metadata.xml un<herd/>-ization, v2

[Andreas K. Huettel]

[-- Attachment #1: Type: text/plain, Size: 657 bytes --]

Am Dienstag 09 Dezember 2014, 00:46:28 schrieb Michał Górny:
> 2. Convert <herd/> to <maintainer type="herd"/>,

Could any <person type="developer"/> please explain to me why this additional 
xmlfication is of any <improvement type="nonzero"/>?

As far as I know the typical way to edit metadata.xml files for all of us is 
still by hand in a text editor, and I prefer a lot typing

<herd>kde</herd>

over

<maintainer type="herd">kde</maintainer>

Cheers,
Andreas

PS. This is a mere syntactical point, and not taking side in any way 
for/against herds. Separate mail.

-- 
Andreas K. Huettel
Gentoo Linux developer
kde, council

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 951 bytes --]

Re: [gentoo-dev] metadata.xml un<herd/>-ization, v2

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1552 bytes --]

Dnia 2014-12-09, o godz. 16:49:24
"Andreas K. Huettel" <dilfridge@gentoo.org> napisał(a):

> Am Dienstag 09 Dezember 2014, 00:46:28 schrieb Michał Górny:
> > 2. Convert <herd/> to <maintainer type="herd"/>,
> 
> Could any <person type="developer"/> please explain to me why this additional 
> xmlfication is of any <improvement type="nonzero"/>?
> 
> As far as I know the typical way to edit metadata.xml files for all of us is 
> still by hand in a text editor, and I prefer a lot typing
> 
> <herd>kde</herd>
> 
> over
> 
> <maintainer type="herd">kde</maintainer>

<maintainer type="herd">
  <email>kde@gentoo.org</email>
  <name>Lovely KDE herd</name>
</maintainer>

to be more precise.

This was already explained in my previous thread. Because:

1. valid <email/> is much more useful than semi-ambiguous <herd/>,

2. bug assignment in order is simpler than magical rules like
'maintainer first, herd second unless description says otherwise',

3. herds.xml is global, metadata.xml is per-repo.

Many of us *read* metadata.xml via cat or simple text editor, and don't
want to be forced to use slow tools like 'equery' to make
semi-meaningful output out of it. And I believe reading happens much
more often than writing.

That said, vim starts with useful template for metadata.xml. It's easy
to copy <maintainer/> element and fill in different e-mail addresses.
I'd honestly drop the whole type="" but a few developers insist on
keeping the extra disambiguation.

-- 
Best regards,
Michał Górny

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] metadata.xml un<herd/>-ization, v2

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1807 bytes --]

Dnia 2014-12-09, o godz. 12:59:26
Ulrich Mueller <ulm@gentoo.org> napisał(a):

> >>>>> On Tue, 9 Dec 2014, Michał Górny wrote:
> 
> > As for the exact details, I've pretty much decided to go for
> > featurism here, IOW making everyone happy. It also proves how absurd
> > typing maintainers is but if you really feel like having it, sure.
> > The default is 'developer', <herd/> tags would be converted into
> > 'herd' and there are other options including 'proxy-maintainer',
> > 'project', 'team' meant to fit all our wannabies. The diff explains
> > the particular options.
> 
> >    <!ELEMENT maintainer ( email, (description| name)* )>
> > +    <!-- maintainer organizational type -->
> > +    <!-- developer: regular Gentoo developer (direct e-mail) -->
> > +    <!-- herd: herd (defined in herds.xml) -->
> 
> As the previously stated goal was to get rid of herds, I don't
> understand why you want to reintroduce them as a value of the
> type attribute. The existing herd elements should become either
> type="project" or type="team" (everything that is not a project,
> I suppose).

As I said, I don't care what final values are. I added a lot of options
to make people happy. As far as I'm concerned, the whole type="" can go
away.

> > +    <!-- project: project (having Wiki/g.o project page) -->
> > +    <!-- proxy-maintainer: maintainer that is not a Gentoo developer
> > +      and commits via a proxy -->
> 
> "proxy-maintainer" is very confusing because you won't put the proxy
> maintainer there, but the user who is being proxied. Please rename
> to something like "proxied" (assuming that this exists as a word in
> English) or "by-proxy".

Because the whole naming in proxy-maintainership is confusing :).

-- 
Best regards,
Michał Górny

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: Re: [gentoo-dev] metadata.xml un<herd/>-ization, v2

[Andreas K. Huettel]

[-- Attachment #1: Type: text/plain, Size: 1664 bytes --]


> > As far as I know the typical way to edit metadata.xml files for all of us
> > is still by hand in a text editor, and I prefer a lot typing
> > 
> > <herd>kde</herd>
> > 
> > over
> > 
> > <maintainer type="herd">kde</maintainer>
> 
> <maintainer type="herd">
>   <email>kde@gentoo.org</email>
>   <name>Lovely KDE herd</name>
> </maintainer>
> 
> to be more precise.

Ugh. Even more ugly stuff from the department of redundancy department.

(Yeah usually I just copy an existing file too...)

Keep it simple. The only result of stuffing more and more requirements into 
auxiliary files is that noone will fill them out completely.

> 1. valid <email/> is much more useful than semi-ambiguous <herd/>,
> 
> 2. bug assignment in order is simpler than magical rules like
> 'maintainer first, herd second unless description says otherwise',
> 
> 3. herds.xml is global, metadata.xml is per-repo.
> 
> Many of us *read* metadata.xml via cat or simple text editor, and don't
> want to be forced to use slow tools like 'equery' to make
> semi-meaningful output out of it.

So the solution is to duplicate all information into every package directory. 
Great. 

(And if I want to send an e-mail to the postgresql team I still have to figure 
out how their alias looks like or find a package that they maintain first.)

Why not instead come up with a new set of rules (and maybe new tags) that 
simplify? 
As example, scratch <herd/>, add <project/> with a 1:1 transformation of 
argument to mail address.
As example, always assign bugs to first entry.



-- 
Andreas K. Huettel
Gentoo Linux developer
kde, council

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 951 bytes --]

Re: [gentoo-dev] metadata.xml un<herd/>-ization, v2

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 2684 bytes --]

Dnia 2014-12-09, o godz. 17:17:35
"Andreas K. Huettel" <dilfridge@gentoo.org> napisał(a):

> 
> > > As far as I know the typical way to edit metadata.xml files for all of us
> > > is still by hand in a text editor, and I prefer a lot typing
> > > 
> > > <herd>kde</herd>
> > > 
> > > over
> > > 
> > > <maintainer type="herd">kde</maintainer>
> > 
> > <maintainer type="herd">
> >   <email>kde@gentoo.org</email>
> >   <name>Lovely KDE herd</name>
> > </maintainer>
> > 
> > to be more precise.
> 
> Ugh. Even more ugly stuff from the department of redundancy department.
> 
> (Yeah usually I just copy an existing file too...)
> 
> Keep it simple. The only result of stuffing more and more requirements into 
> auxiliary files is that noone will fill them out completely.

I'm all for keeping it simple. However, backwards compatibility makes
it hard to keep things simple. I'd love to do, say, metadata.yml
supporting stuff like:

- maintainer: foo@gentoo.org, bar@gentoo.org

- maintainer:
  - name: Foo Bar
    email: foo@gentoo.org
  - bar@gentoo.org

(pseudo-code, not sure if it's 100% valid YAML)

But that's another topic, and a lot of work to support, and most
of the tools will never be fixed to support it.

> > 1. valid <email/> is much more useful than semi-ambiguous <herd/>,
> > 
> > 2. bug assignment in order is simpler than magical rules like
> > 'maintainer first, herd second unless description says otherwise',
> > 
> > 3. herds.xml is global, metadata.xml is per-repo.
> > 
> > Many of us *read* metadata.xml via cat or simple text editor, and don't
> > want to be forced to use slow tools like 'equery' to make
> > semi-meaningful output out of it.
> 
> So the solution is to duplicate all information into every package directory. 
> Great. 

Just the e-mail.

> (And if I want to send an e-mail to the postgresql team I still have to figure 
> out how their alias looks like or find a package that they maintain first.)

How about just using the e-mail? You don't CC all the team members, do
you?

> Why not instead come up with a new set of rules (and maybe new tags) that 
> simplify? 
> As example, scratch <herd/>, add <project/> with a 1:1 transformation of 
> argument to mail address.
> As example, always assign bugs to first entry.

That is *precisely the goal* of this. We scratch <herd/> and <project/>,
and just use well-supported <maintainer/> tag. type="" is entirely
optional, unobtrusive and backwards-compatible with existing apps.
And yes, with just <maintainer/> tag the rule in bug assignment is
always to the first entry.

-- 
Best regards,
Michał Górny

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] metadata.xml un<herd/>-ization, v2

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 649 bytes --]

>>>>> On Tue, 9 Dec 2014, Michał Górny wrote:

> That is *precisely the goal* of this. We scratch <herd/> and
> <project/>, and just use well-supported <maintainer/> tag. type=""
> is entirely optional, unobtrusive and backwards-compatible with
> existing apps. And yes, with just <maintainer/> tag the rule in bug
> assignment is always to the first entry.

This still means that you either
- duplicate in every package dir information that is also maintained
  elsewhere, like mapping from herd or project name to its e-mail
  address, or
- need to do reverse mapping, e.g. from the e-mail address to the
  project's name.

Ulrich

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] metadata.xml un<herd/>-ization, v2

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1071 bytes --]

Dnia 2014-12-09, o godz. 18:59:22
Ulrich Mueller <ulm@gentoo.org> napisał(a):

> >>>>> On Tue, 9 Dec 2014, Michał Górny wrote:
> 
> > That is *precisely the goal* of this. We scratch <herd/> and
> > <project/>, and just use well-supported <maintainer/> tag. type=""
> > is entirely optional, unobtrusive and backwards-compatible with
> > existing apps. And yes, with just <maintainer/> tag the rule in bug
> > assignment is always to the first entry.
> 
> This still means that you either
> - duplicate in every package dir information that is also maintained
>   elsewhere, like mapping from herd or project name to its e-mail
>   address, or
> - need to do reverse mapping, e.g. from the e-mail address to the
>   project's name.

Sure. But usually the e-mail address is more useful than the project
name. Plus, it's unique so we can use it to identify projects without
having to really care about names, project page addresses etc. E-mail
is the common denominator between all different maintainer types.

-- 
Best regards,
Michał Górny

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] metadata.xml un<herd/>-ization, v2

[Michał Górny]

[-- Attachment #1.1: Type: text/plain, Size: 1035 bytes --]

Dnia 2014-12-09, o godz. 10:15:09
Michał Górny <mgorny@gentoo.org> napisał(a):

> Dnia 2014-12-09, o godz. 00:46:28
> Michał Górny <mgorny@gentoo.org> napisał(a):
> 
> > Hello, all.
> > 
> > So considering the previous thread, the Council and QA discussions, I
> > have prepared a new version of the metadata.xml update. To hopefully
> > make everyone happy, I come with this three-step process:
> > 
> > 1. Add type="" attribute to <maintainer/> tag (see attached patch),
> > 
> > 2. Convert <herd/> to <maintainer type="herd"/>,
> 
> If anyone cares, attaching my script doing the conversion. It tries
> hard to preserve the indentation used in metadata.xml. However, it does
> not preserve ' vs " in attribute names.
> 
> The diff can be seen @
> http://dev.gentoo.org/~mgorny/tmp/herds.diff.xz

And a small update to the script attached, that avoids adding duplicate
<maintainer/> tags when herd's already in <maintainer/> as well. Linked
diff updated as well.

-- 
Best regards,
Michał Górny

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.2: herdfix.py --]
[-- Type: text/x-python, Size: 3227 bytes --]

#!/usr/bin/env python

from collections import namedtuple
import errno
import glob
from lxml.builder import E
import lxml.etree
import os
import os.path

def main():
	herdtuple = namedtuple('herdtuple', ('email', 'name'))
	herddb = {}
	portdir = '/var/db/repos/gentoo'
	herdsfile = os.path.join(portdir, 'metadata/herds.xml')
	herdsxml = lxml.etree.parse(herdsfile)
	for h in herdsxml.getroot():
		k = h.find('name').text
		e = h.find('email').text
		d = h.find('description').text
		herddb[k] = herdtuple(e, d)

	intree = portdir
	outtree = '/tmp/1'

	# LAZINESS!
	for f in glob.glob(os.path.join(intree, '*/*/metadata.xml')):
		subpath = os.path.relpath(f, intree)
		print(subpath)
		outf = os.path.join(outtree, subpath)

		xml = lxml.etree.parse(f)
		herds = xml.getroot().findall('herd')
		if not herds: # yay, one file less to care about
			continue
		r = xml.getroot()
		maints = r.findall('maintainer')
		if maints:
			insertpoint = maints[-1]
		else:
			insertpoint = herds[-1]

		# try to guess indentation
		def all_texts(node):
			first = True
			for e in node:
				if first:
					yield node.text
					first = False
				yield e.tail
		def all_indents(node):
			for t in all_texts(node):
				if t is None:
					yield ''
					return
				spl = t.split('\n')
				# go to last line without text
				for l in spl:
					if l.lstrip(' \t') != '':
						break
				# go to the last line
				t = l[:len(l) - len(l.lstrip(' \t'))]
				yield t
		def sub_indents(node):
			for e in node:
				for x in all_indents(e):
					yield x


		# some random defaults
		indent = '\t'
		try:
			indent = max(all_indents(r), key=len)
		except ValueError:
			pass

		inner_indent = indent*2 if indent else '\t'
		try:
			inner_indent = max(sub_indents(r), key=len)
		except ValueError:
			pass

		# start adding new herds after maintainers
		for h in herds:
			he = herddb[h.text.strip()]

			# look for duplicate <herd/> entries
			for m in maints:
				if m.find('email').text.strip() == he.email:
					m.set('type', 'herd')
					r.remove(h)
					break
			else:
				attrs = dict(h.items())
				attrs['type'] = 'herd'
				nm = E.maintainer('\n',
					inner_indent, E.email(he.email), '\n',
					inner_indent, E.name(he.name), '\n',
					indent,
					**attrs
				)
				nextinsert = insertpoint.getnext()
				nm.tail = insertpoint.tail
				if nextinsert is not None:
					r.insert(r.index(nextinsert), nm)
				else:
					# avoid extra indent
					nm.tail = '\n'
					r.append(nm)
				insertpoint = nm

				# now we can remove it safely
				r.remove(h)

				# now fix pre-indent
				prev = nm.getprevious()
				if prev is not None:
					prev.tail = '\n' + indent
				else:
					nm.getparent().text = '\n' + indent

		try:
			os.makedirs(os.path.dirname(outf))
		except OSError as e:
			if e.errno != errno.EEXIST:
				raise
		try:
			os.unlink(outf)
		except OSError as e:
			if e.errno != errno.ENOENT:
				raise
		xml.write(outf, encoding='UTF-8', xml_declaration='1.0')
		# yay, add trailing newline because lxml is dumb
		with open(outf, 'ab') as f:
			f.write(b'\n')

if __name__ == '__main__':
	main()

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] metadata.xml un<herd/>-ization, v2

[Luca Barbato]

On 09/12/14 17:34, Michał Górny wrote:

> I'm all for keeping it simple. However, backwards compatibility makes
> it hard to keep things simple. I'd love to do, say, metadata.yml
> supporting stuff like:
>
> - maintainer: foo@gentoo.org, bar@gentoo.org
>
> - maintainer:
>    - name: Foo Bar
>      email: foo@gentoo.org
>    - bar@gentoo.org
>
> (pseudo-code, not sure if it's 100% valid YAML)
>

Would be neat though.

Back to the discussion would be nice to have just project@gentoo.org 
instead of complex mappings.

lu

Re: [gentoo-dev] metadata.xml un<herd/>-ization, v2

[Michał Górny]

[-- Attachment #1.1: Type: text/plain, Size: 1283 bytes --]

Dnia 2014-12-09, o godz. 00:46:28
Michał Górny <mgorny@gentoo.org> napisał(a):

> So considering the previous thread, the Council and QA discussions, I
> have prepared a new version of the metadata.xml update. To hopefully
> make everyone happy, I come with this three-step process:

And the Council meeting brought a bit of update to this. More
specifically:

1. type="" is now limited to developer, project and team,
corresponding respectively to a single person, a GLEP 39 project (with
webpage) and any other kind of group of people [attached].

2. I've also updated the script to use the new syntax and stop adding
<description/> to make things smaller. The herd descriptions are pretty
bad-suited for maintainer anyway.

3. herds.xml is to be deprecated as well, with existing herds being
replaced by projects (GLEP 39) or teams.

4. We eventually want to migrate to another (simpler, more compact)
metadata format, preferably YAML. The exact format is yet to be specced
(for example, I have no good ideas of doing restrict="".

Does anyone have any more comments or should I proceed with updating
the tree for 1+2?

Oh, and as usual the current diff is at:
http://dev.gentoo.org/~mgorny/tmp/herds.diff.xz

-- 
Best regards,
Michał Górny

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.2: metadata.dtd.diff --]
[-- Type: text/x-patch, Size: 831 bytes --]

Index: metadata.dtd
===================================================================
RCS file: /var/cvsroot/gentoo/xml/htdocs/dtd/metadata.dtd,v
retrieving revision 1.13
diff -u -B -r1.13 metadata.dtd
--- metadata.dtd	9 May 2013 06:58:55 -0000	1.13
+++ metadata.dtd	9 Dec 2014 21:28:03 -0000
@@ -13,6 +13,11 @@
 
   <!-- One tag for each maintainer of a package, multiple allowed-->
   <!ELEMENT maintainer ( email, (description| name)* )>
+    <!-- maintainer organizational type -->
+    <!-- developer: regular person -->
+    <!-- project: GLEP 39 project -->
+    <!-- team: team that is not a project -->
+    <!ATTLIST maintainer type (developer|project|team) "developer" >
 
   <!-- Natural name for package, example: LibreOffice (for app-office/libreoffice) -->
   <!ELEMENT natural-name (#PCDATA) >

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.3: herdfix.py --]
[-- Type: text/x-python, Size: 3184 bytes --]

#!/usr/bin/env python

from collections import namedtuple
import errno
import glob
from lxml.builder import E
import lxml.etree
import os
import os.path

def main():
	herdtuple = namedtuple('herdtuple', ('email', 'name'))
	herddb = {}
	portdir = '/var/db/repos/gentoo'
	herdsfile = os.path.join(portdir, 'metadata/herds.xml')
	herdsxml = lxml.etree.parse(herdsfile)
	for h in herdsxml.getroot():
		k = h.find('name').text
		e = h.find('email').text
		d = h.find('description').text
		herddb[k] = herdtuple(e, d)

	intree = portdir
	outtree = '/tmp/1'

	# LAZINESS!
	for f in glob.glob(os.path.join(intree, '*/*/metadata.xml')):
		subpath = os.path.relpath(f, intree)
		print(subpath)
		outf = os.path.join(outtree, subpath)

		xml = lxml.etree.parse(f)
		herds = xml.getroot().findall('herd')
		if not herds: # yay, one file less to care about
			continue
		r = xml.getroot()
		maints = r.findall('maintainer')
		if maints:
			insertpoint = maints[-1]
		else:
			insertpoint = herds[-1]

		# try to guess indentation
		def all_texts(node):
			first = True
			for e in node:
				if first:
					yield node.text
					first = False
				yield e.tail
		def all_indents(node):
			for t in all_texts(node):
				if t is None:
					yield ''
					return
				spl = t.split('\n')
				# go to last line without text
				for l in spl:
					if l.lstrip(' \t') != '':
						break
				# go to the last line
				t = l[:len(l) - len(l.lstrip(' \t'))]
				yield t
		def sub_indents(node):
			for e in node:
				for x in all_indents(e):
					yield x


		# some random defaults
		indent = '\t'
		try:
			indent = max(all_indents(r), key=len)
		except ValueError:
			pass

		inner_indent = indent*2 if indent else '\t'
		try:
			inner_indent = max(sub_indents(r), key=len)
		except ValueError:
			pass

		# start adding new herds after maintainers
		for h in herds:
			he = herddb[h.text.strip()]

			# look for duplicate <herd/> entries
			for m in maints:
				if m.find('email').text.strip() == he.email:
					m.set('type', 'herd')
					r.remove(h)
					break
			else:
				attrs = dict(h.items())
				attrs['type'] = 'team'
				nm = E.maintainer('\n',
					inner_indent, E.email(he.email), '\n',
					indent,
					**attrs
				)
				nextinsert = insertpoint.getnext()
				nm.tail = insertpoint.tail
				if nextinsert is not None:
					r.insert(r.index(nextinsert), nm)
				else:
					# avoid extra indent
					nm.tail = '\n'
					r.append(nm)
				insertpoint = nm

				# now we can remove it safely
				r.remove(h)

				# now fix pre-indent
				prev = nm.getprevious()
				if prev is not None:
					prev.tail = '\n' + indent
				else:
					nm.getparent().text = '\n' + indent

		try:
			os.makedirs(os.path.dirname(outf))
		except OSError as e:
			if e.errno != errno.EEXIST:
				raise
		try:
			os.unlink(outf)
		except OSError as e:
			if e.errno != errno.ENOENT:
				raise
		xml.write(outf, encoding='UTF-8', xml_declaration='1.0')
		# yay, add trailing newline because lxml is dumb
		with open(outf, 'ab') as f:
			f.write(b'\n')

if __name__ == '__main__':
	main()

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] metadata.xml un<herd/>-ization, v2

[Rich Freeman]

On Tue, Dec 9, 2014 at 11:04 AM, Michał Górny <mgorny@gentoo.org> wrote:
> Dnia 2014-12-09, o godz. 12:59:26
> Ulrich Mueller <ulm@gentoo.org> napisał(a):
>
>>
>> As the previously stated goal was to get rid of herds, I don't
>> understand why you want to reintroduce them as a value of the
>> type attribute. The existing herd elements should become either
>> type="project" or type="team" (everything that is not a project,
>> I suppose).
>
> As I said, I don't care what final values are. I added a lot of options
> to make people happy. As far as I'm concerned, the whole type="" can go
> away.

I thought we were generally agreed we wanted to get rid of herds.  The
goal wasn't to rename them, but to get rid of them.

We could have email aliases for bugs so that people can sign up for
notifications, but they would NOT be considered maintainers.  Of
course, any would be welcome to become actual maintainers, but as far
as treecleaning/etc goes the package is unmaintained.

If we just rename "herd" to "team" then we have the same issue where
nobody can tell if anybody is taking care of anything because it all
goes into some nebulous bin full of packages where nobody is
responsible for anything in particular, and nobody can speak for the
"team" because it isn't really a team.

How about "contact" instead of team.  A package could have any number
of contacts, and they just get CC'ed on bugs, and there is no meaning
to a contact besides being CC'ed on bugs.  They're never assignees -
if there is nobody else in metadata besides a contact then the
assignee is maintainer-wanted.

--
Rich

Re: [gentoo-dev] metadata.xml un<herd/>-ization, v2

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 1229 bytes --]

>>>>> On Tue, 9 Dec 2014, Rich Freeman wrote:

> I thought we were generally agreed we wanted to get rid of herds.
> The goal wasn't to rename them, but to get rid of them.

> We could have email aliases for bugs so that people can sign up for
> notifications, but they would NOT be considered maintainers.  Of
> course, any would be welcome to become actual maintainers, but as
> far as treecleaning/etc goes the package is unmaintained.

> If we just rename "herd" to "team" then we have the same issue where
> nobody can tell if anybody is taking care of anything because it all
> goes into some nebulous bin full of packages where nobody is
> responsible for anything in particular, and nobody can speak for the
> "team" because it isn't really a team.

> How about "contact" instead of team.  A package could have any
> number of contacts, and they just get CC'ed on bugs, and there is no
> meaning to a contact besides being CC'ed on bugs.  They're never
> assignees - if there is nobody else in metadata besides a contact
> then the assignee is maintainer-wanted.

Now sure it I get this, so can you explain with a concrete example?
Let's say, for a package that currently has <herd>xemacs</herd> in its
metadata.

Ulrich

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] metadata.xml un<herd/>-ization, v2

[Sergey Popov]

[-- Attachment #1: Type: text/plain, Size: 576 bytes --]

09.12.2014 14:59, Ulrich Mueller пишет:
> "proxy-maintainer" is very confusing because you won't put the proxy
> maintainer there, but the user who is being proxied. Please rename
> to something like "proxied" (assuming that this exists as a word in
> English) or "by-proxy".

+1 for that.

Proxy maintainer is a Gentoo developer, that commits changes, authored
by proxied maintainer, who does not have commit access to main tree

-- 
Best regards, Sergey Popov
Gentoo developer
Gentoo Desktop Effects project lead
Gentoo Proxy maintainers project lead


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 538 bytes --]

Re: [gentoo-dev] metadata.xml un<herd/>-ization, v2

[Rich Freeman]

On Wed, Dec 10, 2014 at 2:06 AM, Ulrich Mueller <ulm@gentoo.org> wrote:
>>>>>> On Tue, 9 Dec 2014, Rich Freeman wrote:
>
>> I thought we were generally agreed we wanted to get rid of herds.
>> The goal wasn't to rename them, but to get rid of them.
>
>> We could have email aliases for bugs so that people can sign up for
>> notifications, but they would NOT be considered maintainers.  Of
>> course, any would be welcome to become actual maintainers, but as
>> far as treecleaning/etc goes the package is unmaintained.
>
>> If we just rename "herd" to "team" then we have the same issue where
>> nobody can tell if anybody is taking care of anything because it all
>> goes into some nebulous bin full of packages where nobody is
>> responsible for anything in particular, and nobody can speak for the
>> "team" because it isn't really a team.
>
>> How about "contact" instead of team.  A package could have any
>> number of contacts, and they just get CC'ed on bugs, and there is no
>> meaning to a contact besides being CC'ed on bugs.  They're never
>> assignees - if there is nobody else in metadata besides a contact
>> then the assignee is maintainer-wanted.
>
> Now sure it I get this, so can you explain with a concrete example?
> Let's say, for a package that currently has <herd>xemacs</herd> in its
> metadata.
>

That would depend on whether xemacs became a project or not.  The
first part of my proposal [1] was to review the list of herds and
decide which ones were going to become projects, and then review the
list of packages and let developers sign up to maintain packages that
didn't have a non-herd maintainer.

So, if xemacs herd wasn't going to become a project, and nobody signed
up to maintain it, then in your example xemacs@g.o would become a
contact and the package would be assigned to maintainer-needed.

If xemacs decided to become an active project then it would become a
project and would be considered maintained.

If xemacs decided not to become a project but one or more developers
or projects stepped up to maintain it, then xemacs would become a
contact and the maintainers who added themselves to metadata would
become the maintainers.

If you're not actually going to fix the herd problem, then rather than
renaming "herds" to "teams" you might as well leave the broken herds
in place so that somebody else can actually fix them later.  :)

1 - http://article.gmane.org/gmane.linux.gentoo.devel/93587/

--
Rich

[gentoo-dev] Re: metadata.xml un<herd/>-ization, v2

[Michael Palimaka]

On 10/12/14 06:35, Luca Barbato wrote:
> On 09/12/14 17:34, Michał Górny wrote:
> 
>> I'm all for keeping it simple. However, backwards compatibility makes
>> it hard to keep things simple. I'd love to do, say, metadata.yml
>> supporting stuff like:
>>
>> - maintainer: foo@gentoo.org, bar@gentoo.org
>>
>> - maintainer:
>>    - name: Foo Bar
>>      email: foo@gentoo.org
>>    - bar@gentoo.org
>>
>> (pseudo-code, not sure if it's 100% valid YAML)
>>
> 
> Would be neat though.
> 
> Back to the discussion would be nice to have just project@gentoo.org
> instead of complex mappings.

This would be by far the easiest solution. Some herds already have an
alias like this eg. freedesktop -> freedesktop-bugs. Much easier than
mass-editing every single metadata.xml with what amounts to a cosmetic
change.

Re: [gentoo-dev] Re: metadata.xml un<herd/>-ization, v2

[Rich Freeman]

On Thu, Dec 11, 2014 at 1:23 AM, Michael Palimaka <kensington@gentoo.org> wrote:
>
> This would be by far the easiest solution. Some herds already have an
> alias like this eg. freedesktop -> freedesktop-bugs. Much easier than
> mass-editing every single metadata.xml with what amounts to a cosmetic
> change.
>

I think we're not all on the same page about what we're trying to
accomplish.  I think we need to agree on that before we can really
agree on how to do it.

There are many goals here that I can see:
1.  Simplification of the xml schema so that we don't have so many
different kinds of tags with different rules for each.
2.  Simplification of how we track group (ie project/herd/alias/etc)
member lists so that they're not in 5 different places with 5
different ways of determining who is in a group.
3.  Avoiding having large groups of packages maintained by large
groups of devs where the reality is that many packages aren't
maintained by anybody but this is opaque.
4.  If not all of the emails associated with metadata are considered
true maintainers, making it easy to tell which ones are and aren't.

I don't think all of these have equal support, which is why we end up
debating different solutions (obviously a solution which addresses all
of these is going to be more intrusive than a solution that only
addresses some of these).

I've been a proponent of solving all of these, but perhaps it would
make more sense to start smaller than that.  The only catch is that if
we remove the distinction in metadata between
maintainers/proxies/projects/herds/etc then if that distinction
becomes more important in the future it becomes harder to tell which
ones are which.

Part of me is wondering if worrying about #3-4 is actually all that
productive.  Does it really matter if a package is maintained or not,
when it all comes down to it?  Does it make more sense to focus on
whether packages have serious problems?  Maybe if a package is
completely unmaintained it makes it easier for developers to drop in
and make changes without asking anybody about it first, versus logging
a bug, waiting for the maintainer to drop the ball, and then making
the change anyway.

--
Rich

Re: [gentoo-dev] metadata.xml un<herd/>-ization, v2

[Aaron W. Swenson]

[-- Attachment #1: Type: text/plain, Size: 1393 bytes --]

On 2014-12-10 10:41, Sergey Popov wrote:
> 09.12.2014 14:59, Ulrich Mueller пишет:
> > "proxy-maintainer" is very confusing because you won't put the proxy
> > maintainer there, but the user who is being proxied. Please rename
> > to something like "proxied" (assuming that this exists as a word in
> > English) or "by-proxy".
> 
> +1 for that.
> 
> Proxy maintainer is a Gentoo developer, that commits changes, authored
> by proxied maintainer, who does not have commit access to main tree

The Gentoo Dev that's committing the work isn't a maintainer, in
relation to the package. S/he is a committer. So...:

  Gentoo Dev doing the commit: Proxy Committer (or just Proxy)
  Person doing the work: Maintainer-By-Proxy.

Sure, the dev may do a bit of maintenance or cleanup before committing,
but the bulk of the work is done by the maintainer-by-proxy. However, we
can clear up a lot of confusion if we just change our terms slightly to
more strongly indicate the dev's relationship to a package.

And, "proxied" is a word, but it does not mean what we need it to mean. We
could make it mean what we want it to mean, but I think that's kind of
mean. You know what I mean?

-- 
Mr. Aaron W. Swenson
Gentoo Linux Developer
PostgreSQL Herd Bull
Email : titanofold@gentoo.org
GnuPG FP : 2C00 7719 4F85 FB07 A49C 0E31 5713 AA03 D1BB FDA0
GnuPG ID : D1BBFDA0

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 213 bytes --]

Re: [gentoo-dev] metadata.xml un<herd/>-ization, v2

[Tom Wijsman]

On Tue, 9 Dec 2014 16:23:57 +0100
Jeroen Roovers <jer@gentoo.org> wrote:

> On Tue, 9 Dec 2014 12:59:26 +0100
> Ulrich Mueller <ulm@gentoo.org> wrote:
> 
> > >>>>> On Tue, 9 Dec 2014, Michał Górny wrote:
> > 
> > > As for the exact details, I've pretty much decided to go for
> > > featurism here, IOW making everyone happy. It also proves how
> > > absurd typing maintainers is but if you really feel like having
> > > it, sure. The default is 'developer', <herd/> tags would be
> > > converted into 'herd' and there are other options including
> > > 'proxy-maintainer', 'project', 'team' meant to fit all our
> > > wannabies. The diff explains the particular options.
> > 
> > >    <!ELEMENT maintainer ( email, (description| name)* )>
> > > +    <!-- maintainer organizational type -->
> > > +    <!-- developer: regular Gentoo developer (direct e-mail) -->
> > > +    <!-- herd: herd (defined in herds.xml) -->
> > 
> > As the previously stated goal was to get rid of herds, I don't
> > understand why you want to reintroduce them as a value of the
> > type attribute. The existing herd elements should become either
> > type="project" or type="team" (everything that is not a project,
> > I suppose).
> 
> Probably because of the rather verbose criticism brought forward by a
> single developer.

Probably because of the rather verbose burden of proof by another one.

Re: [gentoo-dev] metadata.xml un<herd/>-ization, v2

[Peter Stuge]

Rich Freeman wrote:
> How about "contact" instead of team.
> there is no meaning to a contact besides being CC'ed on bugs.

Please simply call it cc then? :)


//Peter