[gentoo-dev] arch teams and better tools

[gentoo-dev] arch teams and better tools

["Paweł Hajdan, Jr."]

[-- Attachment #1: Type: text/plain, Size: 1067 bytes --]

Recently there was a discussion about better tools for arch teams, to
speed up stabilizations and make them less error-prone.

Here's my answer to that, still in very early development:
<http://git.overlays.gentoo.org/gitweb/?p=proj/arch-tools.git;a=summary>

You can just launch it like this:

./bugzilla-viewer.py --arch x86

It has more options, and can even try to do the stabilization in cvs
tree and capture repoman output (it has to be enabled from the
command-line, see --help).

I think the key thing for arch teams is batching. Compilation takes
time, viewing web pages has latency, so we should maximize the work that
a developer can do within his limited time. To do that, we should
prepare as much information as possible up-front (this is the goal of
bugzilla-viewer.py).

Then the developer can quickly process the list of bugs, and decide what
he's going to test. And then we can have tools for batch-testing,
batch-committing, and so on. Some of them already exist.

Patches welcome, and please let me know what you think!


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 194 bytes --]

Re: [gentoo-dev] arch teams and better tools

[Thomas Kahle]

[-- Attachment #1: Type: text/plain, Size: 2146 bytes --]

Hi

On 18:44 Sat 21 May     , "Paweł Hajdan, Jr." wrote:
> Recently there was a discussion about better tools for arch teams, to
> speed up stabilizations and make them less error-prone.
> 
> Here's my answer to that, still in very early development:
> <http://git.overlays.gentoo.org/gitweb/?p=proj/arch-tools.git;a=summary>
> 
> You can just launch it like this:
> 
> ./bugzilla-viewer.py --arch x86

I like it.  Funnily It digged up some stable bugs from the stone-age
that have been processed, but x86 was still on the CC-list.

> It has more options, and can even try to do the stabilization in cvs
> tree and capture repoman output (it has to be enabled from the
> command-line, see --help).

Have you seen app-portage/tatt ? I think there is a huge overlap between
your project and tatt.  tatt can already fetch bugs from bugzilla,
create USE-flags and reverse dependency, as well as cvs commit scripts.
You can use your own templates for the scripts, but tatt also provides
some.  The next thing would be to clean up a few bugs and do a release,
unfortunately I have not yet found the time for this.  I'd be happy to
give you access to the git-tree your you can just clone it on github.

https://github.com/tom111/tatt

I also use aliases like
x86bugs='bugz search --cc x86@gentoo.org -a x86@gentoo.org'
secbugs='x86bugs | grep security'

> I think the key thing for arch teams is batching. Compilation takes
> time, viewing web pages has latency, so we should maximize the work that
> a developer can do within his limited time. To do that, we should
> prepare as much information as possible up-front (this is the goal of
> bugzilla-viewer.py).

Regarding the work-order I intentionally try to be agnostic.  I do
security bugs first, and then work on the oldest bug.

> Then the developer can quickly process the list of bugs, and decide what
> he's going to test. And then we can have tools for batch-testing,
> batch-committing, and so on. Some of them already exist.
> 
> Patches welcome, and please let me know what you think!



-- 
Thomas Kahle
http://dev.gentoo.org/~tomka/

[-- Attachment #2: Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-dev] arch teams and better tools

[Markos Chandras]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 22/05/2011 09:13 ??, Thomas Kahle wrote:
> Hi
> 
> On 18:44 Sat 21 May     , "Pawe? Hajdan, Jr." wrote:
>> Recently there was a discussion about better tools for arch teams, to
>> speed up stabilizations and make them less error-prone.
>>
>> Here's my answer to that, still in very early development:
>> <http://git.overlays.gentoo.org/gitweb/?p=proj/arch-tools.git;a=summary>
>>
>> You can just launch it like this:
>>
>> ./bugzilla-viewer.py --arch x86
> 
> I like it.  Funnily It digged up some stable bugs from the stone-age
> that have been processed, but x86 was still on the CC-list.
> 
>> It has more options, and can even try to do the stabilization in cvs
>> tree and capture repoman output (it has to be enabled from the
>> command-line, see --help).
> 
> Have you seen app-portage/tatt ? I think there is a huge overlap between
> your project and tatt.  tatt can already fetch bugs from bugzilla,
> create USE-flags and reverse dependency, as well as cvs commit scripts.
> You can use your own templates for the scripts, but tatt also provides
> some.  The next thing would be to clean up a few bugs and do a release,
> unfortunately I have not yet found the time for this.  I'd be happy to
> give you access to the git-tree your you can just clone it on github.
> 
> https://github.com/tom111/tatt
> 
> I also use aliases like
> x86bugs='bugz search --cc x86@gentoo.org -a x86@gentoo.org'
> secbugs='x86bugs | grep security'
> 
>> I think the key thing for arch teams is batching. Compilation takes
>> time, viewing web pages has latency, so we should maximize the work that
>> a developer can do within his limited time. To do that, we should
>> prepare as much information as possible up-front (this is the goal of
>> bugzilla-viewer.py).
> 
> Regarding the work-order I intentionally try to be agnostic.  I do
> security bugs first, and then work on the oldest bug.
> 
>> Then the developer can quickly process the list of bugs, and decide what
>> he's going to test. And then we can have tools for batch-testing,
>> batch-committing, and so on. Some of them already exist.
>>
>> Patches welcome, and please let me know what you think!
> 
> 
> 
Thanks for sharing your scripts. I would prefer to have a single script
doing everything, instead of multiple scripts here and there. I would
also like to see a minimal webpage ( under
http://www.gentoo.org/proj/en/base/tools? ) which would describe how to
use these tools on daily basis to deal with the bug workload. That will
help the rest of us integrate your scripts to our workflow and be more
efficient.

Regards,
- -- 
Markos Chandras / Gentoo Linux Developer / Key ID: B4AFF2C2
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (GNU/Linux)

iQIcBAEBCgAGBQJN2NhoAAoJEPqDWhW0r/LCu0MP/AqRaW5FvHEIF2Ixb60atDj9
CJ/hCvFer8aWoo7ckkhWIQTlW7tYwRAgVj3H/rltx/olSEm7aMxqpH1W0VwrB8pU
8Vobec/lme5aBYIqbsnGyqyp2myZ8He53mFj5jb8zwC8J5k6dxJTO5VOzdsC5b5z
cL6mJImQjNuyeF5YjfGbXhMxDNIrAe9ckGepiQLXjTp2Hn0YOJqQEI04WMmusZ3w
9SLqJ2s5n8puLIbwKZ4cNvwjilMD4xBuBT6zzMZ4kFWZhgDjY0Cqo9mF9g+kcp0j
YiIvFS63+59007UKthwHAurg/8gzE6PLQnMG6hna031b+92RzFEeQVKW/W/AyOSS
a92WdbrThyyYsiokmNv6w1aq2UlDNKbbdqVzaNw/xtm0ry3CkbVy0YPalySAAzYs
UV7C8pdiwkJnhI+yibg6X19GLoPQvPPR9+3+pwqxoIhkfF9lIv2zBcHBbiRlteF/
9GJBN+jfRVwW1UDgAT6BAhv0UULaYg+ZPABWj77JKO4wGz/zqonYCzEC7tUeN30U
Yo8KQPrXDL5QxAVE7CSHFORRuXCSKdiCfWy9cbeszSLizHJBUB5NQa2UCxura5Ib
NaCCGMa0eI3EQFMzxsoEY0MNgkwXZGJk0cKXSR54YxG2tn1PXUV0jGWBunt3qd05
Vx7Dzd7svvyPRco45s3l
=S9dR
-----END PGP SIGNATURE-----

Re: [gentoo-dev] arch teams and better tools

["Paweł Hajdan, Jr."]

[-- Attachment #1: Type: text/plain, Size: 1060 bytes --]

On 5/22/11 11:33 AM, Markos Chandras wrote:
> Thanks for sharing your scripts. I would prefer to have a single
> script doing everything, instead of multiple scripts here and there.

Right, having to complete pieces from various sources can be
frustrating. I'm planning to make arch-tools a complete solution (i.e.
no need to fetch any additional scripts, and only a minimal setup), but
I think it'll consist of multiple scripts (probably the bugzilla viewer,
emerge launcher, and mass committer).

> I would also like to see a minimal webpage ( under 
> http://www.gentoo.org/proj/en/base/tools? ) which would describe how
> to use these tools on daily basis to deal with the bug workload. That
> will help the rest of us integrate your scripts to our workflow and
> be more efficient.

I'll think about that, and it's definitely a good idea. It's just still
early in development, but after the discussion about more automated
testing I decided to post what I have ("release early, release often")
to avoid possible duplication of work.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 194 bytes --]

Re: [gentoo-dev] arch teams and better tools

[Rich Freeman]

On Tue, May 24, 2011 at 3:39 AM, "Paweł Hajdan, Jr."
<phajdan.jr@gentoo.org> wrote:
> On 5/22/11 11:33 AM, Markos Chandras wrote:
>> I would also like to see a minimal webpage ( under
>> http://www.gentoo.org/proj/en/base/tools? ) which would describe how
>> to use these tools on daily basis to deal with the bug workload. That
>> will help the rest of us integrate your scripts to our workflow and
>> be more efficient.
>
> I'll think about that, and it's definitely a good idea. It's just still
> early in development, but after the discussion about more automated
> testing I decided to post what I have ("release early, release often")
> to avoid possible duplication of work.
>

A guide of some sort would definitely be appreciated.  If a few devs
have worked out a super-efficient way of doing stabilizations without
making mistakes it would be nice if everybody else had the option to
do the same without having to stumble around finding it.

Along the same lines - some kind of gentoo dev tips page might be
useful if anybody cared to start creating one.  I have a few
scripts/aliases that I've made use of (often passed along from
somebody else or snipped from a list).  Perhaps not every new
developer knows that they can run "qlist ${1} | xargs scanelf -L -n -q
-F '%n #F' | tr , ' ' | xargs qfile -C | sort -u" to get a starting
list of package dependencies.

Rich

Re: [gentoo-dev] arch teams and better tools

[Peter Volkov]

Hi!

В Вск, 22/05/2011 в 10:13 +0200, Thomas Kahle пишет:
> On 18:44 Sat 21 May     , "Paweł Hajdan, Jr." wrote:

> > Here's my answer to that, still in very early development:
> > <http://git.overlays.gentoo.org/gitweb/?p=proj/arch-tools.git;a=summary>

> Have you seen app-portage/tatt ? 
> https://github.com/tom111/tatt

It looks that quite similar functionality is required to open
stabilization bugs. It's really takes time to check that there is no
bugs opened in the package and all dependent libraries, then to copy all
maintainers and create list of packages with archs like:

cate-gory/library-ver  amd64 ppc ppc64 x86 
cate-gory/pkg-ver amd64 x86

Have anybody thought/programmed such tool? :)

--
Peter.

Re: [gentoo-dev] arch teams and better tools

[Nirbheek Chauhan]

On Mon, May 23, 2011 at 12:54 AM, Peter Volkov <pva@gentoo.org> wrote:
> Hi!
>
> В Вск, 22/05/2011 в 10:13 +0200, Thomas Kahle пишет:
>> On 18:44 Sat 21 May     , "Paweł Hajdan, Jr." wrote:
>
>> > Here's my answer to that, still in very early development:
>> > <http://git.overlays.gentoo.org/gitweb/?p=proj/arch-tools.git;a=summary>
>
>> Have you seen app-portage/tatt ?
>> https://github.com/tom111/tatt
>
> It looks that quite similar functionality is required to open
> stabilization bugs. It's really takes time to check that there is no
> bugs opened in the package and all dependent libraries, then to copy all
> maintainers and create list of packages with archs like:
>
> cate-gory/library-ver  amd64 ppc ppc64 x86
> cate-gory/pkg-ver amd64 x86
>
> Have anybody thought/programmed such tool? :)
>

One part of this, i.e. the generation of that list of packages with
arches, is done by a script that the gnome and gstreamer herds use:

http://git.overlays.gentoo.org/gitweb/?p=proj/gnome.git;a=blob;f=scripts/gen_archlist.py;hb=HEAD

Usage: ./gen_archlist.py <file with atoms>

Reading the script leads to eye-bleeds (it needs to be rewritten), but
it works quite well. Here's some example output:
http://bugs.gentoo.org/show_bug.cgi?id=368281

-- 
~Nirbheek Chauhan

Gentoo GNOME+Mozilla Team

Re: [gentoo-dev] arch teams and better tools

["Paweł Hajdan, Jr."]

[-- Attachment #1: Type: text/plain, Size: 1405 bytes --]

On 5/22/11 10:13 AM, Thomas Kahle wrote:
> I like it.  Funnily It digged up some stable bugs from the stone-age
> that have been processed, but x86 was still on the CC-list.

Thanks. Those old bugs being displayed was a bug, I fixed it.

> Have you seen app-portage/tatt ? I think there is a huge overlap between
> your project and tatt.

Yeah, the project was in fact inspired by tatt and the earlier gatt.

> tatt can already fetch bugs from bugzilla,
> create USE-flags and reverse dependency, as well as cvs commit scripts.
> You can use your own templates for the scripts, but tatt also provides
> some.

One of the main things I wanted was being able to see all bug comments,
bug dependencies, related bugs, and repoman output in the bugzilla
viewer. Can tatt do that?

I think I'm going to either re-use or copy tatt's parts responsible for
compiling and committing the changes. For now I just focused on the
first part of the workflow (identifying bugs to work on, and identifying
possible problems as early as possible).

> Regarding the work-order I intentionally try to be agnostic.  I do
> security bugs first, and then work on the oldest bug.

Right, but sometimes there are bugs with no input from the maintainer.
Another thing I'd like to do is to mark such a bug as ignored, or even
"ignored until the next update" (e.g. when someone replies on that bug).


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 194 bytes --]