[gentoo-dev] The Plethora of Patches

[gentoo-dev] The Plethora of Patches

[Andrew D Kirch]

It has become abundantly clear that distribution maintainers should have 
as few patches as possible.  Patches waste time due to duplicate work, 
resources (portage disk space and bandwidth), and as the Debian project 
recently found out after a major vulnerability was discovered in the 
OpenSSH packages (see http://www.milw0rm.com/exploits/6094, and 
http://www.securityfocus.com/bid/30276 among others), they can become a 
source of great embarrassment, and liability since they are not nearly 
so well audited as code in heavily used mainstream projects (an 
unintentional Cathedral if you will).  I therefore propose the following 
changes:

Patches in the metadata.xml should have some sort of status tracking for 
each patch, repoman should flag any that don't, and warn on any that 
have not been submitted upstream unless the status is signed off on by a 
herd leader (such as Gentoo specific patches). This would provide visual 
feedback for users and developers with regard to a pretty important 
metric on how successful Gentoo is at getting patches pushed back to 
developers.

Developers who consistantly clear a large quantity of patches upstream 
should also be recognized in the Gentoo Monthly Newsletter, and 
otherwise as appropriate.
Obviously the software needs to work, and therefore we need patches, but 
Gentoo has not done enough to date to get them pushed upstream.  Lets 
look at some cringeworthy statistics on outstanding patches. (NB these 
are only patches in portage, and not patches which don't meet portage's 
maximum size)

app-accessibility   48 app-admin          178
app-antivirus       10 app-arch           101
app-backup          55 app-benchmarks      20
app-cdr             58 app-crypt           90
app-dicts           28 app-doc             26
app-editors         90 app-emacs           51
app-emulation      186 app-forensics       21
app-i18n            77 app-laptop          23
app-misc           181 app-mobilephone     34
app-office          64 app-pda             50
app-portage         36 app-shells          91
app-text           334 app-vim             13
app-xemacs           4 dev-ada              1
dev-cpp             30 dev-db             141
dev-dotnet          27 dev-embedded        17
dev-games           27 dev-haskell         12
dev-java           264 dev-lang           313
dev-libs           391 dev-lisp           112
dev-ml              15 dev-perl            78
dev-php              6 dev-php5            11
dev-python         202 dev-ruby            63
dev-scheme          37 dev-tcltk           33
dev-tex             24 dev-tinyos           3
dev-util           328 distfiles           26
eclass              21 games-action        58
games-arcade        76 games-board         58
games-emulation     88 games-engines        8
games-fps           58 games-kids           9
games-misc          15 games-mud           19
games-puzzle        65 games-roguelike     26
games-rpg           15 games-server         7
games-simulation    14 games-sports        17
games-strategy      54 games-util          31
gnome-base          45 gnome-extra         60
gnustep-apps        22 gnustep-base         3
gnustep-libs         9 kde-base           146
kde-misc            52 mail-client         71
mail-filter         49 mail-mta            21
media-fonts          5 media-gfx          188
media-libs         494 media-plugins      273
media-radio          2 media-sound        411
media-tv            44 media-video        253
metadata            72 net-analyzer       213
net-dialup         121 net-dns             45
net-firewall        33 net-fs              47
net-ftp             76 net-im              91
net-irc             68 net-libs           111
net-mail           113 net-misc           428
net-nds             11 net-news            16
net-nntp            21 net-p2p             67
net-print           49 net-proxy           53
net-voip             9 net-wireless        89
net-www             14 net-zope             6
perl-core            2 rox-base            11
rox-extra            6 sci-astronomy       32
sci-biology         32 sci-calculators     31
sci-chemistry      104 sci-electronics     21

Re: [gentoo-dev] The Plethora of Patches

[Jeremy Olexa]

Andrew D Kirch wrote:
<snip all>

Good points, I take it that you have found a mentor and are becoming a 
dev to drive this project then?

-Jeremy

Re: [gentoo-dev] The Plethora of Patches

[Rémi Cardona]

Andrew D Kirch wrote:
> Obviously the software needs to work, and therefore we need patches, but
> Gentoo has not done enough to date to get them pushed upstream. Lets
> look at some cringeworthy statistics on outstanding patches.

Have you even _looked_ at the patches? Can you tell which ones are :
  - backports?
  - Gentoo-specific (for build issues)?
  - shared with other distros?

Did you count the patches we apply because upstream is either dead, 
unresponsive or downright wrong?

Yes, we have a lot of patches, but then again, we have a lot of open 
bugs too. I, for one, am much more worried about the latter.

Please back this up with _real_ statistics.

Thanks

Rémi

Re: [gentoo-dev] The Plethora of Patches

[Denis Dupeyron]

On Thu, Aug 14, 2008 at 4:17 AM, Andrew D Kirch <trelane@trelane.net> wrote:
[...]

Looks like you counted the number of files in the files/
subdirectories. Not all of these are patches. Also, you probably
forgot to count seds, as some of us use sed more than patches.

Oh, and like Jeremy was hinting, please contact QA. They need somebody like you.

Denis.

Re: [gentoo-dev] The Plethora of Patches

[Luca Barbato]

Andrew D Kirch wrote:
[...patches...]

Common practice is to work with upstream (if alive) and have the patches 
merged asap. Nothing _that_ strange IMHO.

lu

-- 

Luca Barbato
Gentoo Council Member
Gentoo/linux Gentoo/PPC
http://dev.gentoo.org/~lu_zero

Re: [gentoo-dev] The Plethora of Patches

[Jeremy Olexa]

On Thu, Aug 14, 2008 at 12:00 AM, Andrew D Kirch <trelane@trelane.net> wrote:
> Jeremy Olexa wrote:
>>
>> Andrew D Kirch wrote:
>> <snip all>
>>
>> Good points, I take it that you have found a mentor and are becoming a dev
>> to drive this project then?
>>
>> -Jeremy
>
> I've spoken in the past with both Elfyn McBratney, and Homer Parker. I have
> also submitted quite a few ebuilds via the bugs.gentoo.org system.  I have
> decided that I'm pretty much not willing to fight the uphill battle with the
> red tape that presently exists at Gentoo.  I found a problem, and wrote up a
> pretty sensible, and not difficult to implement solution for a problem that
> is both negatively affecting the code base Gentoo maintains, the quality of
> all FOSS code, and that has the potential to embarrass the Gentoo
> development community.
>
> Andrew

Ok, well.. If you have extra time available to write up a solution for
IMO, a non-issue, then you have time to devote to "dev stuff" - but,
if you don't want to become a dev then I would rather you didn't
anyway. Put it this way, maybe some of us have decided that our time
could be better used fixing Gentoo than fighting the uphill battle
with the red tape that presently exists at $UPSTREAM. It is a two-way
street for everyone in the FOSS world. Like I said, I like your
points. Personally, I try to submit everything upstream but I only
maintain very few ebuilds. I think this is common practice amongst
Gentoo devs.

List replies preferred. Have a good day,

-Jeremy

Re: [gentoo-dev] The Plethora of Patches

[Santiago M. Mola]

On Thu, Aug 14, 2008 at 4:17 AM, Andrew D Kirch <trelane@trelane.net> wrote:
>
> Patches in the metadata.xml should have some sort of status tracking for
> each patch, repoman should flag any that don't, and warn on any that have
> not been submitted upstream unless the status is signed off on by a herd
> leader (such as Gentoo specific patches). This would provide visual feedback
> for users and developers with regard to a pretty important metric on how
> successful Gentoo is at getting patches pushed back to developers.

It was proposed recently to add some standarized headers to all new
patches for maintenance purposes. Something like:

Source: patch by John Foo, backported from upstream, whatever.
Upstream: In revision 245, rejected, foo.
Reason: Build system sucks

I think that's all we need in order to know how were things when the
patch was added and if it needs to be pushed/tracked upstream, removed
in the next version of the package, etc.

Some of us already put these kind of headers, or at least an URL to
upstream bug or a meaningful source of info about the patch.

However, tracking the status of every patch since its inclusion in
portage until it's removed would be a huge work overhead... and I
doubt it's worthy.

Regards,
-- 
Santiago M. Mola
Jabber ID: cooldwind@gmail.com

Re: [gentoo-dev] The Plethora of Patches

[Mario Fetka]

Am Donnerstag, 14. August 2008 17:24:41 schrieb Santiago M. Mola:
> On Thu, Aug 14, 2008 at 4:17 AM, Andrew D Kirch <trelane@trelane.net> wrote:
> > Patches in the metadata.xml should have some sort of status tracking for
> > each patch, repoman should flag any that don't, and warn on any that have
> > not been submitted upstream unless the status is signed off on by a herd
> > leader (such as Gentoo specific patches). This would provide visual
> > feedback for users and developers with regard to a pretty important
> > metric on how successful Gentoo is at getting patches pushed back to
> > developers.
>
> It was proposed recently to add some standarized headers to all new
> patches for maintenance purposes. Something like:
>
> Source: patch by John Foo, backported from upstream, whatever.
> Upstream: In revision 245, rejected, foo.
> Reason: Build system sucks
>
> I think that's all we need in order to know how were things when the
> patch was added and if it needs to be pushed/tracked upstream, removed
> in the next version of the package, etc.
>
> Some of us already put these kind of headers, or at least an URL to
> upstream bug or a meaningful source of info about the patch.
>
> However, tracking the status of every patch since its inclusion in
> portage until it's removed would be a huge work overhead... and I
> doubt it's worthy.

i am using the lfs tool to create my patches:
http://www.linuxfromscratch.org/patches/downloads/MAINTAINER/lfspatch

it creates patches with patch version number:
irtrans-irserver-5.11.08-arm_remotes-1.patch


and the header it creates looks like this:
Submitted By: Mario Fetka (mario-fetka at gmx dot at) 
Date: 2008-07-18 
Initial Package Version: 5.11.08 
Origin: me 
Upstream Status: unknown 
Description: add back remotes and correct makefile arm dir location

i think some rules for patches would be a good thing.
i would also suggest naming rules for the patches 

Mario

Re: [gentoo-dev] The Plethora of Patches

[Andrew D Kirch]

Rémi Cardona wrote:
> Andrew D Kirch wrote:
>> Obviously the software needs to work, and therefore we need patches, but
>> Gentoo has not done enough to date to get them pushed upstream. Lets
>> look at some cringeworthy statistics on outstanding patches.
>
> Have you even _looked_ at the patches? Can you tell which ones are :
>  - backports?
>  - Gentoo-specific (for build issues)?
>  - shared with other distros?
>
> Did you count the patches we apply because upstream is either dead, 
> unresponsive or downright wrong?
>
> Yes, we have a lot of patches, but then again, we have a lot of open 
> bugs too. I, for one, am much more worried about the latter.
>
> Please back this up with _real_ statistics.
>
> Thanks
>
> Rémi
>
Here's the script that I used to generate this.  I have not manually 
reviewed all of thousands of patches to determine the unique situation 
of each patch, however I would like a suggestion on how to demonstrate 
_real_ statistics short of auditing each and every patch in portage 
which I personally don't have time to do.
for I in `ls`; do
    PATCH=`ls -R ${I} | grep patch | wc -l`
    DIFF=`ls -R ${I} | grep diff | wc -l`
    COUNT=$(( ${PATCH} + ${DIFF} ))
        if ! [ ${COUNT} == 0 ]
        then
                echo $I $COUNT
        fi
done


Andrew

Re: [gentoo-dev] The Plethora of Patches

[Andrew D Kirch]

Denis Dupeyron wrote:
> On Thu, Aug 14, 2008 at 4:17 AM, Andrew D Kirch <trelane@trelane.net> wrote:
> [...]
>
> Looks like you counted the number of files in the files/
> subdirectories. Not all of these are patches. Also, you probably
> forgot to count seds, as some of us use sed more than patches.
>
> Oh, and like Jeremy was hinting, please contact QA. They need somebody like you.
>
> Denis.
>
>   
How would one get ahold of this QA?

Andrew

Re: [gentoo-dev] The Plethora of Patches

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 5186 bytes --]

On Fri, Aug 15, 2008 at 12:55:12AM -0400, Andrew D Kirch wrote:
> Here's the script that I used to generate this.  I have not manually 
> reviewed all of thousands of patches to determine the unique situation of 
> each patch, however I would like a suggestion on how to demonstrate _real_ 
> statistics short of auditing each and every patch in portage which I 
> personally don't have time to do.
Ok, even this misses lots of patches.

As a suggestion on count improvements, look for invocations of epatch.
They will take the following variations:
1. Use of a single file from $FILESDIR
2. Use of a single file from some distfile
3. Use of a directory from $FILESDIR
4. Use of a directory from some distfile

Cases #1 and #2 will cover probably 95% of packages.
Cases #3 and #4 will however contribute a lot to your stats.

For MySQL as an example: 
http://git.overlays.gentoo.org/gitweb/?p=proj/mysql-extras.git;a=summary
We carry 64 different patches. In some cases like 080_all_slot_script-*,
it's 10 different versions of the same patch, that apply to different
upstream releases. Lots of the patches are backports from upstream for
bugs or security, because their release schedule isn't co-operative with
often. 15 of the total 64 files have comment headers, esp. where the
patch was a newer respin of some old one - I've been adding comment
blocks. None of the patches in the mysql set are new features.

Tracking epatch with grep (not hooking it, because of conditional
patching) will get you a better count of overall patches, but not the
distribution of patches in ebuilds.

One other common case to watch for, is cases where we just borrow some
of all of the debian patchset for a package.

In general, I do see your interest in pushing patches upstream, but as
both an upstream author and a downstream maintainer, I ask you to
consider all distributions equally.

Per my experiences as upstream (spanning 6+ packages I've written over
the years):
- Ubuntu seems to have a particularly bad track record with sending
  patches upstream. As the author of readahead-list (which is in a lot
  of distros), I've specifically mailed the Ubuntu maintainer and asked
  that he send me tidied up versions of the patches (I had some specific
  concerns) - and they totally ignored the newer version released a year
  later. I've heard precisely nothing back from them ever. In the case
  of readahead-list, they have two nice feature patches, and one bugfix.
  For a couple of my other packages, I have neither asked nor received
  anything from them, but I do see them carrying feature patches.
- FreeBSD has a decent track record, I've received patches for a few
  different bugs and new features. RedHat/Fedora are pretty good as
  well.

Negative experiences as a downstream maintainer:
- Some upstreams ignored you - For Gitosis (see gitosis-gentoo and
  gitosis packages). Upstream has never accepted or even acknowledged
  any of the feature or bugfix patches I've sent him. This won't show up
  as a patch in your counts, because I now maintain gitosis-gentoo as a
  fork of the original because of the lack of upstream input.
- Some upstreams attack you - For foo2zjs, I was outright textually
  assaulted by the author when I submitted a patch for a page rotation
  bug - he demands that nobody use any third-party packaging and install
  it entirely manually if they want any support - despite the fact I was
  submitting a bugfix and not asking for anything.
- Stubborn upstreams - an upstream denied for a long time that one thing
  I suspected of being an issue was really at fault. It took me several
  months of detailed debugging to conclusively prove it (the upstream in
  question ran on a BSD system and didn't realize a subtle Linux
  difference).
- Requirements of paperwork and contribution rules - Some GNU/FSF
  projects are pretty bad in this regard, wanting signed paperwork to
  have a patch included in the upstream, even if it's a bugfix.
- Strenuous submission process, this is a less degree of a problem, but
  some larger upstreams are extremely picky about submissions. I've had
  to revise a bugfix 5-6 times before from using the style of the
  existing code to match the style of the new requirements instead.
- Effective contribution channels - For MySQL, I've never had a patch
  that I submitted directly to the upstream bug tracking system
  accepted, but all the patches I addressed directly to a developer that
  I knew was reasonable for that portion of the codebase made it in.

Some positive experiences as downstream:
- Linux Kernel. Good patch submission review and ease of inclusion. I've
  got two good bugfixes you'll find in current kernels, plus I've done
  prototypes or spins of other patchsets that have been well received
  even if they weren't suitable for inclusion.
- Danga (MogileFS). A couple of good patches then you get commit access.
  All commits are reviewed anyway.

-- 
Robin Hugh Johnson
Gentoo Linux Developer & Infra Guy
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

[-- Attachment #2: Type: application/pgp-signature, Size: 329 bytes --]

Re: [gentoo-dev] The Plethora of Patches

[Josh Saddler]

[-- Attachment #1: Type: text/plain, Size: 527 bytes --]

Andrew D Kirch wrote:
> Denis Dupeyron wrote:
>> On Thu, Aug 14, 2008 at 4:17 AM, Andrew D Kirch <trelane@trelane.net> 
>> wrote:
>> [...]
>>
>> Looks like you counted the number of files in the files/
>> subdirectories. Not all of these are patches. Also, you probably
>> forgot to count seds, as some of us use sed more than patches.
>>
>> Oh, and like Jeremy was hinting, please contact QA. They need somebody 
>> like you.
>>
>> Denis.
>>
>>   
> How would one get ahold of this QA?


qa@gentoo.org ?


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

[gentoo-dev] Re: The Plethora of Patches

[Steve Long]

Andrew D Kirch wrote:

> Here's the script that I used to generate this.
Just some bash hints. In a nutshell: please don't use ls in scripts.
> I have not manually 
> reviewed all of thousands of patches to determine the unique situation
> of each patch, however I would like a suggestion on how to demonstrate
> _real_ statistics short of auditing each and every patch in portage
> which I personally don't have time to do.
I think it's a great idea, and the other reply from robbat gives you a great
spec to start from in terms of classification.

> for I in `ls`; do
for f in *; do

Globs have a lot to recommend them: see http://wooledge.org:8000/glob
>     PATCH=`ls -R ${I} | grep patch | wc -l`
>     DIFF=`ls -R ${I} | grep diff | wc -l`
>     COUNT=$(( ${PATCH} + ${DIFF} ))
    while read -rd ''; do let count++
    done < <(find "$dir" \( -name '*.diff' -o -name '*.patch' \) -print0)

..in the general case, where you actually need a recursive descend. (We
don't here.)
>         if ! [ ${COUNT} == 0 ]
>         then
>                 echo $I $COUNT
>         fi
    ((count)) && { echo "$dir : $count" }

See http://bash-hackers.org/wiki/doku.php?id=syntax:words for an explanation
of why the quotes make a difference.

Putting it together you end up with this:

#!/bin/bash
# ./countPatchFiles > patchCount
# sed -nr '/^Category: (.*): (.*)/s//\1\2/p' patchCount |sort -n -k 2
PORTDIR=$(portageq envvar PORTDIR)
declare -i count tot=0 cTot
shopt -s nullglob
for d in "$PORTDIR"/*/; do
  c=${d#"$PORTDIR/"}; c=${c%/}
  [[ $c = *-* ]] || continue
  cTot=0
  echo "$c" >&2
  for p in "$d"*/; do
    files=("$p"files/*.patch "$p"files/*.diff)
    count=${#files[@]}
    ((count)) || continue
    p=${p#"$d"}; echo "$c/${p%/} : $count"
    ((tot+=count,cTot+=count))
  done
  echo "Category: $c : $cTot"
done
echo "Total: $tot"

-- HTH,
igli.
(The files are in that array, if their names should be needed.)

Re: [gentoo-dev] The Plethora of Patches

[Rémi Cardona]

Andrew D Kirch a écrit :
> Here's the script that I used to generate this.  I have not manually 
> reviewed all of thousands of patches to determine the unique situation 
> of each patch, however I would like a suggestion on how to demonstrate 
> _real_ statistics short of auditing each and every patch in portage 
> which I personally don't have time to do.

Then how can you come to the conclusion that all the patches we carry 
are somehow bad? I won't reiterate what Robin and others have said about 
various upstreams, but this is a _reality_ we have to work with.

Now instead of doubting your stats once again, here's my suggestion :

Pick a set of packages you like/use, contact its Gentoo maintainers and 
ask for more info about the patches we carry in Portage. Chances are, 
most patches will already be in upstream's BTS or mailing list's 
archives. But some patches may have been forgotten.

Saying we are doing an awful job because $nb_patches > 0 is *not* the 
way to get things moving in the right direction.

I for one, want to get $nb_patches closer to 0, but it's an ongoing 
process, not a goal.

And yes, there *will* have to be a review of all "epatch" and "sed" 
lines in all our ebuilds if we want to go in the right direction.

Cheers

Rémi

Re: [gentoo-dev] The Plethora of Patches

[Tobias Scherbaum]

[-- Attachment #1: Type: text/plain, Size: 1226 bytes --]

Santiago M. Mola wrote:
> I think that's all we need in order to know how were things when the
> patch was added and if it needs to be pushed/tracked upstream, removed
> in the next version of the package, etc.
> 
> Some of us already put these kind of headers, or at least an URL to
> upstream bug or a meaningful source of info about the patch.

A short description possibly including a reference to an upstream or
Gentoo bugreport prefixed to every epatch call should be our minimum
standard. Working on packages whose maintainers are MIA isn't always
that simple - but it's even worse if you have to check a number of
patches to find out what they are for, since when they are in and what
their status is ...

> However, tracking the status of every patch since its inclusion in
> portage until it's removed would be a huge work overhead... and I
> doubt it's worthy.

I don't think it's a huge work overhead, it'll take an additional minute
per included patch to include a minimal description into the ebuild(s)
and use a standardized header for the patch. Compared to the time one
needs to spend when searching for information on that patch somewhen
later on it's worth every minute.

  Tobias

[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] The Plethora of Patches

[Santiago M. Mola]

On Tue, Aug 19, 2008 at 12:02 AM, Tobias Scherbaum
<dertobi123@gentoo.org> wrote:
> Santiago M. Mola wrote:
>
>> However, tracking the status of every patch since its inclusion in
>> portage until it's removed would be a huge work overhead... and I
>> doubt it's worthy.
>
> I don't think it's a huge work overhead, it'll take an additional minute
> per included patch to include a minimal description into the ebuild(s)
> and use a standardized header for the patch. Compared to the time one
> needs to spend when searching for information on that patch somewhen
> later on it's worth every minute.
>

Of course, puting a header with info in every patch is not a work
overhead and I'd say it should be policy. What I meant is that it's no
worth to track the status of every patch after it's added, as was
suggested.

Regards,
-- 
Santiago M. Mola
Jabber ID: cooldwind@gmail.com

Re: [gentoo-dev] The Plethora of Patches

[Tobias Scherbaum]

[-- Attachment #1: Type: text/plain, Size: 1206 bytes --]

Santiago M. Mola wrote:
> On Tue, Aug 19, 2008 at 12:02 AM, Tobias Scherbaum
> <dertobi123@gentoo.org> wrote:
> > Santiago M. Mola wrote:
> >
> >> However, tracking the status of every patch since its inclusion in
> >> portage until it's removed would be a huge work overhead... and I
> >> doubt it's worthy.
> >
> > I don't think it's a huge work overhead, it'll take an additional minute
> > per included patch to include a minimal description into the ebuild(s)
> > and use a standardized header for the patch. Compared to the time one
> > needs to spend when searching for information on that patch somewhen
> > later on it's worth every minute.
> >
> 
> Of course, puting a header with info in every patch is not a work
> overhead and I'd say it should be policy. What I meant is that it's no
> worth to track the status of every patch after it's added, as was
> suggested.

Agreed. Everyone of us is doing some kind of status tracking for each
and every patch at least for every version bump, additional status
tracking like Andrew suggested would be a good thing (tm) but is plain
impossible to realize for now given the fact we're lacking the needed
manpower.

  Tobias

[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 197 bytes --]