[gentoo-dev] prepalldocs is now banned

[gentoo-dev] prepalldocs is now banned

[Thomas Anderson]

Hi Everyone,

    This is a note that in the council meeting on 02/12/2009 the
    function 'prepalldocs' is banned for use in ebuilds with EAPIs 0 1
    and 2. If you want some functionality from this function, please
    propose a new function or clearly defined behavior for prepalldocs
    for a *new* EAPI.

Regards,
Thomas Anderson

-- 
---------
Thomas Anderson
Gentoo Developer
/////////
Areas of responsibility:
AMD64, Secretary to the Gentoo Council
---------

Re: [gentoo-dev] prepalldocs is now banned

[Thilo Bangert]

[-- Attachment #1: Type: text/plain, Size: 9003 bytes --]

Thomas Anderson <gentoofan23@gentoo.org> said:
> Hi Everyone,
>
>     This is a note that in the council meeting on 02/12/2009 the
>     function 'prepalldocs' is banned for use in ebuilds with EAPIs 0 1
>     and 2. If you want some functionality from this function, please
>     propose a new function or clearly defined behavior for prepalldocs
>     for a *new* EAPI.

we have a tracker bug at:
http://bugs.gentoo.org/show_bug.cgi?id=259422

the following 99 packages currently still use 'prepalldocs'. some time in 
the near future i will start filing individual bugs for each of these. 
feel free to beat me to the punch.

thanks
kind regards
Thilo

app-arch/gtk-splitter                                                                             
app-arch/rar                                                                                      
app-arch/xdms                                                                                     
app-backup/amanda                                                                                 
app-cdr/cdcover                                                                                   
app-crypt/gnupg-pkcs11-scd                                                                        
app-crypt/steghide                                                                                
app-doc/howto-text                                                                                
app-doc/phrack                                                                                    
app-emacs/ess                                                                                     
app-misc/ca-certificates                                                                          
app-misc/emelfm2                                                                                  
app-misc/g15daemon                                                                                
app-misc/g15macro                                                                                 
app-misc/g15message                                                                               
app-misc/g15mpd                                                                                   
app-misc/g15stats                                                                                 
app-office/gnucash                                                                                
app-text/htag                                                                                     
app-text/robodoc                                                                                  
app-text/sloccount                                                                                
app-text/ttf2pt1                                                                                  
dev-db/myodbc                                                                                     
dev-db/mysql++                                                                                    
dev-db/unixODBC                                                                                   
dev-embedded/sdcc                                                                                 
dev-embedded/uisp                                                                                 
dev-games/flatzebra                                                                               
dev-lang/gpc                                                                                      
dev-libs/libg15render                                                                             
dev-libs/libgringotts                                                                             
dev-libs/libmcrypt                                                                                
dev-libs/pkcs11-helper                                                                            
dev-libs/ppl                                                                                      
dev-python/gst-python                                                                             
dev-python/yolk                                                                                   
dev-tcltk/mysqltcl                                                                                
dev-tex/cjk-latex                                                                                 
dev-tex/frakturx                                                                                  
dev-util/anjuta                                                                                   
dev-util/geany                                                                                    
dev-util/ltrace                                                                                   
dev-util/pretrace                                                                                 
games-arcade/afternoonstalker                                                                     
games-emulation/fceultra                                                                          
games-fps/lsdldoom                                                                                
games-kids/tuxtype                                                                                
games-misc/bsd-games                                                                              
games-misc/bsd-games-non-free                                                                     
games-misc/typespeed                                                                              
games-puzzle/picpuz                                                                               
games-rpg/openglad                                                                                
games-simulation/lincity-ng                                                                       
games-strategy/liquidwar                                                                          
mail-filter/libdomainkeys                                                                         
media-gfx/surf                                                                                    
media-libs/libtheora                                                                              
media-sound/ezstream                                                                              
media-sound/gbsplay                                                                               
media-sound/gmusicbrowser                                                                         
media-sound/lingot                                                                                
media-sound/muse                                                                                  
media-sound/ncmpc                                                                                 
media-sound/ncmpcpp                                                                               
media-sound/twolame                                                                               
media-sound/vorbis-tools                                                                          
media-video/smplayer                                                                              
net-dialup/diald                                                                                  
net-dialup/freeradius                                                                             
net-dialup/rp-pppoe                                                                               
net-irc/irssi-otr                                                                                 
net-libs/libfwbuilder                                                                             
net-mail/mhonarc                                                                                  
net-misc/mknbi                                                                                    
net-misc/openvpn                                                                                  
net-misc/portfwd                                                                                  
net-nntp/slrn                                                                                     
sys-apps/acl                                                                                      
sys-apps/attr
sys-apps/dmapi
sys-apps/dmidecode
sys-apps/ipmitool
sys-apps/most
sys-apps/pcsc-lite
sys-apps/pcsc-tools
sys-apps/x86info
sys-block/scsirastools
sys-boot/grub
sys-boot/sysload
sys-devel/distcc
sys-fs/dosfstools
sys-fs/ntfs3g
sys-fs/xfsdump
sys-fs/xfsprogs
sys-power/nvram-wakeup
sys-power/suspend
sys-process/at
x11-misc/wbarconf
x11-wm/awesome

find -name '*.ebuild' -exec grep -nH prepalldocs \{\} \; | sort | awk -
F"/" '{print $2"/"$3}'| uniq

>
> Regards,
> Thomas Anderson




[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] prepalldocs is now banned

[Michael Sterrett]

Why would people want to blindly remove prepalldocs when it clearly
results in an inferior build of the package?

Take the recent change to the (already marked stable)
gnupg-2.0.9.ebuild as an example:

Before:

$ epm -qi gnupg | grep Size
Size        : 2845754

$ epm -qi gnupg | grep Size
Size        : 3089515

So everybody who emerges gnupg since this change is wasting space for
no good reason.

Michael Sterrett
 -Mr. Bones.-
mr_bones_@gentoo.org

Re: [gentoo-dev] prepalldocs is now banned

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 474 bytes --]

On Tue, 17 Feb 2009 19:10:33 -0500
Michael Sterrett <mr_bones_@gentoo.org> wrote:
> So everybody who emerges gnupg since this change is wasting space for
> no good reason.

If you care about a couple of hundred kilobytes, relying upon
individual ebuilds to ask the package manager to compress documentation
in some arbitrary manner is the wrong solution.

This was already discussed at length prior to the Council reaching
their decision.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] prepalldocs is now banned

[Alexis Ballier]

[-- Attachment #1: Type: text/plain, Size: 1160 bytes --]

On Wed, 18 Feb 2009 00:18:06 +0000
Ciaran McCreesh <ciaran.mccreesh@googlemail.com> wrote:

> On Tue, 17 Feb 2009 19:10:33 -0500
> Michael Sterrett <mr_bones_@gentoo.org> wrote:
> > So everybody who emerges gnupg since this change is wasting space
> > for no good reason.
> 
> If you care about a couple of hundred kilobytes, relying upon
> individual ebuilds to ask the package manager to compress
> documentation in some arbitrary manner is the wrong solution.

Then, for the nth time, what would be the good solution? How would one
convert prepalldocs usage to something allowed? I've failed to find
anything about it in the relevant bug and the only answer I've seen is
"remove it". You can count on me for marking any prepalldocs removal bug
I'll be the assignee as wontfix as long as there won't be any
alternative solution.

Note that I would consider a viable solution banning prepalldocs and
simply removing it if portage was compressing docs by its own or
calling prepalldocs after src_install... but then IMHO that's the
removal of prepalldocs that would require an EAPI bump not its
reintroduction.


Regards,

Alexis.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] prepalldocs is now banned

[Michael Sterrett]

I added a prepalldocs function to eutils.eclass to provide the
functionality.  It implements the
behavior of the current stable sys-apps/portage-2.1.6.4.

Have fun,

Michael Sterrett
 -Mr. Bones.-
mr_bones_@gentoo.org

On Fri, Feb 13, 2009 at 4:30 PM, Thomas Anderson <gentoofan23@gentoo.org> wrote:
> Hi Everyone,
>
>    This is a note that in the council meeting on 02/12/2009 the
>    function 'prepalldocs' is banned for use in ebuilds with EAPIs 0 1
>    and 2. If you want some functionality from this function, please
>    propose a new function or clearly defined behavior for prepalldocs
>    for a *new* EAPI.

[gentoo-dev] prepalldocs implementation in eutils.eclass (was: prepalldocs is now banned)

[Torsten Veller]

* Michael Sterrett <mr_bones_@gentoo.org>:
> I added a prepalldocs function to eutils.eclass to provide the
> functionality.  It implements the
> behavior of the current stable sys-apps/portage-2.1.6.4.

ecompressdir is more portage internal than prepalldocs ever was.
This must be fixed.

Re: [gentoo-dev] prepalldocs implementation in eutils.eclass (was: prepalldocs is now banned)

[Michael Sterrett]

Patches welcome.

On Wed, Feb 18, 2009 at 3:13 AM, Torsten Veller <ml-en@veller.net> wrote:
> * Michael Sterrett <mr_bones_@gentoo.org>:
>> I added a prepalldocs function to eutils.eclass to provide the
>> functionality.  It implements the
>> behavior of the current stable sys-apps/portage-2.1.6.4.
>
> ecompressdir is more portage internal than prepalldocs ever was.
> This must be fixed.
>
>

Re: [gentoo-dev] prepalldocs is now banned

[Ulrich Mueller]

>>>>> On Wed, 18 Feb 2009, Alexis Ballier wrote:

> Then, for the nth time, what would be the good solution? How would
> one convert prepalldocs usage to something allowed? I've failed to
> find anything about it in the relevant bug and the only answer I've
> seen is "remove it". You can count on me for marking any prepalldocs
> removal bug I'll be the assignee as wontfix as long as there won't
> be any alternative solution.

> Note that I would consider a viable solution banning prepalldocs and
> simply removing it if portage was compressing docs by its own or
> calling prepalldocs after src_install... but then IMHO that's the
> removal of prepalldocs that would require an EAPI bump not its
> reintroduction.

I think a viable solution would consist of two parts:
1. Add some exclude mechanism to prepalldocs, as suggested in
   bug 164114 [1].
2. Have Portage call prepalldocs by default (in prepall).

This way, everything installed under /usr/share/doc would be
compressed by default (honouring the user's setting of
PORTAGE_COMPRESS). Any package that needs literal, uncompressed files
in /usr/share/doc could specify this via the exclude mechanism.

Obviously, this is an incompatible change and would require an EAPI
bump.

Ulrich

[1] <http://bugs.gentoo.org/show_bug.cgi?id=164114>

[gentoo-dev] Re: prepalldocs implementation in eutils.eclass

[Torsten Veller]

* Michael Sterrett <mr_bones_@gentoo.org>:
> Patches welcome.

--- eutils.eclass
+++ eutils.eclass
@@ -1823,21 +1823,3 @@
 		newbin "${tmpwrapper}" "${wrapper}" || die
 	fi
 }
-
-# @FUNCTION: prepalldocs
-# @USAGE:
-# @DESCRIPTION:
-# Compress files in /usr/share/doc which are not already
-# compressed, excluding /usr/share/doc/${PF}/html.
-# Uses the ecompressdir to do the compression.
-prepalldocs() {
-	if [[ -n $1 ]] ; then
-		ewarn "prepalldocs: invalid usage; takes no arguments"
-	fi
-
-	cd "${D}"
-	[[ -d usr/share/doc ]] || exit 0
-
-	ecompressdir --ignore /usr/share/doc/${PF}/html
-	ecompressdir --queue /usr/share/doc
-}

Re: [gentoo-dev] prepalldocs is now banned

[Petteri Räty]

[-- Attachment #1: Type: text/plain, Size: 403 bytes --]

Michael Sterrett wrote:
> I added a prepalldocs function to eutils.eclass to provide the
> functionality.  It implements the
> behavior of the current stable sys-apps/portage-2.1.6.4.
> 
> Have fun,
> 
> Michael Sterrett
>  -Mr. Bones.-
> mr_bones_@gentoo.org
> 

I don't think developers should add stuff to eutils.eclass without prior
review on this mailing list.

Regards,
Petteri


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: [gentoo-dev] prepalldocs is now banned

[Petteri Räty]

[-- Attachment #1: Type: text/plain, Size: 1210 bytes --]

Thilo Bangert wrote:
> Thomas Anderson <gentoofan23@gentoo.org> said:
>> Hi Everyone,
>>
>>     This is a note that in the council meeting on 02/12/2009 the
>>     function 'prepalldocs' is banned for use in ebuilds with EAPIs 0 1
>>     and 2. If you want some functionality from this function, please
>>     propose a new function or clearly defined behavior for prepalldocs
>>     for a *new* EAPI.
> 
> we have a tracker bug at:
> http://bugs.gentoo.org/show_bug.cgi?id=259422
> 
> the following 99 packages currently still use 'prepalldocs'. some time in 
> the near future i will start filing individual bugs for each of these. 
> feel free to beat me to the punch.
> 
> thanks
> kind regards
> Thilo
> 

The council decision should probably have had more text on what to do to
existing usage bug I don't think we spent time on that yet and probably
should in the next meeting. Until we decide how things need to be
changed IMHO the existing ebuilds can stay as they are as I don't think
we should be spending time removing the usage just to find out that we
will for example move the function to eclasses so the ebuilds won't need
to be modified.

Regards,
Petteri




[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: [gentoo-dev] prepalldocs is now banned

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 478 bytes --]

On Wed, 18 Feb 2009 08:39:58 +0100
Alexis Ballier <aballier@gentoo.org> wrote:
> Then, for the nth time, what would be the good solution?

If you explicitly need compression, do it by hand, since there aren't
any mechanisms for guaranteed compression anyway.

If you don't explicitly need compression, don't do anything.

And if you're trying to make a space-critical distribution, start
looking at the big things, not the 4% things.

Easy.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] prepalldocs is now banned

[Ulrich Mueller]

>>>>> On Wed, 18 Feb 2009, Ciaran McCreesh wrote:

> If you don't explicitly need compression, don't do anything.

You mean if the user has requested compression with PORTAGE_COMPRESS,
ignore it? What about FEATURES="nodoc", ignore it too?

> And if you're trying to make a space-critical distribution, start
> looking at the big things, not the 4% things.

Typical documentation consists of text files, so I would expect to
save of the order of 50 % by compressing them. How did you obtain
above number of 4 %?

Ulrich

Re: [gentoo-dev] prepalldocs is now banned

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 937 bytes --]

On Wed, 18 Feb 2009 16:18:03 +0100
Ulrich Mueller <ulm@gentoo.org> wrote:
> > If you don't explicitly need compression, don't do anything.
> 
> You mean if the user has requested compression with PORTAGE_COMPRESS,
> ignore it?

Yup. It's a misfeature. You can see this by considering the proportion
of ebuilds that honour it...

> What about FEATURES="nodoc", ignore it too?

FEATURES is purely a Portage internal, so ebuilds shouldn't be messing
with it.

> > And if you're trying to make a space-critical distribution, start
> > looking at the big things, not the 4% things.
> 
> Typical documentation consists of text files, so I would expect to
> save of the order of 50 % by compressing them. How did you obtain
> above number of 4 %?

The proportion of ebuild-managed content in /usr/share/doc (with
USE=doc, so it's a massive overestimate for space-relevant systems) vs
elsewhere.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] prepalldocs is now banned

[Ulrich Mueller]

>>>>> On Wed, 18 Feb 2009, Ciaran McCreesh wrote:

>> You mean if the user has requested compression with
>> PORTAGE_COMPRESS, ignore it?

> Yup. It's a misfeature. You can see this by considering the
> proportion of ebuilds that honour it...

All ebuilds that install things with "dodoc" at least. That must be
quite a few.

Ulrich

Re: [gentoo-dev] prepalldocs is now banned

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 765 bytes --]

On Wed, 18 Feb 2009 16:46:30 +0100
Ulrich Mueller <ulm@gentoo.org> wrote:
> > Yup. It's a misfeature. You can see this by considering the
> > proportion of ebuilds that honour it...
> 
> All ebuilds that install things with "dodoc" at least. That must be
> quite a few.

But it's not universal, nor consistent. That's part of the problem.

If you really, genuinely think you have a case for compression of docs,
backed up with statistics showing that it's a relevant change, then you
should write a proposal for future EAPIs for handling it, and you
should do it in such a way that it works automatically for all ebuilds,
without any developer intervention (but providing some way for ebuilds
to disable it where necessary).

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] prepalldocs is now banned

[Alexis Ballier]

[-- Attachment #1: Type: text/plain, Size: 898 bytes --]

On Wed, 18 Feb 2009 14:06:46 +0000
Ciaran McCreesh <ciaran.mccreesh@googlemail.com> wrote:

> On Wed, 18 Feb 2009 08:39:58 +0100
> Alexis Ballier <aballier@gentoo.org> wrote:
> > Then, for the nth time, what would be the good solution?
> 
> If you explicitly need compression, do it by hand, since there aren't
> any mechanisms for guaranteed compression anyway.

Yes of course.

> If you don't explicitly need compression, don't do anything.

... and mark relevant stuff as "ok to be compressed with whatever suits
you best"... sounds familiar? :)

> And if you're trying to make a space-critical distribution, start
> looking at the big things, not the 4% things.

When I do something space critical I exclude /usr/share/doc and a
couple of others... that doesn't mean I do like wasting space on
something not space critical when compression algorithms exist.


Alexis.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] prepalldocs is now banned

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1224 bytes --]

On Wed, 18 Feb 2009 19:28:46 +0100
Alexis Ballier <aballier@gentoo.org> wrote:
> > If you don't explicitly need compression, don't do anything.
> 
> ... and mark relevant stuff as "ok to be compressed with whatever
> suits you best"... sounds familiar? :)

No. That's entirely the wrong approach, because it relies upon every
ebuild having support for it, and most don't. The right approach, if
you can demonstrate that there's a genuine benefit to compressing
documentation, is to make a proposal for a future EAPI for compression
by default for certain directories, with an override available for
ebuilds that need specific behaviour.

> > And if you're trying to make a space-critical distribution, start
> > looking at the big things, not the 4% things.
> 
> When I do something space critical I exclude /usr/share/doc and a
> couple of others... that doesn't mean I do like wasting space on
> something not space critical when compression algorithms exist.

If you care about space, focus on something relevant. You are wondering
whether turning the radio off will make your car more fuel efficient
whilst driving with flat tyres, the windows open and in the wrong gear.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] prepalldocs is now banned

[Donnie Berkholz]

[-- Attachment #1: Type: text/plain, Size: 486 bytes --]

On 03:07 Wed 18 Feb     , Michael Sterrett wrote:
> I added a prepalldocs function to eutils.eclass to provide the 
> functionality.  It implements the behavior of the current stable 
> sys-apps/portage-2.1.6.4.

FYI, this addition broke a number of X packages. Hopefully it didn't 
break much else that's yet undiscovered.

http://bugs.gentoo.org/show_bug.cgi?id=259491

-- 
Thanks,
Donnie

Donnie Berkholz
Developer, Gentoo Linux
Blog: http://dberkholz.wordpress.com

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] prepalldocs is now banned

[Alexis Ballier]

[-- Attachment #1: Type: text/plain, Size: 954 bytes --]

> If you really, genuinely think you have a case for compression of
> docs, backed up with statistics showing that it's a relevant change,

I fail to see why you need statistics for something that is clearly a
waste of space, but this could be a start:
http://archives.gentoo.org/gentoo-dev/msg_9018a9f64cd32ba85494887ffe3edf78.xml

> then you should write a proposal for future EAPIs for handling it,

I don't understand why something that has been there for ages has to
die. For what I've seen, the major (and only) problem with prepalldocs
is its definition and I'm sure we can find one that everybody will
agree with.

> and you should do it in such a way that it works automatically for
> all ebuilds, without any developer intervention (but providing some
> way for ebuilds to disable it where necessary).

This is probably a good start:
http://archives.gentoo.org/gentoo-dev/msg_eb1f7952eb2f0fe725bde331a4d9ae30.xml

Alexis.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] prepalldocs is now banned

[Michael Sterrett]

It's already fixed.

Re: [gentoo-dev] prepalldocs is now banned

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1605 bytes --]

On Wed, 18 Feb 2009 20:01:04 +0100
Alexis Ballier <aballier@gentoo.org> wrote:
> > If you really, genuinely think you have a case for compression of
> > docs, backed up with statistics showing that it's a relevant change,
> 
> I fail to see why you need statistics for something that is clearly a
> waste of space, but this could be a start:
> http://archives.gentoo.org/gentoo-dev/msg_9018a9f64cd32ba85494887ffe3edf78.xml

Because if you look at the statistics, it's pretty obvious that it's a
stupid idea. If you've got USE=doc, somewhere around 4% of managed
files are in /usr/share/doc, and once you take inode sizes into
account, you can knock that down to about 3% -- and this is for people
who are already turning on a use flag that wouldn't be on on
space-relevant systems.

> > then you should write a proposal for future EAPIs for handling it,
> 
> I don't understand why something that has been there for ages has to
> die. For what I've seen, the major (and only) problem with prepalldocs
> is its definition and I'm sure we can find one that everybody will
> agree with.

Because killing it is better than keeping it. It's solving an
irrelevant problem the wrong way.

> > and you should do it in such a way that it works automatically for
> > all ebuilds, without any developer intervention (but providing some
> > way for ebuilds to disable it where necessary).
> 
> This is probably a good start:
> http://archives.gentoo.org/gentoo-dev/msg_eb1f7952eb2f0fe725bde331a4d9ae30.xml

Can you demonstrate that it's even remotely useful?

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] prepalldocs is now banned

[Tobias Scherbaum]

[-- Attachment #1: Type: text/plain, Size: 598 bytes --]

Am Mittwoch, den 18.02.2009, 12:26 +0200 schrieb Petteri Räty:
> Michael Sterrett wrote:
> > I added a prepalldocs function to eutils.eclass to provide the
> > functionality.  It implements the
> > behavior of the current stable sys-apps/portage-2.1.6.4.
> > 
> > Have fun,
> > 
> > Michael Sterrett
> >  -Mr. Bones.-
> > mr_bones_@gentoo.org
> > 
> 
> I don't think developers should add stuff to eutils.eclass without prior
> review on this mailing list.

Agreed. Besides that, replacing a workaround with another workaround
isn't a sexy way to solve something ...

  Tobias

[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] prepalldocs is now banned

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 251 bytes --]

On Wed, 18 Feb 2009 14:01:44 -0500
Michael Sterrett <mr_bones_@gentoo.org> wrote:
> It's already fixed.

And have you learned not to try such blatantly irresponsible and
childish behaviour on a tree used by other people?

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] prepalldocs is now banned

[Alexis Ballier]

[-- Attachment #1: Type: text/plain, Size: 1528 bytes --]

On Wed, 18 Feb 2009 18:36:11 +0000
Ciaran McCreesh <ciaran.mccreesh@googlemail.com> wrote:

> On Wed, 18 Feb 2009 19:28:46 +0100
> Alexis Ballier <aballier@gentoo.org> wrote:
> > > If you don't explicitly need compression, don't do anything.
> > 
> > ... and mark relevant stuff as "ok to be compressed with whatever
> > suits you best"... sounds familiar? :)
> 
> No. That's entirely the wrong approach,

I would call it an "unperfect yet useful approach" that unfortunately
we've been using for eapi 0-2.

> because it relies upon every
> ebuild having support for it, and most don't.

That's why it needs to be improved.

> The right approach, if
> you can demonstrate that there's a genuine benefit to compressing
> documentation, is to make a proposal for a future EAPI for compression
> by default for certain directories, with an override available for
> ebuilds that need specific behaviour.

And I agree this is the right solution but yet unimplemented...

> > > And if you're trying to make a space-critical distribution, start
> > > looking at the big things, not the 4% things.
> > 
> > When I do something space critical I exclude /usr/share/doc and a
> > couple of others... that doesn't mean I do like wasting space on
> > something not space critical when compression algorithms exist.
> 
> If you care about space, focus on something relevant. 
[...]

You got me wrong there it seems: I do not care about space, however I
do care about useless waste of space.

Alexis.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] prepalldocs is now banned

[Ulrich Mueller]

>>>>> On Wed, 18 Feb 2009, Ciaran McCreesh wrote:

> Because if you look at the statistics, it's pretty obvious that it's
> a stupid idea. If you've got USE=doc, somewhere around 4% of managed
> files are in /usr/share/doc, and once you take inode sizes into
> account, you can knock that down to about 3% -- and this is for
> people who are already turning on a use flag that wouldn't be on on
> space-relevant systems.

I cannot reproduce this number, but of course it will be different for
every system. Here I have:
   9.5 GB installed files in total
   1.6 GB in /usr/share/doc

That's some 17 %, _with_ PORTAGE_COMPRESS enabled.

Ulrich

Re: [gentoo-dev] prepalldocs is now banned

[Donnie Berkholz]

[-- Attachment #1: Type: text/plain, Size: 453 bytes --]

On 19:08 Wed 18 Feb     , Ciaran McCreesh wrote:
> On Wed, 18 Feb 2009 14:01:44 -0500
> Michael Sterrett <mr_bones_@gentoo.org> wrote:
> > It's already fixed.
> 
> And have you learned not to try such blatantly irresponsible and
> childish behaviour on a tree used by other people?

This email would better be sent in private, if at all.

-- 
Thanks,
Donnie

Donnie Berkholz
Developer, Gentoo Linux
Blog: http://dberkholz.wordpress.com

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] prepalldocs implementation in eutils.eclass (was: prepalldocs is now banned)

[Tiziano Müller]

[-- Attachment #1: Type: text/plain, Size: 2409 bytes --]

Am Mittwoch, den 18.02.2009, 03:22 -0500 schrieb Michael Sterrett:
> Patches welcome.
> 
> On Wed, Feb 18, 2009 at 3:13 AM, Torsten Veller <ml-en@veller.net> wrote:
> > * Michael Sterrett <mr_bones_@gentoo.org>:
> >> I added a prepalldocs function to eutils.eclass to provide the
> >> functionality.  It implements the
> >> behavior of the current stable sys-apps/portage-2.1.6.4.
> >
> > ecompressdir is more portage internal than prepalldocs ever was.
> > This must be fixed.
> >
> >
Instead of breaking things you could have tried to come up with a real
solution.

Even though I share the opinion that compressing docs is unnecessary I
started looking through the use cases.
I'm not done yet, but I think I already identified a couple of common
cases:

[1]
src_install() {
  emake DESTDIR="${D}" docdir="/usr/share/doc/${PF}"
  prepalldocs
}
(or similar)
will change to:
src_install() {
  emake DESTDIR="${D}" docdir="${T}/doc"
  dodoc "${T}/doc"
}

[2]
Not needed at all since every doc gets installed using dodoc or newdoc

[3]
Package installs doc to wrong location.
Solution: dodoc <OLDLOCATION>/* ; rm -rf <OLDLOCATION>

[4]
Assuming that "make install" installs something in /usr/share/doc/${PF}.
Assumption is wrong since "make install" will install
to /usr/share/doc/${P} if at all (since the package doesn't know
anything about -rX). So it should be changed anyway and then case [1] or
[3] applies.

Example packages for case [1]: app-arch/gtk-splitter, app-cdr/cdcover,
app-crypt/steghide, app-editors/lpe, app-emacs/ess
Example packages for case [2]: app-arch/rar, app-backup/amanda
Example packages for case [3]: app-crypt/gnupg
Example packages for case [4]: app-arch/xdms, app-crypt/gnupg-pkcs11-scd

The only problem I see here is that either <OLDLOCATION> or ${T}/doc
contains subdirs. So my proposal for the next EAPI is to allow dodoc and
newdoc to operate on dirs. Which also gives the benefit to reduce this
idiom:
  insinto /usr/share/doc/${PF}
  doins -r examples
to:
  dodoc examples


Your comments?

Cheers,
Tiziano 



-- 
-------------------------------------------------------
Tiziano Müller
Gentoo Linux Developer, Council Member
Areas of responsibility:
  Samba, PostgreSQL, CPP, Python, sysadmin
E-Mail     : dev-zero@gentoo.org
GnuPG FP   : F327 283A E769 2E36 18D5  4DE2 1B05 6A63 AE9C 1E30

[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] prepalldocs implementation in eutils.eclass (was: prepalldocs is now banned)

[Ulrich Mueller]

>>>>> On Thu, 19 Feb 2009, Tiziano Müller wrote:

> Instead of breaking things you could have tried to come up with a
> real solution.

This could be said about the council's decision as well.

> [1]
> src_install() {
>   emake DESTDIR="${D}" docdir="/usr/share/doc/${PF}"
>   prepalldocs
> }
> (or similar)
> will change to:
> src_install() {
>   emake DESTDIR="${D}" docdir="${T}/doc"
>   dodoc "${T}/doc"
> }

This will not work. dodoc operates only on files, not on directories.

Besides, what do we win by this unnecessary copying?

Ulrich

Re: [gentoo-dev] prepalldocs implementation in eutils.eclass (was: prepalldocs is now banned)

[Nirbheek Chauhan]

On Thu, Feb 19, 2009 at 3:07 PM, Ulrich Mueller <ulm@gentoo.org> wrote:
>>>>>> On Thu, 19 Feb 2009, Tiziano Müller wrote:
> This will not work. dodoc operates only on files, not on directories.
>

Tiziano mentioned that later in the email. Perhaps you missed it? He
proposed allowing dodoc to apply on directories as well.

> Besides, what do we win by this unnecessary copying?
>

What copying?


-- 
~Nirbheek Chauhan

Re: [gentoo-dev] prepalldocs implementation in eutils.eclass (was: prepalldocs is now banned)

[Ulrich Mueller]

>>>>> On Thu, 19 Feb 2009, Nirbheek Chauhan wrote:

>> This will not work. dodoc operates only on files, not on directories.

> Tiziano mentioned that later in the email. Perhaps you missed it?

I didn't miss it. Read his message again:

>>> The only problem I see here is that either <OLDLOCATION> or
>>> ${T}/doc contains subdirs.

Whereas the following:

  emake DESTDIR="${D}" docdir="${T}/doc"
  dodoc "${T}/doc"

will not work, even if ${T}/doc contains only files.

>> Besides, what do we win by this unnecessary copying?

> What copying?

Files are first installed under ${T} and then copied to ${D}.

Ulrich

Re: [gentoo-dev] prepalldocs implementation in eutils.eclass

[Luca Barbato]

Tiziano Müller wrote:
> The only problem I see here is that either <OLDLOCATION> or ${T}/doc
> contains subdirs. So my proposal for the next EAPI is to allow dodoc and
> newdoc to operate on dirs. Which also gives the benefit to reduce this
> idiom:
>   insinto /usr/share/doc/${PF}
>   doins -r examples
> to:
>   dodoc examples
> 
> 
> Your comments?

I like the idea, not sure if dodoc could take an -r or just accept 
automatically directories.

lu

-- 

Luca Barbato
Gentoo Council Member
Gentoo/linux Gentoo/PPC
http://dev.gentoo.org/~lu_zero

[gentoo-dev] Proposal for compression handling (was: Re: prepalldocs is now banned)

[Ulrich Mueller]

>>>>> On Wed, 18 Feb 2009, Alexis Ballier wrote:

>> If you really, genuinely think you have a case for compression of
>> docs, backed up with statistics showing that it's a relevant
>> change, then you should write a proposal for future EAPIs for
>> handling it,

> I don't understand why something that has been there for ages has to
> die. For what I've seen, the major (and only) problem with
> prepalldocs is its definition and I'm sure we can find one that
> everybody will agree with.

>> and you should do it in such a way that it works automatically for
>> all ebuilds, without any developer intervention (but providing some
>> way for ebuilds to disable it where necessary).

> This is probably a good start:
> http://archives.gentoo.org/gentoo-dev/msg_eb1f7952eb2f0fe725bde331a4d9ae30.xml

Let's fill this with some life. What do you think about the following
proposal:

Package managers supporting compression would internally maintain two
lists of paths, both having a default value, plus the possibility to
modify them from ebuilds/eclasses:

   - an inclusion list, initially containing:
     /usr/share/doc /usr/share/info /usr/share/man
   - an exclusion list, initially containing:
     /usr/share/doc/${PF}/html

Then we need a new command, let's call it "docompress" for now (name
to be discussed), to manipulate the lists:

   - "docompress" (without option or with option "-a"): add paths
      (directories or files) to the inclusion list
   - "docompress -d": delete paths from the inclusion list
   - "docompress -x": add paths to the exclusion list
   - "docompress -x -d": remove paths from the exclusion list
     (maybe this is not needed?)

(For package managers not supporting compression, "docompress" would
be a no-op.)

Between the src_install and pkg_postinst phases, the package manager
would then go through the inclusion list and optionally (depending on
the user's setting, i.e. PORTAGE_COMPRESS in the Portage case)
compress anything found there, excluding the stuff from the exclusion
list.

The two most common usage cases:

   1. dodoc <file1> <file2> ...

and

   2. emake ... docdir="/usr/share/doc/${PF}" install

would just work as expected, without any need for calling docompress.

A package that needs uncompressed documentation would simply call
"docompress -d /usr/share/doc" or
"docompress -x /usr/share/doc/${PF}/<some subdir>".

In addition, I think that Tiziano's suggestion to have "dodoc" operate
on directories (or to add a "-r" option as "doins" has) is also a good
idea.

Ulrich

Re: [gentoo-dev] prepalldocs implementation in eutils.eclass (was: prepalldocs is now banned)

[Tiziano Müller]

[-- Attachment #1: Type: text/plain, Size: 1005 bytes --]

Am Donnerstag, den 19.02.2009, 11:00 +0100 schrieb Ulrich Mueller:
> >>>>> On Thu, 19 Feb 2009, Nirbheek Chauhan wrote:
> 
> >> This will not work. dodoc operates only on files, not on directories.
> 
> > Tiziano mentioned that later in the email. Perhaps you missed it?
> 
> I didn't miss it. Read his message again:
> 
> >>> The only problem I see here is that either <OLDLOCATION> or
> >>> ${T}/doc contains subdirs.
> 
> Whereas the following:
> 
>   emake DESTDIR="${D}" docdir="${T}/doc"
>   dodoc "${T}/doc"
should be dodoc "${T}/doc/*"

> 
> will not work, even if ${T}/doc contains only files.
Yes, but prepalldocs does it's "magic" on the complete /usr/share/doc.
How about that?

-- 
-------------------------------------------------------
Tiziano Müller
Gentoo Linux Developer, Council Member
Areas of responsibility:
  Samba, PostgreSQL, CPP, Python, sysadmin
E-Mail     : dev-zero@gentoo.org
GnuPG FP   : F327 283A E769 2E36 18D5  4DE2 1B05 6A63 AE9C 1E30

[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] prepalldocs implementation in eutils.eclass (was: prepalldocs is now banned)

[Tiziano Müller]

[-- Attachment #1: Type: text/plain, Size: 764 bytes --]

Am Donnerstag, den 19.02.2009, 10:37 +0100 schrieb Ulrich Mueller:
> >>>>> On Thu, 19 Feb 2009, Tiziano Müller wrote:
> 
> > Instead of breaking things you could have tried to come up with a
> > real solution.
> 
> This could be said about the council's decision as well.
> 
Right. Two seconds after the council meeting every user got a broken
tree because we blindly removed prepalldocs.
Or, wait, removing prepalldocs wouldn't break anything and we didn't.

-- 
-------------------------------------------------------
Tiziano Müller
Gentoo Linux Developer, Council Member
Areas of responsibility:
  Samba, PostgreSQL, CPP, Python, sysadmin
E-Mail     : dev-zero@gentoo.org
GnuPG FP   : F327 283A E769 2E36 18D5  4DE2 1B05 6A63 AE9C 1E30

[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] prepalldocs implementation in eutils.eclass (was: prepalldocs is now banned)

[Ulrich Mueller]

>>>>> On Thu, 19 Feb 2009, Tiziano Müller wrote:

>> emake DESTDIR="${D}" docdir="${T}/doc"
>> dodoc "${T}/doc"

> should be dodoc "${T}/doc/*"

O.K., that looks better.

However, you still copy files around, for the sole purpose to enable
dodoc's "magic" side effect (which also is not mentioned in PMS, by
the way). IMHO, this is only a workaround, not a solution of the
problem.

Ulrich

Re: [gentoo-dev] prepalldocs implementation in eutils.eclass (was: prepalldocs is now banned)

[Matti Bickel]

[-- Attachment #1: Type: text/plain, Size: 656 bytes --]

Tiziano Müller <dev-zero@gentoo.org> wrote:
> The only problem I see here is that either <OLDLOCATION> or ${T}/doc
> contains subdirs. So my proposal for the next EAPI is to allow dodoc and
> newdoc to operate on dirs. Which also gives the benefit to reduce this
> idiom:
>   insinto /usr/share/doc/${PF}
>   doins -r examples
> to:
>   dodoc examples
> 
> 
> Your comments?

Yes, please. It will simplify dozens of ebuilds and feels 'natural' to
me. Keeping with doins, etc. i would propose to make it

    dodoc -r $something

And thanks for your analysis.
-- 
Regards, Matti Bickel
Signed/Encrypted email preferred (key 4849EC6C)

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Proposal for compression handling (was: Re: prepalldocs is now banned)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1383 bytes --]

On Thu, 19 Feb 2009 12:06:17 +0100
Ulrich Mueller <ulm@gentoo.org> wrote:
> Package managers supporting compression would internally maintain two
> lists of paths, both having a default value, plus the possibility to
> modify them from ebuilds/eclasses:
> 
>    - an inclusion list, initially containing:
>      /usr/share/doc /usr/share/info /usr/share/man
>    - an exclusion list, initially containing:
>      /usr/share/doc/${PF}/html
> 
> Then we need a new command, let's call it "docompress" for now (name
> to be discussed), to manipulate the lists:
> 
>    - "docompress" (without option or with option "-a"): add paths
>       (directories or files) to the inclusion list
>    - "docompress -d": delete paths from the inclusion list
>    - "docompress -x": add paths to the exclusion list
>    - "docompress -x -d": remove paths from the exclusion list
>      (maybe this is not needed?)

If there's an inclusion list and an exclusion list, there's only a need
to delete things from the exclusion list if the exclusion list has bad
initial values, and never any need to delete things from the inclusion
list. And since reliably rewriting links in HTML when compressing is at
best tricky and at worst impossible (think JavaScript navigationy
things), is there ever going to be any need for removing from the
exclusion list?
-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Proposal for compression handling

[Ulrich Mueller]

>>>>> On Thu, 19 Feb 2009, Ciaran McCreesh wrote:

> If there's an inclusion list and an exclusion list, there's only a
> need to delete things from the exclusion list if the exclusion list
> has bad initial values, and never any need to delete things from the
> inclusion list. And since reliably rewriting links in HTML when
> compressing is at best tricky and at worst impossible (think
> JavaScript navigationy things), is there ever going to be any need
> for removing from the exclusion list?

Right, probably there isn't. So we will need only two options:

   - "docompress" (without option or with option "-a"): add paths
     (directories or files) to the inclusion list
   - "docompress -x": add paths to the exclusion list

Ulrich

Re: [gentoo-dev] Proposal for compression handling

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 963 bytes --]

On Thu, Feb 19, 2009 at 11:03:28PM +0100, Ulrich Mueller wrote:
> >>>>> On Thu, 19 Feb 2009, Ciaran McCreesh wrote:
> 
> > If there's an inclusion list and an exclusion list, there's only a
> > need to delete things from the exclusion list if the exclusion list
> > has bad initial values, and never any need to delete things from the
> > inclusion list. And since reliably rewriting links in HTML when
> > compressing is at best tricky and at worst impossible (think
> > JavaScript navigationy things), is there ever going to be any need
> > for removing from the exclusion list?
> 
> Right, probably there isn't. So we will need only two options:
> 
>    - "docompress" (without option or with option "-a"): add paths
>      (directories or files) to the inclusion list
>    - "docompress -x": add paths to the exclusion list

Globbing support?  Also I assume compliant implementations do their 
updating between install/preinst?

~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Proposal for compression handling

[Ulrich Mueller]

>>>>> On Thu, 19 Feb 2009, Brian Harring wrote:

>> - "docompress" (without option or with option "-a"): add paths
>> (directories or files) to the inclusion list
>> - "docompress -x": add paths to the exclusion list

> Globbing support?

No, I would say. The other do* commands don't have it, and
"docompress -x foo*" will work anyway since the argument is expanded
before calling the function.

> Also I assume compliant implementations do their updating between
> install/preinst?

Indeed, that's what I meant. ;-) Thanks.

Ulrich

Re: [gentoo-dev] Proposal for compression handling

[Zac Medico]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Ulrich Mueller wrote:
>>>>>> On Thu, 19 Feb 2009, Brian Harring wrote:
> 
>>> - "docompress" (without option or with option "-a"): add paths
>>> (directories or files) to the inclusion list
>>> - "docompress -x": add paths to the exclusion list
> 
>> Globbing support?
> 
> No, I would say. The other do* commands don't have it, and
> "docompress -x foo*" will work anyway since the argument is expanded
> before calling the function.

Keep in mind that using a glob that way is going to mean that
docompress has to join each path with the current directory and then
strip $D from the left side. Doing it that way complicates the
implementation details a bit, and it's not quite as flexible as
regular expression support would be. With regular expression support
you could even control which file extensions are compressed or not.
- --
Thanks,
Zac
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.10 (GNU/Linux)

iEYEARECAAYFAkmgibEACgkQ/ejvha5XGaO3IwCfW+qTR7yMYo1D7XYk6QnUsM+s
4hYAoL4efSmN/xxZog1RGLuWtIqaN/h3
=YWcZ
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Proposal for compression handling

[Ulrich Mueller]

>>>>> On Sat, 21 Feb 2009, Zac Medico wrote:

> Keep in mind that using a glob that way is going to mean that
> docompress has to join each path with the current directory and
> then strip $D from the left side. Doing it that way complicates the
> implementation details a bit, and it's not quite as flexible as
> regular expression support would be. With regular expression support
> you could even control which file extensions are compressed or not.

Following our discussion in #-portage and after thinking about it:
I assume that there would be very few cases where globbing or regexp
support is required. And these could always be handled by the ebuild
itself or by an eclass function.

So, we should follow the KISS principle here, and allow only literal
paths (files or directories) as arguments for docompress.

Ulrich