[gentoo-portage-dev] xattr wrapper for install, bug #465000

[gentoo-portage-dev] xattr wrapper for install, bug #465000

[Anthony G. Basile]

Hi everyone,

A while back, I wrote a python wrapper for install to preserve xattrs. 
Its installed in LIBDIR/portage/bin/install.py.  It is *painfully* slow. 
  For a package like moodle with 16650 .php files, none of which 
probably need any xattr's set, it takes about 30 mins to install.  I 
rewrote the wrapper in C.  Replacing the python wrapper with the C 
wrapper, the same example reduces from about 30 mins to 2 mins.

Mike and I did some back and forth about how best to write it.  The 
latest version is pretty much done at

https://bugs.gentoo.org/show_bug.cgi?id=465000#c56

We need to get that integrated into portage.  1) I'm not 100% sure how 
to do that.  2) We may want to install it at /usr/bin/install-xattr 
because I'm sure it will be useful for more than just portage.

Comments?

-- 
Anthony G. Basile, Ph. D.
Chair of Information Technology
D'Youville College
Buffalo, NY 14201
(716) 829-8197

Re: [gentoo-portage-dev] xattr wrapper for install, bug #465000

[vivo75]

On 01/26/14 23:53, Anthony G. Basile wrote:
> Hi everyone,
>
> A while back, I wrote a python wrapper for install to preserve xattrs.
> Its installed in LIBDIR/portage/bin/install.py.  It is *painfully*
> slow.  For a package like moodle with 16650 .php files, none of which
> probably need any xattr's set, it takes about 30 mins to install.  I
> rewrote the wrapper in C.  Replacing the python wrapper with the C
> wrapper, the same example reduces from about 30 mins to 2 mins.
>
> Mike and I did some back and forth about how best to write it.  The
> latest version is pretty much done at
>
> https://bugs.gentoo.org/show_bug.cgi?id=465000#c56
>
> We need to get that integrated into portage.  1) I'm not 100% sure how
> to do that.  2) We may want to install it at /usr/bin/install-xattr
> because I'm sure it will be useful for more than just portage.
>
> Comments?
>
patch install from coreutils (and then upstream changes) is not an option?
they already support selinux contexts

anyway install-xattr could be useful and /usr/bin would be a good option
IMHO

Re: [gentoo-portage-dev] xattr wrapper for install, bug #465000

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 266 bytes --]

On Monday, January 27, 2014 15:02:30 vivo75@gmail.com wrote:
> patch install from coreutils (and then upstream changes) is not an option?

that route is being pursued independently.  we already have a wrapper and will 
have one for the foreseeable future.
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-portage-dev] xattr wrapper for install, bug #465000

[Anthony G. Basile]

On 01/27/2014 09:02 AM, vivo75@gmail.com wrote:
> On 01/26/14 23:53, Anthony G. Basile wrote:
>> Hi everyone,
>>
>> A while back, I wrote a python wrapper for install to preserve xattrs.
>> Its installed in LIBDIR/portage/bin/install.py.  It is *painfully*
>> slow.  For a package like moodle with 16650 .php files, none of which
>> probably need any xattr's set, it takes about 30 mins to install.  I
>> rewrote the wrapper in C.  Replacing the python wrapper with the C
>> wrapper, the same example reduces from about 30 mins to 2 mins.
>>
>> Mike and I did some back and forth about how best to write it.  The
>> latest version is pretty much done at
>>
>> https://bugs.gentoo.org/show_bug.cgi?id=465000#c56
>>
>> We need to get that integrated into portage.  1) I'm not 100% sure how
>> to do that.  2) We may want to install it at /usr/bin/install-xattr
>> because I'm sure it will be useful for more than just portage.
>>
>> Comments?
>>
> patch install from coreutils (and then upstream changes) is not an option?
> they already support selinux contexts
>
> anyway install-xattr could be useful and /usr/bin would be a good option
> IMHO
>

Been there and I even had a patch ready.  Upstream answer was '\0'.  The 
only people who engaged the discussion were gentoo devs.

Would patching coreutils have been the better approach?  Some people 
might argue install and cp and mv etc should just copy contents to keep 
these utilities as simple as possible.  Although, as you say, install 
can copy selinux contexts, and cp can copy xattr attributes.  So what's 
the problem with extending installs functionality to include arbitrary 
xattr attributes?

Anyhow, seeing as upstream is uninterested, I prefer this wrapper to 
maintaining a local patch against coreutils.


-- 
Anthony G. Basile, Ph. D.
Chair of Information Technology
D'Youville College
Buffalo, NY 14201
(716) 829-8197

Re: [gentoo-portage-dev] xattr wrapper for install, bug #465000

[Francesco R.]

Il 29/01/2014 17:33, Anthony G. Basile ha scritto:
> On 01/27/2014 09:02 AM, vivo75@gmail.com wrote:
>> On 01/26/14 23:53, Anthony G. Basile wrote:
>>> Hi everyone,
>>>
>>> A while back, I wrote a python wrapper for install to preserve xattrs.
>>> Its installed in LIBDIR/portage/bin/install.py.  It is *painfully*
>>> slow.  For a package like moodle with 16650 .php files, none of which
>>> probably need any xattr's set, it takes about 30 mins to install.  I
>>> rewrote the wrapper in C.  Replacing the python wrapper with the C
>>> wrapper, the same example reduces from about 30 mins to 2 mins.
>>>
>>> Mike and I did some back and forth about how best to write it.  The
>>> latest version is pretty much done at
>>>
>>> https://bugs.gentoo.org/show_bug.cgi?id=465000#c56
>>>
>>> We need to get that integrated into portage.  1) I'm not 100% sure how
>>> to do that.  2) We may want to install it at /usr/bin/install-xattr
>>> because I'm sure it will be useful for more than just portage.
>>>
>>> Comments?
>>>
>> patch install from coreutils (and then upstream changes) is not an
>> option?
>> they already support selinux contexts
>>
>> anyway install-xattr could be useful and /usr/bin would be a good option
>> IMHO
>>
>
> Been there and I even had a patch ready.  Upstream answer was '\0'. 
> The only people who engaged the discussion were gentoo devs.
>
> Would patching coreutils have been the better approach?  Some people
> might argue install and cp and mv etc should just copy contents to
> keep these utilities as simple as possible.  Although, as you say,
> install can copy selinux contexts, and cp can copy xattr attributes. 
> So what's the problem with extending installs functionality to include
> arbitrary xattr attributes?
>
> Anyhow, seeing as upstream is uninterested, I prefer this wrapper to
> maintaining a local patch against coreutils.
>
>
thanks for answering, so yes the wrapper seem the only sensible thing,
good luck, since there are no other questions in 3 days the plan must be
perfect ;)

on a totally unrelated and ipotetical note kde is discussing a metadata
engine that use xattr instead of databases, so this could become
important even for desktops

Re: [gentoo-portage-dev] xattr wrapper for install, bug #465000

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 661 bytes --]

On Sunday, January 26, 2014 17:53:44 Anthony G. Basile wrote:
> 1) I'm not 100% sure how to do that.

we'll have to (re-)introduce support for building C files.  shouldn't be too 
hard to start a new directory to hold all the C code.  and a simple Makefile 
to build them all.  i think the tbz2tool.c was the only C program we had 
before and it was compiled via the ebuild (which we should avoid).

> 2) We may want to install it at /usr/bin/install-xattr
> because I'm sure it will be useful for more than just portage.

if that's the case, then i don't think it belongs in portage.  make it a sep 
package that portage will just depend on.
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-portage-dev] xattr wrapper for install, bug #465000

[Brian Dolbec]

On Thu, 30 Jan 2014 23:43:53 -0500
Mike Frysinger <vapier@gentoo.org> wrote:

> On Sunday, January 26, 2014 17:53:44 Anthony G. Basile wrote:
> > 1) I'm not 100% sure how to do that.
> 
> we'll have to (re-)introduce support for building C files.  shouldn't
> be too hard to start a new directory to hold all the C code.  and a
> simple Makefile to build them all.  i think the tbz2tool.c was the
> only C program we had before and it was compiled via the ebuild
> (which we should avoid).
> 
> > 2) We may want to install it at /usr/bin/install-xattr
> > because I'm sure it will be useful for more than just portage.
> 
> if that's the case, then i don't think it belongs in portage.  make
> it a sep package that portage will just depend on.
> -mike

Yes, I agree, make it a separate pkg.  That way it is useful for other
pkgs and utilities.  I am doing the same for several other small
utility libs to avoid copy/pasting the same code over and over again.

Besides, portage will have the python code as an emergency backup.

-- 
Brian Dolbec <dolsen>

Re: [gentoo-portage-dev] xattr wrapper for install, bug #465000

[Greg Turner]

On Thu, Jan 30, 2014 at 8:53 PM, Brian Dolbec <dolsen@gentoo.org> wrote:
> On Thu, 30 Jan 2014 23:43:53 -0500 Mike Frysinger <vapier@gentoo.org> wrote:
>> On Sunday, January 26, 2014 17:53:44 Anthony G. Basile wrote:
[snip]
>> > 2) We may want to install it at /usr/bin/install-xattr
>> > because I'm sure it will be useful for more than just portage.
>>
>> if that's the case, then i don't think it belongs in portage.  make
>> it a sep package that portage will just depend on.
>> -mike
>
> Yes, I agree, make it a separate pkg.  That way it is useful for other
> pkgs and utilities.  I am doing the same for several other small
> utility libs to avoid copy/pasting the same code over and over again.

I took a crack at this:

https://github.com/gmt/xattr-install
https://github.com/gmt/portage
https://github.com/gmt/xattr-overlay

more details are in bug 465000

-gmt