public inbox for gentoo-portage-dev@lists.gentoo.org
 help / color / mirror / Atom feed
* [gentoo-portage-dev] xattr wrapper for install, bug #465000
@ 2014-01-26 22:53 Anthony G. Basile
  2014-01-27 14:02 ` vivo75
  2014-01-31  4:43 ` Mike Frysinger
  0 siblings, 2 replies; 8+ messages in thread
From: Anthony G. Basile @ 2014-01-26 22:53 UTC (permalink / raw
  To: gentoo-portage-dev

Hi everyone,

A while back, I wrote a python wrapper for install to preserve xattrs. 
Its installed in LIBDIR/portage/bin/install.py.  It is *painfully* slow. 
  For a package like moodle with 16650 .php files, none of which 
probably need any xattr's set, it takes about 30 mins to install.  I 
rewrote the wrapper in C.  Replacing the python wrapper with the C 
wrapper, the same example reduces from about 30 mins to 2 mins.

Mike and I did some back and forth about how best to write it.  The 
latest version is pretty much done at

https://bugs.gentoo.org/show_bug.cgi?id=465000#c56

We need to get that integrated into portage.  1) I'm not 100% sure how 
to do that.  2) We may want to install it at /usr/bin/install-xattr 
because I'm sure it will be useful for more than just portage.

Comments?

-- 
Anthony G. Basile, Ph. D.
Chair of Information Technology
D'Youville College
Buffalo, NY 14201
(716) 829-8197


^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [gentoo-portage-dev] xattr wrapper for install, bug #465000
  2014-01-26 22:53 [gentoo-portage-dev] xattr wrapper for install, bug #465000 Anthony G. Basile
@ 2014-01-27 14:02 ` vivo75
  2014-01-28  4:36   ` Mike Frysinger
  2014-01-29 16:33   ` Anthony G. Basile
  2014-01-31  4:43 ` Mike Frysinger
  1 sibling, 2 replies; 8+ messages in thread
From: vivo75 @ 2014-01-27 14:02 UTC (permalink / raw
  To: gentoo-portage-dev

On 01/26/14 23:53, Anthony G. Basile wrote:
> Hi everyone,
>
> A while back, I wrote a python wrapper for install to preserve xattrs.
> Its installed in LIBDIR/portage/bin/install.py.  It is *painfully*
> slow.  For a package like moodle with 16650 .php files, none of which
> probably need any xattr's set, it takes about 30 mins to install.  I
> rewrote the wrapper in C.  Replacing the python wrapper with the C
> wrapper, the same example reduces from about 30 mins to 2 mins.
>
> Mike and I did some back and forth about how best to write it.  The
> latest version is pretty much done at
>
> https://bugs.gentoo.org/show_bug.cgi?id=465000#c56
>
> We need to get that integrated into portage.  1) I'm not 100% sure how
> to do that.  2) We may want to install it at /usr/bin/install-xattr
> because I'm sure it will be useful for more than just portage.
>
> Comments?
>
patch install from coreutils (and then upstream changes) is not an option?
they already support selinux contexts

anyway install-xattr could be useful and /usr/bin would be a good option
IMHO


^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [gentoo-portage-dev] xattr wrapper for install, bug #465000
  2014-01-27 14:02 ` vivo75
@ 2014-01-28  4:36   ` Mike Frysinger
  2014-01-29 16:33   ` Anthony G. Basile
  1 sibling, 0 replies; 8+ messages in thread
From: Mike Frysinger @ 2014-01-28  4:36 UTC (permalink / raw
  To: gentoo-portage-dev

[-- Attachment #1: Type: text/plain, Size: 266 bytes --]

On Monday, January 27, 2014 15:02:30 vivo75@gmail.com wrote:
> patch install from coreutils (and then upstream changes) is not an option?

that route is being pursued independently.  we already have a wrapper and will 
have one for the foreseeable future.
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [gentoo-portage-dev] xattr wrapper for install, bug #465000
  2014-01-27 14:02 ` vivo75
  2014-01-28  4:36   ` Mike Frysinger
@ 2014-01-29 16:33   ` Anthony G. Basile
  2014-01-29 23:50     ` Francesco R.
  1 sibling, 1 reply; 8+ messages in thread
From: Anthony G. Basile @ 2014-01-29 16:33 UTC (permalink / raw
  To: gentoo-portage-dev

On 01/27/2014 09:02 AM, vivo75@gmail.com wrote:
> On 01/26/14 23:53, Anthony G. Basile wrote:
>> Hi everyone,
>>
>> A while back, I wrote a python wrapper for install to preserve xattrs.
>> Its installed in LIBDIR/portage/bin/install.py.  It is *painfully*
>> slow.  For a package like moodle with 16650 .php files, none of which
>> probably need any xattr's set, it takes about 30 mins to install.  I
>> rewrote the wrapper in C.  Replacing the python wrapper with the C
>> wrapper, the same example reduces from about 30 mins to 2 mins.
>>
>> Mike and I did some back and forth about how best to write it.  The
>> latest version is pretty much done at
>>
>> https://bugs.gentoo.org/show_bug.cgi?id=465000#c56
>>
>> We need to get that integrated into portage.  1) I'm not 100% sure how
>> to do that.  2) We may want to install it at /usr/bin/install-xattr
>> because I'm sure it will be useful for more than just portage.
>>
>> Comments?
>>
> patch install from coreutils (and then upstream changes) is not an option?
> they already support selinux contexts
>
> anyway install-xattr could be useful and /usr/bin would be a good option
> IMHO
>

Been there and I even had a patch ready.  Upstream answer was '\0'.  The 
only people who engaged the discussion were gentoo devs.

Would patching coreutils have been the better approach?  Some people 
might argue install and cp and mv etc should just copy contents to keep 
these utilities as simple as possible.  Although, as you say, install 
can copy selinux contexts, and cp can copy xattr attributes.  So what's 
the problem with extending installs functionality to include arbitrary 
xattr attributes?

Anyhow, seeing as upstream is uninterested, I prefer this wrapper to 
maintaining a local patch against coreutils.


-- 
Anthony G. Basile, Ph. D.
Chair of Information Technology
D'Youville College
Buffalo, NY 14201
(716) 829-8197


^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [gentoo-portage-dev] xattr wrapper for install, bug #465000
  2014-01-29 16:33   ` Anthony G. Basile
@ 2014-01-29 23:50     ` Francesco R.
  0 siblings, 0 replies; 8+ messages in thread
From: Francesco R. @ 2014-01-29 23:50 UTC (permalink / raw
  To: gentoo-portage-dev

Il 29/01/2014 17:33, Anthony G. Basile ha scritto:
> On 01/27/2014 09:02 AM, vivo75@gmail.com wrote:
>> On 01/26/14 23:53, Anthony G. Basile wrote:
>>> Hi everyone,
>>>
>>> A while back, I wrote a python wrapper for install to preserve xattrs.
>>> Its installed in LIBDIR/portage/bin/install.py.  It is *painfully*
>>> slow.  For a package like moodle with 16650 .php files, none of which
>>> probably need any xattr's set, it takes about 30 mins to install.  I
>>> rewrote the wrapper in C.  Replacing the python wrapper with the C
>>> wrapper, the same example reduces from about 30 mins to 2 mins.
>>>
>>> Mike and I did some back and forth about how best to write it.  The
>>> latest version is pretty much done at
>>>
>>> https://bugs.gentoo.org/show_bug.cgi?id=465000#c56
>>>
>>> We need to get that integrated into portage.  1) I'm not 100% sure how
>>> to do that.  2) We may want to install it at /usr/bin/install-xattr
>>> because I'm sure it will be useful for more than just portage.
>>>
>>> Comments?
>>>
>> patch install from coreutils (and then upstream changes) is not an
>> option?
>> they already support selinux contexts
>>
>> anyway install-xattr could be useful and /usr/bin would be a good option
>> IMHO
>>
>
> Been there and I even had a patch ready.  Upstream answer was '\0'. 
> The only people who engaged the discussion were gentoo devs.
>
> Would patching coreutils have been the better approach?  Some people
> might argue install and cp and mv etc should just copy contents to
> keep these utilities as simple as possible.  Although, as you say,
> install can copy selinux contexts, and cp can copy xattr attributes. 
> So what's the problem with extending installs functionality to include
> arbitrary xattr attributes?
>
> Anyhow, seeing as upstream is uninterested, I prefer this wrapper to
> maintaining a local patch against coreutils.
>
>
thanks for answering, so yes the wrapper seem the only sensible thing,
good luck, since there are no other questions in 3 days the plan must be
perfect ;)

on a totally unrelated and ipotetical note kde is discussing a metadata
engine that use xattr instead of databases, so this could become
important even for desktops



^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [gentoo-portage-dev] xattr wrapper for install, bug #465000
  2014-01-26 22:53 [gentoo-portage-dev] xattr wrapper for install, bug #465000 Anthony G. Basile
  2014-01-27 14:02 ` vivo75
@ 2014-01-31  4:43 ` Mike Frysinger
  2014-01-31  4:53   ` Brian Dolbec
  1 sibling, 1 reply; 8+ messages in thread
From: Mike Frysinger @ 2014-01-31  4:43 UTC (permalink / raw
  To: gentoo-portage-dev

[-- Attachment #1: Type: text/plain, Size: 661 bytes --]

On Sunday, January 26, 2014 17:53:44 Anthony G. Basile wrote:
> 1) I'm not 100% sure how to do that.

we'll have to (re-)introduce support for building C files.  shouldn't be too 
hard to start a new directory to hold all the C code.  and a simple Makefile 
to build them all.  i think the tbz2tool.c was the only C program we had 
before and it was compiled via the ebuild (which we should avoid).

> 2) We may want to install it at /usr/bin/install-xattr
> because I'm sure it will be useful for more than just portage.

if that's the case, then i don't think it belongs in portage.  make it a sep 
package that portage will just depend on.
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [gentoo-portage-dev] xattr wrapper for install, bug #465000
  2014-01-31  4:43 ` Mike Frysinger
@ 2014-01-31  4:53   ` Brian Dolbec
  2014-01-31 19:55     ` Greg Turner
  0 siblings, 1 reply; 8+ messages in thread
From: Brian Dolbec @ 2014-01-31  4:53 UTC (permalink / raw
  To: gentoo-portage-dev

On Thu, 30 Jan 2014 23:43:53 -0500
Mike Frysinger <vapier@gentoo.org> wrote:

> On Sunday, January 26, 2014 17:53:44 Anthony G. Basile wrote:
> > 1) I'm not 100% sure how to do that.
> 
> we'll have to (re-)introduce support for building C files.  shouldn't
> be too hard to start a new directory to hold all the C code.  and a
> simple Makefile to build them all.  i think the tbz2tool.c was the
> only C program we had before and it was compiled via the ebuild
> (which we should avoid).
> 
> > 2) We may want to install it at /usr/bin/install-xattr
> > because I'm sure it will be useful for more than just portage.
> 
> if that's the case, then i don't think it belongs in portage.  make
> it a sep package that portage will just depend on.
> -mike

Yes, I agree, make it a separate pkg.  That way it is useful for other
pkgs and utilities.  I am doing the same for several other small
utility libs to avoid copy/pasting the same code over and over again.

Besides, portage will have the python code as an emergency backup.

-- 
Brian Dolbec <dolsen>



^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [gentoo-portage-dev] xattr wrapper for install, bug #465000
  2014-01-31  4:53   ` Brian Dolbec
@ 2014-01-31 19:55     ` Greg Turner
  0 siblings, 0 replies; 8+ messages in thread
From: Greg Turner @ 2014-01-31 19:55 UTC (permalink / raw
  To: gentoo-portage-dev

On Thu, Jan 30, 2014 at 8:53 PM, Brian Dolbec <dolsen@gentoo.org> wrote:
> On Thu, 30 Jan 2014 23:43:53 -0500 Mike Frysinger <vapier@gentoo.org> wrote:
>> On Sunday, January 26, 2014 17:53:44 Anthony G. Basile wrote:
[snip]
>> > 2) We may want to install it at /usr/bin/install-xattr
>> > because I'm sure it will be useful for more than just portage.
>>
>> if that's the case, then i don't think it belongs in portage.  make
>> it a sep package that portage will just depend on.
>> -mike
>
> Yes, I agree, make it a separate pkg.  That way it is useful for other
> pkgs and utilities.  I am doing the same for several other small
> utility libs to avoid copy/pasting the same code over and over again.

I took a crack at this:

https://github.com/gmt/xattr-install
https://github.com/gmt/portage
https://github.com/gmt/xattr-overlay

more details are in bug 465000

-gmt


^ permalink raw reply	[flat|nested] 8+ messages in thread

end of thread, other threads:[~2014-01-31 19:55 UTC | newest]

Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2014-01-26 22:53 [gentoo-portage-dev] xattr wrapper for install, bug #465000 Anthony G. Basile
2014-01-27 14:02 ` vivo75
2014-01-28  4:36   ` Mike Frysinger
2014-01-29 16:33   ` Anthony G. Basile
2014-01-29 23:50     ` Francesco R.
2014-01-31  4:43 ` Mike Frysinger
2014-01-31  4:53   ` Brian Dolbec
2014-01-31 19:55     ` Greg Turner

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox