[gentoo-dev] RFC : New ebuild function pkg_create for creating corespondent sorce tarball

[gentoo-dev] RFC : New ebuild function pkg_create for creating corespondent sorce tarball

[Alin Năstac]

[-- Attachment #1: Type: text/plain, Size: 590 bytes --]

I was asked to discuss here a portage enhancement proposed by me [1].

Basically I need a pkg_create() that will be executed only in the
context of the upcoming "ebuild ${PF}.ebuild create" command.

The package where I need it is app-mobilephone/bitpim. The upstream doesn't
offer a source tarball, so I need to construct it myself from their svn
repository. Up till recently, I used some hackery in pkg_setup() to
create it
(see the ebuild), but now "ebuild $PF.ebuild setup" verify the digest before
running pkg_setup().

[1] http://bugs.gentoo.org/show_bug.cgi?id=185567


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 249 bytes --]

Re: [gentoo-dev] RFC : New ebuild function pkg_create for creating corespondent sorce tarball

[Marius Mauch]

[-- Attachment #1: Type: text/plain, Size: 1100 bytes --]

On Tue, 17 Jul 2007 07:25:02 +0300
Alin Năstac <mrness@gentoo.org> wrote:

> I was asked to discuss here a portage enhancement proposed by me [1].
> 
> Basically I need a pkg_create() that will be executed only in the
> context of the upcoming "ebuild ${PF}.ebuild create" command.
> 
> The package where I need it is app-mobilephone/bitpim. The upstream
> doesn't offer a source tarball, so I need to construct it myself from
> their svn repository. Up till recently, I used some hackery in
> pkg_setup() to create it
> (see the ebuild), but now "ebuild $PF.ebuild setup" verify the digest
> before running pkg_setup().
> 
> [1] http://bugs.gentoo.org/show_bug.cgi?id=185567

Two questions:
- are there more packages that could benefit from this?
- is there a particular reason this has to be integrated into the
ebuild and should not be handled by an ordinary script?

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] RFC : New ebuild function pkg_create for creating corespondent sorce tarball

[Alin Năstac]

[-- Attachment #1: Type: text/plain, Size: 917 bytes --]

Marius Mauch wrote:
> Two questions:
> - are there more packages that could benefit from this?
>   

None that I know of. However, there might be other similar packages
without a source tarball (slim chance, but quite possible). At first, I
asked upstream to provide such tarball, but I got refused because
"SourceForge file release process is far too annoying".
As a side note, if bitpim wasn't such a fairly popular package, I
wouldn't even bother with it (personally I don't use it).

> - is there a particular reason this has to be integrated into the
> ebuild and should not be handled by an ordinary script?
>   

There are 2 reasons:
  a) convenience - no need to pass version to the script
  b) maintainability - easy to take over when I will be gone

P.S: The name proposed by me isn't exactly right, as Mike already
remarked on the bug. I suggest to use src_create as function name.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 249 bytes --]

Re: [gentoo-dev] RFC : New ebuild function pkg_create for creating corespondent sorce tarball

[Petteri Räty]

[-- Attachment #1: Type: text/plain, Size: 611 bytes --]

Alin Năstac kirjoitti:
> Marius Mauch wrote:
>> Two questions:
>> - are there more packages that could benefit from this?
>>   
> 
> None that I know of. However, there might be other similar packages
> without a source tarball (slim chance, but quite possible). At first, I
> asked upstream to provide such tarball, but I got refused because
> "SourceForge file release process is far too annoying".
> As a side note, if bitpim wasn't such a fairly popular package, I
> wouldn't even bother with it (personally I don't use it).
> 

This is not uncommon in the Java land.

Regards,
Petteri


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: [gentoo-dev] RFC : New ebuild function pkg_create for creating corespondent sorce tarball

[Stelian Ionescu]

[-- Attachment #1: Type: text/plain, Size: 824 bytes --]

On Tue, Jul 17, 2007 at 09:41:04AM +0300, Petteri Räty wrote:
>Alin Năstac kirjoitti:
>> Marius Mauch wrote:
>>> Two questions:
>>> - are there more packages that could benefit from this?
>>>   
>> 
>> None that I know of. However, there might be other similar packages
>> without a source tarball (slim chance, but quite possible). At first, I
>> asked upstream to provide such tarball, but I got refused because
>> "SourceForge file release process is far too annoying".
>> As a side note, if bitpim wasn't such a fairly popular package, I
>> wouldn't even bother with it (personally I don't use it).
>> 
>
>This is not uncommon in the Java land.
it's also quite common in CommonLisp land

-- 
(sign :name "Stelian Ionescu" :aka "fe[nl]ix"
      :quote "Quidquid latine dictum sit, altum videtur.")

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] RFC : New ebuild function pkg_create for creating corespondent sorce tarball

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 382 bytes --]

On Tuesday 17 July 2007, Alin Năstac wrote:
> P.S: The name proposed by me isn't exactly right, as Mike already
> remarked on the bug. I suggest to use src_create as function name.

Marius proposed src_makedist which is pretty good ... i wonder though if there 
may be a better prefix than src_ or pkg_ considering neither quite fit into 
this realm of expansion ...
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 827 bytes --]

Re: [gentoo-dev] RFC : New ebuild function pkg_create for creating corespondent sorce tarball

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 1373 bytes --]

On Tuesday 17 July 2007, Marius Mauch wrote:
> Alin Năstac <mrness@gentoo.org> wrote:
> > I was asked to discuss here a portage enhancement proposed by me [1].
> >
> > Basically I need a pkg_create() that will be executed only in the
> > context of the upcoming "ebuild ${PF}.ebuild create" command.
> >
> > The package where I need it is app-mobilephone/bitpim. The upstream
> > doesn't offer a source tarball, so I need to construct it myself from
> > their svn repository. Up till recently, I used some hackery in
> > pkg_setup() to create it
> > (see the ebuild), but now "ebuild $PF.ebuild setup" verify the digest
> > before running pkg_setup().
> >
> > [1] http://bugs.gentoo.org/show_bug.cgi?id=185567
>
> Two questions:
> - are there more packages that could benefit from this?
> - is there a particular reason this has to be integrated into the
> ebuild and should not be handled by an ordinary script?

i'll have to ponder for a while, but in the past, i could have made use of 
something like this ... lacking it, i had hand written scripts to do it for 
me.  every case where i needed this though it was a similar situation -- 
upstream did not provide a release, but did provide SCM access.  actually, 
netpbm is another example of where this could be utilized as they only tag 
releases in their svn, not roll tarballs ...
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 827 bytes --]

Re: [gentoo-dev] RFC : New ebuild function pkg_create for creating corespondent sorce tarball

[Luca Barbato]

Marius Mauch wrote:
> On Tue, 17 Jul 2007 07:25:02 +0300
> Alin Năstac <mrness@gentoo.org> wrote:
> 
>> I was asked to discuss here a portage enhancement proposed by me [1].
>>
>> Basically I need a pkg_create() that will be executed only in the
>> context of the upcoming "ebuild ${PF}.ebuild create" command.
>>
>> The package where I need it is app-mobilephone/bitpim. The upstream
>> doesn't offer a source tarball, so I need to construct it myself from
>> their svn repository. Up till recently, I used some hackery in
>> pkg_setup() to create it
>> (see the ebuild), but now "ebuild $PF.ebuild setup" verify the digest
>> before running pkg_setup().
>>
>> [1] http://bugs.gentoo.org/show_bug.cgi?id=185567
> 
> Two questions:
> - are there more packages that could benefit from this?

ffmpeg, mplayer, libdvdread, mlt...

> - is there a particular reason this has to be integrated into the
> ebuild and should not be handled by an ordinary script?

overall nicer

lu

-- 

Luca Barbato

Gentoo/linux Gentoo/PPC
http://dev.gentoo.org/~lu_zero
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] RFC : New ebuild function pkg_create for creating corespondent sorce tarball

[Elias Probst]

[-- Attachment #1: Type: text/plain, Size: 488 bytes --]

On Tuesday 17 July 2007 09:57:53 Luca Barbato wrote:
> Marius Mauch wrote:
> > Two questions:
> > - are there more packages that could benefit from this?
>
> ffmpeg, mplayer, libdvdread, mlt...

Finally, this bug could be closed too, since the development in SVN is 
proceeded very far, but no versioned tarball was released yet.
http://bugs.gentoo.org/show_bug.cgi?id=139432

Regards, Elias P.

-- 
A really nice number:
"09:F9:11:02:9D:74:E3:5B:D8:41:56:C5:63:56:88:C0"

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] RFC : New ebuild function pkg_create for creating corespondent sorce tarball

[Donnie Berkholz]

[-- Attachment #1: Type: text/plain, Size: 1450 bytes --]

On Tue, 17 Jul 2007 06:37:43 +0200
Marius Mauch <genone@gentoo.org> wrote:

> On Tue, 17 Jul 2007 07:25:02 +0300
> Alin Năstac <mrness@gentoo.org> wrote:
> 
> > I was asked to discuss here a portage enhancement proposed by me
> > [1].
> > 
> > Basically I need a pkg_create() that will be executed only in the
> > context of the upcoming "ebuild ${PF}.ebuild create" command.
> > 
> > The package where I need it is app-mobilephone/bitpim. The upstream
> > doesn't offer a source tarball, so I need to construct it myself
> > from their svn repository. Up till recently, I used some hackery in
> > pkg_setup() to create it
> > (see the ebuild), but now "ebuild $PF.ebuild setup" verify the
> > digest before running pkg_setup().
> > 
> > [1] http://bugs.gentoo.org/show_bug.cgi?id=185567
> 
> Two questions:
> - are there more packages that could benefit from this?
> - is there a particular reason this has to be integrated into the
> ebuild and should not be handled by an ordinary script?

Sure. Every package with an unversioned upstream tarball that we're
forced to mirror a reversioned tarball of, or can't mirror at all.

It makes a lot of sense to put this sort of package maintenance and
creation information in the same place as the package.

But to do it right, we might need a new type of dependency. This will
require more research into whether any packages would have a use for it.

Thanks,
Donnie

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] RFC : New ebuild function pkg_create for creating corespondent sorce tarball

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 462 bytes --]

On Tue, 17 Jul 2007 07:25:02 +0300
Alin Năstac <mrness@gentoo.org> wrote:
> Basically I need a pkg_create() that will be executed only in the
> context of the upcoming "ebuild ${PF}.ebuild create" command.

This isn't something that should be done in an ebuild. It's more
suitable to an external script maintained by the appropriate herd.
There's no point cluttering up the tree with something that is of no
use to end users.

-- 
Ciaran McCreesh


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] RFC : New ebuild function pkg_create for creating corespondent sorce tarball

[Luca Barbato]

Ciaran McCreesh wrote:
> On Tue, 17 Jul 2007 07:25:02 +0300
> Alin Năstac <mrness@gentoo.org> wrote:
>> Basically I need a pkg_create() that will be executed only in the
>> context of the upcoming "ebuild ${PF}.ebuild create" command.
> 
> This isn't something that should be done in an ebuild. It's more
> suitable to an external script maintained by the appropriate herd.
> There's no point cluttering up the tree with something that is of no
> use to end users.
> 

Having something that interacts with the live eclasses gives you all you
need to end up with a good tarball with the minimal number of lines of
code. Obviously it could be a gentoolkit-dev/portage-utils script
instead of src_create/pack, since you just need to repack what the
src_unpack did in the ebuild.

I'm not sure how many special cases/option would be necessary for such
script, like exclude dir, one tar per dir and so on.

lu

-- 

Luca Barbato

Gentoo/linux Gentoo/PPC
http://dev.gentoo.org/~lu_zero
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] RFC : New ebuild function pkg_create for creating corespondent sorce tarball

[Alin Năstac]

[-- Attachment #1: Type: text/plain, Size: 371 bytes --]

Luca Barbato wrote:
> I'm not sure how many special cases/option would be necessary for such
> script, like exclude dir, one tar per dir and so on.
>   

The process of creating the tarball is quite specific to the package in
question, therefore you cannot make an universal function that does that.
See the app-mobilephone/bitpim maketarball() and you'll see.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: [gentoo-dev] RFC : New ebuild function pkg_create for creating corespondent sorce tarball

[Luca Barbato]

Alin Năstac wrote:
> Luca Barbato wrote:
>> I'm not sure how many special cases/option would be necessary for such
>> script, like exclude dir, one tar per dir and so on.
>>   
> 
> The process of creating the tarball is quite specific to the package in
> question, therefore you cannot make an universal function that does that.
> See the app-mobilephone/bitpim maketarball() and you'll see.
> 

My idea is to have your overlay with live ebuilds and a tool to make a
release out of it.

All the messy stuff remains in pkg_setup/src_unpack all you need to do
then is to iterate the workdir and make tarballs of what you find there.

Sounds good enough?

lu

-- 

Luca Barbato

Gentoo/linux Gentoo/PPC
http://dev.gentoo.org/~lu_zero
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] RFC : New ebuild function pkg_create for creating corespondent sorce tarball

[Alin Năstac]

[-- Attachment #1: Type: text/plain, Size: 861 bytes --]

Luca Barbato wrote:
> My idea is to have your overlay with live ebuilds and a tool to make a
> release out of it.
>
> All the messy stuff remains in pkg_setup/src_unpack all you need to do
> then is to iterate the workdir and make tarballs of what you find there.
>
> Sounds good enough?
>   
Basically you propose to have a src_whatever function that will work as
a substitute for src_unpack() in case you don't find the tarball on
mirrors, isn't it?
That could work too, but what if, depending on some useflag, I need to
patch the source tree? Because the new function must be capable to
replace src_unpack, you will have to add a command like "use foo &&
epatch ..." at the end of the new function. If foo useflag is enabled on
your system, the external tool that build the tarball will create it
based on a wrongly patched source tree. 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: [gentoo-dev] RFC : New ebuild function pkg_create for creating corespondent sorce tarball

[Luca Barbato]

Alin Năstac wrote:
> Luca Barbato wrote:
>> My idea is to have your overlay with live ebuilds and a tool to make a
>> release out of it.
>>
>> All the messy stuff remains in pkg_setup/src_unpack all you need to do
>> then is to iterate the workdir and make tarballs of what you find there.
>>
>> Sounds good enough?
>>   
> Basically you propose to have a src_whatever function that will work as
> a substitute for src_unpack() in case you don't find the tarball on
> mirrors, isn't it?

More or less a tool that runs the steps till src_unpack included and
then repacks what it finds in workdir.

> That could work too, but what if, depending on some useflag, I need to
> patch the source tree? Because the new function must be capable to
> replace src_unpack, you will have to add a command like "use foo &&
> epatch ..." at the end of the new function. If foo useflag is enabled on
> your system, the external tool that build the tarball will create it
> based on a wrongly patched source tree. 

Well much depends on what is the seed ebuild.

I make the assumption we have 2 kind of ebuilds

live ebuilds that we use to produce tarballs and normal ebuilds that use
those tarballs.

it is a little more complex than your propose but keeps things separated.

lu


-- 

Luca Barbato

Gentoo/linux Gentoo/PPC
http://dev.gentoo.org/~lu_zero
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] RFC : New ebuild function pkg_create for creating corespondent sorce tarball

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 951 bytes --]

On Tuesday 17 July 2007, Ciaran McCreesh wrote:
> Alin Năstac <mrness@gentoo.org> wrote:
> > Basically I need a pkg_create() that will be executed only in the
> > context of the upcoming "ebuild ${PF}.ebuild create" command.
>
> This isn't something that should be done in an ebuild. It's more
> suitable to an external script maintained by the appropriate herd.

except that it makes transition of maintenance of packages from person to 
person, herd to herd, whatever to monkey, a lot easier.  not everything has a 
herd and not every herd has a standard (or anything close) as to how these 
things are done.  having a useful helper function in an ebuild eases and 
standardizes all of these things.

> There's no point cluttering up the tree with something that is of no
> use to end users.

incorrect.  end users can utilize these functions just as easily as a 
developer to help test a new tarball not yet in the tree.
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 827 bytes --]

Re: [gentoo-dev] RFC : New ebuild function pkg_create for creating corespondent sorce tarball

[Jan Kundrát]

[-- Attachment #1: Type: text/plain, Size: 667 bytes --]

Alin Năstac wrote:
> The upstream doesn't offer a source tarball, so I need to construct
> it myself from their svn repository.

If you're creating a live ebuild, there are already existing eclasses
that works from the user's POV.

If your aim is to create an ebuild for a specific version, you might as
well checkout stuff yourself and let Gentoo mirror the generated tarball
(your mail doesn't talk about RESTRICT=fetch). If you let Gentoo mirror
the tarball, users are likely to be happier because they'll get the file
faster and in a more reliable way.

Or am I missing something?

Cheers,
-jkt

-- 
cd /local/pub && more beer > /dev/mouth


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: [gentoo-dev] RFC : New ebuild function pkg_create for creating corespondent sorce tarball

[Alin Năstac]

[-- Attachment #1: Type: text/plain, Size: 1288 bytes --]

Jan Kundrát wrote:
> Alin Năstac wrote:
>   
>> The upstream doesn't offer a source tarball, so I need to construct
>> it myself from their svn repository.
>>     
>
> If you're creating a live ebuild, there are already existing eclasses
> that works from the user's POV.
>   
I'm not speaking about live ebuilds, only about building a source
tarball for a specific version where upstream doesn't offer such thing.
> If your aim is to create an ebuild for a specific version, you might as
> well checkout stuff yourself and let Gentoo mirror the generated tarball
> (your mail doesn't talk about RESTRICT=fetch). If you let Gentoo mirror
> the tarball, users are likely to be happier because they'll get the file
> faster and in a more reliable way.
>   
What I want is a function that creates the source tarball, based on
whatever developer needs for that (the most usual case is a tag export
from a svn/cvs repository). The tarball will be placed on Gentoo mirrors
at the end (users will not run this new function, only the maintainer).
See app-mobilephone/bitpim ebuild example, it will speak for itself.

This can be solved also through an external script, but IMO this
solution is ugly. Ebuilds should contain whatever is needed for their
maintenance.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: [gentoo-dev] RFC : New ebuild function pkg_create for creating corespondent sorce tarball

[Vlastimil Babka]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Jan Kundrát wrote:
> Alin Năstac wrote:
>> The upstream doesn't offer a source tarball, so I need to construct
>> it myself from their svn repository.
> If your aim is to create an ebuild for a specific version, you might as
> well checkout stuff yourself and let Gentoo mirror the generated tarball
> (your mail doesn't talk about RESTRICT=fetch). If you let Gentoo mirror
> the tarball, users are likely to be happier because they'll get the file
> faster and in a more reliable way.

I think he wants to do exactly that, but having the code needed for this
right in the ebuild, so it can benefit from varibles like PV and
versionator eclass for converting PV to e.g. CVS tags... I think it's
quite elegant for this, and that you don't need another script
maintained somewhere else. If there was also switch in the respective
new 'ebuild foo.ebuild src_create' command to automatically scp files
specified by mirror://gentoo in SRC_URI to the right place... mmm :)

The only downside is that users will download something that they won't
find often useful (but think local overlay bumps and bugzilla reports on
version bump that just renaming the ebuild works?). OTOH, while this
might be useful for more than few packages (I can also think of some),
it's not too many to clutter the tree significantly.

> Or am I missing something?
> 
> Cheers,
> -jkt
> 
- --
Vlastimil Babka (Caster)
Gentoo/Java
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFGnJkEtbrAj05h3oQRArf4AJ4n/nvrxsDV1hFixnf9HcGNlscUcgCeJaG8
1Rkm4mQ0HKeJX39P+vwwPz8=
=jTzj
-----END PGP SIGNATURE-----
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] RFC : New ebuild function pkg_create for creating corespondent sorce tarball

[Mike Auty]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Vlastimil Babka wrote:
> I think he wants to do exactly that, but having the code needed for this
> right in the ebuild, so it can benefit from varibles like PV and
> versionator eclass for converting PV to e.g. CVS tags... I think it's
> quite elegant for this, and that you don't need another script
> maintained somewhere else. If there was also switch in the respective
> new 'ebuild foo.ebuild src_create' command to automatically scp files
> specified by mirror://gentoo in SRC_URI to the right place... mmm :)

It also means that if a developer has had to move files around or in
some way create the specific layout of the tarball for the ebuild, they
won't be lost if the dev goes away, or retires, etc.  So attaching the
specific package creation code to the specific package unpacking code
seems fairly sensible.  Although, I can see why there may be objections,
given the very specific circumstances in which it's useful...

Mike  5:)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.5 (GNU/Linux)

iD8DBQFGnKUqu7rWomwgFXoRAsGzAJ9Qx8Qg4zInhXSJsoOy3C8L7ZVwjgCfS+dh
fUx8fdYlqBTPX6TSgrSLQnQ=
=kWPM
-----END PGP SIGNATURE-----
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] RFC : New ebuild function pkg_create for creating corespondent sorce tarball

[Robert Buchholz]

On Tuesday, 17. July 2007 13:16, Mike Auty wrote:
> It also means that if a developer has had to move files around or in
> some way create the specific layout of the tarball for the ebuild,
> they won't be lost if the dev goes away, or retires, etc.  So
> attaching the specific package creation code to the specific package
> unpacking code seems fairly sensible.  Although, I can see why there
> may be objections, given the very specific circumstances in which
> it's useful...

Having it inside the ebuild could also be useful for people who want to 
bump packages themselves and file bugs: They can use the dev's original 
create function and package the sources for testing.

Regards,

Robert
-- 
gentoo-dev@gentoo.org mailing list