[gentoo-dev] Portage 1.5_pre Update

[gentoo-dev] Portage 1.5_pre Update

[drobbins]

Hello all,

Right now, there's a new version of Portage (1.5_pre1) on CVS that features
many major improvements (several that were suggested on this list).  It breaks
compatibility with our older .tbz2 package format, so at this point it's
intended for testers only.  Here's an outline of all changes:

Second-generation .tbz2 format
==============================

Our new second-generation .tbz2 format features a new double-envelope design
for enhanced performance.  Package metadata is encoded in our own lightweight
and fast "xpak" format and appended to the end of the .tbz2 archive, so that it
can be quickly and easily read by our package installation programs.  Packages
can still be installed this way if you're in a pinch (although nothing will be
recorded in the package database):

# cat mypkg-1.0.tbz2 | bzip2 -dq | tar xvf - -C /

All the ebuild components have been updated to to use this second-generation 
.tbz2 format, which will be the default package format for Gentoo Linux 1.0 and
beyond. ***Portage 1.5_pre1 is incompatible with old-style .tbz2 archives***

Enhanced package metadata
=========================

Binary packages as well as packages merged from sources now record additional
helpful metadata in /var/db/pkg.  Right now, we're recording the following information:

/var/db/pkg/sys-apps/foo-1.0
	
	CFLAGS		Optimization flags used to compile foo-1.0
	CXXFLAGS	C++ optimization flags used to compile foo-1.0
	CHOST		The setting of CHOST used to compile foo-1.0
	USE		The "use" settings enabled at the time foo-1.0 was compiled
	PROVIDES	The virtual packages that foo-1.0 provides
	CATEGORY	Package category
	PF		Full package name
	RDEPEND		Runtime dependencies
	LICENSE		License information

You'll notice a new "LICENSE" setting.  Developers can begin to add a LICENSE field
to their ebuilds and use it to record the licensing information for the package. 
Examples:

#you'll use this most of the time
LICENSE="gpl"

#for BSD-like stuff
LICENSE="bsd"

#if a package offers multiple licensing options
LICENSE="gpl artistic"

Recursive pkgmerge
==================

Thanks to our second-generation .tbz2 format, "recursive pkgmerge" now becomes
easy.  Our new, enhanced pkgmerge command now performs the following steps:

# pkgmerge foo-1.0.tbz2

	1. Scan foo-1.0.tbz2 for runtime dependencies
	2. If any, scan PKGDIR, creating a directed graph of all runtime dependency information
	3. Begin installing all necessary packages in optimal order

To use pkgmerge, you'll need to set PKGDIR to point to the location of your
.tbz2 packages.  PKGDIR defaults to /usr/portage/packages.  Portage expects to
find an "All" directory inside PKGDIR that contains all the packages.  Please
remember that the new pkgmerge is incompatible with our old package format, so
you can't use this to recursively merge your current (old-style) packages.
However, Gentoo Linux 1.0_rc4 will use this new packaging format, and recursive
pkgmerges will be fully supported :)

pkgmerge also supports a new option, "--pretend".  If you specify it as the first
argument, pkgmerge will tell you the packages it would install (in order), rather
than actually installing them.

Safe auto-unmerge
=================

You probably know that if you try to merge an already-installed package, the
current (old) version of ebuild doesn't let you.  Instead, it asks you to
unmerge the currently-installed version first.  This doesn't happen anymore.
Instead, the original instance of the package will be automatically safely
unmerged after the new instance has been merged.  So, if you have gimp-1.2
already installed, but you update your gimp-1.2.ebuild with a few fixes, you
can simply type:

# ebuild gimp-1.2.ebuild merge

The new version will be compiled and merged, and then the older instance of
gimp-1.2 will be automatically and safely unmerged, removing any files no
longer used by the new instance.

ebuild updates
==============

The "ebuild" program itself has been updated in several ways.  First, the
"remerge" command has been removed (it's no longer needed, just use "merge").
Secondly, Achim's FHS 2.1 fixes have been incorporated into Portage, so that
all documentation and man pages (installed using doman and dobin only) will end
up in /usr/share rather than /usr/.

python updates
==============

We have a new module called xpak.py that handles the creation and parsing of
xpak metadata.  The portage.py code has been majorly overhauled.  We now use an
object-based "dblink" system to access the /var/db/pkg object database.  By
providing this object API, in the future we will be able to re-implement the
database code in C/C++, write a mySQL package database backend, or do whatever
we like.  Of course, our new .tbz2 format is package database independent,
unlike old ebuilds that directly reference /var/db/pkg.

There are also a host of other cleanups and improvements, resulting in a
portage.py that has a bunch of new functionality, but is actually somewhat
_smaller_ than the original version.

new file locations
==================

All Portage binaries now reside in /usr/lib/portage/bin, and symlinks are
created in /usr/sbin and /usr/bin for popular binaries.  ebuild is now in
/usr/sbin, along with pkgmerge.  The new tbz2tool and xpak binaries live in
/usr/bin.  If you're going to test Portage 1.5_pre1, be sure to delete 
your old /usr/bin/ebuild script!

That's all for now :)

Best Regards,

-- 
Daniel Robbins					<drobbins@gentoo.org>
President/CEO					http://www.gentoo.org 
Gentoo Technologies, Inc.			

Re: [gentoo-dev] Portage 1.5_pre Update

[Thomas Flavel]

All good stuff :) One question though:

On Fri, Feb 16, 2001 at 09:18:11AM -0700, drobbins@gentoo.org wrote:
> 
> You'll notice a new "LICENSE" setting.  Developers can begin to add a LICENSE field
> to their ebuilds and use it to record the licensing information for the package. 
> Examples:
> 
> #you'll use this most of the time
> LICENSE="gpl"
> 
> #for BSD-like stuff
> LICENSE="bsd"
> 
> #if a package offers multiple licensing options
> LICENSE="gpl artistic"
> 

If I want to specify my own home-made license, which nobody knows the name of, could
I put a uri here? Something like

LICENSE="gpl"

would be valid because it's already known, but for unknown licenses:

LICENSE="URI:someaddress"

which could be parsed by whatever front end is used to provide the details of the
license? Just a thought,

- Tom

Re: [gentoo-dev] Portage 1.5_pre Update

[drobbins]

On Fri, Feb 16, 2001 at 04:35:28PM +0000, Thomas Flavel wrote:

> > #if a package offers multiple licensing options
> > LICENSE="gpl artistic"
> > 
> 
> If I want to specify my own home-made license, which nobody knows the name of, could
> I put a uri here? Something like
> 
> LICENSE="gpl"
> 
> would be valid because it's already known, but for unknown licenses:
> 
> LICENSE="URI:someaddress"
> 
> which could be parsed by whatever front end is used to provide the details of the
> license? Just a thought,

Yes, that could work.  Just put the URI in there verbatim if necessary.

-- 
Daniel Robbins					<drobbins@gentoo.org>
President/CEO					http://www.gentoo.org 
Gentoo Technologies, Inc.			

Re: [gentoo-dev] Portage 1.5_pre Update

[Jerry A!]

On Fri, Feb 16, 2001 at 09:18:11AM -0700, drobbins@gentoo.org wrote:
: Hello all,
: 
: Right now, there's a new version of Portage (1.5_pre1) on CVS that features
: many major improvements (several that were suggested on this list).  It breaks
: compatibility with our older .tbz2 package format, so at this point it's
: intended for testers only.  Here's an outline of all changes:

Okay, two questions.

1.  Will this format be used for 1.4?  Or will 1.4_rc2 just become 1.4
and then we begin moving forward?

2.  Is there still a text readable CONTENTS file?  One of the things I
really like is that I can quickly scan to see what has been put where.
If not, is there an easy way to get the contents out of xpak?

        --Jerry

name:  Jerry Alexandratos         ||  Open-Source software isn't a
phone: 703.599.6023               ||  matter of life or death...
email: jerry@thehutt.org          ||  ...It's much more important
                                  ||  than that!

Re: [gentoo-dev] Portage 1.5_pre Update

[drobbins]

On Fri, Feb 16, 2001 at 11:37:39AM -0500, Jerry A! wrote:
> On Fri, Feb 16, 2001 at 09:18:11AM -0700, drobbins@gentoo.org wrote:
> : Hello all,
> : 
> : Right now, there's a new version of Portage (1.5_pre1) on CVS that features
> : many major improvements (several that were suggested on this list).  It breaks
> : compatibility with our older .tbz2 package format, so at this point it's
> : intended for testers only.  Here's an outline of all changes:
> 
> Okay, two questions.
> 
> 1.  Will this format be used for 1.4?  Or will 1.4_rc2 just become 1.4
> and then we begin moving forward?

The new format will be used for all future Gentoo Linux releases.
 
> 2.  Is there still a text readable CONTENTS file?  One of the things I
> really like is that I can quickly scan to see what has been put where.
> If not, is there an easy way to get the contents out of xpak?

Yes, CONTENTS is still there, in an identical format.

-- 
Daniel Robbins					<drobbins@gentoo.org>
President/CEO					http://www.gentoo.org 
Gentoo Technologies, Inc.			

Re: [gentoo-dev] Portage 1.5_pre Update

[Achim Gottinger]

drobbins@gentoo.org wrote:

> Hello all,
>
> Right now, there's a new version of Portage (1.5_pre1) on CVS that features
> many major improvements (several that were suggested on this list).  It breaks
> compatibility with our older .tbz2 package format, so at this point it's
> intended for testers only.  Here's an outline of all changes:
>
> Second-generation .tbz2 format
> ==============================
>
> Our new second-generation .tbz2 format features a new double-envelope design
> for enhanced performance.  Package metadata is encoded in our own lightweight
> and fast "xpak" format and appended to the end of the .tbz2 archive, so that it
> can be quickly and easily read by our package installation programs.  Packages
> can still be installed this way if you're in a pinch (although nothing will be
> recorded in the package database):
>
> # cat mypkg-1.0.tbz2 | bzip2 -dq | tar xvf - -C /
>
> All the ebuild components have been updated to to use this second-generation
> .tbz2 format, which will be the default package format for Gentoo Linux 1.0 and
> beyond. ***Portage 1.5_pre1 is incompatible with old-style .tbz2 archives***
>
> Enhanced package metadata
> =========================
>
> Binary packages as well as packages merged from sources now record additional
> helpful metadata in /var/db/pkg.  Right now, we're recording the following information:
>
> /var/db/pkg/sys-apps/foo-1.0
>
>         CFLAGS          Optimization flags used to compile foo-1.0
>         CXXFLAGS        C++ optimization flags used to compile foo-1.0
>         CHOST           The setting of CHOST used to compile foo-1.0
>         USE             The "use" settings enabled at the time foo-1.0 was compiled
>         PROVIDES        The virtual packages that foo-1.0 provides
>         CATEGORY        Package category
>         PF              Full package name
>         RDEPEND         Runtime dependencies
>         LICENSE         License information
>

You dropped the revision from the package-dir name I think. Good.

>
> You'll notice a new "LICENSE" setting.  Developers can begin to add a LICENSE field
> to their ebuilds and use it to record the licensing information for the package.
> Examples:
>
> #you'll use this most of the time
> LICENSE="gpl"
>
> #for BSD-like stuff
> LICENSE="bsd"
>
> #if a package offers multiple licensing options
> LICENSE="gpl artistic"
>
> Recursive pkgmerge
> ==================
>
> Thanks to our second-generation .tbz2 format, "recursive pkgmerge" now becomes
> easy.  Our new, enhanced pkgmerge command now performs the following steps:
>
> # pkgmerge foo-1.0.tbz2
>
>         1. Scan foo-1.0.tbz2 for runtime dependencies
>         2. If any, scan PKGDIR, creating a directed graph of all runtime dependency information
>         3. Begin installing all necessary packages in optimal order
>
> To use pkgmerge, you'll need to set PKGDIR to point to the location of your
> .tbz2 packages.  PKGDIR defaults to /usr/portage/packages.  Portage expects to
> find an "All" directory inside PKGDIR that contains all the packages.  Please
> remember that the new pkgmerge is incompatible with our old package format, so
> you can't use this to recursively merge your current (old-style) packages.
> However, Gentoo Linux 1.0_rc4 will use this new packaging format, and recursive
> pkgmerges will be fully supported :)

Should we write a converter script for old-sytle packages?

>
>
> pkgmerge also supports a new option, "--pretend".  If you specify it as the first
> argument, pkgmerge will tell you the packages it would install (in order), rather
> than actually installing them.

This is definately a good thing. Does it output the dependencie from DEPEND or the resolved
depedencie.

>
>
> Safe auto-unmerge
> =================
>
> You probably know that if you try to merge an already-installed package, the
> current (old) version of ebuild doesn't let you.  Instead, it asks you to
> unmerge the currently-installed version first.  This doesn't happen anymore.
> Instead, the original instance of the package will be automatically safely
> unmerged after the new instance has been merged.  So, if you have gimp-1.2
> already installed, but you update your gimp-1.2.ebuild with a few fixes, you
> can simply type:
>
> # ebuild gimp-1.2.ebuild merge
>
> The new version will be compiled and merged, and then the older instance of
> gimp-1.2 will be automatically and safely unmerged, removing any files no
> longer used by the new instance.
>
> ebuild updates
> ==============
>
> The "ebuild" program itself has been updated in several ways.  First, the
> "remerge" command has been removed (it's no longer needed, just use "merge").
> Secondly, Achim's FHS 2.1 fixes have been incorporated into Portage, so that
> all documentation and man pages (installed using doman and dobin only) will end

My changes where made to doman, doinfo and dodoc (newman newinfo newdoc).
If your man info doc dirs are in /usr rigth now and not symlinks from /usr/share to /usr, please
move them to /usr/share and symlink them to /usr

>
> up in /usr/share rather than /usr/.
>
> python updates
> ==============
>
> We have a new module called xpak.py that handles the creation and parsing of
> xpak metadata.  The portage.py code has been majorly overhauled.  We now use an
> object-based "dblink" system to access the /var/db/pkg object database.  By
> providing this object API, in the future we will be able to re-implement the
> database code in C/C++, write a mySQL package database backend, or do whatever
> we like.  Of course, our new .tbz2 format is package database independent,
> unlike old ebuilds that directly reference /var/db/pkg.
>
> There are also a host of other cleanups and improvements, resulting in a
> portage.py that has a bunch of new functionality, but is actually somewhat
> _smaller_ than the original version.
>
> new file locations
> ==================
>
> All Portage binaries now reside in /usr/lib/portage/bin, and symlinks are
> created in /usr/sbin and /usr/bin for popular binaries.  ebuild is now in
> /usr/sbin, along with pkgmerge.  The new tbz2tool and xpak binaries live in
> /usr/bin.  If you're going to test Portage 1.5_pre1, be sure to delete
> your old /usr/bin/ebuild script!
>
> That's all for now :)

Thank you for that big peace of work daniel.

achim~

>
>
> Best Regards,
>
> --
> Daniel Robbins                                  <drobbins@gentoo.org>
> President/CEO                                   http://www.gentoo.org
> Gentoo Technologies, Inc.
>
> _______________________________________________
> gentoo-dev mailing list
> gentoo-dev@gentoo.org
> http://www.gentoo.org/mailman/listinfo/gentoo-dev

Re: [gentoo-dev] Portage 1.5_pre Update

[drobbins]

On Fri, Feb 16, 2001 at 05:23:07PM +0100, Achim Gottinger wrote:

> > /var/db/pkg/sys-apps/foo-1.0
> >
> >         CFLAGS          Optimization flags used to compile foo-1.0
> >         CXXFLAGS        C++ optimization flags used to compile foo-1.0
> >         CHOST           The setting of CHOST used to compile foo-1.0
> >         USE             The "use" settings enabled at the time foo-1.0 was compiled
> >         PROVIDES        The virtual packages that foo-1.0 provides
> >         CATEGORY        Package category
> >         PF              Full package name
> >         RDEPEND         Runtime dependencies
> >         LICENSE         License information
> >

> You dropped the revision from the package-dir name I think. Good.

Not yet, but we should do that.  Let's get a list of minor fixes such as this
and then I'll apply them all in one go.

> Should we write a converter script for old-sytle packages?

Good idea.

> > pkgmerge also supports a new option, "--pretend".  If you specify it as the first
> > argument, pkgmerge will tell you the packages it would install (in order), rather
> > than actually installing them.
> 
> This is definately a good thing. Does it output the dependencie from DEPEND or the resolved
> depedencie.

# pkgmerge --pretend libmng-1.0.0.tbz2
These are the packages that I would merge, in order:
/usr/portage/packages/All/lcms-1.06.tbz2
/usr/portage/packages/All/libmng-1.0.0.tbz2   

> Thank you for that big peace of work daniel.

You're welcome!  I've just committed a pkgmerge fix to Portage; a last-minute change to
portage.py caused it to die.

-- 
Daniel Robbins					<drobbins@gentoo.org>
President/CEO					http://www.gentoo.org 
Gentoo Technologies, Inc.