[gentoo-user] How does Portage prioritze emerges in emerge world?

[gentoo-user] How does Portage prioritze emerges in emerge world?

[Holly Bostick]

OK, I take it back.

I said that the situation of upgrading a kernel with the 'symlink' USE
flag active occurring at the same time as a (particular) program needing
to compile against a configured kernel was not likely to occur all that
often, but I was wrong. It's happened again today, but with a different
program than the ones I normally keep an eye on.

The good thing is that I (think I) see what the problem is.

The problem is that Portage emerges the new kernel before (almost)
everything else, without regard for whether the 'symlink' USE flag is
active, and whether or not any of the other programs proposed to emerge
need to compile against a configured kernel source-- or rather, the
currently-running kernel, which the symlink most likely pointed to
before Portage changed it via a previous emerge.

Honestly, there's really no reason (that I can see) to emerge a kernel
source before everything else, since the kernel source is useless until
at the very least configured, and preferably compiled and installed, and
since you're in the middle of an emerge -uwhatever world, you can't
reasonably configure and compile the new source until the entire
operation is finished. Yeah, OK, technically you can, but it's not
really something that an ordinary person would do, I think.

And if the 'symlink' USE flag is active, emerging the kernel sources
before everything else-- which may include packages that must compile
against a configured kernel, with the assumption that the
/usr/src/symlink points to such a kernel, which it no longer does
because the symlink has been changed during a previous emerge and you
have not had time to configure the newly-emerged kernel-- is a real
problem. I just had to open another term, su to root, run MC to change
the symlink-- and got it wrong because I had a second unconfigured
kernel  (2.6.14-r2; 2.6.14-r3 was being installed) that I forgot I had
not yet upgraded to), so had to change the link again to the *real*
running kernel (2.6.14) and emerge --resume. And of course I'll have to
eventually change the symlink back manually in order to actually manage
the new kernel. Which means I have to remember to do that-- which is not
the point of having the 'symlink' USE flag active.

It seems to me that this could all be avoided if Portage emerged a new
kernel *last* in the list if the 'symlink' USE flag is active for kernel
emerges-- then everything in the list that needed a configured kernel
would have one (the currently-running kernel), the emerge would complete
normally, and the symlink would be changed at the end of the procedure
so that my next operation could be to upgrade the kernel, which seems to
me a reasonable and ordinary order of operation (emerge -u** world, then
configure and compile new kernel and run module-rebuild).

Am I doing things wrong, or is this a valid enhancement request for b.g.o?

Holly
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] How does Portage prioritze emerges in emerge world?

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 399 bytes --]

On Sat, 26 Nov 2005 14:06:02 +0100 Holly Bostick <motub@planet.nl>
wrote:
| Am I doing things wrong, or is this a valid enhancement request for
| b.g.o?

Unlikely to happen... Dependency resolution doesn't work that way.

-- 
Ciaran McCreesh : Gentoo Developer (The one that looks before leaping)
Mail            : ciaranm at gentoo.org
Web             : http://dev.gentoo.org/~ciaranm


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-user] How does Portage prioritze emerges in emerge world?

[Zac Medico]

Holly Bostick wrote:
> OK, I take it back.
> 
> I said that the situation of upgrading a kernel with the 'symlink' USE
> flag active occurring at the same time as a (particular) program needing
> to compile against a configured kernel was not likely to occur all that
> often, but I was wrong. It's happened again today, but with a different
> program than the ones I normally keep an eye on.
> 
> The good thing is that I (think I) see what the problem is.
> 
> The problem is that Portage emerges the new kernel before (almost)
> everything else, without regard for whether the 'symlink' USE flag is
> active, and whether or not any of the other programs proposed to emerge
> need to compile against a configured kernel source-- or rather, the
> currently-running kernel, which the symlink most likely pointed to
> before Portage changed it via a previous emerge.
> 
> Honestly, there's really no reason (that I can see) to emerge a kernel
> source before everything else, since the kernel source is useless until
> at the very least configured, and preferably compiled and installed, and
> since you're in the middle of an emerge -uwhatever world, you can't
> reasonably configure and compile the new source until the entire
> operation is finished. Yeah, OK, technically you can, but it's not
> really something that an ordinary person would do, I think.

I like to manage the kernel sources myself so I always keep a kernel package in package.provided.

mkdir -p /etc/portage/profile
echo sys-kernel/gentoo-sources-2.6.14-r3 >> /etc/portage/profile/package.provided

> 
> And if the 'symlink' USE flag is active, emerging the kernel sources
> before everything else-- which may include packages that must compile
> against a configured kernel, with the assumption that the
> /usr/src/symlink points to such a kernel, which it no longer does
> because the symlink has been changed during a previous emerge and you
> have not had time to configure the newly-emerged kernel-- is a real
> problem. I just had to open another term, su to root, run MC to change
> the symlink-- and got it wrong because I had a second unconfigured
> kernel  (2.6.14-r2; 2.6.14-r3 was being installed) that I forgot I had
> not yet upgraded to), so had to change the link again to the *real*
> running kernel (2.6.14) and emerge --resume. And of course I'll have to
> eventually change the symlink back manually in order to actually manage
> the new kernel. Which means I have to remember to do that-- which is not
> the point of having the 'symlink' USE flag active.
> 
> It seems to me that this could all be avoided if Portage emerged a new
> kernel *last* in the list if the 'symlink' USE flag is active for kernel
> emerges-- then everything in the list that needed a configured kernel
> would have one (the currently-running kernel), the emerge would complete
> normally, and the symlink would be changed at the end of the procedure
> so that my next operation could be to upgrade the kernel, which seems to
> me a reasonable and ordinary order of operation (emerge -u** world, then
> configure and compile new kernel and run module-rebuild).
> 
> Am I doing things wrong, or is this a valid enhancement request for b.g.o?
> 
> Holly

The portage developers will not add a special case for kernel packages, so any reordering/prioritization would have to be done in a generic way that is usable for any type of package.  Also, it seems desireable to compile against the latest kernel that is installed.

Perhaps it would make sense to have a default kernel config that is used to configure the kernel sources automatically (make oldconfig; make modules_prepare) after a new kernel is installed?  Something like this could be done ebuild postinst phase (when symlink is created).  It is planned for future versions of portage to have pre/post phase hooks, which will allow users to define actions such as this via /etc/portage/bashrc:

http://thread.gmane.org/gmane.linux.gentoo.portage.devel/1107

Zac
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] How does Portage prioritze emerges in emerge world?

[Holly Bostick]

Zac Medico schreef:
> Holly Bostick wrote:
> 
>> I said that the situation of upgrading a kernel with the 'symlink' 
>> USE flag active occurring at the same time as a (particular) 
>> program needing to compile against a configured kernel was not 
>> likely to occur all that often, but I was wrong. It's happened 
>> again today, but with a different program than the ones I normally 
>> keep an eye on.
>> 
>> The good thing is that I (think I) see what the problem is.
>> 
>> The problem is that Portage emerges the new kernel before (almost)
>>  everything else, without regard for whether the 'symlink' USE flag
>>  is active, and whether or not any of the other programs proposed
>> to emerge need to compile against a configured kernel source-- or 
>> rather, the currently-running kernel, which the symlink most likely
>>  pointed to before Portage changed it via a previous emerge.
>> 
<snip>
> 
> The portage developers will not add a special case for kernel 
> packages, so any reordering/prioritization would have to be done in a
>  generic way that is usable for any type of package.  Also, it seems 
> desireable to compile against the latest kernel that is installed.

..... OK, I understand this to some extent (meaning I get it that
Portage is not going to be revised in this way), but I must question
that last statement, "it seems desirable to compile against the latest
kernel that is installed."

The latest kernel that is *installed* (as opposed to the latest kernel
whose source is emerged, regardless of whether it's configured,
compiled, or installed) is the one I'm booted into, and while I
presumably intend/want to upgrade to the newly emerged kernel at
some reasonably soon point, I don't necessarily want to do it *right
that minute*, nor am I necessarily going to avoid rebooting until such
time as I have installed the upgraded kernel.

> 
> Perhaps it would make sense to have a default kernel config that is 
> used to configure the kernel sources automatically (make oldconfig; 
> make modules_prepare) after a new kernel is installed?  Something 
> like this could be done ebuild postinst phase (when symlink is 
> created).  It is planned for future versions of portage to have 
> pre/post phase hooks, which will allow users to define actions such 
> as this via /etc/portage/bashrc:

This sounds great, but what about the kernel I'm booted into, against
which the module will *not* be compiled, if I have to reboot before
actually configuring/compiling/installing the new kernel?

The kernel modules will not be upgraded for that kernel (because the
upgrades compiled only against the future kernel), and while that won't
precisely break the old kernel (hopefully, since the old modules should
still be good, though I cannot vouch for all circumstances), it means I
don't have the upgraded modules for the currently-running kernel.

After all, module-rebuild will re-build all the modules against a
newly-compiled kernel; I don't need to build some limited subset of said
modules against the new kernel source at the time I emerge the new
kernel source, since I will build all of them at the end of the
operations which make the new kernel actually available for use. What I
do want is to build the upgraded modules against the currently-running
kernel, which I expect to be using for some short additional period of
time (until I compile and install the new kernel, which may be hours or
days in the future). It would be nice to then have the future kernel
source prepared for compilation and installation automatically (by
redirecting the symlink), so that said compilation and installation goes
on a "next-to-do, asap" list of sorts, but I'm not essentially forced to
drop everything in order to compile the new kernel source *right now* in
order to use the upgraded modules, which may be mission-critical in some
respect (if the upgrade fixes functionality that I need working).

Maybe the issue is really that the 'symlink' USE flag is obsolete in
some respect, since it appears that automatic redirection of the
/usr/src/linux symlink can often cause more problems than it solves,
since the user cannot really know ahead of time whether a kernel module
is going to upgrade in the same operation as a new kernel source is
going to be emerged (which is not the same as installing a new kernel,
of course).

I guess I'll turn off the USE flag and manage the symlink directly, but
it seems like there ought to be a better way.

> 
> http://thread.gmane.org/gmane.linux.gentoo.portage.devel/1107

Thanks for the link.

Holly
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] How does Portage prioritze emerges in emerge world?

[Zac Medico]

Holly Bostick wrote:
> 
> This sounds great, but what about the kernel I'm booted into, against
> which the module will *not* be compiled, if I have to reboot before
> actually configuring/compiling/installing the new kernel?
> 

You can get pretty close to your desired behavior (merge kernel last) if you simple mask kernel package versions greater than the one that is currently installed.

mkdir -p /etc/portage
echo ">sys-kernel/gentoo-sources-2.6.14-r2" >> /etc/portage/package.mask

That way, portage will not attempt to upgrade it until you tell it that you are ready, by removing the mask.  And yeah, if USE=symlink causes problems, don't use it (in my suggested scenario above it might be useful though).

Zac
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] How does Portage prioritze emerges in emerge world?

[Holly Bostick]

Zac Medico schreef:
> Holly Bostick wrote:
> 
>> 
>> This sounds great, but what about the kernel I'm booted into, 
>> against which the module will *not* be compiled, if I have to 
>> reboot before actually configuring/compiling/installing the new 
>> kernel?
>> 
> 
> You can get pretty close to your desired behavior (merge kernel last)
>  if you simple mask kernel package versions greater than the one that
>  is currently installed.
> 
> mkdir -p /etc/portage echo ">sys-kernel/gentoo-sources-2.6.14-r2" >> 
> /etc/portage/package.mask
> 
> That way, portage will not attempt to upgrade it until you tell it 
> that you are ready, by removing the mask.  And yeah, if USE=symlink 
> causes problems, don't use it (in my suggested scenario above it 
> might be useful though).

So the ultimate conclusion is that I can either

1) disable the symlink USE flag and manage the redirect manually, which
would enable me to download any kernel at any time without concern for
whether a kernel module was upgrading in the same operation; or

2) manually mask kernels, which would enable me to upgrade any kernel
modules at any time but force me to manually oversee the availability of
kernel upgrades and manually enable them (and re-disable them following
said upgrade).

I guess I'll go for option 1, but the long and the short of it is that
complete automation is unavailable and my only choice is what I prefer
to manage manually.

OK, then. <sigh> Thanks.

Holly
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] How does Portage prioritze emerges in emerge world?

[Glenn Enright]

On Sun, 27 Nov 2005 06:47, Holly Bostick wrote:
> 1) disable the symlink USE flag and manage the redirect manually, which
> would enable me to download any kernel at any time without concern for
> whether a kernel module was upgrading in the same operation; or
<snip
> I guess I'll go for option 1, but the long and the short of it is that
> complete automation is unavailable and my only choice is what I prefer
> to manage manually.
>
> OK, then. <sigh> Thanks.
>
> Holly

Yeah it would be nice if there was a system to manage this, but your solution 
works for me too and its relatively easy to remember what you need to fix 
when you do a kernel upgrade. I get the feeling that automating this sort of 
thing is non-trivial. Besides... are we gentoo users or what?! ;)
-- 
Ozmosis:
	The inability of one's job to live up to one's self-image.
		-- Douglas Coupland, "Generation X: Tales for an Accelerated
		   Culture"
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] How does Portage prioritze emerges in emerge world?

[Abhay Kedia]

[-- Attachment #1: Type: text/plain, Size: 2036 bytes --]

On Saturday 26 Nov 2005 10:12 pm, Holly Bostick wrote:
> ..... OK, I understand this to some extent (meaning I get it that
> Portage is not going to be revised in this way), but I must question
> that last statement, "it seems desirable to compile against the latest
> kernel that is installed."
>
> The latest kernel that is *installed* (as opposed to the latest kernel
> whose source is emerged, regardless of whether it's configured,
> compiled, or installed) is the one I'm booted into, and while I
> presumably intend/want to upgrade to the newly emerged kernel at
> some reasonably soon point, I don't necessarily want to do it *right
> that minute*, nor am I necessarily going to avoid rebooting until such
> time as I have installed the upgraded kernel.
>
I don't know how portage designing works but what you are saying can probably 
never happen. What you want is that give "kernel" a special status and leave 
it out of dependency checking. How can that happen? If you follow the normal 
dependency checking then portage is working exactly how it should.

If we go by the way you want things to work then just imagine this scenario. 
Program abc depends on xyz. You have abc-1.0.0 as well as xyz-1.0.0 installed 
and configured on your system. Today both programs have been updated to 
versions 1.0.1 and you do emrge -uDNv world. What would be the desired action 
that portage should perform? The desired action would be to first update 
xyz-1.0.0 to xyz-1.0.1 and then build abc-1.0.1 against the newly installed 
libraries. What you want is that abc-1.0.1 should install against. xyz-1.0.0 
and then  you will revdep-rebuild later to build abc once again but this time 
against the newer xyz-1.0.1.

imho that is certainly not the way things should work. Why not build with 
latest libraries when you already have them? To do what you want, all kernel 
packages will have to be left alone from dependency tracking and I don't know 
whether it is possible or not. Just my 2 cents.

Abhay

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-user] How does Portage prioritze emerges in emerge world?

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 837 bytes --]

On Sat, 26 Nov 2005 18:47:38 +0100, Holly Bostick wrote:

> 1) disable the symlink USE flag and manage the redirect manually, which
> would enable me to download any kernel at any time without concern for
> whether a kernel module was upgrading in the same operation; or

That's one approach. In some ways it is the easiest, especially if you
already have a script that deals with copying the config, running
make oldconfig and mounting /boot to avoid forgetting this later. The
route I chose was to emerge the kernel sources first when they show up in
emerge -u world. Then compile and install the new kernel before rebooting
and running emerge world again. Thinking about it, removing the symlink
USE flag may be easier.


-- 
Neil Bothwick

Bus: (n.) a connector you plug money into, something like a slot machine.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-user] How does Portage prioritze emerges in emerge world?

[Graham Murray]

Abhay Kedia <abhay.ilugd@gmail.com> writes:

> imho that is certainly not the way things should work. Why not build with 
> latest libraries when you already have them? To do what you want, all kernel 
> packages will have to be left alone from dependency tracking and I don't know 
> whether it is possible or not. Just my 2 cents.

Another way of doing it would be for portage to stop after
'installing' a kernel upgrade. Then you would configure and (possibly
build) the new kernel before restarting portage.
-- 
gentoo-user@gentoo.org mailing list

[gentoo-user] Re: How does Portage prioritze emerges in emerge world?

[James]

Holly Bostick <motub <at> planet.nl> writes:

> I said that the situation of upgrading a kernel with the 'symlink' USE
> flag active occurring at the same time as a (particular) program needing
> to compile against a configured kernel was not likely to occur all that
> often, but I was wrong. It's happened again today, but with a different
> program than the ones I normally keep an eye on.

> The good thing is that I (think I) see what the problem is.

> The problem is that Portage emerges the new kernel before (almost)
> everything else, without regard for whether the 'symlink' USE flag is
> active, and whether or not any of the other programs proposed to emerge
> need to compile against a configured kernel source-- or rather, the
> currently-running kernel, which the symlink most likely pointed to
> before Portage changed it via a previous emerge.

Folks either I'm missing the  boat entirely, or this is far simpler 
than these correspondances indicate.

/etc/src contains this:
rw-r--r--   1 root root    0 Aug 24 14:54 .keep
lrwxrwxrwx   1 root root   22 Nov  3 16:35 linux -> linux-2.6.13-gentoo-r5
drwxr-xr-x  19 root root 1280 Oct  5 16:42 linux-2.6.12-gentoo-r9
drwxr-xr-x  19 root root 1280 Nov 22 16:21 linux-2.6.13-gentoo-r5
drwxr-xr-x  18 root root  688 Oct 30 01:02 linux-2.6.14-gentoo
drwxr-xr-x  18 root root  688 Nov 14 02:45 linux-2.6.14-gentoo-r2
drwxr-xr-x  18 root root  688 Nov 17 21:04 linux-2.6.14.2


But since my symlink is to the 'linux-2.6.13-gentoo-r5' kernel
source, that is what all of my packages use as a default to
compile against, unless I use some of these techniques that 
others have listed?  Am I correct?

I have several kernel-sources later than 'linux-2.6.13-gentoo-r5'
some vanilla some gentoo. How would portage know which one to
compile against if not the symlink set in /usr/src?

Did I miss something or this conversation only relevant if
one chooses to set USE flags in lieu of a symlink in /usr/src?

Why would you want to control this behavior with a USE flag 
versus an 'old fashion symlink' in /usr/src?


confused............
James

-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Re: How does Portage prioritze emerges in emerge world?

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 891 bytes --]

On Mon, 28 Nov 2005 16:26:56 +0000 (UTC), James wrote:

> I have several kernel-sources later than 'linux-2.6.13-gentoo-r5'
> some vanilla some gentoo. How would portage know which one to
> compile against if not the symlink set in /usr/src?

uname -r returns the currently running kernel version.
 
> Did I miss something or this conversation only relevant if
> one chooses to set USE flags in lieu of a symlink in /usr/src?
>
> Why would you want to control this behavior with a USE flag 
> versus an 'old fashion symlink' in /usr/src?

The sylink is still used, but maintained by portage when you install
new kernel sources. The USE flag in question is 'symlink'

# useflag symlink
/usr/portage/profiles/use.desc:symlink - Force kernel ebuilds to automatically update the /usr/src/linux symlink.


-- 
Neil Bothwick

OPERATOR ERROR: Nyah, Nyah, Nyah, Nyah, Nyah!

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-user] Re: How does Portage prioritze emerges in emerge world?

[Francesco R.]

Alle 17:51, lunedì 28 novembre 2005, Neil Bothwick el ga butta:
> |On Mon, 28 Nov 2005 16:26:56 +0000 (UTC), James wrote:
> |> I have several kernel-sources later than 'linux-2.6.13-gentoo-r5'
> |> some vanilla some gentoo. How would portage know which one to
> |> compile against if not the symlink set in /usr/src?
> |
> |uname -r returns the currently running kernel version.

disclaimer: _not_ followed the whole thread and _not_ checked the 
source.

But how can be created packages that work on different machines if it 
use uname -r ?
I strongly hope that the symlink is still used.

Cheers

-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Re: How does Portage prioritze emerges in emerge world?

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 572 bytes --]

On Mon, 28 Nov 2005 18:20:28 +0100, Francesco R. wrote:

> disclaimer: _not_ followed the whole thread and _not_ checked the 
> source.
> 
> But how can be created packages that work on different machines if it 
> use uname -r ?

This thread is all about compiling kernel modules, which must be specific
to either the running kernel or the intended kernel. Most packages take
the latter option, using the symlink, which causes problems if the
symlink is updated during an emerge -u world.


-- 
Neil Bothwick

Always remember to pillage before you burn.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-user] Re: How does Portage prioritze emerges in emerge world?

[Scott Stoddard]

Neil Bothwick wrote:
> 
> This thread is all about compiling kernel modules, which must be specific
> to either the running kernel or the intended kernel. Most packages take
> the latter option, using the symlink, which causes problems if the
> symlink is updated during an emerge -u world.
> 

IMO exactly why no one should ever really use the symlink USE flag...

(Haven't read the whole thread either...long since deleted from my 
mailserver.)

Depending on the output of uname -r is foolish, as witnessed by the 
debacle of windows versions over the years.  (Several prominent pieces 
of software install only on versions matching a string returned by the 
OS, thereby essentially eliminating the possibility that they will work 
on future versions.  I still have several perfectly stable programs that 
won't run on windows xp 64 bit because of this problem.)

Who is to say what that [uname -r] return might look like on different 
systems with custom-patched kernels? from different distros? multi-years 
into the future?


Scott.
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Re: How does Portage prioritze emerges in emerge world?

[Richard Fish]

On 11/28/05, Scott Stoddard <scott@cs.ubishops.ca> wrote:
> Neil Bothwick wrote:
> >
> > This thread is all about compiling kernel modules, which must be specific
> > to either the running kernel or the intended kernel. Most packages take
> > the latter option, using the symlink, which causes problems if the
> > symlink is updated during an emerge -u world.
> >
>
> IMO exactly why no one should ever really use the symlink USE flag...
>
> (Haven't read the whole thread either...long since deleted from my
> mailserver.)
>
> Depending on the output of uname -r is foolish, as witnessed by the

I don't think anybody was advocating _depending_ upon the output of
uname -r, but using that to find the sources for the currently running
kernel.  As in:

ls -l /lib/modules/`uname -r`/build

Using uname -r in this context is unlikely to ever break, as then
module loading would be broken.

-Richard

-- 
gentoo-user@gentoo.org mailing list