[gentoo-dev] [RFC] A proposal regarding packages that provide kernel modules

[gentoo-dev] [RFC] A proposal regarding packages that provide kernel modules

[Georgi Georgiev]

Installing packages that provide kernel modules in Gentoo is a real pain.
Several packages that are among the annoying ones are mplayer and svgalib. They
only provide a single module, but would require complete recompilation on a
kernel upgrade. Since they are not SLOTted, reverting to an older version of
the kernel requires another recompilation.

I want to suggest a different system in respect to installing kernel modules.

- Kernel modules are not to be recorded in /var/db/pkg/category/package/CONTENT
  at all. This is the point that may be the most conflicting, but I personally
  do not think that this would affect things a great deal. The
  /lib/modules/<version> directories have plenty of files that are not recorded
  anywhere anyway (files created by "make -C /usr/src/linux/ modules_install)

- Introduce new functions that can be included in ebuilds called something like
  compile_modules() install_modules(). Also introduce a --modules (or similar)
  switch to portage that will only execute those functions and thus effectively
  only install the modules that a package provides.

- emerge --modules cannot be executed if a package is not emerged already.

- Emerging a package would, of course, also install the modules. (i.e. "emerge
  --modules" is a subset of "emerge")

- Since emerge --modules can only be run on an already emerged package, and
  since the files it installs are not recorded in CONTENTS there is no need to
  touch the files in /var/db/pkg on an "emerge --modules".

- Emerging a package that provides kernel modules records the package to
  /var/cache/edb/modules in a manner similar to world. This way the user can
  simply "emerge --modules modules" and they can install modules to their new
  kernel.

Expecting your comments and I hope we reach some kind of solution.

-- 
/\   Georgi Georgiev   /\ Ignorance must certainly be bliss or there   /\
\/    chutz@gg3.net    \/ wouldn't be so many people so resolutely     \/
/\  +81(90)6266-1163   /\ pursuing it.                                 /\

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] [RFC] A proposal regarding packages that provide kernel modules

[Chris Gianelloni]

[-- Attachment #1: Type: text/plain, Size: 2577 bytes --]

On Tue, 2003-09-16 at 07:33, Georgi Georgiev wrote:
> Installing packages that provide kernel modules in Gentoo is a real pain.
> Several packages that are among the annoying ones are mplayer and svgalib. They
> only provide a single module, but would require complete recompilation on a
> kernel upgrade. Since they are not SLOTted, reverting to an older version of
> the kernel requires another recompilation.
> 
> I want to suggest a different system in respect to installing kernel modules.
> 
> - Kernel modules are not to be recorded in /var/db/pkg/category/package/CONTENT
>   at all. This is the point that may be the most conflicting, but I personally
>   do not think that this would affect things a great deal. The
>   /lib/modules/<version> directories have plenty of files that are not recorded
>   anywhere anyway (files created by "make -C /usr/src/linux/ modules_install)
> 
> - Introduce new functions that can be included in ebuilds called something like
>   compile_modules() install_modules(). Also introduce a --modules (or similar)
>   switch to portage that will only execute those functions and thus effectively
>   only install the modules that a package provides.
> 
> - emerge --modules cannot be executed if a package is not emerged already.
> 
> - Emerging a package would, of course, also install the modules. (i.e. "emerge
>   --modules" is a subset of "emerge")
> 
> - Since emerge --modules can only be run on an already emerged package, and
>   since the files it installs are not recorded in CONTENTS there is no need to
>   touch the files in /var/db/pkg on an "emerge --modules".
> 
> - Emerging a package that provides kernel modules records the package to
>   /var/cache/edb/modules in a manner similar to world. This way the user can
>   simply "emerge --modules modules" and they can install modules to their new
>   kernel.
> 
> Expecting your comments and I hope we reach some kind of solution.

I like it.

As it stands right now, when I use a new kernel, I have to remerge
nvidia-kernel, lm-sensors, cisco-vpnclient-3des, pcmcia-cs, and re-run
/opt/vmware/bin/vmware-config.pl to rebuild all the modules.  This can
be a serious pain and something I forget to do quite often, especially
when testing out kernels.  My current solution is to move all of the
modules that are built into a new location under /lib/modules/<ver> to
keep them from being automatically removed with the next merge.

-- 
Chris Gianelloni
Developer, Gentoo Linux
Games Team

Is your power animal a pengiun?

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] [RFC] A proposal regarding packages that provide kernel modules

[Thomas de Grenier de Latour]

En réponse à Georgi Georgiev <chutz@gg3.net>:

> Expecting your comments and I hope we reach some kind of
solution.

I was also often upset by portage making some cleanup in my
/lib/modules directory, so I've wrote sometime ago a small 
patch which allow to protect some files from cleaning if 
they are in one of the paths defined in a make.conf var: 
CLEAN_PROTECT. I've submit it as bug #24990. The version on 
bugzilla is not really good though, because protected files
will be completly forgotten by portage after the package 
unmerge/clean.
  I've got another version which does the following:
 - protection is done against "clean" (when 


-- 
TGL.

En réponse à Chris Gianelloni <wolf31o2@gentoo.org>:

> I like it.
> 
> As it stands right now, when I use a new kernel, I have to
remerge
> nvidia-kernel, lm-sensors, cisco-vpnclient-3des,
pcmcia-cs, and re-run
> /opt/vmware/bin/vmware-config.pl to rebuild all the
modules.  This can
> be a serious pain and something I forget to do quite
often, especially
> when testing out kernels.  My current solution is to move
all of the
> modules that are built into a new location under
/lib/modules/<ver> to
> keep them from being automatically removed with the next
merge.
> 

I also think that something really need to be done about
drivers in 
portage. The solution proposed on bug #3120 (multiple slots
for a same 
pkg-ver, and slotting drivers to $KV as some are already) is
elegant, 
but it seems hard to implement. It may be a too deep change
for a 
very specific issue. (I've not found any other example where
multiple
slots may be usefull, but if you have some, please share).

Solution proposed by Georgi may be easier, and the "modules" 
counterpart to "world" would be nice. I also like it.

That said, I've started myself to implement yet another,
easier, a 
solution to this problem, and it's almost already finished.
It's not
very sofisticated though:
 
 - user can define a list of path to protect in his make.conf.
 CLEAN_PROTECT="/lib/modules" is a good value in our case.
 
 - when a package is "safely unmerged" (what happens if you
install
 foo/bar-1.0 twice), then files within CLEAN_PROTECT path
and that 
 should have been removed are instead protected and added to
the 
 new CONTENTS file of foo/bar-1.0.
 
 - when you upgrade foo/bar-1.0 to foo/bar-1.1, and have
"autoclean"
 feature, or do a manual "emerge -c", the same happens:
protected 
 files from 1.0 are added to the CONTENTS of 1.1, instead of
being 
 deleted. (if the package is in several version in several
slots, 
 then the protecting pkg will always be the one in the same
slot, 
 but in case of "prune". The logic is exactly the one of the
clean or
 prune in emerge.)

 - when you really want to uninstall foo/bar, "emerge -C"
won't take 
 the protection into account. Because the CONTENTS file is
up to date
 with all previous versions of the files, it will remove
then all. 
 Hence the day you switch from nvidia to ati, "emerge -C
nvidia-kernel"
 will do the cleanup for all kernels modules directories.
 
So far it seems to work for me. I have files from several
modules dirs
in my alsa-driver and nvidia-kernel CONTENTS files. But
there are a few
limitations: I've not implemented protection for objects
other than 
files (dirs, symlinks, fifo). Mainly because I don't need it
though. 
That's why I've not submitted it so far. (In fact, the first
version 
of this patch is submitted as bug #24990, but it's really
not good 
because it makes portage completly forget the existence of
files that 
have been protected.)

In short, this approach is okay for keeping modules
installed in 
several kernels without manual backup (well, at least I use
it), but is
not specific to modules, so for this precise use it is less
powerful 
that what Georgi suggested (no "modules" world-like file and no 
"install modules only" ebuild functions). But it may also be
useful for
a few other task (for instance keeping all icons/backgrounds
that come 
with a WM theme package, accross several versions).

Any comments on the above ideas are welcome. I will also
submit an 
up-to-date patch in bug #24990, but not before the next few
hours 
(I'm not at home right now).

-- 
TGL.

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] [RFC] A proposal regarding packages that provide kernel modules

[Thomas de Grenier de Latour]

En réponse à Chris Gianelloni <wolf31o2@gentoo.org>:

> I like it.
> 
> As it stands right now, when I use a new kernel, I have to
remerge
> nvidia-kernel, lm-sensors, cisco-vpnclient-3des,
pcmcia-cs, and re-run
> /opt/vmware/bin/vmware-config.pl to rebuild all the
modules.  This can
> be a serious pain and something I forget to do quite
often, especially
> when testing out kernels.  My current solution is to move
all of the
> modules that are built into a new location under
/lib/modules/<ver> to
> keep them from being automatically removed with the next
merge.
> 

I also think that something really need to be done about
drivers in 
portage. The solution proposed on bug #3120 (multiple slots
for a same 
pkg-ver, and slotting drivers to $KV as some are already) is
elegant, 
but it seems hard to implement. It may be a too deep change
for a 
very specific issue. (I've not found any other example where
multiple
slots may be usefull, but if you have some, please share).

Solution proposed by Georgi may be easier, and the "modules" 
counterpart to "world" would be nice. I also like it.

That said, I've started myself to implement yet another,
easier, a 
solution to this problem, and it's almost already finished.
It's not
very sofisticated though:
 
 - user can define a list of path to protect in his make.conf.
 CLEAN_PROTECT="/lib/modules" is a good value in our case.
 
 - when a package is "safely unmerged" (what happens if you
install
 foo/bar-1.0 twice), then files within CLEAN_PROTECT path
and that 
 should have been removed are instead protected and added to
the 
 new CONTENTS file of foo/bar-1.0.
 
 - when you upgrade foo/bar-1.0 to foo/bar-1.1, and have
"autoclean"
 feature, or do a manual "emerge -c", the same happens:
protected 
 files from 1.0 are added to the CONTENTS of 1.1, instead of
being 
 deleted. (if the package is in several version in several
slots, 
 then the protecting pkg will always be the one in the same
slot, 
 but in case of "prune". The logic is exactly the one of the
clean or
 prune in emerge.)

 - when you really want to uninstall foo/bar, "emerge -C"
won't take 
 the protection into account. Because the CONTENTS file is
up to date
 with all previous versions of the files, it will remove
then all. 
 Hence the day you switch from nvidia to ati, "emerge -C
nvidia-kernel"
 will do the cleanup for all kernels modules directories.
 
So far it seems to work for me. I have files from several
modules dirs
in my alsa-driver and nvidia-kernel CONTENTS files. But
there are a few
limitations: I've not implemented protection for objects
other than 
files (dirs, symlinks, fifo). Mainly because I don't need it
though. 
That's why I've not submitted it so far. (In fact, the first
version 
of this patch is submitted as bug #24990, but it's really
not good 
because it makes portage completly forget the existence of
files that 
have been protected.)

In short, this approach is okay for keeping modules
installed in 
several kernels without manual backup (well, at least I use
it), but is
not specific to modules, so for this precise use it is less
powerful 
that what Georgi suggested (no "modules" world-like file and no 
"install modules only" ebuild functions). But it may also be
useful for
a few other task (for instance keeping all icons/backgrounds
that come 
with a WM theme package, accross several versions).

Any comments on the above ideas are welcome. I will also
submit an 
up-to-date patch in bug #24990, but not before the next few
hours 
(I'm not at home right now).

-- 
TGL.

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] [RFC] A proposal regarding packages that provide kernel modules

[Thomas de Grenier de Latour]

Sorry for the wrong formatted double-post... I really should
not use this IMP stuff when I'm at work :|

-- 
TGL.

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] [RFC] A proposal regarding packages that provide kernel modules

[Thomas de Grenier de Latour]

On Tue, 16 Sep 2003 17:13:00 +0200 (CEST)
Thomas de Grenier de Latour <degrenier@easyconnect.fr> wrote:

> Any comments on the above ideas are welcome. I will also
> submit an up-to-date patch in bug #24990, but not before
> the next few hours 

Patch uploaded if someone wants to give it a try:
http://bugs.gentoo.org/show_bug.cgi?id=24990

-- 
TGL.

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] [RFC] A proposal regarding packages that provide kernel modules

[Mark Francis]

Thomas de Grenier de Latour wrote:

>On Tue, 16 Sep 2003 17:13:00 +0200 (CEST)
>Thomas de Grenier de Latour <degrenier@easyconnect.fr> wrote:
>
>  
>
>>Any comments on the above ideas are welcome. I will also
>>submit an up-to-date patch in bug #24990, but not before
>>the next few hours 
>>    
>>
>
>Patch uploaded if someone wants to give it a try:
>http://bugs.gentoo.org/show_bug.cgi?id=24990
>  
>

I placed a script to list packages that need to be emerged after 
compiling a new kernel in bug http://bugs.gentoo.org/show_bug.cgi?id=24990.

It's usage is:
kernel-packages | xargs emerge -pv



--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] [RFC] A proposal regarding packages that provide kernel modules

[Thomas de Grenier de Latour]

On Thu, 18 Sep 2003 08:50:20 +1000
Mark Francis <mark@canb2c.com> wrote:

> I placed a script to list packages that need to be emerged after 
> compiling a new kernel in bug
> http://bugs.gentoo.org/show_bug.cgi?id=24990.

Simple but effective :)
Thanks.

-- 
TGL.

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] [RFC] A proposal regarding packages that provide kernel modules

[Georgi Georgiev]

On 18/09/2003 at 08:50:20(+1000), Mark Francis used 0.6K just to say:
> I placed a script to list packages that need to be emerged after 
> compiling a new kernel in bug http://bugs.gentoo.org/show_bug.cgi?id=24990.
> 
> It's usage is:
> kernel-packages | xargs emerge -pv

------- Additional Comment #4 From Mark Francis  2003-09-17 15:44 EST -------
> 
> Created an attachment (id=17924)
> /usr/local/sbin/kernel-packages
> 
> This small script I developed lists all packages that need to be emerged
> after installing a new kernel. I use it this way:
> kernel-packages | xargs emerge -pv

It's no big deal, but I don't like the your word choice in that bug comment
above. The link shows what I mean.

http://marc.theaimsgroup.com/?l=gentoo-dev&m=105451683428007&w=2

-- 
 /   Georgi Georgiev    / People are beginning to notice you. Try       /
\     chutz@gg3.net    \  dressing before you leave the house.         \
 /  +81(90)6266-1163    /                                               /

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] [RFC] A proposal regarding packages that provide kernel modules

[Mark Francis]

Georgi Georgiev wrote:

>On 18/09/2003 at 08:50:20(+1000), Mark Francis used 0.6K just to say:
>  
>
>>I placed a script to list packages that need to be emerged after 
>>compiling a new kernel in bug http://bugs.gentoo.org/show_bug.cgi?id=24990.
>>
>>It's usage is:
>>kernel-packages | xargs emerge -pv
>>    
>>
>
>------- Additional Comment #4 From Mark Francis  2003-09-17 15:44 EST -------
>  
>
>>Created an attachment (id=17924)
>>/usr/local/sbin/kernel-packages
>>
>>This small script I developed lists all packages that need to be emerged
>>after installing a new kernel. I use it this way:
>>kernel-packages | xargs emerge -pv
>>    
>>
>
>It's no big deal, but I don't like the your word choice in that bug comment
>above. The link shows what I mean.
>
>http://marc.theaimsgroup.com/?l=gentoo-dev&m=105451683428007&w=2
>  
>

You are perfectly correct. I should start to comment my code so that I 
do not misremember the source of my many small scripts.

My apologies to the author of this script.

Mark


--
gentoo-dev@gentoo.org mailing list