[gentoo-user] Upgrading old kernel

[gentoo-user] Upgrading old kernel

[james]

Hello,


So I have a gentoo system, with a 3.18.25 kernel.

It works fabulously, but it is time to upgrade, as most codes dependent 
on old software, have been migrated.

So should I skip to a version 5 kernel?
If so which one? I usually run hundreds of testing packages so maybe 
make the new system all testing?


Note: Not interested in using genkernel; prefer discrete, old-fashions 
method of understandable steps;
but perhaps scripted...


I run many old gentoo systems, from embedded hacks on 32 bit micros to 
AMD centric workstations (old but reliable). So, I really would like a 
system/guide that is general, but where I can add details for the 
various hardware/architectures/ issues. It seems like for 18 years, it's 
been a 'whatever reboots' incoherent system
and I'm tire of it.


I need structure and consistency, as I might not upgrade/install any 
system for months/years
and then I need to do a dozen or more, not necessarily of the same 
hardware.  AMD and ARM are what they are mostly, but other variants are 
in my collective. Hardware is like kids, it's hard for me to let go; ymmv.

BTW, a guide that is BTRFS centric, would really be cool....

All discussion/suggestions are welcome, including comments on the links 
below.

Is this the best guide to follow?


https://www.kernel.org/doc/html/latest/

or this one?

https://wiki.gentoo.org/wiki/Kernel/Upgrade

https://wiki.gentoo.org/wiki/Kernel

and this I found this noob_simplified_doc:

https://blog.khmersite.net/2019/01/upgrade-kernel-on-gentoo/

Really looking for input for one generalized guide that can have options 
for various hardware combos.


curiously,
James

Re: [gentoo-user] Upgrading old kernel

[Rich Freeman]

On Wed, Apr 15, 2020 at 11:27 AM james <garftd@verizon.net> wrote:
>
> It works fabulously, but it is time to upgrade, as most codes dependent
> on old software, have been migrated.
>
> So should I skip to a version 5 kernel?
> If so which one? I usually run hundreds of testing packages so maybe
> make the new system all testing?

If you're more of the mindset of stability over features (as seems to
be the case) then I'd stick with a longterm kernel.  That means years
of updates that basically shouldn't require anything more than running
make oldconfig to deal with.  Once in a VERY rare while a new option
shows up.

You should be updating your kernel regularly to address security
issues and other regressions.  If you stay within the same major.minor
series you shouldn't be getting anything other than bugfixes.

I personally use the latest longterm, but not until it has been out
for a few months.  Mainly this is because I use zfs and don't want to
deal with what versions of the one are compatible with what versions
of the other.

Right now I'm on the 4.19 longterm, and I'm getting to the point where
I'm contemplating switching to the 5.4 longterm.  If I were in your
shoes i'd be looking at 5.4 unless there is a reason not to.

If you're asking how to actually compile/install/etc a kernel just
follow the docs, but you should be doing this regularly.  Jumping from
3.18 you're somewhat more likely to run into issues - your biggest
headache though will be dealing with the 30,000 prompts you get from
make oldconfig and making sure you set all the new options correctly.
You won't get that problem going between two patch-level releases (eg
5.4.31 -> 5.4.32).

-- 
Rich

Re: [gentoo-user] Upgrading old kernel

[Andreas Stiasny]

On 15.04.20 17:50, Rich Freeman wrote:

> Jumping from
> 3.18 you're somewhat more likely to run into issues - your biggest
> headache though will be dealing with the 30,000 prompts you get from
> make oldconfig and making sure you set all the new options correctly.

That's why I use make olddefconfig in such a case. This takes all the 
old config values and uses the default for the new ones. If you know 
that you need one or more of the new config options you can fine tune 
them afterwards with make menuconfig.


Andreas

Re: [gentoo-user] Upgrading old kernel

[james]

On 4/15/20 1:40 PM, Andreas Stiasny wrote:
> On 15.04.20 17:50, Rich Freeman wrote:
> 
>> Jumping from
>> 3.18 you're somewhat more likely to run into issues - your biggest
>> headache though will be dealing with the 30,000 prompts you get from
>> make oldconfig and making sure you set all the new options correctly.
> 
> That's why I use make olddefconfig in such a case. This takes all the 
> old config values and uses the default for the new ones. If you know 
> that you need one or more of the new config options you can fine tune 
> them afterwards with make menuconfig.
> 
> 
> Andreas

Ah. never used olddefconfig, I'll give it a spin.

Re: [gentoo-user] Upgrading old kernel

[james]

On 4/15/20 11:40 AM, Rich Freeman wrote:
> On Wed, Apr 15, 2020 at 11:27 AM james <garftd@verizon.net> wrote:
>>
>> It works fabulously, but it is time to upgrade, as most codes dependent
>> on old software, have been migrated.
>>
>> So should I skip to a version 5 kernel?
>> If so which one? I usually run hundreds of testing packages so maybe
>> make the new system all testing?
> 
> If you're more of the mindset of stability over features (as seems to
> be the case) then I'd stick with a longterm kernel.  That means years
> of updates that basically shouldn't require anything more than running
> make oldconfig to deal with.  Once in a VERY rare while a new option
> shows up.

Traditionally yes, but not going forward. About 1/2 are on (going to be) 
the latest and I'll probably just default to every package being the 
latest testing, github or whatever version.


> You should be updating your kernel regularly to address security
> issues and other regressions.  If you stay within the same major.minor
> series you shouldn't be getting anything other than bugfixes.

Agreed, but most of my systems  rarely have a route to the internet or 
are mostly not connected to any ethernet, continuously.


> I personally use the latest longterm, but not until it has been out
> for a few months.  Mainly this is because I use zfs and don't want to
> deal with what versions of the one are compatible with what versions
> of the other.

Yep, for the main system, but using btrfs with redundant drives. I'd 
like zfs, but not certain about it's future being open, open-source, 
etc. btrfs has bee great, for what I have done recently.


> Right now I'm on the 4.19 longterm, and I'm getting to the point where
> I'm contemplating switching to the 5.4 longterm.  If I were in your
> shoes i'd be looking at 5.4 unless there is a reason not to.

5.4 sounds good.

> If you're asking how to actually compile/install/etc a kernel just
> follow the docs, but you should be doing this regularly.  Jumping from
> 3.18 you're somewhat more likely to run into issues - your biggest
> headache though will be dealing with the 30,000 prompts you get from
> make oldconfig and making sure you set all the new options correctly.
> You won't get that problem going between two patch-level releases (eg
> 5.4.31 -> 5.4.32).

Agreed. I was bad sick, off and on for 3 years. Rare blood sugar.....

80% protein diet fixed it all. NO medications, no sugar very few slow 
carbs, finally. So, basically my mind was 80% erased. Good thing I kept 
notes and a myriad of sporadic 'howto docs'.

Kernel hacking was void for 3 years. Now I feel GREAT and have many 
gentoo ambitions, 5G and embedded centric stuff; but also a mail and a 
web server, with very tight security. DNS primaries on little, ram 
intensive arm boards, are pretty sweet when combined  with cloudflare's 
free, secure dns.


Thank for all the help/ideas,
James

Re: [gentoo-user] Upgrading old kernel

[Rich Freeman]

On Wed, Apr 15, 2020 at 2:31 PM james <garftd@verizon.net> wrote:
>
> On 4/15/20 11:40 AM, Rich Freeman wrote:
>
> > I personally use the latest longterm, but not until it has been out
> > for a few months.  Mainly this is because I use zfs and don't want to
> > deal with what versions of the one are compatible with what versions
> > of the other.
>
> Yep, for the main system, but using btrfs with redundant drives. I'd
> like zfs, but not certain about it's future being open, open-source,
> etc. btrfs has bee great, for what I have done recently.
>

So, a few comments here:

First, I used to use btrfs and I'd say it is just as important to
stick with a longterm using btrfs because that project has a terrible
history of introducing regressions in new kernel releases.  If I were
using btrfs I'm not sure I'd even go with 5.4 over 4.19 as it has only
been around a few months and I'd be concerned they haven't worked out
all the btrfs bugs yet.

Now, I haven't used btrfs recently, so maybe things have gotten much
better, but I'm skeptical on that front.  I've had to do complete
btrfs reinstalls more than once from backups, and this was on btrfs
raid1 only.  I REALLY like the feature set and design/etc of btrfs and
think it definitely could be the future of linux mainstream storage,
but for whatever reason QA has been a big problem and it has taken way
longer than I expected to mature.  It was btrfs QA problems that drove
me to pay such close attention to what kernel series I was running.

That is why I've mainly moved to zfs as my main general-purpose
filesystem on hosts where restoring from backup isn't about popping an
SDcard out of a Pi and flashing a couple GB backup image onto it.

I'm not entirely happy with some of the limitations of zfs and of
course it not being in-mainline is a huge hassle.  There really is no
risk of it not being FOSS - it is FOSS and of course it always will be
as is the case with anything FOSS.  Whether anybody is contributing to
it in 10 years is another matter, but it isn't like the license has an
expiry date on it.  The #3 openzfs contributor is a Gentoo dev.  I
suspect the main risk to zfs is that btrfs finally gets its quality
level up sufficiently that people switch over, which would be great.
Either that or zfs gets sloppy with QA and people abandon it, which
would be terrible, but probably unlikely at this point.

For larger-scale storage I'm using Lizardfs and greatly admire Ceph as
well in this space.  MooseFS is another option (which Lizardfs is a
fork of).  These distributed filesystems are generally more flexible
than zfs and give you redundancy above the host level.  Right now the
bulk of my storage is on lizardfs with the lizardfs chunkservers being
implemented on top of zfs.  That gives me the data security benefits
of zfs but without the inflexibility, since I don't pool drives so I'm
not limited by the ability to add/remove/etc drives from zfs pools.
That said, vdev removal has become a thing in v0.8 and perhaps we'll
see increased flexibility in the future.

Overall zfs and btrfs are actually converging somewhat, just from
different target audiences.  IMO with the growing importance of
distributed filesystems I think that the main niche for zfs and btrfs
will be as a general-purpose filesystem similar to ext4 but with
additional flexibility (volume management) and robustness
(raid/checksums/etc).  Once you get bigger than a few drives Ceph will
become the gold standard for storage, or at least that is the leading
technology right now.  Lizardfs is more of a ghetto Ceph that doesn't
require dozens of GB of RAM per server.

If you haven't been upgrading kernels you may have just missed all the
fun of btrfs regressions over recent years.  :)  In any case unless
things have changed a lot I'd seriously consider longterms and then
carefully checking for regressions before doing upgrades between major
versions.

-- 
Rich

Re: [gentoo-user] Upgrading old kernel

[Thomas Mueller]

On 4/15/20 1:40 PM, Andreas Stiasny wrote:
> On 15.04.20 17:50, Rich Freeman wrote:
 
>> Jumping from
>> 3.18 you're somewhat more likely to run into issues - your biggest
>> headache though will be dealing with the 30,000 prompts you get from
>> make oldconfig and making sure you set all the new options correctly.
 
> That's why I use make olddefconfig in such a case. This takes all the 
> old config values and uses the default for the new ones. If you know 
> that you need one or more of the new config options you can fine tune 
> them afterwards with make menuconfig.
 
 
> Andreas

james responded:

> Ah. never used olddefconfig, I'll give it a spin.

That raises the question, what if you have no kernel config, as may be the case if you are going to Gentoo for the first time, or are cross-compiling from FreeBSD or NetBSD?

I have tried with OpenADK (www.openadk.org), which got as far as successfully building cross-gcc some of the time, but never succeeded at building the kernel.

Is defconfig the best starting point?  One would want to maximize the probability of success building the kernel while retaining a functional system that would support vital hardware including ethernet, wi-fi, hard drives and USB, and I would need to be able to read a NetBSD or FreeBSD file system (UFS/FFSv1 or 2).  I use GPT, so there are no traditional now-deprecated BSD disklabels that Linux would not recognize.

If I just start with menuconfig, I could miss some vital parts.

OpenADK started with a minimal kernel config, maybe it was too minimal?

I have successfully compiled kernels and userlands on FreeBSD and NetBSD (no menuconfig, defconfig, etc; kernel configs start with a GENERIC config).  NetBSD kernel config is much longer than FreeBSD kernel config but is dwarfed by Linux kernel config.

Tom

[gentoo-user] Re: Upgrading old kernel

[Nikos Chantziaras]

On 15/04/2020 21:09, james wrote:
> On 4/15/20 1:40 PM, Andreas Stiasny wrote:
>>
>> That's why I use make olddefconfig in such a case. This takes all the 
>> old config values and uses the default for the new ones. If you know 
>> that you need one or more of the new config options you can fine tune 
>> them afterwards with make menuconfig.
> 
> Ah. never used olddefconfig, I'll give it a spin.

Note that there's sys-kernel/gentoo-kernel-bin now. You can use that 
without having to build anything. Or, you can use that at first, and 
then use its config in sys-kernel/gentoo-sources and tweak it to your needs.

There's also sys-kernel/gentoo-kernel but it's description is confusing 
as hell: "Linux kernel built with Gentoo patches". Which to me sounds 
exactly like gentoo-kernel-bin just with slightly different wording... :-/

Re: [gentoo-user] Re: Upgrading old kernel

[Ashley Dixon]

[-- Attachment #1: Type: text/plain, Size: 944 bytes --]

On Thu, Apr 16, 2020 at 10:08:45AM +0300, Nikos Chantziaras wrote:
> There's also sys-kernel/gentoo-kernel but it's description is confusing as
> hell: "Linux kernel built with Gentoo patches". Which to me sounds exactly
> like gentoo-kernel-bin just with slightly different wording... :-/

The difference can be seen in the ebuild source U.R.I.s:
        gentoo-kernel-bin: https://dev.gentoo.org/~mgorny/binpkg/amd64/kernel/ \
                sys-kernel/gentoo-kernel/${MY_P}.xpak
        gentoo-kernel: https://cdn.kernel.org/pub/linux/kernel/ ... + 
                patches from https://dev.gentoo.org/~mpagano/dist/genpatches

The wording is horrible, but as the suffix ("-bin") would suggest, the former is
a binary  .xpak  package,  whereas  the  latter  is  base  kernel  sources  with
additional Gentoo developers' patches.

-- 

Ashley Dixon
suugaku.co.uk

2A9A 4117
DA96 D18A
8A7B B0D2
A30E BF25
F290 A8AA


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[gentoo-user] Re: Upgrading old kernel

[Nikos Chantziaras]

On 16/04/2020 10:21, Ashley Dixon wrote:
> On Thu, Apr 16, 2020 at 10:08:45AM +0300, Nikos Chantziaras wrote:
>> There's also sys-kernel/gentoo-kernel but it's description is confusing as
>> hell: "Linux kernel built with Gentoo patches". Which to me sounds exactly
>> like gentoo-kernel-bin just with slightly different wording... :-/
> 
> The difference can be seen in the ebuild source U.R.I.s:
>          gentoo-kernel-bin: https://dev.gentoo.org/~mgorny/binpkg/amd64/kernel/ \
>                  sys-kernel/gentoo-kernel/${MY_P}.xpak
>          gentoo-kernel: https://cdn.kernel.org/pub/linux/kernel/ ... +
>                  patches from https://dev.gentoo.org/~mpagano/dist/genpatches
> 
> The wording is horrible, but as the suffix ("-bin") would suggest, the former is
> a binary  .xpak  package,  whereas  the  latter  is  base  kernel  sources  with
> additional Gentoo developers' patches.

I still don't understand what that means. If it's a pre-built kernel, 
then it sounds the same as gentoo-kernel-bin. If it's sources, then it 
sounds the same as gentoo-sources.

Re: [gentoo-user] Upgrading old kernel

[james]

On 4/15/20 10:59 PM, Thomas Mueller wrote:
> On 4/15/20 1:40 PM, Andreas Stiasny wrote:
>> On 15.04.20 17:50, Rich Freeman wrote:
>   
>>> Jumping from
>>> 3.18 you're somewhat more likely to run into issues - your biggest
>>> headache though will be dealing with the 30,000 prompts you get from
>>> make oldconfig and making sure you set all the new options correctly.
>   
>> That's why I use make olddefconfig in such a case. This takes all the
>> old config values and uses the default for the new ones. If you know
>> that you need one or more of the new config options you can fine tune
>> them afterwards with make menuconfig.
>   
>   
>> Andreas
> 
> james responded:
> 
>> Ah. never used olddefconfig, I'll give it a spin.
> 
> That raises the question, what if you have no kernel config, as may be the case if you are going to Gentoo for the first time, or are cross-compiling from FreeBSD or NetBSD?
> 
> I have tried with OpenADK (www.openadk.org), which got as far as successfully building cross-gcc some of the time, but never succeeded at building the kernel.
> 
> Is defconfig the best starting point?  One would want to maximize the probability of success building the kernel while retaining a functional system that would support vital hardware including ethernet, wi-fi, hard drives and USB, and I would need to be able to read a NetBSD or FreeBSD file system (UFS/FFSv1 or 2).  I use GPT, so there are no traditional now-deprecated BSD disklabels that Linux would not recognize.
> 
> If I just start with menuconfig, I could miss some vital parts.
> 
> OpenADK started with a minimal kernel config, maybe it was too minimal?
> 
> I have successfully compiled kernels and userlands on FreeBSD and NetBSD (no menuconfig, defconfig, etc; kernel configs start with a GENERIC config).  NetBSD kernel config is much longer than FreeBSD kernel config but is dwarfed by Linux kernel config.
> 
> Tom

OK, time to spill 'the beans'.

OpenADK does not look like a kernel building tool. Booting a minimal 
state-machine for an embedded device, starts at the bottom of the code 
blocks.


Building a linux kernel, that runs on the bottom of processors, guaged 
by resources and capabilities has always been a 'pita' that is nothing 
but duress. As you down the tree of what micro-processors can do, and 
the limited (kernel/system) resources, limited instruction sets, etc, 
etc  that need is not common and you are best off following a well worn 
path. Folks that do not deeply understand the lack or limited (uP) 
resources and the subsequent limitied options available, need to get 
into a good, university program or go write assemble code on uPs for a 
few years. In essence, that sort of approach is a giant waste of time. 
I.E. follow a well worn path and learn to code in C and assembler. 
Executive, minimal OS and such, written in forth or other such languages 
are shear folly.  C and Assembler, for find something else to do with 
your time, wisely.


However, that said, integrating certain processor family trees into what 
other, more sophisticated 64 bit arm projects are doing, particularly 
with low level codes, is a wise idea. Pick your battles wisely. Caveat 
Emptor!


Back to my thread::
An older, existing system is very rich in unique work and codes, at 
least for me, so I keep old image-systems, around for decades. A Gentoo 
packrat, as I can quote from very smart people back to 2004, when 
necessary. Their words were and are true, but, let's focus on 
virgin/noob (kernel-centric) systems issues.


Fast Forward.
I use the install disk from CloverOS which does a wonderful job of auto* 
for recent kernels and many packages. However it is not portage_raw or 
the myriad of other places to download and install or hack ebuilds; not 
necessarily of the Gentoo-approved feedstock. ymmv.


CloverOS will give you a clean, new kernel, but lacks a window system (I 
do not use anything big, regardless of system resources) that is 
functional but not robust, imho.


But 10 (15 max) minutes for a gentoo install is just freaking awesome, 
btrfs and a new kernel 5.* kernel.
I have not tried to just copy over a kernel and associated file, but 
that is on the list as I have (3) identical AMD systems, 64bit, with AMD 
video cards and 32 G of DDR3 memory. But I shall just try to copy over a 
kernel derived from the CloverOS gentoo to an indentical hardware system 
running a version 3.18 james-derived-and-build linux kernel, just to see 
what happens.


So a hybrid technique to rapidly test pre-built kernels, in an automated 
fashion, then going back and duplicating the same kernel-builds-tests 
from a kernel-gentoo-source-package, would be and attractive experiment 
to me.


One off (which is what we do as a gentoo collective of hacks) kernels 
with the build-test-repeat cycle seems like an arcane semantic for a 
collective of experts (or fledgling gentoo-soon-to-be-experts)in this 
day and age (strictly of my opine).


Where I'm going, managing a collective of hundreds or thousands of 
gentoo derivative systems (from small arm embedded) to workstations and 
servers, is way, way past due an automated and auto-managed tool-system. 
So, I'm not the best person to weigh in here. This problem has been 
solved, hundreds of times, by really smart people, but most all have 
gone commercial::


Gentoo spawned CoreOS (multi-booting-many-kernel choices-automated) 
which was purchased by Redhat, which experienced a hostile takeover by 
IBM. IBM is now filing bankruptcy, to get billions of tax payer dollars 
from the US taxpayers, whilst running full steam ahead with a 
gentoo-derivative OS. I'm not sure why the good work at CoreOS, did not 
make it back into the Gentoo mainstream.
(Rich or some other smart Gentooer could/should enlighten us?).


So is it not time for a few, very smart Gentoo devs to build us a simple 
tool? Not one that does everything, but that does enough to make/track 
new kernel builds streamlined and easy? Every system should have a dozen 
or more viable kernels for a wide variety of reasons (imho.).


Then the greater gentoo community could go off, each in their own, 
tractor inspired, direction with 'tuned, optimized and secure' unique 
kernels to plow the fields they plant?  (a plea for help?)? leadership?


Was Andrew Gaffney booted out of Gentoo, for similar ideas as James? 
Surely he has superior skills.....

https://blogs.gentoo.org/agaffney/2006/01/18/installer_philosophy/



I could tagged hundreds, but what's the point.....


James

[gentoo-user] Re: Upgrading old kernel

[Jonathan Callen]

[-- Attachment #1.1: Type: text/plain, Size: 2037 bytes --]

On 4/16/20 3:43 AM, Nikos Chantziaras wrote:
> On 16/04/2020 10:21, Ashley Dixon wrote:
>> On Thu, Apr 16, 2020 at 10:08:45AM +0300, Nikos Chantziaras wrote:
>>> There's also sys-kernel/gentoo-kernel but it's description is
>>> confusing as
>>> hell: "Linux kernel built with Gentoo patches". Which to me sounds
>>> exactly
>>> like gentoo-kernel-bin just with slightly different wording... :-/
>>
>> The difference can be seen in the ebuild source U.R.I.s:
>>          gentoo-kernel-bin:
>> https://dev.gentoo.org/~mgorny/binpkg/amd64/kernel/ \
>>                  sys-kernel/gentoo-kernel/${MY_P}.xpak
>>          gentoo-kernel: https://cdn.kernel.org/pub/linux/kernel/ ... +
>>                  patches from
>> https://dev.gentoo.org/~mpagano/dist/genpatches
>>
>> The wording is horrible, but as the suffix ("-bin") would suggest, the
>> former is
>> a binary  .xpak  package,  whereas  the  latter  is  base  kernel 
>> sources  with
>> additional Gentoo developers' patches.
> 
> I still don't understand what that means. If it's a pre-built kernel,
> then it sounds the same as gentoo-kernel-bin. If it's sources, then it
> sounds the same as gentoo-sources.
> 

It's neither.  The package installs a kernel (not just sources for the
kernel), like gentoo-kernel-bin, but it does so by building it locally,
just like just about every other package that doesn't have a name ending
in "-bin". On the other hand, gentoo-kernel-bin downloads a precompiled
kernel and just installs it locally, not compiling anything.  The
gentoo-sources package also doesn't compile anything, it just dumps some
source code on your system for you to compile outside of portage.

In short:
                  |  Downloaded  |  Installed
------------------+--------------+-------------
gentoo-sources    | source code  | source code
gentoo-kernel     | source code  | built kernel
gentoo-kernel-bin | built kernel | built kernel

-- 
Jonathan Callen


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 981 bytes --]