[gentoo-server] Challenging Update Question

[gentoo-server] Challenging Update Question

[Collin Starkweather]

I have a server that is in need of a significant update, but it's  
proving a challenge.  I have a big picture question, then provide some  
details below.

(I originally thought the gentoo-admin list would be the best place to  
ask this, but based on the stats, it seems to be relatively inactive.   
Let me know if there is a better list to ask the question on.)

Please excuse the length of the question, but as you can see, there  
are a variety of variables in play.

The Big Picture
---------------

The server has not been updated since late 2005 or so.  It just runs  
Apache, mod_perl, and an application server.  So far, it has just  
hummed along doing its work without complaint, solid as a rock, which  
is why no one has bothered with it.

As you doubtless know, if you miss a couple of upgrade cycles with  
Gentoo, there can be (and has been) breakage when trying to emerge -u  
world.

There are two identical drives, and I've mirrored (manually, not RAID)  
one onto the other.

The challenge is that the box is in the U.S. and I live in China.   
There is no one there who can administer it; however, if something  
goes really wrong, someone can just swap the drives and reboot.  The  
key factor is that I want this to be as low-risk as possible since  
swapping the drives is about the extent of on-site support available.

The big picture question:  Would it (1) be simpler and easier to  
rebuild from scratch on the redundant drive, or (2) is it simpler and  
easier to deal with the current issues updating from 2005.0 and a  
2.4.x kernel?

The Details
-----------

Option (1):  Rebuilding on the Redundant Drive

Pros -- It seems this would be the easiest way to do things, and I get  
a fresh kernel and build.

Cons -- If I rebuild on the redundant drive, I lose the ability to  
swap drives if there is breakage.  Also, the application server  
(Apache Pagekit) is solid as a rock, but a real bitch to configure.   
Last time I tried to upgrade Pagekit, due to Apache versioning issues,  
configuration changes, etc., it took me a full weekend.  Not fun.

Option (2):  Upgrading from 2005.0

Pros -- Perhaps less risky (advice on this would be appreciated!) and  
I maintain another drive I can use to compare configurations,  
selectively roll things back, etc.

Cons -- The gory details.  When I did an emerge --sync, the  
/etc/make.profile symlink broke.  It used to point at

   /usr/portage/profiles/default-linux/x86/no-nptl/2.4

which no longer exists, I suppose since the 2.4.x kernels seem to no  
longer be supported.

So the questions that arise are:

1) Would it be less risky to upgrade from 2005.0 to 2007.0 than  
rebuild from scratch on the redundant drive?
2) What is no-nptl?  I don't know why the old portage profile was  
/usr/portage/.../no-nptl.  It seems to have something to do with  
glibc-2.4.  Do I still need it?  Or can I just point /etc/make.profile  
at /usr/portage/.../2007.0?
3) I pointed /etc/make.profile at /usr/portage/.../2007.0, and tried  
emerge -pu world.  I was told that mail-mta/qmail no longer existed  
and is required by sudo (?!?) which is required by libapreq2.  Perhaps  
mail-mta/netqmail is the new qmail?  Anyway, this gives me the  
impression that there are deep dependencies that may have changed  
significantly.  Is this what you would (subjectively) characterize as  
a bad sign?
4) What would be the best order of operations?  emerge -u world, then  
update the kernel, or update the kernel, then emerge -u world?
5) 2008.0 is due on March 17.  Is it worthwhile putting off the  
upgrade for 2008.0?  I wouldn't want to deal with two difficult  
upgrades if there is breakage between 2007.0 and 2008.0.

Thanks in advance,

-Collin

-- 
Collin Starkweather, Ph.D.
http://www.linkedin.com/in/collinstarkweather

--
gentoo-server@lists.gentoo.org mailing list

Re: [gentoo-server] Challenging Update Question

[RijilV]

[-- Attachment #1: Type: text/plain, Size: 504 bytes --]

On 11/02/2008, Collin Starkweather <gentoo@collinstarkweather.com> wrote:
>
> [lots of words]

Option (1):  Rebuilding on the Redundant Drive
> [still more words]



I think that pretty much sums up my view.  If you have an application that
is difficult to install, you could build out a new gentoo server somewhere
(locally, on amazon, etc) and get it working there.  Then take that
configuration (you could even just rsync+ssh the whole os over....) to that
other drive in your server and reboot.

.r'

[-- Attachment #2: Type: text/html, Size: 971 bytes --]

Re: [gentoo-server] Challenging Update Question

[W.Kenworthy]

Rebuild/upgrade on the redundant drive in a chroot
Rebuild elsewhere (local to you) on similar hardware and copy OS over.
I suspect though, that building a new system, getting it working and
shipping it as a black box would be the most low risk/effective
strategy.

Hint:
Setup grub to boot either os so local support only has to select which
disk to boot from if there is a failure.


BillK


On Mon, 2008-02-11 at 23:30 -0700, Collin Starkweather wrote:
> I have a server that is in need of a significant update, but it's  
> proving a challenge.  I have a big picture question, then provide some  
> details below.

-- 
gentoo-server@lists.gentoo.org mailing list

Re: [gentoo-server] Challenging Update Question

[Collin Starkweather]

Quoting "W.Kenworthy" <billk@iinet.net.au>:

> Rebuild/upgrade on the redundant drive in a chroot
> Rebuild elsewhere (local to you) on similar hardware and copy OS over.
> I suspect though, that building a new system, getting it working and
> shipping it as a black box would be the most low risk/effective
> strategy.
>
> Hint:
> Setup grub to boot either os so local support only has to select which
> disk to boot from if there is a failure.

Thanks for the advice.

This may seem a novice question, but can you build a 2.6 kernel and  
use it to boot a system built against 2.4?  That is, to divide the  
move into two testable components, kernel and everything else,

1) Build a full new system on the redundant drive with a 2.6 kernel
2) Copy *just* the kernel over and test it (with a menu in grub as you  
suggest in case it barfs)
3) If the kernel works, then move the rest over

Or does the kernel change enough between major iterations that you'd  
have to, say, rebuild glibc or somesuch?

Cheers,

-Collin

-- 
Collin Starkweather, Ph.D.
http://www.linkedin.com/in/collinstarkweather

--
gentoo-server@lists.gentoo.org mailing list

Re: [gentoo-server] Challenging Update Question

[Andrew Gaffney]

Collin Starkweather wrote:
> This may seem a novice question, but can you build a 2.6 kernel and use 
> it to boot a system built against 2.4?  That is, to divide the move into 
> two testable components, kernel and everything else,

This will probably bite you in the ass. Most notably, support for devfs was 
dropped around 2.6.13, so unless you are managing a static /dev, you're going to 
need udev. However, I'm not sure if the baselayout from that long ago had any 
notion of udev or if you can even build udev while running a 2.4 kernel (this is 
probably less of an issue).

Really, the "best" way to do this is to just do a new install on the second 
drive. However, you might have some of the problems noted above, since you're 
still running a 2.4 kernel while building a system intended for 2.6.

I guess that your only remaining "sane" option is to build a new install on a 
"local" machine, and then rsync the entire thing over to the 2nd drive in your 
existing machine. Then setup grub to boot from the install on the 2nd drive by 
default.

-- 
Andrew Gaffney                                 http://dev.gentoo.org/~agaffney/
Gentoo Linux Developer             Catalyst/Installer + x86 release coordinator
-- 
gentoo-server@lists.gentoo.org mailing list

Re: [gentoo-server] Challenging Update Question

[William Kenworthy]

The other way around - update the toolchain first.  That is gcc,
binutils, glibc, ...

When this is all working do a emerge world -ep to make sure everything
is up to date for 2.4 (Dont forget the expat upgrade gotchas - read the
guide!)  Use revdep-rebuild liberally!

Then do the kernel and reboot

Then bring up anything that was held back by 2.4.

It will actually be the toolchain (and probably expat) that will cause
the most grief!  Stay with a 3.x series gcc until the end - there are
upgrade guides if you want a 4 series, but wait.

I did this some time back on a large system (~1000 packages) and it does
work - and the system can stay online for almost all of it as well.  If
you have a relatively small system, it is much easier and quicker as
there are far fewer complications.

BillK

On Tue, 2008-02-12 at 05:00 -0700, Collin Starkweather wrote:
> Quoting "W.Kenworthy" <billk@iinet.net.au>:
> 
> > Rebuild/upgrade on the redundant drive in a chroot
> > Rebuild elsewhere (local to you) on similar hardware and copy OS over.
> > I suspect though, that building a new system, getting it working and
> > shipping it as a black box would be the most low risk/effective
> > strategy.
> >
> > Hint:
> > Setup grub to boot either os so local support only has to select which
> > disk to boot from if there is a failure.
> 
> Thanks for the advice.
> 
> This may seem a novice question, but can you build a 2.6 kernel and  
> use it to boot a system built against 2.4?  That is, to divide the  
> move into two testable components, kernel and everything else,
> 
> 1) Build a full new system on the redundant drive with a 2.6 kernel
> 2) Copy *just* the kernel over and test it (with a menu in grub as you  
> suggest in case it barfs)
> 3) If the kernel works, then move the rest over
> 
> Or does the kernel change enough between major iterations that you'd  
> have to, say, rebuild glibc or somesuch?
> 
> Cheers,
> 
> -Collin
> 
> -- 
> Collin Starkweather, Ph.D.
> http://www.linkedin.com/in/collinstarkweather
> 
-- 
William Kenworthy <billk@iinet.net.au>
Home in Perth!
-- 
gentoo-server@lists.gentoo.org mailing list

Re: [gentoo-server] Challenging Update Question

[Benjamen R. Meyer]

How about option #3: Build a hard drive (or hard drive set) locally for
the system and ship them to be installed?

I think that would give you the best of both of your previous options.
Granted, you'd have to go through export compliance stuff and all that,
but it might be worth the extra hassle.

Just a thought.

Ben

Collin Starkweather wrote:
> I have a server that is in need of a significant update, but it's
> proving a challenge.  I have a big picture question, then provide some
> details below.
> 
> (I originally thought the gentoo-admin list would be the best place to
> ask this, but based on the stats, it seems to be relatively inactive. 
> Let me know if there is a better list to ask the question on.)
> 
> Please excuse the length of the question, but as you can see, there are
> a variety of variables in play.
> 
> The Big Picture
> ---------------
> 
> The server has not been updated since late 2005 or so.  It just runs
> Apache, mod_perl, and an application server.  So far, it has just hummed
> along doing its work without complaint, solid as a rock, which is why no
> one has bothered with it.
> 
> As you doubtless know, if you miss a couple of upgrade cycles with
> Gentoo, there can be (and has been) breakage when trying to emerge -u
> world.
> 
> There are two identical drives, and I've mirrored (manually, not RAID)
> one onto the other.
> 
> The challenge is that the box is in the U.S. and I live in China.  There
> is no one there who can administer it; however, if something goes really
> wrong, someone can just swap the drives and reboot.  The key factor is
> that I want this to be as low-risk as possible since swapping the drives
> is about the extent of on-site support available.
> 
> The big picture question:  Would it (1) be simpler and easier to rebuild
> from scratch on the redundant drive, or (2) is it simpler and easier to
> deal with the current issues updating from 2005.0 and a 2.4.x kernel?
> 
> The Details
> -----------
> 
> Option (1):  Rebuilding on the Redundant Drive
> 
> Pros -- It seems this would be the easiest way to do things, and I get a
> fresh kernel and build.
> 
> Cons -- If I rebuild on the redundant drive, I lose the ability to swap
> drives if there is breakage.  Also, the application server (Apache
> Pagekit) is solid as a rock, but a real bitch to configure.  Last time I
> tried to upgrade Pagekit, due to Apache versioning issues, configuration
> changes, etc., it took me a full weekend.  Not fun.
> 
> Option (2):  Upgrading from 2005.0
> 
> Pros -- Perhaps less risky (advice on this would be appreciated!) and I
> maintain another drive I can use to compare configurations, selectively
> roll things back, etc.
> 
> Cons -- The gory details.  When I did an emerge --sync, the
> /etc/make.profile symlink broke.  It used to point at
> 
>   /usr/portage/profiles/default-linux/x86/no-nptl/2.4
> 
> which no longer exists, I suppose since the 2.4.x kernels seem to no
> longer be supported.
> 
> So the questions that arise are:
> 
> 1) Would it be less risky to upgrade from 2005.0 to 2007.0 than rebuild
> from scratch on the redundant drive?
> 2) What is no-nptl?  I don't know why the old portage profile was
> /usr/portage/.../no-nptl.  It seems to have something to do with
> glibc-2.4.  Do I still need it?  Or can I just point /etc/make.profile
> at /usr/portage/.../2007.0?
> 3) I pointed /etc/make.profile at /usr/portage/.../2007.0, and tried
> emerge -pu world.  I was told that mail-mta/qmail no longer existed and
> is required by sudo (?!?) which is required by libapreq2.  Perhaps
> mail-mta/netqmail is the new qmail?  Anyway, this gives me the
> impression that there are deep dependencies that may have changed
> significantly.  Is this what you would (subjectively) characterize as a
> bad sign?
> 4) What would be the best order of operations?  emerge -u world, then
> update the kernel, or update the kernel, then emerge -u world?
> 5) 2008.0 is due on March 17.  Is it worthwhile putting off the upgrade
> for 2008.0?  I wouldn't want to deal with two difficult upgrades if
> there is breakage between 2007.0 and 2008.0.
> 
> Thanks in advance,
> 
> -Collin
> 


-- 
gentoo-server@lists.gentoo.org mailing list

Re: [gentoo-server] Challenging Update Question

[Randy Barlow]

Collin Starkweather wrote:
> 5) 2008.0 is due on March 17.  Is it worthwhile putting off the upgrade
> for 2008.0?  I wouldn't want to deal with two difficult upgrades if
> there is breakage between 2007.0 and 2008.0.

Remember that Gentoo is not a versioned OS.  2007.0 and 2008.0 are just
live cd's.  You can keep updating a Gentoo machine indefinitely forever,
so there is no concept of "my machine is running 2008.0" because my
machine is as updated as the last time that I synced and ran emerge
--update --deep world.  Make sense?

-- 
Randy Barlow
http://electronsweatshop.com
-- 
gentoo-server@lists.gentoo.org mailing list

Re: [gentoo-server] Challenging Update Question

[RijilV]

[-- Attachment #1: Type: text/plain, Size: 1713 bytes --]

yeah, however if you leave a system alone without updating it for a few
years, its totally a 2006.0 or wherever you left off at. You probably aren't
going to succeed at an 'emerge -Du world' either.  The version-less system
only works if you do the work to keep yours updated.  Don't get me wrong, I
think this is way better than the rest of the dead-end distros.

And, FWIW, there is some breakage between 2007.0 and 2008.0.  Off the top of
my head the way apache modules are compiled has changed.


I would like to suggest that if you (Collin) have no intention of updating
your system, you pick another distro.  Dealing with the a broken package
once or moth or so and updating your system alteast as frequently are just
part of how Gentoo was designed.  Failure to do so is going to leave you in
a situation very much like that of any other dead-end-distro, however there
are going to be many less resources within the Gentoo community for helping
you support your system.

.r'

On 12/02/2008, Randy Barlow <randy@electronsweatshop.com> wrote:
>
> Collin Starkweather wrote:
> > 5) 2008.0 is due on March 17.  Is it worthwhile putting off the upgrade
> > for 2008.0?  I wouldn't want to deal with two difficult upgrades if
> > there is breakage between 2007.0 and 2008.0.
>
> Remember that Gentoo is not a versioned OS.  2007.0 and 2008.0 are just
> live cd's.  You can keep updating a Gentoo machine indefinitely forever,
> so there is no concept of "my machine is running 2008.0" because my
> machine is as updated as the last time that I synced and ran emerge
> --update --deep world.  Make sense?
>
> --
> Randy Barlow
> http://electronsweatshop.com
> --
> gentoo-server@lists.gentoo.org mailing list
>
>

[-- Attachment #2: Type: text/html, Size: 2263 bytes --]