From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from pigeon.gentoo.org ([69.77.167.62] helo=lists.gentoo.org) by finch.gentoo.org with esmtp (Exim 4.60) (envelope-from ) id 1KkzMb-0008AT-EO for garchives@archives.gentoo.org; Wed, 01 Oct 2008 10:55:25 +0000 Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id AF3FEE036E; Wed, 1 Oct 2008 10:55:23 +0000 (UTC) Received: from yw-out-1718.google.com (yw-out-1718.google.com [74.125.46.158]) by pigeon.gentoo.org (Postfix) with ESMTP id 65B90E036E for ; Wed, 1 Oct 2008 10:55:23 +0000 (UTC) Received: by yw-out-1718.google.com with SMTP id 5so84084ywm.46 for ; Wed, 01 Oct 2008 03:55:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:in-reply-to:mime-version:content-type:references; bh=i4UcIQnH5zvv1ZBY7tSJPw4J/wrOLYbbQGaT2n3TAX0=; b=NFnmGU179dKwKAq0r78tOw/xm5YlPSqeV81s4G/6XVLAWJbFADZdtd39Ii9SB9bBMg 0aqK+Vb/SI2Uzyb7KGpnoVIcNsINqWgEy9s6Awrdix8rczHvwVQle+H71Kocs/DL4gm1 HjzuO15hAyJt2yjLe7JaHnTTOm5aeY8Nb01Jo= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:in-reply-to:mime-version :content-type:references; b=N86PXQyAFDbxUi2qk2dJd3I4y4Tw5LNKSjSWGtI8V7Tw/S/ZreJXEQRijqpAKbYqkn pZ4hZkf0W7YhPpxBvZtn/cyk4Q6I8vd47CKjCZIy+996jh2u5dEqh09ebf2hrSmmmoGx XTLEdbaYsEqFm7N62imEEUNdGYUucsl9wfGAo= Received: by 10.100.202.14 with SMTP id z14mr7399111anf.138.1222858521614; Wed, 01 Oct 2008 03:55:21 -0700 (PDT) Received: by 10.100.163.17 with HTTP; Wed, 1 Oct 2008 03:55:21 -0700 (PDT) Message-ID: <279fbba40810010355q5da249b0k47edad2306422afc@mail.gmail.com> Date: Wed, 1 Oct 2008 11:55:21 +0100 From: "Kerin Millar" To: gentoo-server@lists.gentoo.org Subject: Re: [gentoo-server] Server Packages for Gentoo In-Reply-To: <20080930182846.1e5856fb@robbieab.com> Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-server@lists.gentoo.org Reply-to: gentoo-server@lists.gentoo.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_88326_21861570.1222858521591" References: <908514.71571.qm@web65403.mail.ac4.yahoo.com> <20080930182846.1e5856fb@robbieab.com> X-Archives-Salt: d849a82e-a295-4776-900a-3185df0c1ba9 X-Archives-Hash: 041af784a2ea28e63dad4ea9842976da ------=_Part_88326_21861570.1222858521591 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline 2008/9/30 Robert Bridge On Tue, 30 Sep 2008 09:17:42 -0700 (PDT) BRM wrote: > That's a matter of choosing what you install; but that's not specific > to Gentoo. > > MySQL on Gentoo is not going to be any different than MySQL on RHEL > or SLES. However, stability - due to differences in versions, > patches, etc. - might be different; but should be close to the same. Or better ... Except the Gentoo version will move a lot faster, potentially causing problems... This is potentially true but (a) the term "problems" can be interpreted in different ways (b) it actually cuts both ways (warning: long anecdote follows before leading up to my point) ... Recently I volunteered some time to help a friend deal with some serious issues he was having in running a popular community site. He'd recently migrated to a dedicated host running CentOS and had assumed that this would address all of the scalability issues he was encountering beforehand. In fact, the situation became worse. When I investigated I discovered that apache/mod_php was running the server into the ground, eventually causing the kernel's OOM killer to spring into action. This situation was not helped by the rather horrid bespoke configuration, with core software having been re-packaged adly by the ISP and effectively held together with rubber-bands and sticky tape. Simply put, it was a complete and utter mess and hopelessly unstable. Due to the comparitively limited amount of physical RAM and the behaviour exhibited by apache, I suggested that he run lighttpd and php-cgi. I wasn't particularly suprised to find that CentOS did not have official packages and I had to resort to using third-party repositories containing hoplessly outdated packages to find what I needed (or face building from source). I was effectively fighting to be able to make the distro do what I wanted it to do. After addressing that, he continued to encounter stability issues. I the suggested that he might consider moving to a more flexible distro with a broader range of packages on offer. After learning of the options made available to him by the ISP, the only one that seemed remotely palatable was Debian. He conducted a full re-installation accordingly, and I set up lighttpd, php5-cgi and a number of other components in the stack. Interestingly, not everything he wanted was available - namely the apc opcode cache. Cue messing around installing build-essential and a number of other dependencies manually before having to manually build apc from source. Anyway, after setting everything up, things seemed to go well initially. But it wasn't before long before disaster struck - after a certain load various php-cgi processes would "run away" and consume inordinate amounts of processor time, with lighttpd unable to service further requests as a result. The only way to address the problem would be to run pkill -9 php5-cgi && pkill lighttpd. Worse, after doing so, the MySQL database that powered his backend would be subtly corrupted - enough to break the bulletin board software at the heart of the site! This would simply happen again and again. I pursued every angle that I could possibly think of. This is where Debian started to seriously get in my way. I knew that it was a bug, but I hadn't yet identifed which. I wanted to update the key components in the stack to see if the problem had already been addressed. I pinned a newer version of lighttpd from lenny to no avail. I wanted to try a newer version of php but Debian simply does not offer an up-to-date package. Furthermore, it became apparent that "unmasking" (to use a Gentoo-centric term) new software in Debian is very much an all or nothing affair, which is decidedly not what I wanted. To cut a long story short, I became throughly fed up with the situation and realised that something needed to be done. I therefore conducted a precarious - but ultimately successful - remote migration to Gentoo in-situ and, guess what? After setting up a lean and mean base system and installing lighttpd-1.4.19 and php-5.2.6 fresh out of portage, the site proceeded to work beautifully and without a hitch. And MySQL, which had been a CPU hog on Debian, now runs noticeably more efficiently. Incidentally, after doing a bit more digging I figured out that the system had probably been affected by PHP bug 40286 [1]. At the time of writing, Debian have done nothing about this bug [2] and, I suspect, not a greal deal concerning the 180 or so other bugs that have been fixed upstream in PHP since the 5.2.0 release. Simply put, Gentoo enabled me to get to where we needed to go - on a fast track to stability no less. And it didn't get in my whilst doing it. In fact, it enabled me to simplify the complexity of the base system to a significant extent through the discriminating employment of USE flags. And, with fantastic components such as openrc/baselayout-2, eselect, webapp-config and, - not least - portage itself, it's a joy to manage. In actual fact, the components of the base system are _not_ really updated all that often in Gentoo, despite a lot of nonsense that one often hears to the contrary. Since this deployment, there have been 3 minor package updates (one of which was a system package, man-pages) and - what do you know - today a new version of lighttpd is released which fixed 4 security bugs and it's already in the tree. I glanced over the upstream ChangeLog and had no hesitation in applying it to the system in question immediately. Incidentally, I wonder how long it will take the "enterprise" distros to backport the necessary fixes, assuming they even bother at all? And this leads me up to the point I'm trying to make. There are other distros out there that like to position themselves as the natural choice for sysadmins who seek "stability" or require "enterprise" class packages. They would effectively have you believe that it's viable to run a bunch of frozen packages on a general-purpose system because they are doing the heavy lifting and claim to be backporting the fixes that matter. My view is that this is largely a sham - there are countless security bugs are never backported, and that's before you even get to the non-security bugs that have a high impact. Take the kernel for example - it's probably not an exaggeration to say that Gentoo has one of the most pro-actively maintained kernel patchsets around [3] in terms of maintaining branches that upstream like to drop like yesterday's bad news, largely thanks to the combined efforts of the kernel herd on genpatches [4], and the maintainer of hardened-extras. I'd invite anyone who doubts this to take a look at, say, the work that was done on the 2.6.23 branch of hardened-sources [5], above and beyond the related genpatches set, then to compare and contrast with your favourite "enterprise" distro and see exactly how good a job they are doing of looking after your interests. Sure, it's recently been dropped from the tree because we only have the manpower to maintain to maintain so many releases, but it's _still_ probably a far safer kernel than you're getting in the likes of RHEL or Debian! And I'm not even talking about the grsecurity/PaX related stuff here, but actual fixes that come from the stable-queue upstream or, in some cases, are not to be found in the stable queue at all (or are not submitted because upstream don't care anymore). >From my perspective, all these distros do is provide the illusion that you are safe in not pro-actively managing your system and completely avoiding the fact of the matter that, yes, there comes a time when software really should be upgraded. For pretty much all of the open-source software that I use on the backend, upgrades typically go very smoothly and fix a heck of a lot more than is ever broken. Well, this post turned out to be a lot longer than I had anticipated. But I've seen so many comments that allude to Gentoo somehow being unfit for purpose because it doesn't freeze off a so-called "stable" tree so many times that, frankly, I get fed up with it and figured that something had to be said. Gentoo, whilst certainly having its fair share of foibles, doesn't get enough credit for the things that it does well and the things that it does right. If one doesn't like the way that Gentoo does things then there are surely other distros out there that will meet one's expectations, such as they are. My take: Gentoo is so much more pleasant to manage and administer that I feel like a duck out of water whenever I'm charged with managing anything else. The technology is generally light-years ahead of its contemporaries and I honestly do sleep a lot easier at night knowing that my systems are powered by it. Finally, any extra time expended in managing it is for me (a) well within the margins of what I consider a reasonable amount of effort (b) time well spent (c) produces tangible benefits (more than I could possibly mention here). Cheers, --Kerin [1] http://bugs.php.net/bug.php?id=40286 [2] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=431799 [3] http://bugs.gentoo.org/show_bug.cgi?id=185022#c3 [4] http://dev.gentoo.org/~dsd/genpatches/ [5] http://confucius.dh.bytemark.co.uk/~kerin.millar/trunk/2.6.23/ ------=_Part_88326_21861570.1222858521591 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline
2008/9/30 Robert Bridge <robert@robbieab.com>

    On Tue, 30 S= ep 2008 09:17:42 -0700 (PDT)
    BRM <bm_witness@yahoo.com> wrote:

    > That's a matter of choosing what you instal= l; but that's not specific
    > to Gentoo.
&nb= sp;   >
    > MySQL on Gentoo is not goin= g to be any different than MySQL on RHEL
    > or SLES= . However, stability - due to differences in versions,
    > patches, etc. - might be different; but should be c= lose to the same.

Or better ...

    Except the= Gentoo version will move a lot faster, potentially causing
  =   problems...

This is potentially true but (a) the term "p= roblems" can be interpreted in different ways (b) it actually cuts bot= h ways (warning: long anecdote follows before leading up to my point) ...
Recently I volunteered some time to help a friend deal with some seriou= s issues he was having in running a popular community site. He'd recent= ly migrated to a dedicated host running CentOS and had assumed that this wo= uld address all of the scalability issues he was encountering beforehand. I= n fact, the situation became worse. When I investigated I discovered that a= pache/mod_php was running the server into the ground, eventually causing th= e kernel's OOM killer to spring into action. This situation was not hel= ped by the rather horrid bespoke configuration, with core software having b= een re-packaged adly by the ISP and effectively held together with rubber-b= ands and sticky tape. Simply put, it was a complete and utter mess and hope= lessly unstable.

Due to the comparitively limited amount of physical RAM and the behavio= ur exhibited by apache, I suggested that he run lighttpd and php-cgi. I was= n't particularly suprised to find that CentOS did not have official pac= kages and I had to resort to using third-party repositories containing hopl= essly outdated packages to find what I needed (or face building from source= ). I was effectively fighting to be able to make the distro do what I wante= d it to do.

After addressing that, he continued to encounter stability issues. I th= e suggested that he might consider moving to a more flexible distro with a = broader range of packages on offer. After learning of the options made avai= lable to him by the ISP, the only one that seemed remotely palatable was De= bian. He conducted a full re-installation accordingly, and I set up lighttp= d, php5-cgi and a number of other components in the stack. Interestingly, n= ot everything he wanted was available - namely the apc opcode cache. Cue me= ssing around installing build-essential and a number of other dependencies = manually before having to manually build apc from source.

Anyway, after setting everything up, things seemed to go well initially= . But it wasn't before long before disaster struck - after a certain lo= ad various php-cgi processes would "run away" and consume inordin= ate amounts of processor time, with lighttpd unable to service further requ= ests as a result. The only way to address the problem would be to run pkill= -9 php5-cgi && pkill lighttpd. Worse, after doing so, the MySQL da= tabase that powered his backend would be subtly corrupted - enough to break= the bulletin board software at the heart of the site! This would simply ha= ppen again and again.

I pursued every angle that I could possibly think of. This is where Deb= ian started to seriously get in my way. I knew that it was a bug, but I had= n't yet identifed which. I wanted to update the key components in the s= tack to see if the problem had already been addressed. I pinned a newer ver= sion of lighttpd from lenny to no avail. I wanted to try a newer version of= php but Debian simply does not offer an up-to-date package. Furthermore, i= t became apparent that "unmasking" (to use a Gentoo-centric term)= new software in Debian is very much an all or nothing affair, which is dec= idedly not what I wanted.

To cut a long story short, I became throughly fed up with the situation= and realised that something needed to be done. I therefore conducted a pre= carious - but ultimately successful - remote migration to Gentoo in-situ an= d, guess what? After setting up a lean and mean base system and installing = lighttpd-1.4.19 and php-5.2.6 fresh out of portage, the site proceeded to w= ork beautifully and without a hitch. And MySQL, which had been a CPU hog on= Debian, now runs noticeably more efficiently. Incidentally, after doing a = bit more digging I figured out that the system had probably been affected b= y PHP bug 40286 [1]. At the time of writing, Debian have done nothing about= this bug [2] and, I suspect, not a greal deal concerning the 180 or so oth= er bugs that have been fixed upstream in PHP since the 5.2.0 release.

Simply put, Gentoo enabled me to get to where we needed to go - on a fa= st track to stability no less. And it didn't get in my whilst doing it.= In fact, it enabled me to simplify the complexity of the base system to a = significant extent through the discriminating employment of USE flags. And,= with fantastic components such as openrc/baselayout-2, eselect, webapp-con= fig and, - not least - portage itself, it's a joy to manage.

In actual fact, the components of the base system are _not_ really upda= ted all that often in Gentoo, despite a lot of nonsense that one often hear= s to the contrary. Since this deployment, there have been 3 minor package u= pdates (one of which was a system package, man-pages) and - what do you kno= w - today a new version of lighttpd is released which fixed 4 security bugs= and it's already in the tree. I glanced over the upstream ChangeLog an= d had no hesitation in applying it to the system in question immediately. I= ncidentally, I wonder how long it will take the "enterprise" dist= ros to backport the necessary fixes, assuming they even bother at all?

And this leads me up to the point I'm trying to make. There are oth= er distros out there that like to position themselves as the natural choice= for sysadmins who seek "stability" or require "enterprise&q= uot; class packages. They would effectively have you believe that it's = viable to run a bunch of frozen packages on a general-purpose system becaus= e they are doing the heavy lifting and claim to be backporting the fixes th= at matter. My view is that this is largely a sham  - there are countle= ss security bugs are never backported, and that's before you even get t= o the non-security bugs that have a high impact.

Take the kernel for example - it's probably not an exaggeration to = say that Gentoo has one of the most pro-actively maintained kernel patchset= s around [3] in terms of maintaining branches that upstream like to drop li= ke yesterday's bad news, largely thanks to the combined efforts of the = kernel herd on genpatches [4], and the maintainer of hardened-extras. I'= ;d invite anyone who doubts this to take a look at, say, the work that was = done on the 2.6.23 branch of hardened-sources [5], above and beyond the rel= ated genpatches set, then to compare and contrast with your favourite "= ;enterprise" distro and see exactly how good a job they are doing of l= ooking after your interests. Sure, it's recently been dropped from the = tree because we only have the manpower to maintain to maintain so many rele= ases, but it's _still_ probably a far safer kernel than you're gett= ing in the likes of RHEL or Debian! And I'm not even talking about the = grsecurity/PaX related stuff here, but actual fixes that come from the stab= le-queue upstream or, in some cases, are not to be found in the stable queu= e at all (or are not submitted because upstream don't care anymore).
From my perspective, all these distros do is provide the illusion that = you are safe in not pro-actively managing your system and completely avoidi= ng the fact of the matter that, yes, there comes a time when software reall= y should be upgraded. For pretty much all of the open-source software that = I use on the backend, upgrades typically go very smoothly and fix a heck of= a lot more than is ever broken.

Well, this post turned out to be a lot longer than I had anticipated. B= ut I've seen so many comments that allude to Gentoo somehow being unfit= for purpose because it doesn't freeze off a so-called "stable&quo= t; tree so many times that, frankly, I get fed up with it and figured that = something had to be said. Gentoo, whilst certainly having its fair share of= foibles, doesn't get enough credit for the things that it does well an= d the things that it does right. If one doesn't like the way that Gento= o does things then there are surely other distros out there that will meet = one's expectations, such as they are.

My take: Gentoo is so much more pleasant to manage and administer that = I feel like a duck out of water whenever I'm charged with managing anyt= hing else. The technology is generally light-years ahead of its contemporar= ies and I honestly do sleep a lot easier at night knowing that my systems a= re powered by it. Finally, any extra time expended in managing it is for me= (a) well within the margins of what I consider a reasonable amount of effo= rt (b) time well spent (c) produces tangible benefits (more than I could po= ssibly mention here).

Cheers,

--Kerin

[1] http://bugs.php.net/bug.php?id=3D40286
[2] http://bugs.debi= an.org/cgi-bin/bugreport.cgi?bug=3D431799
[3] http://b= ugs.gentoo.org/show_bug.cgi?id=3D185022#c3
[4] http://dev.gentoo.org/~dsd/genpatches/[5] http://confucius.dh.bytemark.co.uk/~kerin.millar/trunk/2.6.23/ ------=_Part_88326_21861570.1222858521591--