[gentoo-dev] Init systems portage category

[gentoo-dev] Init systems portage category

[Victor Ostorga]

Lately I have stepeed into bug 216461 "init systems in sys-apps as well
as in sys-process and even app-admin" and was about to moving
sys-process/minit to sys/apps-minit , but stepped into bug 190982
"move sys-process/{minit,runit} and app-admin/jinit to sys-aps" which
is the same and closed WONTFIX in 2009-08-09 .

I don't know the history about init systems category, but obviously is
necessary to stablish a category into which init systems should live
happy forever (sys-init ? app-init? foobar?).

Regards,

Víctor.

Re: [gentoo-dev] Init systems portage category

[Robert Bradbury]

[-- Attachment #1: Type: text/plain, Size: 1473 bytes --]

On Mon, Oct 12, 2009 at 11:39 AM, Victor Ostorga <vostorga@gentoo.org>wrote:

>
> I don't know the history about init systems category, but obviously is
> necessary to stablish a category into which init systems should live
> happy forever (sys-init ? app-init? foobar?).
>
>
I don't know what you want to call it, "sys-init" perhaps.  But it, and a
number of other packages, e.g. sys-apps/util-linux (which includes mount and
fsck), openrc, bash, udev, etc. belong in a "special" category for "packages
which could prevent the system from booting or corrupt file systems" if the
emerges do not work perfectly.  I get hung up once or twice a year by
semi-auto-emerging a package not realizing that it is a potential
show-stopper that should be closely monitored (or which should require an
immediate system reboot to see if it broke anything).  In contrast, you
could break any of the various X libraries, browsers, etc. and still have a
system from which one could fall back/forward.

Right now one only knows if an emerge is "N"ew or an "U"pgrade with little
indication as to how badly it could go wrong.

As far as I know there is no "critical packages" list (or class) which
include those that are likely to create much bigger headaches than common
emerge failures (for example this would include all executables used by the
init/openrc processes) which under ideal circumstances would be part of a
single package that could be compiled with a "static" option.

Robert

[-- Attachment #2: Type: text/html, Size: 1853 bytes --]

Re: [gentoo-dev] Init systems portage category

[Jesús Guerrero]

On Mon, 12 Oct 2009 12:45:27 -0400, Robert Bradbury
<robert.bradbury@gmail.com> wrote:
> On Mon, Oct 12, 2009 at 11:39 AM, Victor Ostorga
> <vostorga@gentoo.org>wrote:
> 
>>
>> I don't know the history about init systems category, but obviously is
>> necessary to stablish a category into which init systems should live
>> happy forever (sys-init ? app-init? foobar?).
>>
>>
> I don't know what you want to call it, "sys-init" perhaps.  But it, and
a
> number of other packages, e.g. sys-apps/util-linux (which includes mount
> and
> fsck), openrc, bash, udev, etc. belong in a "special" category for
> "packages
> which could prevent the system from booting or corrupt file systems" if
the
> emerges do not work perfectly.  I get hung up once or twice a year by
> semi-auto-emerging a package not realizing that it is a potential
> show-stopper that should be closely monitored (or which should require
an
> immediate system reboot to see if it broke anything).  In contrast, you
> could break any of the various X libraries, browsers, etc. and still
have a
> system from which one could fall back/forward.
> 
> Right now one only knows if an emerge is "N"ew or an "U"pgrade with
little
> indication as to how badly it could go wrong.
> 
> As far as I know there is no "critical packages" list (or class) which
> include those that are likely to create much bigger headaches than
common
> emerge failures (for example this would include all executables used by
the
> init/openrc processes) which under ideal circumstances would be part of
a
> single package that could be compiled with a "static" option.

But there's one... That what the "system" set is about in first place. We
could argue if creating a new category would be any good or not, that's a
different issue. But there's already a list of packages that's considered
critical for a Gentoo system. That's what "system" is, and you will get a
big red waning when trying to uninstall one package belonging to this
category.

-- 
Jesús Guerrero

Re: [gentoo-dev] Init systems portage category

[Wyatt Epp]

[-- Attachment #1: Type: text/plain, Size: 1294 bytes --]

2009/10/12 Jesús Guerrero <i92guboj@terra.es>

>  But there's one... That what the "system" set is about in first place. We
> could argue if creating a new category would be any good or not, that's a
> different issue. But there's already a list of packages that's considered
> critical for a Gentoo system. That's what "system" is, and you will get a
> big red waning when trying to uninstall one package belonging to this
> category.
>
>
Seeing as we understand @system to be "critical for a functional Gentoo
system", the phrase "critical packages" may have been poorly chosen for
communicating the concept of "things that, should I be cavalier in playing
with them, may leave me with a system that is incapable of playing again
without intervention from one of those lovely LiveCD things".

Nevertheless, there is a class of "packages that I need to watch out for,
because they'll make my life miserable in ways X can only dream about and
THEN stab me in the kidneys with a rusty javelin if I'm not careful" under
discussion that could probably use some action.  It's unfortunate that
there's no good way of encoding arbitrary semantic metadata about a small
set of packages such that it can be leveraged by various sources to achieve
this end...

Regards,
Wyatt

[-- Attachment #2: Type: text/html, Size: 1716 bytes --]

Re: [gentoo-dev] Init systems portage category

[Robert Bradbury]

[-- Attachment #1: Type: text/plain, Size: 2182 bytes --]

2009/10/12 Jesús Guerrero <i92guboj@terra.es>

>
> But there's one... That what the "system" set is about in first place. We
> could argue if creating a new category would be any good or not, that's a
> different issue. But there's already a list of packages that's considered
> critical for a Gentoo system. That's what "system" is, and you will get a
> big red waning when trying to uninstall one package belonging to this
> category.
>

My point would be that the selection criteria isn't particularly
robust/strict.  Iptunnel, df or du for example are not required to the best
of my knowledge for system booting or emerges.  Dev-lang/python is on the
other hand required for emerging (and is not a "sys" package).  I'm also not
sure that the warnings are strict enough.  In order to upgrade a package
(util-linux I think) recently I had to unmerge a library package on which it
depended but which conflicted with an upgrade to said library.  The unmerge
of the library package broke either fsck or mount (I can't remember which).
Had I tried to reboot before the upgrade was completed there would have been
problems.

Big "red warnings" are of no use when one is doing semi-automatic-upgrades
(and colored encodings are normally disabled when one dumps the emerge
output to a file).  What is needed is a separate indicator in emerge outputs
indicating that an upgrade is potentially "Dangerous" or should require
"Manual" intervention.  Anyone who is not a full time developer but who
wants to maintain a relatively up-to-date Gentoo system (which IMO is its
primary advantage over "packaged" releases such as RedHat, Ubuntu, etc.) is
going to want to automate the nightly emerges as much as possible such that
no user intervention is required.  And that probably works 90% of the time.
But there are those 5% of emerges that fail "reasonably" and require some
intervention (e.g. bug reports) and those 0.1-1% of emerges that fail (or
even succeed) with potential problems that could cost the user days.  It is
that final category (and it isn't every binary produced by a sys* package)
that I am suggesting warrants more attention.

Robert

[-- Attachment #2: Type: text/html, Size: 2616 bytes --]

Re: [gentoo-dev] Init systems portage category

[Robert Bradbury]

[-- Attachment #1: Type: text/plain, Size: 772 bytes --]

I agree with Wyatt's point.

Wouldn't there be an easy way to reset the last access date on all of the
files to say 1/1/2009 on a system then execute a relatively robust
multi-user boot (and maybe a world emerge upgrade) and record which files
are actually used during that process, then determine which package they
belong to and label those with some "level of criticality"?

Its probably also true that this list of files could be part of a "critical
system backup" that one could just keep around in a bz2 file for fast
recoveries (or even as md5sum's to determine when they might have been hosed
by disk errors or viruses) -- or is this something that is already done by
some of the selinux options?  [I've never used selinux so am unsure of
everything it does.]

R.

[-- Attachment #2: Type: text/html, Size: 839 bytes --]

Re: [gentoo-dev] Init systems portage category

[Jesús Guerrero]

On Mon, 12 Oct 2009 13:52:49 -0400, Robert Bradbury
<robert.bradbury@gmail.com> wrote:
> I agree with Wyatt's point.
> 
> Wouldn't there be an easy way to reset the last access date on all of
the
> files to say 1/1/2009 on a system then execute a relatively robust
> multi-user boot (and maybe a world emerge upgrade) and record which
files
> are actually used during that process, then determine which package they
> belong to and label those with some "level of criticality"?

In my opinion, if we really want to speak about a way to implement that
kind of snapshoting, we should start thinking about providing a better
integration with lvm, from the root. lvm can take care of the snapshots on
a non-expensive way, and it would be relatively easy to implement. However
a lot of stuff would need to be re-documented, starting from the handbook,
and the init system.

Into my eyes, it's the only serious way to do this at least until btrfs is
ready for the masses, and there's a long way until we reach that point
still.

As for the package bits, it's true that the semantic and delimitation
about what's part of the system and what isn't, and the mechanism to handle
some things could be better, but I've grown accustomed to the way it is and
I really don't care if that changes or not.
-- 
Jesús Guerrero

Re: [gentoo-dev] Init systems portage category

[Richard Freeman]

Jesús Guerrero wrote:
> 
> In my opinion, if we really want to speak about a way to implement that
> kind of snapshoting, we should start thinking about providing a better
> integration with lvm, from the root. lvm can take care of the snapshots on
> a non-expensive way, and it would be relatively easy to implement. However
> a lot of stuff would need to be re-documented, starting from the handbook,
> and the init system.

LVM snapshotting is extremely wasteful - it has no knowledge of the 
underlying structure of a filesystem.  For example, if I moved all the 
files around on a fairly full ext3 filesystem, an LVM snapshot would 
consume the full size of the filesystem.  However, a filesystem-level 
solution wouldn't need to store a single byte of data since nothing 
actually changed.

Also - a snapshot restoration obliterates ALL data on the partition that 
has changed since the snapshot was taken - so unless the essential files 
are on a separate partition it won't work out well.

LVM snapshots really seem to be a solution to atomic backups - if you 
unmount, snapshot, and remount a filesystem then you can run a 
self-consistent backup off of the snapshot with minimal downtime.  The 
wasted space isn't a big deal since the snapshot would be deleted before 
it grew too far.

Finally - I'm not to eager to try out lvm2 again anytime soon - I lost a 
ton of data when something glitched and wiped out data across multiple 
lvm partitions.  I know that the error must have been in the lvm layer 
(not md or the filesystem), because when I fscked and repaired one 
filesystem, another filesystem instantly became hosed (on a separate lvm 
partition).  Somehow the partitions had gotten scrambled together and 
the fsck was crossing partition boundaries.  Plus, dmesg was dumping all 
kinds of compliants at the md layer about the lvm device trying to 
access out-of-range clusters.  Googling I found a few other reports of 
similar behavior - it seems extremely rare, but very nasty.

Fortunately the most important stuff on my PC was backed up (good 
planning), but it was still a pain - I lost tons of DVR recordings since 
I don't back that up (not worth the cost vs the value of the data).  Now 
I just run ext3 on top of md and I haven't had any problems.

You're right that btrfs will definitely help.  However, being able to 
create a personal stage1 tarball at will would certainly also be useful, 
and it wouldn't actually consume much disk space.

Re: [gentoo-dev] Init systems portage category

[Tobias Klausmann]

Hi! 

Here's another chance to be reminded that Gentoo is about choice:
how about a config file that comes with a pre-set list of packages
that are important (if they're installed), for example e2fsprogs,
the init system, stuff like that. But the user can add to this
list (cryptsetup if it's needed for boot, the critical service
this machine provides (say, apache), openssh because the machine
is in a small metal box just off William's Field, AQ). Then, come
update day, those packages, if to-be-emerged, are highlighted
and/or there's a message at the bottom of emerge output (you /do/
a pretend run before you upgrade world, right?) warning that
special care must be taken.

Just my €0.012 (adjusted for inflation)

Regards,
Tobias

Re: [gentoo-dev] Init systems portage category

[Thilo Bangert]

[-- Attachment #1: Type: Text/Plain, Size: 1027 bytes --]

Victor Ostorga <vostorga@gentoo.org> said:
> Lately I have stepeed into bug 216461 "init systems in sys-apps as well
> as in sys-process and even app-admin" and was about to moving
> sys-process/minit to sys/apps-minit , but stepped into bug 190982
> "move sys-process/{minit,runit} and app-admin/jinit to sys-aps" which
> is the same and closed WONTFIX in 2009-08-09 .
> 
> I don't know the history about init systems category, but obviously is
> necessary to stablish a category into which init systems should live
> happy forever (sys-init ? app-init? foobar?).
> 

i would stick all inits into sys-process - its not crowded in there and 
"process" fits the init concept well IMHO.

generally i thought, we wanted to move stuff out of sys-apps, as its really 
not a good description of what a package is about.

sys-apps/ucspi-tcp and sys-apps/ucspi-ssl could move to net-misc or 
preferably net-libs. sys-apps/ucspi-proxy to net-proxy.

thanks
kind regards
Thilo

> Regards,
> 
> Víctor.
> 


[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]