[gentoo-dev] About suggesting to create a separate partition for portage tree in handbook

[gentoo-dev] About suggesting to create a separate partition for portage tree in handbook

[Pacho Ramos]

[-- Attachment #1: Type: text/plain, Size: 742 bytes --]

Hello

I am a bit surprised handbook still doesn't suggest people to create a
separate partition for /usr/portage tree. I remember my first Gentoo
systems had it inside / and that lead to a lot of fragmentation, much
slower "emerge -pvuDN world" (I benchmarked it when I changed my
partitioning scheme to put /usr/portage) separate and a lot of disk
space lost (I remember portage tree reached around 3 GB of disk space
while I am now running with 300MB)

Could handbook suggest people to put /usr/portage on a different
partition then? The only doubt I have is what filesystem would be better
for it, in my case I am using reiserfs with tail enabled, but maybe you
have other different setups.

Thanks for discussing this :)

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] About suggesting to create a separate partition for portage tree in handbook

[Sven Vermeulen]

On Tue, Mar 27, 2012 at 07:49:00PM +0200, Pacho Ramos wrote:
> I am a bit surprised handbook still doesn't suggest people to create a
> separate partition for /usr/portage tree. I remember my first Gentoo
> systems had it inside / and that lead to a lot of fragmentation, much
> slower "emerge -pvuDN world" (I benchmarked it when I changed my
> partitioning scheme to put /usr/portage) separate and a lot of disk
> space lost (I remember portage tree reached around 3 GB of disk space
> while I am now running with 300MB)
> 
> Could handbook suggest people to put /usr/portage on a different
> partition then? The only doubt I have is what filesystem would be better
> for it, in my case I am using reiserfs with tail enabled, but maybe you
> have other different setups.

To be honest, I don't think it is wise to describe it in the Gentoo Handbook
just yet. I don't mind having it documented elsewhere, but the separate
partition is not mandatory for getting Gentoo up and running. The
instructions currently also just give an example partition layout and tell
users that different layouts are perfectly possible.

We need to take into consideration what is needed (must) for a Gentoo
installation, what is seriously recommended (should), what is nice to have
(could), etc. And for me, having a separate /usr/portage is a nice-to-have
imo.

Wkr,
	Sven Vermeulen

Re: [gentoo-dev] About suggesting to create a separate partition for portage tree in handbook

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 605 bytes --]

On Tue, 27 Mar 2012 19:49:00 +0200
Pacho Ramos <pacho@gentoo.org> wrote:
> I am a bit surprised handbook still doesn't suggest people to create a
> separate partition for /usr/portage tree.

I don't know whether you've heard, but PackageKit (a hard dependency of
udev as of 185, to allow automatic installation of the appropriate
firmware) no longer supports /usr/portage on its own partition. But
that's ok, because extensive studies have shown that the only possible
reasons for putting /usr/portage on its own partition are historical,
since everyone has an SSD now.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] About suggesting to create a separate partition for portage tree in handbook

[Aaron W. Swenson]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 03/27/2012 02:01 PM, Sven Vermeulen wrote:
> On Tue, Mar 27, 2012 at 07:49:00PM +0200, Pacho Ramos wrote:
>> I am a bit surprised handbook still doesn't suggest people to
>> create a separate partition for /usr/portage tree. I remember my
>> first Gentoo systems had it inside / and that lead to a lot of
>> fragmentation, much slower "emerge -pvuDN world" (I benchmarked
>> it when I changed my partitioning scheme to put /usr/portage)
>> separate and a lot of disk space lost (I remember portage tree
>> reached around 3 GB of disk space while I am now running with
>> 300MB)
>> 
>> Could handbook suggest people to put /usr/portage on a different 
>> partition then? The only doubt I have is what filesystem would be
>> better for it, in my case I am using reiserfs with tail enabled,
>> but maybe you have other different setups.
> 
> To be honest, I don't think it is wise to describe it in the Gentoo
> Handbook just yet. I don't mind having it documented elsewhere, but
> the separate partition is not mandatory for getting Gentoo up and
> running. The instructions currently also just give an example
> partition layout and tell users that different layouts are
> perfectly possible.
> 
> We need to take into consideration what is needed (must) for a
> Gentoo installation, what is seriously recommended (should), what
> is nice to have (could), etc. And for me, having a separate
> /usr/portage is a nice-to-have imo.
> 
> Wkr, Sven Vermeulen
> 

Definitely. The handbook should only cover simple, straightforward
setups. New users are already overwhelmed by the handbook as it is.
Going into details about alternate setups would only increase the
number of "Is there a quick start guide somewhere that I can follow"
or "which setup is best" questions that we currently get in #gentoo
and friends.

If anything, I'd recommend we remove some details, like getting rid of
the 'mirrorselect' command. (Too many people run into a non-starter
because of it.)

- - Aaron
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iF4EAREIAAYFAk9yBREACgkQVxOqA9G7/aBPzAD+J7Lt3nmXDvKx9S2yMyLgM36B
/ANwxzr/S/HwY+Zq8JwA/jY7m+Dp47150IUiSfZkyJBB0Wjc1uCCRy/x5SgR7+J9
=gy2m
-----END PGP SIGNATURE-----

Re: [gentoo-dev] About suggesting to create a separate partition for portage tree in handbook

[Alexandre Rostovtsev]

On Tue, 2012-03-27 at 20:01 +0200, Sven Vermeulen wrote:
> On Tue, Mar 27, 2012 at 07:49:00PM +0200, Pacho Ramos wrote:
> > I am a bit surprised handbook still doesn't suggest people to create a
> > separate partition for /usr/portage tree. I remember my first Gentoo
> > systems had it inside / and that lead to a lot of fragmentation, much
> > slower "emerge -pvuDN world" (I benchmarked it when I changed my
> > partitioning scheme to put /usr/portage) separate and a lot of disk
> > space lost (I remember portage tree reached around 3 GB of disk space
> > while I am now running with 300MB)
> > 
> > Could handbook suggest people to put /usr/portage on a different
> > partition then? The only doubt I have is what filesystem would be better
> > for it, in my case I am using reiserfs with tail enabled, but maybe you
> > have other different setups.
> 
> To be honest, I don't think it is wise to describe it in the Gentoo Handbook
> just yet. I don't mind having it documented elsewhere, but the separate
> partition is not mandatory for getting Gentoo up and running. The
> instructions currently also just give an example partition layout and tell
> users that different layouts are perfectly possible.
> 
> We need to take into consideration what is needed (must) for a Gentoo
> installation, what is seriously recommended (should), what is nice to have
> (could), etc. And for me, having a separate /usr/portage is a nice-to-have
> imo.

The partitioning scheme is something that the user needs to decide on
*before* getting Gentoo up and running. After the user had finished
installing the operating system, it's too late to inform him about the
advantages of a separate /usr/portage.

IMHO, chapter 4 of the handbook needs the following changes:

1. ext4, not ext3, needs to be recommended as the default filesystem. We
have kernel 3.2 marked stable, there is no need to keep talking about
ext4 as if it's something experimental.

2. The handbook should mention that a separate small /usr/portage
partition can noticeably improve performance for users with a rotational
hard drive, and that it's not needed for solid-state drives. It should
also mention that using Gentoo with a separate /usr/portage partition
will require some additional configuration (such as changing DISTDIR and
PKGDIR to avoid running out of space).

-Alexandre.

Re: [gentoo-dev] About suggesting to create a separate partition for portage tree in handbook

[Rich Freeman]

On Tue, Mar 27, 2012 at 2:34 PM, Alexandre Rostovtsev <tetromino@gentoo.org>
> The partitioning scheme is something that the user needs to decide on
> *before* getting Gentoo up and running. After the user had finished
> installing the operating system, it's too late to inform him about the
> advantages of a separate /usr/portage.

Yes and no (if you have free space, you could easily move /usr/portage
- some other changes are harder).

However, you could extend this line of argument to raid, lvm, and even
stuff like the use of systemd or an alternative package manager.  All
of those things are much easier to implement if you just start out
with them.

I'm all for creating a wiki to talk about some alternative options.
Perhaps even link to it at the start of the handbook in the intro (if
you're not in a rush and want to read about more advanced
configurations, check out ...).

However, I tend to agree that the handbook should be a
nearly-foolproof no-frills Gentoo installation.

> 1. ext4, not ext3, needs to be recommended as the default filesystem. We
> have kernel 3.2 marked stable, there is no need to keep talking about
> ext4 as if it's something experimental.

I tend to agree here.  Not sure we need the full discussion of
filesystems either.  Ext4 is probably good enough for everybody, and
mention ext3/2 as more established alternatives.

I tend to feel the same way about stuff like LILO.

Then again, Gentoo is about choice.  It just seems like we're
presenting users with more choices than makes sense for a newbie.  If
there is a choice between something that 99.99% of users will want,
and some ancient piece of cruft that still works and is better for
0.01% of the userbase, does that really have to be in the handbook?

Rich

Re: [gentoo-dev] About suggesting to create a separate partition for portage tree in handbook

[Ian Stakenvicius]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 27/03/12 02:47 PM, Rich Freeman wrote:
> On Tue, Mar 27, 2012 at 2:34 PM, Alexandre Rostovtsev
> <tetromino@gentoo.org>
>> The partitioning scheme is something that the user needs to
>> decide on *before* getting Gentoo up and running. After the user
>> had finished installing the operating system, it's too late to
>> inform him about the advantages of a separate /usr/portage.
> 
> Yes and no (if you have free space, you could easily move
> /usr/portage - some other changes are harder).
> 
> However, you could extend this line of argument to raid, lvm, and
> even stuff like the use of systemd or an alternative package
> manager.  All of those things are much easier to implement if you
> just start out with them.
> 
> I'm all for creating a wiki to talk about some alternative
> options. Perhaps even link to it at the start of the handbook in
> the intro (if you're not in a rush and want to read about more
> advanced configurations, check out ...).
> 
> However, I tend to agree that the handbook should be a 
> nearly-foolproof no-frills Gentoo installation.
> 


You know, we have "Code Listing 2.1: Filesystem Example" in Section 4,
we could always adjust that to have a /usr/portage partition in it
(take a bit of space away from /home, or something)

It doesn't recommend/require anything, but when users see it they'll
think about it.


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (GNU/Linux)

iF4EAREIAAYFAk9yDK4ACgkQAJxUfCtlWe19QgEA22gRFMmyaxVpJp+LeaPsTWOq
RqF2z9fZvebtBiSdLSUA/R4c10HtDeBpjEJyHCKbQkKJWc+ilRw8bilOgHgAvKT5
=egsm
-----END PGP SIGNATURE-----

Re: [gentoo-dev] About suggesting to create a separate partition for portage tree in handbook

[vivo75]

Il 27/03/2012 20:53, Ian Stakenvicius ha scritto:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> On 27/03/12 02:47 PM, Rich Freeman wrote:
>> On Tue, Mar 27, 2012 at 2:34 PM, Alexandre Rostovtsev
>> <tetromino@gentoo.org>
>>> The partitioning scheme is something that the user needs to
>>> decide on *before* getting Gentoo up and running. After the user
>>> had finished installing the operating system, it's too late to
>>> inform him about the advantages of a separate /usr/portage.
>> Yes and no (if you have free space, you could easily move
>> /usr/portage - some other changes are harder).
>>
>> However, you could extend this line of argument to raid, lvm, and
>> even stuff like the use of systemd or an alternative package
>> manager.  All of those things are much easier to implement if you
>> just start out with them.
>>
>> I'm all for creating a wiki to talk about some alternative
>> options. Perhaps even link to it at the start of the handbook in
>> the intro (if you're not in a rush and want to read about more
>> advanced configurations, check out ...).
>>
>> However, I tend to agree that the handbook should be a
>> nearly-foolproof no-frills Gentoo installation.
>>
>
> You know, we have "Code Listing 2.1: Filesystem Example" in Section 4,
> we could always adjust that to have a /usr/portage partition in it
> (take a bit of space away from /home, or something)
>
> It doesn't recommend/require anything, but when users see it they'll
> think about it.

Boh ... IMHO /usr/portage should be a squashfs filesystem, rsynced from 
some kind server out there, auto(un)mounted, so it releases resources 
after use.
No needs for any additional partition (which sound kinda lame for a 
package manager)

However the devs are right here, handbook should be stripped down, not 
bloated with details that could be fulfilled later

Rgds,
Francesco

Re: [gentoo-dev] About suggesting to create a separate partition for portage tree in handbook

[Aaron W. Swenson]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 03/27/2012 02:53 PM, Ian Stakenvicius wrote:
> On 27/03/12 02:47 PM, Rich Freeman wrote:
>> On Tue, Mar 27, 2012 at 2:34 PM, Alexandre Rostovtsev 
>> <tetromino@gentoo.org>
>>> The partitioning scheme is something that the user needs to 
>>> decide on *before* getting Gentoo up and running. After the
>>> user had finished installing the operating system, it's too
>>> late to inform him about the advantages of a separate
>>> /usr/portage.
> 
>> Yes and no (if you have free space, you could easily move 
>> /usr/portage - some other changes are harder).
...
>> However, I tend to agree that the handbook should be a 
>> nearly-foolproof no-frills Gentoo installation.
> 
> 
> 
> You know, we have "Code Listing 2.1: Filesystem Example" in Section
> 4, we could always adjust that to have a /usr/portage partition in
> it (take a bit of space away from /home, or something)
> 
> It doesn't recommend/require anything, but when users see it
> they'll think about it.

That isn't the way users read it, though. They read it and assume that
is precisely how they *need* to configure their disk layout.

- - Aaron
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iF4EAREIAAYFAk9yD0kACgkQVxOqA9G7/aAdKQD/WnFh36QVL1tV/kfHdPcUyebQ
W0nvjhngMEU09fW8bWwA/A+A6wnhbm3DUA0Pl2dTmOY20rW9ceLE7qM3PSqM5tw1
=coW5
-----END PGP SIGNATURE-----

Re: [gentoo-dev] About suggesting to create a separate partition for portage tree in handbook

[Ian Stakenvicius]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 27/03/12 03:04 PM, Aaron W. Swenson wrote:
> 
>> You know, we have "Code Listing 2.1: Filesystem Example" in
>> Section 4, we could always adjust that to have a /usr/portage
>> partition in it (take a bit of space away from /home, or
>> something)
> 
>> It doesn't recommend/require anything, but when users see it 
>> they'll think about it.
> 
> That isn't the way users read it, though. They read it and assume
> that is precisely how they *need* to configure their disk layout.
> 
> - Aaron
> 

Really?  It's been a while since i hung out in #gentoo, but i was
there pretty solidly for a couple of years and i don't recall any new
user (to gentoo or linux) reporting in, saying they set up their
disk(s) with all of those partitions.  They pretty well always
followed the "default partitioning scheme" listed in the table in 4.b
(which is used for every other example on that chapter).

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (GNU/Linux)

iF4EAREIAAYFAk9yEVoACgkQAJxUfCtlWe3EtgEAwr62YTL812ehPurzTJWT1sqr
SUQhJzybaLlY0Rf2T6ABANqOtXDK+IbRTjLw1fcfjGHqWuYUAfqYnYtniN5ztwHK
=Vi2z
-----END PGP SIGNATURE-----

Re: [gentoo-dev] About suggesting to create a separate partition for portage tree in handbook

[Sven Vermeulen]

On Tue, Mar 27, 2012 at 02:47:15PM -0400, Rich Freeman wrote:
> On Tue, Mar 27, 2012 at 2:34 PM, Alexandre Rostovtsev <tetromino@gentoo.org>
> > 1. ext4, not ext3, needs to be recommended as the default filesystem. We
> > have kernel 3.2 marked stable, there is no need to keep talking about
> > ext4 as if it's something experimental.
> 
> I tend to agree here.  Not sure we need the full discussion of
> filesystems either.  Ext4 is probably good enough for everybody, and
> mention ext3/2 as more established alternatives.

I see no issue putting ext4 as the suggested file system. However, it must
be checked on a per-architecture basis (I can only test x86 and amd64 myself
- I know, I'm missing all the fun) and preferably brought on by the
responsible teams of those architectures.

Dropping the (elaborate) explanation on file systems won't win us much. It's
not like it is that long - a paragraph per file system type. Even the online
help in recent distribution installations provide more information.

> I tend to feel the same way about stuff like LILO.

I would *really* like to drop LILO and while we are at it, get grub2 working
on all systems/architectures and stable ;-) But I'm not going to drop LILO
without group consent.

> Then again, Gentoo is about choice.  It just seems like we're
> presenting users with more choices than makes sense for a newbie.  If
> there is a choice between something that 99.99% of users will want,
> and some ancient piece of cruft that still works and is better for
> 0.01% of the userbase, does that really have to be in the handbook?

Welcome to documentation development. The Gentoo Handbook has always been a
difficult source for such discussions. If we truely want to provide
information towards our users on all possible choices, you'll need a totally
different approach.

I once started (before I left Gentoo, rejoined, left again) on a "complete
gentoo handbook" that covered much more in greater detail (you'll find the
last version at
http://www.gentoo.org/doc/en/handbook/draft/complete/handbook.xml) but I've
since moved away from that. Perhaps I should work again on it...

Wkr,
	Sven Vermeulen

Re: [gentoo-dev] About suggesting to create a separate partition for portage tree in handbook

[Kent Fredric]

On 28 March 2012 07:53, Ian Stakenvicius <axs@gentoo.org> wrote:
> You know, we have "Code Listing 2.1: Filesystem Example" in Section 4,
> we could always adjust that to have a /usr/portage partition in it
> (take a bit of space away from /home, or something)
>
> It doesn't recommend/require anything, but when users see it they'll
> think about it.


I'd be careful with that logic, users may just copy it without
thinking and then wonder why it doesn't work ( because they didn't do
all the other steps required to make it work such as making sure you
don't run out of space, set DIST_DIR etc etc )

-- 
Kent

perl -e  "print substr( \"edrgmaM  SPA NOcomil.ic\\@tfrken\", \$_ * 3,
3 ) for ( 9,8,0,7,1,6,5,4,3,2 );"

Re: [gentoo-dev] About suggesting to create a separate partition for portage tree in handbook

[Kent Fredric]

On 28 March 2012 08:15, Sven Vermeulen <swift@gentoo.org> wrote:
>> Then again, Gentoo is about choice.  It just seems like we're
>> presenting users with more choices than makes sense for a newbie.  If
>> there is a choice between something that 99.99% of users will want,
>> and some ancient piece of cruft that still works and is better for
>> 0.01% of the userbase, does that really have to be in the handbook?
>
> Welcome to documentation development. The Gentoo Handbook has always been a
> difficult source for such discussions. If we truely want to provide
> information towards our users on all possible choices, you'll need a totally
> different approach.
>
> I once started (before I left Gentoo, rejoined, left again) on a "complete
> gentoo handbook" that covered much more in greater detail (you'll find the
> last version at
> http://www.gentoo.org/doc/en/handbook/draft/complete/handbook.xml) but I've
> since moved away from that. Perhaps I should work again on it...
>
> Wkr,
>        Sven Vermeulen
>


An idea is a javascripty-dynamic-slidey thing that makes more details
and advanced stuff visible to people who want it, so you can adjust
the documentation to suit your skill level.

-- 
Kent

perl -e  "print substr( \"edrgmaM  SPA NOcomil.ic\\@tfrken\", \$_ * 3,
3 ) for ( 9,8,0,7,1,6,5,4,3,2 );"

Re: [gentoo-dev] About suggesting to create a separate partition for portage tree in handbook

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 1480 bytes --]

On Wed, Mar 28, 2012 at 08:20:45AM +1300, Kent Fredric wrote:
> On 28 March 2012 08:15, Sven Vermeulen <swift@gentoo.org> wrote:
> >> Then again, Gentoo is about choice.  It just seems like we're
> >> presenting users with more choices than makes sense for a newbie.  If
> >> there is a choice between something that 99.99% of users will want,
> >> and some ancient piece of cruft that still works and is better for
> >> 0.01% of the userbase, does that really have to be in the handbook?
> >
> > Welcome to documentation development. The Gentoo Handbook has always been a
> > difficult source for such discussions. If we truely want to provide
> > information towards our users on all possible choices, you'll need a totally
> > different approach.
> >
> > I once started (before I left Gentoo, rejoined, left again) on a "complete
> > gentoo handbook" that covered much more in greater detail (you'll find the
> > last version at
> > http://www.gentoo.org/doc/en/handbook/draft/complete/handbook.xml) but I've
> > since moved away from that. Perhaps I should work again on it...
> >
> > Wkr,
> >        Sven Vermeulen
> >
> 
> 
> An idea is a javascripty-dynamic-slidey thing that makes more details
> and advanced stuff visible to people who want it, so you can adjust
> the documentation to suit your skill level.

Why not just the separate "quick install" guide like we have that lists
steps and the handbook if yu want more details?

William

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] About suggesting to create a separate partition for portage tree in handbook

[Aaron W. Swenson]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 03/27/2012 03:13 PM, Ian Stakenvicius wrote:
> On 27/03/12 03:04 PM, Aaron W. Swenson wrote:
> 
>>> You know, we have "Code Listing 2.1: Filesystem Example" in 
>>> Section 4, we could always adjust that to have a /usr/portage 
>>> partition in it (take a bit of space away from /home, or 
>>> something)
> 
>>> It doesn't recommend/require anything, but when users see it 
>>> they'll think about it.
> 
>> That isn't the way users read it, though. They read it and
>> assume that is precisely how they *need* to configure their disk
>> layout.
> 
>> - Aaron
> 
> 
> Really?  It's been a while since i hung out in #gentoo, but i was 
> there pretty solidly for a couple of years and i don't recall any
> new user (to gentoo or linux) reporting in, saying they set up
> their disk(s) with all of those partitions.  They pretty well
> always followed the "default partitioning scheme" listed in the
> table in 4.b (which is used for every other example on that
> chapter).

Yes, really.

I've seen it often in #gentoo where a new user said that they did it
just like the example told them to, despite it being marked 'Optional'
they still thought it was required. It would take several of the
experienced users to say 'skip it' in four-part harmony to convince
the novice to move on.

The less we show regarding advanced setups, the less likely we'll have
support a new installation. We can hint at them somewhere in the
paragraphs, but we should avoid them in the code samples because
people don't read. They skim.

- - Aaron
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iF4EAREIAAYFAk9yFZsACgkQVxOqA9G7/aCUrQD/UU0WysiyQg2CDRRtOCAAY0rR
s53sHazRFILXEtey2/wA/jhMCobeu4n1YJmL2+Wz/txuClMmoY+gDHaW+O4CfZuf
=fw9Z
-----END PGP SIGNATURE-----

Re: [gentoo-dev] About suggesting to create a separate partition for portage tree in handbook

[Sven Vermeulen]

On Tue, Mar 27, 2012 at 02:29:34PM -0500, William Hubbs wrote:
> Why not just the separate "quick install" guide like we have that lists
> steps and the handbook if yu want more details?

We came from that. It means we need to start managing "just the commands"
for each architecture. After a while, people start asking more information
for "just the necessary bits", making the guides longer and longer, after
which they'll eventually need to be made multi-page.

Wkr,
	Sven Vermeulen

Re: [gentoo-dev] About suggesting to create a separate partition for portage tree in handbook

[Richard Yao]

[-- Attachment #1: Type: text/plain, Size: 1787 bytes --]

On 03/27/12 14:34, Alexandre Rostovtsev wrote:
> The partitioning scheme is something that the user needs to decide on
> *before* getting Gentoo up and running. After the user had finished
> installing the operating system, it's too late to inform him about the
> advantages of a separate /usr/portage.
> 
> IMHO, chapter 4 of the handbook needs the following changes:
> 
> 1. ext4, not ext3, needs to be recommended as the default filesystem. We
> have kernel 3.2 marked stable, there is no need to keep talking about
> ext4 as if it's something experimental.
> 
> 2. The handbook should mention that a separate small /usr/portage
> partition can noticeably improve performance for users with a rotational
> hard drive, and that it's not needed for solid-state drives. It should
> also mention that using Gentoo with a separate /usr/portage partition
> will require some additional configuration (such as changing DISTDIR and
> PKGDIR to avoid running out of space).
> 
> -Alexandre.
> 
> 

Could we amend this to also include the benefits of ZFS and why you
would want to use XFS or reiserfs instead of ext{2,3,4} as your
filesystem in situations where ZFS is not yet appropriate (e.g. using it
on Gentoo stable)? We could also include documentation on Reiser4 while
we are at it.

With that said, I don't think that this is appropriate for the handbook.
It is meant to get users started, not to set things in stone. The
partitioning can always be redone later via a stage4 backup.

On the note, I would like to suggest that we make a separate disk
partitioning and filesystem handbook, which would seem to be a more
appropriate location for this information. I should also say that I do
agree about recommending ext4 instead of ext3 by default.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 900 bytes --]

Re: [gentoo-dev] About suggesting to create a separate partition for portage tree in handbook

[Alec Moskvin]

On Tuesday 27 March 2012 14:34:03, Alexandre Rostovtsev wrote:
> On Tue, 2012-03-27 at 20:01 +0200, Sven Vermeulen wrote:
> > On Tue, Mar 27, 2012 at 07:49:00PM +0200, Pacho Ramos wrote:
> > > I am a bit surprised handbook still doesn't suggest people to create a
> > > separate partition for /usr/portage tree. I remember my first Gentoo
> > > systems had it inside / and that lead to a lot of fragmentation, much
> > > slower "emerge -pvuDN world" (I benchmarked it when I changed my
> > > partitioning scheme to put /usr/portage) separate and a lot of disk
> > > space lost (I remember portage tree reached around 3 GB of disk space
> > > while I am now running with 300MB)
> > > 
> > > Could handbook suggest people to put /usr/portage on a different
> > > partition then? The only doubt I have is what filesystem would be better
> > > for it, in my case I am using reiserfs with tail enabled, but maybe you
> > > have other different setups.
> > 
> > To be honest, I don't think it is wise to describe it in the Gentoo Handbook
> > just yet. I don't mind having it documented elsewhere, but the separate
> > partition is not mandatory for getting Gentoo up and running. The
> > instructions currently also just give an example partition layout and tell
> > users that different layouts are perfectly possible.
> > 
> > We need to take into consideration what is needed (must) for a Gentoo
> > installation, what is seriously recommended (should), what is nice to have
> > (could), etc. And for me, having a separate /usr/portage is a nice-to-have
> > imo.
> 
> The partitioning scheme is something that the user needs to decide on
> *before* getting Gentoo up and running. After the user had finished
> installing the operating system, it's too late to inform him about the
> advantages of a separate /usr/portage.

It does not have to be a separate *physical* partition. It could be set
up as a loop device without any real downsides:

/usr/portage/tree.ext4	/usr/portage/tree	ext4	loop,noatime	0 0

An advantage is that it can be easily resized if necessary.

> IMHO, chapter 4 of the handbook needs the following changes:
> 
> 1. ext4, not ext3, needs to be recommended as the default filesystem. We
> have kernel 3.2 marked stable, there is no need to keep talking about
> ext4 as if it's something experimental.
> 
> 2. The handbook should mention that a separate small /usr/portage
> partition can noticeably improve performance for users with a rotational
> hard drive, and that it's not needed for solid-state drives. It should
> also mention that using Gentoo with a separate /usr/portage partition
> will require some additional configuration (such as changing DISTDIR and
> PKGDIR to avoid running out of space).
> 
> -Alexandre.
> 
> 

Re: [gentoo-dev] About suggesting to create a separate partition for portage tree in handbook

[Kent Fredric]

On 28 March 2012 08:57, Richard Yao <ryao@cs.stonybrook.edu> wrote:
>
> Could we amend this to also include the benefits of ZFS and why you
> would want to use XFS or reiserfs instead of ext{2,3,4} as your
> filesystem in situations where ZFS is not yet appropriate (e.g. using it
> on Gentoo stable)? We could also include documentation on Reiser4 while
> we are at it.

Thats probably asking a bit much, I've done my experimenting with
XFS/reiserfs , the benefits aren't that substantial to be worth the
hassle of the negatives. And as for Reiser4, if there's any
documentation mentioning that I think it being simply "Don't use
Reiser4" adequate enough.

Noob Level: Just Use Ext4
Intermediate: Just Use Ext4, use Ext3 or 2 if you want more something
else, but ext4 should do the trick
Advanced: Entertain the ideas of XFS/reiser if you want, but you're
not likely going to see a *lot* of difference over ext4 on its own
partition. Not in the long term.

I used to advocate JFS, but long term experience with it taught me JFS
is fast for new file systems, and gets progressively slower over time.
The original IBM JFS had a defrag tool nobody managed to port to Linux
so JFS just gets crufty and stays that way.


-- 
Kent

perl -e  "print substr( \"edrgmaM  SPA NOcomil.ic\\@tfrken\", \$_ * 3,
3 ) for ( 9,8,0,7,1,6,5,4,3,2 );"

Re: [gentoo-dev] About suggesting to create a separate partition for portage tree in handbook

[Brian Dolbec]

[-- Attachment #1: Type: text/plain, Size: 431 bytes --]

On Tue, 2012-03-27 at 19:16 +0100, Ciaran McCreesh wrote:
> But that's ok, because extensive studies have shown that the only possible
> reasons for putting /usr/portage on its own partition are historical,
> since everyone has an SSD now.
> 

Yeah, right.  Since I must be the only one out there that doesn't yet
have an SSD, you'll give me (and anyone else that still doesn't) one?
-- 
Brian Dolbec <dolsen@gentoo.org>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] About suggesting to create a separate partition for portage tree in handbook

[Rich Freeman]

On Wed, Mar 28, 2012 at 3:16 AM, Brian Dolbec <dolsen@gentoo.org> wrote:
> On Tue, 2012-03-27 at 19:16 +0100, Ciaran McCreesh wrote:
>> But that's ok, because extensive studies have shown that the only possible
>> reasons for putting /usr/portage on its own partition are historical,
>> since everyone has an SSD now.
>>
>
> Yeah, right.  Since I must be the only one out there that doesn't yet
> have an SSD, you'll give me (and anyone else that still doesn't) one?

Woosh...

Re: [gentoo-dev] About suggesting to create a separate partition for portage tree in handbook

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 825 bytes --]

On 28 March 2012 20:16, Brian Dolbec <dolsen@gentoo.org> wrote:
> On Tue, 2012-03-27 at 19:16 +0100, Ciaran McCreesh wrote:
>> But that's ok, because extensive studies have shown that the only
possible
>> reasons for putting /usr/portage on its own partition are historical,
>> since everyone has an SSD now.
>>
>
> Yeah, right.  Since I must be the only one out there that doesn't yet
> have an SSD, you'll give me (and anyone else that still doesn't) one?
> --
> Brian Dolbec <dolsen@gentoo.org>

https://en.wikipedia.org/wiki/Sarcasm

Unfortunately, when on the internet, this often transmutes into :

https://en.wiktionary.org/wiki/dick

[image: Inline images 1]

Sorry.


-- 
Kent

perl -e  "print substr( \"edrgmaM  SPA NOcomil.ic\\@tfrken\", \$_ * 3, 3 )
for ( 9,8,0,7,1,6,5,4,3,2 );"

http://kent-fredric.fox.geek.nz

[-- Attachment #2: Type: text/html, Size: 1333 bytes --]

Re: [gentoo-dev] About suggesting to create a separate partition for portage tree in handbook

[Richard Yao]

[-- Attachment #1: Type: text/plain, Size: 902 bytes --]

On 03/28/12 03:16, Brian Dolbec wrote:
> On Tue, 2012-03-27 at 19:16 +0100, Ciaran McCreesh wrote:
>> But that's ok, because extensive studies have shown that the only possible
>> reasons for putting /usr/portage on its own partition are historical,
>> since everyone has an SSD now.
>>
> 
> Yeah, right.  Since I must be the only one out there that doesn't yet
> have an SSD, you'll give me (and anyone else that still doesn't) one?

In response to the people who don't like what Brian had to say, I would
like to say that we can't start making assumptions about what hardware
people have and ignore anyone who does not fit those assumptions.

I support Brian on this. If you guys want to have documentation on more
advanced disk tricks, make a separate handbook that specializes in
partitioning and filesystems. The main handbook can include a reference
to it for advanced users.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 900 bytes --]

Re: [gentoo-dev] About suggesting to create a separate partition for portage tree in handbook

[Matt Turner]

On Wed, Mar 28, 2012 at 10:59 AM, Richard Yao <ryao@cs.stonybrook.edu> wrote:
> On 03/28/12 03:16, Brian Dolbec wrote:
>> On Tue, 2012-03-27 at 19:16 +0100, Ciaran McCreesh wrote:
>>> But that's ok, because extensive studies have shown that the only possible
>>> reasons for putting /usr/portage on its own partition are historical,
>>> since everyone has an SSD now.
>>>
>>
>> Yeah, right.  Since I must be the only one out there that doesn't yet
>> have an SSD, you'll give me (and anyone else that still doesn't) one?
>
> In response to the people who don't like what Brian had to say, I would
> like to say that we can't start making assumptions about what hardware
> people have and ignore anyone who does not fit those assumptions.
>
> I support Brian on this. If you guys want to have documentation on more
> advanced disk tricks, make a separate handbook that specializes in
> partitioning and filesystems. The main handbook can include a reference
> to it for advanced users.

You seem to have missed it too, so let's someone just spell it out
before this goes farther.

Ciaran was mocking the argument that's given by proponents of merging
/ into /usr. He *doesn't* actually feel like that.

So let's stop this.

Re: [gentoo-dev] About suggesting to create a separate partition for portage tree in handbook

[Rich Freeman]

On Wed, Mar 28, 2012 at 10:59 AM, Richard Yao <ryao@cs.stonybrook.edu> wrote:
> On 03/28/12 03:16, Brian Dolbec wrote:
>> On Tue, 2012-03-27 at 19:16 +0100, Ciaran McCreesh wrote:
>>> But that's ok, because extensive studies have shown that the only possible
>>> reasons for putting /usr/portage on its own partition are historical,
>>> since everyone has an SSD now.
>>
>> Yeah, right.  Since I must be the only one out there that doesn't yet
>> have an SSD, you'll give me (and anyone else that still doesn't) one?
>
> In response to the people who don't like what Brian had to say, I would
> like to say that we can't start making assumptions about what hardware
> people have and ignore anyone who does not fit those assumptions.

Nobody doesn't like what Brian had to say.  Most everybody around here
including Ciaran likely agrees with him.

The issue is that Ciaran said the complete opposite of what he was
trying to communicate (sarcasm), and that likely due to
language/culture/etc that might not have been clear to somebody who
isn't a native English speaker in a western culture.

The allusion was clearly to the larger udev/systemd/usr issues and the
point he was making is that many of these boil down to disagreements
about what use cases you consider important.

So, just take everything Ciaran said in that particular post, assume
he meant the exact opposite, and now you'll see where he is coming
from.

Yes, I do agree that sarcasm tends to cause problems on international
email lists, but his post did at least make me smile.  :)

Rich

Re: [gentoo-dev] About suggesting to create a separate partition for portage tree in handbook

[Joshua Kinard]

[-- Attachment #1: Type: text/plain, Size: 981 bytes --]

On 03/28/2012 03:16, Brian Dolbec wrote:

> On Tue, 2012-03-27 at 19:16 +0100, Ciaran McCreesh wrote:
>> But that's ok, because extensive studies have shown that the only possible
>> reasons for putting /usr/portage on its own partition are historical,
>> since everyone has an SSD now.
>>
> 
> Yeah, right.  Since I must be the only one out there that doesn't yet
> have an SSD, you'll give me (and anyone else that still doesn't) one?


Does this mean that those of us who still use old mechanical disks are like
a bunch of steampunk fans?

Come to think of it, a disk drive that blew steam and made a whistling sound
without totally destroying the disk would be pretty awesome...

-- 
Joshua Kinard
Gentoo/MIPS
kumba@gentoo.org
4096R/D25D95E3 2011-03-28

"The past tempts us, the present confuses us, the future frightens us.  And
our lives slip away, moment by moment, lost in that vast, terrible in-between."

--Emperor Turhan, Centauri Republic


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 834 bytes --]

Re: [gentoo-dev] About suggesting to create a separate partition for portage tree in handbook

[Brian Dolbec]

[-- Attachment #1: Type: text/plain, Size: 1913 bytes --]

On Wed, 2012-03-28 at 11:37 -0400, Rich Freeman wrote:
> On Wed, Mar 28, 2012 at 10:59 AM, Richard Yao <ryao@cs.stonybrook.edu> wrote:
> > On 03/28/12 03:16, Brian Dolbec wrote:
> >> On Tue, 2012-03-27 at 19:16 +0100, Ciaran McCreesh wrote:
> >>> But that's ok, because extensive studies have shown that the only possible
> >>> reasons for putting /usr/portage on its own partition are historical,
> >>> since everyone has an SSD now.
> >>
> >> Yeah, right.  Since I must be the only one out there that doesn't yet
> >> have an SSD, you'll give me (and anyone else that still doesn't) one?
> >
> > In response to the people who don't like what Brian had to say, I would
> > like to say that we can't start making assumptions about what hardware
> > people have and ignore anyone who does not fit those assumptions.
> 
> Nobody doesn't like what Brian had to say.  Most everybody around here
> including Ciaran likely agrees with him.
> 
> The issue is that Ciaran said the complete opposite of what he was
> trying to communicate (sarcasm), and that likely due to
> language/culture/etc that might not have been clear to somebody who
> isn't a native English speaker in a western culture.
> 
> The allusion was clearly to the larger udev/systemd/usr issues and the
> point he was making is that many of these boil down to disagreements
> about what use cases you consider important.
> 
> So, just take everything Ciaran said in that particular post, assume
> he meant the exact opposite, and now you'll see where he is coming
> from.
> 
> Yes, I do agree that sarcasm tends to cause problems on international
> email lists, but his post did at least make me smile.  :)
> 
> Rich
> 

I didn't miss that his statements were sarcasm.  I just failed at
sarcastic reply without it being clear that it was.  

Sorry, Not my best work :/

-- 
Brian Dolbec <dolsen@gentoo.org>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] About suggesting to create a separate partition for portage tree in handbook

[Joshua Saddler]

[-- Attachment #1: Type: text/plain, Size: 1095 bytes --]

On Tue, 27 Mar 2012 19:49:00 +0200
Pacho Ramos <pacho@gentoo.org> wrote:

> Hello
> 
> I am a bit surprised handbook still doesn't suggest people to
> create a separate partition for /usr/portage tree. I remember my
> first Gentoo systems had it inside / and that lead to a lot of
> fragmentation, much slower "emerge -pvuDN world" (I benchmarked it
> when I changed my partitioning scheme to put /usr/portage) separate
> and a lot of disk space lost (I remember portage tree reached
> around 3 GB of disk space while I am now running with 300MB)
> 
> Could handbook suggest people to put /usr/portage on a different
> partition then? The only doubt I have is what filesystem would be
> better for it, in my case I am using reiserfs with tail enabled,
> but maybe you have other different setups.
> 
> Thanks for discussing this :)

not gonna happen, for reasons that SwifT & others already mentioned.
this is the sort of non-simple, non-trivial text/info/instructions
that would be better suited to an "optimizing your FS layout" article
on the gentoo wiki, or similar.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] About suggesting to create a separate partition for portage tree in handbook

[Dale]

Joshua Saddler wrote:
> On Tue, 27 Mar 2012 19:49:00 +0200
> Pacho Ramos <pacho@gentoo.org> wrote:
> 
>> Hello
>>
>> I am a bit surprised handbook still doesn't suggest people to
>> create a separate partition for /usr/portage tree. I remember my
>> first Gentoo systems had it inside / and that lead to a lot of
>> fragmentation, much slower "emerge -pvuDN world" (I benchmarked it
>> when I changed my partitioning scheme to put /usr/portage) separate
>> and a lot of disk space lost (I remember portage tree reached
>> around 3 GB of disk space while I am now running with 300MB)
>>
>> Could handbook suggest people to put /usr/portage on a different
>> partition then? The only doubt I have is what filesystem would be
>> better for it, in my case I am using reiserfs with tail enabled,
>> but maybe you have other different setups.
>>
>> Thanks for discussing this :)
> 
> not gonna happen, for reasons that SwifT & others already mentioned.
> this is the sort of non-simple, non-trivial text/info/instructions
> that would be better suited to an "optimizing your FS layout" article
> on the gentoo wiki, or similar.


Well, way back when I first installed Gentoo, I actually read some
before I even started.  I learned through all that reading that /,
/boot, /home, /usr, /usr/portage and /var are best on their own
partition.  Each of those are for different reasons.

The root partition is obvious, I would hope anyway.  ;-)  The boot
partitions comes in handy if you don't automount it or have more than
one distro installed.  Home is obvious.  People recommended /usr because
it could a) be mounted read only and b)  it can be enlarged if needed
since it tends to grow a lot.  Portage since it is tons of small files
and tends to fragment a lot.  The var partition is so that if some error
message repeats itself overnight and fills up the partition it at least
doesn't lock up the whole system.  I actually had this one happen to me
once.  For some reason, even logrotate didn't catch it, tar up and
delete the old ones.  I woke up to a mess that only going to single user
would fix.  The best thing I did was to have /var on its own partition.

When people are planning to install Gentoo and they have not done at
least some research, I think they should get to keep the pieces.
Installing Gentoo is not something to do on a whim.  It should be
planned and thought through even if the person is completely new to
Gentoo.  I read up for at least a month before ever even starting.

I agree with having a simple manual for the folks that want to install
just to look and then have a separate manual, wiki even, for more
serious set ups.  This can include things like RAID, LVM and having more
than a couple partitions.  Of course, Gentoo is almost endless in options.

Back to my hole.

Dale

:-)  :-)

-- 
I am only responsible for what I said ... Not for what you understood or
how you interpreted my words!

Miss the compile output?  Hint:
EMERGE_DEFAULT_OPTS="--quiet-build=n"

[gentoo-dev] Re: About suggesting to create a separate partition for portage tree in handbook

[Duncan]

Dale posted on Wed, 28 Mar 2012 19:35:40 -0500 as excerpted:

> Joshua Saddler wrote:
>> On Tue, 27 Mar 2012 19:49:00 +0200 Pacho Ramos <pacho@gentoo.org>
>> wrote:
>> 
>>> I am a bit surprised handbook still doesn't suggest people to create a
>>> separate partition for /usr/portage tree. I remember my first Gentoo
>>> systems had it inside / and that lead to a lot of fragmentation, much
>>> slower "emerge -pvuDN world" [and] a lot of disk space lost

>>> Could handbook suggest people to put /usr/portage on a different
>>> partition then? The only doubt I have is what filesystem would be
>>> better for it, in my case I am using reiserfs with tail enabled, but
>>> maybe you have other different setups.

>> not gonna happen, for reasons that SwifT & others already mentioned.
>> this is the sort of non-simple, non-trivial text/info/instructions that
>> would be better suited to an "optimizing your FS layout" article on the
>> gentoo wiki, or similar.

Agreed, tho ACTUALLY having the documentation available, AND LINKING to 
it in the handbook ("For an in-depth discussion, read..."), would be a 
good thing.

> Well, way back when I first installed Gentoo, I actually read some
> before I even started.  I learned through all that reading that /,
> /boot, /home, /usr, /usr/portage and /var are best on their own
> partition.  Each of those are for different reasons.

Same here.  It's a bit of a point of pride for me that before I even had 
my own gentoo system installed (some problem due to my wanting posix 
threading, then relatively new to Linux, over Linux threads; didn't work 
for me with 2004.0, worked great with 2004.1), I had read the handbook, 
etc, and was replying on the lists to questions from folks who obviously 
hadn't read up...

But I already had a good idea what I wanted my partition layout to look 
like based on my Mandrake experience.  The questions I needed to ask, 
because they were NOT covered in the manual (or anywhere else in the 
documentation I could find at the time), and because they were self-
evidently going to have rather different answers on gentoo than on 
mandrake, were things like:

Just how big IS the portage tree?

What about the package tree?

What about the sources tree?

After a couple partition reorganizations, I ended up with sources inside 
the portage tree, but packages on its own partition, making it easier to 
keep packages backed up, something the portage tree and sources don't 
need as the net's a far more sufficient backup for them than I could ever 
manage locally.


For years I've thought that a bit more emphasis should be placed on 
FEATURES=binpkg, given the many ways it can save your ass and/or make 
troubleshooting a current version issue far easier.  And while I agree 
that the installation section of the handbook, in any case, isn't the 
place for complex discussion of the many system partitioning schemes and 
their positives/negatives, information such as the above, exactly what 
sort of realistic sizes can be expected for the portage tree itself, for 
sources, and for binpkgs (if the feature is enabled), should be covered.

That's because most gentoo users have at least some experience on other 
distros before they come to gentoo, and thus likely already have a 
preferred partitioning setup... if they care about it at all.  All they 
really need is information about the relative sizes of gentoo-specific 
features, the ebuild tree, sources, and binpkgs, and perhaps a bit better 
coverage of the binpkgs option (which I'd simply link-punt in the install 
section as well, but cover it a bit better under the working with portage 
section, with the install-section link pointing there).

> The root partition is obvious, I would hope anyway.  ;-)  The boot
> partitions comes in handy if you don't automount it or have more than
> one distro installed.  Home is obvious.  People recommended /usr because
> it could a) be mounted read only and b)  it can be enlarged if needed
> since it tends to grow a lot.  Portage since it is tons of small files
> and tends to fragment a lot.  The var partition is so that if some error
> message repeats itself overnight and fills up the partition it at least
> doesn't lock up the whole system.  I actually had this one happen to me
> once.  For some reason, even logrotate didn't catch it, tar up and
> delete the old ones.  I woke up to a mess that only going to single user
> would fix.  The best thing I did was to have /var on its own partition.

FWIW, that's /var/log on it's own partition here, for exactly the reason 
you mention.  But /var itself is on rootfs here, these days.

> When people are planning to install Gentoo and they have not done at
> least some research, I think they should get to keep the pieces.
> Installing Gentoo is not something to do on a whim.  It should be
> planned and thought through even if the person is completely new to
> Gentoo.  I read up for at least a month before ever even starting.

Again agreed,

But really, to some degree it's something that's only learned from 
experience.  If anything, what I'd suggest for the installation manual 
partitioning section would be a variant on the programmer's dictum:

"Plan to throw one away, because you're either going to end up doing it 
anyway after you make your mistakes and figure out the way you /should/ 
have done it, or putting up with a sub-optimum setup if you don't throw 
one away, and planning for it from the beginning will make the process 
easier when the time comes."

I know I've gone thru several partition layout iterations here, before I 
came up with something very close to what I'd consider optimal... that 
has stayed that way for several years.

=:^)

> I agree with having a simple manual for the folks that want to install
> just to look and then have a separate manual, wiki even, for more
> serious set ups.  This can include things like RAID, LVM and having more
> than a couple partitions.  Of course, Gentoo is almost endless in
> options.

Agreed.  The only thing I'd add would be that the simple installation 
should have "for more information" type links to the more complex 
discussions of each step/decision, at the appropriate place.  Then people 
like Dale and I will read them, and but they'll be clearly marked "for 
more information" or similar, so those uninterested in that sort of 
discussion can easily skip it. =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: About suggesting to create a separate partition for portage tree in handbook

[Dale]

Duncan wrote:
> Dale posted on Wed, 28 Mar 2012 19:35:40 -0500 as excerpted:
> 
>> Joshua Saddler wrote:

> Agreed, tho ACTUALLY having the documentation available, AND LINKING to 
> it in the handbook ("For an in-depth discussion, read..."), would be a 
> good thing.
> 
>> Well, way back when I first installed Gentoo, I actually read some
>> before I even started.  I learned through all that reading that /,
>> /boot, /home, /usr, /usr/portage and /var are best on their own
>> partition.  Each of those are for different reasons.
> 
> Same here.  It's a bit of a point of pride for me that before I even had 
> my own gentoo system installed (some problem due to my wanting posix 
> threading, then relatively new to Linux, over Linux threads; didn't work 
> for me with 2004.0, worked great with 2004.1), I had read the handbook, 
> etc, and was replying on the lists to questions from folks who obviously 
> hadn't read up...


I started with 1.4.  Dang I'm getting old.  It's one reason I would hate
to leave Gentoo.


> 
> But I already had a good idea what I wanted my partition layout to look 
> like based on my Mandrake experience.  The questions I needed to ask, 
> because they were NOT covered in the manual (or anywhere else in the 
> documentation I could find at the time), and because they were self-
> evidently going to have rather different answers on gentoo than on 
> mandrake, were things like:
> 
> Just how big IS the portage tree?
> 
> What about the package tree?
> 
> What about the sources tree?


I think I found that info somewhere before I installed.  Plus, one could
ask someone on IRC too.  If they have it on a separate partition, the
results of df would be good enough and quick.


> 
> After a couple partition reorganizations, I ended up with sources inside 
> the portage tree, but packages on its own partition, making it easier to 
> keep packages backed up, something the portage tree and sources don't 
> need as the net's a far more sufficient backup for them than I could ever 
> manage locally.
> 
> 
> For years I've thought that a bit more emphasis should be placed on 
> FEATURES=binpkg, given the many ways it can save your ass and/or make 
> troubleshooting a current version issue far easier.  And while I agree 
> that the installation section of the handbook, in any case, isn't the 
> place for complex discussion of the many system partitioning schemes and 
> their positives/negatives, information such as the above, exactly what 
> sort of realistic sizes can be expected for the portage tree itself, for 
> sources, and for binpkgs (if the feature is enabled), should be covered.
> 
> That's because most gentoo users have at least some experience on other 
> distros before they come to gentoo, and thus likely already have a 
> preferred partitioning setup... if they care about it at all.  All they 
> really need is information about the relative sizes of gentoo-specific 
> features, the ebuild tree, sources, and binpkgs, and perhaps a bit better 
> coverage of the binpkgs option (which I'd simply link-punt in the install 
> section as well, but cover it a bit better under the working with portage 
> section, with the install-section link pointing there).
> 
>> The root partition is obvious, I would hope anyway.  ;-)  The boot
>> partitions comes in handy if you don't automount it or have more than
>> one distro installed.  Home is obvious.  People recommended /usr because
>> it could a) be mounted read only and b)  it can be enlarged if needed
>> since it tends to grow a lot.  Portage since it is tons of small files
>> and tends to fragment a lot.  The var partition is so that if some error
>> message repeats itself overnight and fills up the partition it at least
>> doesn't lock up the whole system.  I actually had this one happen to me
>> once.  For some reason, even logrotate didn't catch it, tar up and
>> delete the old ones.  I woke up to a mess that only going to single user
>> would fix.  The best thing I did was to have /var on its own partition.
> 
> FWIW, that's /var/log on it's own partition here, for exactly the reason 
> you mention.  But /var itself is on rootfs here, these days.


That would work too.  At the time, /var was recommended.


> 
>> When people are planning to install Gentoo and they have not done at
>> least some research, I think they should get to keep the pieces.
>> Installing Gentoo is not something to do on a whim.  It should be
>> planned and thought through even if the person is completely new to
>> Gentoo.  I read up for at least a month before ever even starting.
> 
> Again agreed,
> 
> But really, to some degree it's something that's only learned from 
> experience.  If anything, what I'd suggest for the installation manual 
> partitioning section would be a variant on the programmer's dictum:
> 
> "Plan to throw one away, because you're either going to end up doing it 
> anyway after you make your mistakes and figure out the way you /should/ 
> have done it, or putting up with a sub-optimum setup if you don't throw 
> one away, and planning for it from the beginning will make the process 
> easier when the time comes."
> 
> I know I've gone thru several partition layout iterations here, before I 
> came up with something very close to what I'd consider optimal... that 
> has stayed that way for several years.
> 
> =:^)


If someone told me they was going to do a install similar to mine,
without all the udev, init thingy and /usr confusion, I would give a lot
of info but also recommend this.  Install onto a spare drive then run du
on /usr, /var and such.  Take that information and create partitions on
the permanant OS drive then copy the install over.  That way you have
the best possible info based on your own install and what you plan to use.

They would then have a nice and neat system.  If they use LVM, they can
even enlarge/shrink things as needed.  This could be very true
considering the talk of moving portage files around.  Sounds like /var
may be about to grow.


> 
>> I agree with having a simple manual for the folks that want to install
>> just to look and then have a separate manual, wiki even, for more
>> serious set ups.  This can include things like RAID, LVM and having more
>> than a couple partitions.  Of course, Gentoo is almost endless in
>> options.
> 
> Agreed.  The only thing I'd add would be that the simple installation 
> should have "for more information" type links to the more complex 
> discussions of each step/decision, at the appropriate place.  Then people 
> like Dale and I will read them, and but they'll be clearly marked "for 
> more information" or similar, so those uninterested in that sort of 
> discussion can easily skip it. =:^)
> 


No problem there.  If folks could put info on a wiki, it would be nice.
 Example, 'I have KDE, Fluxbox, and this is the space required  xxxx
Gbs.'  Or 'I have Fluxbox and apache installed and I use xxxxGbs.'  Then
list what /usr, /var and others are in Gbs.  That would save some people
from having to redo their drives.  Let the docs point to that for the
ones interested.

What a wish list.  ;-)

Dale

:-)  :-)

-- 
I am only responsible for what I said ... Not for what you understood or
how you interpreted my words!

Miss the compile output?  Hint:
EMERGE_DEFAULT_OPTS="--quiet-build=n"

Re: [gentoo-dev] About suggesting to create a separate partition for portage tree in handbook

[Pacho Ramos]

[-- Attachment #1: Type: text/plain, Size: 2393 bytes --]

Will start to reply but will take some time as I don't have much this
days :(

El mar, 27-03-2012 a las 20:01 +0200, Sven Vermeulen escribió:
> On Tue, Mar 27, 2012 at 07:49:00PM +0200, Pacho Ramos wrote:
> > I am a bit surprised handbook still doesn't suggest people to create a
> > separate partition for /usr/portage tree. I remember my first Gentoo
> > systems had it inside / and that lead to a lot of fragmentation, much
> > slower "emerge -pvuDN world" (I benchmarked it when I changed my
> > partitioning scheme to put /usr/portage) separate and a lot of disk
> > space lost (I remember portage tree reached around 3 GB of disk space
> > while I am now running with 300MB)
> > 
> > Could handbook suggest people to put /usr/portage on a different
> > partition then? The only doubt I have is what filesystem would be better
> > for it, in my case I am using reiserfs with tail enabled, but maybe you
> > have other different setups.
> 
> To be honest, I don't think it is wise to describe it in the Gentoo Handbook
> just yet. I don't mind having it documented elsewhere, but the separate
> partition is not mandatory for getting Gentoo up and running. The
> instructions currently also just give an example partition layout and tell
> users that different layouts are perfectly possible.
> 
> We need to take into consideration what is needed (must) for a Gentoo
> installation, what is seriously recommended (should), what is nice to have
> (could), etc. And for me, having a separate /usr/portage is a nice-to-have
> imo.
> 
> Wkr,
> 	Sven Vermeulen
> 
> 

My idea is to add a comment about this because it's not obvious having
portage tree in a "common" partition with the rest of the system has
some problems like high fragmentation, waste of disk space and also
performance problems. I discovered it empirically when trying to get
"emerge -pvuDN world" a bit faster. 

Also, once a partition scheme is chosen when installing Gentoo at first
time, it's sometimes difficult to modify (for example, I was luck in my
cases because I had big swap partitions I shrinked a bit for portage
tree.

You can probably see it's "nice-to-have" (as partition scheme that is
shown in handbook showing partitions for /var, /home...), but it's
better than letting people put their portage trees in a standard
partition with the rest of the system

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] About suggesting to create a separate partition for portage tree in handbook

[Pacho Ramos]

[-- Attachment #1: Type: text/plain, Size: 2067 bytes --]

El mar, 27-03-2012 a las 14:34 -0400, Alexandre Rostovtsev escribió:
> On Tue, 2012-03-27 at 20:01 +0200, Sven Vermeulen wrote:
> > On Tue, Mar 27, 2012 at 07:49:00PM +0200, Pacho Ramos wrote:
> > > I am a bit surprised handbook still doesn't suggest people to create a
> > > separate partition for /usr/portage tree. I remember my first Gentoo
> > > systems had it inside / and that lead to a lot of fragmentation, much
> > > slower "emerge -pvuDN world" (I benchmarked it when I changed my
> > > partitioning scheme to put /usr/portage) separate and a lot of disk
> > > space lost (I remember portage tree reached around 3 GB of disk space
> > > while I am now running with 300MB)
> > > 
> > > Could handbook suggest people to put /usr/portage on a different
> > > partition then? The only doubt I have is what filesystem would be better
> > > for it, in my case I am using reiserfs with tail enabled, but maybe you
> > > have other different setups.
> > 
> > To be honest, I don't think it is wise to describe it in the Gentoo Handbook
> > just yet. I don't mind having it documented elsewhere, but the separate
> > partition is not mandatory for getting Gentoo up and running. The
> > instructions currently also just give an example partition layout and tell
> > users that different layouts are perfectly possible.
> > 
> > We need to take into consideration what is needed (must) for a Gentoo
> > installation, what is seriously recommended (should), what is nice to have
> > (could), etc. And for me, having a separate /usr/portage is a nice-to-have
> > imo.
[...]
> 2. The handbook should mention that a separate small /usr/portage
> partition can noticeably improve performance for users with a rotational
> hard drive, and that it's not needed for solid-state drives. It should
> also mention that using Gentoo with a separate /usr/portage partition
> will require some additional configuration (such as changing DISTDIR and
> PKGDIR to avoid running out of space).
> 
> -Alexandre.
> 
> 
> 

This would be nice :D

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] About suggesting to create a separate partition for portage tree in handbook

[Pacho Ramos]

[-- Attachment #1: Type: text/plain, Size: 2007 bytes --]

El mar, 27-03-2012 a las 14:53 -0400, Ian Stakenvicius escribió:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
> 
> On 27/03/12 02:47 PM, Rich Freeman wrote:
> > On Tue, Mar 27, 2012 at 2:34 PM, Alexandre Rostovtsev
> > <tetromino@gentoo.org>
> >> The partitioning scheme is something that the user needs to
> >> decide on *before* getting Gentoo up and running. After the user
> >> had finished installing the operating system, it's too late to
> >> inform him about the advantages of a separate /usr/portage.
> > 
> > Yes and no (if you have free space, you could easily move
> > /usr/portage - some other changes are harder).
> > 
> > However, you could extend this line of argument to raid, lvm, and
> > even stuff like the use of systemd or an alternative package
> > manager.  All of those things are much easier to implement if you
> > just start out with them.
> > 
> > I'm all for creating a wiki to talk about some alternative
> > options. Perhaps even link to it at the start of the handbook in
> > the intro (if you're not in a rush and want to read about more
> > advanced configurations, check out ...).
> > 
> > However, I tend to agree that the handbook should be a 
> > nearly-foolproof no-frills Gentoo installation.
> > 
> 
> 
> You know, we have "Code Listing 2.1: Filesystem Example" in Section 4,
> we could always adjust that to have a /usr/portage partition in it
> (take a bit of space away from /home, or something)
> 
> It doesn't recommend/require anything, but when users see it they'll
> think about it.

This would be a good option, but I would anyway add a note warning
people about the cons of having portage tree in a normal partition with
the rest of the system, otherwise people could simply ignore that code
listing because they could thing it's there simply on a try to get all
system "splitted" ;) (for example, I don't usually have a separate
partition for all what is listed there, only /, /home and /usr/portage)


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] About suggesting to create a separate partition for portage tree in handbook

[Pacho Ramos]

[-- Attachment #1: Type: text/plain, Size: 3409 bytes --]

El mar, 27-03-2012 a las 16:05 -0400, Alec Moskvin escribió:
> On Tuesday 27 March 2012 14:34:03, Alexandre Rostovtsev wrote:
> > On Tue, 2012-03-27 at 20:01 +0200, Sven Vermeulen wrote:
> > > On Tue, Mar 27, 2012 at 07:49:00PM +0200, Pacho Ramos wrote:
> > > > I am a bit surprised handbook still doesn't suggest people to create a
> > > > separate partition for /usr/portage tree. I remember my first Gentoo
> > > > systems had it inside / and that lead to a lot of fragmentation, much
> > > > slower "emerge -pvuDN world" (I benchmarked it when I changed my
> > > > partitioning scheme to put /usr/portage) separate and a lot of disk
> > > > space lost (I remember portage tree reached around 3 GB of disk space
> > > > while I am now running with 300MB)
> > > > 
> > > > Could handbook suggest people to put /usr/portage on a different
> > > > partition then? The only doubt I have is what filesystem would be better
> > > > for it, in my case I am using reiserfs with tail enabled, but maybe you
> > > > have other different setups.
> > > 
> > > To be honest, I don't think it is wise to describe it in the Gentoo Handbook
> > > just yet. I don't mind having it documented elsewhere, but the separate
> > > partition is not mandatory for getting Gentoo up and running. The
> > > instructions currently also just give an example partition layout and tell
> > > users that different layouts are perfectly possible.
> > > 
> > > We need to take into consideration what is needed (must) for a Gentoo
> > > installation, what is seriously recommended (should), what is nice to have
> > > (could), etc. And for me, having a separate /usr/portage is a nice-to-have
> > > imo.
> > 
> > The partitioning scheme is something that the user needs to decide on
> > *before* getting Gentoo up and running. After the user had finished
> > installing the operating system, it's too late to inform him about the
> > advantages of a separate /usr/portage.
> 
> It does not have to be a separate *physical* partition. It could be set
> up as a loop device without any real downsides:
> 
> /usr/portage/tree.ext4	/usr/portage/tree	ext4	loop,noatime	0 0
> 
> An advantage is that it can be easily resized if necessary.
> 
> > IMHO, chapter 4 of the handbook needs the following changes:
> > 
> > 1. ext4, not ext3, needs to be recommended as the default filesystem. We
> > have kernel 3.2 marked stable, there is no need to keep talking about
> > ext4 as if it's something experimental.
> > 
> > 2. The handbook should mention that a separate small /usr/portage
> > partition can noticeably improve performance for users with a rotational
> > hard drive, and that it's not needed for solid-state drives. It should
> > also mention that using Gentoo with a separate /usr/portage partition
> > will require some additional configuration (such as changing DISTDIR and
> > PKGDIR to avoid running out of space).
> > 
> > -Alexandre.
> > 
> > 
> 
> 

(I think this last reply can complete my replies to this thread for
now :))

Looks then that there are several alternatives for portage tree, then,
maybe the option would be to add a note to Gentoo Handbook explaining
the cons of having portage tree on a standard partition and, then, put a
link to a wiki page (for example) where all this alternatives are
explained.

What do you think about this approach? 

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] About suggesting to create a separate partition for portage tree in handbook

[Sven Vermeulen]

On Fri, Mar 30, 2012 at 10:06:18AM +0200, Pacho Ramos wrote:
> Looks then that there are several alternatives for portage tree, then,
> maybe the option would be to add a note to Gentoo Handbook explaining
> the cons of having portage tree on a standard partition and, then, put a
> link to a wiki page (for example) where all this alternatives are
> explained.
> 
> What do you think about this approach? 

I don't like the "cons" approach, as it gives the impression that users are
pushed into a negative solution, whereas the current situation works just
fine for almost all users. The approach for a different partition is for
performance reasons (which most users don't have any negative feelings
about) and as such might be read as a "ricer" approach.

But perhaps it would be more "lean" to just start with a wiki page (or
document) for alternative / better partitioning layouts, and when that has
stabilized then we can talk about Handbook integration, not?

Wkr,
	Sven Vermeulen

Re: [gentoo-dev] About suggesting to create a separate partition for portage tree in handbook

[Brian Harring]

On Sat, Mar 31, 2012 at 08:44:02AM +0000, Sven Vermeulen wrote:
> On Fri, Mar 30, 2012 at 10:06:18AM +0200, Pacho Ramos wrote:
> > Looks then that there are several alternatives for portage tree, then,
> > maybe the option would be to add a note to Gentoo Handbook explaining
> > the cons of having portage tree on a standard partition and, then, put a
> > link to a wiki page (for example) where all this alternatives are
> > explained.
> > 
> > What do you think about this approach? 
> 
> I don't like the "cons" approach, as it gives the impression that users are
> pushed into a negative solution, whereas the current situation works just
> fine for almost all users. The approach for a different partition is for
> performance reasons (which most users don't have any negative feelings
> about) and as such might be read as a "ricer" approach.

For modern hardware w/ a modern kernel (or at least >=2.6.38 for the 
dcache resolution optimizations)... does anyone actually have real 
performance stats for this?

If the notion is a seperate FS, one tailored to the portage tree's 
usage models (tail packing for example), sure, grok that although I 
question how much people really are getting out of it.

In the past, situation definitely differed- I'm just wondering if the 
gain is actually worth debating it, rather than just ignoring it (or 
sticking it in a foot note for people trying to use durons).
~harring

Re: [gentoo-dev] About suggesting to create a separate partition for portage tree in handbook

[Pacho Ramos]

[-- Attachment #1: Type: text/plain, Size: 1690 bytes --]

El sáb, 31-03-2012 a las 08:44 +0000, Sven Vermeulen escribió:
> On Fri, Mar 30, 2012 at 10:06:18AM +0200, Pacho Ramos wrote:
> > Looks then that there are several alternatives for portage tree, then,
> > maybe the option would be to add a note to Gentoo Handbook explaining
> > the cons of having portage tree on a standard partition and, then, put a
> > link to a wiki page (for example) where all this alternatives are
> > explained.
> > 
> > What do you think about this approach? 
> 
> I don't like the "cons" approach, as it gives the impression that users are
> pushed into a negative solution, whereas the current situation works just
> fine for almost all users. The approach for a different partition is for
> performance reasons (which most users don't have any negative feelings
> about) and as such might be read as a "ricer" approach.
> 
> But perhaps it would be more "lean" to just start with a wiki page (or
> document) for alternative / better partitioning layouts, and when that has
> stabilized then we can talk about Handbook integration, not?
> 
> Wkr,
> 	Sven Vermeulen
> 
> 

Current solution works but causes a really slow portage tree when ages
passes (I still have a machine with tree in / and is really really slow
but, since it's used by my father at his job, I am unable to solve
it :( ). And not, I don't think it's a ricer approach at all, it's for
performance and for save a lot of disk space too.

About the wiki page, I can only document reiserfs+tail usage as it's the
one I use and I know, about other alternatives like using squashfs, loop
mount... I cannot promise anything as I simply don't know how to set
them.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] About suggesting to create a separate partition for portage tree in handbook

[Pacho Ramos]

[-- Attachment #1: Type: text/plain, Size: 1851 bytes --]

El sáb, 31-03-2012 a las 02:35 -0700, Brian Harring escribió:
> On Sat, Mar 31, 2012 at 08:44:02AM +0000, Sven Vermeulen wrote:
> > On Fri, Mar 30, 2012 at 10:06:18AM +0200, Pacho Ramos wrote:
> > > Looks then that there are several alternatives for portage tree, then,
> > > maybe the option would be to add a note to Gentoo Handbook explaining
> > > the cons of having portage tree on a standard partition and, then, put a
> > > link to a wiki page (for example) where all this alternatives are
> > > explained.
> > > 
> > > What do you think about this approach? 
> > 
> > I don't like the "cons" approach, as it gives the impression that users are
> > pushed into a negative solution, whereas the current situation works just
> > fine for almost all users. The approach for a different partition is for
> > performance reasons (which most users don't have any negative feelings
> > about) and as such might be read as a "ricer" approach.
> 
> For modern hardware w/ a modern kernel (or at least >=2.6.38 for the 
> dcache resolution optimizations)... does anyone actually have real 
> performance stats for this?
> 
> If the notion is a seperate FS, one tailored to the portage tree's 
> usage models (tail packing for example), sure, grok that although I 
> question how much people really are getting out of it.
> 
> In the past, situation definitely differed- I'm just wondering if the 
> gain is actually worth debating it, rather than just ignoring it (or 
> sticking it in a foot note for people trying to use durons).
> ~harring
> 
> 

I did performance stats one year ago or so, but I don't have time to
redo all of them to simply confirm how behave now with recent kernel (in
that time, I checked reiserfs, ext2 with multiple block sizes).
Regarding disk space usage, it's still valid today for sure

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] About suggesting to create a separate partition for portage tree in handbook

[Zac Medico]

On 03/31/2012 06:34 AM, Pacho Ramos wrote:
> About the wiki page, I can only document reiserfs+tail usage as it's the
> one I use and I know, about other alternatives like using squashfs, loop
> mount... I cannot promise anything as I simply don't know how to set
> them.

Squashfs is really simple to use:

   mksquashfs /usr/portage portage.squashfs
   mount -o loop portage.squashfs /usr/portage

-- 
Thanks,
Zac

Re: [gentoo-dev] About suggesting to create a separate partition for portage tree in handbook

[Walter Dnes]

On Sat, Mar 31, 2012 at 10:42:50AM -0700, Zac Medico wrote
> On 03/31/2012 06:34 AM, Pacho Ramos wrote:
> > About the wiki page, I can only document reiserfs+tail usage as it's the
> > one I use and I know, about other alternatives like using squashfs, loop
> > mount... I cannot promise anything as I simply don't know how to set
> > them.
> 
> Squashfs is really simple to use:
> 
>    mksquashfs /usr/portage portage.squashfs
>    mount -o loop portage.squashfs /usr/portage

  Don't the "space-saving filesystems" (squashfs, reiserfs-with-tail,
etc) run more slowly due to their extra finicky steps to save space?  If
you really want to save a gigabyte or 2, run "eclean -d distfiles" and
"localepurge" after every emerge update.  I've also cobbled together my
own "autodepclean" script that check for, and optionally unmerges
unneeded stuff that was pulled in as a dependancy of a package that has
since been removed.

-- 
Walter Dnes <waltdnes@waltdnes.org>

Re: [gentoo-dev] About suggesting to create a separate partition for portage tree in handbook

[Zac Medico]

On 03/31/2012 04:25 PM, Walter Dnes wrote:
> On Sat, Mar 31, 2012 at 10:42:50AM -0700, Zac Medico wrote
>> On 03/31/2012 06:34 AM, Pacho Ramos wrote:
>>> About the wiki page, I can only document reiserfs+tail usage as it's the
>>> one I use and I know, about other alternatives like using squashfs, loop
>>> mount... I cannot promise anything as I simply don't know how to set
>>> them.
>>
>> Squashfs is really simple to use:
>>
>>    mksquashfs /usr/portage portage.squashfs
>>    mount -o loop portage.squashfs /usr/portage
> 
>   Don't the "space-saving filesystems" (squashfs, reiserfs-with-tail,
> etc) run more slowly due to their extra finicky steps to save space?  If
> you really want to save a gigabyte or 2, run "eclean -d distfiles" and
> "localepurge" after every emerge update.  I've also cobbled together my
> own "autodepclean" script that check for, and optionally unmerges
> unneeded stuff that was pulled in as a dependancy of a package that has
> since been removed.

Well, in this case squashfs is more about improving access time than
saving space. You end up with the whole tree stored in a mostly
contiguous chunk of disk space, which minimizes seek time.
-- 
Thanks,
Zac

Re: [gentoo-dev] About suggesting to create a separate partition for portage tree in handbook

[Pacho Ramos]

[-- Attachment #1: Type: text/plain, Size: 1195 bytes --]

El sáb, 31-03-2012 a las 19:25 -0400, Walter Dnes escribió:
> On Sat, Mar 31, 2012 at 10:42:50AM -0700, Zac Medico wrote
> > On 03/31/2012 06:34 AM, Pacho Ramos wrote:
> > > About the wiki page, I can only document reiserfs+tail usage as it's the
> > > one I use and I know, about other alternatives like using squashfs, loop
> > > mount... I cannot promise anything as I simply don't know how to set
> > > them.
> > 
> > Squashfs is really simple to use:
> > 
> >    mksquashfs /usr/portage portage.squashfs
> >    mount -o loop portage.squashfs /usr/portage
> 
>   Don't the "space-saving filesystems" (squashfs, reiserfs-with-tail,
> etc) run more slowly due to their extra finicky steps to save space?  If
> you really want to save a gigabyte or 2, run "eclean -d distfiles" and
> "localepurge" after every emerge update.  I've also cobbled together my
> own "autodepclean" script that check for, and optionally unmerges
> unneeded stuff that was pulled in as a dependancy of a package that has
> since been removed.
> 

I have distfiles on a completely different dir and, using different
partition for ages for portage tree hasn't show that space saving
problems

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[gentoo-dev] Re: About suggesting to create a separate partition for portage tree in handbook

[Steven J Long]

Walter Dnes wrote:
> I've also cobbled together my
> own "autodepclean" script that check for, and optionally unmerges
> unneeded stuff that was pulled in as a dependancy of a package that has
> since been removed.
>
What advantage does it have over a standard --depclean?
 
-- 
#friendly-coders -- We're friendly, but we're not /that/ friendly ;-)

Re: [gentoo-dev] Re: About suggesting to create a separate partition for portage tree in handbook

[Walter Dnes]

On Sun, Apr 01, 2012 at 04:28:15PM +0100, Steven J Long wrote
> Walter Dnes wrote:
> > I've also cobbled together my
> > own "autodepclean" script that check for, and optionally unmerges
> > unneeded stuff that was pulled in as a dependancy of a package that has
> > since been removed.
> >
> What advantage does it have over a standard --depclean?

  It reads the output of "emerge --pretend --depclean" and creates an
executable script "cleanscript" in the current directory.  cleanscript
is a list of "emerge --depclean" commands, followed by "revdep-rebuild"
at the very end.  The advantage is that you get to see ahead of time
what would be removed.  Even edit it before running, if you so desire.
Here it is...

#!/bin/bash
# autodepclean script v 0.03 released under GPL v3 by Walter Dnes 2012/01/16
# Generates a file "cleanscript" to remove unused ebuilds, including
# buildtime-only dependancies.
#
# Warning; this script is still beta.  I recommend that you check the output
# in cleanscript before running it.
#
# With the arrival of "virtual/editor", the script now suggests removing
# app-editors/nano, which may not be what you want.  If you want to keep
# nano, put it into world
#
# version 0.03 disables the removal of gentoo-sources.  Your current kernel
# is not always the most recent one in /usr/src.
#
echo "#!/bin/bash" > cleanscript
echo "#" >> cleanscript
emerge --pretend --depclean |\
  grep -A1 "^ .*/" |\
  grep -v "^ \*" |\
  grep -v "^--" |\
  sed ":/: {
N
s:\n::
s/    selected: /-/
s/^ /emerge --depclean =/
}" | grep -v "gentoo-sources" >> cleanscript
echo "revdep-rebuild" >> cleanscript
chmod 744 cleanscript

-- 
Walter Dnes <waltdnes@waltdnes.org>

Re: [gentoo-dev] About suggesting to create a separate partition for portage tree in handbook

[Pacho Ramos]

El sáb, 31-03-2012 a las 17:33 -0700, Zac Medico escribió:
> On 03/31/2012 04:25 PM, Walter Dnes wrote:
> > On Sat, Mar 31, 2012 at 10:42:50AM -0700, Zac Medico wrote
> >> On 03/31/2012 06:34 AM, Pacho Ramos wrote:
> >>> About the wiki page, I can only document reiserfs+tail usage as it's the
> >>> one I use and I know, about other alternatives like using squashfs, loop
> >>> mount... I cannot promise anything as I simply don't know how to set
> >>> them.
> >>
> >> Squashfs is really simple to use:
> >>
> >>    mksquashfs /usr/portage portage.squashfs
> >>    mount -o loop portage.squashfs /usr/portage
> > 
> >   Don't the "space-saving filesystems" (squashfs, reiserfs-with-tail,
> > etc) run more slowly due to their extra finicky steps to save space?  If
> > you really want to save a gigabyte or 2, run "eclean -d distfiles" and
> > "localepurge" after every emerge update.  I've also cobbled together my
> > own "autodepclean" script that check for, and optionally unmerges
> > unneeded stuff that was pulled in as a dependancy of a package that has
> > since been removed.
> 
> Well, in this case squashfs is more about improving access time than
> saving space. You end up with the whole tree stored in a mostly
> contiguous chunk of disk space, which minimizes seek time.

Would be possible to generate and provide squashed files at the same
time tarballs with portage tree snapshots are generated? mksquashfs can
take a lot of resources depending on the machine, but providing the
squashed images would still benefit people allowing them to download and
mount them

Re: [gentoo-dev] About suggesting to create a separate partition for portage tree in handbook

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 3484 bytes --]

Dnia 2013-07-21, o godz. 13:42:17
Pacho Ramos <pacho@gentoo.org> napisał(a):

> El sáb, 31-03-2012 a las 17:33 -0700, Zac Medico escribió:
> > On 03/31/2012 04:25 PM, Walter Dnes wrote:
> > > On Sat, Mar 31, 2012 at 10:42:50AM -0700, Zac Medico wrote
> > >> On 03/31/2012 06:34 AM, Pacho Ramos wrote:
> > >>> About the wiki page, I can only document reiserfs+tail usage as it's the
> > >>> one I use and I know, about other alternatives like using squashfs, loop
> > >>> mount... I cannot promise anything as I simply don't know how to set
> > >>> them.
> > >>
> > >> Squashfs is really simple to use:
> > >>
> > >>    mksquashfs /usr/portage portage.squashfs
> > >>    mount -o loop portage.squashfs /usr/portage
> > > 
> > >   Don't the "space-saving filesystems" (squashfs, reiserfs-with-tail,
> > > etc) run more slowly due to their extra finicky steps to save space?  If
> > > you really want to save a gigabyte or 2, run "eclean -d distfiles" and
> > > "localepurge" after every emerge update.  I've also cobbled together my
> > > own "autodepclean" script that check for, and optionally unmerges
> > > unneeded stuff that was pulled in as a dependancy of a package that has
> > > since been removed.
> > 
> > Well, in this case squashfs is more about improving access time than
> > saving space. You end up with the whole tree stored in a mostly
> > contiguous chunk of disk space, which minimizes seek time.
> 
> Would be possible to generate and provide squashed files at the same
> time tarballs with portage tree snapshots are generated? mksquashfs can
> take a lot of resources depending on the machine, but providing the
> squashed images would still benefit people allowing them to download and
> mount them

I'm experimenting with squashfs lately and here's a few notes:

1. I didn't find a good way of generating incremental images with
squashfs itself. I didn't try tools like diffball (those that were used
in emerge-delta-webrsync) but I recall they were very slow (you'd have
to use 56K modem to get them faster than rsync) and I doubt they'll fit
squashfs specifics.

2. squashfs is best used with union filesystem like aufs3. However,
that basically requires patching the kernel since FUSE-based union
filesystems simply don't work.

a) unionfs-fuse doesn't support replacing files from read-only branch,

b) funinonfs gets broken with rsync somehow.

I haven't tested le ol' unionfs, but aufs3 I get working great.

3. squashfs+aufs3 really benefits from '--omit-dir-times' rsync option.
Otherwise, it recreates the whole directory structure on each rsync.
This also causes much less output. We should think about making this
the default.

4. 'emerge --sync' is ultra-fast with this combo. very big sync goes
in less than a minute.

5. I have doubts about 'emerge -1vDtu @world' speed. It is very
subjective feeling but I feel like reiserfs was actually faster in this
regard. However, space savings would surely benefit our users.

6. if we're to do squahfs+aufs3, we need a clean dir structure for all
of it, including squashfs files, intermediate mounts and r/w branches.

7. we could probably get incremential squashfs+aufs3 through squashing
old r/w branches and adding new ones on top of them. But considering
the 'emerge --sync' speed gain, I don't know if this is really worth
the effort, and if increase in branches wouldn't make it slow.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

Re: [gentoo-dev] About suggesting to create a separate partition for portage tree in handbook

[Pacho Ramos]

El dom, 21-07-2013 a las 13:57 +0200, Michał Górny escribió:
[...]
> 5. I have doubts about 'emerge -1vDtu @world' speed. It is very
> subjective feeling but I feel like reiserfs was actually faster in this
> regard. However, space savings would surely benefit our users.
> 

I also feel it faster (or, at least, not slower) with reiserfs, but
going from ~300 MB to 79. Not sure if it would benefit from putting
squashed image in a different filesystem (it was placed in /root, that
is ext4 in my case). Maybe it would be faster if generated image was put
in /var/tmp/portage (that is tmpfs in my case)

But I am testing it with plain squashfs (without write support)

Re: [gentoo-dev] About suggesting to create a separate partition for portage tree in handbook

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 897 bytes --]

Dnia 2013-07-21, o godz. 14:06:12
Pacho Ramos <pacho@gentoo.org> napisał(a):

> El dom, 21-07-2013 a las 13:57 +0200, Michał Górny escribió:
> [...]
> > 5. I have doubts about 'emerge -1vDtu @world' speed. It is very
> > subjective feeling but I feel like reiserfs was actually faster in this
> > regard. However, space savings would surely benefit our users.
> > 
> 
> I also feel it faster (or, at least, not slower) with reiserfs, but
> going from ~300 MB to 79. Not sure if it would benefit from putting
> squashed image in a different filesystem (it was placed in /root, that
> is ext4 in my case). Maybe it would be faster if generated image was put
> in /var/tmp/portage (that is tmpfs in my case)

Using different block size may make a difference. I suspect that most
important reason for the slowdown is due to random accesses.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

Re: [gentoo-dev] About suggesting to create a separate partition for portage tree in handbook

[Michael Weber]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 07/21/2013 01:42 PM, Pacho Ramos wrote:
> Would be possible to generate and provide squashed files at the
> same time tarballs with portage tree snapshots are generated?
> mksquashfs can take a lot of resources depending on the machine,
> but providing the squashed images would still benefit people
> allowing them to download and mount them

I've establish a cron job on my server to generate gzip and xz
squashed snapshots. I sync distfiles from utwente at 6:05 and generate
the squashfs at 6:35 after verifying the gpg signatures.
There's a 10,5h lag between snapshots and squashfs files - we could
improve if I'm allowed to sync against master rsync/dinstfiles.

[1] http://lore.xmw.de/gentoo/genberry/snapshots/

- -- 
Michael Weber
Gentoo Developer
web: https://xmw.de/
mailto: Michael Weber <xmw@gentoo.org>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.20 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iF4EAREIAAYFAlHr744ACgkQknrdDGLu8JAuNAD/YB8f+Pee7FNkjnNfnjaCYyMM
kdYw2JnbGyH4Srvqlj8A/A/yC37W7MFOZSESLFipkvG01zQ6EvTM0576dC1Z9kdI
=lBLB
-----END PGP SIGNATURE-----

Re: [gentoo-dev] About suggesting to create a separate partition for portage tree in handbook

[justin]

[-- Attachment #1: Type: text/plain, Size: 1022 bytes --]

On 7/21/13 4:26 PM, Michael Weber wrote:
> On 07/21/2013 01:42 PM, Pacho Ramos wrote:
>> Would be possible to generate and provide squashed files at the
>> same time tarballs with portage tree snapshots are generated?
>> mksquashfs can take a lot of resources depending on the machine,
>> but providing the squashed images would still benefit people
>> allowing them to download and mount them
> 
> I've establish a cron job on my server to generate gzip and xz
> squashed snapshots. I sync distfiles from utwente at 6:05 and generate
> the squashfs at 6:35 after verifying the gpg signatures.
> There's a 10,5h lag between snapshots and squashfs files - we could
> improve if I'm allowed to sync against master rsync/dinstfiles.
> 
> [1] http://lore.xmw.de/gentoo/genberry/snapshots/
> 
> 

I am creating them as well. Perhaps we can bundle the effort.

What I also found out that using zsync is quite efficient with squashfs
images. I normally don't sync more then 20-30% of the image.

Justin


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 268 bytes --]

Re: [gentoo-dev] About suggesting to create a separate partition for portage tree in handbook

[Pacho Ramos]

El dom, 21-07-2013 a las 16:46 +0200, justin escribió:
> On 7/21/13 4:26 PM, Michael Weber wrote:
> > On 07/21/2013 01:42 PM, Pacho Ramos wrote:
> >> Would be possible to generate and provide squashed files at the
> >> same time tarballs with portage tree snapshots are generated?
> >> mksquashfs can take a lot of resources depending on the machine,
> >> but providing the squashed images would still benefit people
> >> allowing them to download and mount them
> > 
> > I've establish a cron job on my server to generate gzip and xz
> > squashed snapshots. I sync distfiles from utwente at 6:05 and generate
> > the squashfs at 6:35 after verifying the gpg signatures.
> > There's a 10,5h lag between snapshots and squashfs files - we could
> > improve if I'm allowed to sync against master rsync/dinstfiles.
> > 
> > [1] http://lore.xmw.de/gentoo/genberry/snapshots/
> > 
> > 
> 
> I am creating them as well. Perhaps we can bundle the effort.
> 
> What I also found out that using zsync is quite efficient with squashfs
> images. I normally don't sync more then 20-30% of the image.
> 
> Justin
> 

Maybe infra could be contacted to try to share the effort (and also
offer the snapshot in a bit more "official" way, I mean, similar to
tarballs with snapshots)

Re: [gentoo-dev] About suggesting to create a separate partition for portage tree in handbook

[Zac Medico]

On 07/21/2013 04:57 AM, Michał Górny wrote:
> a) unionfs-fuse doesn't support replacing files from read-only branch,

Maybe you've got some kind of configuration problem (did you forget to
enable the cow option?), because unionfs-fuse seems to work fine for me.
-- 
Thanks,
Zac

Re: [gentoo-dev] About suggesting to create a separate partition for portage tree in handbook

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 419 bytes --]

Dnia 2013-07-21, o godz. 11:00:46
Zac Medico <zmedico@gentoo.org> napisał(a):

> On 07/21/2013 04:57 AM, Michał Górny wrote:
> > a) unionfs-fuse doesn't support replacing files from read-only branch,
> 
> Maybe you've got some kind of configuration problem (did you forget to
> enable the cow option?), because unionfs-fuse seems to work fine for me.

It is possible.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]