[gentoo-dev] Large scale deployments - and portage

[gentoo-dev] Large scale deployments - and portage

[Ron O'Hara]

Hi,

I may have missed something in all the docs, but as far as I can see 
there is what should be a fairly simple enhancement to portage that 
would help considerably in making Gentoo more practical in larger 
deployments.

I came to gentoo to get the advantages of being able to tailor my 
system(s) and still have a simple, consistent upgrade path for all the 
component software - portage is great for that. From the developer and 
tinkerer point of view it is hard to see improvements in the overall 
strategy behind portage.

BUT - for larger deployments, there are extra needs which are not 
currently handled by portage - but should be a snap to implement.

In large deployments, it is critical to maintain strong change control 
and have predictable environments that you can validate application 
performance in. Many places use a strategy like:  "All production 
servers are RedHat 7.1 (or Solaris 2.7, or Windows 2000 +SP5 ... etc)" 

At present I cant see a way to easily do that with gentoo. Eg. Gentoo 
1.4 final is really just a starting point - the minute you do an 'emerge 
sync', you have an unknown mix of software. And worse, if you do it an 
hour later on a different machine, it could be a slight different mix 
again - different from the first box.
It is relatively easy to compare the portage trees and work out what the 
differences are, but thats not much help

If it were possible to 'tag' the portage tree with labels at regular 
intervals, and and do an 'emerge sync' with a nominated 'tag' - then you 
would have the equivelant of the fixed points that other distributions 
have when they cut a release CD.




As an example scenario for someone building deployment images in a huge 
telco.

After some experimentation, they decide that their basic gentoo system 
will be 'gentoo with rsync tag gentoo-1.4-Aug-2003'

They take the first box and do a standard gentoo install, but at the 
'emerge sync' point in the instructions, they do 'emerge sync 
tag=gentoo-1.4-Aug-2003'.
They then complete all the package installations and tweaking that they 
want - site specific choices.


This is a repeatable process!! - critical for duplicating an environment.

As each of the systems is deployed - they are in a known state. Even 
nicer is that deployed systems can be easily upgraded to some other 
'known environment' by doing an 'emerge sync tag=xxxx' to some later tag.

For 'hotfixes', deployed boxes probably need the normal 'emerge sync' 
which will give them acvcess to the latest and greatest bug fix - but 
you would probably not do an 'emerge -u world' on these boxes unless 
moving it from one 'tag level' to another.

Now the only question is - did I miss something and this already exists?

Regards
Ron O'Hara








.


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Large scale deployments - and portage

[Stuart Herbert]

[-- Attachment #1: signed data --]
[-- Type: text/plain, Size: 1567 bytes --]

Hi Ron,

On Sunday 17 August 2003 11:09 pm, Ron O'Hara wrote:
> If it were possible to 'tag' the portage tree with labels at regular
> intervals, and and do an 'emerge sync' with a nominated 'tag' - then you
> would have the equivelant of the fixed points that other distributions
> have when they cut a release CD.

It's an interesting idea ;-)

You might want to have a look at running your own (internal) rsync mirror for 
your Gentoo boxes - and your own (internal) distfiles mirror too.  This gives 
you complete control over what ebuilds your many Gentoo machines will get, 
and ensures that you can roll out software long after it has been removed 
from Gentoo's portage (we delete 'old' files all the time, when we believe 
they're obsolete, which really screws up consistency across a large 
deployment.  However, with your own internal mirror, you can keep these 
ebuilds and their distfiles around long after Gentoo has dropped them!)

Tbh, it's a hack, rather than a nice solid server/client enterprise-ready 
Portage solution, but it's one that does work.

Best regards,
Stu
-- 
Stuart Herbert                                              stuart@gentoo.org
Gentoo Developer                                       http://www.gentoo.org/
Beta packages for download            http://dev.gentoo.org/~stuart/packages/
Come and meet me in March 2004                 http://www.phparch.com/cruise/

GnuGP key id# F9AFC57C available from http://pgp.mit.edu
Key fingerprint = 31FB 50D4 1F88 E227 F319  C549 0C2F 80BA F9AF C57C
--

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Large scale deployments - and portage

[Ron O'Hara]

Hi Stuart,

>You might want to have a look at running your own (internal) rsync mirror for 
>your Gentoo boxes - and your own (internal) distfiles mirror too.  This gives 
>you complete control over what ebuilds your many Gentoo machines will get, 
>and ensures that you can roll out software long after it has been removed 
>from Gentoo's portage (we delete 'old' files all the time, when we believe 
>they're obsolete, which really screws up consistency across a large 
>deployment.  However, with your own internal mirror, you can keep these 
>ebuilds and their distfiles around long after Gentoo has dropped them!)
>

Having your own rsync mirror is only really like having a single extra 
'tag'... although it does address the issue of ebuilds that disappear. 
(so does /usr/local/portage)
The other deficiency with running your own rsyncmirror is just that it's 
effectively a private 'fork' - that pushes lots of the maintenance 
issues back into your own lap. A primary benefit of  using any 'distro' 
is that a bigger team  is keeping an eye on all the changes going on. 
With a commercially backed distro like RedHat you are hoping that they 
have the financial resources to keep on supporting that distro and 
creating RPMS to match the old releases. You also have a defined 
statement about how long a particular old release is supported.

Personally, I prefer the open source community approach for real 
guarantees that support will continue -even if I need to do a little bit 
of it myself on behalf of the community - and this is a different issue 
from the idea of supporting 'tags' against the emerge sync.  The 
community (collectively) has more resources than any individual or company.

You are also right that removing 'old' files from portage is an issue... 
in fact probably a show stopper in some instances. Perhaps the solution 
is to look at it as if the portage tree is under CVS control. That would 
make the unstable "~arch" stuff associated with the HEAD of the tree. 
You would need a label to identify the equivelant of the 'stable' 
branch.  Other labels would represent the 'tag's available for using 
with emerge sync.
In this style of  setup, old ebuilds are not 'deleted' - just removed 
from the stable and HEAD parts of the tree. IE. They still exists within 
the 'tag' that represents the historical development of gentoo. They are 
no longer maintained, but are still part of an 'old release' (or tag)
(I'm really thinking of it more in terms of  a Clearcase style version 
control file system where a user has a 'view' of the files at a 
particular point in time on a particular branch of development - maybe 
use http://www.gnu.org/directory/sysadmin/vc/cvsfs.html ???)

The associated space and bandwidth issues for mirror sites can be 
addressed by only mirroring the HEAD and 'stable' parts - this is the 
same as the current level.


It then becomes easy to implement a policy for support of 'back 
releases' - you could just choose something like - "gentoo maintains 
four years worth of tagged versions." Not only that, if any company 
wants to have a longer timeframe supported, then they only need to offer 
disk space and bandwidth support to achieve this - or THEN run their own 
rsync mirror.
Since gentoo is a source level distro, many of the hassles that RedHat, 
Mandrake etc have making RPMS for old releases dont apply.

I'm NOT implying developer support for 'old' ebuilds - but if package 
'xzy' used to be available with Gentoo in Aug2002, it should still be 
available when you are building a system at that version level. This 
would also mean that if someone really must have it supported - then 
they can do it themselves and get it moved back into the main branch. If 
this 'tag' idea makes it attactive for companies to use Gentoo, then I 
would expect an explosion in the number of active (company paid) 
developers maintaining the ebuilds of little programs that the core team 
could not justify supporting.


Another thing that running your own rsync mirror can never achieve is 
third party vendor certification. Having defined 'tags' would allow 
companies like Oracle to certify a particular version of Gentoo as being 
supported. Thats not currently possible. The very power and flexibility 
that Gentoo gives developers is also totally incompatible with the major 
software vendor QA techniques. Using a 'tag' would go a long way to 
overcoming this.
It would become possible for people like SAP to certify their products 
as supported with specific 'make.conf' settings for a specific tag. 
Thats enough to make Gentoo fine for both the Quality Assured type large 
deployment and yet still retain brilliant upgrade and security patch 
capabilities.

Cheers
Ron

>
>Tbh, it's a hack, rather than a nice solid server/client enterprise-ready 
>Portage solution, but it's one that does work.
>
>Best regards,
>Stu
>  
>



--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Large scale deployments - and portage

[Matt Thrailkill]

I've been using my own rsync mirror for my handful of Gentoo boxes, and
for a long time I've been lusting after something like tagged branches
of the Portage tree like how you have Woody or Slink or Sarge in Debian,
or how you have -CURRENT, -STABLE, -RELENG_N_M in FreeBSD.

This discussion has come up a few times on this list before, in short
form.  It sounds like the core Gentoo team is busier with other things,
and the zeitgeist is that people like us would just use the GRP.

I'd LOVE to start/work on some sort of project to maintain different
Portage trees for people to rsync against.  I think it could work out
really well, it'd be a symbiotic relationship to the current Gentoo -
the herd of Gentoo developers can continue advancing the distro, adding
bleeding edge ebuilds to Portage, etc.  Basically, advancing the Gentoo
analogue of sid/-CURRENT.  Then the other Portage Branches project can
siphon off the ebuilds they want and apply their own pessimism to create
stable branches, versioned releases of the tree and so on.  Then people
pick the branch they want to sync against, and use it.

I wrote a short rant about it here:
http://branched.modestolan.com

On Mon, 18 Aug 2003 09:27:36 +1000
Ron O'Hara <rono@sentuny.com.au> wrote:

> Having your own rsync mirror is only really like having a single extra
> 
> 'tag'... although it does address the issue of ebuilds that disappear.
> 
> (so does /usr/local/portage)
> The other deficiency with running your own rsyncmirror is just that
> it's effectively a private 'fork' - that pushes lots of the
> maintenance issues back into your own lap. A primary benefit of  using
> any 'distro' is that a bigger team  is keeping an eye on all the
> changes going on. With a commercially backed distro like RedHat you
> are hoping that they have the financial resources to keep on
> supporting that distro and creating RPMS to match the old releases.
> You also have a defined statement about how long a particular old
> release is supported.
> 
> Personally, I prefer the open source community approach for real 
> guarantees that support will continue -even if I need to do a little
> bit of it myself on behalf of the community - and this is a different
> issue from the idea of supporting 'tags' against the emerge sync.  The
> 
> community (collectively) has more resources than any individual or
> company.
> 
> You are also right that removing 'old' files from portage is an
> issue... in fact probably a show stopper in some instances. Perhaps
> the solution is to look at it as if the portage tree is under CVS
> control. That would make the unstable "~arch" stuff associated with
> the HEAD of the tree. You would need a label to identify the
> equivelant of the 'stable' branch.  Other labels would represent the
> 'tag's available for using with emerge sync.
> In this style of  setup, old ebuilds are not 'deleted' - just removed 
> from the stable and HEAD parts of the tree. IE. They still exists
> within the 'tag' that represents the historical development of gentoo.
> They are no longer maintained, but are still part of an 'old release'
> (or tag)(I'm really thinking of it more in terms of  a Clearcase style
> version control file system where a user has a 'view' of the files at
> a particular point in time on a particular branch of development -
> maybe use http://www.gnu.org/directory/sysadmin/vc/cvsfs.html ???)
> 
> The associated space and bandwidth issues for mirror sites can be 
> addressed by only mirroring the HEAD and 'stable' parts - this is the 
> same as the current level.
> 
> 
> It then becomes easy to implement a policy for support of 'back 
> releases' - you could just choose something like - "gentoo maintains 
> four years worth of tagged versions." Not only that, if any company 
> wants to have a longer timeframe supported, then they only need to
> offer disk space and bandwidth support to achieve this - or THEN run
> their own rsync mirror.
> Since gentoo is a source level distro, many of the hassles that
> RedHat, Mandrake etc have making RPMS for old releases dont apply.
> 
> I'm NOT implying developer support for 'old' ebuilds - but if package 
> 'xzy' used to be available with Gentoo in Aug2002, it should still be 
> available when you are building a system at that version level. This 
> would also mean that if someone really must have it supported - then 
> they can do it themselves and get it moved back into the main branch.
> If this 'tag' idea makes it attactive for companies to use Gentoo,
> then I would expect an explosion in the number of active (company
> paid) developers maintaining the ebuilds of little programs that the
> core team could not justify supporting.
> 
> 
> Another thing that running your own rsync mirror can never achieve is 
> third party vendor certification. Having defined 'tags' would allow 
> companies like Oracle to certify a particular version of Gentoo as
> being supported. Thats not currently possible. The very power and
> flexibility that Gentoo gives developers is also totally incompatible
> with the major software vendor QA techniques. Using a 'tag' would go a
> long way to overcoming this.
> It would become possible for people like SAP to certify their products
> 
> as supported with specific 'make.conf' settings for a specific tag. 
> Thats enough to make Gentoo fine for both the Quality Assured type
> large deployment and yet still retain brilliant upgrade and security
> patch capabilities.
> 
> Cheers
> Ron
> 
> >
> >Tbh, it's a hack, rather than a nice solid server/client
> >enterprise-ready Portage solution, but it's one that does work.
> >
> >Best regards,
> >Stu
> >  
> >
> 
> 
> 
> --
> gentoo-dev@gentoo.org mailing list
> 
> 



--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Large scale deployments - and portage

[Stuart Herbert]

[-- Attachment #1: signed data --]
[-- Type: text/plain, Size: 4423 bytes --]

Matt, Ron,

I agree with you both - to a point.  The current CVS strategy has gotten 
Gentoo this far, but it needs to evolve if Gentoo is to become more friendly 
to large companies.  It is being talked about amongst key Gentoo developers, 
but at the moment I don't know the current status, or any timescales.

One thing I'd like to see is the eradication of the word 'stable' from 
anything to do with Portage.  Right now, I don't see any clear or consistent 
understanding of what 'stable' means, and I don't think that the term is 
actually very helpful.  Does 'stable' mean fit for production use?  Does it 
simply mean 'the ebuild works, and the app doesn't segfault when you fire it 
up'?  It's not helpful.

If Gentoo moves to a multiple-tree strategy, -CURRENT is fair enough I guess, 
as are -RELEASE branches (although the whole idea of a RELEASE needs 
re-considering too I believe).  But what would -STABLE in between actually 
mean?  We need a better word.  Heck, even -UNMASKED would be a better start - 
at least it's more accurate, and something that we could reach a clear 
definition on.

I believe that ebuilds *should* have a guaranteed minimum life expectency in 
the Portage tree; I also believe that developers *should* announce the 
removal of an ebuild before it is removed, to give time for any necessary 
debate or local archiving into /usr/local/portage.  Right now, a) it doesn't 
happen, and b) there's nowhere suitable to make the announcement anyway.  You 
can find out more about my opinion on managing individual ebuilds by reading 
a draft document on dev.gentoo.org/~stuart/sponsorship/.  You'll need a PDF 
reader to read it.

Getting back to managing the larger Portage tree ... I don't agree that 
providing support for 'tagged branches' is just as simple as making the 
tagged branches available for x period of time.  A 'tagged branch' wouldn't 
be suitable for any large software provider to certify against - because it's 
an incomplete target.  A tagged branch is just a bunch of ebuilds.  It 
doesn't include the compiler you used, the CFLAGS you used, or the 
architecture you're running on.  And the moment a branch is tagged, it 
becomes a liability.

Let's look at a traditional SCM (software change management) problem for a 
moment.  What happens to the 'tagged branch' when security problems are 
found?  What about when important bug fixes are needed?  Would we backport 
those ebuilds into the tagged branches?  Where would that effort come from?  
Who would do that work?

Look at the alternative - withdraw a particular tagged branch, and tell users 
to upgrade to a different tagged branch.  Unfortunately, you're going to have 
an unpredictable amount of change between those branches.  The effort of 
moving could be high, and would be difficult to plan for in advance.  We'd 
end up like - shall we say - certain older distributions, where from time to 
time wiping the box is less hassle than performing the upgrade.

What does a certification program involve?  Any QA practice that is more than 
cosmetic would include SCM, and the SCM team would require that the exact 
versions of libraries and supporting utilities are identified and documented.  
Which is exactly how ebuilds work in practice.  You'd have to throw in 
environmental factors - hardware, file systems & disk layouts, etc etc - but 
the ebuild is at the heart of it, and not the wider distribution.  If all the 
technical factors are satisfied, then the QA practice would approve 
certification.  Although a tag is a convenient way of saying that all these 
factors *should* be present, Gentoo's ebuilds provide a viable alternative.

I really don't see how the Gentoo approach is incompatible with that QA 
practice.  In reality, certifications are a political tool as much as a 
technical one - and there Gentoo will face new challenges for sure.

Best regards,
Stu
-- 
Stuart Herbert                                              stuart@gentoo.org
Gentoo Developer                                       http://www.gentoo.org/
Beta packages for download            http://dev.gentoo.org/~stuart/packages/
Come and meet me in March 2004                 http://www.phparch.com/cruise/

GnuGP key id# F9AFC57C available from http://pgp.mit.edu
Key fingerprint = 31FB 50D4 1F88 E227 F319  C549 0C2F 80BA F9AF C57C
--

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Large scale deployments - and portage

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 1058 bytes --]

On Mon, Aug 18, 2003 at 08:09:03AM +1000, Ron O'Hara wrote:
> I may have missed something in all the docs, but as far as I can see
> there is what should be a fairly simple enhancement to portage that
> would help considerably in making Gentoo more practical in larger
> deployments.
I do it in the same way that I handle FreeBSD systems. I have a single
copy of the tree on a read-only NFS mount. I also have a minimal overlay
tree that is also on a read-only NFS mount with some custom packages. I
can manipulate that copy as I see fit, and it's already at all the
machines. 

I have 26 machines at work currently running Gentoo, 6 of those are
workstations and the rest servers. I'm actually going to be migrating
the workstations back to Redhat due to pressure from users, but the
servers are staying Gentoo.

-- 
Robin Hugh Johnson
E-Mail     : robbat2@orbis-terrarum.net
Home Page  : http://www.orbis-terrarum.net/?l=people.robbat2
ICQ#       : 30269588 or 41961639
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

[-- Attachment #2: Type: application/pgp-signature, Size: 232 bytes --]

Re: [gentoo-dev] Large scale deployments - and portage

[Don Seiler]

Yes I was going to suggest something similar.  We only have 6 desktop
users at work right now but we all share the same /usr/portage via NFS.
So all ebuilds and distfiles are from the same source.  It seems silly
to be re-downloading distfiles to every machine.

Don.

On Sun, Aug 17, 2003 at 04:17:02PM -0700, Robin H. Johnson wrote:
> On Mon, Aug 18, 2003 at 08:09:03AM +1000, Ron O'Hara wrote:
> > I may have missed something in all the docs, but as far as I can see
> > there is what should be a fairly simple enhancement to portage that
> > would help considerably in making Gentoo more practical in larger
> > deployments.
> I do it in the same way that I handle FreeBSD systems. I have a single
> copy of the tree on a read-only NFS mount. I also have a minimal overlay
> tree that is also on a read-only NFS mount with some custom packages. I
> can manipulate that copy as I see fit, and it's already at all the
> machines. 
> 
> I have 26 machines at work currently running Gentoo, 6 of those are
> workstations and the rest servers. I'm actually going to be migrating
> the workstations back to Redhat due to pressure from users, but the
> servers are staying Gentoo.
> 
> -- 
> Robin Hugh Johnson
> E-Mail     : robbat2@orbis-terrarum.net
> Home Page  : http://www.orbis-terrarum.net/?l=people.robbat2
> ICQ#       : 30269588 or 41961639
> GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85



--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Large scale deployments - and portage

[Paul de Vrieze]

>
> I have 26 machines at work currently running Gentoo, 6 of those are
> workstations and the rest servers. I'm actually going to be migrating
> the workstations back to Redhat due to pressure from users, but the
> servers are staying Gentoo.
>

Would you mind sharing with us what the users don't like about gentoo? If
we know it, we might be able to improve.

Paul

-- 
Paul de Vrieze
Researcher
Mail: pauldv@cs.kun.nl
Homepage: http://www.devrieze.net




--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Large scale deployments - and portage

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 740 bytes --]

On Mon, Aug 18, 2003 at 10:24:03AM +0200, Paul de Vrieze wrote:
> Would you mind sharing with us what the users don't like about gentoo?
> If we know it, we might be able to improve.
This is a research enviroment here, and a large number of users have
custom rpm packages with specialized binaries in them for various
research work, alas, no source available for many of them :-(.
There have been a lot less hassles installing them on a pure RPM
machine, instead of trying fufill the deps via portage.

-- 
Robin Hugh Johnson
E-Mail     : robbat2@orbis-terrarum.net
Home Page  : http://www.orbis-terrarum.net/?l=people.robbat2
ICQ#       : 30269588 or 41961639
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

[-- Attachment #2: Type: application/pgp-signature, Size: 232 bytes --]

Re: [gentoo-dev] Large scale deployments - and portage

[Ron O'Hara]

Hi Chris

Chris Smith wrote:
[chop]

>I think the real trick to it, is having a central server for you. There is a 
>great post on it in the forums somewhere. Having one box for 500 gentoo boxes 
>downloading all the distfiles, and doing all the rsyncs is a big bandwidth 
>saver!
>
>  
>
>>Now the only question is - did I miss something and this already exists?
>>    
>>
>
>Not really. Here is that forum post if you are interested:
>http://forums.gentoo.org/viewtopic.php?t=59134
>

Thanks for the link.  Ok I've read it and along with you, a few others 
have explained how they are using their own mirror of files (in one form 
or other) .. I am already assumming that that is the approach that every 
large site would setup.
My issue is more about version control and change control in a complex 
large deployment - the bandwidth issue is solved by the local mirror, 
but versioning is not.

I put a lot more about this in a reply to Stuart and Cc'd the list. In 
essence, big sites want things to operate at a known software level for 
all components. Gentoo gives brilliant support for deployment and 
upgrades, but does not directly support locking things down at a 
'specific release'. The local mirror is just a hack for that aspect (it 
is still needed for performance reasons)

One thing I didn't point out in the earlier mail, is another consequence 
of change control issues in large deployments. Many huge sites ALWAYS 
run multiple release levels of all sorts of things, for the simple 
reason that logistics prevent them ever having all systems running at 
the same level. Change windows are constrained and there are multiple 
teams of developers and admins working through changes all the time. 
Organized hell!!

Eg. BT in the Uk deploys a new Cisco router about every 30 minutes... 
every day of the year!!  Thats an operation with SCALE. By definition, 
their routers are never ALL running the same release of IOS.. they are 
always managing the operation of multiple versions in production - and 
multiple migration paths are being tested all the time. - Continuous 
organized hell!!
A similar situation exists for their Solaris deployments - multiple 
versions in production.

You also get the situation where one business application is only 
certified to run on version x.y.z with patches a+b+c of an operating 
system, and another (also essential) business application is only 
certifed to run on some other version with some other set of patches.

How do you support that using the current Gentoo? Unfortunately not as 
easily as you can with a distro like RedHat. But I suspect that if some 
form of 'tag' or 'label' becomes supported through the portage tree, 
then Gentoo will in fact offer the equivelant support in this area and 
have a superior ability in the upgrade/security fix area.


As for how this could be done... hmmmm.  I suspect putting the portage 
tree into CVS and then using something like 'cvsfs' 
(http://www.gnu.org/directory/sysadmin/vc/cvsfs.html) to present it to 
rsync would be good enough..



Thinking out loud::::

backward compatibility - current rsync updates would continue unchanged 
so 'migration' is a non-event.

New features: If you mirrored the CVS repository to your local miror (I 
think the cvs client can do that?), then your local mirror can offer 
different views of the portage tree at any of the labels it holds. 
Multiple concurrent versions could be presented under different 'host 
names' for the internal rsync process.
In fact you could use that to achieve the same thing for the global 
situation too.


Eg.

Assume I have set up a fake host called   'Aug2003.acme.com' as one way 
to access my local rsync mirror. In this host name, rsyncd runs chrooted 
to the CVS view of the files against the 'tag' for August-2003.  The 
same can happen for any other 'tag' to allow multiple externally 
accessible views of the portage tree via rsync against my local mirror.

All I need to do for my Aug2003 version gentoo machines is point them at 
that fake host name. The same goes for other versions.


Ok - where is that different to running multiple local mirrors?  It's 
different in one important way (and there is a saving on disk 
space/bandwidth too).
The primary difference is that the 'tags' are publicly defined by the 
Gentoo project. That means that external vendors can take a 'tag' and QA 
their software against it. Their customers can then install using the 
predefined 'tag' and use that vendors software ...

Plus if, the gentoo project choses to offer the same type of central 
repository, then a 'release' also implies the creation of an rsync 
hostname for accessing that release (and only  that release). It also 
provides a way to remove 'supported' access to that release by removing 
the hostname -- simple DNS changes.
For those companies that take a mirror - there is no need to remove the 
internal CVS labels so old 'releases' in  the portage tree can live for 
a long time after they are publicly accessible from Gentoo.


end of thinking through my hat!!

more ramblings from me.... but still - worth discussing to help bring 
Gentoo into the corporate realm



Regards
Ron O'Hara
http://www.openconsulting.us



--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Large scale deployments - and portage

[Paul de Vrieze]

Hi Ron,

Currently there is no support yet for what you want, except when you do it
yourself. But we are actively looking into providing for an "enterprise
gentoo". This version of gentoo would have real releases every say half
year (the terms have not been decided yet), further it would provide
security fixes and fixes for serious bugs. But it would not provide newer
versions of applications to provide more stability. Security fixes would
be backported if at all possible.

Expect work on this to become visible in the comming months.

For now, for enterprise systems you probably want to have a server that
maintains private snapshots. (It is easy to set the specific tree in the
make.conf file, and multiple trees can be offered on the server). You
would then need to monitor security problems, and merge the fixed ebuilds
in the snapshot trees. This indeed is some work, but I believe in most
enterprise enviroments such an update would involve some testing too.

For syncing the server trees, please take a look at the rsync mirror
documentation. In those environments it is most often more applicable to
use rsync directly instead of through portage. That also allows for custom
locations. You might also want to exclude parts of the tree like games.

Paul

-- 
Paul de Vrieze
Gentoo developer
Mail: pauldv@cs.kun.nl
Homepage: http://www.devrieze.net




--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Large scale deployments - and portage

[Brian Jackson]

You could pin packages in the world file (/var/cache/edb/world).

<apache-2
<courier-0.42.3
<mysql-4.0.5

etc. etc. Then copy that file to each server, and tada, you've got a standard 
profile type thing.

--Brian Jackson



On Sunday 17 August 2003 05:09 pm, Ron O'Hara wrote:
> Hi,
>
> I may have missed something in all the docs, but as far as I can see
> there is what should be a fairly simple enhancement to portage that
> would help considerably in making Gentoo more practical in larger
> deployments.
>
> I came to gentoo to get the advantages of being able to tailor my
> system(s) and still have a simple, consistent upgrade path for all the
> component software - portage is great for that. From the developer and
> tinkerer point of view it is hard to see improvements in the overall
> strategy behind portage.
>
> BUT - for larger deployments, there are extra needs which are not
> currently handled by portage - but should be a snap to implement.
>
> In large deployments, it is critical to maintain strong change control
> and have predictable environments that you can validate application
> performance in. Many places use a strategy like:  "All production
> servers are RedHat 7.1 (or Solaris 2.7, or Windows 2000 +SP5 ... etc)"
>
> At present I cant see a way to easily do that with gentoo. Eg. Gentoo
> 1.4 final is really just a starting point - the minute you do an 'emerge
> sync', you have an unknown mix of software. And worse, if you do it an
> hour later on a different machine, it could be a slight different mix
> again - different from the first box.
> It is relatively easy to compare the portage trees and work out what the
> differences are, but thats not much help
>
> If it were possible to 'tag' the portage tree with labels at regular
> intervals, and and do an 'emerge sync' with a nominated 'tag' - then you
> would have the equivelant of the fixed points that other distributions
> have when they cut a release CD.
>
>
>
>
> As an example scenario for someone building deployment images in a huge
> telco.
>
> After some experimentation, they decide that their basic gentoo system
> will be 'gentoo with rsync tag gentoo-1.4-Aug-2003'
>
> They take the first box and do a standard gentoo install, but at the
> 'emerge sync' point in the instructions, they do 'emerge sync
> tag=gentoo-1.4-Aug-2003'.
> They then complete all the package installations and tweaking that they
> want - site specific choices.
>
>
> This is a repeatable process!! - critical for duplicating an environment.
>
> As each of the systems is deployed - they are in a known state. Even
> nicer is that deployed systems can be easily upgraded to some other
> 'known environment' by doing an 'emerge sync tag=xxxx' to some later tag.
>
> For 'hotfixes', deployed boxes probably need the normal 'emerge sync'
> which will give them acvcess to the latest and greatest bug fix - but
> you would probably not do an 'emerge -u world' on these boxes unless
> moving it from one 'tag level' to another.
>
> Now the only question is - did I miss something and this already exists?
>
> Regards
> Ron O'Hara
>
>
>
>
>
>
>
>
> .
>
>
> --
> gentoo-dev@gentoo.org mailing list

-- 
Home -- http://www.brianandsara.net
Gentoo -- http://gentoo.brianandsara.net

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Large scale deployments - and portage

[Stuart Bouyer]

[-- Attachment #1: Type: text/plain, Size: 838 bytes --]

On Sun, 17 Aug 2003 19:42:30 -0500
Brian Jackson <iggy@gentoo.org> wrote:

> You could pin packages in the world file (/var/cache/edb/world).
> 
> <apache-2
> <courier-0.42.3
> <mysql-4.0.5
> 
> etc. etc. Then copy that file to each server, and tada, you've got a
> standard profile type thing.
> 
> --Brian Jackson
> 

That works fine UNTIL a package is removed from the portage tree, then
you have to go and try to get it (and the various others that went with
it) back using the cvsweb interface - lots of fun I can tell you.

There was some mention a month or so ago about tagging the cvs tree, but
it seems to have been lost in the hype over 1.4 release. Maybe someday??

StuBear

-- 
GnuPG KeyID 1607E7F7
Key fingerprint = 5C38 AA94 A4C1 6AAF 0EE4  C089 EE01 193D 1607 E7F7
gpg --keyserver search.keyserver.net --recv-keys 1607E7F7

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]