[gentoo-user] How to update portage offline with minimal impact?

[gentoo-user] How to update portage offline with minimal impact?

[BRM]

I have a couple Sparc systems. One has been running Gentoo for a long
time - installed using Gentoo 2006, not updated since due to the issue
I'm about the discuss - and the other is a near identical system that
might get Gentoo 2007 installed. Both are on two separate networks and
have no communication between them.

The first system does have some Internet access through a firewall, but
it doesn't really work, at least for this purpose; so it's just as good
as not having any access at all for this purpose.

The second system may or may not have Internet access, so for now let's
just assume it doesn't. It's really this second system that I want to
figure the problem out for.

In either case, I can't update portage using the normal method of
'emerge --sync'. So, I'm trying to figure out a solution that would
enable me to update the systems. Under Slackware, I'd just point
pkgtool to the CD media and install from that, just like during
installation. Is there a similar approach for Gentoo? How do I overcome
the source mirror issue too so that the systems don't try to download
stuff from the web?

I could probably host an rsync server on the local systems to host
portage, but how would I keep it up to date? Would I simply be able to
extract a tarball into the directory rsync is serving up?

I can get large tarballs or ISOs from other systems to these systems;
but they won't be able to download them themselves.

I noticed the Manual Download info in the FAQ:
http://www.gentoo.org/doc/en/faq.xml#manualdownload

However, that is not a solution I can use as I might not be the long
term maintainer, and I'd like an easier solution as it requires a lot
of work to download stuff. I'd like a solution similar to the
following:

# tar xvjf /portage-sources-<data>.tar.bz2 -C /my-portage-sources
# tar xvjf /portage-<date>.tar.bz2 -C /my-portage
# emerge --sync --portage-source /my-portage
# emerge world -vuD --sources /my-portage-sources

If there isn't a solution, I might look into how to make a solution
(not sure).

TIA,

Ben
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] How to update portage offline with minimal impact?

[Iain Buchanan]

On Thu, 2008-01-03 at 13:16 -0800, BRM wrote:

[snip]

> However, that is not a solution I can use as I might not be the long
> term maintainer, and I'd like an easier solution as it requires a lot
> of work to download stuff. I'd like a solution similar to the
> following:
> 
> # tar xvjf /portage-sources-<data>.tar.bz2 -C /my-portage-sources
> # tar xvjf /portage-<date>.tar.bz2 -C /my-portage
> # emerge --sync --portage-source /my-portage
> # emerge world -vuD --sources /my-portage-sources

This has come up before, so I know some people here have a bit of
experience with doing it.  Essentially it's possible.  I think the steps
required are:

- download a portage snapshot as you would in an initial install, or
create your own from another gentoo machine (more info here:
http://www.gentoo.org/doc/en/handbook/handbook-x86.xml?part=1&chap=5#installing_portage)
- you may need to run `emerge --metadata` after unpacking it, or
something like that - someone else can comment here :)
- then follow your manual download instructions: `emerge -ufpDN world`
and download all these packages on another box
- transfer the packages to /usr/portage/distfiles, and check you got
them all with `emerge -ufDN world`.  You should have no fetch errors.
- then go! `emerge -uvaDN world`

Notes:
- when creating your own snapshot,
exclude /usr/portage/distfiles, /usr/portage/packages, (and others?)
- I would recommend -N as you might see some new features since you
haven't updated in a while

Hopefully someone will fix any holes I left, otherwise this should work!
-- 
Iain Buchanan <iaindb at netspace dot net dot au>

There's a lot to be said for not saying a lot.

-- 
gentoo-user@lists.gentoo.org mailing list

Re: [gentoo-user] How to update portage offline with minimal impact?

[BRM]

--- Iain Buchanan <iaindb@netspace.net.au> wrote:
> On Thu, 2008-01-03 at 13:16 -0800, BRM wrote:
> [snip]
> > However, that is not a solution I can use as I might not be the
> long
> > term maintainer, and I'd like an easier solution as it requires a
> lot
> > of work to download stuff. I'd like a solution similar to the
> > following:
> > # tar xvjf /portage-sources-<data>.tar.bz2 -C /my-portage-sources
> > # tar xvjf /portage-<date>.tar.bz2 -C /my-portage
> > # emerge --sync --portage-source /my-portage
> > # emerge world -vuD --sources /my-portage-sources
> This has come up before, so I know some people here have a bit of
> experience with doing it.  Essentially it's possible.  I think the
> steps
> required are:
> 
> - download a portage snapshot as you would in an initial install, or
> create your own from another gentoo machine (more info here:
>
http://www.gentoo.org/doc/en/handbook/handbook-x86.xml?part=1&chap=5#installing_portage)
> - you may need to run `emerge --metadata` after unpacking it, or
> something like that - someone else can comment here :)
> - then follow your manual download instructions: `emerge -ufpDN
> world`
> and download all these packages on another box
> - transfer the packages to /usr/portage/distfiles, and check you got
> them all with `emerge -ufDN world`.  You should have no fetch errors.
> - then go! `emerge -uvaDN world`
> 
> Notes:
> - when creating your own snapshot,
> exclude /usr/portage/distfiles, /usr/portage/packages, (and others?)
> - I would recommend -N as you might see some new features since you
> haven't updated in a while
> 
> Hopefully someone will fix any holes I left, otherwise this should
> work!

Thanks. I'll have to see about giving it a try; not sure how well it'll
work on the one system; and I'll have to run it past my other admins
for the other system. Any how...

Thanks.

Ben
-- 
gentoo-user@lists.gentoo.org mailing list

Re: [gentoo-user] How to update portage offline with minimal impact?

[Per-Erik Westerberg]

tor 2008-01-03 klockan 13:16 -0800 skrev BRM:
> I have a couple Sparc systems. One has been running Gentoo for a long
> time - installed using Gentoo 2006, not updated since due to the issue
> I'm about the discuss - and the other is a near identical system that
> might get Gentoo 2007 installed. Both are on two separate networks and
> have no communication between them.
> 
> The first system does have some Internet access through a firewall, but
> it doesn't really work, at least for this purpose; so it's just as good
> as not having any access at all for this purpose.
> 
> The second system may or may not have Internet access, so for now let's
> just assume it doesn't. It's really this second system that I want to
> figure the problem out for.
> 
> In either case, I can't update portage using the normal method of
> 'emerge --sync'. So, I'm trying to figure out a solution that would
> enable me to update the systems. Under Slackware, I'd just point
> pkgtool to the CD media and install from that, just like during
> installation. Is there a similar approach for Gentoo? How do I overcome
> the source mirror issue too so that the systems don't try to download
> stuff from the web?
> 
> I could probably host an rsync server on the local systems to host
> portage, but how would I keep it up to date? Would I simply be able to
> extract a tarball into the directory rsync is serving up?
> 
> I can get large tarballs or ISOs from other systems to these systems;
> but they won't be able to download them themselves.
> 
> I noticed the Manual Download info in the FAQ:
> http://www.gentoo.org/doc/en/faq.xml#manualdownload
> 
> However, that is not a solution I can use as I might not be the long
> term maintainer, and I'd like an easier solution as it requires a lot
> of work to download stuff. I'd like a solution similar to the
> following:
> 
> # tar xvjf /portage-sources-<data>.tar.bz2 -C /my-portage-sources
> # tar xvjf /portage-<date>.tar.bz2 -C /my-portage
> # emerge --sync --portage-source /my-portage
> # emerge world -vuD --sources /my-portage-sources
> 
> If there isn't a solution, I might look into how to make a solution
> (not sure).
> 
> TIA,
> 
> Ben

Hi,

Have you tried to use a proxy (adjust accordingly)?

export http_proxy=http://proxy.company.com:8080
export ftp_proxy=http://proxy.company.com:8080
export RSYNC_PROXY=proxy.company.com:8080

  BR / P-E

-- 
gentoo-user@lists.gentoo.org mailing list

Re: [gentoo-user] How to update portage offline with minimal impact?

[BRM]

--- Per-Erik Westerberg <per-erik.westerberg@bredband.net> wrote:
> tor 2008-01-03 klockan 13:16 -0800 skrev BRM:
> > I have a couple Sparc systems. One has been running Gentoo for a
> long
> > time - installed using Gentoo 2006, not updated since due to the
> issue
> > I'm about the discuss - and the other is a near identical system
> that
> > might get Gentoo 2007 installed. Both are on two separate networks
> and
> > have no communication between them.
> > 
> > The first system does have some Internet access through a firewall,
> but
> > it doesn't really work, at least for this purpose; so it's just as
> good
> > as not having any access at all for this purpose.
<snip>
> > In either case, I can't update portage using the normal method of
> > 'emerge --sync'. So, I'm trying to figure out a solution that would
> > enable me to update the systems. Under Slackware, I'd just point
> > pkgtool to the CD media and install from that, just like during
> > installation. Is there a similar approach for Gentoo? How do I
> overcome
> > the source mirror issue too so that the systems don't try to
> download
> > stuff from the web?
> > 
> Have you tried to use a proxy (adjust accordingly)?
> export http_proxy=http://proxy.company.com:8080
> export ftp_proxy=http://proxy.company.com:8080
> export RSYNC_PROXY=proxy.company.com:8080

Yes, I tried using the proxy on the one system. (The other system won't
even have that as an option.) The problem came there that the proxy is
an authenticated proxy, primarily designed to work with Windows. It
works fine from Firefox/Netscape in X Windows, but causes problems for
command-line tools and console browsers. So, in addition to my trying
to find a solution where a proxy is not an option, it is, for all
intents and purposes, a non-option any way.

Additionally, because it is an authenticated proxy, it is not an ideal
solution as it would leave the username/password for a user in plain
site of all users on the system as the info would be either in the
environment variables and/or the command-line options of a program. So,
from a security stand-point, it's not an option either since it
sometimes takes a day or so to perform updates.

TIA,

Ben
-- 
gentoo-user@lists.gentoo.org mailing list

Re: [gentoo-user] How to update portage offline with minimal impact?

[Daniel da Veiga]

On Jan 8, 2008 7:13 PM, BRM <bm_witness@yahoo.com> wrote:
> --- Per-Erik Westerberg <per-erik.westerberg@bredband.net> wrote:
> > tor 2008-01-03 klockan 13:16 -0800 skrev BRM:
> > > I have a couple Sparc systems. One has been running Gentoo for a
> > long
> > > time - installed using Gentoo 2006, not updated since due to the
> > issue
> > > I'm about the discuss - and the other is a near identical system
> > that
> > > might get Gentoo 2007 installed. Both are on two separate networks
> > and
> > > have no communication between them.
> > >
> > > The first system does have some Internet access through a firewall,
> > but
> > > it doesn't really work, at least for this purpose; so it's just as
> > good
> > > as not having any access at all for this purpose.
> <snip>
> > > In either case, I can't update portage using the normal method of
> > > 'emerge --sync'. So, I'm trying to figure out a solution that would
> > > enable me to update the systems. Under Slackware, I'd just point
> > > pkgtool to the CD media and install from that, just like during
> > > installation. Is there a similar approach for Gentoo? How do I
> > overcome
> > > the source mirror issue too so that the systems don't try to
> > download
> > > stuff from the web?
> > >
> > Have you tried to use a proxy (adjust accordingly)?
> > export http_proxy=http://proxy.company.com:8080
> > export ftp_proxy=http://proxy.company.com:8080
> > export RSYNC_PROXY=proxy.company.com:8080
>
> Yes, I tried using the proxy on the one system. (The other system won't
> even have that as an option.) The problem came there that the proxy is
> an authenticated proxy, primarily designed to work with Windows. It
> works fine from Firefox/Netscape in X Windows, but causes problems for
> command-line tools and console browsers. So, in addition to my trying
> to find a solution where a proxy is not an option, it is, for all
> intents and purposes, a non-option any way.
>

If you really don't wanna use the network, you can easily transfer a
tarball and rsync locally (gentoo forums have little nifty scripts for
syncing locally and emerging metadata). The foruns also have lots of
scripts designed to create a list of needed distfiles and download
them at another machine, you can transfer this and update. With a
little set of scripts you can automate the whole process using the
network, or require minor user intervention to transfer the list and
later the files to and from a networkless machine.

> Additionally, because it is an authenticated proxy, it is not an ideal
> solution as it would leave the username/password for a user in plain
> site of all users on the system as the info would be either in the
> environment variables and/or the command-line options of a program. So,
> from a security stand-point, it's not an option either since it
> sometimes takes a day or so to perform updates.
>

There's no problem in using an authenticated proxy for
emerge-webrsync, as you can keep a script in a directory with
restricted permissions, only root would be able to see it anyway, and
you can use this machine as an rsync and distfiles mirror for any
other in the network, crontab would work as well, as only the user who
creates it can see it (if you set it). You can even set a special
username/password at your proxy that can only access rsync port and
mirrors for distfiles for increased security.

OK, those are some of MANY options available. Gentoo is very flexible,
even in a controlled environment.

-- 
Daniel da Veiga

Filosofia de TI: Programadores de verdade consideram o conceito "o que
você vê é o que você tem" tão ruim em editores de texto quanto em
mulheres. Não, o programador de verdade quer um editor de texto do
estilo "você pediu, você levou" - complicado, indecifrável, poderoso,
impiedoso, perigoso.

Re: [gentoo-user] How to update portage offline with minimal impact?

[BRM]

--- Daniel da Veiga <danieldaveiga@gmail.com> wrote:
> On Jan 8, 2008 7:13 PM, BRM <bm_witness@yahoo.com> wrote:
> > --- Per-Erik Westerberg <per-erik.westerberg@bredband.net> wrote:
> > > tor 2008-01-03 klockan 13:16 -0800 skrev BRM:
> > > > I have a couple Sparc systems. One has been running Gentoo for
> a
> > > long
> > > > time - installed using Gentoo 2006, not updated since due to
> the
> > > issue
> > > > I'm about the discuss - and the other is a near identical
> system
> > > that
> > > > might get Gentoo 2007 installed. Both are on two separate
> networks
> > > and
> > > > have no communication between them.
> > > >
> > > > The first system does have some Internet access through a
> firewall,
> > > but
> > > > it doesn't really work, at least for this purpose; so it's just
> as
> > > good
> > > > as not having any access at all for this purpose.
> > <snip>
> > > > In either case, I can't update portage using the normal method
> of
> > > > 'emerge --sync'. So, I'm trying to figure out a solution that
> would
> > > > enable me to update the systems. Under Slackware, I'd just
> point
> > > > pkgtool to the CD media and install from that, just like during
> > > > installation. Is there a similar approach for Gentoo? How do I
> > > overcome
> > > > the source mirror issue too so that the systems don't try to
> > > download
> > > > stuff from the web?
> > > >
> > > Have you tried to use a proxy (adjust accordingly)?
> > > export http_proxy=http://proxy.company.com:8080
> > > export ftp_proxy=http://proxy.company.com:8080
> > > export RSYNC_PROXY=proxy.company.com:8080
> >
> > Yes, I tried using the proxy on the one system. (The other system
> won't
> > even have that as an option.) The problem came there that the proxy
> is
> > an authenticated proxy, primarily designed to work with Windows. It
> > works fine from Firefox/Netscape in X Windows, but causes problems
> for
> > command-line tools and console browsers. So, in addition to my
> trying
> > to find a solution where a proxy is not an option, it is, for all
> > intents and purposes, a non-option any way.
> >
> If you really don't wanna use the network, you can easily transfer a
> tarball and rsync locally (gentoo forums have little nifty scripts
> for
> syncing locally and emerging metadata). The foruns also have lots of
> scripts designed to create a list of needed distfiles and download
> them at another machine, you can transfer this and update. With a
> little set of scripts you can automate the whole process using the
> network, or require minor user intervention to transfer the list and
> later the files to and from a networkless machine.

Any that you recommend? This sounds like what I want.
 
> > Additionally, because it is an authenticated proxy, it is not an
> ideal
> > solution as it would leave the username/password for a user in
> plain
> > site of all users on the system as the info would be either in the
> > environment variables and/or the command-line options of a program.
> So,
> > from a security stand-point, it's not an option either since it
> > sometimes takes a day or so to perform updates.
> There's no problem in using an authenticated proxy for
> emerge-webrsync, as you can keep a script in a directory with
> restricted permissions, only root would be able to see it anyway, and
> you can use this machine as an rsync and distfiles mirror for any
> other in the network, crontab would work as well, as only the user
> who
> creates it can see it (if you set it). You can even set a special
> username/password at your proxy that can only access rsync port and
> mirrors for distfiles for increased security.
> 
> OK, those are some of MANY options available. Gentoo is very
> flexible,
> even in a controlled environment.

True - gentoo is very flexible, and its emerging management is why I
chose it for the first system behind the proxy. When I had originally
set up the system, the proxies weren't authenticated and things worked.
Unfortunately, I don't have any control of the proxies and the only
thing I can do is use my own username and password - thus putting some
personal liability on the line as the company would hold me
responsible.  I am aware I can do a restricted script - but I still end
up with the problem (which is documented) that someone could possibly
sniff the environment of the script and get the username/password, or
sniff the program names - as listed by 'ps' and other sources (e.g. the
kernel) - and get it there too, depending on how ftp/wget/etc. are
called.

Unfortunately, the system behind the proxy may have other issues.
Apparently some of the primary software for the system (Apache,
Subversion, Trac) didn't ever get emerged. I know I can list it as
already provided, but that would cause a problem with updating that
software via emerging, no? (Which is what I really want!) So, the
system may need a complete rebuild to do it right, and I'm not sure how
I would be able to do that at the moment for a number of reasons beyond
the scope of my problem here. So that system will likely sit as it is
for a long time to come...

Any how...I still have another system that has not yet been setup that
I need to figure this out for - and that one won't likely have Internet
access at all, so the proxy issue doesn't matter.

Thanks!

Ben
-- 
gentoo-user@lists.gentoo.org mailing list