[gentoo-user] rsync internal mirror configuration

[gentoo-user] rsync internal mirror configuration

[James]

Hello,

I just set up a rsync internal mirror following:
//gentoo-wiki.com/HOWTO_Local_Rsync_Mirror

I did not see what determines what (during a 24 hour
period) controls the time the internal rsync mirror
goes out to  update the files? Obviously I only
want the rsync internal server to update once a day.


Any other documents that I missed on setting up
an internal rsync server?

What the amount of disk space I can expect this 
new internal rsync mirror to use?

James



-- 
gentoo-user@gentoo.org mailing list

RE: [gentoo-user] rsync internal mirror configuration

[Dave Nebinger]

> I did not see what determines what (during a 24 hour
> period) controls the time the internal rsync mirror
> goes out to  update the files? Obviously I only
> want the rsync internal server to update once a day.

Following that document means that you are exporting the /usr/portage
directory on the local box for your internal servers.

This has some implications:

1. the /usr/portage directory is not updated by this process.  You still
need to emerge --sync the local system.  This is QED by a cron task.

2. This only syncs up the /usr/portage tree but does not help reduce the
number of distfiles downloads.  Look at http://gentoo-wiki.com/Http_proxy
and http://gentoo-wiki.com/HOWTO_Download_Cache_for_LAN-Http-Replicator for
instructions there.  By using the http-replicator you only will be
downloading distfiles/patch files once.

3. There is no additional disk space requirements for the rsync mirror;
you're serving up your existing /usr/portage directory so it's not like you
wouldn't have those files around anyway.



-- 
gentoo-user@gentoo.org mailing list

[gentoo-user] Re: rsync internal mirror configuration

[James]

Dave Nebinger <dnebinger <at> joat.com> writes:


> This has some implications:

Ok, let's take it from the begining.

A. I followed step 4 in this document:

http://www.gentoo.org/doc/en/rsync.xml?style=printable

I was able to 'emerge sync' up a local client on the rsync server 
successfully,so that part is done. Here are my client entries in 
/etc/make.conf:

GENTOO_MIRRORS="192.168.2.9"
SYNC="rsync://192.168.2.9/gentoo-portage"

Test, per this document was successful, So now I can manually rsync
 up one system and then emerge sync all of the local gentoo 
clients manuallyor via cron (with crontab entries).

> 1. the /usr/portage directory is not updated by this process. 
> You still need to emerge --sync the local system.  This is QED 
> by a cron task.

OK on the local rsync server I added this to automate the daily 
task of rsync(ing)
# Rsync entries
#
30 1 * * * root emerge sync

Spartan but hopefully sufficent.

I have (3) other documents to follow (this should be one doc)
B) http://gentoo-wiki.com/HOWTO_Local_Rsync_Mirror.
C) http://gentoo-wiki.com/Http_proxy
D) 
http://gentoo-wiki.com/HOWTO_Download_Cache_for_LAN-Http-Replicator

(B) tells me how to edit the /etc/rsyncd.conf file and the
/etc/rsync/rsyncd.motd file. Also  I did these:
etc/init.d/rsyncd start
rc-update add rsyncd default

This mod was already made to each client's /etc/make.conf file:
SYNC="rsync://192.168.2.9/gentoo-portage

> 2. This only syncs up the /usr/portage tree but does not help reduce 
> the number of distfiles downloads. 
> Look at http://gentoo-wiki.com/Http_proxy

OK, now I use document (C) to create the proxy entry on each client:
Editing File: /etc/env.d/99local to look like this 
http_proxy="192.168.2.9:8080" 
instead of this  
http_proxy="proxy.server.com:8080"

I check it on the client by issuing "echo $http_proxy"
which give the correct results:
192.168.2.9:8080

So far so good???

> and http://gentoo-wiki.com/HOWTO_Download_Cache_for_
> LAN-Http-Replicator for
> instructions there.  By using the http-replicator you only will be
> downloading distfiles/patch files once.

OK here's where I use doc (D). http-replicator is masked so I 
# echo "net-proxy/http-replicator" >> /etc/portage/package.keywords
# emerge http-replicator
Which worked fine.

Then I added this line to the clients and the rsync(distfile) server:
http_proxy="http://192.168.2.9:8080"  per the doc (D).

On the server I ran:
'repcacheman --user portage --dir /var/cache/http-replicator' 
Then http-replicator start at boot:
rc-update add http-replicator default


and last run this daily on the server to keep it current?
emerge -uDva world && repcacheman

Anything I missed? The last part of doc (D) is a little fuzzy, 
especially the part about deleting duplicate and deleting the 
distfile directory.

???
Surely parts of all (4) documents belong in one master howto?


> 3. There is no additional disk space requirements for the rsync mirror;
> you're serving up your existing /usr/portage directory so it's not like you
> wouldn't have those files around anyway.
> 

Yea, well I'll keep a close eye on this. I have the server set up, and one
client to update manually off of the server, It all goes well, I'll add more
systems, and automate via cron the system that stay up 24 hours a day.

Is there a script to update workstation systems, automatically,
 say 10 minutes after booting? How best to do this?

James


-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Re: rsync internal mirror configuration

[A. Khattri]

On Tue, 12 Jul 2005, James wrote:

> Is there a script to update workstation systems, automatically,
>  say 10 minutes after booting? How best to do this?

I run this in my servers at 3am (they sync off a local rsync server,
similar to your setup):

http://www.panhorst.com/glcu/


-- 
Aj.
-- 
gentoo-user@gentoo.org mailing list

RE: [gentoo-user] Re: rsync internal mirror configuration

[Dave Nebinger]

> OK on the local rsync server I added this to automate the daily
> task of rsync(ing)
> # Rsync entries
> #
> 30 1 * * * root emerge sync

Unless you want to receive the daily email full of all kinds of funky
characters, I'd redirect the output from emerge to a file.  On my boxen I
call 'dailysync.sh' from cron.  The script has this:

cornholio ~ # cat bin/dailysync.sh
#!/bin/sh
#
# sync.sh: Script to handle the portage syncing.
#

emerge --sync 1>>/var/log/autosync.log 2>&1

# Update eix and edb...
if [ -e /usr/portage/profiles/default-linux/x86/2005.0/make.defaults ]
then
  echo Not recreating make.defaults, it exists... >>/var/log/autosync.log
else
  echo Recreating make.defaults... >>/var/log/autosync.log
  ln -sf ../make.defaults
/usr/portage/profiles/default-linux/x86/2005.0/make.defaults
1>>/var/log/autosync.log 2>&1
fi

eix -u 1>>/var/log/autosync.log 2>&1
eupdatedb 1>>/var/log/autosync.log 2>&1

cornholio ~ #

The if/then logic is needed because eix expects a <version>/make.defaults
file, but the make.defaults has been pushed up one directory level.  The
emerge --sync wipes the file out so I have to recreate it before updating
eix.  And the output from the script goes to a log file rather than
stdout/stderr (so when run via cron no email is generated).

> OK, now I use document (C) to create the proxy entry on each client:
> Editing File: /etc/env.d/99local to look like this
> http_proxy="192.168.2.9:8080"
> instead of this
> http_proxy="proxy.server.com:8080"
> 
> I check it on the client by issuing "echo $http_proxy"
> which give the correct results:
> 192.168.2.9:8080
> 
> So far so good???

Hmm, the documentation might have changed since I did my setup.  My internal
boxes have 'http_proxy=http://192.168.0.1:8084' directly in the make.conf
file; I didn't make any changes to /etc/conf.d/local.start for the http
proxy.

> and last run this daily on the server to keep it current?
> emerge -uDva world && repcacheman

Well if you have the time and inclination you can do it daily.  Once you get
into maintaining the system you'll realize that this kind of attention to
the box is frequently overkill - a working box is a working box and there is
no need to fix what isn't broken.

Rather than do this on a daily basis I generate a daily email of packages to
be updated; if I see one I feel is critical I'll deal with the update
otherwise I'll let the system go for awhile.  The script I use for this is:

cornholio ~ # cat bin/updatereport.sh
#!/bin/sh
#
# updatereport.sh: Script to send an update report to root.
#

#
# Remove the old report file.
#
if [ -e /var/log/update.report ] ; then
  /bin/rm -f /var/log/update.report
fi
if [ -e /var/log/update.rpt.txt ] ; then
  /bin/rm -f /var/log/update.rpt.txt
fi

#
# Run the command to generate the output file.
#
emerge --pretend --update --deep world 1>/var/log/update.report

#
# Put a standard header at the top of the text file.
#
echo "Current gentoo update report." > /var/log/update.rpt.txt
echo "" >> /var/log/update.rpt.txt
date >> /var/log/update.rpt.txt
echo "" >> /var/log/update.rpt.txt

#
# Extract a clean text file
#
strings /var/log/update.report >> /var/log/update.rpt.txt

#
# Mail the clean text file.
#
mail -s "Cornholio Portage Update Report" root < /var/log/update.rpt.txt

cornholio ~ #

> Anything I missed?

Well sure.  The part I should have mentioned is that if your systems are all
similar in that they have the same hardware and same use flags, package
installs, etc., then you might want to have one system build binary packages
for distribution to the internal systems.  That way you only suffer the
build once.

Personnaly I've got different architectures (PIII, P4, and AMD) mixed in
and, as each system has a different purpose they typically don't have
similar use flags or package installations, so a binary package distribution
doesn't work for me.

> Surely parts of all (4) documents belong in one master howto?

I'm sure it could but there's probably folks on the other side of the fence
that only want a local rsync mirror or a local portage download cache and
not both.
 
> Is there a script to update workstation systems, automatically,
>  say 10 minutes after booting? How best to do this?

Sure.  Generate a script that sleeps for 10 minutes then kicks off the
emerge process.  Call the script from /etc/conf.d/local.start (be sure to
include the & to spawn it in the background) and you're golden.

However, you need to consider if this is something that you really want to
do.  Automated updates are frowned upon by folks on the list.

I tend to stagger my system emerges because I use distcc.  The individual
boxes have a smaller time impact handling the package recompiles.



-- 
gentoo-user@gentoo.org mailing list

[gentoo-user] Re: rsync internal mirror configuration

[James]

Dave Nebinger <dnebinger <at> joat.com> writes:


> > OK on the local rsync server I added this to automate the daily

> > 30 1 * * * root emerge sync

> Unless you want to receive the daily email full of all kinds of funky
> characters, I'd redirect the output from emerge to a file.  On my boxen I
> call 'dailysync.sh' from cron.  The script has this:

OK but can't I do that with a simple mode to the /etc/crontab ?:
30 1 * * * root emerge sync 2>&1
?

> cornholio ~ # cat bin/dailysync.sh
> #!/bin/sh
> #
> # sync.sh: Script to handle the portage syncing.

> emerge --sync 1>>/var/log/autosync.log 2>&1

> # Update eix and edb...
> if [ -e /usr/portage/profiles/default-linux/x86/2005.0/make.defaults ]
> then
>   echo Not recreating make.defaults, it exists... >>/var/log/autosync.log
> else
>   echo Recreating make.defaults... >>/var/log/autosync.log
>   ln -sf ../make.defaults
> /usr/portage/profiles/default-linux/x86/2005.0/make.defaults
> 1>>/var/log/autosync.log 2>&1
> fi

> eix -u 1>>/var/log/autosync.log 2>&1
> eupdatedb 1>>/var/log/autosync.log 2>&1

OK, I can try this script. I guess eix is an ebuild index
& searching tool, that I need to read up on...

This script is for the internal rsync/replicator server?


> The if/then logic is needed because eix expects a <version>/make.defaults
> file, but the make.defaults has been pushed up one directory level.  The
> emerge --sync wipes the file out so I have to recreate it before updating
> eix.  And the output from the script goes to a log file rather than
> stdout/stderr (so when run via cron no email is generated).

> > OK, now I use document (C) to create the proxy entry on each client:
> > Editing File: /etc/env.d/99local to look like this
> > http_proxy="192.168.2.9:8080"
> > instead of this
> > http_proxy="proxy.server.com:8080"

> > I check it on the client by issuing "echo $http_proxy"
> > which give the correct results:
> > 192.168.2.9:8080

> > So far so good???

> Hmm, the documentation might have changed since I did my setup.  My internal
> boxes have 'http_proxy=http://192.168.0.1:8084' directly in the make.conf
> file; I didn't make any changes to /etc/conf.d/local.start for the http
> proxy.

OK, the port has changed. What's the deal about that?

> > and last run this daily on the server to keep it current?
> > emerge -uDva world && repcacheman

> Well if you have the time and inclination you can do it daily.  Once you get
> into maintaining the system you'll realize that this kind of attention to
> the box is frequently overkill - a working box is a working box and there is
> no need to fix what isn't broken.

> Rather than do this on a daily basis I generate a daily email of packages to
> be updated; if I see one I feel is critical I'll deal with the update
> otherwise I'll let the system go for awhile.  The script I use for this is:

OK, monthly on a manual basis. I'll try your script/email approach.

> cornholio ~ # cat bin/updatereport.sh
> #!/bin/sh

> # updatereport.sh: Script to send an update report to root.

> # Remove the old report file.

> if [ -e /var/log/update.report ] ; then
>   /bin/rm -f /var/log/update.report
> fi
> if [ -e /var/log/update.rpt.txt ] ; then
>   /bin/rm -f /var/log/update.rpt.txt
> fi

> # Run the command to generate the output file.

> emerge --pretend --update --deep world 1>/var/log/update.report

> # Put a standard header at the top of the text file.

> echo "Current gentoo update report." > /var/log/update.rpt.txt
> echo "" >> /var/log/update.rpt.txt
> date >> /var/log/update.rpt.txt
> echo "" >> /var/log/update.rpt.txt

> # Extract a clean text file

> strings /var/log/update.report >> /var/log/update.rpt.txt

> # Mail the clean text file.

> mail -s "Cornholio Portage Update Report" root < /var/log/update.rpt.txt


OK thanks for the script. I'll have to study it, test it and let you know
how it goes...


> > Anything I missed?

> Well sure.  The part I should have mentioned is that if your systems are all
> similar in that they have the same hardware and same use flags, package
> installs, etc., then you might want to have one system build binary packages
> for distribution to the internal systems.  That way you only suffer the
> build once.

In the future I'll be distributing binaries to lots of old pI and pII systems,
so that kind of automation will have to wait.


> Personnaly I've got different architectures (PIII, P4, and AMD) mixed in
> and, as each system has a different purpose they typically don't have
> similar use flags or package installations, so a binary package distribution
> doesn't work for me.

ditto.

> > Surely parts of all (4) documents belong in one master howto?

> I'm sure it could but there's probably folks on the other side of the fence
> that only want a local rsync mirror or a local portage download cache and
> not both.

Well the simple cases are easy. What I need, including your scripts is
to be in a single document. I'll write something, and you can look it
over. That helps me fully understand what everything does, and provides
a nice little document. I tend to 'dumb down' documents, so they are 
easy to understand.... I also include more background material
so novices and seasoned persons can use them.

> > Is there a script to update workstation systems, automatically,
> >  say 10 minutes after booting? How best to do this?

> Sure.  Generate a script that sleeps for 10 minutes then kicks off the
> emerge process.  Call the script from /etc/conf.d/local.start (be sure to
> include the & to spawn it in the background) and you're golden.

> However, you need to consider if this is something that you really want to
> do.  Automated updates are frowned upon by folks on the list.


Agreed. But I intend to set up a test-quarrantine system that is fully
automated, and if all goes well for (1) 24-hour period, then push the
updates to a bunch of PI and PII systems for a bunch of kids (hoodlums.... 
some of which are mine.)

Thanks for the help and scripts...
James



-- 
gentoo-user@gentoo.org mailing list

[gentoo-user] Re: rsync internal mirror configuration

[James]

A. Khattri <ajai <at> bway.net> writes:


> > Is there a script to update workstation systems, automatically,
> >  say 10 minutes after booting? How best to do this?

> I run this in my servers at 3am (they sync off a local rsync server,
> similar to your setup):
> http://www.panhorst.com/glcu/


This looks interesting. 
For now I'm going to keep it manual, for a variety of reasons.
When I get ready to 'automate' clients (gentoo workstations)
I'll take a deeper look at this software.

For now, I'm going to sit back and tweak the manual process
to gain a deeper understanding of the issues with automation.

Thanks for the information.

James




-- 
gentoo-user@gentoo.org mailing list

RE: [gentoo-user] Re: rsync internal mirror configuration

[Dave Nebinger]

> OK but can't I do that with a simple mode to the /etc/crontab ?:
> 30 1 * * * root emerge sync 2>&1
> ?

Yes, but I do a little more in my script and want the pieces to be
synchronized.  As you saw in the script not only do I emerge sync but I also
update the eix and esearch databases (both are cached package search/query
tools that kick the crap out of "emerge --search").

I've been considering adding an "emerge --fetchonly" line to the script but
don't yet have the warm and fuzzies about the http-replicator script.

> This script is for the internal rsync/replicator server?

Runs on all of them.  All of them need to sync and I like being able to
query from whatever system I'm on.

> OK, the port has changed. What's the deal about that?

Most local http daemons run on 8080 or 8088; I just didn't want to deal with
conflicts so I chose my own number.
 
> OK thanks for the script. I'll have to study it, test it and let you know
> how it goes...

They've been working out great for me.  I've recently added a
'revdep-rebuild' script so I can be reported of packages in need of fixing
(in case I miss the step manually).

> Agreed. But I intend to set up a test-quarrantine system that is fully
> automated, and if all goes well for (1) 24-hour period, then push the
> updates to a bunch of PI and PII systems for a bunch of kids (hoodlums....
> some of which are mine.)

Ah, but how long would it take to test out the new package(s) in the
quarantine system?  When it's something significant (i.e. pam) are you going
to test out every system component using pam before deploying internally?

In your case I'd probably ensure the kids logins are typical user logins (no
update capability).  Run the scripts (or similar scripts) to automate the
syncing and reporting.  Keep ssh running on their systems.  Hold off on
updates until they run into something that breaks or until a critical
package update is released.  Then you can ssh in and emerge stuff until it
works.

Basically it's a question that even windows folks have to face - is it more
important to have a working yet somewhat out of date system or a
cutting-edge updated system that can sometimes be unstable?

In my home I've chosen to keep my boxen up to date but the kids systems tend
to lag well behind the curve.



-- 
gentoo-user@gentoo.org mailing list

[gentoo-user] Re: rsync internal mirror configuration

[James]

Dave Nebinger <dnebinger <at> joat.com> writes:


> > OK but can't I do that with a simple mode to the /etc/crontab ?:
> > 30 1 * * * root emerge sync 2>&1

> Yes, but I do a little more in my script and want the pieces to be
> synchronized.  As you saw in the script not only do I emerge sync but I also
> update the eix and esearch databases (both are cached package search/query
> tools that kick the crap out of "emerge --search").

I'm going to test your scripts and look at this for a while....

> I've been considering adding an "emerge --fetchonly" line to the script but
> don't yet have the warm and fuzzies about the http-replicator script.

Yes, well I'm going to go slowly with this
until I have the warm feeling and some
happy experience over time.
I'd be interested in keeping up with your scipt enhancements.
If it's not too much trouble, when you are happy with
some enhancements, send them to me for testing....

> > This script is for the internal rsync/replicator server?

> Runs on all of them.  All of them need to sync and I like being able to
> query from whatever system I'm on.

Yea, I was not sure, but it makes
sense.

> > OK, the port has changed. What's the deal about that?

> Most local http daemons run on 8080 or 8088; I just didn't want to deal with
> conflicts so I chose my own number.

Yea, I might change the port number...if I see problems. 

> > OK thanks for the script. I'll have to study it, test it and let you know
> > how it goes...

> They've been working out great for me.  I've recently added a
> 'revdep-rebuild' script so I can be reported of packages in need of fixing
> (in case I miss the step manually).

Can you send me a copy of that script or post it to this thread?


> In your case I'd probably ensure the kids logins are typical user logins (no
> update capability).  Run the scripts (or similar scripts) to automate the
> syncing and reporting.  Keep ssh running on their systems.  Hold off on
> updates until they run into something that breaks or until a critical
> package update is released.  Then you can ssh in and emerge stuff until it
> works.

I'll take this under consideration. My purpose in extending automated
updates, is to be able to manage a large number of embedded gentoo 
devices in the future.  Roll-back and Recovery mechanism will be 
added later. Besides Gentoo needs to leave the laboratory
(purvey of experts) and enter the world of normal humanoids. That will
force the Gentoo community to make Gentoo a commodity technology for
the world's normal folks.....gentoo's destiny in my opinion.

Using kids as experiments and developing baseline analysis semantics, is
of great fun... (Hell, that's what God gave them to us for: to 
experiment with them and then explain where dad messed up and dad's
newest ideas.) Even their friends like the approach of experimental
parenting... Furthermore, as an engineer, I often tell my clients 
that something works with the children or that children understand the
new technology, so maybe management can adapt too...)

YMMV

James

-- 
gentoo-user@gentoo.org mailing list

RE: [gentoo-user] Re: rsync internal mirror configuration

[Dave Nebinger]

> > I've been considering adding an "emerge --fetchonly" line to the script
> but
> > don't yet have the warm and fuzzies about the http-replicator script.
> 
> Yes, well I'm going to go slowly with this
> until I have the warm feeling and some
> happy experience over time.
> I'd be interested in keeping up with your scipt enhancements.
> If it's not too much trouble, when you are happy with
> some enhancements, send them to me for testing....

Sure, but I don't have any plans for updates in the near future.  They all
seem to be functioning well enough for my intents and purposes.

> > They've been working out great for me.  I've recently added a
> > 'revdep-rebuild' script so I can be reported of packages in need of
> fixing
> > (in case I miss the step manually).
> 
> Can you send me a copy of that script or post it to this thread?

Sure.  It's nothing fancy, but does the job.  The biggest headache is the
gentoo script developers expectation that the only time you're going to call
their script is when you're sitting at a color-supporting terminal.  When
trying to build an email message all of the color-coding crap still seeps
through.  If I could get them to do anything it would be to add a common
"--nocolor" command line argument to disable that stuff.

cornholio ~ # cat bin/revdep.sh
#!/bin/sh
#
# revdep.sh Script to report on revdep/rebuild tasks.
#

# Remove existing files.
/bin/rm -f /root/.revdep* >> /var/log/revdep.log 2>&1

# Now do the revdep stuff
revdep-rebuild -p > /var/log/revdep.rpt 2>>/var/log/revdep.log

# now mail the report to root
strings /var/log/revdep.rpt > /var/log/revdep.txt
mail -s "Cornholio revdep report" root < /var/log/revdep.txt
2>>/var/log/revdep.log

cornholio ~ #

> > In your case I'd probably ensure the kids logins are typical user logins
> (no
> > update capability).  Run the scripts (or similar scripts) to automate
> the
> > syncing and reporting.  Keep ssh running on their systems.  Hold off on
> > updates until they run into something that breaks or until a critical
> > package update is released.  Then you can ssh in and emerge stuff until
> it
> > works.
> 
> I'll take this under consideration. My purpose in extending automated
> updates, is to be able to manage a large number of embedded gentoo
> devices in the future.  Roll-back and Recovery mechanism will be
> added later. Besides Gentoo needs to leave the laboratory
> (purvey of experts) and enter the world of normal humanoids. That will
> force the Gentoo community to make Gentoo a commodity technology for
> the world's normal folks.....gentoo's destiny in my opinion.

Oh, I'd tend to disagree.  Source-based distributions IMHO would be a huge
pain in the neck for those responsible for a large number of systems.  Twer
it up to me I'd be using a binary-based distro such as suse or redhat; let
them work out the kinks so I wouldn't have to.

I prefer running gentoo because I have total control over the box.  Downtime
at home simply means I can't surf or receive email; it also means that I
have some good old down and dirty work ahead to get the box back up, which I
enjoy.  It's also a hassle that only I have to deal with, not a whole floor
of workers sitting idle waiting for the problem to be resolved.

But to be an admin over an office full of gentoo systems?  Not sure I'd like
that or that they would pay me what it would take to keep it functional.
Unless, of course, all systems were exactly the same (hardware and software
installations) and the users had minimal permissions to the systems (i.e.
they could only write files to their home directories and nowhere else on
the system, including /tmp & /var/tmp).




-- 
gentoo-user@gentoo.org mailing list

[gentoo-user] Re: rsync internal mirror configuration

[James]

Dave Nebinger <dnebinger <at> joat.com> writes:


Thanks for the scripts and help!

> > I'll take this under consideration. My purpose in extending automated
> > updates, is to be able to manage a large number of embedded gentoo
> > devices in the future.  Roll-back and Recovery mechanism will be
> > added later. Besides Gentoo needs to leave the laboratory
> > (purvey of experts) and enter the world of normal humanoids. That will
> > force the Gentoo community to make Gentoo a commodity technology for
> > the world's normal folks.....gentoo's destiny in my opinion.

> Oh, I'd tend to disagree.  Source-based distributions IMHO would be a huge
> pain in the neck for those responsible for a large number of systems.  Twer
> it up to me I'd be using a binary-based distro such as suse or redhat; let
> them work out the kinks so I wouldn't have to.

> I prefer running gentoo because I have total control over the box.  Downtime
> at home simply means I can't surf or receive email; it also means that I
> have some good old down and dirty work ahead to get the box back up, which I
> enjoy.  It's also a hassle that only I have to deal with, not a whole floor
> of workers sitting idle waiting for the problem to be resolved.

> But to be an admin over an office full of gentoo systems?  Not sure I'd like
> that or that they would pay me what it would take to keep it functional.
> Unless, of course, all systems were exactly the same (hardware and software
> installations) and the users had minimal permissions to the systems (i.e.
> they could only write files to their home directories and nowhere else on
> the system, including /tmp & /var/tmp).

Well, I do not think we disagree at all. I'm ramping up to support 5-15
gentoo user systems, to get some experience. My real goal is to learn
gentoo at a deeper level. When I roll out a new TCP/IP based data-logger
(hopefully this fall) it'll take sensor inputs and control a few outputs. 
No humans will use the devices for anything other than interfacing 
sensors and collecting data. 

The experience I gain form helping kids/adults use Gentoo, will help me
manage thousands of dataloggers across public and private networks. 
Besides if I screw it up, no big deal. They can always use a winblows
box until I get it fixed. Strict user control semantics will be
used to limit what they can screw up.

I'm also using jffnms to update and manage all sorts of routers and 
industrial contols embedded devices. After some
time, I'm sure I'll roll my own solution, but for now, managing Gentoo
user systems and customizing JFFNMS for router and other snmp devices
is enough of to keep me busy.

Thanks again for all of your help!

sincerely,
James Horton




-- 
gentoo-user@gentoo.org mailing list

RE: [gentoo-user] Re: rsync internal mirror configuration

[Dave Nebinger]

> Thanks for the scripts and help!

That's what we're here for ;-)

> When I roll out a new TCP/IP based data-logger
> (hopefully this fall) it'll take sensor inputs and control a few outputs.
> No humans will use the devices for anything other than interfacing
> sensors and collecting data.

As with most other embedded devices, wouldn't these remain fairly static at
the OS level?  I mean, would you really want to try to debug why (all of a
sudden) the remote devices stopped functioning because of an errant package
emerge?

As for learning gentoo at a deeper level, we encourage that.  I would
suggest, only because it really made things clear for me, that you look at
Linux from Scratch http://www.linuxfromscratch.org.  It will make the entire
process for how to custom build a linux box a lot clearer.

> The experience I gain form helping kids/adults use Gentoo, will help me
> manage thousands of dataloggers across public and private networks.
> Besides if I screw it up, no big deal. They can always use a winblows
> box until I get it fixed. Strict user control semantics will be
> used to limit what they can screw up.

>From an administrative point of view, however, I think you'd be better
served by having one system working, freeze it and then clone it to the
other systems.  Skip updates as much as possible.

Basically you'd be taking the same route as other embedded linux products.
My linksys routers, all linux based, have the ability to handle new firmware
(linux distribution) but they do not auto-update themselves.

> I'm also using jffnms to update and manage all sorts of routers and
> industrial contols embedded devices. After some
> time, I'm sure I'll roll my own solution, but for now, managing Gentoo
> user systems and customizing JFFNMS for router and other snmp devices
> is enough of to keep me busy.

Exactly my point - why introduce even more administrative headaches to have
thousands of gentoo systems automatically emerging packages on their own?



-- 
gentoo-user@gentoo.org mailing list

[gentoo-user] Re: rsync internal mirror configuration

[James]

Dave Nebinger <dnebinger <at> joat.com> writes:

> 
> > Thanks for the scripts and help!
> 
> That's what we're here for 
> 
> > When I roll out a new TCP/IP based data-logger
> > (hopefully this fall) it'll take sensor inputs and control a few outputs.
> > No humans will use the devices for anything other than interfacing
> > sensors and collecting data.
> 
> As with most other embedded devices, wouldn't these remain fairly static at
> the OS level?  I mean, would you really want to try to debug why (all of a
> sudden) the remote devices stopped functioning because of an errant package
> emerge?

Well, yes and no. Yes you do not update remote device as often, but, they
do need RTOS/executive/kernel/state-machine updates over time. It's
a huge problem for devices with a microprocessor/microcontroller.
We often design device to store 2 or more bootstrap programs. That way
a remote device can get new bootstrap firmware, and if it fails use the
old boot code. DISTRIBUTION aka over ppp or TCP/IP is very much needed.
Most embedded companies, particulary in the 8-16 bit micro space, do 
a pathetic job at longevity maintenance planning on their embedded devices.
Suppose your remote 8/16 bit micro supports a fat file system on an
8 bit dsp. It's a nightmare support all of the different vendors
flash based cards, so the driver(s) usually use a few select vendors. But
the company with those devices, cuts a deal with a new 'hong-kong'
vendor. Naturally, their flash card does not work. The driver code,
part of the RTOS/executive/kernel/StateMachine has to be updated. Also,
newer versions of processors end up in products, so you get your 
maintenace code now spread over several versions of a processor,
much like the AMD-Intel-Via.....saga. 

There is a need for a robust binary and remote compilation solution
for remotely located embedded devices. Flexibility to mix and
match is the key. The applications that run on the
RTOS/kernel/executive/StateMachine need more freequent updates than
the raw OS engine code, but, both need to be maintained. This would also
lead to less disposal of microelectronic devices, world wide. The
garbage dumps of the world, need a little reprieve, in my opinion, too.


> As for learning gentoo at a deeper level, we encourage that.  I would
> suggest, only because it really made things clear for me, that you look at
> Linux from Scratch http://www.linuxfromscratch.org.  It will make the entire
> process for how to custom build a linux box a lot clearer.

Yes this is on my 'todo' list. I consult and work 40-80 hours per week.
Soon I'll have a couple of months off.....

> 
> > The experience I gain form helping kids/adults use Gentoo, will help me
> > manage thousands of dataloggers across public and private networks.
> > Besides if I screw it up, no big deal. They can always use a winblows
> > box until I get it fixed. Strict user control semantics will be
> > used to limit what they can screw up.
> 
> From an administrative point of view, however, I think you'd be better
> served by having one system working, freeze it and then clone it to the
> other systems.  Skip updates as much as possible.
Sure but updates for microprocessors are the simili for eduction. Do 
you think we should minimize educcation? minimize education for adults?

Methinks your indescretions and prejudices against micros, is insensitive
to the future, sensate possibilitys of micros and the things they inhabit.
micros are entering the human body at an alarming rate. Don't you thing
we need mechanisms to keep their interal code robust and current?
Nanobots are real and the very near future of medicine.



> Basically you'd be taking the same route as other embedded linux products.
> My linksys routers, all linux based, have the ability to handle new firmware
> (linux distribution) but they do not auto-update themselves.

I'm thinking much bigger than router code. My wife if in the middle of
a phD in Biomedical. Her area of expertise is firmware. There is a HUGE
need to further the paradyme.


> > I'm also using jffnms to update and manage all sorts of routers and
> > industrial contols embedded devices. After some
> > time, I'm sure I'll roll my own solution, but for now, managing Gentoo
> > user systems and customizing JFFNMS for router and other snmp devices
> > is enough of to keep me busy.

> Exactly my point - why introduce even more administrative headaches to have
> thousands of gentoo systems automatically emerging packages on their own?

Well let me remind you of a lawyer story (you know lawyers make much more
money than engineers and computer scientists?) 

There was a small town with with one lawyer. He was financially starving.
Another lawyer move to town, business boomed, and both retired wealthy.
Quit canabalizing computer science and electrical engineer so that we 
die of starvation. The more complex the systems, the more money we can
all make..... Beside complex feature-rich systems are wonderful and employ
lots of people. Just look at the space-shuttle....

(HUGH GRIN)

James 




-- 
gentoo-user@gentoo.org mailing list

[gentoo-user] Re: rsync internal mirror configuration

[James]

Dave Nebinger <dnebinger <at> joat.com> writes:

> > Thanks for the scripts and help!

> That's what we're here for 

BACKGROUND:
I did not use your scripts yet...today I'll implement them.

I recently set up an internal server for rsync and distfile distribution.
How do I check to ensure that this internal server actually was successful
at downloading the rsync files and the appropriate distfiles for the other
sytems?

I followed part 4 of this document:
http://www.gentoo.org/doc/en/rsync.xml?style=printable

and these 3 documents:

http://gentoo-wiki.com/HOWTO_Local_Rsync_Mirror.
http://gentoo-wiki.com/Http_proxy
http://gentoo-wiki.com/HOWTO_Download_Cache_for_LAN-Http-Replicator


PROBLEM?
I manually ran 'emerge -uDp world' today on both a client and the
sever. The client updated a few files, the server returned nothing to update.
I kept another p4 system using rsync and downloading the distfiles separate.

emerge -uDp world shows this file (among others) on the P4 system using the old
external update method:
[ebuild  NS   ] sys-kernel/gentoo-sources-2.6.12-r5

The new internal-AMD rsync/distfile server, issueing
'emerge -s gentoo-sources' shows:
      sys-kernel/gentoo-sources
      Latest version available: 2.6.12-r4
      Latest version installed: 2.6.12-r4

Is this evidence that the nightly updates, are not working on the
internal server?
Is there a simple test to determine if the updates are working
on the rsync and distfiles?

Did I miss something?
Ideas?



-- 
gentoo-user@gentoo.org mailing list

RE: [gentoo-user] Re: rsync internal mirror configuration

[Dave Nebinger]

> Methinks your indescretions and prejudices against micros, is insensitive
> to the future, sensate possibilitys of micros and the things they inhabit.
> micros are entering the human body at an alarming rate. Don't you thing
> we need mechanisms to keep their interal code robust and current?
> Nanobots are real and the very near future of medicine.

Yes but we get back to the point that I had been trying to make - the
embedded devices (even nanobots) would not need to build from source.  I
know if I had embedded nanobots I would want them focusing on the job they
were embedded for, not recompiling source from the latest software updates.

That's the distinction between a source based an binary based distribution.
Use the source based distro to optimize for hardware then that is the
template for the binary based distro to the embedded systems.



-- 
gentoo-user@gentoo.org mailing list

RE: [gentoo-user] Re: rsync internal mirror configuration

[Dave Nebinger]

> I recently set up an internal server for rsync and distfile distribution.
> How do I check to ensure that this internal server actually was successful
> at downloading the rsync files and the appropriate distfiles for the other
> sytems?

/usr/portage/metadata/timestamp contains the timestamp for when the sync was
completed.
 
> PROBLEM?
> I manually ran 'emerge -uDp world' today on both a client and the
> sever. The client updated a few files, the server returned nothing to
> update.
> I kept another p4 system using rsync and downloading the distfiles
> separate.

Since the server is the local rsync mirror it must do it's emerge --sync
first.  After it has completed (note: not during the run), the client(s) can
emerge --sync.
 
> emerge -uDp world shows this file (among others) on the P4 system using
> the old external update method:
> [ebuild  NS   ] sys-kernel/gentoo-sources-2.6.12-r5
> 
> The new internal-AMD rsync/distfile server, issueing
> 'emerge -s gentoo-sources' shows:
>       sys-kernel/gentoo-sources
>       Latest version available: 2.6.12-r4
>       Latest version installed: 2.6.12-r4
> 
> Is this evidence that the nightly updates, are not working on the
> internal server?

No, it is merely an indication that at some point between the server's sync
and the older sync that -r5 of gentoo sources was released.  My systems, as
of last night around midnight thought only -r4 was available.  Just now I
ran emerge --sync on the server and it now sees that -r5 is available.

It's strictly a timing issue.

> Is there a simple test to determine if the updates are working
> on the rsync and distfiles?

Sure.  New packages are released every day.  I can't remember a single day
in the last month where emerge -uDp reported no packages to update.

So if you run a few days and constantly see emerge -uDp reporting no
packages, there's probably a sync problem.




-- 
gentoo-user@gentoo.org mailing list

[gentoo-user] Re: rsync internal mirror configuration

[James]

Dave Nebinger <dnebinger <at> joat.com> writes:



> Yes but we get back to the point that I had been trying to make - the
> embedded devices (even nanobots) would not need to build from source.  I
> know if I had embedded nanobots I would want them focusing on the job they
> were embedded for, not recompiling source from the latest software updates.

> That's the distinction between a source based an binary based distribution.
> Use the source based distro to optimize for hardware then that is the
> template for the binary based distro to the embedded systems.

Well you may be right the majority of the time, but as complex sytems grow
and the number of difference in peripherals, sensors, and other variable
hardrware details, make issuing a single binary for upgrade, problematic.
Just like the kernel on a workstation can be compile specifically for a 
target mix of devices and the applications then are compiled, sometimes
dependant on the kernel/rtos/executive/statemachine details for the binaries that
result. Furthermore, often you are resource constrained on an embedded
micro:timing, interrupts, code space, processor, cache, or ram limits
(just to name a few). It's very common, particulary in the 8/16 bit micro
world to make sifnification changes to the statemachine, just because
the bloating of features has now created performance or code space issues.
Often, Ansi C code is replaced with assembler to compact the executables.

You have a valid point, but, things are more complicated than that. Often
haredware gets replaced or upgraded because the increasing diversity of
the code bloats beyond what the resources can handle. Hareware designers
often screw up the amount of computational resources needed on an embedded
device.  That's way so many Cell phones are thrown away each year, they
cannot properly handle software (rtos and apps) upgrades....

Statistically you are correct. In actuality, a significant and growing protion
of embedded devices lack the ability (resources) to have their software upgraded
or not viable mechanism (like attaching your gameboy to a linux system)
exists to extend the life of the micro based product.....

Think about it. Microsoft and the OS vendors are abandoning  586, p1 and 
p2 machines in droves. Yet linux/BSD et. al. offer extended life to
these aging machines. Embedded devices are only limited by the
forward thinking of their designers.....

Besides what else would I do with my fleeting free time?

James



-- 
gentoo-user@gentoo.org mailing list

[gentoo-user] Re: rsync internal mirror configuration

[James]

Dave Nebinger <dnebinger <at> joat.com> writes:

> 
> > I recently set up an internal server for rsync and distfile distribution.
> > How do I check to ensure that this internal server actually was successful
> > at downloading the rsync files and the appropriate distfiles for the other
> > sytems?
> 
> /usr/portage/metadata/timestamp contains the timestamp for when the sync was
> completed.
> 
Ahhh that's it:
# cat /usr/portage/metadata/timestamp
Tue Jul 12 13:06:52 UTC 2005
 # date
Wed Jul 13 12:05:26 UTC 2005

The clock (hw and OS) were off, by about25 minutes but that has 
been correct now. /etc/crontab was set for 1:30 
30 1 * * * root emerge sync

> > PROBLEM?
> > I manually ran 'emerge -uDp world' today on both a client and the
> > sever. The client updated a few files, the server returned nothing to
> > update.
> > I kept another p4 system using rsync and downloading the distfiles
> > separate.
> 
> Since the server is the local rsync mirror it must do it's emerge --sync
> first.  After it has completed (note: not during the run), the client(s) can
> emerge --sync.
> 
> > emerge -uDp world shows this file (among others) on the P4 system using
> > the old external update method:
> > [ebuild  NS   ] sys-kernel/gentoo-sources-2.6.12-r5
> > 
> > The new internal-AMD rsync/distfile server, issueing
> > 'emerge -s gentoo-sources' shows:
> >       sys-kernel/gentoo-sources
> >       Latest version available: 2.6.12-r4
> >       Latest version installed: 2.6.12-r4
> > 
> > Is this evidence that the nightly updates, are not working on the
> > internal server?
> 
> No, it is merely an indication that at some point between the server's sync
> and the older sync that -r5 of gentoo sources was released.  My systems, as
> of last night around midnight thought only -r4 was available.  Just now I
> ran emerge --sync on the server and it now sees that -r5 is available.
> 
> It's strictly a timing issue.

Well now I'm starting to 'get it'. Before when I ran rsync manually,
it did not update (rsync was launched last night (EST), so
I got some error message about not  being available or something
like that. Now I just ran 'emerge syncc' maually on the  server
machine and it did update the rsync files (portage cache).
On the internal server
emerge -uDp world now does not reveal any new files to update...

> 
> > Is there a simple test to determine if the updates are working
> > on the rsync and distfiles?
> 
> Sure.  New packages are released every day.  I can't remember a single day
> in the last month where emerge -uDp reported no packages to update.
> 
> So if you run a few days and constantly see emerge -uDp reporting no
> packages, there's probably a sync problem.

Well 2 things.

I'm going to implement your scripts now. And I shall have patiences 
for a few days to see how the updates proceed on the internal server
and the clients using the internal server. Checking against the 
internal clients still using remote servers for rsync and distfile
updates....

Thanks,

James






-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Re: rsync internal mirror configuration

[A. Khattri]

On Wed, 13 Jul 2005, James wrote:

> Ahhh that's it:
> # cat /usr/portage/metadata/timestamp
> Tue Jul 12 13:06:52 UTC 2005
>  # date
> Wed Jul 13 12:05:26 UTC 2005

On my machines, the master rsync server does a sync at 3am.
The rest of the machines sync with the master server at 4am.

This way there's no timing problems.

> The clock (hw and OS) were off, by about25 minutes but that has
> been correct now. /etc/crontab was set for 1:30

I run ntpd on ALL my machines (If you want to get fancy, you could setup
your rsync server to also be a local NTP master server and have the other
machines sync their clocks with this server too - makes even more sense
if you use NFS).


-- 
Aj.
-- 
gentoo-user@gentoo.org mailing list

[gentoo-user] Re: rsync internal mirror configuration

[James]

A. Khattri <ajai <at> bway.net> writes:

> 
> On Wed, 13 Jul 2005, James wrote:
> 
> > Ahhh that's it:
> > # cat /usr/portage/metadata/timestamp
> > Tue Jul 12 13:06:52 UTC 2005
> >  # date
> > Wed Jul 13 12:05:26 UTC 2005
> 
> On my machines, the master rsync server does a sync at 3am.
> The rest of the machines sync with the master server at 4am.
> 
> This way there's no timing problems.
> 
The timing problems have been eliminated. All dates and times
are  within a few second of each other now.
System/OS clocks have all been sync to cell phone time.
Bios/Hardware clocks are now sync'd to the System/OS clocks.

> > The clock (hw and OS) were off, by about25 minutes but that has
> > been correct now. /etc/crontab was set for 1:30

> I run ntpd on ALL my machines (If you want to get fancy, you could setup
> your rsync server to also be a local NTP master server and have the other
> machines sync their clocks with this server too - makes even more sense
> if you use NFS).

I don't want to run NTP right now, but later I will.
I do not use NFS, as each system has plenty of disk space.
I'm not a big fan of NFS.... YMMV.


Thanks for the ideas, although these are not the source of
my problems. I'm implementing Dave's scripts tonight. Then
I'll see what happens....

I think the inherent entropy associated with patience, is my greatest
source of errors, presently... If I'm wrong, we'll adjust things
externally, and then continue the epic battle with entropy.

Thanks,

James







-- 
gentoo-user@gentoo.org mailing list