[gentoo-user] most of /usr/portage gone--why? and what now}

[gentoo-user] most of /usr/portage gone--why? and what now}

[Allan Gottlieb]

This fine morning I awoke to find that in my fairly new gentoo install
nearly all of /usr/portage is gone.

    bash-2.05b# ls -lA /mnt/gentoo-new/usr/portage
    total 20
    drwxrwsr-x  4 root portage 16384 Jul 12 22:10 distfiles
    drwxr-xr-x  4 root root     4096 Jul 11 23:04 packages
    bash-2.05b#

The distfiles directory has many (appropriate) files and packages has
the one package it should.  The partition has plenty of room (almost
7GB).

I run esync from cron (anacron) when I boot and it failed (see below
for the mail sent by anacron) telling me to see emerge-sync.log.  Here
I see some failures and a statement that a new version of portage is
available (again see below).

I still have my previous gentoo (using it right now) on a different
partition.  It started from one of the 2004 profiles, but has been
kept up to date until a very few weeks ago when I started cutting over
to my new system.  I can certainly run an
   emerge --tree --ask --verbose --newuse --update --deep world
but wanted to do as little as possible before asking for advice.

My main question is whether I should copy files from the old
/usr/portage to the new one or is there better method to proceed.
I keep extensive backups of all my files and all of /etc, but do not
backup the system files (other than having an older version of gentoo
that is kept reasonably up to date).

Thanks for any help.
allan

================  Mail from anacron concerning today's esync ================

>From root@localhost.localdomain  Sat Jul 16 09:27:45 2005
Return-Path: <root@localhost.localdomain>
X-Original-To: root
Delivered-To: root@localhost.localdomain
Received: by localhost.localdomain (Postfix, from userid 0)
        id 5E33E544B5; Tue, 12 Jul 2005 11:49:51 -0400 (EDT)
From: root@localhost.localdomain (Anacron)
To: root@localhost.localdomain
Subject: Anacron job 'anacron-daily-2'
Message-Id: <20050712154951.5E33E544B5@localhost.localdomain>
Date: Tue, 12 Jul 2005 11:49:51 -0400 (EDT)

 * Importing old portage tree
 * Doing '/usr/bin/emerge sync' now
 * Error: '/usr/bin/emerge sync' failed, see /var/log/emerge-sync.log for errors
run-parts: /local/etc/anacron-daily-2/esync-cron exited with return code 1

>From root@localhost.localdomain  Sat Jul 16 09:34:29 2005
Return-Path: <root@localhost.localdomain>
X-Original-To: root
Delivered-To: root@localhost.localdomain
Received: by localhost.localdomain (Postfix, from userid 0)
        id 869A7544B7; Sat, 16 Jul 2005 09:34:26 -0400 (EDT)
From: root@localhost.localdomain (Anacron)
To: root@localhost.localdomain
Subject: Anacron job 'anacron-daily-2'
Message-Id: <20050716133426.869A7544B7@localhost.localdomain>
Date: Sat, 16 Jul 2005 09:34:26 -0400 (EDT)

 * Importing old portage tree
 * Doing '/usr/bin/emerge sync' now

 * Doing 'eupdatedb' now


 * esearch-index generated in 1 second(s)
 * indexed 0 ebuilds
 * size of esearch-index: 0 kB

 * Importing new portage tree
 * Preparing databases
 * Searching for changes

 * No updates found

================ Contents of emerge-sync.log ================

rsync: failed to connect to 128.118.99.31: Connection timed out
rsync error: error in socket IO (code 10) at clientserver.c(88)
==============================
CLARKSON OPEN SOURCE INSTITUTE
http://cosi.clarkson.edu
==============================
IP:             mirror.clarkson.edu
                rsync10.us.gentoo.org
Specs:          Sun Ultra60
                2 x 296Mhz CPUs
                1796 MB RAM
                250 GB RAID Storage
Bandwidth:      10 Mbit (when limited)
User Limit:     35 off-campus connections
Location:       Clarkson University
                http://www.clarkson.edu
                Potsdam, NY USA
Contact:        mirror@cosi.clarkson.edu
==============================
Welcome to our rsync server.

receiving file list ...
1 file to consider

Number of files: 1
Number of files transferred: 0
Total file size: 32 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 32
Total bytes written: 203
Total bytes read: 550

wrote 203 bytes  read 550 bytes  301.20 bytes/sec
total size is 32  speedup is 0.04
==============================
CLARKSON OPEN SOURCE INSTITUTE
http://cosi.clarkson.edu
==============================
IP:             mirror.clarkson.edu
                rsync10.us.gentoo.org
Specs:          Sun Ultra60
                2 x 296Mhz CPUs
                1796 MB RAM
                250 GB RAID Storage
Bandwidth:      10 Mbit (when limited)
User Limit:     35 off-campus connections
Location:       Clarkson University
                http://www.clarkson.edu
                Potsdam, NY USA
Contact:        mirror@cosi.clarkson.edu
==============================
Welcome to our rsync server.

receiving file list ...
1 file to consider
 0 files...
Number of files: 1
Number of files transferred: 0
Total file size: 0 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 20
Total bytes written: 181
Total bytes read: 538

wrote 181 bytes  read 538 bytes  287.60 bytes/sec
total size is 0  speedup is 0.00
>>> starting rsync with rsync://128.118.99.31/gentoo-portage...
>>> checking server timestamp ...
>>> retry ...


>>> Starting retry 1 of 3 with rsync://128.153.144.22/gentoo-portage
>>> checking server timestamp ...


 * An update to portage is available. It is _highly_ recommended
 * that you update portage now, before any other packages are updated.
 * Please do so and then update ALL of your configuration files.

-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] most of /usr/portage gone--why? and what now}

[William Kenworthy]

If files have gone missing first step is fsck the disk just in case.
Next emerge sync if it works.  (dont use the fancy options until its
working again.)  Whats "esync" ??  I wouldnt trust it until you know
what happened.

You can copy the complete portage from another machine over the top
without problems - the rsync part of emerge does just that. If its not
to old, just change the profile link if necessary and emerge sync to
upgrade.

I upgrade portage as the first step after the sync whenever it appears:
portage upgrades often change some of the underlying structure (as well
as break things ...) 

There is also a rescue portage hidden somewhere in portage that can also
be downloaded - its a tarball of a basic portage with enough "bits" to
rescue a system thats lost its portage all together (I did this once
with rm -rf * in /usr/portage !)

BillK



On Sat, 2005-07-16 at 11:16 -0400, Allan Gottlieb wrote:
> This fine morning I awoke to find that in my fairly new gentoo install
> nearly all of /usr/portage is gone.
> 
>     bash-2.05b# ls -lA /mnt/gentoo-new/usr/portage
>     total 20
>     drwxrwsr-x  4 root portage 16384 Jul 12 22:10 distfiles
>     drwxr-xr-x  4 root root     4096 Jul 11 23:04 packages
>     bash-2.05b#
> 
> The distfiles directory has many (appropriate) files and packages has
> the one package it should.  The partition has plenty of room (almost
> 7GB).
> 
> I run esync from cron (anacron) when I boot and it failed (see below
> for the mail sent by anacron) telling me to see emerge-sync.log.  Here
> I see some failures and a statement that a new version of portage is
> available (again see below).
> 
> I still have my previous gentoo (using it right now) on a different
> partition.  It started from one of the 2004 profiles, but has been
> kept up to date until a very few weeks ago when I started cutting over
> to my new system.  I can certainly run an
>    emerge --tree --ask --verbose --newuse --update --deep world
> but wanted to do as little as possible before asking for advice.
> 
> My main question is whether I should copy files from the old
> /usr/portage to the new one or is there better method to proceed.
> I keep extensive backups of all my files and all of /etc, but do not
> backup the system files (other than having an older version of gentoo
> that is kept reasonably up to date).
> 
> Thanks for any help.
> allan
> 
> ================  Mail from anacron concerning today's esync ================
> 
> From root@localhost.localdomain  Sat Jul 16 09:27:45 2005
> Return-Path: <root@localhost.localdomain>
> X-Original-To: root
> Delivered-To: root@localhost.localdomain
> Received: by localhost.localdomain (Postfix, from userid 0)
>         id 5E33E544B5; Tue, 12 Jul 2005 11:49:51 -0400 (EDT)
> From: root@localhost.localdomain (Anacron)
> To: root@localhost.localdomain
> Subject: Anacron job 'anacron-daily-2'
> Message-Id: <20050712154951.5E33E544B5@localhost.localdomain>
> Date: Tue, 12 Jul 2005 11:49:51 -0400 (EDT)
> 
>  * Importing old portage tree
>  * Doing '/usr/bin/emerge sync' now
>  * Error: '/usr/bin/emerge sync' failed, see /var/log/emerge-sync.log for errors
> run-parts: /local/etc/anacron-daily-2/esync-cron exited with return code 1
> 
> From root@localhost.localdomain  Sat Jul 16 09:34:29 2005
> Return-Path: <root@localhost.localdomain>
> X-Original-To: root
> Delivered-To: root@localhost.localdomain
> Received: by localhost.localdomain (Postfix, from userid 0)
>         id 869A7544B7; Sat, 16 Jul 2005 09:34:26 -0400 (EDT)
> From: root@localhost.localdomain (Anacron)
> To: root@localhost.localdomain
> Subject: Anacron job 'anacron-daily-2'
> Message-Id: <20050716133426.869A7544B7@localhost.localdomain>
> Date: Sat, 16 Jul 2005 09:34:26 -0400 (EDT)
> 
>  * Importing old portage tree
>  * Doing '/usr/bin/emerge sync' now
> 
>  * Doing 'eupdatedb' now
> 
> 
>  * esearch-index generated in 1 second(s)
>  * indexed 0 ebuilds
>  * size of esearch-index: 0 kB
> 
>  * Importing new portage tree
>  * Preparing databases
>  * Searching for changes
> 
>  * No updates found
> 
> ================ Contents of emerge-sync.log ================
> 
> rsync: failed to connect to 128.118.99.31: Connection timed out
> rsync error: error in socket IO (code 10) at clientserver.c(88)
> ==============================
> CLARKSON OPEN SOURCE INSTITUTE
> http://cosi.clarkson.edu
> ==============================
> IP:             mirror.clarkson.edu
>                 rsync10.us.gentoo.org
> Specs:          Sun Ultra60
>                 2 x 296Mhz CPUs
>                 1796 MB RAM
>                 250 GB RAID Storage
> Bandwidth:      10 Mbit (when limited)
> User Limit:     35 off-campus connections
> Location:       Clarkson University
>                 http://www.clarkson.edu
>                 Potsdam, NY USA
> Contact:        mirror@cosi.clarkson.edu
> ==============================
> Welcome to our rsync server.
> 
> receiving file list ...
> 1 file to consider
> 
> Number of files: 1
> Number of files transferred: 0
> Total file size: 32 bytes
> Total transferred file size: 0 bytes
> Literal data: 0 bytes
> Matched data: 0 bytes
> File list size: 32
> Total bytes written: 203
> Total bytes read: 550
> 
> wrote 203 bytes  read 550 bytes  301.20 bytes/sec
> total size is 32  speedup is 0.04
> ==============================
> CLARKSON OPEN SOURCE INSTITUTE
> http://cosi.clarkson.edu
> ==============================
> IP:             mirror.clarkson.edu
>                 rsync10.us.gentoo.org
> Specs:          Sun Ultra60
>                 2 x 296Mhz CPUs
>                 1796 MB RAM
>                 250 GB RAID Storage
> Bandwidth:      10 Mbit (when limited)
> User Limit:     35 off-campus connections
> Location:       Clarkson University
>                 http://www.clarkson.edu
>                 Potsdam, NY USA
> Contact:        mirror@cosi.clarkson.edu
> ==============================
> Welcome to our rsync server.
> 
> receiving file list ...
> 1 file to consider
>  0 files...
> Number of files: 1
> Number of files transferred: 0
> Total file size: 0 bytes
> Total transferred file size: 0 bytes
> Literal data: 0 bytes
> Matched data: 0 bytes
> File list size: 20
> Total bytes written: 181
> Total bytes read: 538
> 
> wrote 181 bytes  read 538 bytes  287.60 bytes/sec
> total size is 0  speedup is 0.00
> >>> starting rsync with rsync://128.118.99.31/gentoo-portage...
> >>> checking server timestamp ...
> >>> retry ...
> 
> 
> >>> Starting retry 1 of 3 with rsync://128.153.144.22/gentoo-portage
> >>> checking server timestamp ...
> 
> 
>  * An update to portage is available. It is _highly_ recommended
>  * that you update portage now, before any other packages are updated.
>  * Please do so and then update ALL of your configuration files.
> 
-- 
William Kenworthy <billk@iinet.net.au>
Home!

-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] most of /usr/portage gone--why? and what now}

[Holly Bostick]

William Kenworthy schreef:
> If files have gone missing first step is fsck the disk just in case.
> Next emerge sync if it works.  (dont use the fancy options until its
> working again.)  Whats "esync" ??  I wouldnt trust it until you know
> what happened.

Esync is a part of gentoolkit-- it runs emerge sync and outputs or mails
you the updated package list (new and upgradeable packages). It's really
not dangerous in and of itself.

Isn't what happened that the sync failed before it was able to import
the new portage tree, but after it had removed the old one?

I could be wrong-- although I've had several failures to sync in the
past few days myself (but I didn't check /usr/portage afterwards, since
I knew it would be borked, but it does support some kind of weird server
issue), but it seems to me that all that's needed is a successful sync
to re-populate the Portage tree.

Afaics, this is one of those "head wound" kinda problems (bleeds a lot
and looks very scary, but not as serious as it seems at first glance).

But I could be mistaken.

Holly
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] most of /usr/portage gone--why? and what now}

[Allan Gottlieb]

At Sun, 17 Jul 2005 03:57:14 +0200 Holly Bostick <motub@planet.nl> wrote:

> William Kenworthy schreef:
>> If files have gone missing first step is fsck the disk just in case.
>> Next emerge sync if it works.  (dont use the fancy options until its
>> working again.)

Done.  Fsck clean.
Emerge --sync has repopulated /usr/portage ... but it gave an error at
the end.  Here is the tail of the output


     118600 files...
    Number of files: 118670
    Number of files transferred: 98725
    Total file size: 93601075 bytes
    Total transferred file size: 93601075 bytes
    Literal data: 93601075 bytes
    Matched data: 0 bytes
    File list size: 2882564
    Total bytes written: 1974681
    Total bytes read: 50760948

    wrote 1974681 bytes  read 50760948 bytes  259143.14 bytes/sec
    total size is 93601075  speedup is 1.77

    >>> Updating Portage cache:
    Traceback (most recent call last):
      File "/usr/bin/emerge", line 2705, in ?
        oldcat = portage.catsplit(cp_list[0])[0]
    IndexError: list index out of range
    ajglap ~ #

>> Whats "esync" ??  I wouldnt trust it until you know what happened.
>
> Esync is a part of gentoolkit-- it runs emerge sync and outputs or mails
> you the updated package list (new and upgradeable packages). It's really
> not dangerous in and of itself.

It also runs eupdatedb, which indexes the packages so that a subsequent
  esearch <package-name>
goes much faster than
  emerge --search <package-name>

> Isn't what happened that the sync failed before it was able to import
> the new portage tree, but after it had removed the old one?

That is my theory as well

> Afaics, this is one of those "head wound" kinda problems (bleeds a lot
> and looks very scary, but not as serious as it seems at first glance).

Great description.

Thank you both.

I am guessing that the the "updating portage cache" error is due
to the previous failed emerge --sync.
Hence I ran an
  emerge --metadata
This succeeded.  I then ran
  eupdatedb
which also succeeded.

Running both
  esearch portage
and
  emerge --ask portage
showed that portage was up to date so I then proceeded to

   emerge --tree --ask --verbose --newuse --update --deep world

which showed firefox, glibc, and a few others.

This is now chugging along successfully.

Thank you again; once once this group has proven to be one of the
great gentoo assets.

Sincerly,
allan
-- 
gentoo-user@gentoo.org mailing list