[gentoo-dev] cvs.gentoo.org hard drive failure

[gentoo-dev] cvs.gentoo.org hard drive failure

[drobbins]

Important information:
======================

At about 10:00 AM this morning, I walked into my home office and noticed that
cvs.gentoo.org's hard drive light was stuck on.  Turning on the monitor, I
found a number of IDE drive error kernel messages on the screen.  I quickly
rebooted the server, and found that the hard drive (a quite new IBM 45Gb
Ultra/100 drive) was making a weird chirping sound and was unable to spin up...
the system wouldn't recognize the drive, it was dead.

So, it's 11:51 AM now, and the system has been restored onto another IBM 30Gb
Ultra/100 drive.  *Fortunately for everone*, I performed a full system backup
to tape about a day ago.  Ironically, I'm in the process of writing a backup
tutorial for IBM developerWorks :/

Here's what we lost:
====================

About a day's worth of received mail

About a day's worth of cvs commits (Achim, could you "cvs commit" from your
mirror so we can get most of these lost changes back?)

About 1500 words from my backup tutorial :/

About 3 hours of excellent work on the new Portage system (a bummer, but
easily re-do-able)

Basically, any files or emails or anything created in the last day have been
lost.  No incoming mail when the server was dead was lost, since we have a
backup MX record and account at my ISP.

Here's what you need to know:
=============================

When I restored everything this morning, I think I had some kind of umask
problem so that some file permissions are incorrect.  You may want to check
your files to make sure that they have the right perms.  Ownership should
be correct.  I had to fix up mailman so that our mailing list would again
work correctly.

All in all, we were incredibly fortunate that it wasn't worse!!!!!!
Have you ever noticed that all system crashes happen on Sunday morning?

Best Regards,

-- 
Daniel Robbins					<drobbins@gentoo.org>
President/CEO					http://www.gentoo.org 
Gentoo Technologies, Inc.			

Re: [gentoo-dev] cvs.gentoo.org hard drive failure

[Achim Gottinger]

drobbins@gentoo.org wrote:
> 
> Important information:
> ======================
> 
> At about 10:00 AM this morning, I walked into my home office and noticed that
> cvs.gentoo.org's hard drive light was stuck on.  Turning on the monitor, I
> found a number of IDE drive error kernel messages on the screen.  I quickly
> rebooted the server, and found that the hard drive (a quite new IBM 45Gb
> Ultra/100 drive) was making a weird chirping sound and was unable to spin up...
> the system wouldn't recognize the drive, it was dead.
> 
> So, it's 11:51 AM now, and the system has been restored onto another IBM 30Gb
> Ultra/100 drive.  *Fortunately for everone*, I performed a full system backup
> to tape about a day ago.  Ironically, I'm in the process of writing a backup
> tutorial for IBM developerWorks :/
> 
> Here's what we lost:
> ====================
> 
> About a day's worth of received mail
> 
> About a day's worth of cvs commits (Achim, could you "cvs commit" from your
> mirror so we can get most of these lost changes back?)

Hmm the mirror runns "cvs update -d" every hour. I tried to commit but
only a few missing digest where commited.
Fortunately I did not on gentoo this weekend. 

> 
> About 1500 words from my backup tutorial :/

But one new chapter for free. :-)

> 
> About 3 hours of excellent work on the new Portage system (a bummer, but
> easily re-do-able)
> 
> Basically, any files or emails or anything created in the last day have been
> lost.  No incoming mail when the server was dead was lost, since we have a
> backup MX record and account at my ISP.
> 
> Here's what you need to know:
> =============================
> 
> When I restored everything this morning, I think I had some kind of umask
> problem so that some file permissions are incorrect.  You may want to check
> your files to make sure that they have the right perms.  Ownership should
> be correct.  I had to fix up mailman so that our mailing list would again
> work correctly.
> 
> All in all, we were incredibly fortunate that it wasn't worse!!!!!!
> Have you ever noticed that all system crashes happen on Sunday morning?

I did not notice any problems. :-)

achim~

> 
> Best Regards,
> 
> --
> Daniel Robbins                                  <drobbins@gentoo.org>
> President/CEO                                   http://www.gentoo.org
> Gentoo Technologies, Inc.
> 
> _______________________________________________
> gentoo-dev mailing list
> gentoo-dev@gentoo.org
> http://www.gentoo.org/mailman/listinfo/gentoo-dev