[gentoo-project] dipper.gentoo.org hardware failure impacts: rsync, releases, distfiles, blogs, project webhosting (& VM temporary outage)

[gentoo-project] dipper.gentoo.org hardware failure impacts: rsync, releases, distfiles, blogs, project webhosting (& VM temporary outage)

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 1071 bytes --]

Hi everybody,

The hardware of dipper.gentoo.org has failed unexpectedly.

The following services are impacted:
- primary rsync mirror data generation (git to rsync)
- primary hosting of releases
- primary hosting of distfiles
- blogs database
- project website hosting

Root cause on the failure seems to be Dell motherboard well past
warranty, so we're going to recover any remaining data from it, and then
move the services to somewhere else.

We're specing out new hardware to buy as a permanent replacement for
dipper, and we have a short-term plan that's being put into action.

As the quickest path, I'm temporarily taking the following VMs offline:
devbox.amd64.dev.gentoo.org
roverlay.dev.gentoo.org
tinderbox.amd64.dev.gentoo.org

Then the dipper disks will be added to the VM host that had these VMs.

-- 
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Trustee & Treasurer
E-Mail   : robbat2@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 445 bytes --]

Re: [gentoo-project] dipper.gentoo.org hardware failure impacts: rsync, releases, distfiles, blogs, project webhosting (& VM temporary outage)

[Chí-Thanh Christopher Nguyễn]

[-- Attachment #1: Type: text/plain, Size: 286 bytes --]

Robin H. Johnson schrieb:
> - primary hosting of distfiles

For understanding, does that mean that no new files will be added to Gentoo 
mirrors now, and if we bump a package then hordes of Gentoo users will DDoS 
${SRC_URI}?


Best regards,
Chí-Thanh Christopher Nguyễn


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[gentoo-project] dipper.gentoo.org service RESTORED: rsync, releases, distfiles, blogs, project webhosting; VM temporary outage ongoing

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 939 bytes --]

On Fri, May 13, 2016 at 09:36:34PM +0000, Robin H. Johnson wrote:
> The hardware of dipper.gentoo.org has failed unexpectedly.
The dipper drives have been transplanted successfully to the gentoo2 VM
host, and dipper lives on for now.

The following VMs will remain temporarily unavailable (the data has been
backed up in a compressed form):
devbox.amd64.dev.gentoo.org
roverlay.dev.gentoo.org
tinderbox.amd64.dev.gentoo.org

A post-mortem report will follow, including:
- quicker return-to-operation for some of the affected services
- hardware purchase planning

A big thank you to Justin Dugger (pwnguin) at OSUOSL for being hands-on
for the hardware switchover this Saturday morning.

-- 
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Trustee & Treasurer
E-Mail   : robbat2@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 445 bytes --]

[gentoo-project] Re: [gentoo-core] dipper.gentoo.org service RESTORED: rsync, releases, distfiles, blogs, project webhosting; VM temporary outage ongoing

[Kristian Fiskerstrand]

[-- Attachment #1.1: Type: text/plain, Size: 1005 bytes --]

On 05/14/2016 10:53 PM, Robin H. Johnson wrote:
> On Fri, May 13, 2016 at 09:36:34PM +0000, Robin H. Johnson wrote:
>> The hardware of dipper.gentoo.org has failed unexpectedly.
> The dipper drives have been transplanted successfully to the gentoo2 VM
> host, and dipper lives on for now.
> 
> The following VMs will remain temporarily unavailable (the data has been
> backed up in a compressed form):
> devbox.amd64.dev.gentoo.org
> roverlay.dev.gentoo.org
> tinderbox.amd64.dev.gentoo.org
> 
> A post-mortem report will follow, including:
> - quicker return-to-operation for some of the affected services
> - hardware purchase planning
> 
> A big thank you to Justin Dugger (pwnguin) at OSUOSL for being hands-on
> for the hardware switchover this Saturday morning.
> 

Nice job getting it back up (relatively) quickly. Thank you!

-- 
Kristian Fiskerstrand
OpenPGP certificate reachable at hkp://pool.sks-keyservers.net
fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [gentoo-project] dipper.gentoo.org service RESTORED: rsync, releases, distfiles, blogs, project webhosting; VM temporary outage ongoing

[Daniel Campbell]

[-- Attachment #1.1: Type: text/plain, Size: 975 bytes --]

On 05/14/2016 01:53 PM, Robin H. Johnson wrote:
> On Fri, May 13, 2016 at 09:36:34PM +0000, Robin H. Johnson wrote:
>> The hardware of dipper.gentoo.org has failed unexpectedly.
> The dipper drives have been transplanted successfully to the gentoo2 VM
> host, and dipper lives on for now.
> 
> The following VMs will remain temporarily unavailable (the data has been
> backed up in a compressed form):
> devbox.amd64.dev.gentoo.org
> roverlay.dev.gentoo.org
> tinderbox.amd64.dev.gentoo.org
> 
> A post-mortem report will follow, including:
> - quicker return-to-operation for some of the affected services
> - hardware purchase planning
> 
> A big thank you to Justin Dugger (pwnguin) at OSUOSL for being hands-on
> for the hardware switchover this Saturday morning.
> 
Awesome job to everyone involved!

-- 
Daniel Campbell - Gentoo Developer
OpenPGP Key: 0x1EA055D6 @ hkp://keys.gnupg.net
fpr: AE03 9064 AE00 053C 270C  1DE4 6F7A 9091 1EA0 55D6


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[gentoo-project] VMs temporarily restored: roverlay, devbox, tinderbox

[Robin H. Johnson]

The VMs that were displaced by the dipper migration have been
temporarily restored.
- devbox.amd64.dev.gentoo.org
- roverlay.dev.gentoo.org
- tinderbox.amd64.dev.gentoo.org

They are going to be slower than previously due to sharing hardware,
until the new hardware is live.

-- 
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Trustee & Treasurer
E-Mail   : robbat2@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136