[gentoo-project] [FYI] Gentoo Service instability June 28 - 29

[Alec Warner <antarus@gentoo.org>]

[-- Attachment #1: Type: text/plain, Size: 932 bytes --]

Hi,

Many of you reported that various infra-owned services were not functioning
properly these past two days. We had some package upgrades roll out that
were bad, and a subset of services were non-functional while we worked
through the incident.

(a) The upgrades did not appear to cause any data loss, but service
availability was reduced (500s / 502s for some HTTP services.)
(b) Some hosts may have had problems running some commands. E.g. I know
that 'grep' didn't work for a while. If you observed errors related to
missing 'GLIBC_2.33' symbols then this was likely related.
(c) Many replicated services (including infra-status) rely on gitweb to
take updates from git; and gitweb was not up, so they were unable to
receive updates.
(d) More impacted notes will follow in a postmortem that will come later
this week for this incident.

Thanks to all of you who reported problems and apologies for the service
disruptions.

-A

[-- Attachment #2: Type: text/html, Size: 1090 bytes --]