From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id 925A2139335 for ; Tue, 29 Jun 2021 03:12:47 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 8A255E0857; Tue, 29 Jun 2021 03:12:44 +0000 (UTC) Received: from mail-ej1-x630.google.com (mail-ej1-x630.google.com [IPv6:2a00:1450:4864:20::630]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 538CAE083D for ; Tue, 29 Jun 2021 03:12:44 +0000 (UTC) Received: by mail-ej1-x630.google.com with SMTP id hz1so33648640ejc.1 for ; Mon, 28 Jun 2021 20:12:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gentoo-org.20150623.gappssmtp.com; s=20150623; h=mime-version:from:date:message-id:subject:to; bh=a4Y1PQSXDhosJseeT5V4YR8eTllUqoh2zq0xCN535SY=; b=E0JOW5L2t9XvrWzfL79Dl1wzHcyabUD3ONA/s8ArGmyLHepAF/Vw1z0HNXkewEhS6W qdC41R1eUPgIbirbCZOadN4DFmFM0sIBOWS7rLGySC93fpNENsKVy7tYfwRia81IIBBt 5GusY3uMWx1fQIxOzAQFdsj6SehhTTn9da1vNoyzS5c1Itb2TJVlWjiVX6596v4WOXg5 zelEkKHBM8ZbI7MxLmbbstt9KdepIDA7gpH1gJMl8VsDmhJ93EGzx0mufWXFgsnKGb3d 3fq2GanjzhlqDv34hFviqGe4a4bqtw7DtBgnw3A787Tg1QT6P0fBu2+XKAwN9TGM67eg RMhg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=a4Y1PQSXDhosJseeT5V4YR8eTllUqoh2zq0xCN535SY=; b=RtDXZtHGk3Ff+W7Cey96yKhqlSf9N8l9lOynVLRk6nWGFyLjWf5mK7Y7kcfkHUbet2 oa6IYGqoprMuoheU1U+Lvn8iVQuKb5UtQXCvxUdqyJqdnwKHnInqkgNpL0PZW5m5UNTW euEHgM0wHIXwW7NCjreknfkfTwyI/i+mnsueVj33kVq4RuKPtXV/XaHmu0lqBgt7AY0l dmL3GZUKtgRyNvj326hftmmykvQpev4i9SwGJ59Ls56AB2WDaSl7G1FryW801UlhQ0wH EV7CxA0/3XW9xzz2NyMUDQliThjLJmkd4QJSeZZy6KwAxaD16coTGe2WeRwTyWiAo7kR FHvg== X-Gm-Message-State: AOAM530ejxwcV0gPEouVgypjkzaurPbrl/i7YuRLvbR19Yer18gONmn5 rOns7JYRw9hHZOu5Y+wQCNN1I0I955ymP3pek1UDcQ== X-Google-Smtp-Source: ABdhPJzmH3EMikPVu5fb8IGWU1xkqCVY8SfUe7uqO16MfoMKCTKjM3/Ay5DI0W/qU+kwBwj20WiJQdn/LbykXZ00JRE= X-Received: by 2002:a17:907:3e9a:: with SMTP id hs26mr27392970ejc.531.1624936362740; Mon, 28 Jun 2021 20:12:42 -0700 (PDT) Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Project discussion list X-BeenThere: gentoo-project@lists.gentoo.org Reply-To: gentoo-project@lists.gentoo.org X-Auto-Response-Suppress: DR, RN, NRN, OOF, AutoReply MIME-Version: 1.0 From: Alec Warner Date: Mon, 28 Jun 2021 20:12:31 -0700 Message-ID: Subject: [gentoo-project] [FYI] Gentoo Service instability June 28 - 29 To: Gentoo Dev , gentoo-project Content-Type: multipart/alternative; boundary="000000000000af1dbe05c5deface" X-Archives-Salt: 4be2a9de-3810-406e-98ba-580797a2eae8 X-Archives-Hash: 462d4fd7118a9590ea0bbd8648dde24d --000000000000af1dbe05c5deface Content-Type: text/plain; charset="UTF-8" Hi, Many of you reported that various infra-owned services were not functioning properly these past two days. We had some package upgrades roll out that were bad, and a subset of services were non-functional while we worked through the incident. (a) The upgrades did not appear to cause any data loss, but service availability was reduced (500s / 502s for some HTTP services.) (b) Some hosts may have had problems running some commands. E.g. I know that 'grep' didn't work for a while. If you observed errors related to missing 'GLIBC_2.33' symbols then this was likely related. (c) Many replicated services (including infra-status) rely on gitweb to take updates from git; and gitweb was not up, so they were unable to receive updates. (d) More impacted notes will follow in a postmortem that will come later this week for this incident. Thanks to all of you who reported problems and apologies for the service disruptions. -A --000000000000af1dbe05c5deface Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi,

Many of you reported that various i= nfra-owned services were not functioning properly these past two days. We h= ad some package upgrades roll out that were bad, and a subset of services w= ere non-functional while we worked through the incident.

(a) The upgrades did not appear=C2=A0to cause any data loss, but ser= vice availability was reduced (500s / 502s for some HTTP services.)
(b= ) Some hosts may have had problems running some commands. E.g. I know that = 'grep' didn't work for a while. If you observed errors related = to missing 'GLIBC_2.33' symbols then this was likely related.
(= c) Many replicated services (including infra-status) rely on gitweb to take= updates from git; and gitweb was not up, so they were unable to receive up= dates.
(d) More impacted notes will follow in a postmortem that w= ill come later this week for this incident.

Thanks to al= l of you who reported problems and apologies for the service disruptions.

-A
--000000000000af1dbe05c5deface--