[gentoo-dev] archives.gentoo.org linking brokenness

[gentoo-dev] archives.gentoo.org linking brokenness

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 515 bytes --]

Hi folks,

Due to some mhonarc brokenness, all existing external links to archives
messages have become invalidated. We're looking at fixing it in such a
way that it doesn't happen again (consistent hashes instead of
sequential numbers), but it will be a day or two - restoring the old
ones is not possible at all.

Sorry for any inconvenience.

-- 
Robin Hugh Johnson
Gentoo Linux Developer & Infra Guy
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

[-- Attachment #2: Type: application/pgp-signature, Size: 329 bytes --]

Re: [gentoo-dev] archives.gentoo.org linking brokenness

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 1100 bytes --]

On Sun, Jan 27, 2008 at 01:49:46AM -0800, Robin H. Johnson wrote:
> Due to some mhonarc brokenness, all existing external links to archives
> messages have become invalidated. We're looking at fixing it in such a
> way that it doesn't happen again (consistent hashes instead of
> sequential numbers), but it will be a day or two - restoring the old
> ones is not possible at all.
Ok, all of the archives are fixed.

Here's an example of a new link:
http://archives.gentoo.org/gentoo-gwn/msg_916064a3a0913217d311541108d04782.xml

That's going to be constant forever now, regardless of how many times
mhonarc rebuilds it's entire index.

However, please note that any attachments linked from a mail might
change their URLs depending on mhonarc, so you should only link directly
to the mail itself.

If you see any problems with mails, if you see any new problems with
archive mails, please file a bug for Infrastructure.

-- 
Robin Hugh Johnson
Gentoo Linux Developer & Infra Guy
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

[-- Attachment #2: Type: application/pgp-signature, Size: 329 bytes --]

Re: [gentoo-dev] archives.gentoo.org linking brokenness

[Chris Gianelloni]

[-- Attachment #1: Type: text/plain, Size: 451 bytes --]

On Sun, 2008-01-27 at 01:49 -0800, Robin H. Johnson wrote:
> sequential numbers), but it will be a day or two - restoring the old
> ones is not possible at all.

Umm... OK.  What's the new format?  Do we just have to go an manually
change every single link we've ever made to anything on archives?  How
are we going to find the articles if the old links don't work?

-- 
Chris Gianelloni
Release Engineering Strategic Lead
Games Developer

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] archives.gentoo.org linking brokenness

[Alec Warner]

On 1/27/08, Chris Gianelloni <wolf31o2@gentoo.org> wrote:
> On Sun, 2008-01-27 at 01:49 -0800, Robin H. Johnson wrote:
> > sequential numbers), but it will be a day or two - restoring the old
> > ones is not possible at all.
>
> Umm... OK.  What's the new format?  Do we just have to go an manually
> change every single link we've ever made to anything on archives?  How
> are we going to find the articles if the old links don't work?
>

So slight postmortem:

The url per e-mail depended on the order of processing for messages:
the first message got 1, the second got 2 and so forth.  The upgrade
ruined that database.

To prevent this in the future, Robin created a scheme that is not
dependent on the order of processing (which is harder that it sounds
as we discussed on -infra last night).  We don't have a mapping from
old e-mails to new ones.  If you know what the title was you can
probably fish the message out of the archives; otherwise you are
probably SOL.

The bonus out of all this is that the new storage format could get
destroyed and we wouldn't have this problem (archives would just be
down for two days while infra regenerates the indexes.

-Alec
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] archives.gentoo.org linking brokenness

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 1091 bytes --]

On Sun, Jan 27, 2008 at 05:31:36PM -0800, Chris Gianelloni wrote:
> On Sun, 2008-01-27 at 01:49 -0800, Robin H. Johnson wrote:
> > sequential numbers), but it will be a day or two - restoring the old
> > ones is not possible at all.
> Umm... OK.  What's the new format?
For example:
http://archives.gentoo.org/gentoo-nfp/msg_d358a8903925e46412a6dd5659f3429a.xml

> Do we just have to go an manually change every single link we've ever
> made to anything on archives?  How are we going to find the articles
> if the old links don't work?
Unfortunately the hard way, by hand.

The new ones use a hash of (salt, Received, Date, Message-ID, From,
Subject, List-Id), and it's guaranteed to be the same regardless of
mhonarc deciding to ignore it's existing database.

Previously, mhonarc generated sequential ids before based on listdir()
order, which changed as archives have lived on ~7 boxes over it's
lifetime.

-- 
Robin Hugh Johnson
Gentoo Linux Developer & Infra Guy
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

[-- Attachment #2: Type: application/pgp-signature, Size: 329 bytes --]

Re: [gentoo-dev] archives.gentoo.org linking brokenness

[Chris Gianelloni]

[-- Attachment #1: Type: text/plain, Size: 738 bytes --]

On Sun, 2008-01-27 at 17:57 -0800, Robin H. Johnson wrote:
> > Do we just have to go an manually change every single link we've ever
> > made to anything on archives?  How are we going to find the articles
> > if the old links don't work?
> Unfortunately the hard way, by hand.

Ouch.

> The new ones use a hash of (salt, Received, Date, Message-ID, From,
> Subject, List-Id), and it's guaranteed to be the same regardless of
> mhonarc deciding to ignore it's existing database.

OK.  At least now they're designed so that this cannot happen again.
Thanks for taking the time to code that part up.  I hope that you sent
that patch upstream!  ;]

-- 
Chris Gianelloni
Release Engineering Strategic Lead
Games Developer

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]