[gentoo-dev] upcoming mirror cleansing

[gentoo-dev] upcoming mirror cleansing

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 1422 bytes --]

Hola all.

So mirror-dist is ready to go, infra side of it being set for a final 
testing run then switching it live if things go fine after next 
weekend.

Why this matters- around 10,000 files out of 28,600 files will be 
removed from the mirrors network.  Either

A) no ebuild claims that distfile.  it's orphaned on our mirrors
B) RESTRICT="fetch" is set for the ebuild.  We don't mirror those 
   files.
C) RESTRICT="mirror" is set for the ebuild.  Again, we don't mirror 
   those files, in this case we defer to another network (I don't make 
   the rules, that's just how it's done)

Bugs may exist, but I'd suggest you take a hard look at the reasons 
above to verify it's actually a bug.  Should you find one, kindly 
email me.

A quicky report of flies that'll be ixnayed is available at 
http://dev.gentoo.org/~ferringb/failure.xml 
Sorry about the size, it'll come down after the 10,000 deletion is 
swallowed.  The deletion dates should be ignored.  They're accurate 
for the testing, but aren't the actual dates when the files will be 
removed from the mirrors (the deletion dates will be off till this is 
live).

Aside from that, kindly check the failed fetches on the list.  If one 
of your ebuilds is in that list, either no valid URI could be found, 
or you doffed the SRC_URI for it (mirror:// w/out a mirror has been 
common).

That's all.  Eat the pudding.
~brian

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] upcoming mirror cleansing

[Alin Nastac]

[-- Attachment #1: Type: text/plain, Size: 687 bytes --]

Brian Harring wrote:

>Why this matters- around 10,000 files out of 28,600 files will be 
>removed from the mirrors network.  Either
>
>A) no ebuild claims that distfile.  it's orphaned on our mirrors
>B) RESTRICT="fetch" is set for the ebuild.  We don't mirror those 
>   files.
>C) RESTRICT="mirror" is set for the ebuild.  Again, we don't mirror 
>   those files, in this case we defer to another network (I don't make 
>   the rules, that's just how it's done)
>
>  
>
You should not erase files newer than an arbitrary amount of time (a
week maybe?).
Don't forget about dev.g.o:/space/distfiles-local; a dev first put the
tarball in that folder _then_ submit the ebuild who use it.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 256 bytes --]

Re: [gentoo-dev] upcoming mirror cleansing

[Brian Harring]

On Sat, Apr 23, 2005 at 05:54:40PM +0300, Alin Nastac wrote:
> Brian Harring wrote:
> 
> >Why this matters- around 10,000 files out of 28,600 files will be 
> >removed from the mirrors network.  Either
> >
> >A) no ebuild claims that distfile.  it's orphaned on our mirrors
> >B) RESTRICT="fetch" is set for the ebuild.  We don't mirror those 
> >   files.
> >C) RESTRICT="mirror" is set for the ebuild.  Again, we don't mirror 
> >   those files, in this case we defer to another network (I don't make 
> >   the rules, that's just how it's done)
> >
> >  
> >
> You should not erase files newer than an arbitrary amount of time (a
> week maybe?).
> Don't forget about dev.g.o:/space/distfiles-local; a dev first put the
> tarball in that folder _then_ submit the ebuild who use it.
Files that are orphaned from ebuilds (this includes distfiles-local 
uploaded files) are marked for death, and have a week till they're 
removed from the mirrors.  They're stored for an additional 2 weeks, 
then deleted.

There's a bit more to it then that, but the short version is that 
there are reasaonable delays built into the auto cleansing.
~brian
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] upcoming mirror cleansing

[Donnie Berkholz]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Brian Harring wrote:
> A quicky report of flies that'll be ixnayed is available at 
> http://dev.gentoo.org/~ferringb/failure.xml 

Thanks for making that page! I just searched it and found two packages I
cared about with wrong SRC_URI's, fixed one and filed a bug for the other.

It would be even cooler if it matched stuff up with metadata.xml!

Thanks,
Donnie
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)

iD8DBQFCap+GXVaO67S1rtsRAsFqAJ9YazXDJ6iNQ0+Ovy9JoGDjR5UqKwCePEcd
cc5x8VYmEPFxgO9YNsu0nVk=
=k96f
-----END PGP SIGNATURE-----
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] upcoming mirror cleansing

[Brian Harring]

On Sat, Apr 23, 2005 at 12:18:30PM -0700, Donnie Berkholz wrote:
> Brian Harring wrote:
> > A quicky report of flies that'll be ixnayed is available at 
> > http://dev.gentoo.org/~ferringb/failure.xml 
> 
> Thanks for making that page! I just searched it and found two packages I
> cared about with wrong SRC_URI's, fixed one and filed a bug for the other.
> 
> It would be even cooler if it matched stuff up with metadata.xml!
Intending on pulling herd/maintainer information and mapping it in.

That said, my intention for it is to bind it to the packages.g.o 
database.  I need persistance, since it only notes when a file is 
orphaned; need to be able to look at a general tree history, and know 
what ebuilds owned a file, thus getting to the metadata.xml.  I could 
write this myself, track the data, but I'm not writing Yet Another 
!*@#*ing Local TreeDB.  Would rather unify the script (and a few 
other ideas/scripts I'm kicking around) against a generalized packages 
db...

So yeah, moving in that direction :)
~brian
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] upcoming mirror cleansing

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 435 bytes --]

On Sat, 23 Apr 2005 08:49:59 -0500 Brian Harring <ferringb@gentoo.org>
wrote:
| Why this matters- around 10,000 files out of 28,600 files will be 
| removed from the mirrors network.

I just had a random thought. Have our GLEP 19 people thought about this
at all?

-- 
Ciaran McCreesh : Gentoo Developer (Vim, Fluxbox, shell tools)
Mail            : ciaranm at gentoo.org
Web             : http://dev.gentoo.org/~ciaranm


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] upcoming mirror cleansing

[Olivier Crête]

[-- Attachment #1: Type: text/plain, Size: 573 bytes --]

On Sat, 2005-23-04 at 23:29 +0100, Ciaran McCreesh wrote:
> On Sat, 23 Apr 2005 08:49:59 -0500 Brian Harring <ferringb@gentoo.org>
> wrote:
> | Why this matters- around 10,000 files out of 28,600 files will be 
> | removed from the mirrors network.
> 
> I just had a random thought. Have our GLEP 19 people thought about this
> at all?

Ideally the clearing script would also very that the file isnt needed by
something in the GLEP19 tree, right ? But since there is no official
GLEP19 tree yet...

-- 
Olivier Crête
tester@gentoo.org
x86 Security Liaison

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] upcoming mirror cleansing

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 633 bytes --]

On Sat, Apr 23, 2005 at 08:49:59AM -0500, Brian Harring wrote:
> Hola all.
[snip]

under 'Deletions for Sunday May 01 2005'
unknown:
	portage-2.0.51.20.tar.bz2 sandbox-1.2.tar.bz2
Perhaps a major glitch here, since portage-2.0.51.20 is the latest version?

Also, will the script be re-run before actual deletions take place? (I'm
tracking down instances of nomirror that shouldn't be there).

-- 
Robin Hugh Johnson
E-Mail     : robbat2@orbis-terrarum.net
Home Page  : http://www.orbis-terrarum.net/?l=people.robbat2
ICQ#       : 30269588 or 41961639
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

[-- Attachment #2: Type: application/pgp-signature, Size: 241 bytes --]

Re: [gentoo-dev] upcoming mirror cleansing

[Daniel]

[-- Attachment #1: Type: text/plain, Size: 890 bytes --]


> Aside from that, kindly check the failed fetches on the list.  If one
> of your ebuilds is in that list, either no valid URI could be found,
> or you doffed the SRC_URI for it (mirror:// w/out a mirror has been
> common).

The following from the list report no valid uri although they WORKFORME.

app-shells/tcsh-6.14 tcsh-6.14.00.tar.gz fetcher return no uris succeeded
ftp://ftp.astron.com/pub/tcsh/tcsh-6.14.00.tar.gz
net-analyzer/netperf-2.4.0_rc3 netperf-2.4.0-rc3.tar.gz fetcher return no uris 
succeeded
ftp://ftp.cup.hp.com/dist/networking/benchmarks/netperf/experimental/netperf-2.4.0-rc3.tar.gz
net-wireless/aircrack-2.1 aircrack-2.1.tgz fetcher return no uris succeeded

http://www.cr0.net:8040/code/network/aircrack-2.1.tgz

Problem with ftp/no port 80 parsing?

-- 
Daniel Black <dragonheart@gentoo.org>
Gentoo Crypto/PPC/dev-embedded/Forensics/NetMon

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] upcoming mirror cleansing

[Brian Harring]

On Sat, Apr 23, 2005 at 04:55:14PM -0700, Robin H. Johnson wrote:
> On Sat, Apr 23, 2005 at 08:49:59AM -0500, Brian Harring wrote:
> > Hola all.
> [snip]
> 
> under 'Deletions for Sunday May 01 2005'
> unknown:
> 	portage-2.0.51.20.tar.bz2 sandbox-1.2.tar.bz2
> Perhaps a major glitch here, since portage-2.0.51.20 is the latest version?
Jason deployed .20 via distfiles-local to get it into the mirrors prior to 
adding the ebuild.
So why is that file in the list of files that are marked for deletion?  
Becuase at the time of the run, _no_ ebuild claimed that file.  We 
didn't push the portage ebuild into the tree until .20 tarball was in 
the mirrors.

So it's valid.  It's also the reason we wait a full week before 
actually removing any file from the mirror tier.


> Also, will the script be re-run before actual deletions take place? (I'm
> tracking down instances of nomirror that shouldn't be there).
Yes.  I'll be restaggering the deletions to run during the first week 
it's live, so you've got a week. :)
What *can* be done, but requires a damn good reason, is that 
individual files can have their deletion times screwed with- same way 
I'm staggering the deletes.

That said, I don't care to do it unless requested.  Mentioning it, 
because in special cases/circumstances it may be needed (just the same 
as in special cases/circumstances, cvs->rsync can be turned off if 
someone breaks the tree).  If a file is marked for deletion, you've 
got a week from detection to either fix the ebuild, or add an ebuild 
in- this however is valid.  The mirror tier isn't a dumping ground :)
~brian
-- 
gentoo-dev@gentoo.org mailing list