[gentoo-dev] Adding large files to the mirrors?

[gentoo-dev] Adding large files to the mirrors?

[Mike Auty]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi there,

I'm updating the app-crypt/ophcrack-tables package to include the new
tables available from their site.  These are basically just additional
data packages that can be useful with the app-crypt/ophcrack package,
but they're very large.  Including all of them will come just short of
30Gb.  I don't know whether it's ok to add that to our mirrors, or add
RESTRICT="mirror"?

Also, these take up a lot of space in distfiles.  Does anyone know of
a clever way to allow them to be installed without also stashing a
copy in distfiles (symlinking to the distfiles directory is a no go,
because each "set" needs its own files to have the same name as the
other sets)?

I guess it's a bit of an infra question, but thought I'd ask here in
case anyone else has found themselves in a similar situation.  For
more specifics about what the package is like/for see the test ebuild
at [1]...

Mike  5:)

[1]
http://git.overlays.gentoo.org/gitweb/?p=dev/ikelos.git;a=blob;f=app-crypt/ophcrack-tables/ophcrack-tables-1.1.ebuild
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)

iEYEARECAAYFAlJdzaQACgkQu7rWomwgFXqq1wCePSA0S03pVVxeesqAbd2faO4h
2wgAn0P0y5dvhsArRnFdFajrApM5C4YV
=fim1
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Adding large files to the mirrors?

[Vicente Olivert Riera]

[-- Attachment #1: Type: text/plain, Size: 1612 bytes --]

Can't the users download and install those tables manually from the
official ophcrack website/server? If so, maybe adding the instructions
in the pkg_postinst() phase of app-crypt/ophcrack ebuild would be
enough. In that case, it wouldn't make sense having
app-crypt/ophcrack-tables in the portage tree and should be treecleaned.

**************************
Vicente Olivert Riera
Gentoo Linux Developer
ID GnuPG: 5AE9E7B2E9BBCBA8
**************************

On 10/16/2013 01:20 AM, Mike Auty wrote:
> Hi there,
> 
> I'm updating the app-crypt/ophcrack-tables package to include the new
> tables available from their site.  These are basically just additional
> data packages that can be useful with the app-crypt/ophcrack package,
> but they're very large.  Including all of them will come just short of
> 30Gb.  I don't know whether it's ok to add that to our mirrors, or add
> RESTRICT="mirror"?
> 
> Also, these take up a lot of space in distfiles.  Does anyone know of
> a clever way to allow them to be installed without also stashing a
> copy in distfiles (symlinking to the distfiles directory is a no go,
> because each "set" needs its own files to have the same name as the
> other sets)?
> 
> I guess it's a bit of an infra question, but thought I'd ask here in
> case anyone else has found themselves in a similar situation.  For
> more specifics about what the package is like/for see the test ebuild
> at [1]...
> 
> Mike  5:)
> 
> [1]
> http://git.overlays.gentoo.org/gitweb/?p=dev/ikelos.git;a=blob;f=app-crypt/ophcrack-tables/ophcrack-tables-1.1.ebuild
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 263 bytes --]

Re: [gentoo-dev] Adding large files to the mirrors?

[Patrick Lauer]

On 10/16/2013 07:20 AM, Mike Auty wrote:
> Hi there,
> 
> I'm updating the app-crypt/ophcrack-tables package to include the new
> tables available from their site.  These are basically just additional
> data packages that can be useful with the app-crypt/ophcrack package,
> but they're very large.  Including all of them will come just short of
> 30Gb.  I don't know whether it's ok to add that to our mirrors, or add
> RESTRICT="mirror"?
Add a ophcrack-tables-big package that is mirror-restricted
(that way the rest of ophcrack-tables is unaffected)

Pull it in with a useflag ("big-tables" ?)

The "double size" problem with a copy in distfiles and livefs might be
unavoidable - I have no good idea how to fix that

Re: [gentoo-dev] Adding large files to the mirrors?

[Rick "Zero_Chaos" Farina]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 10/15/2013 07:29 PM, Vicente Olivert Riera wrote:
> Can't the users download and install those tables manually from the
> official ophcrack website/server? If so, maybe adding the instructions
> in the pkg_postinst() phase of app-crypt/ophcrack ebuild would be
> enough. In that case, it wouldn't make sense having
> app-crypt/ophcrack-tables in the portage tree and should be treecleaned.

I think this may be the sanest option *for the large tables*.  The small
tables are great, and you can einfo a message about the large ones.

- -Zero
> 
> **************************
> Vicente Olivert Riera
> Gentoo Linux Developer
> ID GnuPG: 5AE9E7B2E9BBCBA8
> **************************
> 
> On 10/16/2013 01:20 AM, Mike Auty wrote:
>> Hi there,
>>
>> I'm updating the app-crypt/ophcrack-tables package to include the new
>> tables available from their site.  These are basically just additional
>> data packages that can be useful with the app-crypt/ophcrack package,
>> but they're very large.  Including all of them will come just short of
>> 30Gb.  I don't know whether it's ok to add that to our mirrors, or add
>> RESTRICT="mirror"?
>>
>> Also, these take up a lot of space in distfiles.  Does anyone know of
>> a clever way to allow them to be installed without also stashing a
>> copy in distfiles (symlinking to the distfiles directory is a no go,
>> because each "set" needs its own files to have the same name as the
>> other sets)?
>>
>> I guess it's a bit of an infra question, but thought I'd ask here in
>> case anyone else has found themselves in a similar situation.  For
>> more specifics about what the package is like/for see the test ebuild
>> at [1]...
>>
>> Mike  5:)
>>
>> [1]
>> http://git.overlays.gentoo.org/gitweb/?p=dev/ikelos.git;a=blob;f=app-crypt/ophcrack-tables/ophcrack-tables-1.1.ebuild
>>
> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJSXertAAoJEKXdFCfdEflKICcP/2I+L61IWFk2Ackp+SqGgCHV
NKuqCAFeoGywXIIgoFI/eWrOfHBQwyNKrNL6lRcBcYQXJy96yR/caRSPS+zPsEoU
B4pJ13KJg0ipCriZUXHMtAOGcBOJRr40ItW0L3qT8ICl8TIFHeaDuqx5Sbl7GuHQ
Ewqt5CdN4g7lpp5XROmz/LKHr3fHi+KNMPj7GrKd/guoALqmJvclmyWmmrovzr91
1ueSPghfSF78AHjEj8X7H7702/waLeOpww2MZkXfAY+H3Chw2AuTR0JC7xiSpMvX
Dq4rtLUmdeobozBwNDkczoVGcfxoR+AguNlZUIz2Vij2A3XxVznqEYHY+VIbwZVm
o5ut+P58GJBBct5TSi1IMdq2Rfrg7xP16e757Ki1FDSbQamaDhl8SETPbPQ0YrjV
J9m8JQ2tlvgMi20Z+UA5QVWT9jZq8s8/26eDkeZNzYilmNZ8sSWUAlpNOgC0xsSG
n4RN1COEVcpyxhyFjjMLB+GjxVa14fCgpZ+mcd/VwRmxU7scnw8YweWALamnAEbB
FmcBB6tedeeA6TWcXtr3HGbfwt37KdHHo8xDJpJVl42dk08FKtV5sSY4vA2MDkL9
pyI4r4qo076KT9N4ZIsljPf3pGAYe3ERnIdNFgEci8UJBapoHh2GnLuTAN+EFjDt
GcEiQLGWWMK9TuVL/9Jj
=kOqk
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Adding large files to the mirrors?

[Chí-Thanh Christopher Nguyễn]

Mike Auty schrieb:
> Hi there,
> 
> I'm updating the app-crypt/ophcrack-tables package to include the new 
> tables available from their site.  These are basically just additional 
> data packages that can be useful with the app-crypt/ophcrack package, 
> but they're very large.  Including all of them will come just short of 
> 30Gb.  I don't know whether it's ok to add that to our mirrors, or add 
> RESTRICT="mirror"?

Currently they are on sourceforge mirrors AFAICT, so I expect that upstream
will not mind if Gentoo users download from there.

> Also, these take up a lot of space in distfiles.  Does anyone know of a
> clever way to allow them to be installed without also stashing a copy in
> distfiles (symlinking to the distfiles directory is a no go, because
> each "set" needs its own files to have the same name as the other
> sets)?

I suggest that you add a "large" USE flag, and if it is enabled, download
and install the whole thing. If disabled, just print the URLs in a log
message, so the user can download themselves if they wish.


Best regards,
Chí-Thanh Christopher Nguyễn

Re: [gentoo-dev] Adding large files to the mirrors?

[Mike Auty]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hiya,

First off, thanks everybody for the suggestions.  It seems like people
aren't keen to mirror the large tables on our mirrors, but no one's
mentioned a reason for not doing so.  Is there an infra reason behind
that or did everyone just take my caution and run with it?

On 16/10/13 02:40, Chí-Thanh Christopher Nguyễn wrote:
> I suggest that you add a "large" USE flag, and if it is enabled,
> download and install the whole thing. If disabled, just print the
> URLs in a log message, so the user can download themselves if they
> wish.

So downloading them manually is a pain (the larger tables aren't in a
single zip, they're split amongst 12 files for each table), and the
ebuild to do the downloading is already built.  I'll include a
postinst note indicating that the tables are getting stored twice, and
I should even be able to give them an indication of how much space
they're wasting.  I'll also provide a URL they can visit if they want
to download manually instead.

The question then becomes, do I split the ebuild into two (giving us
three ebuilds for what is otherwise a tiny package) just so some SRCs
are mirror-restricted and others aren't?  All of that hinges on
whether our servers can handle the extra size...

Mike  5:)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)

iEYEARECAAYFAlJeWR4ACgkQu7rWomwgFXq8bQCfRde/Tg7sAirqT05d5spckC+s
fUIAoIH1SlXvLmM3CqM0x1vQN0oPdiKi
=qqsa
-----END PGP SIGNATURE-----

[gentoo-dev] Re: Adding large files to the mirrors?

[Duncan]

Mike Auty posted on Wed, 16 Oct 2013 10:15:10 +0100 as excerpted:

> First off, thanks everybody for the suggestions.  It seems like people
> aren't keen to mirror the large tables on our mirrors, but no one's
> mentioned a reason for not doing so.  Is there an infra reason behind
> that or did everyone just take my caution and run with it?

Based on previous related discussion, it's simply a reluctance to throw 
that much extra resource requirements at the folks who are after all 
/volunteering/ to host gentoo mirrors.  We're invited guests whose 
continued presence is at their pleasure, and we shouldn't be taking undue 
advantage of the privileges we're given or the resources required to 
support us as such.

If it's a core requirement, no problem.  But the further from core 
requirement it gets the more sensitive infra seems to be about non-
trivial space requirements, and any way you look at it, 30 gigs of 
rainbow tables are both non-core and non-trivial, especially with the 
split package and no-mirror on the big package solution, so...

Beyond that, I suppose it'd be a topic for the mirror list.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Adding large files to the mirrors?

[Francesco R.]

Il 16/10/2013 11:15, Mike Auty ha scritto:
> Hiya,
[snip]
> So downloading them manually is a pain (the larger tables aren't in a
> single zip, they're split amongst 12 files for each table), and the
> ebuild to do the downloading is already built.  I'll include a
> postinst note indicating that the tables are getting stored twice, and
> I should even be able to give them an indication of how much space
> they're wasting.  I'll also provide a URL they can visit if they want
> to download manually instead.
>
> The question then becomes, do I split the ebuild into two (giving us
> three ebuilds for what is otherwise a tiny package) just so some SRCs
> are mirror-restricted and others aren't?  All of that hinges on
> whether our servers can handle the extra size...
>
> Mike  5:)
Add a bash script that can download all the files and mention it in
postinst.
the script will download all the files in current directory and put them
in the correct place.

Re: [gentoo-dev] Re: Adding large files to the mirrors?

[Mike Auty]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1



On 16/10/13 12:01, Duncan wrote:
> If it's a core requirement, no problem.  But the further from core
>  requirement it gets the more sensitive infra seems to be about
> non- trivial space requirements, and any way you look at it, 30
> gigs of rainbow tables are both non-core and non-trivial,
> especially with the split package and no-mirror on the big package
> solution, so...

No, that's fine.  I wanted to check, because I don't take the
resources for granted, but I do see us as providing a mirroring
service for open source projects (to a degree), and no one so far had
put forward an opinion about how infra would feel about it.

Anyway, Francesco made an excellent point, and I think I'll be going
with the additional bash script for grabbing the large tables...

Thanks for you help everyone!  5:)

Mike  5:)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)

iEYEARECAAYFAlJfGKkACgkQu7rWomwgFXo7dwCeIgp2WTS5vaBpG+zNTzdw0Q6g
4ocAoLUlnjIcZzcWzJ8Va1P5mdx65moP
=Y1m5
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Adding large files to the mirrors?

[Mike Auty]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hiya,

On 16/10/13 12:54, Francesco R. wrote:
> Il 16/10/2013 11:15, Mike Auty ha scritto: Add a bash script that
> can download all the files and mention it in postinst. the script
> will download all the files in current directory and put them in
> the correct place.

That's an absolutely awesome suggestion, I'll do that!  Thanks!  5:)

Mike  5:)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)

iEYEARECAAYFAlJfGK8ACgkQu7rWomwgFXrUIACfTTUcmqbApW4CqMlNd3YLC5yp
pLwAn0hUHOEoG+noOMtgHqhBtKFb6tbT
=E9tY
-----END PGP SIGNATURE-----