[gentoo-soc] Sort of "ondemand" mirror for distfiles

[gentoo-soc] Sort of "ondemand" mirror for distfiles

[Łukasz Zubkowicz]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi all,

I'm a CS student from Poland interested in taking part in GSoC 2010.
I've been using Gentoo for about 5 years. I've expanded my home network
recently and found some difficulties with keeping things tidy on 5
different boxes with Gentoo. And that's where my idea for this year's
GSoC has come from.
Most of packages I install are common to all of my machines and
therefore I would be happy to download them only once. It would save
bandwidth and time for me and others. Trying to solve this issue with
some ready software I've found two or three solutions, but none of them
satisfying. Starting with the simplest solution, I could just emerge
package on one machine and then copy sources to the others, but it's at
least a little bit stupid. Being smarter, I could export
/usr/portage/distfiles to other machines through NFS, but as far as I
know something about NFS it could raise some synchronization errors if
tried to merge the same packages on more than one machine
simultaneously. Finally, the designated solution: I could just install a
mirror on one of my local machines and use in within my network (just
like I've done it with rsyncing portage tree), but... isn't it to
expensive? It would be fine if I had a big network with plenty of
storage, but (keeping in mind my small home network) spending about
200GB (that number I've found somewhere) on accumulating all packages
doesn't seem to me a good idea. It's quite substantial amount of disk
space for me.
I feel like there was lack of solution for people like me and I bet
there are more such users who have about 5 to 15 machines under their
control. My idea is to develop kind of mirror which would act just like
the real "big" mirror, but not keep all of packages. Instead of that it
would download a package when it's needed for the first time and make
somehow wait machine which is asking for it (and others which will ask
for it in time before it's fully downloaded). Of course it's only a
draft, but I just wanted you to feel my idea.
I wait for your comments and responses. I realize that I may be terribly
wrong and there is already such thing which I need, but I have spent
some time on seeking and haven't found it. In such case I'm sorry for
wasting your time and... please let me know about it  :) .

- --
Best wishes, Lukasz Zubkowicz
http://students.mimuw.edu.pl/~lz248258

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkuvO1QACgkQ+6A5fji6urEOAgCgigzFa+WY+nz86lA0ErZPpxlj
ZH4AoO9UctF51PAOpLVOzjAibbUYiCSE
=wj/o
-----END PGP SIGNATURE-----

Re: [gentoo-soc] Sort of "ondemand" mirror for distfiles

[Auke Booij]

What are you describing is just a caching proxy. Try Squid.

2010/3/28 Łukasz Zubkowicz <lukasek@students.mimuw.edu.pl>:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hi all,
>
> I'm a CS student from Poland interested in taking part in GSoC 2010.
> I've been using Gentoo for about 5 years. I've expanded my home network
> recently and found some difficulties with keeping things tidy on 5
> different boxes with Gentoo. And that's where my idea for this year's
> GSoC has come from.
> Most of packages I install are common to all of my machines and
> therefore I would be happy to download them only once. It would save
> bandwidth and time for me and others. Trying to solve this issue with
> some ready software I've found two or three solutions, but none of them
> satisfying. Starting with the simplest solution, I could just emerge
> package on one machine and then copy sources to the others, but it's at
> least a little bit stupid. Being smarter, I could export
> /usr/portage/distfiles to other machines through NFS, but as far as I
> know something about NFS it could raise some synchronization errors if
> tried to merge the same packages on more than one machine
> simultaneously. Finally, the designated solution: I could just install a
> mirror on one of my local machines and use in within my network (just
> like I've done it with rsyncing portage tree), but... isn't it to
> expensive? It would be fine if I had a big network with plenty of
> storage, but (keeping in mind my small home network) spending about
> 200GB (that number I've found somewhere) on accumulating all packages
> doesn't seem to me a good idea. It's quite substantial amount of disk
> space for me.
> I feel like there was lack of solution for people like me and I bet
> there are more such users who have about 5 to 15 machines under their
> control. My idea is to develop kind of mirror which would act just like
> the real "big" mirror, but not keep all of packages. Instead of that it
> would download a package when it's needed for the first time and make
> somehow wait machine which is asking for it (and others which will ask
> for it in time before it's fully downloaded). Of course it's only a
> draft, but I just wanted you to feel my idea.
> I wait for your comments and responses. I realize that I may be terribly
> wrong and there is already such thing which I need, but I have spent
> some time on seeking and haven't found it. In such case I'm sorry for
> wasting your time and... please let me know about it  :) .
>
> - --
> Best wishes, Lukasz Zubkowicz
> http://students.mimuw.edu.pl/~lz248258
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.9 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAkuvO1QACgkQ+6A5fji6urEOAgCgigzFa+WY+nz86lA0ErZPpxlj
> ZH4AoO9UctF51PAOpLVOzjAibbUYiCSE
> =wj/o
> -----END PGP SIGNATURE-----
>
>

Re: [gentoo-soc] Sort of "ondemand" mirror for distfiles

[Petteri Räty]

[-- Attachment #1: Type: text/plain, Size: 713 bytes --]

On 03/28/2010 02:19 PM, Łukasz Zubkowicz wrote:

> I feel like there was lack of solution for people like me and I bet
> there are more such users who have about 5 to 15 machines under their
> control. My idea is to develop kind of mirror which would act just like
> the real "big" mirror, but not keep all of packages. Instead of that it
> would download a package when it's needed for the first time and make
> somehow wait machine which is asking for it (and others which will ask
> for it in time before it's fully downloaded). Of course it's only a
> draft, but I just wanted you to feel my idea.

Sounds like you want to setup a HTTP proxy server with long time caching.

Regards,
Petteri


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 900 bytes --]

Re: [gentoo-soc] Sort of "ondemand" mirror for distfiles

[Łukasz Zubkowicz]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Thank you all for your fast and precise responses. They will be
definitely useful.
I will try some ready idea for GSoC then. I guess now I should try to
tackle some mentor about it, am I right?

- --
Lukasz Zubkowicz

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkuvcZYACgkQ+6A5fji6urGoDACgqv4o7/hLWoVz5WhDpGqgwvbv
BKUAn2YErwPcjVrXcwgBYYJNKVlMUnND
=F1ud
-----END PGP SIGNATURE-----

Re: [gentoo-soc] Sort of "ondemand" mirror for distfiles

[Brian Dolbec]

[-- Attachment #1: Type: text/plain, Size: 1312 bytes --]

On Sun, 2010-03-28 at 15:29 +0300, Petteri Räty wrote:
> On 03/28/2010 02:19 PM, Łukasz Zubkowicz wrote:
> 
> > I feel like there was lack of solution for people like me and I bet
> > there are more such users who have about 5 to 15 machines under their
> > control. My idea is to develop kind of mirror which would act just like
> > the real "big" mirror, but not keep all of packages. Instead of that it
> > would download a package when it's needed for the first time and make
> > somehow wait machine which is asking for it (and others which will ask
> > for it in time before it's fully downloaded). Of course it's only a
> > draft, but I just wanted you to feel my idea.
> 
> Sounds like you want to setup a HTTP proxy server with long time caching.
> 
> Regards,
> Petteri
> 


Continuing that, there are many hits on the gentoo forums about this,
one of which seems to run the gammit of possible solutions, including
setting up a proxy.  first page about 13 posts down is a caching setup,
but there are more solutions including ftp, rsync,...

Post subject: HOWTO: Central Gentoo Mirror for your Internal Network

http://forums.gentoo.org/viewtopic-t-59134-start-0-postdays-0-postorder-asc-highlight-caching+distfiles+proxy.html

-- 
Brian Dolbec <brian.dolbec@gmail.com>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-soc] Sort of "ondemand" mirror for distfiles

[Brian Dolbec]

[-- Attachment #1: Type: text/plain, Size: 400 bytes --]

oops, I should have included this one:

 Post subject: HOWTO:Download Cache for your LAN-Http-Replicator (ver 3.0)
http://forums.gentoo.org/viewtopic-t-173226-start-0-postdays-0-postorder-asc-highlight-caching+distfiles+proxy.html

and has an ebuild in the portage tree:

http://packages.gentoo.org/package/net-proxy/http-replicator?arches=all

-- 
Brian Dolbec <brian.dolbec@gmail.com>


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-soc] Sort of "ondemand" mirror for distfiles

[Auke Booij]

No, mentors will be appointed after applications are in. You should
discuss another idea, and write an application for it (which you can
submit starting tomorrow).

PS: fix your timezone.

2010/3/28 Łukasz Zubkowicz <lukasek@students.mimuw.edu.pl>:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Thank you all for your fast and precise responses. They will be
> definitely useful.
> I will try some ready idea for GSoC then. I guess now I should try to
> tackle some mentor about it, am I right?
>
> - --
> Lukasz Zubkowicz
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.9 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAkuvcZYACgkQ+6A5fji6urGoDACgqv4o7/hLWoVz5WhDpGqgwvbv
> BKUAn2YErwPcjVrXcwgBYYJNKVlMUnND
> =F1ud
> -----END PGP SIGNATURE-----
>
>

Re: [gentoo-soc] Sort of "ondemand" mirror for distfiles

[Reith Ameretat]

Hi,
I have not any professional project. i have projects in C/C++, PHP and
Bash and i'm familiar with Python, but I'm afraid of cant complete a
project; if i take a project and don't complete it, what happens? i
lose somebody's chance?
It is legal join Gentoo GSoC from counties that Google closed servers on it?
excuse me if i am wrong place for this question.

regards
//sorry for my poor English

On 3/28/10, Auke Booij <auke@tulcod.com> wrote:
> No, mentors will be appointed after applications are in. You should
> discuss another idea, and write an application for it (which you can
> submit starting tomorrow).
>
> PS: fix your timezone.
>
> 2010/3/28 Łukasz Zubkowicz <lukasek@students.mimuw.edu.pl>:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Thank you all for your fast and precise responses. They will be
>> definitely useful.
>> I will try some ready idea for GSoC then. I guess now I should try to
>> tackle some mentor about it, am I right?
>>
>> - --
>> Lukasz Zubkowicz
>>
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG v2.0.9 (GNU/Linux)
>> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>>
>> iEYEARECAAYFAkuvcZYACgkQ+6A5fji6urGoDACgqv4o7/hLWoVz5WhDpGqgwvbv
>> BKUAn2YErwPcjVrXcwgBYYJNKVlMUnND
>> =F1ud
>> -----END PGP SIGNATURE-----
>>
>>
>
>

Re: [gentoo-soc] Sort of "ondemand" mirror for distfiles

[Reith Ameretat]

Hi,
I have not any professional project. i have projects in C/C++, PHP and
Bash and i'm familiar with Python, but I'm afraid of cant complete a
project; if i take a project and don't complete it, what happens? i
lose somebody's chance?
It is legal join Gentoo GSoC from counties that Google closed servers on it?
excuse me if i am wrong place for this question.

regards
//sorry for my poor English

Re: [gentoo-soc] Sort of "ondemand" mirror for distfiles

[Auke Booij]

On Sun, Mar 28, 2010 at 7:20 PM, Reith Ameretat
<ameretat.reith@gmail.com> wrote:
> Hi,
> I have not any professional project. i have projects in C/C++, PHP and
> Bash and i'm familiar with Python, but I'm afraid of cant complete a
> project; if i take a project and don't complete it, what happens? i
> lose somebody's chance?
> It is legal join Gentoo GSoC from counties that Google closed servers on it?
> excuse me if i am wrong place for this question.
>
> regards
> //sorry for my poor English
>
>

It's hard enough to finish a project *with* proper knowledge, maybe
you should wait a year. As for your questions:

-yes, you would be using a slot someone else won't be able to use
-see the gsoc faq for countries you cannot live in to join (i guess
china is one of them)

Re: [gentoo-soc] Sort of "ondemand" mirror for distfiles

[Reith Ameretat]

OK, i come from Iran (unfortunately Islamic regime!) and can't join. :(
but i try develop one of ideas on my own and going deeper to Gentoo!

thanks dear ;)
//sorry for my poor English

On 3/28/10, Auke Booij <auke@tulcod.com> wrote:
> On Sun, Mar 28, 2010 at 7:20 PM, Reith Ameretat
> <ameretat.reith@gmail.com> wrote:
>> Hi,
>> I have not any professional project. i have projects in C/C++, PHP and
>> Bash and i'm familiar with Python, but I'm afraid of cant complete a
>> project; if i take a project and don't complete it, what happens? i
>> lose somebody's chance?
>> It is legal join Gentoo GSoC from counties that Google closed servers on
>> it?
>> excuse me if i am wrong place for this question.
>>
>> regards
>> //sorry for my poor English
>>
>>
>
> It's hard enough to finish a project *with* proper knowledge, maybe
> you should wait a year. As for your questions:
>
> -yes, you would be using a slot someone else won't be able to use
> -see the gsoc faq for countries you cannot live in to join (i guess
> china is one of them)
>
>