[gentoo-portage-dev] emerge feature request: Downloads managed by lock file system

[gentoo-portage-dev] emerge feature request: Downloads managed by lock file system

[Thomas Horsten]

Hi,

I have a request for an emerge feature that I think would be useful for a lot 
of people.

I have quite a fast machine with RAID and 3GHz CPU, and only a 512kbit/s 
Internet connection. When I do an "emerge -u world" most of the time actually 
goes by waiting for packages to download. For most ebuilds, It takes roughly 
the same time to download as to compile.

So to speed up my builds, I usually start an "emerge -f -u world". Once the 
first couple of packages have downloaded I then start an "emerge -u world" in 
another shell.

This works great as long as the fetcher process is ahead of the builder 
process. But if some packages are built too quickly, the builder process 
catches up. It then starts downloading the same file as the downloader 
process, resuming from wherever the downloader had gotten to, so now two 
downloaders are writing to the same file (waste of bandwidth and a possible 
source of corrupted archives).

What I suggest is that emerge will put a lockfile in place when it's 
downloading a file, eg. "name-of-package.tar.bz2.lock", or by using fcntl 
locks. Then if the "builder" catches up, and finds the output file is already 
open, it waits on the lock for the download to complete and then continues 
unpacking after that.

I already love gentoo and the portage system, it's by far the best available, 
but adding this feature would make it even better IMHO.

Regards,

Thomas


--
gentoo-portage-dev@gentoo.org mailing list

Re: [gentoo-portage-dev] emerge feature request: Downloads managed by lock file system

[Simon Mika]

Although I have a computer that is a bit slower I have the similiar problem.
Downloading does not consume much processorpower and athor machine
resources. But compiling does need a lot. Wy don't use always use 2
parallell processes, one that downloads, and one that compiles. The
compiling process would still need to be synced with the downloading, but
that should not be too hard to implement with some semaphore or similiar
solution.

/Simon Mika


On Sat, Feb 07, 2004 at 01:18:47PM +0000, Thomas Horsten wrote:
> Hi,
> 
> I have a request for an emerge feature that I think would be useful for a lot 
> of people.
> 
> I have quite a fast machine with RAID and 3GHz CPU, and only a 512kbit/s 
> Internet connection. When I do an "emerge -u world" most of the time actually 
> goes by waiting for packages to download. For most ebuilds, It takes roughly 
> the same time to download as to compile.
> 
> So to speed up my builds, I usually start an "emerge -f -u world". Once the 
> first couple of packages have downloaded I then start an "emerge -u world" in 
> another shell.
> 
> This works great as long as the fetcher process is ahead of the builder 
> process. But if some packages are built too quickly, the builder process 
> catches up. It then starts downloading the same file as the downloader 
> process, resuming from wherever the downloader had gotten to, so now two 
> downloaders are writing to the same file (waste of bandwidth and a possible 
> source of corrupted archives).
> 
> What I suggest is that emerge will put a lockfile in place when it's 
> downloading a file, eg. "name-of-package.tar.bz2.lock", or by using fcntl 
> locks. Then if the "builder" catches up, and finds the output file is already 
> open, it waits on the lock for the download to complete and then continues 
> unpacking after that.
> 
> I already love gentoo and the portage system, it's by far the best available, 
> but adding this feature would make it even better IMHO.
> 
> Regards,
> 
> Thomas
> 
> 
> --
> gentoo-portage-dev@gentoo.org mailing list
> 

--
gentoo-portage-dev@gentoo.org mailing list

Re: [gentoo-portage-dev] emerge feature request: Downloads managed by lock file system

[Sven Vermeulen]

[-- Attachment #1: Type: text/plain, Size: 1171 bytes --]

On Sat, Feb 07, 2004 at 06:04:18PM +0100, Simon Mika wrote:
> Although I have a computer that is a bit slower I have the similiar problem.
> Downloading does not consume much processorpower and athor machine
> resources. But compiling does need a lot. Wy don't use always use 2
> parallell processes, one that downloads, and one that compiles. The
> compiling process would still need to be synced with the downloading, but
> that should not be too hard to implement with some semaphore or similiar
> solution.

A quick 'n dirty hack in the FETCHCOMMAND should suffice. Something along the
lines of (pseudocode):

FETCHCOMMAND="if exists \${DISTDIR}/`basename \${URL}.lck`; then skip; else \
              touch \${DISTDIR}/`basename \${URL}.lck`; \
              /usr/bin/wget ...; \
              rm \${DISTDIR}/`basename \${URL}.lck`; fi"

This of course as a temporary solution ;)

Wkr,
	Sven Vermeulen

-- 
  FOSDEM 2004               Free and Open Source Developers European Meeting
  21 - 22 februari          Brussels, Belgium          http://www.fosdem.org

  http://www.gentoo.org     Documentation & PR              swift@gentoo.org

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-portage-dev] emerge feature request: Downloads managed by lock file system

[Jeff Smelser]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Saturday 07 February 2004 11:17 am, Sven Vermeulen wrote:
> A quick 'n dirty hack in the FETCHCOMMAND should suffice. Something along
> the lines of (pseudocode):
>
> FETCHCOMMAND="if exists \${DISTDIR}/`basename \${URL}.lck`; then skip; else
> \ touch \${DISTDIR}/`basename \${URL}.lck`; \
>               /usr/bin/wget ...; \
>               rm \${DISTDIR}/`basename \${URL}.lck`; fi"

I tried this, and got:

Calculating world dependencies ...done!
>>> emerge (1 of 4) net-libs/c-client-2002e-r2 to /
>>> Downloading http://gentoo.oregonstate.edu/distfiles/imap-2002e.tar.Z
[c-client-2002e-r2] bash: line 1: exists: command not found
[c-client-2002e-r2] bash: line 2:  : command not found
[c-client-2002e-r2] bash: line 3:  : command not found
[c-client-2002e-r2] bash: line 4:  : command not found
>>> Downloading 
http://distro.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/imap-2002e.tar.Z
[c-client-2002e-r2] bash: line 1: exists: command not found
[c-client-2002e-r2] bash: line 2:  : command not found
[c-client-2002e-r2] bash: line 3:  : command not found
[c-client-2002e-r2] bash: line 4:  : command not found
>>> Downloading ftp://ftp.cac.washington.edu/imap/imap-2002e.tar.Z
[c-client-2002e-r2] bash: line 1: exists: command not found
[c-client-2002e-r2] bash: line 2:  : command not found
[c-client-2002e-r2] bash: line 3:  : command not found
[c-client-2002e-r2] bash: line 4:  : command not found
!!! Couldn't download imap-2002e.tar.Z. Aborting.


- -- 
Failure is not an option. It comes bundled with your Microsoft product.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)

iD8DBQFAJWGfld4MRA3gEwYRAnLwAJ9WgjQRUWddfcpn5yaGPiRV0KPeZQCcC+9G
2U0sZY2CXBiSVEvurcPBNkk=
=wbaz
-----END PGP SIGNATURE-----


--
gentoo-portage-dev@gentoo.org mailing list

Re: [gentoo-portage-dev] emerge feature request: Downloads managed by lock file system

[Thomas Horsten]

On Saturday 07 February 2004 22:07, Jeff Smelser wrote:
> On Saturday 07 February 2004 11:17 am, Sven Vermeulen wrote:
> > A quick 'n dirty hack in the FETCHCOMMAND should suffice. Something along
> > the lines of (pseudocode):
> >
> > FETCHCOMMAND="if exists \${DISTDIR}/`basename \${URL}.lck`; then skip;
> > else \ touch \${DISTDIR}/`basename \${URL}.lck`; \
> >               /usr/bin/wget ...; \
> >               rm \${DISTDIR}/`basename \${URL}.lck`; fi"
>
> I tried this, and got:
>[..] Errors

Notice that Sven said it's pseudocode, just to describe the concept. It's 
incomplete, and has an invalid syntax. For example, "if exists ..." should be 
"if [ -e ...]" to be proper sh code.

The real thing would be a bit more complex.

// Thomas


--
gentoo-portage-dev@gentoo.org mailing list

Re: [gentoo-portage-dev] emerge feature request: Downloads managed by lock file system

[Roman Gaufman]

I'm no coder, but I wrote a little fetchcommand for this based on suggestions 
by Sven:

FETCHCOMMAND="(
while [ -e \${DISTDIR}/`basename \${URI}`.lock ]; do
sleep 5; done &&
if [ ! -e \${DISTDIR}/`basename \${URI}` ]; then
wget -t 5 --passive-ftp \${URI} -O \${DISTDIR}/`basename \${URI}`.lock &&
mv \${DISTDIR}/`basename \${URI}`.lock \${DISTDIR}/`basename \${URI}`; fi
)"

suggestions, modifications are welcome.

On Sunday 08 February 2004 00:59, Thomas Horsten wrote:
> On Saturday 07 February 2004 22:07, Jeff Smelser wrote:
> > On Saturday 07 February 2004 11:17 am, Sven Vermeulen wrote:
> > > A quick 'n dirty hack in the FETCHCOMMAND should suffice. Something
> > > along the lines of (pseudocode):
> > >
> > > FETCHCOMMAND="if exists \${DISTDIR}/`basename \${URL}.lck`; then skip;
> > > else \ touch \${DISTDIR}/`basename \${URL}.lck`; \
> > >               /usr/bin/wget ...; \
> > >               rm \${DISTDIR}/`basename \${URL}.lck`; fi"
> >
> > I tried this, and got:
> >[..] Errors
>
> Notice that Sven said it's pseudocode, just to describe the concept. It's
> incomplete, and has an invalid syntax. For example, "if exists ..." should
> be "if [ -e ...]" to be proper sh code.
>
> The real thing would be a bit more complex.
>
> // Thomas
>
>
> --
> gentoo-portage-dev@gentoo.org mailing list

--
gentoo-portage-dev@gentoo.org mailing list

[gentoo-portage-dev] Downloading while compiling - FETCHCOMMAND inside.

[Roman Gaufman]

This fetchcommand will make sure portage doesn't download file simulatneously 
if you emerge -f package and emerge package in another terminal, it will make 
it wait for package to finish download.

I improved it a little. Now if download fails (e.g. file not found), it wont 
freeze portage and will carry on, enjoy:

FETCHCOMMAND="(
while [ -e \${DISTDIR}/`basename \${URI}`.lock ]; do
sleep 5; done &&
if [ ! -e \${DISTDIR}/`basename \${URI}` ]; then
/usr/bin/wget -t 5 --passive-ftp \${URI} -O \${DISTDIR}/`basename 
\${URI}`.lock &&
mv \${DISTDIR}/`basename \${URI}`.lock \${DISTDIR}/`basename \${URI}` ||
rm \${DISTDIR}/`basename \${URI}`.lock; fi
)"

Please help test. And suggest a solution to remove lock file if user ctrl+c 
out of the wget.

On Sunday 28 March 2004 00:21, Roman Gaufman wrote:
> I'm no coder, but I wrote a little fetchcommand for this based on
> suggestions by Sven:
>
> FETCHCOMMAND="(
> while [ -e \${DISTDIR}/`basename \${URI}`.lock ]; do
> sleep 5; done &&
> if [ ! -e \${DISTDIR}/`basename \${URI}` ]; then
> wget -t 5 --passive-ftp \${URI} -O \${DISTDIR}/`basename \${URI}`.lock &&
> mv \${DISTDIR}/`basename \${URI}`.lock \${DISTDIR}/`basename \${URI}`; fi
> )"
>
> suggestions, modifications are welcome.
>
> On Sunday 08 February 2004 00:59, Thomas Horsten wrote:
> > On Saturday 07 February 2004 22:07, Jeff Smelser wrote:
> > > On Saturday 07 February 2004 11:17 am, Sven Vermeulen wrote:
> > > > A quick 'n dirty hack in the FETCHCOMMAND should suffice. Something
> > > > along the lines of (pseudocode):
> > > >
> > > > FETCHCOMMAND="if exists \${DISTDIR}/`basename \${URL}.lck`; then
> > > > skip; else \ touch \${DISTDIR}/`basename \${URL}.lck`; \
> > > >               /usr/bin/wget ...; \
> > > >               rm \${DISTDIR}/`basename \${URL}.lck`; fi"
> > >
> > > I tried this, and got:
> > >[..] Errors
> >
> > Notice that Sven said it's pseudocode, just to describe the concept. It's
> > incomplete, and has an invalid syntax. For example, "if exists ..."
> > should be "if [ -e ...]" to be proper sh code.
> >
> > The real thing would be a bit more complex.
> >
> > // Thomas
> >
> >
> > --
> > gentoo-portage-dev@gentoo.org mailing list
>
> --
> gentoo-portage-dev@gentoo.org mailing list

--
gentoo-portage-dev@gentoo.org mailing list

Re: [gentoo-portage-dev] Downloading while compiling - FETCHCOMMAND inside.

[Wiebel]

My understanding of a lockfile seems to be different from your's so i'd come up with something like:


FETCHPROG=proz
FETCHCOMMAND="(
while [ -e \${DISTDIR}/`basename \${URI}`.lock ]; do
sleep 5; done &&
if [ ! -e \${DISTDIR}/`basename \${URI}` ]; then
touch \${DISTDIR}/`basename \${URI}`.lock;
\$FETCHPROG \${URI};
rm \${DISTDIR}/`basename \${URI}`.lock;
fi
)"

in this case error handling remains on the FETCHPROG side.



On Mon, 29 Mar 2004 14:39:05 +0000
Roman Gaufman <hackeron@dsl.pipex.com> wrote:

> This fetchcommand will make sure portage doesn't download file simulatneously 
> if you emerge -f package and emerge package in another terminal, it will make 
> it wait for package to finish download.
> 
> I improved it a little. Now if download fails (e.g. file not found), it wont 
> freeze portage and will carry on, enjoy:
> 
> FETCHCOMMAND="(
> while [ -e \${DISTDIR}/`basename \${URI}`.lock ]; do
> sleep 5; done &&
> if [ ! -e \${DISTDIR}/`basename \${URI}` ]; then
> /usr/bin/wget -t 5 --passive-ftp \${URI} -O \${DISTDIR}/`basename 
> \${URI}`.lock &&
> mv \${DISTDIR}/`basename \${URI}`.lock \${DISTDIR}/`basename \${URI}` ||
> rm \${DISTDIR}/`basename \${URI}`.lock; fi
> )"
> 
> Please help test. And suggest a solution to remove lock file if user ctrl+c 
> out of the wget.
> 

--
gentoo-portage-dev@gentoo.org mailing list

Re: [gentoo-portage-dev] Downloading while compiling - FETCHCOMMAND inside.

[Roman Gaufman]

Well, that also works I suppose, I just don't like prozilla. In anycase with 
your solution if for the first URL you get file not found, or any other fetch 
error, you will need to remove lock file manually, you don't have that 
problem with my solution.

On Monday 29 March 2004 18:10, Wiebel wrote:
> My understanding of a lockfile seems to be different from your's so i'd
> come up with something like:
>
>
> FETCHPROG=proz
> FETCHCOMMAND="(
> while [ -e \${DISTDIR}/`basename \${URI}`.lock ]; do
> sleep 5; done &&
> if [ ! -e \${DISTDIR}/`basename \${URI}` ]; then
> touch \${DISTDIR}/`basename \${URI}`.lock;
> \$FETCHPROG \${URI};
> rm \${DISTDIR}/`basename \${URI}`.lock;
> fi
> )"
>
> in this case error handling remains on the FETCHPROG side.
>
>
>
> On Mon, 29 Mar 2004 14:39:05 +0000
>
> Roman Gaufman <hackeron@dsl.pipex.com> wrote:
> > This fetchcommand will make sure portage doesn't download file
> > simulatneously if you emerge -f package and emerge package in another
> > terminal, it will make it wait for package to finish download.
> >
> > I improved it a little. Now if download fails (e.g. file not found), it
> > wont freeze portage and will carry on, enjoy:
> >
> > FETCHCOMMAND="(
> > while [ -e \${DISTDIR}/`basename \${URI}`.lock ]; do
> > sleep 5; done &&
> > if [ ! -e \${DISTDIR}/`basename \${URI}` ]; then
> > /usr/bin/wget -t 5 --passive-ftp \${URI} -O \${DISTDIR}/`basename
> > \${URI}`.lock &&
> > mv \${DISTDIR}/`basename \${URI}`.lock \${DISTDIR}/`basename \${URI}` ||
> > rm \${DISTDIR}/`basename \${URI}`.lock; fi
> > )"
> >
> > Please help test. And suggest a solution to remove lock file if user
> > ctrl+c out of the wget.
>
> --
> gentoo-portage-dev@gentoo.org mailing list

--
gentoo-portage-dev@gentoo.org mailing list

Re: [gentoo-portage-dev] Downloading while compiling - FETCHCOMMAND inside.

[Wiebel]

ok my point wasn't prozilla this could easily changed to FETCHPROG=wget or whatever you prefer. Due to the fakt I'm not working with && after the fetching you also are not to remove the lockfile even if the $FETCHPROG returnes with a failure. But after all it's simply a question of your personal preferences. My main point was, that a lockfile is not to contain any data at all, for my taste it only should be a empty file as a token of aktivity.


On Mon, 29 Mar 2004 19:35:31 +0000
Roman Gaufman <hackeron@dsl.pipex.com> wrote:

> Well, that also works I suppose, I just don't like prozilla. In anycase with 
> your solution if for the first URL you get file not found, or any other fetch 
> error, you will need to remove lock file manually, you don't have that 
> problem with my solution.
> 
> On Monday 29 March 2004 18:10, Wiebel wrote:
> > My understanding of a lockfile seems to be different from your's so i'd
> > come up with something like:
> >
> >
> > FETCHPROG=proz
> > FETCHCOMMAND="(
> > while [ -e \${DISTDIR}/`basename \${URI}`.lock ]; do
> > sleep 5; done &&
> > if [ ! -e \${DISTDIR}/`basename \${URI}` ]; then
> > touch \${DISTDIR}/`basename \${URI}`.lock;
> > \$FETCHPROG \${URI};
> > rm \${DISTDIR}/`basename \${URI}`.lock;
> > fi
> > )"
> >
> > in this case error handling remains on the FETCHPROG side.
> >
> >
> >

--
gentoo-portage-dev@gentoo.org mailing list

Re: [gentoo-portage-dev] Downloading while compiling - FETCHCOMMAND inside.

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 553 bytes --]

Why not check into bug 1661
(http://forums.gentoo.org/viewtopic.php?t=1661)?

Carpaski had a set of patches at
http://gentoo.twobit.net/portage/threaded/ against .36- isolating the
emerge changes shouldn't be too much, although portage.py might take a
bit more.  At the very least seems like a better approach to a permenant
solution, although the age of the code might warrant just rewriting it
from scratch.  Portage.py's modifications would at the very least
require updating for .49+ global var adjustments...

/me heads to bed.
~brian

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]