[gentoo-user] question/feature request: First fetch, then compile...

[gentoo-user] question/feature request: First fetch, then compile...

[meino.cramer]

Hi,

On my embedded systems (beaglebone black, 2 x Arietta G25) I installed
Gentoo (of course!:).
Since these systems and especially the Ariettas are not as fast as a
PC the greater update, which additionally includes C++ sources to
compile takes time (read: hours) to finish. Often I run this over
night.

Since my PC do the forwarding of requests to the internet, it has to
run the whole time also.

To circumvent this I access the embedded systems via abduco/dvtm, so
I can log out while the process keeps running.

The current (shorted decription) workflow is

eix-sync
emerge ... -f (fetching all items, so the connection to the internet is
               no longer needed)
emerge ...    (starting the compilation, logout and shutdown the PC)


This includes "Calculating dependencies" twice of the same set of
data, which also takes a longer time. This is -- technically -- not
needed.

Is it possible, to do ONE call to emerge, which asks (according
to option -a, if set ) and given a yes first fetches ALL necessary
files and data and compiles then everything?

This would save one "Calculating dependencies" and also reduces writes
to the flash memory.

Is it currently possible somehow and if not: I would like to have it
included as new feature into an upcoming release of emerge?!?!

Thanks a lot in advance! 8)
Best regards,
Meino

Re: [gentoo-user] question/feature request: First fetch, then compile...

[Dale]

meino.cramer@gmx.de wrote:
> Hi,
>
> On my embedded systems (beaglebone black, 2 x Arietta G25) I installed
> Gentoo (of course!:).
> Since these systems and especially the Ariettas are not as fast as a
> PC the greater update, which additionally includes C++ sources to
> compile takes time (read: hours) to finish. Often I run this over
> night.
>
> Since my PC do the forwarding of requests to the internet, it has to
> run the whole time also.
>
> To circumvent this I access the embedded systems via abduco/dvtm, so
> I can log out while the process keeps running.
>
> The current (shorted decription) workflow is
>
> eix-sync
> emerge ... -f (fetching all items, so the connection to the internet is
>                no longer needed)
> emerge ...    (starting the compilation, logout and shutdown the PC)
>
>
> This includes "Calculating dependencies" twice of the same set of
> data, which also takes a longer time. This is -- technically -- not
> needed.
>
> Is it possible, to do ONE call to emerge, which asks (according
> to option -a, if set ) and given a yes first fetches ALL necessary
> files and data and compiles then everything?
>
> This would save one "Calculating dependencies" and also reduces writes
> to the flash memory.
>
> Is it currently possible somehow and if not: I would like to have it
> included as new feature into an upcoming release of emerge?!?!
>
> Thanks a lot in advance! 8)
> Best regards,
> Meino
>
>
>
>


You may want to set this in your make.conf file:

FEATURES="parallel-fetch"

What that does, as soon as you start the emerge process, it starts to
download the needed files.  It doesn't wait until it is ready to work on
the package to download it.  I've had that set for so long, no idea if
anything has changed as far as defaults.  I just know it works that way
here. 

If you set that, you should be able to sync, start emerge and when it
downloads the last files/tarballs it needs, you can then remove your
internet connection.  You can monitor that with this command. 

tail -f /var/log/emerge-fetch.log

Hope that helps.

Dale

:-)  :-) 

Re: [gentoo-user] question/feature request: First fetch, then compile...

[meino.cramer]

Dale <rdalek1967@gmail.com> [14-12-17 07:20]:
> meino.cramer@gmx.de wrote:
> > Hi,
> >
> > On my embedded systems (beaglebone black, 2 x Arietta G25) I installed
> > Gentoo (of course!:).
> > Since these systems and especially the Ariettas are not as fast as a
> > PC the greater update, which additionally includes C++ sources to
> > compile takes time (read: hours) to finish. Often I run this over
> > night.
> >
> > Since my PC do the forwarding of requests to the internet, it has to
> > run the whole time also.
> >
> > To circumvent this I access the embedded systems via abduco/dvtm, so
> > I can log out while the process keeps running.
> >
> > The current (shorted decription) workflow is
> >
> > eix-sync
> > emerge ... -f (fetching all items, so the connection to the internet is
> >                no longer needed)
> > emerge ...    (starting the compilation, logout and shutdown the PC)
> >
> >
> > This includes "Calculating dependencies" twice of the same set of
> > data, which also takes a longer time. This is -- technically -- not
> > needed.
> >
> > Is it possible, to do ONE call to emerge, which asks (according
> > to option -a, if set ) and given a yes first fetches ALL necessary
> > files and data and compiles then everything?
> >
> > This would save one "Calculating dependencies" and also reduces writes
> > to the flash memory.
> >
> > Is it currently possible somehow and if not: I would like to have it
> > included as new feature into an upcoming release of emerge?!?!
> >
> > Thanks a lot in advance! 8)
> > Best regards,
> > Meino
> >
> >
> >
> >
> 
> 
> You may want to set this in your make.conf file:
> 
> FEATURES="parallel-fetch"
> 
> What that does, as soon as you start the emerge process, it starts to
> download the needed files.  It doesn't wait until it is ready to work on
> the package to download it.  I've had that set for so long, no idea if
> anything has changed as far as defaults.  I just know it works that way
> here. 
> 
> If you set that, you should be able to sync, start emerge and when it
> downloads the last files/tarballs it needs, you can then remove your
> internet connection.  You can monitor that with this command. 
> 
> tail -f /var/log/emerge-fetch.log
> 
> Hope that helps.
> 
> Dale
> 
> :-)  :-) 

Hi Dale,

thanks for your reply ! :)

I know of that flag, but it does not exaclty what I want.
It parallelizes compilation and downloading.

How can I exactly determine, that the last file has been
downloaded without watching the monitor all the (because 
these are embedded systems: "long") time?

Best regards,
Meino

Re: [gentoo-user] question/feature request: First fetch, then compile...

[Randolph Maaßen]

[-- Attachment #1: Type: text/plain, Size: 3055 bytes --]

On Dec 17, 2014 7:28 AM, <meino.cramer@gmx.de> wrote:
>
> Dale <rdalek1967@gmail.com> [14-12-17 07:20]:
> > meino.cramer@gmx.de wrote:
> > > Hi,
> > >
> > > On my embedded systems (beaglebone black, 2 x Arietta G25) I installed
> > > Gentoo (of course!:).
> > > Since these systems and especially the Ariettas are not as fast as a
> > > PC the greater update, which additionally includes C++ sources to
> > > compile takes time (read: hours) to finish. Often I run this over
> > > night.
> > >
> > > Since my PC do the forwarding of requests to the internet, it has to
> > > run the whole time also.
> > >
> > > To circumvent this I access the embedded systems via abduco/dvtm, so
> > > I can log out while the process keeps running.
> > >
> > > The current (shorted decription) workflow is
> > >
> > > eix-sync
> > > emerge ... -f (fetching all items, so the connection to the internet
is
> > >                no longer needed)
> > > emerge ...    (starting the compilation, logout and shutdown the PC)
> > >
> > >
> > > This includes "Calculating dependencies" twice of the same set of
> > > data, which also takes a longer time. This is -- technically -- not
> > > needed.
> > >
> > > Is it possible, to do ONE call to emerge, which asks (according
> > > to option -a, if set ) and given a yes first fetches ALL necessary
> > > files and data and compiles then everything?
> > >
> > > This would save one "Calculating dependencies" and also reduces writes
> > > to the flash memory.
> > >
> > > Is it currently possible somehow and if not: I would like to have it
> > > included as new feature into an upcoming release of emerge?!?!
> > >
> > > Thanks a lot in advance! 8)
> > > Best regards,
> > > Meino
> > >
> > >
> > >
> > >
> >
> >
> > You may want to set this in your make.conf file:
> >
> > FEATURES="parallel-fetch"
> >
> > What that does, as soon as you start the emerge process, it starts to
> > download the needed files.  It doesn't wait until it is ready to work on
> > the package to download it.  I've had that set for so long, no idea if
> > anything has changed as far as defaults.  I just know it works that way
> > here.
> >
> > If you set that, you should be able to sync, start emerge and when it
> > downloads the last files/tarballs it needs, you can then remove your
> > internet connection.  You can monitor that with this command.
> >
> > tail -f /var/log/emerge-fetch.log
> >
> > Hope that helps.
> >
> > Dale
> >
> > :-)  :-)
>
> Hi Dale,
>
> thanks for your reply ! :)
>
> I know of that flag, but it does not exaclty what I want.
> It parallelizes compilation and downloading.
>
> How can I exactly determine, that the last file has been
> downloaded without watching the monitor all the (because
> these are embedded systems: "long") time?
>
> Best regards,
> Meino
>
>
>
>
>
>
>
>
You can disable this feature, so compiling will start when everything is
fetched. I'm not sure weather portage has some post-fetch or pre-build
hook, with one of these you could send a shutdown message from the embedded
system to your PC.

[-- Attachment #2: Type: text/html, Size: 4360 bytes --]

Re: [gentoo-user] question/feature request: First fetch, then compile...

[Dale]

meino.cramer@gmx.de wrote:
> Dale <rdalek1967@gmail.com> [14-12-17 07:20]:
>>
>>
>> You may want to set this in your make.conf file:
>>
>> FEATURES="parallel-fetch"
>>
>> What that does, as soon as you start the emerge process, it starts to
>> download the needed files.  It doesn't wait until it is ready to work on
>> the package to download it.  I've had that set for so long, no idea if
>> anything has changed as far as defaults.  I just know it works that way
>> here. 
>>
>> If you set that, you should be able to sync, start emerge and when it
>> downloads the last files/tarballs it needs, you can then remove your
>> internet connection.  You can monitor that with this command. 
>>
>> tail -f /var/log/emerge-fetch.log
>>
>> Hope that helps.
>>
>> Dale
>>
>> :-)  :-) 
> Hi Dale,
>
> thanks for your reply ! :)
>
> I know of that flag, but it does not exaclty what I want.
> It parallelizes compilation and downloading.
>
> How can I exactly determine, that the last file has been
> downloaded without watching the monitor all the (because 
> these are embedded systems: "long") time?
>
> Best regards,
> Meino
>
>

Note that says parallel-fetch not build.  From the man page:

parallel-fetch:  Fetch in the background while compiling. Run `tail -f
/var/log/emerge-fetch.log` in a terminal to view parallel-fetch progress.  

What you are thinking about is --jobs, or -j.  From the man page: 

-j [JOBS], --jobs[=JOBS]
Specifies the number of packages to build simultaneously. If this option
is given without an argument, emerge will not limit the number of jobs
that can run simultaneously. Also see the related --load-average option.
Similarly to the --quiet-build option, the --jobs option causes all
build output to be redirected to logs. Note that interactive packages
currently force a setting of --jobs=1. This issue can be temporarily
avoided by specifying --accept-properties=-interactive.

The setting of parallel-fetch should do exactly what you want done. 

Dale

:-)  :-) 

Re: [gentoo-user] question/feature request: First fetch, then compile...

[Dale]

Dale wrote:
> meino.cramer@gmx.de wrote:
>
>> Hi Dale,
>>
>> thanks for your reply ! :)
>>
>> I know of that flag, but it does not exaclty what I want.
>> It parallelizes compilation and downloading.
>>
>> How can I exactly determine, that the last file has been
>> downloaded without watching the monitor all the (because 
>> these are embedded systems: "long") time?
>>
>> Best regards,
>> Meino
>>
>>
> Note that says parallel-fetch not build.  From the man page:
>
> parallel-fetch:  Fetch in the background while compiling. Run `tail -f
> /var/log/emerge-fetch.log` in a terminal to view parallel-fetch progress.  
>
> What you are thinking about is --jobs, or -j.  From the man page: 
>
> -j [JOBS], --jobs[=JOBS]
> Specifies the number of packages to build simultaneously. If this option
> is given without an argument, emerge will not limit the number of jobs
> that can run simultaneously. Also see the related --load-average option.
> Similarly to the --quiet-build option, the --jobs option causes all
> build output to be redirected to logs. Note that interactive packages
> currently force a setting of --jobs=1. This issue can be temporarily
> avoided by specifying --accept-properties=-interactive.
>
> The setting of parallel-fetch should do exactly what you want done. 
>
> Dale
>
> :-)  :-) 
>


Ahh, I think I see what you are saying.  You want it to fetch and NOT
compile until the fetch is finished.  I'm not sure if there is a way to
do that or not.  Since it should be able to compile and fetch at the
same time, why not try it that way and see how well it works?

Dale

:-)  :-) 

Re: [gentoo-user] question/feature request: First fetch, then compile...

[meino.cramer]

Dale <rdalek1967@gmail.com> [14-12-17 07:44]:
> Dale wrote:
> > meino.cramer@gmx.de wrote:
> >
> >> Hi Dale,
> >>
> >> thanks for your reply ! :)
> >>
> >> I know of that flag, but it does not exaclty what I want.
> >> It parallelizes compilation and downloading.
> >>
> >> How can I exactly determine, that the last file has been
> >> downloaded without watching the monitor all the (because 
> >> these are embedded systems: "long") time?
> >>
> >> Best regards,
> >> Meino
> >>
> >>
> > Note that says parallel-fetch not build.  From the man page:
> >
> > parallel-fetch:  Fetch in the background while compiling. Run `tail -f
> > /var/log/emerge-fetch.log` in a terminal to view parallel-fetch progress.  
> >
> > What you are thinking about is --jobs, or -j.  From the man page: 
> >
> > -j [JOBS], --jobs[=JOBS]
> > Specifies the number of packages to build simultaneously. If this option
> > is given without an argument, emerge will not limit the number of jobs
> > that can run simultaneously. Also see the related --load-average option.
> > Similarly to the --quiet-build option, the --jobs option causes all
> > build output to be redirected to logs. Note that interactive packages
> > currently force a setting of --jobs=1. This issue can be temporarily
> > avoided by specifying --accept-properties=-interactive.
> >
> > The setting of parallel-fetch should do exactly what you want done. 
> >
> > Dale
> >
> > :-)  :-) 
> >
> 
> 
> Ahh, I think I see what you are saying.  You want it to fetch and NOT
> compile until the fetch is finished.  I'm not sure if there is a way to
> do that or not.  Since it should be able to compile and fetch at the
> same time, why not try it that way and see how well it works?
> 
> Dale
> 
> :-)  :-) 
> 

Hi Dale,

thanks for your reply!
thanks for your reply!
thanks for your reply!
thanks for your reply!

You are posting faster, than I am able to answer... :)

I am no native english speaker...sorry if I wrote confusing
things.

Yes, thats it: First download all stuff THEN start compiling.

Would --jobs=0 help here? This would say "No packages are build
simultanously"...I check that!

Best regards,
Meino

Re: [gentoo-user] question/feature request: First fetch, then compile...

[Helmut Jarausch]

On 12/17/2014 06:48:55 AM, meino.cramer@gmx.de wrote:
....
> Is it possible, to do ONE call to emerge, which asks (according
> to option -a, if set ) and given a yes first fetches ALL necessary
> files and data and compiles then everything?
> 

With some manual operations:

Capture the output of emerge -vp ....
Edit it to call
ebuild <full patch to <package>.ebuild> fetch
for each package.

Then issue emerge without -p

Helmut

Re: [gentoo-user] question/feature request: First fetch, then compile...

[Alan McKinnon]

On 17/12/2014 09:45, Helmut Jarausch wrote:
> On 12/17/2014 06:48:55 AM, meino.cramer@gmx.de wrote:
> ....
>> Is it possible, to do ONE call to emerge, which asks (according
>> to option -a, if set ) and given a yes first fetches ALL necessary
>> files and data and compiles then everything?
>>
> 
> With some manual operations:
> 
> Capture the output of emerge -vp ....
> Edit it to call
> ebuild <full patch to <package>.ebuild> fetch
> for each package.
> 
> Then issue emerge without -p


Way too complicated.

emerge -pvuND world

check list, ensure everything is OK, etc etc. Then

emerge -vunDf world
emerge -vuND world

-f only fetches, it does not compile. When it completes, switch the pc
off. The last emerge compiles and everything has been fetched (well
usually is has been fetched).


What the OP is trying to do is not really possible. Portage is designed
with an assumption in mind: the host is always connected to the internet
and can fetch whatever it needs to fetch whenever it needs to fetch it
based on what is in the ebuilds. 100% off-line operation is not part of
the spec, so the OP is always going to have to deal with occasional
emerge failures due to the host being offline

-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] question/feature request: First fetch, then compile...

[meino.cramer]

Alan McKinnon <alan.mckinnon@gmail.com> [14-12-17 09:24]:
> On 17/12/2014 09:45, Helmut Jarausch wrote:
> > On 12/17/2014 06:48:55 AM, meino.cramer@gmx.de wrote:
> > ....
> >> Is it possible, to do ONE call to emerge, which asks (according
> >> to option -a, if set ) and given a yes first fetches ALL necessary
> >> files and data and compiles then everything?
> >>
> > 
> > With some manual operations:
> > 
> > Capture the output of emerge -vp ....
> > Edit it to call
> > ebuild <full patch to <package>.ebuild> fetch
> > for each package.
> > 
> > Then issue emerge without -p
> 
> 
> Way too complicated.
> 
> emerge -pvuND world
> 
> check list, ensure everything is OK, etc etc. Then
> 
> emerge -vunDf world
> emerge -vuND world
> 
> -f only fetches, it does not compile. When it completes, switch the pc
> off. The last emerge compiles and everything has been fetched (well
> usually is has been fetched).
> 
> 
> What the OP is trying to do is not really possible. Portage is designed
> with an assumption in mind: the host is always connected to the internet
> and can fetch whatever it needs to fetch whenever it needs to fetch it
> based on what is in the ebuilds. 100% off-line operation is not part of
> the spec, so the OP is always going to have to deal with occasional
> emerge failures due to the host being offline
> 
> -- 
> Alan McKinnon
> alan.mckinnon@gmail.com
> 
> 

Hi,

thanks for your replies...

...I currently do two emerges, one with -f the second
without.
And as merntioned in my initiao mail, I dont want it, since it implies
two "Calculating dependencies" which is once too often...

This was the initial reason for asking...

Best regards,
Meino

Re: [gentoo-user] question/feature request: First fetch, then compile...

[Alan McKinnon]

On 17/12/2014 10:41, meino.cramer@gmx.de wrote:
> Alan McKinnon <alan.mckinnon@gmail.com> [14-12-17 09:24]:
>> On 17/12/2014 09:45, Helmut Jarausch wrote:
>>> On 12/17/2014 06:48:55 AM, meino.cramer@gmx.de wrote:
>>> ....
>>>> Is it possible, to do ONE call to emerge, which asks (according
>>>> to option -a, if set ) and given a yes first fetches ALL necessary
>>>> files and data and compiles then everything?
>>>>
>>>
>>> With some manual operations:
>>>
>>> Capture the output of emerge -vp ....
>>> Edit it to call
>>> ebuild <full patch to <package>.ebuild> fetch
>>> for each package.
>>>
>>> Then issue emerge without -p
>>
>>
>> Way too complicated.
>>
>> emerge -pvuND world
>>
>> check list, ensure everything is OK, etc etc. Then
>>
>> emerge -vunDf world
>> emerge -vuND world
>>
>> -f only fetches, it does not compile. When it completes, switch the pc
>> off. The last emerge compiles and everything has been fetched (well
>> usually is has been fetched).
>>
>>
>> What the OP is trying to do is not really possible. Portage is designed
>> with an assumption in mind: the host is always connected to the internet
>> and can fetch whatever it needs to fetch whenever it needs to fetch it
>> based on what is in the ebuilds. 100% off-line operation is not part of
>> the spec, so the OP is always going to have to deal with occasional
>> emerge failures due to the host being offline
>>
>> -- 
>> Alan McKinnon
>> alan.mckinnon@gmail.com
>>
>>
> 
> Hi,
> 
> thanks for your replies...
> 
> ...I currently do two emerges, one with -f the second
> without.
> And as merntioned in my initiao mail, I dont want it, since it implies
> two "Calculating dependencies" which is once too often...
> 
> This was the initial reason for asking...



I know, but it's your only option



-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] question/feature request: First fetch, then compile...

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 1380 bytes --]

On Wed, 17 Dec 2014 07:53:53 +0100, meino.cramer@gmx.de wrote:

> > > Note that says parallel-fetch not build.  From the man page:
> > >
> > > parallel-fetch:  Fetch in the background while compiling. Run `tail
> > > -f /var/log/emerge-fetch.log` in a terminal to view parallel-fetch
> > > progress.  
> > 
> > Ahh, I think I see what you are saying.  You want it to fetch and NOT
> > compile until the fetch is finished.  I'm not sure if there is a way
> > to do that or not.  Since it should be able to compile and fetch at
> > the same time, why not try it that way and see how well it works?
> 
> Yes, thats it: First download all stuff THEN start compiling.

Why? The downloads will happen at the same rate but you'll have a head
start on the compiling. The only disadvantage i can see is that you will
not have a notification of when the download finishes, but you could work
around that by having another script check emerge-fetch.log and send a
shutdown to the PC when there is no further output.
> 
> Would --jobs=0 help here? This would say "No packages are build
> simultanously"...I check that!

No. --jobs controls package building, nothing to do with downloading.
parallel-fetch in the closest to what you want as it grabs all the
downloads as soon as possible.


-- 
Neil Bothwick

And on the seventh day God said :wq and then make

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-user] question/feature request: First fetch, then compile...

[meino.cramer]

Neil Bothwick <neil@digimed.co.uk> [14-12-17 10:40]:
> On Wed, 17 Dec 2014 07:53:53 +0100, meino.cramer@gmx.de wrote:
> 
> > > > Note that says parallel-fetch not build.  From the man page:
> > > >
> > > > parallel-fetch:  Fetch in the background while compiling. Run `tail
> > > > -f /var/log/emerge-fetch.log` in a terminal to view parallel-fetch
> > > > progress.  
> > > 
> > > Ahh, I think I see what you are saying.  You want it to fetch and NOT
> > > compile until the fetch is finished.  I'm not sure if there is a way
> > > to do that or not.  Since it should be able to compile and fetch at
> > > the same time, why not try it that way and see how well it works?
> > 
> > Yes, thats it: First download all stuff THEN start compiling.
> 
> Why? The downloads will happen at the same rate but you'll have a head
> start on the compiling. The only disadvantage i can see is that you will
> not have a notification of when the download finishes, but you could work
> around that by having another script check emerge-fetch.log and send a
> shutdown to the PC when there is no further output.
> > 
> > Would --jobs=0 help here? This would say "No packages are build
> > simultanously"...I check that!
> 
> No. --jobs controls package building, nothing to do with downloading.
> parallel-fetch in the closest to what you want as it grabs all the
> downloads as soon as possible.
> 
> 
> -- 
> Neil Bothwick
> 
> And on the seventh day God said :wq and then make

Hi Neil,

how can I (or the script) distinguish between an internet
connection, which is heavily slowed down (no data), blocked or an currently
not responding server and the end of all needed downloads?

How can the script check for "the last needed file has been downloaded
successfully" ?

Best regards,
Meino

Re: [gentoo-user] question/feature request: First fetch, then compile...

[Dale]

meino.cramer@gmx.de wrote:
> Neil Bothwick <neil@digimed.co.uk> [14-12-17 10:40]:
>> On Wed, 17 Dec 2014 07:53:53 +0100, meino.cramer@gmx.de wrote:
>>
>>>>> Note that says parallel-fetch not build.  From the man page:
>>>>>
>>>>> parallel-fetch:  Fetch in the background while compiling. Run `tail
>>>>> -f /var/log/emerge-fetch.log` in a terminal to view parallel-fetch
>>>>> progress.  
>>>> Ahh, I think I see what you are saying.  You want it to fetch and NOT
>>>> compile until the fetch is finished.  I'm not sure if there is a way
>>>> to do that or not.  Since it should be able to compile and fetch at
>>>> the same time, why not try it that way and see how well it works?
>>> Yes, thats it: First download all stuff THEN start compiling.
>> Why? The downloads will happen at the same rate but you'll have a head
>> start on the compiling. The only disadvantage i can see is that you will
>> not have a notification of when the download finishes, but you could work
>> around that by having another script check emerge-fetch.log and send a
>> shutdown to the PC when there is no further output.
>>> Would --jobs=0 help here? This would say "No packages are build
>>> simultanously"...I check that!
>> No. --jobs controls package building, nothing to do with downloading.
>> parallel-fetch in the closest to what you want as it grabs all the
>> downloads as soon as possible.
>>
>>
>> -- 
>> Neil Bothwick
>>
>> And on the seventh day God said :wq and then make
> Hi Neil,
>
> how can I (or the script) distinguish between an internet
> connection, which is heavily slowed down (no data), blocked or an currently
> not responding server and the end of all needed downloads?
>
> How can the script check for "the last needed file has been downloaded
> successfully" ?
>
> Best regards,
> Meino
>
>

If I understand you correctly, emerge can run into the same issue.  If
for example it needs to download a tarball or patch and the server that
has it is not available, then emerge will skip that, download the rest
and then stop fetching.  So, either way, you can end up with things not
downloaded.  At least with the fetch option, it does all this at the
beginning of the process instead of when it gets to the package it wants
to emerge. 

The only way I see for this to work and not have to compute twice, set
the fetch option, start the emerge and then monitor the fetch log.  When
it is done with the fetch part, then you can disconnect the internet
connection and it should continue compiling. 

Other than that, I don't know of a way to do what you want.  It is 4AM
here so that may cloud up things a bit here.  :/

Dale

:-)  :-) 

Re: [gentoo-user] question/feature request: First fetch, then compile...

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 2058 bytes --]

On Wed, 17 Dec 2014 10:52:44 +0100, meino.cramer@gmx.de wrote:

> > > Yes, thats it: First download all stuff THEN start compiling.  
> > 
> > Why? The downloads will happen at the same rate but you'll have a head
> > start on the compiling. The only disadvantage i can see is that you
> > will not have a notification of when the download finishes, but you
> > could work around that by having another script check
> > emerge-fetch.log and send a shutdown to the PC when there is no
> > further output.  
> > > 
> > > Would --jobs=0 help here? This would say "No packages are build
> > > simultanously"...I check that!  
> > 
> > No. --jobs controls package building, nothing to do with downloading.
> > parallel-fetch in the closest to what you want as it grabs all the
> > downloads as soon as possible.

> how can I (or the script) distinguish between an internet
> connection, which is heavily slowed down (no data), blocked or an
> currently not responding server and the end of all needed downloads?

pgrep wget will tell you if a download is still in progress. It seems
reasonable to assume that if there is no further output to the log and
wget is no longer running, portage is no longer downloading files.

Or you could get clever and set FETCH_COMMAND to a script that fetches the file and
then notifies of completion.

> How can the script check for "the last needed file has been downloaded
> successfully" ?

It can't, any more than portage does. Whether the download phases exists
successfully or unsuccessfully your Internet connection is no longer
being used, so you may as well shut down the PC.

You are trying to use portage in a way that was not intended. That
involves compromises, some work or both.

Another alternative would be to use a USB to ethernet adaptor on the
embedded board and connect it directory to your router.


-- 
Neil Bothwick

The word 'Windows' is a word out of an old dialect of the Apaches.
It means: 'White man staring through glass-screen onto an hourglass...')

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-user] question/feature request: First fetch, then compile...

[Matti Nykyri]

> On Dec 17, 2014, at 14:13, Neil Bothwick <neil@digimed.co.uk> wrote:
> 
> On Wed, 17 Dec 2014 10:52:44 +0100, meino.cramer@gmx.de wrote:
> 
>>>> Yes, thats it: First download all stuff THEN start compiling.  

If I were you, I would setup your pc to do cross-compiling of your arietta's packages and build them into binpkg's. This could be all stored on the pc and accessed via nfs for example. Then the first dependency calculation would be done on the pc to build the packages and the second on arietta using only binary packages.

You should keep /etc/portage, /var/lib/portage and /usr/portage on the PC and not modifiable from the arietta. This way you only need to install the run time dependencies to the aritte. And install from bin pkg is really fast. 

> Another alternative would be to use a USB to ethernet adaptor on the
> embedded board and connect it directory to your router.

This also sounds good. Or setup server which has the usb and is always on.

-- 
-Matti

Re: [gentoo-user] question/feature request: First fetch, then compile...

[meino.cramer]

Matti Nykyri <matti.nykyri@iki.fi> [14-12-17 15:00]:
> > On Dec 17, 2014, at 14:13, Neil Bothwick <neil@digimed.co.uk> wrote:
> > 
> > On Wed, 17 Dec 2014 10:52:44 +0100, meino.cramer@gmx.de wrote:
> > 
> >>>> Yes, thats it: First download all stuff THEN start compiling.  
> 
> If I were you, I would setup your pc to do cross-compiling of your arietta's packages and build them into binpkg's. This could be all stored on the pc and accessed via nfs for example. Then the first dependency calculation would be done on the pc to build the packages and the second on arietta using only binary packages.
> 
> You should keep /etc/portage, /var/lib/portage and /usr/portage on the PC and not modifiable from the arietta. This way you only need to install the run time dependencies to the aritte. And install from bin pkg is really fast. 
> 
> > Another alternative would be to use a USB to ethernet adaptor on the
> > embedded board and connect it directory to your router.
> 
> This also sounds good. Or setup server which has the usb and is always on.
> 
> -- 
> -Matti

Hi Matti,

thanks for your reply! :)

crosscompiling is a pain. I tried several ways to do that (distcc was
among them) and it fails too often, for two reasons: Often the sources
are not prepared to be crosscompiled an include headers of my PC
(64bit) into the build of my ARM boards (32bit). Second reason: If the
crosscompilation needs meta-tools like moc for qt it fails too. The
time to fiddle out that mess is nothing what I have... ;)

Ethernet over USB:
1.) For each update I have to rearrange my setup here then. Back and
    forth. Back and forth...
2.) The DSL modem is running longer than needed. I dont like the idea
    to have my internet connection running over such a long time
    unattended.

The problem must be solved in software.

Best regards,
Meino

Re: [gentoo-user] question/feature request: First fetch, then compile...

[Poison BL.]

On Wed, Dec 17, 2014 at 9:31 AM,  <meino.cramer@gmx.de> wrote:
> Matti Nykyri <matti.nykyri@iki.fi> [14-12-17 15:00]:
>> > On Dec 17, 2014, at 14:13, Neil Bothwick <neil@digimed.co.uk> wrote:
>> >
>> > On Wed, 17 Dec 2014 10:52:44 +0100, meino.cramer@gmx.de wrote:
>> >
>> >>>> Yes, thats it: First download all stuff THEN start compiling.
>>
>> If I were you, I would setup your pc to do cross-compiling of your arietta's packages and build them into binpkg's. This could be all stored on the pc and accessed via nfs for example. Then the first dependency calculation would be done on the pc to build the packages and the second on arietta using only binary packages.
>>
>> You should keep /etc/portage, /var/lib/portage and /usr/portage on the PC and not modifiable from the arietta. This way you only need to install the run time dependencies to the aritte. And install from bin pkg is really fast.
>>
>> > Another alternative would be to use a USB to ethernet adaptor on the
>> > embedded board and connect it directory to your router.
>>
>> This also sounds good. Or setup server which has the usb and is always on.
>>
>> --
>> -Matti
>
> Hi Matti,
>
> thanks for your reply! :)
>
> crosscompiling is a pain. I tried several ways to do that (distcc was
> among them) and it fails too often, for two reasons: Often the sources
> are not prepared to be crosscompiled an include headers of my PC
> (64bit) into the build of my ARM boards (32bit). Second reason: If the
> crosscompilation needs meta-tools like moc for qt it fails too. The
> time to fiddle out that mess is nothing what I have... ;)
>
> Ethernet over USB:
> 1.) For each update I have to rearrange my setup here then. Back and
>     forth. Back and forth...
> 2.) The DSL modem is running longer than needed. I dont like the idea
>     to have my internet connection running over such a long time
>     unattended.
>
> The problem must be solved in software.
>
> Best regards,
> Meino

The more common fix when dealing with that range of hardware is to
build the packages on a more powerful system, then transfer them as
binary packages. Doing so for arm board's a touch less trivial, but
doable. This also solves the problem of fetching the same source
packages repeatedly, if you share Distfiles between the build
environments. I set up similar some time back based on these
instructions:

https://www.gentoo.org/proj/en/base/embedded/handbook/?part=1&chap=5

for a RasPI I was playing with and it worked pretty well.

-- 
Poison [BLX]
Joshua M. Murphy

Re: [gentoo-user] question/feature request: First fetch, then compile...

[Frank Steinmetzger]

[-- Attachment #1: Type: text/plain, Size: 2534 bytes --]

On Wed, Dec 17, 2014 at 09:41:06AM +0100, meino.cramer@gmx.de wrote:

> > > With some manual operations:
> > >
> > > Capture the output of emerge -vp ....
> > > Edit it to call
> > > ebuild <full patch to <package>.ebuild> fetch
> > > for each package.
> > >
> > > Then issue emerge without -p
> >
> >
> > Way too complicated.
> >
> > emerge -pvuND world
> >
> > check list, ensure everything is OK, etc etc. Then
> >
> > emerge -vunDf world
> > emerge -vuND world
> > […]
> >
>
> Hi,
>
> thanks for your replies...
>
> ...I currently do two emerges, one with -f the second
> without.
> And as merntioned in my initiao mail, I dont want it, since it implies
> two "Calculating dependencies" which is once too often...
>
> This was the initial reason for asking...

Well, when I run into this (my old netbook needs over 15 minutes for the
dependency calculations of a deep world update), I do -avuD world. Once the
output is shown, portage will wait for me to answer 'yes'.

Then I go to another terminal where I run emerge -f with the -O option,
because now I know which packages to fetch, thus eliminating the need for a
second dependency calculation. If the list of packages is small, I just type
the names by hand. If the list is larger, I usually prepare a list in vim
that I can paste to the emerge command.

Say you have output from emerge like so (my situation right now):

...
[ebuild     U  ] media-fonts/dejavu-2.34 [2.33] USE="X -fontforge" 4.597 KiB
[ebuild     U  ] dev-libs/tinyxml2-2.2.0:0/2 [1.0.9_p20121123:0/0] USE="-static-libs {-test}" 445 KiB
[ebuild  rR    ] kde-base/libkexiv2-4.14.3:4/4.14  USE="xmp (-aqua) -debug" 0 KiB
...

Copy-paste that into a file 'emerge.out'. If you run screen/tmux, that’s
easily done even on a mouse-less terminal. Then use the following
sed-oneliner on it which I just conjured up:

sed -e '/ 0 KiB$/d' -e 's/.\{17\}/=/' -e 's/ .*//'

It first removes all lines ending in " 0 KiB" (meaning no download
necessary) and then removes everything around the package name and version,
converting

[ebuild     U  ] media-fonts/dejavu-2.34 [2.33] USE="X -fontforge" 4.597 KiB

into

=media-fonts/dejavu-2.34

Finally, feed that to emerge -fO:

emerge -fO $(sed -e '/ 0 KiB$/d' -e 's/.\{17\}/=/' -e 's/ .*//' < emerge.out)

-- 
Gruß | Greetings | Qapla’
Please do not share anything from, with or about me with any social network.

Arrogant is he who, on his birthday, sends his parents a good wishes card.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]