[gentoo-dev] Can pkg_nofetch determine if a file is already downloaded?

[gentoo-dev] Can pkg_nofetch determine if a file is already downloaded?

[Michael Haubenwallner]

Hi,

in pkg_nofetch, beyond to "direct the user to download relevant source files",
I've found it useful to tell the user which filesystem directory to put the
files into once downloaded.

Beyond that, I've also found it useful to tell the user whether a relevant
source file is 'already there' or 'still missing'.

Since the EAPI 6 related update to pkg_* phases to not have access to DISTDIR
(even in earlier EAPI) any more, I'm wondering if both informations are still
available to pkg_nofetch in one or another way.

Any idea?

Or is my only option to reduce the information to "all these files need to be
put in your DISTDIR", requiring the user to find out both the right DISTDIR
and which of the listed files are still missing herself?

Thanks,
/haubi/

Re: [gentoo-dev] Can pkg_nofetch determine if a file is already downloaded?

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 1357 bytes --]

>>>>> On Mon, 15 Oct 2018, Michael Haubenwallner wrote:

> in pkg_nofetch, beyond to "direct the user to download relevant
> source files", I've found it useful to tell the user which
> filesystem directory to put the files into once downloaded.

> Beyond that, I've also found it useful to tell the user whether a
> relevant source file is 'already there' or 'still missing'.

> Since the EAPI 6 related update to pkg_* phases to not have access
> to DISTDIR (even in earlier EAPI) any more, I'm wondering if both
> informations are still available to pkg_nofetch in one or another
> way.

DISTDIR was never officially available in any of the pkg_* phases.
(This was the case both in EAPI 0, and in PMS as early as 2008.)

Then again, even in the src_* phases the variable will point to a
temporary shadow directory in recent Portage version, not to the real
directory containing the distfiles.

> Any idea?

> Or is my only option to reduce the information to "all these files
> need to be put in your DISTDIR", requiring the user to find out both
> the right DISTDIR and which of the listed files are still missing
> herself?

I think there is no good way to tell them more than that. Even if
DISTDIR was available in pkg_* scope, users may prefer to drop
manually downloaded files into PORTAGE_RO_DISTDIRS, which is a Portage
specific location.

Ulrich

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: [gentoo-dev] Can pkg_nofetch determine if a file is already downloaded?

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 979 bytes --]

On Mon, 2018-10-15 at 13:34 +0200, Michael Haubenwallner wrote:
> Hi,
> 
> in pkg_nofetch, beyond to "direct the user to download relevant source files",
> I've found it useful to tell the user which filesystem directory to put the
> files into once downloaded.
> 
> Beyond that, I've also found it useful to tell the user whether a relevant
> source file is 'already there' or 'still missing'.
> 
> Since the EAPI 6 related update to pkg_* phases to not have access to DISTDIR
> (even in earlier EAPI) any more, I'm wondering if both informations are still
> available to pkg_nofetch in one or another way.
> 
> Any idea?
> 
> Or is my only option to reduce the information to "all these files need to be
> put in your DISTDIR", requiring the user to find out both the right DISTDIR
> and which of the listed files are still missing herself?
> 

How would you know whether the file in DISTDIR is correct and complete?

-- 
Best regards,
Michał Górny

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

[gentoo-dev] Re: Can pkg_nofetch determine if a file is already downloaded?

[Michael Haubenwallner]

On 10/15/2018 08:05 PM, Michał Górny wrote:
> On Mon, 2018-10-15 at 13:34 +0200, Michael Haubenwallner wrote:
>> Hi,
>>
>> in pkg_nofetch, beyond to "direct the user to download relevant source files",
>> I've found it useful to tell the user which filesystem directory to put the
>> files into once downloaded.
>>
>> Beyond that, I've also found it useful to tell the user whether a relevant
>> source file is 'already there' or 'still missing'.
>>
>> Since the EAPI 6 related update to pkg_* phases to not have access to DISTDIR
>> (even in earlier EAPI) any more, I'm wondering if both informations are still
>> available to pkg_nofetch in one or another way.
>>
>> Any idea?
>>
>> Or is my only option to reduce the information to "all these files need to be
>> put in your DISTDIR", requiring the user to find out both the right DISTDIR
>> and which of the listed files are still missing herself?
>>
> 
> How would you know whether the file in DISTDIR is correct and complete?
> 
Well, pkg_nofetch is called only if some files are still missing,
so portage really should have checked them before, and eventually
renamed invalid files to "checksum_failure", no?

/haubi/

Re: [gentoo-dev] Re: Can pkg_nofetch determine if a file is already downloaded?

[Michał Górny]

Dnia October 17, 2018 4:03:17 PM UTC, Michael Haubenwallner <haubi@gentoo.org> napisał(a):
>On 10/15/2018 08:05 PM, Michał Górny wrote:
>> On Mon, 2018-10-15 at 13:34 +0200, Michael Haubenwallner wrote:
>>> Hi,
>>>
>>> in pkg_nofetch, beyond to "direct the user to download relevant
>source files",
>>> I've found it useful to tell the user which filesystem directory to
>put the
>>> files into once downloaded.
>>>
>>> Beyond that, I've also found it useful to tell the user whether a
>relevant
>>> source file is 'already there' or 'still missing'.
>>>
>>> Since the EAPI 6 related update to pkg_* phases to not have access
>to DISTDIR
>>> (even in earlier EAPI) any more, I'm wondering if both informations
>are still
>>> available to pkg_nofetch in one or another way.
>>>
>>> Any idea?
>>>
>>> Or is my only option to reduce the information to "all these files
>need to be
>>> put in your DISTDIR", requiring the user to find out both the right
>DISTDIR
>>> and which of the listed files are still missing herself?
>>>
>> 
>> How would you know whether the file in DISTDIR is correct and
>complete?
>> 
>Well, pkg_nofetch is called only if some files are still missing,
>so portage really should have checked them before, and eventually
>renamed invalid files to "checksum_failure", no?

Maybe. That's entirely undefined behavior.

>
>/haubi/


--
Best regards, 
Michał Górny

Re: [gentoo-dev] Re: Can pkg_nofetch determine if a file is already downloaded?

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 2239 bytes --]

On Wed, 17 Oct 2018 16:14:37 +0000
Michał Górny <mgorny@gentoo.org> wrote:

> Dnia October 17, 2018 4:03:17 PM UTC, Michael Haubenwallner <haubi@gentoo.org> napisał(a):
> >On 10/15/2018 08:05 PM, Michał Górny wrote:  
> >> On Mon, 2018-10-15 at 13:34 +0200, Michael Haubenwallner wrote:  
> >>> Hi,
> >>>
> >>> in pkg_nofetch, beyond to "direct the user to download relevant  
> >source files",  
> >>> I've found it useful to tell the user which filesystem directory to  
> >put the  
> >>> files into once downloaded.
> >>>
> >>> Beyond that, I've also found it useful to tell the user whether a  
> >relevant  
> >>> source file is 'already there' or 'still missing'.
> >>>
> >>> Since the EAPI 6 related update to pkg_* phases to not have access  
> >to DISTDIR  
> >>> (even in earlier EAPI) any more, I'm wondering if both informations  
> >are still  
> >>> available to pkg_nofetch in one or another way.
> >>>
> >>> Any idea?
> >>>
> >>> Or is my only option to reduce the information to "all these files  
> >need to be  
> >>> put in your DISTDIR", requiring the user to find out both the right  
> >DISTDIR  
> >>> and which of the listed files are still missing herself?
> >>>  
> >> 
> >> How would you know whether the file in DISTDIR is correct and  
> >complete?  
> >>   
> >Well, pkg_nofetch is called only if some files are still missing,
> >so portage really should have checked them before, and eventually
> >renamed invalid files to "checksum_failure", no?  
> 
> Maybe. That's entirely undefined behavior.
> 


Does the construction of the shadow dir happen before checksum etc
validation? Or does it happen afterwards?

One option would potentially be to only create entries in the shadow
dir if they're verified to be complete as per Manifest.

Then a simple -e test is all that's needed to know if its downloaded
yet. ( But PMS will need to stipulate which phases -e tests should be
able to check this )

Outside that, perhaps a future EAPI could have a function for this
purpose, where the function is defined to only return a true value if
the distfile exists *and* is Manifestly correct ( doing on-demand
verification and possibly caching it )


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]