[gentoo-dev] [EAPI 8 RFC] Selective fetch/mirror (un-)restriction

[gentoo-dev] [EAPI 8 RFC] Selective fetch/mirror (un-)restriction

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1815 bytes --]

Hello, everyone.

I'd like to start a series of mails dedicated to features proposed for
including in EAPI 8.  For a start, I'd like to discuss the topic of
selective fetch restriction [1].  It has been discussed at least in 2013
[2], and since it's finally got chance to be included, I think it's
worthwhile to rehash it.


The problem
===========
Fetch/mirror restriction as of now can only be applied to all distfiles
at once.  This causes problems in the (rather rare) case when we'd like
to add some additional files to SRC_URI that do not require the big
restriction.  As a result, the user is forced to manually fetch all
the files even if only one truly requires it.


Proposed solution
=================
The current proposal is based on extending the current URI syntax to
permit excluding individual files from the restriction.  The idea is to
prepend 'fetch+' to protocol to undo fetch restriction, or to prepend
'mirror+' to undo fetch & mirror restrictions.

Example 1: removing mirror restriction from files

RESTRICT="mirror"
SRC_URI="https://example.com/you-cant-mirror-this.tar.bz2
  mirror+https://example.com/but-you-can-mirror-this.tar.gz"

Example 2: removing fetch & mirror restriction from files

RESTRICT="fetch"
SRC_URI="https://example.com/you-cant-fetch-this.zip
  mirror+https://example.com/but-you-can-mirror-this.tar.gz"

Example 3: removing fetch restriction while leaving mirror restriction

RESTRICT="fetch"
SRC_URI="https://example.com/you-cant-fetch-this.zip
   fetch+https://example.com/you-cant-mirror-this.tar.bz2"


Comments
========
WDYT?


References
==========
[1] https://bugs.gentoo.org/371413
[2] https://archives.gentoo.org/gentoo-dev/message/b0823618d5d3cc61bbed1e88dc2f144d

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: [gentoo-dev] [EAPI 8 RFC] Selective fetch/mirror (un-)restriction

[Francesco Riosa]

[-- Attachment #1: Type: text/plain, Size: 345 bytes --]

Il giorno lun 16 dic 2019 alle ore 13:39 Michał Górny <mgorny@gentoo.org>
ha scritto:

>
>
> Comments
> ========
> WDYT?
>
> what about getting rid of RESTRICT="fetch" and manage everything inside
SRC_URI? Would that be technically feasible?
Ideally marking only the not re-distributable download and leaving
untouched the others

[-- Attachment #2: Type: text/html, Size: 676 bytes --]

Re: [gentoo-dev] [EAPI 8 RFC] Selective fetch/mirror (un-)restriction

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 1614 bytes --]

>>>>> On Mon, 16 Dec 2019, Michał Górny wrote:

> Proposed solution
> =================
> The current proposal is based on extending the current URI syntax to
> permit excluding individual files from the restriction.  The idea is to
> prepend 'fetch+' to protocol to undo fetch restriction, or to prepend
> 'mirror+' to undo fetch & mirror restrictions.

> Example 1: removing mirror restriction from files

> RESTRICT="mirror"
> SRC_URI="https://example.com/you-cant-mirror-this.tar.bz2
>   mirror+https://example.com/but-you-can-mirror-this.tar.gz"

> Example 2: removing fetch & mirror restriction from files

> RESTRICT="fetch"
> SRC_URI="https://example.com/you-cant-fetch-this.zip
>   mirror+https://example.com/but-you-can-mirror-this.tar.gz"

> Example 3: removing fetch restriction while leaving mirror restriction

> RESTRICT="fetch"
> SRC_URI="https://example.com/you-cant-fetch-this.zip
>    fetch+https://example.com/you-cant-mirror-this.tar.bz2"

Looks good, but what is slightly confusing is that it doesn't map
one-to-one to the RESTRICT tokens:

- RESTRICT="mirror" enables mirror restriction, and it is undone by
  "mirror+", as expected.

- RESTRICT="fetch" enables both fetch and mirror restriction, but it is
  undone by "mirror+" as well, not by "fetch+" (which disables only
  fetch restriction).

I had already asked this in bug 371413 [1], but is there an actual usage
case for example 3? Because if there isn't, we might get away with only
supporting "mirror+", which should be less error prone.

Ulrich

> [1] https://bugs.gentoo.org/371413

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: [gentoo-dev] [EAPI 8 RFC] Selective fetch/mirror (un-)restriction

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 412 bytes --]

>>>>> On Mon, 16 Dec 2019, Francesco Riosa wrote:

> what about getting rid of RESTRICT="fetch" and manage everything
> inside SRC_URI? Would that be technically feasible? Ideally marking
> only the not re-distributable download and leaving untouched the
> others

That would have the disadvantage that mirror and bindist restrictions
(which are strongly correlated) would be listed in different places.

Ulrich

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: [gentoo-dev] [EAPI 8 RFC] Selective fetch/mirror (un-)restriction

[Rich Freeman]

On Mon, Dec 16, 2019 at 8:33 AM Ulrich Mueller <ulm@gentoo.org> wrote:
>
> >>>>> On Mon, 16 Dec 2019, Francesco Riosa wrote:
>
> > what about getting rid of RESTRICT="fetch" and manage everything
> > inside SRC_URI? Would that be technically feasible? Ideally marking
> > only the not re-distributable download and leaving untouched the
> > others
>
> That would have the disadvantage that mirror and bindist restrictions
> (which are strongly correlated) would be listed in different places.

An obvious way to address that is to also move the mirror restriction
into SRC_URI, so that they are both exclusively in this location as
well.  I believe those are the only two redistribution-oriented
options for RESTRICT currently.

-- 
Rich

Re: [gentoo-dev] [EAPI 8 RFC] Selective fetch/mirror (un-)restriction

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1927 bytes --]

On Mon, 2019-12-16 at 14:16 +0100, Ulrich Mueller wrote:
> > > > > > On Mon, 16 Dec 2019, Michał Górny wrote:
> > Proposed solution
> > =================
> > The current proposal is based on extending the current URI syntax to
> > permit excluding individual files from the restriction.  The idea is to
> > prepend 'fetch+' to protocol to undo fetch restriction, or to prepend
> > 'mirror+' to undo fetch & mirror restrictions.
> > Example 1: removing mirror restriction from files
> > RESTRICT="mirror"
> > SRC_URI="https://example.com/you-cant-mirror-this.tar.bz2
> >   mirror+https://example.com/but-you-can-mirror-this.tar.gz"
> > Example 2: removing fetch & mirror restriction from files
> > RESTRICT="fetch"
> > SRC_URI="https://example.com/you-cant-fetch-this.zip
> >   mirror+https://example.com/but-you-can-mirror-this.tar.gz"
> > Example 3: removing fetch restriction while leaving mirror restriction
> > RESTRICT="fetch"
> > SRC_URI="https://example.com/you-cant-fetch-this.zip
> >    fetch+https://example.com/you-cant-mirror-this.tar.bz2"
> 
> Looks good, but what is slightly confusing is that it doesn't map
> one-to-one to the RESTRICT tokens:
> 
> - RESTRICT="mirror" enables mirror restriction, and it is undone by
>   "mirror+", as expected.
> 
> - RESTRICT="fetch" enables both fetch and mirror restriction, but it is
>   undone by "mirror+" as well, not by "fetch+" (which disables only
>   fetch restriction).
> 
> I had already asked this in bug 371413 [1], but is there an actual usage
> case for example 3? Because if there isn't, we might get away with only
> supporting "mirror+", which should be less error prone.
> 

Does this really solve the problem?  The labels are still the other way
around, except that you throw 'fetch+' away as invalid.

While at it, I'm wondering if 'mirror+mirror://foo' can be confusing.

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: [gentoo-dev] [EAPI 8 RFC] Selective fetch/mirror (un-)restriction

[Michał Górny]

Dnia December 16, 2019 1:16:12 PM UTC, Ulrich Mueller <ulm@gentoo.org> napisał(a):
>>>>>> On Mon, 16 Dec 2019, Michał Górny wrote:
>
>> Proposed solution
>> =================
>> The current proposal is based on extending the current URI syntax to
>> permit excluding individual files from the restriction.  The idea is
>to
>> prepend 'fetch+' to protocol to undo fetch restriction, or to prepend
>> 'mirror+' to undo fetch & mirror restrictions.
>
>> Example 1: removing mirror restriction from files
>
>> RESTRICT="mirror"
>> SRC_URI="https://example.com/you-cant-mirror-this.tar.bz2
>>   mirror+https://example.com/but-you-can-mirror-this.tar.gz"
>
>> Example 2: removing fetch & mirror restriction from files
>
>> RESTRICT="fetch"
>> SRC_URI="https://example.com/you-cant-fetch-this.zip
>>   mirror+https://example.com/but-you-can-mirror-this.tar.gz"
>
>> Example 3: removing fetch restriction while leaving mirror
>restriction
>
>> RESTRICT="fetch"
>> SRC_URI="https://example.com/you-cant-fetch-this.zip
>>    fetch+https://example.com/you-cant-mirror-this.tar.bz2"
>
>Looks good, but what is slightly confusing is that it doesn't map
>one-to-one to the RESTRICT tokens:
>
>- RESTRICT="mirror" enables mirror restriction, and it is undone by
>  "mirror+", as expected.
>
>- RESTRICT="fetch" enables both fetch and mirror restriction, but it is
>  undone by "mirror+" as well, not by "fetch+" (which disables only
>  fetch restriction).
>
>I had already asked this in bug 371413 [1], but is there an actual
>usage
>case for example 3? Because if there isn't, we might get away with only
>supporting "mirror+", which should be less error prone.

Actually, what about the original example provided by Vadim? It's a game + translations, all rights reserved. We can't mirror them but we can fetch them.

>
>Ulrich
>
>> [1] https://bugs.gentoo.org/371413


--
Best regards, 
Michał Górny

Re: [gentoo-dev] [EAPI 8 RFC] Selective fetch/mirror (un-)restriction

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 696 bytes --]

>>>>> On Fri, 20 Dec 2019, Michał Górny wrote:

>>> Example 3: removing fetch restriction while leaving mirror
>>> restriction

>>> RESTRICT="fetch"
>>> SRC_URI="https://example.com/you-cant-fetch-this.zip
>>> fetch+https://example.com/you-cant-mirror-this.tar.bz2"

>> I had already asked this in bug 371413 [1], but is there an actual
>> usage case for example 3? Because if there isn't, we might get away
>> with only supporting "mirror+", which should be less error prone.

> Actually, what about the original example provided by Vadim? It's a
> game + translations, all rights reserved. We can't mirror them but we
> can fetch them.

Thank you. That answers my question.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]