[gentoo-dev] [Future EAPI] src_fetch() phase function to support VCS fetching

[gentoo-dev] [Future EAPI] src_fetch() phase function to support VCS fetching

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 2412 bytes --]

Hello,

As Sid Hayn raised today on #gentoo-portage, it would be useful to
finally have portage able to fetch updates from VCS-es independently
of src_unpack(). This could be used, for example, on machines
temporarily connected to the network -- one would then fetch files
while connected to the network, and perform the updates later.

There are a few ways how we could handle that but the cleanest and most
universal one seems to be defining a src_fetch() phase function
in a future EAPI.

In the EAPIs supporting src_fetch(), that phase function would be used
by PM when requesting the files to be fetched. A default_src_fetch()
will be declared as well, providing implementation-defined code
fetching files like they are fetched now. Older EAPIs will simply
always use that default.

The phase function would be disjoint from the normal merge process,
much like pkg_pretend(). In portage, it will be called as 'portage'
user if FEATURES=userfetch is enabled.

VCS eclasses supporting separated fetching would define two phase
functions:
- src_fetch() which would be responsible for fetching updates,
- src_unpack() which would be responsible for checking out the source
  to work directory.

The remaining issue is handling dependencies on the tools necessary to
do fetching. For default_src_fetch(), we can assume that the package
manager provides the necessary tools. For custom src_fetch(), we would
need either to:

1) require satisfying whole DEPEND when fetching -- probably pointless,
   as it will make --fetchonly almost impossible when doing initial
   installs;

2) introduce a new dependency type (please do not get into details how
   we do it -- we will discuss that another time, at the moment please
   just keep it as 'new dependency type') -- and we probably end up
   having a switch for --fetchonly without installing deps (thus
   omitting packages where they are not satisfied), and with deps;

3) [ugly!] assume that src_fetch() should check for its deps and fail
   if they are not satisfied. If that's mostly for live ebuilds, it may
   be acceptable. Then the package manager will just have one 'fetch
   failed' on --fetchonly (or early pre-fetch), and it will have to
   invoke src_fetch() after satisfying the deps, before src_unpack().

What do you think? What are your ideas, suggestions?

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-dev] [Future EAPI] src_fetch() phase function to support VCS fetching

[Rick "Zero_Chaos" Farina]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 09/04/2012 12:43 PM, Michał Górny wrote:
> Hello,
> 
> As Sid Hayn raised today on #gentoo-portage, it would be useful to
If you insist on using real names mine is Rick ;-)
> finally have portage able to fetch updates from VCS-es independently
> of src_unpack(). This could be used, for example, on machines
> temporarily connected to the network -- one would then fetch files
> while connected to the network, and perform the updates later.
> 
> There are a few ways how we could handle that but the cleanest and most
> universal one seems to be defining a src_fetch() phase function
> in a future EAPI.
> 
> In the EAPIs supporting src_fetch(), that phase function would be used
> by PM when requesting the files to be fetched. A default_src_fetch()
> will be declared as well, providing implementation-defined code
> fetching files like they are fetched now. Older EAPIs will simply
> always use that default.
> 
> The phase function would be disjoint from the normal merge process,
> much like pkg_pretend(). In portage, it will be called as 'portage'
> user if FEATURES=userfetch is enabled.
> 
> VCS eclasses supporting separated fetching would define two phase
> functions:
> - src_fetch() which would be responsible for fetching updates,
> - src_unpack() which would be responsible for checking out the source
>   to work directory.
> 
> The remaining issue is handling dependencies on the tools necessary to
> do fetching. For default_src_fetch(), we can assume that the package
> manager provides the necessary tools. For custom src_fetch(), we would
> need either to:
> 
> 1) require satisfying whole DEPEND when fetching -- probably pointless,
>    as it will make --fetchonly almost impossible when doing initial
>    installs;
> 
> 2) introduce a new dependency type (please do not get into details how
>    we do it -- we will discuss that another time, at the moment please
>    just keep it as 'new dependency type') -- and we probably end up
>    having a switch for --fetchonly without installing deps (thus
>    omitting packages where they are not satisfied), and with deps;
> 
> 3) [ugly!] assume that src_fetch() should check for its deps and fail
>    if they are not satisfied. If that's mostly for live ebuilds, it may
>    be acceptable. Then the package manager will just have one 'fetch
>    failed' on --fetchonly (or early pre-fetch), and it will have to
>    invoke src_fetch() after satisfying the deps, before src_unpack().
> 
> What do you think? What are your ideas, suggestions?
> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBAgAGBQJQRjQeAAoJEKXdFCfdEflKdDAP/jyV8+NwEBX+etKq7e+qzbVg
oIgjqOLfqYEmbEkO9WMsUltsf8YATy9zGB1itRiLUb5v+eptjEDvp2dx/AA4YVio
RLYwtMBJsESXd6d+3PbCTeoZHTZwb7ue6yY+qC0auyC4mD6nrkDY8swungcDwFHN
GZ99gAtRnQ5h8+phrKHyWzePTkBN5+32GCIn2XfRwMo+T0tGyTeMGqYfPJPlEu0y
LNJmdBU0TfzE7N8QA4sp/IDQQvBmZNpH6aQM+zkJWpvBUXBATdIPZZHBAy55TWaJ
t9KxiHD6XF/h0ShlTIpsbfSj31FTil2l/LfhAQdXSrB4GPxLaoovOJb2vl9L+ZyP
QJclNeL1kNaOO1MIm20tGJOV6Wh0iHsn/fJlbh4YYn4PaNF+F+UXyp69uGcniuCe
PxFC/aosq39LoFRrZdyRfx3DWPXnhYfcFWRwEDosa/Qb/Mbljs+BZRRPIB2Y8ItV
9j5WokM6BkNPNeoNnaS1JvCaac7xiurizs7IaXxfYC5q8aZja1OdDrV9Jh7KxIUe
q8rsKMk6y9KqGvSBfW3Y9JDgIdUUG8x0bVM/j9+gqOtPH5uFgPRnZxX/Hfb0+1J5
iLjFXgwLl2AtEuflY07KfKB9xyEV58x8gkCo/BUX8Y8E6re5/4o+kNiArQL2Z+YS
SOD2BmvyVHOlurug9Lq4
=FOao
-----END PGP SIGNATURE-----

Re: [gentoo-dev] [Future EAPI] src_fetch() phase function to support VCS fetching

[Michael Mol]

On Tue, Sep 4, 2012 at 12:43 PM, Michał Górny <mgorny@gentoo.org> wrote:
> Hello,
>
> As Sid Hayn raised today on #gentoo-portage, it would be useful to
> finally have portage able to fetch updates from VCS-es independently
> of src_unpack(). This could be used, for example, on machines
> temporarily connected to the network -- one would then fetch files
> while connected to the network, and perform the updates later.
>
> There are a few ways how we could handle that but the cleanest and most
> universal one seems to be defining a src_fetch() phase function
> in a future EAPI.
>
> In the EAPIs supporting src_fetch(), that phase function would be used
> by PM when requesting the files to be fetched. A default_src_fetch()
> will be declared as well, providing implementation-defined code
> fetching files like they are fetched now. Older EAPIs will simply
> always use that default.
>
> The phase function would be disjoint from the normal merge process,
> much like pkg_pretend(). In portage, it will be called as 'portage'
> user if FEATURES=userfetch is enabled.
>
> VCS eclasses supporting separated fetching would define two phase
> functions:
> - src_fetch() which would be responsible for fetching updates,
> - src_unpack() which would be responsible for checking out the source
>   to work directory.

The 'checking out' language for src_unpack() sounds like it assumes a
DVCS such as mercurial or git. What about cvs or svn, where fetching
is also checking out? (This is probably a trivial thing to clear up,
though.)

Also, where would the local copy go? distfiles? It's common for
distfiles to be stored on, e.g. an NFS mount, so you may need to be
careful not to place repositories there which have filesystem
semantics that are disagreeable to NFS. (The only example I know of
off the top of my head is svn, where the documentation warns against
using the dbd backend on top of NFS.)

Other common remote mounts (such as cifs) may have restrictions that
could force munging of filenames, too, and VCS infrastructures (or
even unpacked checkouts with strange filenames) placed on those
filesystems may have unanticipated results.

It may be helpful to have some kind of adapter mount in place, or even
generate a tarball of the local copy and store that instead. (That'd
be problematic if multiple boxes were modifying the local copy on the
same share, but that's obviously problematic anyway.)

-- 
:wq

Re: [gentoo-dev] [Future EAPI] src_fetch() phase function to support VCS fetching

[Rick "Zero_Chaos" Farina]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 09/04/2012 12:43 PM, Michał Górny wrote:
> Hello,
> 
> As Sid Hayn raised today on #gentoo-portage, it would be useful to
> finally have portage able to fetch updates from VCS-es independently
> of src_unpack(). This could be used, for example, on machines
> temporarily connected to the network -- one would then fetch files
> while connected to the network, and perform the updates later.
> 
> There are a few ways how we could handle that but the cleanest and most
> universal one seems to be defining a src_fetch() phase function
> in a future EAPI.
> 
> In the EAPIs supporting src_fetch(), that phase function would be used
> by PM when requesting the files to be fetched. A default_src_fetch()
> will be declared as well, providing implementation-defined code
> fetching files like they are fetched now. Older EAPIs will simply
> always use that default.
> 
> The phase function would be disjoint from the normal merge process,
> much like pkg_pretend(). In portage, it will be called as 'portage'
> user if FEATURES=userfetch is enabled.
> 
> VCS eclasses supporting separated fetching would define two phase
> functions:
> - src_fetch() which would be responsible for fetching updates,
> - src_unpack() which would be responsible for checking out the source
>   to work directory.
> 
> The remaining issue is handling dependencies on the tools necessary to
> do fetching. For default_src_fetch(), we can assume that the package
> manager provides the necessary tools. For custom src_fetch(), we would
> need either to:
> 
> 1) require satisfying whole DEPEND when fetching -- probably pointless,
>    as it will make --fetchonly almost impossible when doing initial
>    installs;
> 
> 2) introduce a new dependency type (please do not get into details how
>    we do it -- we will discuss that another time, at the moment please
>    just keep it as 'new dependency type') -- and we probably end up
>    having a switch for --fetchonly without installing deps (thus
>    omitting packages where they are not satisfied), and with deps;
> 
> 3) [ugly!] assume that src_fetch() should check for its deps and fail
>    if they are not satisfied. If that's mostly for live ebuilds, it may
>    be acceptable. Then the package manager will just have one 'fetch
>    failed' on --fetchonly (or early pre-fetch), and it will have to
>    invoke src_fetch() after satisfying the deps, before src_unpack().

I believe the easiest (and honestly most sane) method is to simply have
src_fetch in the live classes check for needed deps and die (with a
"please emerge blah") if deps are not found.  Adding something like
FDEPEND just seems to be getting way too crazy on the dependency tree
AND would require things to build during fetch-only which doesn't make
sense.

Thanks,
Zero
> 
> What do you think? What are your ideas, suggestions?
> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBAgAGBQJQRjTjAAoJEKXdFCfdEflKGPoP+wYhzeFNcXO9Lv4qm5/oPXuC
RIrCs2q6erWxowFbiTO//XILVbiH1nHBWx/uJV9S0hTM+dBRIh/zZaCXy5PYnLrK
dTKiHgmpNPgTQ41QGI7BZl4lQFGLgGfdJnCncSTLmLZQtMGwXD4jZ8SQ/QE1wbwK
rpHtYQ/Z5kUKFJG25MRFtFZ2ifZQVpVIPmmrFfyDh+1l9Oh8AtF6XKZTriwX1ppG
osPz8jo2XtoYejqvD0kFEZwc5C7FKmULYyrB+tcl5dmOgf60LvcDTBkUGi7U2ewk
DWhVoLX0zXWMZfoEi3c6GRsJgZ81yXIMfPC+SUYipdfBvdH6iWYuxfgJpSioUtbV
s4xC4IyOGQehnt8OAe5PpHHJcqxlDVXidmVz0sKkmJB3dM6rIsamiNZajmKVrnyh
1zle8g4XBP8AFg/fB+OiOSuYNoI5GnC2D/rp7zAtajl+0GtVWvApPUFEa0DP8LlR
bYXMIGpx+LAi9rYlEdNs3tW1C4OVjZpCKYU7cjNO15f2ZOVX4WNmAWtRmpoG6O5l
zL1Mv6DRgdtoITfYfogupZBgjynHsCzKL1Mb2skqtmJ853ZjV4HGgXqxcica12O7
EoJ65EU5rrzCGSUz5rqunNwK9A11VBjxXWJ/e29NKt2vACxRuiwMXkl7EK2a/oCA
pTBk5q6CvQFJCyzw2bee
=ueCf
-----END PGP SIGNATURE-----

Re: [gentoo-dev] [Future EAPI] src_fetch() phase function to support VCS fetching

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 2474 bytes --]

On Tue, 4 Sep 2012 13:02:36 -0400
Michael Mol <mikemol@gmail.com> wrote:

> On Tue, Sep 4, 2012 at 12:43 PM, Michał Górny <mgorny@gentoo.org>
> wrote:
> > Hello,
> >
> > As Sid Hayn raised today on #gentoo-portage, it would be useful to
> > finally have portage able to fetch updates from VCS-es independently
> > of src_unpack(). This could be used, for example, on machines
> > temporarily connected to the network -- one would then fetch files
> > while connected to the network, and perform the updates later.
> >
> > There are a few ways how we could handle that but the cleanest and
> > most universal one seems to be defining a src_fetch() phase function
> > in a future EAPI.
> >
> > In the EAPIs supporting src_fetch(), that phase function would be
> > used by PM when requesting the files to be fetched. A
> > default_src_fetch() will be declared as well, providing
> > implementation-defined code fetching files like they are fetched
> > now. Older EAPIs will simply always use that default.
> >
> > The phase function would be disjoint from the normal merge process,
> > much like pkg_pretend(). In portage, it will be called as 'portage'
> > user if FEATURES=userfetch is enabled.
> >
> > VCS eclasses supporting separated fetching would define two phase
> > functions:
> > - src_fetch() which would be responsible for fetching updates,
> > - src_unpack() which would be responsible for checking out the
> > source to work directory.
> 
> The 'checking out' language for src_unpack() sounds like it assumes a
> DVCS such as mercurial or git. What about cvs or svn, where fetching
> is also checking out? (This is probably a trivial thing to clear up,
> though.)

They either stay with src_unpack() or do 'cvs up' in src_fetch()
and just copy files over in src_unpack(). Anyway, that's what they do
now -- update the copy in distfiles/cvs-src and then copy it.

> Also, where would the local copy go? distfiles? It's common for
> distfiles to be stored on, e.g. an NFS mount, so you may need to be
> careful not to place repositories there which have filesystem
> semantics that are disagreeable to NFS. (The only example I know of
> off the top of my head is svn, where the documentation warns against
> using the dbd backend on top of NFS.)

The actual code will be eclass-dependent, so it will go whatever it
goes now. No magic, no exherbo. Just plain phase function.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-dev] [Future EAPI] src_fetch() phase function to support VCS fetching

[Rick "Zero_Chaos" Farina]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 09/04/2012 01:02 PM, Michael Mol wrote:
> On Tue, Sep 4, 2012 at 12:43 PM, Michał Górny <mgorny@gentoo.org> wrote:
>> Hello,
>>
>> As Sid Hayn raised today on #gentoo-portage, it would be useful to
>> finally have portage able to fetch updates from VCS-es independently
>> of src_unpack(). This could be used, for example, on machines
>> temporarily connected to the network -- one would then fetch files
>> while connected to the network, and perform the updates later.
>>
>> There are a few ways how we could handle that but the cleanest and most
>> universal one seems to be defining a src_fetch() phase function
>> in a future EAPI.
>>
>> In the EAPIs supporting src_fetch(), that phase function would be used
>> by PM when requesting the files to be fetched. A default_src_fetch()
>> will be declared as well, providing implementation-defined code
>> fetching files like they are fetched now. Older EAPIs will simply
>> always use that default.
>>
>> The phase function would be disjoint from the normal merge process,
>> much like pkg_pretend(). In portage, it will be called as 'portage'
>> user if FEATURES=userfetch is enabled.
>>
>> VCS eclasses supporting separated fetching would define two phase
>> functions:
>> - src_fetch() which would be responsible for fetching updates,
>> - src_unpack() which would be responsible for checking out the source
>>   to work directory.
> 
> The 'checking out' language for src_unpack() sounds like it assumes a
> DVCS such as mercurial or git. What about cvs or svn, where fetching
> is also checking out? (This is probably a trivial thing to clear up,
> though.)
> 
> Also, where would the local copy go? distfiles? It's common for
> distfiles to be stored on, e.g. an NFS mount, so you may need to be
> careful not to place repositories there which have filesystem
> semantics that are disagreeable to NFS. (The only example I know of
> off the top of my head is svn, where the documentation warns against
> using the dbd backend on top of NFS.)
> 
> Other common remote mounts (such as cifs) may have restrictions that
> could force munging of filenames, too, and VCS infrastructures (or
> even unpacked checkouts with strange filenames) placed on those
> filesystems may have unanticipated results.
> 
> It may be helpful to have some kind of adapter mount in place, or even
> generate a tarball of the local copy and store that instead. (That'd
> be problematic if multiple boxes were modifying the local copy on the
> same share, but that's obviously problematic anyway.)
> 

All the live eclasses already drop a checkout (or whatever term you
like) of the repo in /usr/portage/distfiles.  What we are talking about
here is separating the "download/checkout into /usr/portage/distfiles"
from the "copy from /usr/portage/distfiles to ${S}". Making this two
separate phases would allow a reasonably sane "emerge -f blah" support
to fetch the live sources before build.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBAgAGBQJQRjk9AAoJEKXdFCfdEflKeocQAJgSMaycMkvmdalYWUBU4lRJ
eNilSxWoMAuln+1FMRv9HxavQCMzkw/h9o3Lwy75Kv2+Z/MZy60caG7yplexSKEf
Z3apTwU1Cbj7WasDNkNRXgoHHsn55Fnr28lueFmcAlvMWDnBV0TmXCYFiRh961hN
7Co3wbHT4ahk1TTTURFDoWbE2R24e1UJFzrWRRO4oyoIDqPqOIOqhR+4Nnl5vsXa
/UQmvIqgzxlWgdjeaWceQxgexPt0g+ZwFUFIF6FhL3xqoF6kokBj64VxPRAgK05X
x78LSsul+A0H6sAU4S9e5CmZekxFyNanF3PJ5aZaAfp8dPTWaUFIhJHh3jDFktdT
zaO1LVpEOVnUenuV6XgpETN8XxThZHzXFRokuyR3Aza41/p3eg/jo07DGmUBLRP2
fu9h/ZVwrcPixgpTNd2+8YhxIxuoka7CmJiVb4ncAJ/Yp95G64dM57DXf46rKPTC
6LX/5jQgXcR+WbQswLumZJ+EJ7aIzGhJRFGXZmbNvWVrzFxfB7vtgUl6NUsKmzTo
gffrxrZexzlnvkCThrdM1C89BdAdkWATICiAAEkf2P9gzEdtB+p65VBqYcNzFc+Q
X9kw+X6lnAn6D/KqJdSAG9plhjRuZ9JTGnZ/ZlucyP7uVnmHV5+NOVn5MYvZ/Sso
z2F4QPIJi3C9LQkOSyKF
=vy05
-----END PGP SIGNATURE-----

Re: [gentoo-dev] [Future EAPI] src_fetch() phase function to support VCS fetching

[Zac Medico]

On 09/04/2012 10:05 AM, Rick "Zero_Chaos" Farina wrote:
> I believe the easiest (and honestly most sane) method is to simply have
> src_fetch in the live classes check for needed deps and die (with a
> "please emerge blah") if deps are not found.  Adding something like
> FDEPEND just seems to be getting way too crazy on the dependency tree
> AND would require things to build during fetch-only which doesn't make
> sense.

I think it's nicer to have FDEPEND because it makes the deps more
complete, so the package manager can bail out when necessary, without
even executing src_fetch. In the case of --fetchonly the package manager
could simply bail out if the deps are not installed (like how it bails
out for --buildpkgonly when the deps aren't installed).
-- 
Thanks,
Zac

Re: [gentoo-dev] [Future EAPI] src_fetch() phase function to support VCS fetching

[Ian Stakenvicius]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 04/09/12 01:32 PM, Zac Medico wrote:
> On 09/04/2012 10:05 AM, Rick "Zero_Chaos" Farina wrote:
>> I believe the easiest (and honestly most sane) method is to
>> simply have src_fetch in the live classes check for needed deps
>> and die (with a "please emerge blah") if deps are not found.
>> Adding something like FDEPEND just seems to be getting way too
>> crazy on the dependency tree AND would require things to build
>> during fetch-only which doesn't make sense.
> 
> I think it's nicer to have FDEPEND because it makes the deps more 
> complete, so the package manager can bail out when necessary,
> without even executing src_fetch. In the case of --fetchonly the
> package manager could simply bail out if the deps are not installed
> (like how it bails out for --buildpkgonly when the deps aren't
> installed).

Just looking into the future here; would things like archivers or
other helpers used by src_unpack move to FDEPEND as well?  or would
this be limited solely to tools that data transfer?



-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)

iF4EAREIAAYFAlBGXQYACgkQ2ugaI38ACPDKGwEAlvqkQHBv2vDtGfYNYrIIMIV9
1le8HgtXRhIu9FOhw1EA/3m1xc4zNcWMslK6RiV5sn4mBVXUQt6Ry2m/Nar+ccFq
=DqcL
-----END PGP SIGNATURE-----

Re: [gentoo-dev] [Future EAPI] src_fetch() phase function to support VCS fetching

[Rick "Zero_Chaos" Farina]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 09/04/2012 03:56 PM, Ian Stakenvicius wrote:
> On 04/09/12 01:32 PM, Zac Medico wrote:
>> On 09/04/2012 10:05 AM, Rick "Zero_Chaos" Farina wrote:
>>> I believe the easiest (and honestly most sane) method is to
>>> simply have src_fetch in the live classes check for needed deps
>>> and die (with a "please emerge blah") if deps are not found.
>>> Adding something like FDEPEND just seems to be getting way too
>>> crazy on the dependency tree AND would require things to build
>>> during fetch-only which doesn't make sense.
> 
>> I think it's nicer to have FDEPEND because it makes the deps more 
>> complete, so the package manager can bail out when necessary,
>> without even executing src_fetch. In the case of --fetchonly the
>> package manager could simply bail out if the deps are not installed
>> (like how it bails out for --buildpkgonly when the deps aren't
>> installed).
> 
> Just looking into the future here; would things like archivers or
> other helpers used by src_unpack move to FDEPEND as well?  or would
> this be limited solely to tools that data transfer?

We are talking about things required for src_fetch (the download) so no,
things required for src_unpack have no real place in this as far as I am
concerned.
> 
> 
> 
> 
> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBAgAGBQJQRl8nAAoJEKXdFCfdEflKoA0P/152JYx2/62tou9kcUCEILkz
DXGfO5bY+A1DYrCsVAR9H7C4dzmrULU0JNjUl3nMpGFKXMAvWQbai2uDICMECZVg
QCwHVzxyHD9TzdVAyp928dtn/FtnPvaUV8WHIugFuaZZM/mmACsyEYxFqusMXu51
KkMzs1F9TatL3uoAMuvT0KLiMWHmxw6OrmMrqRX1T7gP4AIy7cD5h8v7DOzWORLQ
x12/HGAA2HLDgfhwrNymRs+OIqSLXE54DmAfvOglB6Jfw/I/N6HoY1PQ1vZaW7jU
P+t2l7VbRsINmHUk+tbl+7A5F2XBE5GphSh8oIN7yuToSwtbstlGFmTGhb/V0gpM
HUpI5v/6B40fc+DgEuAMAZjhUBT+1YIbIO9WVUaXN/I9JBA22FBZSm4qu59qWg/U
DX+Jhul9M+A5QPdNBios0K5BYxFXamx/i52w/361NdLV2Onn1zHQT95oC7RKOGJG
fZu6PhtkW7Z/CaYSja7TKa8H5pl/+Bj5Mgz7cGlwz41AP4Q6+omfnxlLf+iJME07
rBfKe4RuhZuUXamu+SfW7OdNY8Zqm9skSGw1HbHFVHDQIXKtt8Pv9tr1m364h+W9
r6SUE1xdJaCbtzh2/ajpu9EIVLoVuW/NEFvQIMd7wn3R2PTKgNHOnkLB6c8j/qaa
nWe+jCpIt2uSd2Syceqh
=4kd2
-----END PGP SIGNATURE-----

Re: [gentoo-dev] [Future EAPI] src_fetch() phase function to support VCS fetching

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1359 bytes --]

On Tue, 04 Sep 2012 15:56:54 -0400
Ian Stakenvicius <axs@gentoo.org> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
> 
> On 04/09/12 01:32 PM, Zac Medico wrote:
> > On 09/04/2012 10:05 AM, Rick "Zero_Chaos" Farina wrote:
> >> I believe the easiest (and honestly most sane) method is to
> >> simply have src_fetch in the live classes check for needed deps
> >> and die (with a "please emerge blah") if deps are not found.
> >> Adding something like FDEPEND just seems to be getting way too
> >> crazy on the dependency tree AND would require things to build
> >> during fetch-only which doesn't make sense.
> > 
> > I think it's nicer to have FDEPEND because it makes the deps more 
> > complete, so the package manager can bail out when necessary,
> > without even executing src_fetch. In the case of --fetchonly the
> > package manager could simply bail out if the deps are not installed
> > (like how it bails out for --buildpkgonly when the deps aren't
> > installed).
> 
> Just looking into the future here; would things like archivers or
> other helpers used by src_unpack move to FDEPEND as well?  or would
> this be limited solely to tools that data transfer?

src_fetch() only. Archivers belong to src_unpack(), which belongs to
the main phase function cycle -> DEPEND.


-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: Re: [gentoo-dev] [Future EAPI] src_fetch() phase function to support VCS fetching

[Andreas K. Huettel]

[-- Attachment #1: Type: text/plain, Size: 398 bytes --]

> Just looking into the future here; would things like archivers or
> other helpers used by src_unpack move to FDEPEND as well?  or would
> this be limited solely to tools that data transfer?

We should keep the data transfer and the unpack phase clearly separated. So, 
this would best really be for data transfer only. 

-- 
Andreas K. Huettel
Gentoo Linux developer
kde, sci, arm, tex, printing

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] [Future EAPI] src_fetch() phase function to support VCS fetching

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 916 bytes --]

On Tue, 4 Sep 2012 19:23:51 +0200
Michał Górny <mgorny@gentoo.org> wrote:
> > The 'checking out' language for src_unpack() sounds like it assumes
> > a DVCS such as mercurial or git. What about cvs or svn, where
> > fetching is also checking out? (This is probably a trivial thing to
> > clear up, though.)
> 
> They either stay with src_unpack() or do 'cvs up' in src_fetch()
> and just copy files over in src_unpack(). Anyway, that's what they do
> now -- update the copy in distfiles/cvs-src and then copy it.

This doesn't work if we have, for example, foo:1 and foo:2 both using
the same SCM repository, but different branches. Much as we'd like to
pretend that everyone uses Git, we can't really ignore this case...

So we have to decide: do we make the src_fetch copy the data somewhere
after all, or do we require that eclasses do something obscene to avoid
this?

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] [Future EAPI] src_fetch() phase function to support VCS fetching

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1737 bytes --]

On Wed, 5 Sep 2012 08:25:54 +0100
Ciaran McCreesh <ciaran.mccreesh@googlemail.com> wrote:

> On Tue, 4 Sep 2012 19:23:51 +0200
> Michał Górny <mgorny@gentoo.org> wrote:
> > > The 'checking out' language for src_unpack() sounds like it
> > > assumes a DVCS such as mercurial or git. What about cvs or svn,
> > > where fetching is also checking out? (This is probably a trivial
> > > thing to clear up, though.)
> > 
> > They either stay with src_unpack() or do 'cvs up' in src_fetch()
> > and just copy files over in src_unpack(). Anyway, that's what they
> > do now -- update the copy in distfiles/cvs-src and then copy it.
> 
> This doesn't work if we have, for example, foo:1 and foo:2 both using
> the same SCM repository, but different branches. Much as we'd like to
> pretend that everyone uses Git, we can't really ignore this case...
> 
> So we have to decide: do we make the src_fetch copy the data somewhere
> after all, or do we require that eclasses do something obscene to
> avoid this?

Eclasses have to handle themselves, if we are supporting this. There's
no point in bloating the solution for 1 theoretical package on 1000
which doesn't even exist.

And yes, it is *very* unlikely that someone uses a slotted live ebuild
with two branches being meaningful and managed in the same repo. Even
if such thing exists, it is broken anyway because you can't say that
re-fetching the branches back and forth is a correct solution. And it
breaks existing tools anyway.

Well, even I doubt eclasses need to do something obscene. The need for
that is so small that I believe if someone was crazy enough, he could
just set an appropriate storedir in ebuild.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-dev] [Future EAPI] src_fetch() phase function to support VCS fetching

[Ulrich Mueller]

>>>>> On Wed, 5 Sep 2012, Michał Górny wrote:

> And yes, it is *very* unlikely that someone uses a slotted live
> ebuild with two branches being meaningful and managed in the same
> repo.

We do that all the time for app-editors/emacs-vcs. For example, we
used to have live ebuilds for the trunk and for the emacs-24 branch,
sharing a common repository.

> Even if such thing exists, it is broken anyway because you can't say
> that re-fetching the branches back and forth is a correct solution.
> And it breaks existing tools anyway.

I haven't notice any breakage with it.

Ulrich

Re: [gentoo-dev] [Future EAPI] src_fetch() phase function to support VCS fetching

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 587 bytes --]

On Wed, 5 Sep 2012 11:07:44 +0200
Ulrich Mueller <ulm@gentoo.org> wrote:

> >>>>> On Wed, 5 Sep 2012, Michał Górny wrote:
> 
> > And yes, it is *very* unlikely that someone uses a slotted live
> > ebuild with two branches being meaningful and managed in the same
> > repo.
> 
> We do that all the time for app-editors/emacs-vcs. For example, we
> used to have live ebuilds for the trunk and for the emacs-24 branch,
> sharing a common repository.

Doesn't bzr actually work alike git? i.e. keep all commits when
switching branches?

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-dev] [Future EAPI] src_fetch() phase function to support VCS fetching

[Ulrich Mueller]

>>>>> On Wed, 5 Sep 2012, Michał Górny wrote:

> On Wed, 5 Sep 2012 11:07:44 +0200
> Ulrich Mueller <ulm@gentoo.org> wrote:

>> >>>>> On Wed, 5 Sep 2012, Michał Górny wrote:
>> 
>> > And yes, it is *very* unlikely that someone uses a slotted live
>> > ebuild with two branches being meaningful and managed in the same
>> > repo.
>> 
>> We do that all the time for app-editors/emacs-vcs. For example, we
>> used to have live ebuilds for the trunk and for the emacs-24 branch,
>> sharing a common repository.

> Doesn't bzr actually work alike git? i.e. keep all commits when
> switching branches?

Actually, bzr.eclass uses branches without a working tree (what would
be called a "bare repository" in the Git world). There's no need to
switch branches.

Ulrich

Re: Re: [gentoo-dev] [Future EAPI] src_fetch() phase function to support VCS fetching

[Andreas K. Huettel]

[-- Attachment #1: Type: text/plain, Size: 690 bytes --]

> And yes, it is *very* unlikely that someone uses a slotted live ebuild
> with two branches being meaningful and managed in the same repo. Even
> if such thing exists, it is broken anyway because you can't say that
> re-fetching the branches back and forth is a correct solution. And it
> breaks existing tools anyway.

This is done large-scale for all KDE ebuilds (in the KDE overlay) to support 
master and KDE/4.x stable branch. Most use git, so no problem; some (still) 
use subversion but will be migrated upstream soon(?).

Other examples are libreoffice (main tree, git) and cups (main tree, 
subversion).

-- 
Andreas K. Huettel
Gentoo Linux developer
kde, sci, arm, tex, printing

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] [Future EAPI] src_fetch() phase function to support VCS fetching

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 841 bytes --]

On Wed, 05 Sep 2012 12:45:16 +0200
"Andreas K. Huettel" <dilfridge@gentoo.org> wrote:
> > And yes, it is *very* unlikely that someone uses a slotted live
> > ebuild with two branches being meaningful and managed in the same
> > repo. Even if such thing exists, it is broken anyway because you
> > can't say that re-fetching the branches back and forth is a correct
> > solution. And it breaks existing tools anyway.
> 
> This is done large-scale for all KDE ebuilds (in the KDE overlay) to
> support master and KDE/4.x stable branch. Most use git, so no
> problem; some (still) use subversion but will be migrated upstream
> soon(?).
> 
> Other examples are libreoffice (main tree, git) and cups (main tree, 
> subversion).

I guess that's a pretty comprehensive "we need to do this properly"
then.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] [Future EAPI] src_fetch() phase function to support VCS fetching

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1104 bytes --]

On Wed, 5 Sep 2012 11:49:03 +0100
Ciaran McCreesh <ciaran.mccreesh@googlemail.com> wrote:

> On Wed, 05 Sep 2012 12:45:16 +0200
> "Andreas K. Huettel" <dilfridge@gentoo.org> wrote:
> > > And yes, it is *very* unlikely that someone uses a slotted live
> > > ebuild with two branches being meaningful and managed in the same
> > > repo. Even if such thing exists, it is broken anyway because you
> > > can't say that re-fetching the branches back and forth is a
> > > correct solution. And it breaks existing tools anyway.
> > 
> > This is done large-scale for all KDE ebuilds (in the KDE overlay) to
> > support master and KDE/4.x stable branch. Most use git, so no
> > problem; some (still) use subversion but will be migrated upstream
> > soon(?).
> > 
> > Other examples are libreoffice (main tree, git) and cups (main
> > tree, subversion).
> 
> I guess that's a pretty comprehensive "we need to do this properly"
> then.

Did I say we don't need to? We have the two eclasses which need to do
this properly, right? So what's your problem?

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-dev] [Future EAPI] src_fetch() phase function to support VCS fetching

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 573 bytes --]

On Wed, 5 Sep 2012 13:00:05 +0200
Michał Górny <mgorny@gentoo.org> wrote:
> > I guess that's a pretty comprehensive "we need to do this properly"
> > then.
> 
> Did I say we don't need to? We have the two eclasses which need to do
> this properly, right? So what's your problem?

The problem is that we need to work out how to deal with this at least
for Subversion, and probably for CVS too... As much as we'd like to, we
can't just roll out a src_fetch without eclass changes. This doesn't
appear to be a trivial feature to provide.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] [Future EAPI] src_fetch() phase function to support VCS fetching

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1418 bytes --]

On Wed, 5 Sep 2012 12:07:22 +0100
Ciaran McCreesh <ciaran.mccreesh@googlemail.com> wrote:

> On Wed, 5 Sep 2012 13:00:05 +0200
> Michał Górny <mgorny@gentoo.org> wrote:
> > > I guess that's a pretty comprehensive "we need to do this
> > > properly" then.
> > 
> > Did I say we don't need to? We have the two eclasses which need to
> > do this properly, right? So what's your problem?
> 
> The problem is that we need to work out how to deal with this at least
> for Subversion, and probably for CVS too... As much as we'd like to,
> we can't just roll out a src_fetch without eclass changes. This
> doesn't appear to be a trivial feature to provide.

First of all, subversion isn't a problem here. Subversion doesn't have
native branches. It's just directories in the tree, with tree having
linear history. In other words, it just works.

So it remains CVS and possibly darcs. But considering that the issue is
so unlikely and so small, we can as well assume that the affected
ebuilds should set ECVS_LOCALNAME to something with the branch. This is
in line with how we handle SRC_URI collisions.

And in any case, we may just decide to use a ECVS_LOCALNAME with branch
appended by default.

Finally, I don't think eclasses are really forced to use src_fetch()
from day one. src_unpack() will still work for them, and we can adjust
them gradually.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-dev] [Future EAPI] src_fetch() phase function to support VCS fetching

[Ian Stakenvicius]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 05/09/12 07:25 AM, Michał Górny wrote:
> Finally, I don't think eclasses are really forced to use
> src_fetch() from day one. src_unpack() will still work for them,
> and we can adjust them gradually.
> 

...except for the fact that the whole point of this is so that live
ebuilds will download their sources via src_fetch() when emerge -f is
called...  :)

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)

iF4EAREIAAYFAlBHUtcACgkQ2ugaI38ACPA7YAEAmuGb+BY67admkDhAAO5sLDvI
iChvTWdTFgRclPxeylEA/iUWFv3eCTzVBBhbvpGc44F2P8IO010OhEu1PrFj2mfC
=fO5h
-----END PGP SIGNATURE-----

Re: [gentoo-dev] [Future EAPI] src_fetch() phase function to support VCS fetching

[James Cloos]

>>>>> "CM" == Ciaran McCreesh <ciaran.mccreesh@googlemail.com> writes:

CM> This doesn't work if we have, for example, foo:1 and foo:2 both using
CM> the same SCM repository, but different branches.

The subversion eclass already handles that; it stores in $distfiles/$P/$branch.

The cvs eclass also could do so.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

Re: [gentoo-dev] [Future EAPI] src_fetch() phase function to support VCS fetching

[Brian Harring]

On Wed, Sep 05, 2012 at 12:07:22PM +0100, Ciaran McCreesh wrote:
> On Wed, 5 Sep 2012 13:00:05 +0200
> Micha?? G??rny <mgorny@gentoo.org> wrote:
> > > I guess that's a pretty comprehensive "we need to do this properly"
> > > then.
> > 
> > Did I say we don't need to? We have the two eclasses which need to do
> > this properly, right? So what's your problem?
> 
> The problem is that we need to work out how to deal with this at least
> for Subversion, and probably for CVS too... As much as we'd like to, we
> can't just roll out a src_fetch without eclass changes. This doesn't
> appear to be a trivial feature to provide.

You're conflating the required phase/hook w/ existing bad 
implementations; they're separate.

The bad implementations can't use the hook till they sort out their 
mess; straight forward enough.

Worth noting, last I looked, git eclass actually didn't do this right 
either; basically merges the namespace of each remote/fetch source 
into the local namespace, no prefixing.  Fixable, but the issue mostly 
comes about because of the fact we do *not* have a src_fetch in the 
first place.

Either way, if the hook was in place, I expect the eclass level issues 
to be sorted shortly after; right now there isn't a reason to do that 
work (chicken/egg).

Consider that my +1 for src_fetch also, although FDEPEND is needed or 
dependencies; I don't care which, getting a src_fetch has been on my 
todo for chrome-os for a while now.

One additional thought- re: the scenarios where we don't fetch to an 
intermediate location, then transfer that contents into ${WORKDIR}, 
while a better name is needed, something along the lines of 
RESTRICT=fetches-into-workdir comes to mind.

Basically that restriction would be interpretted as "$WORKDIR must be 
setup/preserved from invocation of src_fetch to actual building".  

Via that restrict, both scenarios should be addressed in full.
~harring

Re: [gentoo-dev] [Future EAPI] src_fetch() phase function to support VCS fetching

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 661 bytes --]

On Thu, 6 Sep 2012 09:49:13 -0700
Brian Harring <ferringb@gmail.com> wrote:

> One additional thought- re: the scenarios where we don't fetch to an 
> intermediate location, then transfer that contents into ${WORKDIR}, 
> while a better name is needed, something along the lines of 
> RESTRICT=fetches-into-workdir comes to mind.
> 
> Basically that restriction would be interpretted as "$WORKDIR must be 
> setup/preserved from invocation of src_fetch to actual building".  
> 
> Via that restrict, both scenarios should be addressed in full.

Does separate src_fetch() provide any benefit in that scenario?

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-dev] [Future EAPI] src_fetch() phase function to support VCS fetching

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 859 bytes --]

Yes.  The manager can still parallelize prefetching, only consuming a build
job slot post fetch.
On Sep 6, 2012 11:49 AM, "Michał Górny" <mgorny@gentoo.org> wrote:

> On Thu, 6 Sep 2012 09:49:13 -0700
> Brian Harring <ferringb@gmail.com> wrote:
>
> > One additional thought- re: the scenarios where we don't fetch to an
> > intermediate location, then transfer that contents into ${WORKDIR},
> > while a better name is needed, something along the lines of
> > RESTRICT=fetches-into-workdir comes to mind.
> >
> > Basically that restriction would be interpretted as "$WORKDIR must be
> > setup/preserved from invocation of src_fetch to actual building".
> >
> > Via that restrict, both scenarios should be addressed in full.
>
> Does separate src_fetch() provide any benefit in that scenario?
>
> --
> Best regards,
> Michał Górny
>

[-- Attachment #2: Type: text/html, Size: 1223 bytes --]

Re: [gentoo-dev] [Future EAPI] src_fetch() phase function to support VCS fetching

[Rick "Zero_Chaos" Farina]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 09/06/2012 02:50 PM, Michał Górny wrote:
> On Thu, 6 Sep 2012 09:49:13 -0700
> Brian Harring <ferringb@gmail.com> wrote:
> 
>> One additional thought- re: the scenarios where we don't fetch to an 
>> intermediate location, then transfer that contents into ${WORKDIR}, 
>> while a better name is needed, something along the lines of 
>> RESTRICT=fetches-into-workdir comes to mind.

Realizing this is a late response I would like to add....  Um, what?
src_fetch should only be putting things into /usr/portage/distfiles,
never into ${WORKDIR}, that's for src_unpack to handle.

Am I missing something here?

- -Zero
>>
>> Basically that restriction would be interpretted as "$WORKDIR must be 
>> setup/preserved from invocation of src_fetch to actual building".  
>>
>> Via that restrict, both scenarios should be addressed in full.
> 
> Does separate src_fetch() provide any benefit in that scenario?
> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBAgAGBQJQT7z+AAoJEKXdFCfdEflKVPQQALGcLGAgfo8U+M6TdW5Edksf
oqaXE+NSTeFe2DE0G2mgKYSdSIZgMiFp5mLFwpdfAT1gjzFAc+34+5SY8X/0uaaG
OU7fafdUmOlqgD7rCvX56kSWZVPmTV3oZDghmwB1SUIQpL9PNSZoz5uKoatt4UL8
mBMiTmnsYou8f+wDCJoN8eLVoQb/Hm2inobGUCozCsqU6ASgk1eVePpAmJNNVKp6
wrKuVbj4FeDS17Q6xc2g8exXlkxhGmdS1MmugKBR9csYC9P82fh4bXqVzxG15h9O
YZDU5nagJQ9fY6M7oeKg6etVe6PvwOd/FH0Z4wtQJ2NicOsu6DiBf7J+yLvadJdF
M5fE1kxjtR+rm+6bfNgBl2hP5DPdUYxPnZChftPNRpiVe0P0YZGuRgy+GqfXSmMh
8Zf37hJylauBDy397yGapIJ4ergpYVb3Z1ZfU6uW7n8k0apqjqk+TYLEv2tajtTc
WBdRoBAT1LxvjFIHj2Bf5uzNtqev4l19vJv1AnALgs1v1Z8/TiSPB/B/2DvvIawR
Ys+mAEKzQOCezaCjVOpjq7pvi80/8PMcw6Txk9WpsAezrxdAj2X24kwsptVugCAO
lG3agqmOQgH+vCf0PkplxMSGBlofh4hZ3mNbuDaFlaRfiSlzFX//rkQTtOjK4YQu
6SiRnKxiDzJEn+Q1SNUr
=xgcT
-----END PGP SIGNATURE-----

Re: [gentoo-dev] [Future EAPI] src_fetch() phase function to support VCS fetching

[Brian Harring]

On Tue, Sep 11, 2012 at 06:36:46PM -0400, Rick "Zero_Chaos" Farina wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On 09/06/2012 02:50 PM, Micha?? G??rny wrote:
> > On Thu, 6 Sep 2012 09:49:13 -0700
> > Brian Harring <ferringb@gmail.com> wrote:
> > 
> >> One additional thought- re: the scenarios where we don't fetch to an 
> >> intermediate location, then transfer that contents into ${WORKDIR}, 
> >> while a better name is needed, something along the lines of 
> >> RESTRICT=fetches-into-workdir comes to mind.
> 
> Realizing this is a late response I would like to add....  Um, what?
> src_fetch should only be putting things into /usr/portage/distfiles,
> never into ${WORKDIR}, that's for src_unpack to handle.
> 
> Am I missing something here?

WORKDIR was an example; punting it directly into the pkgs distfiles is 
also fine.

Basically, you're assuming that the content *is* cachable- cause if it 
isn't, then dumping in ${DISTDIR} is wasteful (both since it'll 
require a copy into the ebuild's workdir, and since it means 
increasing crap accumulating in the workdir).

Further, there are cases where content *is* available, but is 
fundamentally outside ${DISTDIR}- in cros, they have the 
chrome/chromium source available in certain cases.

Now, either we can store 13GB int ${DISTDIR} that's a copy of that 
external developer checkout, then copy that into ${WORKDIR}, or w/ a 
restrict marker, copy that sucker into ${WORKDIR} directly.

Is it a corner case?  Yep, but it's easy enough to deal w/- the only 
constraint there is that the src_fetch's target dir isn't ${DISTDIR}, 
it's ${WORKDIR}, and the PM is required to preserve ${WORKDIR} from 
src_fetch to the time of that pkg actually running.

~harring