[gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources

[gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 298 bytes --]

All,

we have several Go ebuilds in the tree that bundle multiple separate
upstream sources. One example is app-admin/consul-0.5.2.

My thought is that we shouldn't bundle like this, but we should figure
out how to write ebuilds for the dependent packages as well.

What do others think?

William


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources

[Zac Medico]

On 06/29/2015 02:27 PM, William Hubbs wrote:
> All,
> 
> we have several Go ebuilds in the tree that bundle multiple separate
> upstream sources. One example is app-admin/consul-0.5.2.
> 
> My thought is that we shouldn't bundle like this, but we should figure
> out how to write ebuilds for the dependent packages as well.
> 
> What do others think?

Maybe we should take into account the number of consumers of said
libraries? If there's only one consumer of a given library, then what's
the advantage of splitting out a separate ebuild? Also, in our
discussion, it may be useful to distinguish between bundling via "one
big tarball" versus bundling via multiple tarballs in SRC_URI.
-- 
Thanks,
Zac

Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources

[Zac Medico]

On 06/29/2015 05:27 PM, wireless@tampabay.rr.com wrote:
> On 06/29/2015 05:50 PM, Zac Medico wrote:
>> On 06/29/2015 02:27 PM, William Hubbs wrote:
>>> All,
>>>
>>> we have several Go ebuilds in the tree that bundle multiple separate
>>> upstream sources. One example is app-admin/consul-0.5.2.
>>>
>>> My thought is that we shouldn't bundle like this, but we should figure
>>> out how to write ebuilds for the dependent packages as well.
>>>
>>> What do others think?
>>
>> Maybe we should take into account the number of consumers of said
>> libraries? If there's only one consumer of a given library, then what's
>> the advantage of splitting out a separate ebuild? Also, in our
>> discussion, it may be useful to distinguish between bundling via "one
>> big tarball" versus bundling via multiple tarballs in SRC_URI.
> 
> You have much to consider. Consul, like zookeeper (ultrabug overlay) is
> very useful for building clusters on (gentoo) linux. It would be very
> cool to split consul into a separate build. That way one can experiment
> with combining  a wide variety of sys-cluster builds with other packages.

While it would certainly be possible to split out a number of separate
ebuilds for Go libraries that are used *exclusively* by consul, what
advantages would it have? You mention "a wide variety of sys-cluster
builds," but I'm not sure what packages you're talking about. For
example, are you aware of any other packages that use hashicorp's raft
library [1]?

> Regardless of which way you go, it would be great to have some detail
> documents about the various (software) components if you stay with one
> large build.

You can see all of the components (including github.com/hashicorp/raft)
in the SRC_URI variable of the ebuild [2].

[1] https://github.com/hashicorp/raft
[2]
https://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/app-admin/consul/consul-0.5.2.ebuild?view=markup
-- 
Thanks,
Zac

Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources

[wireless]

On 06/29/2015 05:50 PM, Zac Medico wrote:
> On 06/29/2015 02:27 PM, William Hubbs wrote:
>> All,
>>
>> we have several Go ebuilds in the tree that bundle multiple separate
>> upstream sources. One example is app-admin/consul-0.5.2.
>>
>> My thought is that we shouldn't bundle like this, but we should figure
>> out how to write ebuilds for the dependent packages as well.
>>
>> What do others think?
>
> Maybe we should take into account the number of consumers of said
> libraries? If there's only one consumer of a given library, then what's
> the advantage of splitting out a separate ebuild? Also, in our
> discussion, it may be useful to distinguish between bundling via "one
> big tarball" versus bundling via multiple tarballs in SRC_URI.

You have much to consider. Consul, like zookeeper (ultrabug overlay) is 
very useful for building clusters on (gentoo) linux. It would be very 
cool to split consul into a separate build. That way one can experiment 
with combining  a wide variety of sys-cluster builds with other packages.


Regardless of which way you go, it would be great to have some detail 
documents about the various (software) components if you stay with one
large build.


hth,
James

Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources

[wireless]

On 06/29/2015 06:50 PM, Zac Medico wrote:
> On 06/29/2015 05:27 PM, wireless@tampabay.rr.com wrote:
>> On 06/29/2015 05:50 PM, Zac Medico wrote:
>>> On 06/29/2015 02:27 PM, William Hubbs wrote:
>>>> All,
>>>>
>>>> we have several Go ebuilds in the tree that bundle multiple separate
>>>> upstream sources. One example is app-admin/consul-0.5.2.
>>>>
>>>> My thought is that we shouldn't bundle like this, but we should figure
>>>> out how to write ebuilds for the dependent packages as well.
>>>>
>>>> What do others think?
>>>
>>> Maybe we should take into account the number of consumers of said
>>> libraries? If there's only one consumer of a given library, then what's
>>> the advantage of splitting out a separate ebuild? Also, in our
>>> discussion, it may be useful to distinguish between bundling via "one
>>> big tarball" versus bundling via multiple tarballs in SRC_URI.
>>
>> You have much to consider. Consul, like zookeeper (ultrabug overlay) is
>> very useful for building clusters on (gentoo) linux. It would be very
>> cool to split consul into a separate build. That way one can experiment
>> with combining  a wide variety of sys-cluster builds with other packages.
>
> While it would certainly be possible to split out a number of separate
> ebuilds for Go libraries that are used *exclusively* by consul, what
> advantages would it have? You mention "a wide variety of sys-cluster
> builds," but I'm not sure what packages you're talking about. For
> example, are you aware of any other packages that use hashicorp's raft
> library [1]?

First of all, I'm not sure  why my  nntp interface to gentoo-dev is not 
following the thread (sorry, I'm still working out how to use nntp to 
gentoo-dev).

I'm not up on raft, although it looks very interesting [FSM] and all.
I've been working on apache-mesos a bit. Consul is used frequently
with mesos; here is one example [A]. My experience is that current 
clusters/clouds are mostly a unique mix of different software, consul 
being but one of many common components. Perhaps I did not have a 
sufficiently deep understanding of raft, but my comment was meant to 
encourage a consul package for gentoo, I guess dependant on a raft 
package too.

>> Regardless of which way you go, it would be great to have some detail
>> documents about the various (software) components if you stay with one
>> large build.
>
> You can see all of the components (including github.com/hashicorp/raft)
> in the SRC_URI variable of the ebuild [2].

Yea, I need to read up on raft; it does look promising as it took mesos 
a while to become popular.  Is raft as a separate ebuild useful; I'm not 
sure, but it does look interesting from what I've seen. Many projects 
within the cluster/cloud space have morphed, so raft has just as good a 
chance to diversify it's appeal and usefulness. Surely the convenience 
of the dev that maintains the package(s) is also keenly important.

> [1] https://github.com/hashicorp/raft
> [2]
> https://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/app-admin/consul/consul-0.5.2.ebuild?view=markup


[A] https://github.com/CiscoCloud/mesos-consul

James

Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources

[Michael Orlitzky]

On 06/29/2015 07:44 PM, Zac Medico wrote:
> 
> While it would certainly be possible to split out a number of separate
> ebuilds for Go libraries that are used *exclusively* by consul, what
> advantages would it have?

Even in this limiting case,

  1. You avoid pointless rebuilds. You rebuild the library (and
     probably the binary, for Go packages) when the library is upgraded
     rather than rebuilding everything whenever anything is updated.

  2. Security. If upstream treats the packages as separate, a user
     might hear that there's a security issue in libfoo but then run
     eix and see that he doesn't have libfoo installed (because it's
     bundled).

  3. Chicken and egg problem. If the library only has one consumer and
     you keep it bundled with that consumer forever, then it will
     probably only ever have one consumer. If somebody wants to use
     it in an overlay or something he'd have to pull in the whole
     program.

  4. Ebuild complexity. Now you have to compile e.g. three packages in
     src_compile, install three packages in src_install, etc. The result
     is more complicated than building once, three times.

  5. One maintainer has to commit to maintaining all of the dependencies
     in addition to the program that he cares about.

Someone actually has to do the work to split out the libraries, so it
may not be a clear-cut win in some cases. But it's nicer to have them
split out should that happen by magic.

Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources

[Zac Medico]

On 06/29/2015 07:08 PM, wireless@tampabay.rr.com wrote:
> On 06/29/2015 06:50 PM, Zac Medico wrote:
>> On 06/29/2015 05:27 PM, wireless@tampabay.rr.com wrote:
>>> On 06/29/2015 05:50 PM, Zac Medico wrote:
>>>> On 06/29/2015 02:27 PM, William Hubbs wrote:
>>>>> All,
>>>>>
>>>>> we have several Go ebuilds in the tree that bundle multiple separate
>>>>> upstream sources. One example is app-admin/consul-0.5.2.
>>>>>
>>>>> My thought is that we shouldn't bundle like this, but we should figure
>>>>> out how to write ebuilds for the dependent packages as well.
>>>>>
>>>>> What do others think?
>>>>
>>>> Maybe we should take into account the number of consumers of said
>>>> libraries? If there's only one consumer of a given library, then what's
>>>> the advantage of splitting out a separate ebuild? Also, in our
>>>> discussion, it may be useful to distinguish between bundling via "one
>>>> big tarball" versus bundling via multiple tarballs in SRC_URI.
>>>
>>> You have much to consider. Consul, like zookeeper (ultrabug overlay) is
>>> very useful for building clusters on (gentoo) linux. It would be very
>>> cool to split consul into a separate build. That way one can experiment
>>> with combining  a wide variety of sys-cluster builds with other
>>> packages.
>>
>> While it would certainly be possible to split out a number of separate
>> ebuilds for Go libraries that are used *exclusively* by consul, what
>> advantages would it have? You mention "a wide variety of sys-cluster
>> builds," but I'm not sure what packages you're talking about. For
>> example, are you aware of any other packages that use hashicorp's raft
>> library [1]?
> 
> First of all, I'm not sure  why my  nntp interface to gentoo-dev is not
> following the thread (sorry, I'm still working out how to use nntp to
> gentoo-dev).
> 
> I'm not up on raft, although it looks very interesting [FSM] and all.
> I've been working on apache-mesos a bit. Consul is used frequently
> with mesos; here is one example [A]. My experience is that current
> clusters/clouds are mostly a unique mix of different software, consul
> being but one of many common components. Perhaps I did not have a
> sufficiently deep understanding of raft,

Understanding raft is beyond the scope of this discussion. The question
is, "Do we know of any packages other than consul that consume the
hashicorp/raft library?"

> but my comment was meant to
> encourage a consul package for gentoo,

We already have a consul package for gentoo, so there's no encouragement
needed there. ;)

>  I guess dependant on a raft package too.

Are you sure about that, given that consul would be the only consumer of
the hashicorp/raft library?

> 
>>> Regardless of which way you go, it would be great to have some detail
>>> documents about the various (software) components if you stay with one
>>> large build.
>>
>> You can see all of the components (including github.com/hashicorp/raft)
>> in the SRC_URI variable of the ebuild [2].
> 
> Yea, I need to read up on raft; it does look promising as it took mesos
> a while to become popular.  Is raft as a separate ebuild useful; I'm not
> sure, but it does look interesting from what I've seen.

It's not useful unless there are at least 2 ebuilds that can use it.

> Many projects
> within the cluster/cloud space have morphed, so raft has just as good a
> chance to diversify it's appeal and usefulness. Surely the convenience
> of the dev that maintains the package(s) is also keenly important.

Unless you are writing an ebuild which uses the hashicorp/raft library,
you really don't need an ebuild for it. So, maybe we should wait and see
if the need arises.
-- 
Thanks,
Zac

Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources

[Zac Medico]

On 06/29/2015 07:24 PM, Michael Orlitzky wrote:
> On 06/29/2015 07:44 PM, Zac Medico wrote:
>>
>> While it would certainly be possible to split out a number of separate
>> ebuilds for Go libraries that are used *exclusively* by consul, what
>> advantages would it have?
> 
> Even in this limiting case,
> 
>   1. You avoid pointless rebuilds. You rebuild the library (and
>      probably the binary, for Go packages) when the library is upgraded
>      rather than rebuilding everything whenever anything is updated.

From my experience, Go packages don't take very long to build.

>   2. Security. If upstream treats the packages as separate, a user
>      might hear that there's a security issue in libfoo but then run
>      eix and see that he doesn't have libfoo installed (because it's
>      bundled).

That's a reasonable motivation. However, many of these libraries don't
have any tags. So, you'll have to use the commit hashes if you want to
test for vulnerabilities. In the case of the consul ebuild, the commit
hashes of the libraries are available in the SRC_URI. I suppose that we
could standardize a way to expose these.

>   3. Chicken and egg problem. If the library only has one consumer and
>      you keep it bundled with that consumer forever, then it will
>      probably only ever have one consumer. If somebody wants to use
>      it in an overlay or something he'd have to pull in the whole
>      program.

If a Go developer wants to use the libraries in question, then he'll
probably use 'go get' to install them. I doubt the existence of an
ebuild will have much relevance in people's decision to adopt a given Go
library.

>   4. Ebuild complexity. Now you have to compile e.g. three packages in
>      src_compile, install three packages in src_install, etc. The result
>      is more complicated than building once, three times.

In the case of the consul ebuild, all of the libraries are automatically
built when the ebuild calls the emake. Even without a Makefile, Go makes
it trivial to build the dependencies.

>   5. One maintainer has to commit to maintaining all of the dependencies
>      in addition to the program that he cares about.

I guess that's a reasonable argument, depending on how much maintenance
the dependencies require.

> Someone actually has to do the work to split out the libraries, so it
> may not be a clear-cut win in some cases. But it's nicer to have them
> split out should that happen by magic.

Considering that Go binaries are statically linked, you'll end up with a
bunch of Go libraries installed that you don't need during run-time.
-- 
Thanks,
Zac

Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources

[Ultrabug]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 30/06/2015 05:25, Zac Medico wrote:
> On 06/29/2015 07:24 PM, Michael Orlitzky wrote:
>> On 06/29/2015 07:44 PM, Zac Medico wrote:
>>> 

Having faced the exact same problem I have to say I agree 100% with
Zac. I'd like to say that Gentoo needs this kind of packages to stay
actual and that our NOGO (yes that's an actual joke) on Go packages is
not good for us nowdays.

>>> While it would certainly be possible to split out a number of
>>> separate ebuilds for Go libraries that are used *exclusively*
>>> by consul, what advantages would it have?
>> 
>> Even in this limiting case,
>> 
>> 1. You avoid pointless rebuilds. You rebuild the library (and 
>> probably the binary, for Go packages) when the library is
>> upgraded rather than rebuilding everything whenever anything is
>> updated.
> 
> From my experience, Go packages don't take very long to build.
> 

+1, Go is not C, I have the same feeling

>> 2. Security. If upstream treats the packages as separate, a user 
>> might hear that there's a security issue in libfoo but then run 
>> eix and see that he doesn't have libfoo installed (because it's 
>> bundled).
> 
> That's a reasonable motivation. However, many of these libraries
> don't have any tags. So, you'll have to use the commit hashes if
> you want to test for vulnerabilities. In the case of the consul
> ebuild, the commit hashes of the libraries are available in the
> SRC_URI. I suppose that we could standardize a way to expose
> these.
> 

+1, there is no strong tagging on every upstream. Maybe that's another
topic but handling git sub modules et al could be made easier while
satisfying our QA (or maybe make some exceptions)

>> 3. Chicken and egg problem. If the library only has one consumer
>> and you keep it bundled with that consumer forever, then it will 
>> probably only ever have one consumer. If somebody wants to use it
>> in an overlay or something he'd have to pull in the whole 
>> program.
> 
> If a Go developer wants to use the libraries in question, then
> he'll probably use 'go get' to install them. I doubt the existence
> of an ebuild will have much relevance in people's decision to adopt
> a given Go library.
> 
>> 4. Ebuild complexity. Now you have to compile e.g. three packages
>> in src_compile, install three packages in src_install, etc. The
>> result is more complicated than building once, three times.
> 
> In the case of the consul ebuild, all of the libraries are
> automatically built when the ebuild calls the emake. Even without a
> Makefile, Go makes it trivial to build the dependencies.
> 

Non live GIT ebuilds already make ebuilds more complex, this should
indeed be enough.

>> 5. One maintainer has to commit to maintaining all of the
>> dependencies in addition to the program that he cares about.
> 
> I guess that's a reasonable argument, depending on how much
> maintenance the dependencies require.
> 

Since there is no real Go support as such, this would be a pain ...

>> Someone actually has to do the work to split out the libraries,
>> so it may not be a clear-cut win in some cases. But it's nicer to
>> have them split out should that happen by magic.
> 
> Considering that Go binaries are statically linked, you'll end up
> with a bunch of Go libraries installed that you don't need during
> run-time.
> 

+1, this defeats Go's main advantage imho (not that I think it's
smart, but it's the actual fact)

Cheers

Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 1265 bytes --]

All,

we have digressed a bit, so I want to bring the discussion back to what
my main concerns are about this issue.

1. Should we bundle Go packages with Go software?

If we do, except for the Go standard library which is part of
dev-lang/go, do we need to bother with installing Go sources and
packages at all?

The down side of the whole bundling idea is that every consumer
on someone's system could potentially have a different version of the
Go package, which doesn't lend itself well to security concerns.

This is why bundling is generally discouraged in Gentoo.

Also, if we bundle, most of dev-go/* doesn't need to exist because these
libraries would be bundled into and statically linked into the software
that needs them.

2. How should we bundle?

This is where my concern about consul and some other ebuilds comes in.

The way the consul ebuild is written (putting the commit hashes of
dependencies in SRC_URI) assumes that all of the dependencies will stay
on github. This makes the ebuild far less flexable than go itself is.

If we are going to bundle, I would rather have one tarball that includes
all of the sources for consul and the dependent libraries dropped on the
Gentoo mirrors. Such a tarball is very easy to create.

Thoughts?

William


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources

[Michael Orlitzky]

On 06/29/2015 11:25 PM, Zac Medico wrote:
> 
> Considering that Go binaries are statically linked, you'll end up with a
> bunch of Go libraries installed that you don't need during run-time.
> 

They'll eventually give this up, because everyone does when their
language starts seeing serious use. I won't pretend that's a real
argument though.

Suppose ten years from now everything is written in Go. I have 500
statically linked Go packages on my system, all of whose dependencies
were built and compiled-in at install time. Now someone finds a remote
root vulnerability in the go-openssl library. I know some of the
packages I have installed were built against it. What do I do?

At least with the useless dev-go/go-openssl installed, I can use
subslots to rebuild everything after an upgrade to the fixed version.

Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources

[Zac Medico]

On 06/30/2015 08:35 AM, William Hubbs wrote:
> All,
> 
> we have digressed a bit, so I want to bring the discussion back to what
> my main concerns are about this issue.
> 
> 1. Should we bundle Go packages with Go software?
> 
> If we do, except for the Go standard library which is part of
> dev-lang/go, do we need to bother with installing Go sources and
> packages at all?
> 
> The down side of the whole bundling idea is that every consumer
> on someone's system could potentially have a different version of the
> Go package, which doesn't lend itself well to security concerns.
> 
> This is why bundling is generally discouraged in Gentoo.

Yes, as a general rule, bundling is sub-optimal. However, there are
often exceptions to general rules like these, especially when there are
competing concerns to contend with.

> Also, if we bundle, most of dev-go/* doesn't need to exist because these
> libraries would be bundled into and statically linked into the software
> that needs them.

Some static libraries are commonly used enough that it might be
reasonable to install them. Alternatively, we might invent a notion of
having one ebuild execute another ebuild in order to install static
dependencies into a temporary build directory.

> 2. How should we bundle?
> 
> This is where my concern about consul and some other ebuilds comes in.
> 
> The way the consul ebuild is written (putting the commit hashes of
> dependencies in SRC_URI) assumes that all of the dependencies will stay
> on github. This makes the ebuild far less flexable than go itself is.

Agreed. However, there's no rule which says that we have to force all
ebuilds to fit into common templates.

> If we are going to bundle, I would rather have one tarball that includes
> all of the sources for consul and the dependent libraries dropped on the
> Gentoo mirrors. Such a tarball is very easy to create.

I would prefer to use separate tarballs for each dependency, preferably
with the commit hash encoded in the tarball name. This makes the ebuild
dependencies transparent in the sense that the commit hashes of the
dependencies are readily available. The "one big tarball" is opaque
rather than transparent, and it will have a tendency bloat the mirrors.
By keeping the dependencies in separate tarballs, we can easily do a
revbump that updates a subset of the dependencies, without having to
re-pack everything into a big bloated tarball.
-- 
Thanks,
Zac

Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources

[Zac Medico]

On 06/30/2015 08:49 AM, Michael Orlitzky wrote:
> On 06/29/2015 11:25 PM, Zac Medico wrote:
>>
>> Considering that Go binaries are statically linked, you'll end up with a
>> bunch of Go libraries installed that you don't need during run-time.
>>
> 
> They'll eventually give this up, because everyone does when their
> language starts seeing serious use. I won't pretend that's a real
> argument though.

Yeah, we'll see. We need to deal with the current version of reality
though...

> Suppose ten years from now everything is written in Go. I have 500
> statically linked Go packages on my system, all of whose dependencies
> were built and compiled-in at install time. Now someone finds a remote
> root vulnerability in the go-openssl library. I know some of the
> packages I have installed were built against it. What do I do?

Use slot-operator := deps, together with the emerge --with-bdeps=y
option. Then, if you bump the sub-slot of the go-openssl library, all of
your go packages that have it in DEPEND with a slot-operator :=
dependency will be rebuilt automatically.

> At least with the useless dev-go/go-openssl installed, I can use
> subslots to rebuild everything after an upgrade to the fixed version.

As I mentioned in my reply to William [1],  we might invent a notion of
having one ebuild execute another ebuild in order to install static
dependencies into a temporary build directory. That way, static
libraries would be built on-demand, and discarded as soon as possible.

[1]
https://archives.gentoo.org/gentoo-dev/message/4b150fe36bf9e0ba1eb29b1d695a3193
-- 
Thanks,
Zac

Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources

[Zac Medico]

On 06/30/2015 11:12 AM, Zac Medico wrote:
> As I mentioned in my reply to William [1],  we might invent a notion of
> having one ebuild execute another ebuild in order to install static
> dependencies into a temporary build directory. That way, static
> libraries would be built on-demand, and discarded as soon as possible.
> 
> [1]
> https://archives.gentoo.org/gentoo-dev/message/4b150fe36bf9e0ba1eb29b1d695a3193
> 

I should note that I'm not very fond of this idea. If the dependencies
have separate ebuilds (like dev-go/*), then you can already use
something like 'emerge --depclean --with-bdeps=n' to remove the static
libraries that aren't needed at run-time.
-- 
Thanks,
Zac

Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources

[Michael Orlitzky]

On 06/30/2015 02:12 PM, Zac Medico wrote:
> 
>> Suppose ten years from now everything is written in Go. I have 500
>> statically linked Go packages on my system, all of whose dependencies
>> were built and compiled-in at install time. Now someone finds a remote
>> root vulnerability in the go-openssl library. I know some of the
>> packages I have installed were built against it. What do I do?
> 
> Use slot-operator := deps, together with the emerge --with-bdeps=y
> option. Then, if you bump the sub-slot of the go-openssl library, all of
> your go packages that have it in DEPEND with a slot-operator :=
> dependency will be rebuilt automatically.
> 

Right, and now what if go-openssl was built on-the-fly 500 times and
there's no package for it?

Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources

[Zac Medico]

On 06/30/2015 11:25 AM, Michael Orlitzky wrote:
> On 06/30/2015 02:12 PM, Zac Medico wrote:
>>
>>> Suppose ten years from now everything is written in Go. I have 500
>>> statically linked Go packages on my system, all of whose dependencies
>>> were built and compiled-in at install time. Now someone finds a remote
>>> root vulnerability in the go-openssl library. I know some of the
>>> packages I have installed were built against it. What do I do?
>>
>> Use slot-operator := deps, together with the emerge --with-bdeps=y
>> option. Then, if you bump the sub-slot of the go-openssl library, all of
>> your go packages that have it in DEPEND with a slot-operator :=
>> dependency will be rebuilt automatically.
>>
> 
> Right, and now what if go-openssl was built on-the-fly 500 times and
> there's no package for it?

Yeah that's obviously sub-optimal, and it's the reason why I created the
dev-go/* ebuilds. However, we may want to distinguish between libraries
that would only have a single consumer and libraries that would have
multiple consumers. Using the same rules regardless of the number of
consumers is not necessarily optimal.
-- 
Thanks,
Zac

Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 4233 bytes --]

On Tue, Jun 30, 2015 at 10:53:58AM -0700, Zac Medico wrote:
> On 06/30/2015 08:35 AM, William Hubbs wrote:
> > All,
> > 
> > we have digressed a bit, so I want to bring the discussion back to what
> > my main concerns are about this issue.
> > 
> > 1. Should we bundle Go packages with Go software?
> > 
> > If we do, except for the Go standard library which is part of
> > dev-lang/go, do we need to bother with installing Go sources and
> > packages at all?
> > 
> > The down side of the whole bundling idea is that every consumer
> > on someone's system could potentially have a different version of the
> > Go package, which doesn't lend itself well to security concerns.
> > 
> > This is why bundling is generally discouraged in Gentoo.
> 
> Yes, as a general rule, bundling is sub-optimal. However, there are
> often exceptions to general rules like these, especially when there are
> competing concerns to contend with.
 
 I don't really see what the competing concerns are in this case.

> > Also, if we bundle, most of dev-go/* doesn't need to exist because these
> > libraries would be bundled into and statically linked into the software
> > that needs them.
> 
> Some static libraries are commonly used enough that it might be
> reasonable to install them. Alternatively, we might invent a notion of
> having one ebuild execute another ebuild in order to install static
> dependencies into a temporary build directory.

Why do we need to worry about how many projects use a library? Upstream
has it as a library for good reason, so that multiple projects can use
 it. If upstream installs it as a library, that's how we should install
 it if we install it.

The problem I see with the argument about "commonly used enough" is the
vagueness of it. If we have two packages that use a library, it is
commonly used enough that it should be installed separately.

If we start out bundling libraries, especially libraries from different
upstreams than the package we are working on, that forces all go
maintainers to check all go ebuilds in the tree to see if multiple
bundling is going on and open bugs to create separate ebuilds for
libraries that were only used before by one package but now are used by
more than one.

> > 2. How should we bundle?
> > 
> > This is where my concern about consul and some other ebuilds comes in.
> > 
> > The way the consul ebuild is written (putting the commit hashes of
> > dependencies in SRC_URI) assumes that all of the dependencies will stay
> > on github. This makes the ebuild far less flexable than go itself is.
> 
> Agreed. However, there's no rule which says that we have to force all
> ebuilds to fit into common templates.
 
We do when they deal with common issues; that's the whole point of
language-based eclasses, e.g. ruby* perl* and python*.

> > If we are going to bundle, I would rather have one tarball that includes
> > all of the sources for consul and the dependent libraries dropped on the
> > Gentoo mirrors. Such a tarball is very easy to create.
> 
> I would prefer to use separate tarballs for each dependency, preferably
> with the commit hash encoded in the tarball name. This makes the ebuild
> dependencies transparent in the sense that the commit hashes of the
> dependencies are readily available. The "one big tarball" is opaque
> rather than transparent, and it will have a tendency bloat the mirrors.
> By keeping the dependencies in separate tarballs, we can easily do a
> revbump that updates a subset of the dependencies, without having to
> re-pack everything into a big bloated tarball.

I can agree with part of this; one big tarball is less transparent than
multiple tarballs.

Another thing to consider is,  with one big tarball, you can name the
extraction directory to match ${S}, which means there would be no need
for magic in the ebuilds to deal with putting extracted directories in
the right place.

All I'm saying is, if we are going to bundle, lets go all in and not
download multiple upstream packages in src_uri but put them in big
tarballs.

If we are not going to bundle, the best way to handle it is to not
bundle at all imo.

William


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources

[Zac Medico]

On 06/30/2015 01:30 PM, William Hubbs wrote:
> On Tue, Jun 30, 2015 at 10:53:58AM -0700, Zac Medico wrote:
>> On 06/30/2015 08:35 AM, William Hubbs wrote:
>>> All,
>>>
>>> we have digressed a bit, so I want to bring the discussion back to what
>>> my main concerns are about this issue.
>>>
>>> 1. Should we bundle Go packages with Go software?
>>>
>>> If we do, except for the Go standard library which is part of
>>> dev-lang/go, do we need to bother with installing Go sources and
>>> packages at all?
>>>
>>> The down side of the whole bundling idea is that every consumer
>>> on someone's system could potentially have a different version of the
>>> Go package, which doesn't lend itself well to security concerns.
>>>
>>> This is why bundling is generally discouraged in Gentoo.
>>
>> Yes, as a general rule, bundling is sub-optimal. However, there are
>> often exceptions to general rules like these, especially when there are
>> competing concerns to contend with.
>  
>  I don't really see what the competing concerns are in this case.

The competing concern is that un-bundling has some possibly undesirable
consequences, mainly that it means we'll be installing static libraries
that were only intended to be temporary build artifacts. It makes sense
to install them if there are multiple consumers, otherwise it doesn't
make much sense.

>>> Also, if we bundle, most of dev-go/* doesn't need to exist because these
>>> libraries would be bundled into and statically linked into the software
>>> that needs them.
>>
>> Some static libraries are commonly used enough that it might be
>> reasonable to install them. Alternatively, we might invent a notion of
>> having one ebuild execute another ebuild in order to install static
>> dependencies into a temporary build directory.
> 
> Why do we need to worry about how many projects use a library?

If you want to clutter the tree with trivial ebuilds that only have a
single consumer, then I guess that's fine. It's not clear to me that
this is the best course of action, but I'm not going to try to stop you
if that's what you want to do.

>  Upstream
> has it as a library for good reason, so that multiple projects can use
>  it. If upstream installs it as a library, that's how we should install
>  it if we install it.

I don't think that consul upstream "installs it as a library." The last
time that I checked upstream's "build from source" instructions, the
consul binary was the only result of interest, so all of the temporary
build artifacts could simply be discarded after the consul binary had
been built.

> The problem I see with the argument about "commonly used enough" is the
> vagueness of it. If we have two packages that use a library, it is
> commonly used enough that it should be installed separately.

As soon as you have at least two consumers, then you have a good reason
to un-bundle a library. If there's only one consumer, then un-bundling
becomes questionable.

> If we start out bundling libraries, especially libraries from different
> upstreams than the package we are working on, that forces all go
> maintainers to check all go ebuilds in the tree to see if multiple
> bundling is going on and open bugs to create separate ebuilds for
> libraries that were only used before by one package but now are used by
> more than one.

Nobody is being forced to do anything. If a maintainer of a Go package
than bundles libraries is doing a version bump, then it would be a good
time for him to check if ebuilds have been created for any of those
bundled dependencies, and un-bundle them at that point.

>>> 2. How should we bundle?
>>>
>>> This is where my concern about consul and some other ebuilds comes in.
>>>
>>> The way the consul ebuild is written (putting the commit hashes of
>>> dependencies in SRC_URI) assumes that all of the dependencies will stay
>>> on github. This makes the ebuild far less flexable than go itself is.
>>
>> Agreed. However, there's no rule which says that we have to force all
>> ebuilds to fit into common templates.
>  
> We do when they deal with common issues; that's the whole point of
> language-based eclasses, e.g. ruby* perl* and python*.

There can always be outliers that don't fit your existing templates.

>>> If we are going to bundle, I would rather have one tarball that includes
>>> all of the sources for consul and the dependent libraries dropped on the
>>> Gentoo mirrors. Such a tarball is very easy to create.
>>
>> I would prefer to use separate tarballs for each dependency, preferably
>> with the commit hash encoded in the tarball name. This makes the ebuild
>> dependencies transparent in the sense that the commit hashes of the
>> dependencies are readily available. The "one big tarball" is opaque
>> rather than transparent, and it will have a tendency bloat the mirrors.
>> By keeping the dependencies in separate tarballs, we can easily do a
>> revbump that updates a subset of the dependencies, without having to
>> re-pack everything into a big bloated tarball.
> 
> I can agree with part of this; one big tarball is less transparent than
> multiple tarballs.
> 
> Another thing to consider is,  with one big tarball, you can name the
> extraction directory to match ${S}, which means there would be no need
> for magic in the ebuilds to deal with putting extracted directories in
> the right place.

I think this so-called "magic" is a small price to pay for transparency,
and the ease with which one can do a revision bump while updating a
subset of the bundled dependencies (all the tarballs are automatically
generated by github, so I don't have to pack any of them myself).

> All I'm saying is, if we are going to bundle, lets go all in and not
> download multiple upstream packages in src_uri but put them in big
> tarballs.

I would prefer to create separate ebuilds for all of the dependencies,
than be forced to do this.

> If we are not going to bundle, the best way to handle it is to not
> bundle at all imo.

Even if there is only a single consumer of said libraries?
-- 
Thanks,
Zac

Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 2086 bytes --]

On Tue, Jun 30, 2015 at 02:34:52PM -0700, Zac Medico wrote:
> On 06/30/2015 01:30 PM, William Hubbs wrote:
> > On Tue, Jun 30, 2015 at 10:53:58AM -0700, Zac Medico wrote:
> >> On 06/30/2015 08:35 AM, William Hubbs wrote:
> >>> All,
> >>>
> >>> we have digressed a bit, so I want to bring the discussion back to what
> >>> my main concerns are about this issue.
> >>>
> >>> 1. Should we bundle Go packages with Go software?
> >>>
> >>> If we do, except for the Go standard library which is part of
> >>> dev-lang/go, do we need to bother with installing Go sources and
> >>> packages at all?
> >>>
> >>> The down side of the whole bundling idea is that every consumer
> >>> on someone's system could potentially have a different version of the
> >>> Go package, which doesn't lend itself well to security concerns.
> >>>
> >>> This is why bundling is generally discouraged in Gentoo.
> >>
> >> Yes, as a general rule, bundling is sub-optimal. However, there are
> >> often exceptions to general rules like these, especially when there are
> >> competing concerns to contend with.
> >  
> >  I don't really see what the competing concerns are in this case.
> 
> The competing concern is that un-bundling has some possibly undesirable
> consequences, mainly that it means we'll be installing static libraries
> that were only intended to be temporary build artifacts. It makes sense
> to install them if there are multiple consumers, otherwise it doesn't
> make much sense.
 
 Thinking about this, there may be a third option. This would take a
 slight reworking of the golang-build.eclass, but that is easy to do,
 and it would possibly remove the subslot from the dependencies.

 The source code is where the compatibility between versions of Go is,
 not the static objects, so what if, for third-party go packages, we
skip installing the static objects?

The only down side of this would be that there might be longer rebuilds
if the packages have multiple consumers, but it gets rid of the static
objects.

What do you think?

William


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources

[Zac Medico]

On 06/30/2015 03:08 PM, William Hubbs wrote:
> On Tue, Jun 30, 2015 at 02:34:52PM -0700, Zac Medico wrote:
>> On 06/30/2015 01:30 PM, William Hubbs wrote:
>>>  
>>>  I don't really see what the competing concerns are in this case.
>>
>> The competing concern is that un-bundling has some possibly undesirable
>> consequences, mainly that it means we'll be installing static libraries
>> that were only intended to be temporary build artifacts. It makes sense
>> to install them if there are multiple consumers, otherwise it doesn't
>> make much sense.
>  
>  Thinking about this, there may be a third option. This would take a
>  slight reworking of the golang-build.eclass, but that is easy to do,
>  and it would possibly remove the subslot from the dependencies.
> 
>  The source code is where the compatibility between versions of Go is,
>  not the static objects, so what if, for third-party go packages, we
> skip installing the static objects?

If we did this with consul, for example, then the source code for all
those libraries (that have no other consumers) would have to be
installed in order to build consul-template against the consul's api
library. It would be similar to a header dependency. This would
necessitate the introduction of "build-against" dependencies [1], or
equivalent virtuals (like virtual/podofo-build).

> The only down side of this would be that there might be longer rebuilds
> if the packages have multiple consumers, but it gets rid of the static
> objects.
> 
> What do you think?

Considering the similarity to header dependencies, I don't know. The
subslot thing seems slightly more appealing to me.

[1] https://bugs.gentoo.org/show_bug.cgi?id=392239
-- 
Thanks,
Zac

Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 1612 bytes --]

On Tue, Jun 30, 2015 at 04:48:29PM -0700, Zac Medico wrote:
> On 06/30/2015 03:08 PM, William Hubbs wrote:
> >  Thinking about this, there may be a third option. This would take a
> >  slight reworking of the golang-build.eclass, but that is easy to do,
> >  and it would possibly remove the subslot from the dependencies.
> > 
> >  The source code is where the compatibility between versions of Go is,
> >  not the static objects, so what if, for third-party go packages, we
> > skip installing the static objects?
> 
> If we did this with consul, for example, then the source code for all
> those libraries (that have no other consumers) would have to be
> installed in order to build consul-template against the consul's api
> library. It would be similar to a header dependency. This would
> necessitate the introduction of "build-against" dependencies [1], or
> equivalent virtuals (like virtual/podofo-build).

How is this different from DEPEND="dev-go/podofo" for example or
DEPEND=">=dev-go/fodofo-0_prexxxxxxxx"?

> > The only down side of this would be that there might be longer rebuilds
> > if the packages have multiple consumers, but it gets rid of the static
> > objects.
> > 
> > What do you think?
> 
> Considering the similarity to header dependencies, I don't know. The
> subslot thing seems slightly more appealing to me.

I got the idea of not installing the objects from Debian's description
of how they do this [1]; they do not mention installing the objects.

Let me know what you think.

William

[1] http://pkg-go.alioth.debian.org/packaging.html


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources

[Zac Medico]

On 06/30/2015 07:01 PM, William Hubbs wrote:
> On Tue, Jun 30, 2015 at 04:48:29PM -0700, Zac Medico wrote:
>> On 06/30/2015 03:08 PM, William Hubbs wrote:
>>>  Thinking about this, there may be a third option. This would take a
>>>  slight reworking of the golang-build.eclass, but that is easy to do,
>>>  and it would possibly remove the subslot from the dependencies.
>>>
>>>  The source code is where the compatibility between versions of Go is,
>>>  not the static objects, so what if, for third-party go packages, we
>>> skip installing the static objects?
>>
>> If we did this with consul, for example, then the source code for all
>> those libraries (that have no other consumers) would have to be
>> installed in order to build consul-template against the consul's api
>> library. It would be similar to a header dependency. This would
>> necessitate the introduction of "build-against" dependencies [1], or
>> equivalent virtuals (like virtual/podofo-build).
> 
> How is this different from DEPEND="dev-go/podofo" for example or
> DEPEND=">=dev-go/fodofo-0_prexxxxxxxx"?

The virtual/podofo-build package pulls in dev-libs/boost, since packages
which build against podofo will fail to build unless the boost headers
are installed. Since dev-libs/boost is not a run-time dependency of
podofo, and it's not a direct build-time dependency of packages that
build against podofo, we pull it in via virtual/podofo-build.

If we install Go source files without the corresponding static
libraries, they we create a similar situation to the above. For example,
if consul doesn't install its static api library, then anything that
wants to build against that library is going to need indirect
dependencies installed in order to build that library. The indirect
dependencies are not needed if there is an installed instance of
consul's static api library.

>>> The only down side of this would be that there might be longer rebuilds
>>> if the packages have multiple consumers, but it gets rid of the static
>>> objects.
>>>
>>> What do you think?
>>
>> Considering the similarity to header dependencies, I don't know. The
>> subslot thing seems slightly more appealing to me.
> 
> I got the idea of not installing the objects from Debian's description
> of how they do this [1]; they do not mention installing the objects.
> 
> Let me know what you think.
> 
> William
> 
> [1] http://pkg-go.alioth.debian.org/packaging.html
> 

As I understand it, debian does the equivalent of putting the Go
dependencies in both DEPEND and RDEPEND. This means that users are
forced to keep build-time dependencies around after they are no longer
needed.
-- 
Thanks,
Zac

Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources

[Zac Medico]

On 06/30/2015 03:08 PM, William Hubbs wrote:
>  The source code is where the compatibility between versions of Go is,
>  not the static objects, so what if, for third-party go packages, we
> skip installing the static objects?
> 
> The only down side of this would be that there might be longer rebuilds
> if the packages have multiple consumers, but it gets rid of the static
> objects.
> 
> What do you think?

I'll give real example involving go-tools. The go-tools build requires
go-net, which in turn requires go-text. If the go-net *.a files are
installed, then it is possible to build go-tools against go-net without
having go-text installed. If the go-net *.a files are not installed,
then you will have to install go-text before you can build go-tools. It
introduces an indirect build-time dependency between go-tools and go-text.
-- 
Thanks,
Zac

Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 1089 bytes --]

On Sat, Jul 04, 2015 at 12:19:28PM -0700, Zac Medico wrote:
> On 06/30/2015 03:08 PM, William Hubbs wrote:
> >  The source code is where the compatibility between versions of Go is,
> >  not the static objects, so what if, for third-party go packages, we
> > skip installing the static objects?
> > 
> > The only down side of this would be that there might be longer rebuilds
> > if the packages have multiple consumers, but it gets rid of the static
> > objects.
> > 
> > What do you think?
> 
> I'll give real example involving go-tools. The go-tools build requires
> go-net, which in turn requires go-text. If the go-net *.a files are
> installed, then it is possible to build go-tools against go-net without
> having go-text installed. If the go-net *.a files are not installed,
> then you will have to install go-text before you can build go-tools. It
> introduces an indirect build-time dependency between go-tools and go-text.

Sure, but what I'm proposing is that we do not install any *.a files
for Go software that is not part of dev-lang/go.

William



[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources

[Zac Medico]

On 07/04/2015 12:32 PM, William Hubbs wrote:
> On Sat, Jul 04, 2015 at 12:19:28PM -0700, Zac Medico wrote:
>> On 06/30/2015 03:08 PM, William Hubbs wrote:
>>>  The source code is where the compatibility between versions of Go is,
>>>  not the static objects, so what if, for third-party go packages, we
>>> skip installing the static objects?
>>>
>>> The only down side of this would be that there might be longer rebuilds
>>> if the packages have multiple consumers, but it gets rid of the static
>>> objects.
>>>
>>> What do you think?
>>
>> I'll give real example involving go-tools. The go-tools build requires
>> go-net, which in turn requires go-text. If the go-net *.a files are
>> installed, then it is possible to build go-tools against go-net without
>> having go-text installed. If the go-net *.a files are not installed,
>> then you will have to install go-text before you can build go-tools. It
>> introduces an indirect build-time dependency between go-tools and go-text.
> 
> Sure, but what I'm proposing is that we do not install any *.a files
> for Go software that is not part of dev-lang/go.

Exactly the same type of situation can arise for packages that are not
part of dev-lang/go. For example, if consul's static api.a library is
not installed, then it will introduce indirect build-time dependencies
for the consul-template package.
-- 
Thanks,
Zac

Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 1765 bytes --]

On Sat, Jul 04, 2015 at 12:43:37PM -0700, Zac Medico wrote:
> On 07/04/2015 12:32 PM, William Hubbs wrote:
> > On Sat, Jul 04, 2015 at 12:19:28PM -0700, Zac Medico wrote:
> >> On 06/30/2015 03:08 PM, William Hubbs wrote:
> >>>  The source code is where the compatibility between versions of Go is,
> >>>  not the static objects, so what if, for third-party go packages, we
> >>> skip installing the static objects?
> >>>
> >>> The only down side of this would be that there might be longer rebuilds
> >>> if the packages have multiple consumers, but it gets rid of the static
> >>> objects.
> >>>
> >>> What do you think?
> >>
> >> I'll give real example involving go-tools. The go-tools build requires
> >> go-net, which in turn requires go-text. If the go-net *.a files are
> >> installed, then it is possible to build go-tools against go-net without
> >> having go-text installed. If the go-net *.a files are not installed,
> >> then you will have to install go-text before you can build go-tools. It
> >> introduces an indirect build-time dependency between go-tools and go-text.
> > 
> > Sure, but what I'm proposing is that we do not install any *.a files
> > for Go software that is not part of dev-lang/go.
> 
> Exactly the same type of situation can arise for packages that are not
> part of dev-lang/go. For example, if consul's static api.a library is
> not installed, then it will introduce indirect build-time dependencies
> for the consul-template package.

Hmm, I haven't looked at either consul or consul-template yet, but I'm
thinking that if you use golang-build.eclass to install everything and
make sure GOPATH is set correctly, consul-template will pick up
everything it needs.

What am I missing?

William

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources

[Zac Medico]

On 07/04/2015 01:24 PM, William Hubbs wrote:
> On Sat, Jul 04, 2015 at 12:43:37PM -0700, Zac Medico wrote:
>> On 07/04/2015 12:32 PM, William Hubbs wrote:
>>> On Sat, Jul 04, 2015 at 12:19:28PM -0700, Zac Medico wrote:
>>>> On 06/30/2015 03:08 PM, William Hubbs wrote:
>>>>>  The source code is where the compatibility between versions of Go is,
>>>>>  not the static objects, so what if, for third-party go packages, we
>>>>> skip installing the static objects?
>>>>>
>>>>> The only down side of this would be that there might be longer rebuilds
>>>>> if the packages have multiple consumers, but it gets rid of the static
>>>>> objects.
>>>>>
>>>>> What do you think?
>>>>
>>>> I'll give real example involving go-tools. The go-tools build requires
>>>> go-net, which in turn requires go-text. If the go-net *.a files are
>>>> installed, then it is possible to build go-tools against go-net without
>>>> having go-text installed. If the go-net *.a files are not installed,
>>>> then you will have to install go-text before you can build go-tools. It
>>>> introduces an indirect build-time dependency between go-tools and go-text.
>>>
>>> Sure, but what I'm proposing is that we do not install any *.a files
>>> for Go software that is not part of dev-lang/go.
>>
>> Exactly the same type of situation can arise for packages that are not
>> part of dev-lang/go. For example, if consul's static api.a library is
>> not installed, then it will introduce indirect build-time dependencies
>> for the consul-template package.
> 
> Hmm, I haven't looked at either consul or consul-template yet, but I'm
> thinking that if you use golang-build.eclass to install everything and
> make sure GOPATH is set correctly, consul-template will pick up
> everything it needs.
> 
> What am I missing?

You need to recognize that "build-time for package A" is not the same as
"build-time for package B". When you build go-tools, you can't rely on
the build-time dependencies of go-net being present. Likewise, you can't
rely on the build-time dependencies of consul being present when
consul-template is built.
-- 
Thanks,
Zac

Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources

[Zac Medico]

On 07/04/2015 01:33 PM, Zac Medico wrote:
> On 07/04/2015 01:24 PM, William Hubbs wrote:
>> What am I missing?
> 
> You need to recognize that "build-time for package A" is not the same as
> "build-time for package B". When you build go-tools, you can't rely on
> the build-time dependencies of go-net being present. Likewise, you can't
> rely on the build-time dependencies of consul being present when
> consul-template is built.

For example, you should be able to build go-net, unistall go-text, and
then build go-tools without go-text installed. Likewise, you should be
able to build consul, uninstall go-crypto, and then build
consul-template without go-crypto installed.
-- 
Thanks,
Zac