* [gentoo-dev] [patch] golang-vcs-snapshot.eclass: add vendoring of external dependencies
@ 2017-03-09 17:58 William Hubbs
2017-03-09 18:21 ` William L. Thomson Jr.
2017-03-13 14:06 ` William Hubbs
0 siblings, 2 replies; 12+ messages in thread
From: William Hubbs @ 2017-03-09 17:58 UTC (permalink / raw
To: gentoo development
[-- Attachment #1.1: Type: text/plain, Size: 390 bytes --]
All,
things like this are already being done in the tree (by app-admin/consul
for example), and I'n sure by other go ebuilds as well.
I would like to add this functionality to golang-vcs-snapshot.eclass as
a way to remove duplicate code from ebuilds and make this available to
other ebuilds as well.
If there are no objections, I'll commit on Sunday after the council
meeting.
William
[-- Attachment #1.2: golang-vcs-snapshot.patch --]
[-- Type: text/x-diff, Size: 3800 bytes --]
diff --git a/eclass/golang-vcs-snapshot.eclass b/eclass/golang-vcs-snapshot.eclass
index 3ec1d36..f384c38 100644
--- a/eclass/golang-vcs-snapshot.eclass
+++ b/eclass/golang-vcs-snapshot.eclass
@@ -10,9 +10,12 @@
# This eclass provides a convenience src_unpack() which unpacks the
# first tarball mentioned in SRC_URI to its appropriate location in
# ${WORKDIR}/${P}, treating ${WORKDIR}/${P} as a go workspace.
+# Also, it provides a downstream method of vendoring packages.
#
# The location where the tarball is extracted is defined as
-# ${WORKDIR}/${P}/src/${EGO_PN}.
+# ${WORKDIR}/${P}/src/${EGO_PN}. The location of vendored packages is
+# defined as ${WORKDIR}/${P}/src/${EGO_PN%/*}/vendor to match Go's
+# vendoring setup.
#
# The typical use case is VCS snapshots coming from github, bitbucket
# and similar services.
@@ -24,13 +27,20 @@
#
# @CODE
# EGO_PN=github.com/user/package
+# EGO_VENDOR=(
+# "github.com/xenolf/lego 6cac0ea7d8b28c889f709ec7fa92e92b82f490dd"
+# "golang.org/x/crypto 453249f01cfeb54c3d549ddb75ff152ca243f9d8 github.com/golang/crypto"
+# )
+#
# inherit golang-vcs-snapshot
#
-# SRC_URI="http://github.com/example/${PN}/tarball/v${PV} -> ${P}.tar.gz"
+# SRC_URI="http://github.com/example/${PN}/tarball/v${PV} -> ${P}.tar.gz
+# ${EGO_VENDOR_URI}"
# @CODE
#
# The above example will extract the tarball to
# ${WORKDIR}/${P}/src/github.com/user/package
+# and add the vendored tarballs to ${WORKDIR}/src/${EGO_PN}/vendor
inherit golang-base
@@ -41,15 +51,67 @@ esac
EXPORT_FUNCTIONS src_unpack
+# @ECLASS-VARIABLE: EGO_VENDOR
+# @DESCRIPTION:
+# This variable contains a list of vendored packages.
+# The items of this array are strings that contain the
+# import path and the git commit hash for a vendored package.
+# If the import path does not start with github.com, the third argument
+# can be used to point to a github repository.
+
+declare -arg EGO_VENDOR
+
+_golang-vcs-snapshot_set_vendor_uri() {
+ EGO_VENDOR_URI=
+ local lib
+ for lib in "${EGO_VENDOR[@]}"; do
+ lib=(${lib})
+ if [[ -n ${lib[2]} ]]; then
+ EGO_VENDOR_URI+=" https://${lib[2]}/archive/${lib[1]}.tar.gz -> ${lib[2]//\//-}-${lib[1]}.tar.gz"
+ else
+ EGO_VENDOR_URI+=" https://${lib[0]}/archive/${lib[1]}.tar.gz -> ${lib[0]//\//-}-${lib[1]}.tar.gz"
+ fi
+ done
+ readonly EGO_VENDOR_URI
+}
+
+_golang-vcs-snapshot_set_vendor_uri
+unset -f _golang-vcs-snapshot_set_vendor_uri
+
+_golang-vcs-snapshot_dovendor() {
+ local VENDOR_PATH=$1 VENDORPN=$2 TARBALL=$3
+ rm -fr "${VENDOR_PATH}/${VENDORPN}" || die
+ mkdir -p "${VENDOR_PATH}/${VENDORPN}" || die
+ tar -C "${VENDOR_PATH}/${VENDORPN}" -x --strip-components 1\
+ -f "${DISTDIR}"/${TARBALL} || die
+}
+
# @FUNCTION: golang-vcs-snapshot_src_unpack
# @DESCRIPTION:
# Extract the first archive from ${A} to the appropriate location for GOPATH.
golang-vcs-snapshot_src_unpack() {
- local x
+ local lib vendor_path x
ego_pn_check
set -- ${A}
x="$1"
mkdir -p "${WORKDIR}/${P}/src/${EGO_PN%/...}" || die
tar -C "${WORKDIR}/${P}/src/${EGO_PN%/...}" -x --strip-components 1 \
-f "${DISTDIR}/${x}" || die
+
+ if [[ -n "${EGO_VENDOR}" ]]; then
+ vendor_path="${WORKDIR}/${P}/src/${EGO_PN%/...}/vendor"
+ mkdir -p "${vendor_path}" || die
+ for lib in "${EGO_VENDOR[@]}"; do
+ lib=(${lib})
+ if [[ -n ${lib[2]} ]]; then
+ einfo "Vendoring ${lib[0]} ${lib[2]//\//-}-${lib[1]}.tar.gz"
+ _golang-vcs-snapshot_dovendor "${VENDOR_PATH}" ${lib[0]} \
+ ${lib[2]//\//-}-${lib[1]}.tar.gz
+ else
+ einfo "Vendoring ${lib[0]} ${lib[0]//\//-}-${lib[1]}.tar.gz"
+ _golang-vcs-snapshot_dovendor "${VENDOR_PATH}" ${lib[0]} \
+ ${lib[0]//\//-}-${lib[1]}.tar.gz
+ fi
+ done
+ fi
}
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]
^ permalink raw reply related [flat|nested] 12+ messages in thread
* Re: [gentoo-dev] [patch] golang-vcs-snapshot.eclass: add vendoring of external dependencies
2017-03-09 17:58 [gentoo-dev] [patch] golang-vcs-snapshot.eclass: add vendoring of external dependencies William Hubbs
@ 2017-03-09 18:21 ` William L. Thomson Jr.
2017-03-09 18:29 ` Michael Orlitzky
2017-03-13 14:06 ` William Hubbs
1 sibling, 1 reply; 12+ messages in thread
From: William L. Thomson Jr. @ 2017-03-09 18:21 UTC (permalink / raw
To: gentoo-dev
[-- Attachment #1: Type: text/plain, Size: 133 bytes --]
Pretty sure no need for the ||die, rm -fr should never fail.
rm -fr "${VENDOR_PATH}/${VENDORPN}" || die
--
William L. Thomson Jr.
[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 195 bytes --]
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [gentoo-dev] [patch] golang-vcs-snapshot.eclass: add vendoring of external dependencies
2017-03-09 18:21 ` William L. Thomson Jr.
@ 2017-03-09 18:29 ` Michael Orlitzky
2017-03-09 19:00 ` William L. Thomson Jr.
0 siblings, 1 reply; 12+ messages in thread
From: Michael Orlitzky @ 2017-03-09 18:29 UTC (permalink / raw
To: gentoo-dev
On 03/09/2017 01:21 PM, William L. Thomson Jr. wrote:
> Pretty sure no need for the ||die, rm -fr should never fail.
>
> rm -fr "${VENDOR_PATH}/${VENDORPN}" || die
>
$ rm -rf /bin/cp || echo "it failed"
rm: cannot remove '/bin/cp': Permission denied
it failed
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [gentoo-dev] [patch] golang-vcs-snapshot.eclass: add vendoring of external dependencies
2017-03-09 18:29 ` Michael Orlitzky
@ 2017-03-09 19:00 ` William L. Thomson Jr.
2017-03-09 19:28 ` Michael Orlitzky
0 siblings, 1 reply; 12+ messages in thread
From: William L. Thomson Jr. @ 2017-03-09 19:00 UTC (permalink / raw
To: gentoo-dev
[-- Attachment #1: Type: text/plain, Size: 949 bytes --]
On Thursday, March 9, 2017 1:29:55 PM EST Michael Orlitzky wrote:
> On 03/09/2017 01:21 PM, William L. Thomson Jr. wrote:
> > Pretty sure no need for the ||die, rm -fr should never fail.
> >
> > rm -fr "${VENDOR_PATH}/${VENDORPN}" || die
>
> $ rm -rf /bin/cp || echo "it failed"
> rm: cannot remove '/bin/cp': Permission denied
> it failed
Under what circumstances?
It is possible, but not to likely, as the dirs are part of a package unpacked.
User would have permissions. If portage is not being run as root. If run as
root, only extended file attributions ( immutable ) can prevent root from
removing. Not likely set in a packaged unpacked.
This part of the eclass is handling sources from VC. I am not sure how they
can be pulled in with permissions that would prevent what ever user portage is
running under from removing a directory.
Seems like it is not possible to generate the above permission issue.
--
William L. Thomson Jr.
[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 195 bytes --]
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [gentoo-dev] [patch] golang-vcs-snapshot.eclass: add vendoring of external dependencies
2017-03-09 19:00 ` William L. Thomson Jr.
@ 2017-03-09 19:28 ` Michael Orlitzky
2017-03-09 23:16 ` William L. Thomson Jr.
0 siblings, 1 reply; 12+ messages in thread
From: Michael Orlitzky @ 2017-03-09 19:28 UTC (permalink / raw
To: gentoo-dev
On 03/09/2017 02:00 PM, William L. Thomson Jr. wrote:
>
> Under what circumstances?
>
> ...
>
> Seems like it is not possible to generate the above permission issue.
>
I can make them up all day...
* VENDOR_PATH=VENDORPN="" and we try to "rm -rf /"
* A hard drive error occurs.
* Bad memory crashes "rm".
* Somebody is running a recursive chmod on /var/tmp during emerge.
* The tarball contains something you don't expect and can't delete.
...
Ultimately, the die() is there to catch precisely all the things that
should never ("will never") happen.
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [gentoo-dev] [patch] golang-vcs-snapshot.eclass: add vendoring of external dependencies
2017-03-09 19:28 ` Michael Orlitzky
@ 2017-03-09 23:16 ` William L. Thomson Jr.
2017-03-10 0:08 ` Michael Orlitzky
2017-03-12 2:00 ` Kent Fredric
0 siblings, 2 replies; 12+ messages in thread
From: William L. Thomson Jr. @ 2017-03-09 23:16 UTC (permalink / raw
To: gentoo-dev
[-- Attachment #1: Type: text/plain, Size: 2007 bytes --]
On Thursday, March 9, 2017 2:28:47 PM EST Michael Orlitzky wrote:
> On 03/09/2017 02:00 PM, William L. Thomson Jr. wrote:
> > Under what circumstances?
> >
> > ...
> >
> > Seems like it is not possible to generate the above permission issue.
>
> I can make them up all day...
I cannot find the exact comment, but I recall being told before || die used
along with rm -f was incorrect or something along those lines.
Case in point dev-db/firebird use to have a line like
rm -rf "${S}"/extern/{btyacc,editline,icu} || die
But if you look at current ebuild it is now
rm -r "${S}"/extern/{btyacc,editline,icu} || die
The force option/argument was dropped. Why? Seems it could have remained.
Which is why I commented, as I am pretty sure that has been said to me before.
Essentially do not use -f with || die or something to that effect. Maybe for
different reasoning. Still one of those things I never liked. Do it this way,
for this reason, that is not documented. Others may digress and leads to
confusion.
> * VENDOR_PATH=VENDORPN="" and we try to "rm -rf /"
That would likely be an incorrect ebuild and should not have been committed to
tree.
> * A hard drive error occurs.
Likely have much greater issues than rm failing, not sure || die or the rest
would work in that case.
> * Bad memory crashes "rm".
Same as above, if rm is crashing due to memory issues, you likely have serious
issues. I doubt portage would continue on, or the system, etc.
> * Somebody is running a recursive chmod on /var/tmp during emerge.
That sounds like user error if doing updates etc on a system others are
administrating at the same time. I would put that in the stupid category.
Plus doing such would likely cause problems for other things that use that
directory if not portage entirely.
> * The tarball contains something you don't expect and can't delete.
Why would that package be added to tree, or a developer be working with
something blindly?
--
William L. Thomson Jr.
[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 195 bytes --]
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [gentoo-dev] [patch] golang-vcs-snapshot.eclass: add vendoring of external dependencies
2017-03-09 23:16 ` William L. Thomson Jr.
@ 2017-03-10 0:08 ` Michael Orlitzky
2017-03-10 0:43 ` William L. Thomson Jr.
2017-03-12 2:00 ` Kent Fredric
1 sibling, 1 reply; 12+ messages in thread
From: Michael Orlitzky @ 2017-03-10 0:08 UTC (permalink / raw
To: gentoo-dev
On 03/09/2017 06:16 PM, William L. Thomson Jr. wrote:
>
> Case in point dev-db/firebird use to have a line like
>
> rm -rf "${S}"/extern/{btyacc,editline,icu} || die
>
> But if you look at current ebuild it is now
>
> rm -r "${S}"/extern/{btyacc,editline,icu} || die
>
> The force option/argument was dropped. Why? Seems it could have remained.
> Which is why I commented, as I am pretty sure that has been said to me before.
Whether "-f" is appropriate or not depends on the context.
With firebird, you expect all of those directories to exist, and you
want to be notified (so that you can update the ebuild) if one of them
isn't there. If upstream unbundles those three libraries and you're
doing "rm -rf", then you'll never notice, and that (now pointless) line
of code will stay there forever.
In the golang-vcs eclass, you truly don't know that the directory will
exist, so "-f" is needed to avoid a "no such directory" error.
> Essentially do not use -f with || die or something to that effect. Maybe for
> different reasoning. Still one of those things I never liked. Do it this way,
> for this reason, that is not documented. Others may digress and leads to
> confusion.
There's an easy rule: always use "|| die" on commands that don't die
themselves. It's always better to fail as soon as something unexpected
happens. Otherwise it's probably going to fail later anyway, and you'll
waste a whole lot of time trying to figure out why it *really* failed.
There are exceptions to the rule, but it should be really obvious when
you hit one, like if you have a test suite that always returns 1.
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [gentoo-dev] [patch] golang-vcs-snapshot.eclass: add vendoring of external dependencies
2017-03-10 0:08 ` Michael Orlitzky
@ 2017-03-10 0:43 ` William L. Thomson Jr.
2017-03-10 1:03 ` William L. Thomson Jr.
0 siblings, 1 reply; 12+ messages in thread
From: William L. Thomson Jr. @ 2017-03-10 0:43 UTC (permalink / raw
To: gentoo-dev
[-- Attachment #1: Type: text/plain, Size: 2418 bytes --]
On Thursday, March 9, 2017 7:08:12 PM EST Michael Orlitzky wrote:
>
> Whether "-f" is appropriate or not depends on the context.
>
> With firebird, you expect all of those directories to exist, and you
> want to be notified (so that you can update the ebuild) if one of them
> isn't there. If upstream unbundles those three libraries and you're
> doing "rm -rf", then you'll never notice, and that (now pointless) line
> of code will stay there forever.
I was assuming such. However that also means who ever is maintaining said
package is not paying attention. Relying on || die to fail when/if one of
those changes. Which I would think a safer way is to use -rf and not care if
one is removed.
In this specific package is also related to patches. Not just that bit in
ebuild. Thus maintainer like would have noticed when patches failed. Seemed
like needless nitpicking that just leads to confusion.
> In the golang-vcs eclass, you truly don't know that the directory will
> exist, so "-f" is needed to avoid a "no such directory" error.
In that case would you not want to test for existence before removal. Rather
than just forced removal if its not there.
In the case of an ebuild, calling || die just to remove a no longer needed
segment. Seems like a lazy fall back/requirement. I have seen lots of other
cruft remain, so not sure that is a good solution overall for keeping ebuilds
current.
> > Essentially do not use -f with || die or something to that effect. Maybe
> > for different reasoning. Still one of those things I never liked. Do it
> > this way, for this reason, that is not documented. Others may digress and
> > leads to confusion.
>
> There's an easy rule: always use "|| die" on commands that don't die
> themselves. It's always better to fail as soon as something unexpected
> happens. Otherwise it's probably going to fail later anyway, and you'll
> waste a whole lot of time trying to figure out why it *really* failed.
I get the usage of || die , it was just the force option of rm. Since I
recalled being told the -f would negate the || die. Short of reasons stated.
> There are exceptions to the rule, but it should be really obvious when
> you hit one, like if you have a test suite that always returns 1.
Sure, and along those lines I do not believe the rm -f || die stuff is
documented. That may help and clarify usage based on context.
--
William L. Thomson Jr.
[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 195 bytes --]
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [gentoo-dev] [patch] golang-vcs-snapshot.eclass: add vendoring of external dependencies
2017-03-10 0:43 ` William L. Thomson Jr.
@ 2017-03-10 1:03 ` William L. Thomson Jr.
2017-03-10 7:38 ` Ulrich Mueller
0 siblings, 1 reply; 12+ messages in thread
From: William L. Thomson Jr. @ 2017-03-10 1:03 UTC (permalink / raw
To: gentoo-dev
[-- Attachment #1: Type: text/plain, Size: 1307 bytes --]
On Thursday, March 9, 2017 7:43:50 PM EST William L. Thomson Jr. wrote:
> On Thursday, March 9, 2017 7:08:12 PM EST Michael Orlitzky wrote:
> > Whether "-f" is appropriate or not depends on the context.
> >
> > With firebird, you expect all of those directories to exist, and you
> > want to be notified (so that you can update the ebuild) if one of them
> > isn't there. If upstream unbundles those three libraries and you're
> > doing "rm -rf", then you'll never notice, and that (now pointless) line
> > of code will stay there forever.
>
> I was assuming such. However that also means who ever is maintaining said
> package is not paying attention. Relying on || die to fail when/if one of
> those changes. Which I would think a safer way is to use -rf and not care if
> one is removed.
>
> In this specific package is also related to patches. Not just that bit in
> ebuild. Thus maintainer like would have noticed when patches failed. Seemed
> like needless nitpicking that just leads to confusion.
Along the lines of failures. What if a system has rm aliased to prompt before
removal? In that case rm -r would fail, but rm -fr would not. That would cause
failures for the user and not the developer. Assuming rm does not disable
prompt for a non-interactive shell.
--
William L. Thomson Jr.
[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 195 bytes --]
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [gentoo-dev] [patch] golang-vcs-snapshot.eclass: add vendoring of external dependencies
2017-03-09 23:16 ` William L. Thomson Jr.
2017-03-10 0:08 ` Michael Orlitzky
@ 2017-03-12 2:00 ` Kent Fredric
1 sibling, 0 replies; 12+ messages in thread
From: Kent Fredric @ 2017-03-12 2:00 UTC (permalink / raw
To: gentoo-dev
[-- Attachment #1: Type: text/plain, Size: 1294 bytes --]
On Thu, 09 Mar 2017 18:16:51 -0500
"William L. Thomson Jr." <wlt-ml@o-sinc.com> wrote:
> That would likely be an incorrect ebuild and should not have been committed to
> tree.
If people didn't accidentally overlook problems where variables contained the wrong
contents and tried to access files they shouldn't, we wouldn't have a need for Gentoo Maintainers.
More over, we wouldn't need a lot of security we have these days, and PHP features like register_globals
would have never been considered a bad thing.
But the reality is, sometimes you write bugs in your code where you're not looking at it carefully enough.
Sometimes said bugs sneak through code review, tests, and about 100 people installing it without issue.
Hence a defensive approach of "Look, I don't think I've done anything really bad here, but just in case
reality starts bending in on itself, lets put in some sensible BANG points to catch that".
You're not going to remember every time you aught to put such BANG points in place, but the hope is
that having *2* mental toolkits: "Don't make mistakes" and "Put things in place to go bang when I make mistakes",
With a bit of luck, both won't fail a the same time.
But that's just me I guess. || die "Error sending clear message by email"
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [gentoo-dev] [patch] golang-vcs-snapshot.eclass: add vendoring of external dependencies
2017-03-09 17:58 [gentoo-dev] [patch] golang-vcs-snapshot.eclass: add vendoring of external dependencies William Hubbs
2017-03-09 18:21 ` William L. Thomson Jr.
@ 2017-03-13 14:06 ` William Hubbs
1 sibling, 0 replies; 12+ messages in thread
From: William Hubbs @ 2017-03-13 14:06 UTC (permalink / raw
To: gentoo development
[-- Attachment #1: Type: text/plain, Size: 46 bytes --]
This was committed last night.
William
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]
^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2017-03-13 14:06 UTC | newest]
Thread overview: 12+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2017-03-09 17:58 [gentoo-dev] [patch] golang-vcs-snapshot.eclass: add vendoring of external dependencies William Hubbs
2017-03-09 18:21 ` William L. Thomson Jr.
2017-03-09 18:29 ` Michael Orlitzky
2017-03-09 19:00 ` William L. Thomson Jr.
2017-03-09 19:28 ` Michael Orlitzky
2017-03-09 23:16 ` William L. Thomson Jr.
2017-03-10 0:08 ` Michael Orlitzky
2017-03-10 0:43 ` William L. Thomson Jr.
2017-03-10 1:03 ` William L. Thomson Jr.
2017-03-10 7:38 ` Ulrich Mueller
2017-03-12 2:00 ` Kent Fredric
2017-03-13 14:06 ` William Hubbs
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox