[gentoo-dev] Prevent binary/non-compiled packages from binary package creation

[gentoo-dev] Prevent binary/non-compiled packages from binary package creation

[William L. Thomson Jr.]

[-- Attachment #1: Type: text/plain, Size: 1044 bytes --]

I make a lot of binaries for use on other systems, to expedite updates.
It does not make sense for some packages to ever be a binary package.

Packages like -bin packages or gentoo-sources, which are just sources.
Having binary ebuilds of those is of no benefit. I can be the opposite
causing things to be much slower. Like in the case of large things
packages like android-studio.

Just seems odd to make a binary of a binary, or repackage
gentoo-sources into a gentoo ebuild binary/binpkg. There is not really
any benefit either way for such packages.

It would be beneficial if ebuilds could have some internal variable
that prevents it from being a binary package. It should not prevent
merge or fail. Just never create a binary of this package, always use
the ebuild as normal. Even if someone is force installing using binary
flag or otherwise.

As most things I think this would require support in PMS, or next EAPI
at minimum. But I think the EAPI comes from PMS, so they are related.

-- 
William L. Thomson Jr.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] Prevent binary/non-compiled packages from binary package creation

[Rich Freeman]

On Tue, Aug 8, 2017 at 12:37 PM, William L. Thomson Jr.
<wlt-ml@o-sinc.com> wrote:
>
> As most things I think this would require support in PMS, or next EAPI
> at minimum. But I think the EAPI comes from PMS, so they are related.
>

Actually, I'm not sure about this since it doesn't really affect what
is actually built, but I'll defer to others on that.  For example, in
[1] there is a statement "Package managers may recognise other tokens,
but ebuilds may not rely upon them being supported."  This strikes me
as that sort of thing, and indeed this might be the right place to put
it.  There is no real harm if a particular package manager doesn't
support this feature as it has no affect on what is installed.  I
believe we also use non-PMS variables for things like disabling QA
checks in tinderboxes and such, since this has no impact on package
managers.

A new EAPI would be defined in a new version of the PMS.  The PMS is
the specification that includes EAPI among other things.

1 - https://projects.gentoo.org/pms/6/pms.html#x1-920008.2.8

-- 
Rich

Re: [gentoo-dev] Prevent binary/non-compiled packages from binary package creation

[Kristian Fiskerstrand]

[-- Attachment #1.1: Type: text/plain, Size: 525 bytes --]

On 08/08/2017 06:37 PM, William L. Thomson Jr. wrote:
> I make a lot of binaries for use on other systems, to expedite updates.
> It does not make sense for some packages to ever be a binary package.

Any particular reason this decision shouldn't be left to the operator of
the binhost rather than the package maintainer? it can already be
controlled through env files.

-- 
Kristian Fiskerstrand
OpenPGP keyblock reachable at hkp://pool.sks-keyservers.net
fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [gentoo-dev] Prevent binary/non-compiled packages from binary package creation

[Rich Freeman]

On Tue, Aug 8, 2017 at 10:11 AM, Kristian Fiskerstrand <k_f@gentoo.org> wrote:
> On 08/08/2017 06:37 PM, William L. Thomson Jr. wrote:
>> I make a lot of binaries for use on other systems, to expedite updates.
>> It does not make sense for some packages to ever be a binary package.
>
> Any particular reason this decision shouldn't be left to the operator of
> the binhost rather than the package maintainer? it can already be
> controlled through env files.
>

Perhaps, but I could see some value in having some way to mark
packages that don't compile anything.  This could also overlap
somewhat with the desire to track arch-independent packages for
stabilization purposes.  I could see it being useful to be able to
obtain a list of all the binary packages in the Gentoo repo for QA
purposes/etc as well.

Maybe it isn't a flag that outright blocks binary package building,
but a way to mark such packages so that a user can apply a policy on
top of this.

Whether it belongs in the ebuild, or in metadata, is another matter.

-- 
Rich

Re: [gentoo-dev] Prevent binary/non-compiled packages from binary package creation

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1258 bytes --]

On wto, 2017-08-08 at 10:18 -0700, Rich Freeman wrote:
> On Tue, Aug 8, 2017 at 10:11 AM, Kristian Fiskerstrand <k_f@gentoo.org> wrote:
> > On 08/08/2017 06:37 PM, William L. Thomson Jr. wrote:
> > > I make a lot of binaries for use on other systems, to expedite updates.
> > > It does not make sense for some packages to ever be a binary package.
> > 
> > Any particular reason this decision shouldn't be left to the operator of
> > the binhost rather than the package maintainer? it can already be
> > controlled through env files.
> > 
> 
> Perhaps, but I could see some value in having some way to mark
> packages that don't compile anything.  This could also overlap
> somewhat with the desire to track arch-independent packages for
> stabilization purposes.  I could see it being useful to be able to
> obtain a list of all the binary packages in the Gentoo repo for QA
> purposes/etc as well.
> 
> Maybe it isn't a flag that outright blocks binary package building,
> but a way to mark such packages so that a user can apply a policy on
> top of this.
> 
> Whether it belongs in the ebuild, or in metadata, is another matter.

Does a package that builds documentation from sources count?

-- 
Best regards,
Michał Górny

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 988 bytes --]

Re: [gentoo-dev] Prevent binary/non-compiled packages from binary package creation

[William L. Thomson Jr.]

On Tue, 8 Aug 2017 10:18:36 -0700
Rich Freeman <rich0@gentoo.org> wrote:
>
> Whether it belongs in the ebuild, or in metadata, is another matter.

The how, implementation, etc is not as important to me. I just think
there should be some means to prevent such. If there is not currently.

As mentioned there could be other uses/benefits of such depending on
how it is implemented. That is up to others. I just would like to not
have binaries made of packages I feel are a waste to be re-packaged
into a binpkg.

-- 
William L. Thomson Jr.

Re: [gentoo-dev] Prevent binary/non-compiled packages from binary package creation

[William L. Thomson Jr.]

[-- Attachment #1: Type: text/plain, Size: 1184 bytes --]

On Tue, 8 Aug 2017 19:11:18 +0200
Kristian Fiskerstrand <k_f@gentoo.org> wrote:

> On 08/08/2017 06:37 PM, William L. Thomson Jr. wrote:
> > I make a lot of binaries for use on other systems, to expedite
> > updates. It does not make sense for some packages to ever be a
> > binary package.  
> 
> Any particular reason this decision shouldn't be left to the operator
> of the binhost rather than the package maintainer?

Can you think of any? I cannot see any operator wanting a binary of a
binary, or a package of sources. When they already have a sources
tarball. Maybe in the case of shipping binaries without sources. But I
am not sure if an binary ebuild ignores SRC_URI entirely.

I think moving binaries without needing the distfiles would be the
only reason why an operator may prefer binaries of stuff that does not
get compiled, just installed.

> it can already be controlled through env files.

I was thinking it might, but having used them to skip other hooks. I
was thinking they could not be used as such for binary packages. Have
you confirmed such is possible?  Could you provide a link or example?
Thanks!


-- 
William L. Thomson Jr.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] Prevent binary/non-compiled packages from binary package creation

[Kristian Fiskerstrand]

[-- Attachment #1.1: Type: text/plain, Size: 1227 bytes --]

On 08/08/2017 07:23 PM, William L. Thomson Jr. wrote:
> Can you think of any? I cannot see any operator wanting a binary of a
> binary, or a package of sources. When they already have a sources

 - The machine you're installing it on might not have internet access so
you want to have the files stored in a single location for wrapping it up.

 - You might want an audit trail of installed packages, so using the
binary files on specific media ensures same copy is installed everywhere

 - You might be applying local patches through /etc/portage/patches that
are distributed to all clients

On 08/08/2017 07:23 PM, William L. Thomson Jr. wrote:
>> it can already be controlled through env files.
> I was thinking it might, but having used them to skip other hooks. I
> was thinking they could not be used as such for binary packages. Have
> you confirmed such is possible?  Could you provide a link or example?
> Thanks!

try something like:
/etc/portage/env/nobin:
FEATURES="-buildpkg"

/etc/portage/package.env/nobin:
sys-kernel/gentoo-sources nobin

-- 
Kristian Fiskerstrand
OpenPGP keyblock reachable at hkp://pool.sks-keyservers.net
fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [gentoo-dev] Prevent binary/non-compiled packages from binary package creation

[William L. Thomson Jr.]

[-- Attachment #1: Type: text/plain, Size: 1815 bytes --]

On Tue, 8 Aug 2017 19:32:48 +0200
Kristian Fiskerstrand <k_f@gentoo.org> wrote:

> On 08/08/2017 07:23 PM, William L. Thomson Jr. wrote:
> > Can you think of any? I cannot see any operator wanting a binary of
> > a binary, or a package of sources. When they already have a
> > sources  
> 
>  - The machine you're installing it on might not have internet access
> so you want to have the files stored in a single location for
> wrapping it up.

Not sure that would be any different between distfiles and packages.

>  - You might want an audit trail of installed packages, so using the
> binary files on specific media ensures same copy is installed
> everywhere

Doesn't the manifest/hash aspect of ebuilds ensure that for the
tarballs already? If installing same version, same tarball, Its all the
same already.

>  - You might be applying local patches through /etc/portage/patches
> that are distributed to all clients

This might be the strongest reason. Though would only apply to stuff
like say kernel sources. Not sure what patches could be applied to a
binary ebuild, -bin. A patch would not effect src_install per my
understanding.

> On 08/08/2017 07:23 PM, William L. Thomson Jr. wrote:
> >> it can already be controlled through env files.  
> > I was thinking it might, but having used them to skip other hooks. I
> > was thinking they could not be used as such for binary packages.
> > Have you confirmed such is possible?  Could you provide a link or
> > example? Thanks!  
> 
> try something like:
> /etc/portage/env/nobin:
> FEATURES="-buildpkg"
> 
> /etc/portage/package.env/nobin:
> sys-kernel/gentoo-sources nobin

That may work, I was not thinking to negate.  Trying it out now. But
may lead me to another need... :)

-- 
William L. Thomson Jr.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] Prevent binary/non-compiled packages from binary package creation

[William L. Thomson Jr.]

[-- Attachment #1: Type: text/plain, Size: 1334 bytes --]

> > On 08/08/2017 07:23 PM, William L. Thomson Jr. wrote:  
> > >> it can already be controlled through env files.    
> > > I was thinking it might, but having used them to skip other
> > > hooks. I was thinking they could not be used as such for binary
> > > packages. Have you confirmed such is possible?  Could you provide
> > > a link or example? Thanks!    
> > 
> > try something like:
> > /etc/portage/env/nobin:
> > FEATURES="-buildpkg"
> > 
> > /etc/portage/package.env/nobin:
> > sys-kernel/gentoo-sources nobin  
> 
> That may work, I was not thinking to negate.  Trying it out now. But
> may lead me to another need... :)

My other need is that it does not seem you can use env files or patches
in profiles. I think I can do the env stuff via other means. Like
package.use. I do not want it in make.defaults, as this is per package.

The problem with env files is managing them per system. I want to do
things like that in profiles. It is to much work to manage that stuff
on each system. Even using things like ansible. Which is why I have
moved to custom profiles rather than per system stuff.

I will need to experiment a bit more. But IMHO a variable that can be
set in an ebuild, and bypassed if needed by operator at emerge time is
the simplest approach.

-- 
William L. Thomson Jr.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] Prevent binary/non-compiled packages from binary package creation

[Sam Jorna (wraeth)]

[-- Attachment #1.1: Type: text/plain, Size: 835 bytes --]

On 09/08/17 04:20, William L. Thomson Jr. wrote:
> On Tue, 8 Aug 2017 19:32:48 +0200
> Kristian Fiskerstrand <k_f@gentoo.org> wrote:
>>  - You might be applying local patches through /etc/portage/patches
>> that are distributed to all clients
> 
> This might be the strongest reason. Though would only apply to stuff
> like say kernel sources. Not sure what patches could be applied to a
> binary ebuild, -bin. A patch would not effect src_install per my
> understanding.

There's also the fact that binpkgs may be manually installed exactly as
the package manager would have installed it, rather than extracting
whatever upstream supplies verbatim. This includes things like any
wrappers, desktop files or symlinks created by the ebuild, or other such
downstream-isms.

-- 
Sam Jorna (wraeth)
GnuPG ID: D6180C26


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 858 bytes --]

Re: [gentoo-dev] Prevent binary/non-compiled packages from binary package creation

[William L. Thomson Jr.]

[-- Attachment #1: Type: text/plain, Size: 1255 bytes --]

On Wed, 9 Aug 2017 10:29:40 +1000
"Sam Jorna (wraeth)" <wraeth@gentoo.org> wrote:

> On 09/08/17 04:20, William L. Thomson Jr. wrote:
> > On Tue, 8 Aug 2017 19:32:48 +0200
> > Kristian Fiskerstrand <k_f@gentoo.org> wrote:  
> >>  - You might be applying local patches through /etc/portage/patches
> >> that are distributed to all clients  
> > 
> > This might be the strongest reason. Though would only apply to stuff
> > like say kernel sources. Not sure what patches could be applied to a
> > binary ebuild, -bin. A patch would not effect src_install per my
> > understanding.  
> 
> There's also the fact that binpkgs may be manually installed exactly
> as the package manager would have installed it, rather than extracting
> whatever upstream supplies verbatim. 

What then is the benefit? If what is installed is the same from
package manager or binpkg. Also your redistributing another's package
in binary format which may not be legally allowed.

> This includes things like any wrappers, desktop files or symlinks
> created by the ebuild, or other such downstream-isms.

If it was made via package manager. If it was made via quickpkg, it
maybe different than if made by package manager.

-- 
William L. Thomson Jr.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] Prevent binary/non-compiled packages from binary package creation

[Sam Jorna (wraeth)]

[-- Attachment #1.1: Type: text/plain, Size: 1814 bytes --]

On 09/08/17 10:43, William L. Thomson Jr. wrote:
> On Wed, 9 Aug 2017 10:29:40 +1000
> "Sam Jorna (wraeth)" <wraeth@gentoo.org> wrote:
> 
>> On 09/08/17 04:20, William L. Thomson Jr. wrote:
>>> On Tue, 8 Aug 2017 19:32:48 +0200
>>> Kristian Fiskerstrand <k_f@gentoo.org> wrote:  
>>>>  - You might be applying local patches through /etc/portage/patches
>>>> that are distributed to all clients  
>>>
>>> This might be the strongest reason. Though would only apply to stuff
>>> like say kernel sources. Not sure what patches could be applied to a
>>> binary ebuild, -bin. A patch would not effect src_install per my
>>> understanding.  
>>
>> There's also the fact that binpkgs may be manually installed exactly
>> as the package manager would have installed it, rather than extracting
>> whatever upstream supplies verbatim. 
> 
> What then is the benefit? If what is installed is the same from
> package manager or binpkg. Also your redistributing another's package
> in binary format which may not be legally allowed.

The difference is that how the package manager/ebuild installs the
package may be better suited to the environment than what upstream
expects (such as upstreams that install through a .run file)

>> This includes things like any wrappers, desktop files or symlinks
>> created by the ebuild, or other such downstream-isms.
> 
> If it was made via package manager. If it was made via quickpkg, it
> maybe different than if made by package manager.

Using quickpkg can create different binaries depending on your options,
but otherwise it should package any installed files as recorded by the
package manager - provided you use inclusive options, there should be no
appreciable difference as far as I'm aware.

-- 
Sam Jorna (wraeth)
GnuPG ID: D6180C26


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 858 bytes --]

Re: [gentoo-dev] Prevent binary/non-compiled packages from binary package creation

[William L. Thomson Jr.]

[-- Attachment #1: Type: text/plain, Size: 1091 bytes --]

On Wed, 9 Aug 2017 11:07:04 +1000
"Sam Jorna (wraeth)" <wraeth@gentoo.org> wrote:

> > What then is the benefit? If what is installed is the same from
> > package manager or binpkg. Also your redistributing another's
> > package in binary format which may not be legally allowed.  
> 
> The difference is that how the package manager/ebuild installs the
> package may be better suited to the environment than what upstream
> expects (such as upstreams that install through a .run file)

I fail to see how basically skipping src_install and maybe some prepare
stuff that makes it better suited to an environment.
Can you explain that further?

These packages are just exploded tarballs. I fail to see the benefit
to repacking those into another tarball to be exploded. At best
skipping src_install and/or prepare, seems to be the only difference.

I see no difference in installing kernel sources via source ebuild or a
binpkg, pre-built ebuild binary. Other than the time it takes to
re-package the kernel sources into another tarball.


-- 
William L. Thomson Jr.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] Prevent binary/non-compiled packages from binary package creation

[Francesco Riosa]

[-- Attachment #1: Type: text/plain, Size: 1364 bytes --]

2017-08-09 17:33 GMT+02:00 William L. Thomson Jr. <wlt-ml@o-sinc.com>:

> On Wed, 9 Aug 2017 11:07:04 +1000
> "Sam Jorna (wraeth)" <wraeth@gentoo.org> wrote:
>
> > > What then is the benefit? If what is installed is the same from
> > > package manager or binpkg. Also your redistributing another's
> > > package in binary format which may not be legally allowed.
> >
> > The difference is that how the package manager/ebuild installs the
> > package may be better suited to the environment than what upstream
> > expects (such as upstreams that install through a .run file)
>
> I fail to see how basically skipping src_install and maybe some prepare
> stuff that makes it better suited to an environment.
> Can you explain that further?
>
> These packages are just exploded tarballs. I fail to see the benefit
> to repacking those into another tarball to be exploded. At best
> skipping src_install and/or prepare, seems to be the only difference.
>

one such benefit is that the binhost is known and managed by someone you
trust, SRC_URI point to the wider and dangerous internet.
So please leave this as a configurable choice.


>
> I see no difference in installing kernel sources via source ebuild or a
> binpkg, pre-built ebuild binary. Other than the time it takes to
> re-package the kernel sources into another tarball.
>
>
> --
> William L. Thomson Jr.
>

[-- Attachment #2: Type: text/html, Size: 2076 bytes --]

Re: [gentoo-dev] Prevent binary/non-compiled packages from binary package creation

[William L. Thomson Jr.]

[-- Attachment #1: Type: text/plain, Size: 2193 bytes --]

On Wed, 9 Aug 2017 22:23:41 +0200
Francesco Riosa <vivo75@gmail.com> wrote:

> 2017-08-09 17:33 GMT+02:00 William L. Thomson Jr. <wlt-ml@o-sinc.com>:
> 
> > On Wed, 9 Aug 2017 11:07:04 +1000
> > "Sam Jorna (wraeth)" <wraeth@gentoo.org> wrote:
> >
> > > > What then is the benefit? If what is installed is the same from
> > > > package manager or binpkg. Also your redistributing another's
> > > > package in binary format which may not be legally allowed.
> > >
> > > The difference is that how the package manager/ebuild installs the
> > > package may be better suited to the environment than what upstream
> > > expects (such as upstreams that install through a .run file)
> >
> > I fail to see how basically skipping src_install and maybe some
> > prepare stuff that makes it better suited to an environment.
> > Can you explain that further?
> >
> > These packages are just exploded tarballs. I fail to see the benefit
> > to repacking those into another tarball to be exploded. At best
> > skipping src_install and/or prepare, seems to be the only
> > difference.
> >
> one such benefit is that the binhost is known and managed by someone
> you trust, SRC_URI point to the wider and dangerous internet.

Interesting argument, saying upstreams cannot be trusted. Nor Gentoo
with its manifests and hashes of the files referenced in SRC_URI. Given
that most will come from Gentoo mirrors not upstream servers. Ignoring
that the binhost has to use these SRC_URI's just the same. It makes
sense in very strange way.

FYI binpkgs have no hash. If someone did something malicious within the
binhost to the binpkgs. You have no way of knowing. Yes the same can
happen with ebuilds and manifest. But easy to sync portage and see if a
manifest has changed.

Therefore relying on binhost as means of security is possible but odd,
as it leaves things open to potentially larger issues.

> So please leave this as a configurable choice.

For some things configurable would make sense. For things like fetch
restricted, it would not. Since it is not legal to mirror that stuff to
begin with. It surely is not to re-package.

-- 
William L. Thomson Jr.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] Prevent binary/non-compiled packages from binary package creation

[Sam Jorna (wraeth)]

[-- Attachment #1.1: Type: text/plain, Size: 681 bytes --]

On 10/08/17 06:35, William L. Thomson Jr. wrote:
> FYI binpkgs have no hash. If someone did something malicious within the
> binhost to the binpkgs. You have no way of knowing. Yes the same can
> happen with ebuilds and manifest. But easy to sync portage and see if a
> manifest has changed.

This isn't exactly true - see ${PKGDIR}/Packages on the binhost, which
is a manifest of built packages and related metadata. Granted this is
created by the binhost, it does exist and contains SHA1 and MD5 hashes,
as well as package size. In that sense it's no different to how a
package Manifest file works within a repository.

-- 
Sam Jorna (wraeth)
GnuPG ID: D6180C26


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 858 bytes --]

Re: [gentoo-dev] Prevent binary/non-compiled packages from binary package creation

[William L. Thomson Jr.]

[-- Attachment #1: Type: text/plain, Size: 1535 bytes --]

On Thu, 10 Aug 2017 10:50:45 +1000
"Sam Jorna (wraeth)" <wraeth@gentoo.org> wrote:

> On 10/08/17 06:35, William L. Thomson Jr. wrote:
> > FYI binpkgs have no hash. If someone did something malicious within
> > the binhost to the binpkgs. You have no way of knowing. Yes the
> > same can happen with ebuilds and manifest. But easy to sync portage
> > and see if a manifest has changed.  
> 
> This isn't exactly true - see ${PKGDIR}/Packages on the binhost, which
> is a manifest of built packages and related metadata. Granted this is
> created by the binhost, it does exist and contains SHA1 and MD5
> hashes, as well as package size. In that sense it's no different to
> how a package Manifest file works within a repository.

You are correct. I meant to say no verifiable hash. You can hash
anything locally and claim it to be trustworthy. Thus mentioning
syncing portage to compare manifest of ebuild/SRC_URI.

Someone remakes a binpkg tarball, edits ${PKGDIR}/Packages with new
SHA1 and MD5. No way to know.

IMHO SRI_URI is more trustworthy than binhost, in the sense of
verification. If  you have means to verify the binhost stuff it maybe
more trustworthy. That is left to the admin.

I see binpkg as a temporary convenience. I am doing updates across many
of the same systems. Less images, containers, etc. I made binaries on
one system. Immediately used as updated on others. Then discarded on
binhost. Also used for  testing, reverting between slotted versions.

-- 
William L. Thomson Jr.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] Prevent binary/non-compiled packages from binary package creation

[Sam Jorna (wraeth)]

[-- Attachment #1.1: Type: text/plain, Size: 1602 bytes --]

On 10/08/17 11:42, William L. Thomson Jr. wrote:
> On Thu, 10 Aug 2017 10:50:45 +1000
> "Sam Jorna (wraeth)" <wraeth@gentoo.org> wrote:
> 
>> On 10/08/17 06:35, William L. Thomson Jr. wrote:
>>> FYI binpkgs have no hash. If someone did something malicious within
>>> the binhost to the binpkgs. You have no way of knowing. Yes the
>>> same can happen with ebuilds and manifest. But easy to sync portage
>>> and see if a manifest has changed.  
>>
>> This isn't exactly true - see ${PKGDIR}/Packages on the binhost, which
>> is a manifest of built packages and related metadata. Granted this is
>> created by the binhost, it does exist and contains SHA1 and MD5
>> hashes, as well as package size. In that sense it's no different to
>> how a package Manifest file works within a repository.
> 
> You are correct. I meant to say no verifiable hash. You can hash
> anything locally and claim it to be trustworthy. Thus mentioning
> syncing portage to compare manifest of ebuild/SRC_URI<snip>
> IMHO SRI_URI is more trustworthy than binhost, in the sense of
> verification. If  you have means to verify the binhost stuff it maybe
> more trustworthy. That is left to the admin.

This is no greater risk than syncing from a potentially compromised
mirror. You would use a mirror you trust and, similarly (perhaps even
more so) you would use a binhost you trust.

It does raise the idea of some form of signing of the Packages file,
similar to gpg-signed portage snapshots, but that's moving well beyond
the scope of this thread.

-- 
Sam Jorna (wraeth)
GnuPG ID: D6180C26


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 858 bytes --]

Re: [gentoo-dev] Prevent binary/non-compiled packages from binary package creation

[William L. Thomson Jr.]

[-- Attachment #1: Type: text/plain, Size: 1846 bytes --]

On Thu, 10 Aug 2017 13:33:54 +1000
"Sam Jorna (wraeth)" <wraeth@gentoo.org> wrote:
>
> This is no greater risk than syncing from a potentially compromised
> mirror. You would use a mirror you trust and, similarly (perhaps even
> more so) you would use a binhost you trust.

Getting a bit ridiculous now. Let me get my tin foil hat.

So your suggesting Gentoo mirrors are could be compromised? Your saying
that Gentoo repo gets compromised. Which then leaks out onto mirrors. If
a mirror is compromised, clearly it would not match up to other mirrors
or the master Gentoo repo. All with no one in the world noticing. Not a
likely scenario.

Lets go down this rabbit hole. Lets say Gentoo repo was compromised.
You simply look at upstream sources and their hashes. If Gentoo
mirrored sources do not match up to upstream. Then you know something
is wrong.

Thus you have many ways to verify, pull from mirror, compare to mirror,
compared to master Gentoo repo, compare to upstream. None of that can
be done with a binpkg. There are no public binhost. There is no
official Gentoo binhost. That is something people setup.

They may trust their own binhost. But to imply that is more trust
worthy than public stuff that is in more than one verifiable location
against 3rd parties. That logic does not hold up.

> It does raise the idea of some form of signing of the Packages file,
> similar to gpg-signed portage snapshots, but that's moving well beyond
> the scope of this thread.

That still would never give you any 3rd party verification. Why do we
not self sign certificates? Why are those not trusted? Trust tends to
come from 3rd parties.

Even GPG relies on a WOT, without that its pointless. An unsigned GPG
key is pretty worthless. Signing stuff with that means nothing.

-- 
William L. Thomson Jr.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] Prevent binary/non-compiled packages from binary package creation

[Sam Jorna (wraeth)]

[-- Attachment #1.1: Type: text/plain, Size: 147 bytes --]

On 11/08/17 03:08, William L. Thomson Jr. wrote:
> Lets go down this rabbit hole.

Let's not.

-- 
Sam Jorna (wraeth)
GnuPG ID: D6180C26


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 858 bytes --]

Re: [gentoo-dev] Prevent binary/non-compiled packages from binary package creation

[Sam Jorna (wraeth)]

[-- Attachment #1.1: Type: text/plain, Size: 534 bytes --]

On 09/08/17 10:43, William L. Thomson Jr. wrote:
> Also your redistributing another's package
> in binary format which may not be legally allowed.

Just to clarify, I wasn't suggesting redistributing license-encumbered
packages. Since binary packages are managed by the system administrator,
not Gentoo (as a distro), it remains with the administrator to adhere to
any relevant license restrictions. Plus the package manager can't tell
if you're distributing binaries or not.

-- 
Sam Jorna (wraeth)
GnuPG ID: D6180C26


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 858 bytes --]

Re: [gentoo-dev] Prevent binary/non-compiled packages from binary package creation

[William L. Thomson Jr.]

[-- Attachment #1: Type: text/plain, Size: 874 bytes --]

On Thu, 10 Aug 2017 11:25:34 +1000
"Sam Jorna (wraeth)" <wraeth@gentoo.org> wrote:

> On 09/08/17 10:43, William L. Thomson Jr. wrote:
> > Also your redistributing another's package
> > in binary format which may not be legally allowed.  
> 
> Just to clarify, I wasn't suggesting redistributing license-encumbered
> packages. Since binary packages are managed by the system
> administrator, not Gentoo (as a distro), it remains with the
> administrator to adhere to any relevant license restrictions. Plus
> the package manager can't tell if you're distributing binaries or not.

Sure, I was just pointing out that there maybe legal needs to prevent
such. Unless someone wants to circumvent. Their call but not default.

The package manager knows about fetch restricted ebuilds. It should
not to re-package that stuff. IMHO

-- 
William L. Thomson Jr.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] Prevent binary/non-compiled packages from binary package creation

[Sam Jorna (wraeth)]

[-- Attachment #1.1: Type: text/plain, Size: 1209 bytes --]

On 10/08/17 11:47, William L. Thomson Jr. wrote:
> On Thu, 10 Aug 2017 11:25:34 +1000
> "Sam Jorna (wraeth)" <wraeth@gentoo.org> wrote:
> 
>> On 09/08/17 10:43, William L. Thomson Jr. wrote:
>>> Also your redistributing another's package
>>> in binary format which may not be legally allowed.  
>>
>> Just to clarify, I wasn't suggesting redistributing license-encumbered
>> packages. Since binary packages are managed by the system
>> administrator, not Gentoo (as a distro), it remains with the
>> administrator to adhere to any relevant license restrictions. Plus
>> the package manager can't tell if you're distributing binaries or not.
> 
> Sure, I was just pointing out that there maybe legal needs to prevent
> such. Unless someone wants to circumvent. Their call but not default.
> 
> The package manager knows about fetch restricted ebuilds. It should
> not to re-package that stuff. IMHO

Packaging a binary is not redistributing. Building binpkgs does not mean
you're going to redistribute them, and even if you were, the package
manager has no way of determining that aside from an
--im-going-to-redist-this-package option.

-- 
Sam Jorna (wraeth)
GnuPG ID: D6180C26


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 858 bytes --]

Re: [gentoo-dev] Prevent binary/non-compiled packages from binary package creation

[Ian Stakenvicius]

[-- Attachment #1.1: Type: text/plain, Size: 696 bytes --]

On 08/08/17 01:23 PM, William L. Thomson Jr. wrote:
> On Tue, 8 Aug 2017 19:11:18 +0200
> Kristian Fiskerstrand <k_f@gentoo.org> wrote:
> 
>> it can already be controlled through env files.
> 
> I was thinking it might, but having used them to skip other hooks. I
> was thinking they could not be used as such for binary packages. Have
> you confirmed such is possible?  Could you provide a link or example?
> Thanks!

I'm not sure explicitly about environment files, but it's an option to
emerge.  For instance, I've added this to my EMERGE_DEFAULT_OPTS to
ensure none of the following are built:

--buildpkg-exclude "virtual/* sys-kernel/*-sources dev-perl/* perl-core/*"




[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 248 bytes --]

Re: [gentoo-dev] Prevent binary/non-compiled packages from binary package creation

[William L. Thomson Jr.]

[-- Attachment #1: Type: text/plain, Size: 594 bytes --]

On Tue, 8 Aug 2017 13:34:00 -0400
Ian Stakenvicius <axs@gentoo.org> wrote:

> I'm not sure explicitly about environment files, but it's an option to
> emerge.  For instance, I've added this to my EMERGE_DEFAULT_OPTS to
> ensure none of the following are built:
> 
> --buildpkg-exclude "virtual/* sys-kernel/*-sources dev-perl/*
> perl-core/*"

Something like this would NOT be desirable. It would have to be done on
every system. Package env files, if possible, would be a better
approach as that could be distribute to be consistent on all systems.

-- 
William L. Thomson Jr.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] Prevent binary/non-compiled packages from binary package creation

[Kristian Fiskerstrand]

[-- Attachment #1.1: Type: text/plain, Size: 729 bytes --]

On 08/08/2017 08:10 PM, William L. Thomson Jr. wrote:
>> I'm not sure explicitly about environment files, but it's an option to
>> emerge.  For instance, I've added this to my EMERGE_DEFAULT_OPTS to
>> ensure none of the following are built:
>>
>> --buildpkg-exclude "virtual/* sys-kernel/*-sources dev-perl/*
>> perl-core/*"
> Something like this would NOT be desirable. It would have to be done on
> every system. 
It would have to be set on every binhost, not every client system.. that
said, I prefer env approach as it is easier to track changes properly in
a CVS

-- 
Kristian Fiskerstrand
OpenPGP keyblock reachable at hkp://pool.sks-keyservers.net
fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [gentoo-dev] Prevent binary/non-compiled packages from binary package creation

[William L. Thomson Jr.]

[-- Attachment #1: Type: text/plain, Size: 1140 bytes --]

On Tue, 8 Aug 2017 20:15:07 +0200
Kristian Fiskerstrand <k_f@gentoo.org> wrote:

> On 08/08/2017 08:10 PM, William L. Thomson Jr. wrote:
> >> I'm not sure explicitly about environment files, but it's an
> >> option to emerge.  For instance, I've added this to my
> >> EMERGE_DEFAULT_OPTS to ensure none of the following are built:
> >>
> >> --buildpkg-exclude "virtual/* sys-kernel/*-sources dev-perl/*
> >> perl-core/*"  
> > Something like this would NOT be desirable. It would have to be
> > done on every system.   
> It would have to be set on every binhost, not every client system..
> that said, I prefer env approach as it is easier to track changes
> properly in a CVS

That is assuming clients do not have FEATURES="buildpkg". I have that
set just in case I merge some package directly on a system. I have the
binary ready for others. I am trying out the env route now, but may
need some changes there.

Most are made on the binhost, but not all. Also I am not necessarily
using a binhost per se. I have portage on shared NFS. I also rsync some
binaries between NFS servers.

-- 
William L. Thomson Jr.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] Prevent binary/non-compiled packages from binary package creation

[William L. Thomson Jr.]

[-- Attachment #1: Type: text/plain, Size: 787 bytes --]

Just to clarify, the contenders for no binpkg would be the
following, potentially more.

 - ebuilds that are fetch restricted
 - ebuilds that installs files unchanged, like kernel sources
 - Binary ebuilds, -bin, that just use src_install and do not build
   anything

There may be some other cases, but I think that covers the main ones.
The first case, should NEVER, not even optionally be allowed to be
binpkg. That is re-distributing something that is fetch restricted. If
it cannot be mirrored, I doubt it can legally be  re-packaged.

The later 2 could be "optional" defaulted to not build, but could be
forced. There is little benefit at that point but some may prefer those
be a binpkg.

I have no problem with it being optional.

-- 
William L. Thomson Jr.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[gentoo-dev] Re: Prevent binary/non-compiled packages from binary package creation

[Duncan]

William L. Thomson Jr. posted on Tue, 08 Aug 2017 12:37:42 -0400 as
excerpted:

> I make a lot of binaries for use on other systems, to expedite updates.
> It does not make sense for some packages to ever be a binary package.
> 
> Packages like -bin packages or gentoo-sources, which are just sources.
> Having binary ebuilds of those is of no benefit. I can be the opposite
> causing things to be much slower. Like in the case of large things
> packages like android-studio.

There's actually potentially significant differences and potential 
benefit.  In addition to those already mentioned by others...

* Binpkgs packages store the build environment as well.  This includes 
eclass functions, etc, which are used from the binpkg, not from the 
existing tree, which may differ from that used at package creation.

For example, some time ago I upgraded glibc (on ~arch) and had to roll 
back.  Downgrading glibc is normally prohibited due to other dependencies 
that may have been built since that could now depend on the newer glibc, 
but fortunately I caught the problem quickly and only a handful of 
packages had been rebuilt after that, and the problem was bad enough that 
rebuilding the few if necessary was trivial compared to dealing with the 
broken glibc functionality.

Fine, I thought, just emerge the old binpg.  Not so easy because it 
refused to downgrade unless I_KNOW_WHAT_I_AM_DOING was set, and setting 
that was useless because it wasn't in the binpkg environment.  So I ended 
up doing a full rebuild to get it in the binpkg environment.  IIRC I had 
to do it from a backup, however, because glibc was broken enough I 
couldn't do it from the working copy, that being the reason I caught it 
so fast in the first place.

Now I keep that variable set for glibc via package.environment, so it's 
always in my glibc binpkgs in case I need to downgrade and I won't have 
to do a full rebuild of the old version next time.

Obviously glibc's not a pre-built binary, but the same thing could apply 
there.  A variable could be set only on the pkghost, that would apply to 
all installs of that binpkg due to the saved environment, that wouldn't 
apply to a non-binpkg merge of the same ebuild.

IOW, installing from a binpkg and from the existing tree ebuild could 
result in differences in the installed package, due only to the 
environment-saving.

> Just seems odd to make a binary of a binary, or repackage gentoo-sources
> into a gentoo ebuild binary/binpkg. There is not really any benefit
> either way for such packages.

While it does seem odd, there are certainly benefits in some cases...

> It would be beneficial if ebuilds could have some internal variable that
> prevents it from being a binary package. It should not prevent merge or
> fail. Just never create a binary of this package, always use the ebuild
> as normal. Even if someone is force installing using binary flag or
> otherwise.

Having an internal binary-prebuild variable set could indeed be useful, 
but agreed with the others, acting on it should be an option or perhaps 
an optional FEATURE, controllable by the sysadmin/user, not mandatory.

FWIW, I'll almost certainly keep building binpkgs on here, regardless, 
because among other things I've found it just too useful to be able to go 
back and see what that older version I have archived looked like, both in 
terms of the files included, and the saved ebuild and eclass code.  I 
can't count the times I've found it useful to have those old binpkgs for 
reference, and in fact, that's one of major benefits of using 
FEATURES=buildpkg in the first place, regardless of the package, in my 
book.

Meanwhile, having buildpkg on virtual packages[1] is what amuses me.  
There, most of the benefits of binpkgs that arguably apply even to 
prebuilt binary and no-bin packages as long as they package and install 
/some/ file, arguably don't apply at all, tho I suppose there might be 
corner cases where they /could/.  

---
[1] Virtual packages: Including my own occasional null-pkg.  Like
sys-fs/udisks, for instance.  It's a runtime-only dep of
kde-frameworks/solid, used for functionality I don't want/need anyway, so 
I null-pkg it with an overlay version that has no deps and installs no 
files.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: Prevent binary/non-compiled packages from binary package creation

[William L. Thomson Jr.]

[-- Attachment #1: Type: text/plain, Size: 598 bytes --]

On Thu, 10 Aug 2017 23:30:37 +0000 (UTC)
Duncan <1i5t5.duncan@cox.net> wrote:

> Meanwhile, having buildpkg on virtual packages[1] is what amuses me.  
> There, most of the benefits of binpkgs that arguably apply even to 
> prebuilt binary and no-bin packages as long as they package and
> install /some/ file, arguably don't apply at all, tho I suppose there
> might be corner cases where they /could/.  

That is two other cases I did not think of, thanks for mentioning.

 - virtual ebuilds
 - meta ebuilds

binpkgs of those do not make much sense.

-- 
William L. Thomson Jr.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]