[gentoo-dev] News Item: Portage rsync hardlink support

[gentoo-dev] News Item: Portage rsync hardlink support

[Zac Medico]

Please review.

Title: Portage rsync hardlink support
Author: Zac Medico <zmedico@gentoo.org>
Posted: 2018-07-11
Revision: 1
News-Item-Format: 2.0
Display-If-Installed: sys-apps/portage

For users of the rsync tree, beginning with sys-apps/portage-2.3.42,
the default behavior for sync operations will use hardlinks in order
to ensure that a repository remains in a valid state if something
goes wrong [1]. For example, if signature verification fails during a
sync operation, the new hardlink behavior will preserve the previous
state of the repository.

The new behavior may conflict with configurations that restrict the
use of hardlinks, such as overlay filesystems. Therefore, users will
have to set "sync-allow-hardlinks = no" in repos.conf if they have
a configuration that restricts the use of hardlinks, but this should
not be very common:

[DEFAULT]
sync-allow-hardlinks = no

[1] https://bugs.gentoo.org/660410 sys-apps/portage: use rsync
    --link-dest to implement atomic repository updates (and abort if
    signature verification fails)
-- 
Thanks,
Zac

Re: [gentoo-dev] News Item: Portage rsync hardlink support

[Michał Górny]

Dnia 8 lipca 2018 08:08:31 CEST, Zac Medico <zmedico@gentoo.org> napisał(a):
>Please review.
>
>Title: Portage rsync hardlink support
>Author: Zac Medico <zmedico@gentoo.org>
>Posted: 2018-07-11
>Revision: 1
>News-Item-Format: 2.0
>Display-If-Installed: sys-apps/portage
>
>For users of the rsync tree, beginning with sys-apps/portage-2.3.42,
>the default behavior for sync operations will use hardlinks in order
>to ensure that a repository remains in a valid state if something
>goes wrong [1]. For example, if signature verification fails during a
>sync operation, the new hardlink behavior will preserve the previous
>state of the repository.
>
>The new behavior may conflict with configurations that restrict the
>use of hardlinks, such as overlay filesystems. Therefore, users will
>have to set "sync-allow-hardlinks = no" in repos.conf if they have
>a configuration that restricts the use of hardlinks, but this should
>not be very common:
>
>[DEFAULT]
>sync-allow-hardlinks = no
>
>[1] https://bugs.gentoo.org/660410 sys-apps/portage: use rsync
>    --link-dest to implement atomic repository updates (and abort if
>    signature verification fails)

Is safe git syncing implemented already? If not, maybe finish it first and cover both with a single news item. Git is going to be more efficient here, so people may want to learn they have an alternative.

-- 
Best regards,
Michał Górny (by phone)

Re: [gentoo-dev] News Item: Portage rsync hardlink support

[Zac Medico]

[-- Attachment #1.1: Type: text/plain, Size: 1994 bytes --]

On 07/07/2018 11:53 PM, Michał Górny wrote:
> Dnia 8 lipca 2018 08:08:31 CEST, Zac Medico <zmedico@gentoo.org> napisał(a):
>> Please review.
>>
>> Title: Portage rsync hardlink support
>> Author: Zac Medico <zmedico@gentoo.org>
>> Posted: 2018-07-11
>> Revision: 1
>> News-Item-Format: 2.0
>> Display-If-Installed: sys-apps/portage
>>
>> For users of the rsync tree, beginning with sys-apps/portage-2.3.42,
>> the default behavior for sync operations will use hardlinks in order
>> to ensure that a repository remains in a valid state if something
>> goes wrong [1]. For example, if signature verification fails during a
>> sync operation, the new hardlink behavior will preserve the previous
>> state of the repository.
>>
>> The new behavior may conflict with configurations that restrict the
>> use of hardlinks, such as overlay filesystems. Therefore, users will
>> have to set "sync-allow-hardlinks = no" in repos.conf if they have
>> a configuration that restricts the use of hardlinks, but this should
>> not be very common:
>>
>> [DEFAULT]
>> sync-allow-hardlinks = no
>>
>> [1] https://bugs.gentoo.org/660410 sys-apps/portage: use rsync
>>    --link-dest to implement atomic repository updates (and abort if
>>    signature verification fails)
> 
> Is safe git syncing implemented already? If not, maybe finish it first and cover both with a single news item. Git is going to be more efficient here, so people may want to learn they have an alternative.

Yeah there's already a patch for git sync [1] but I'd rather not make
this news item more complicated than it needs to be. I wouldn't have
bothered with a news item except that I want to give people some warning
in case they are using overlayfs [2]. I think the efficiency difference
between rsync and git here are pretty negligible for most people.

[1] https://bugs.gentoo.org/660372
[2]
https://www.brunsware.de/blog/gentoo/portage-tree-squashfs-overlayfs.html
-- 
Thanks,
Zac


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: [gentoo-dev] News Item: Portage rsync hardlink support

[Michał Górny]

Dnia 8 lipca 2018 09:14:06 CEST, Zac Medico <zmedico@gentoo.org> napisał(a):
>On 07/07/2018 11:53 PM, Michał Górny wrote:
>> Dnia 8 lipca 2018 08:08:31 CEST, Zac Medico <zmedico@gentoo.org>
>napisał(a):
>>> Please review.
>>>
>>> Title: Portage rsync hardlink support
>>> Author: Zac Medico <zmedico@gentoo.org>
>>> Posted: 2018-07-11
>>> Revision: 1
>>> News-Item-Format: 2.0
>>> Display-If-Installed: sys-apps/portage
>>>
>>> For users of the rsync tree, beginning with sys-apps/portage-2.3.42,
>>> the default behavior for sync operations will use hardlinks in order
>>> to ensure that a repository remains in a valid state if something
>>> goes wrong [1]. For example, if signature verification fails during
>a
>>> sync operation, the new hardlink behavior will preserve the previous
>>> state of the repository.
>>>
>>> The new behavior may conflict with configurations that restrict the
>>> use of hardlinks, such as overlay filesystems. Therefore, users will
>>> have to set "sync-allow-hardlinks = no" in repos.conf if they have
>>> a configuration that restricts the use of hardlinks, but this should
>>> not be very common:
>>>
>>> [DEFAULT]
>>> sync-allow-hardlinks = no
>>>
>>> [1] https://bugs.gentoo.org/660410 sys-apps/portage: use rsync
>>>    --link-dest to implement atomic repository updates (and abort if
>>>    signature verification fails)
>> 
>> Is safe git syncing implemented already? If not, maybe finish it
>first and cover both with a single news item. Git is going to be more
>efficient here, so people may want to learn they have an alternative.
>
>Yeah there's already a patch for git sync [1] but I'd rather not make
>this news item more complicated than it needs to be. I wouldn't have
>bothered with a news item except that I want to give people some
>warning
>in case they are using overlayfs [2]. I think the efficiency difference
>between rsync and git here are pretty negligible for most people.

Are you sure about that? That might have been the case so far but this hardlink tree may actually tip the balance.

>
>[1] https://bugs.gentoo.org/660372
>[2]
>https://www.brunsware.de/blog/gentoo/portage-tree-squashfs-overlayfs.html


-- 
Best regards,
Michał Górny (by phone)

Re: [gentoo-dev] News Item: Portage rsync hardlink support

[Zac Medico]

[-- Attachment #1.1: Type: text/plain, Size: 2383 bytes --]

On 07/08/2018 02:15 AM, Michał Górny wrote:
> Dnia 8 lipca 2018 09:14:06 CEST, Zac Medico <zmedico@gentoo.org> napisał(a):
>> On 07/07/2018 11:53 PM, Michał Górny wrote:
>>> Dnia 8 lipca 2018 08:08:31 CEST, Zac Medico <zmedico@gentoo.org>
>> napisał(a):
>>>> Please review.
>>>>
>>>> Title: Portage rsync hardlink support
>>>> Author: Zac Medico <zmedico@gentoo.org>
>>>> Posted: 2018-07-11
>>>> Revision: 1
>>>> News-Item-Format: 2.0
>>>> Display-If-Installed: sys-apps/portage
>>>>
>>>> For users of the rsync tree, beginning with sys-apps/portage-2.3.42,
>>>> the default behavior for sync operations will use hardlinks in order
>>>> to ensure that a repository remains in a valid state if something
>>>> goes wrong [1]. For example, if signature verification fails during
>> a
>>>> sync operation, the new hardlink behavior will preserve the previous
>>>> state of the repository.
>>>>
>>>> The new behavior may conflict with configurations that restrict the
>>>> use of hardlinks, such as overlay filesystems. Therefore, users will
>>>> have to set "sync-allow-hardlinks = no" in repos.conf if they have
>>>> a configuration that restricts the use of hardlinks, but this should
>>>> not be very common:
>>>>
>>>> [DEFAULT]
>>>> sync-allow-hardlinks = no
>>>>
>>>> [1] https://bugs.gentoo.org/660410 sys-apps/portage: use rsync
>>>>    --link-dest to implement atomic repository updates (and abort if
>>>>    signature verification fails)
>>>
>>> Is safe git syncing implemented already? If not, maybe finish it
>> first and cover both with a single news item. Git is going to be more
>> efficient here, so people may want to learn they have an alternative.
>>
>> Yeah there's already a patch for git sync [1] but I'd rather not make
>> this news item more complicated than it needs to be. I wouldn't have
>> bothered with a news item except that I want to give people some
>> warning
>> in case they are using overlayfs [2]. I think the efficiency difference
>> between rsync and git here are pretty negligible for most people.
> 
> Are you sure about that? That might have been the case so far but this hardlink tree may actually tip the balance.

Even if it takes twice a long (which it doesn't), the difference is
negligible for most people because they usually don't sync more than
once per day.
-- 
Thanks,
Zac


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: [gentoo-dev] News Item: Portage rsync hardlink support

[Toralf Förster]

[-- Attachment #1.1: Type: text/plain, Size: 591 bytes --]

On 07/08/2018 08:08 AM, Zac Medico wrote:
> Please review.
>
> Title: Portage rsync hardlink support
> Author: Zac Medico <zmedico@gentoo.org>
> Posted: 2018-07-11
> Revision: 1
> News-Item-Format: 2.0
> Display-If-Installed: sys-apps/portage
> IMO there's another heads-up for users having an unsual configuration:

So we do speak about files under /usr/portage itself and not about that
dir (==changing its inode number), right?

B/c  otherwise there's another heads-up for people bind-mounting
/usr/portage onto chrooted images.

-- 
Toralf
PGP 23217DA7 9B888F45



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] News Item: Portage rsync hardlink support

[M. J. Everitt]

[-- Attachment #1.1: Type: text/plain, Size: 495 bytes --]

On 08/07/18 10:21, Zac Medico wrote:
> On 07/08/2018 02:15 AM, Michał Górny wrote:
>> <snip>
>> Are you sure about that? That might have been the case so far but this hardlink tree may actually tip the balance.
> Even if it takes twice a long (which it doesn't), the difference is
> negligible for most people because they usually don't sync more than
> once per day.
It is recommended not to hammer the rsync mirrors with updates, and so
this argument holds true for "most users".


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] News Item: Portage rsync hardlink support

[Zac Medico]

[-- Attachment #1.1: Type: text/plain, Size: 819 bytes --]

On 07/08/2018 02:28 AM, Toralf Förster wrote:
> On 07/08/2018 08:08 AM, Zac Medico wrote:
>> Please review.
>>
>> Title: Portage rsync hardlink support
>> Author: Zac Medico <zmedico@gentoo.org>
>> Posted: 2018-07-11
>> Revision: 1
>> News-Item-Format: 2.0
>> Display-If-Installed: sys-apps/portage
>> IMO there's another heads-up for users having an unsual configuration:
> 
> So we do speak about files under /usr/portage itself and not about that
> dir (==changing its inode number), right?
> 
> B/c  otherwise there's another heads-up for people bind-mounting
> /usr/portage onto chrooted images.

This case will continue to work because the download happens in a
subdirectory, and after it's verified we call rsync again to apply the
changes to the top-level directory.
-- 
Thanks,
Zac


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: [gentoo-dev] News Item: Portage rsync hardlink support

[Kristian Fiskerstrand]

[-- Attachment #1.1: Type: text/plain, Size: 450 bytes --]

On 07/08/2018 08:08 AM, Zac Medico wrote:
>  For example, if signature verification fails during a
> sync operation, the new hardlink behavior will preserve the previous
> state of the repository.

This seems like a nice improvement, thank you for implementing it and
making it the new default.

-- 
Kristian Fiskerstrand
OpenPGP keyblock reachable at hkp://pool.sks-keyservers.net
fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [gentoo-dev] News Item: Portage rsync hardlink support

[Kristian Fiskerstrand]

[-- Attachment #1.1: Type: text/plain, Size: 615 bytes --]

On 07/08/2018 08:53 AM, Michał Górny wrote:
> Is safe git syncing implemented already? If not, maybe finish it first and cover both with a single news item. Git is going to be more efficient here, so people may want to learn they have an alternative.

Why complicate things, and increase wait for something that benefits
most users, just to give alternatives to a few using non-default sync
mechanism. Securing git distribution is a whole different ballpark.

-- 
Kristian Fiskerstrand
OpenPGP keyblock reachable at hkp://pool.sks-keyservers.net
fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [gentoo-dev] News Item: Portage rsync hardlink support

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1039 bytes --]

W dniu nie, 08.07.2018 o godzinie 15∶02 +0200, użytkownik Kristian
Fiskerstrand napisał:
> On 07/08/2018 08:53 AM, Michał Górny wrote:
> > Is safe git syncing implemented already? If not, maybe finish it first and cover both with a single news item. Git is going to be more efficient here, so people may want to learn they have an alternative.
> 
> Why complicate things, and increase wait for something that benefits
> most users, just to give alternatives to a few using non-default sync
> mechanism. Securing git distribution is a whole different ballpark.
> 

Let me rephrase.  Let's say I'm using rsync.  This new feature is
something positive but it breaks my use case (for one of the listed
reasons -- overlayfs, inode use, small fs cache).  After reading this
news item, I learn that my only option is to disable the new feature.

Now, I would appreciate being told that there's an alternate sync method
that handles secure updates without having all those drawbacks.

-- 
Best regards,
Michał Górny

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

Re: [gentoo-dev] News Item: Portage rsync hardlink support

[Rich Freeman]

On Sun, Jul 8, 2018 at 9:02 AM Kristian Fiskerstrand <k_f@gentoo.org> wrote:
>
> On 07/08/2018 08:53 AM, Michał Górny wrote:
> > Is safe git syncing implemented already? If not, maybe finish it first and cover both with a single news item. Git is going to be more efficient here, so people may want to learn they have an alternative.
>
> Why complicate things, and increase wait for something that benefits
> most users, just to give alternatives to a few using non-default sync
> mechanism. Securing git distribution is a whole different ballpark.
>

I'll agree that it is different, but we're talking about verification
of the HEAD signature by infra, not verification of individual
developer keys, which was the topic of the recent thread.

Verification is already built-into portage for git syncing (but off by
default).  The problem is that portage will still checkout the tree if
it fails verification.  The patch is to do the verification before
checking it out so that if it fails the tree is left in a
last-known-good state (at least as seen by tools at the filesystem
level - the fetched bad commits would still be visible to git).

-- 
Rich

Re: [gentoo-dev] News Item: Portage rsync hardlink support

[M. J. Everitt]

[-- Attachment #1.1: Type: text/plain, Size: 1651 bytes --]



On 08/07/18 18:34, Rich Freeman wrote:
> On Sun, Jul 8, 2018 at 9:02 AM Kristian Fiskerstrand <k_f@gentoo.org> wrote:
>> On 07/08/2018 08:53 AM, Michał Górny wrote:
>>> Is safe git syncing implemented already? If not, maybe finish it first and cover both with a single news item. Git is going to be more efficient here, so people may want to learn they have an alternative.
>> Why complicate things, and increase wait for something that benefits
>> most users, just to give alternatives to a few using non-default sync
>> mechanism. Securing git distribution is a whole different ballpark.
>>
> I'll agree that it is different, but we're talking about verification
> of the HEAD signature by infra, not verification of individual
> developer keys, which was the topic of the recent thread.
>
> Verification is already built-into portage for git syncing (but off by
> default).  The problem is that portage will still checkout the tree if
> it fails verification.  The patch is to do the verification before
> checking it out so that if it fails the tree is left in a
> last-known-good state (at least as seen by tools at the filesystem
> level - the fetched bad commits would still be visible to git).
>
Slightly radical thought here, but hear me out ..

Could we use this same functionality to be able to validate the tree
integrity with respect to CI testing? I mean, if the tree is 'broken'
could we have some kind of warning displayed perhaps? Something that
could be toggled (or default Off) would indeed be good, so that
users/devs can choose what level or 'standard' of tree state they're
prepared to accept.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] News Item: Portage rsync hardlink support

[Kristian Fiskerstrand]

[-- Attachment #1.1: Type: text/plain, Size: 962 bytes --]

On 07/08/2018 07:34 PM, Rich Freeman wrote:
>  The patch is to do the verification before
> checking it out so that if it fails the tree is left in a
> last-known-good state (at least as seen by tools at the filesystem
> level - the fetched bad commits would still be visible to git).

there is still a very different key management issue discussed. If a
developers credentials are removed from Gentoo LDAP for some reason it
will be stopped from pushing new commits immediately, but the third
party verification can be valid up until that point and after since the
developer might not have published a revocation certificate.

the git sync method will need a way to distinguish this for end-users,
but the proper rsync synchronization will be able to trust the data at
the point we say it is OK.


-- 
Kristian Fiskerstrand
OpenPGP keyblock reachable at hkp://pool.sks-keyservers.net
fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [gentoo-dev] News Item: Portage rsync hardlink support

[Zac Medico]

[-- Attachment #1.1: Type: text/plain, Size: 1417 bytes --]

On 07/08/2018 06:56 AM, Michał Górny wrote:
> W dniu nie, 08.07.2018 o godzinie 15∶02 +0200, użytkownik Kristian
> Fiskerstrand napisał:
>> On 07/08/2018 08:53 AM, Michał Górny wrote:
>>> Is safe git syncing implemented already? If not, maybe finish it first and cover both with a single news item. Git is going to be more efficient here, so people may want to learn they have an alternative.
>>
>> Why complicate things, and increase wait for something that benefits
>> most users, just to give alternatives to a few using non-default sync
>> mechanism. Securing git distribution is a whole different ballpark.
>>
> 
> Let me rephrase.  Let's say I'm using rsync.  This new feature is
> something positive but it breaks my use case (for one of the listed
> reasons -- overlayfs, inode use, small fs cache).  After reading this
> news item, I learn that my only option is to disable the new feature.
> 
> Now, I would appreciate being told that there's an alternate sync method
> that handles secure updates without having all those drawbacks.

The thing is, the normal git tree doesn't even provide pre-generated
metadata, and I see then gentoo-mirror repo that provides metadata does
not have commits signed with an release key:

https://github.com/gentoo-mirror/gentoo/commits/stable

So I'm really not comfortable recommending git to anyone at this point.
-- 
Thanks,
Zac


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: [gentoo-dev] News Item: Portage rsync hardlink support

[Rich Freeman]

On Sun, Jul 8, 2018 at 1:50 PM Kristian Fiskerstrand <k_f@gentoo.org> wrote:
>
> On 07/08/2018 07:34 PM, Rich Freeman wrote:
> >  The patch is to do the verification before
> > checking it out so that if it fails the tree is left in a
> > last-known-good state (at least as seen by tools at the filesystem
> > level - the fetched bad commits would still be visible to git).
>
> there is still a very different key management issue discussed. If a
> developers credentials are removed from Gentoo LDAP for some reason it
> will be stopped from pushing new commits immediately, but the third
> party verification can be valid up until that point and after since the
> developer might not have published a revocation certificate.
>
> the git sync method will need a way to distinguish this for end-users,
> but the proper rsync synchronization will be able to trust the data at
> the point we say it is OK.

Again, the current portage support for git verification doesn't check
any developer keys.

It just checks the keys in /usr/share/openpgp-keys/gentoo-release.asc
(after fetching updates).  If the HEAD commit verifies, it is good, if
not it is bad.  It doesn't matter whether any commit in the tree other
than the HEAD has a valid signature, and it will reject any signature
from a developer on HEAD.  The logic is somewhat similar to rsync in
that regard.

There is a separate proposal out there which is unimplemented to check
developer keys, and we aren't talking about that.  I agree that it has
the challenge of figuring out whether the key was acceptable at the
time the signature was made.

-- 
Rich

Re: [gentoo-dev] News Item: Portage rsync hardlink support

[Kristian Fiskerstrand]

[-- Attachment #1.1: Type: text/plain, Size: 423 bytes --]

On 07/08/2018 08:10 PM, Rich Freeman wrote:
> Again, the current portage support for git verification doesn't check
> any developer keys.

right, so why would it be material for a news item improving the status
quo for those synching using the official rsync method?


-- 
Kristian Fiskerstrand
OpenPGP keyblock reachable at hkp://pool.sks-keyservers.net
fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [gentoo-dev] News Item: Portage rsync hardlink support

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1927 bytes --]

W dniu nie, 08.07.2018 o godzinie 11∶04 -0700, użytkownik Zac Medico
napisał:
> On 07/08/2018 06:56 AM, Michał Górny wrote:
> > W dniu nie, 08.07.2018 o godzinie 15∶02 +0200, użytkownik Kristian
> > Fiskerstrand napisał:
> > > On 07/08/2018 08:53 AM, Michał Górny wrote:
> > > > Is safe git syncing implemented already? If not, maybe finish it first and cover both with a single news item. Git is going to be more efficient here, so people may want to learn they have an alternative.
> > > 
> > > Why complicate things, and increase wait for something that benefits
> > > most users, just to give alternatives to a few using non-default sync
> > > mechanism. Securing git distribution is a whole different ballpark.
> > > 
> > 
> > Let me rephrase.  Let's say I'm using rsync.  This new feature is
> > something positive but it breaks my use case (for one of the listed
> > reasons -- overlayfs, inode use, small fs cache).  After reading this
> > news item, I learn that my only option is to disable the new feature.
> > 
> > Now, I would appreciate being told that there's an alternate sync method
> > that handles secure updates without having all those drawbacks.
> 
> The thing is, the normal git tree doesn't even provide pre-generated
> metadata, and I see then gentoo-mirror repo that provides metadata does
> not have commits signed with an release key:
> 
> https://github.com/gentoo-mirror/gentoo/commits/stable
> 
> So I'm really not comfortable recommending git to anyone at this point.

Wrong twice.

Firstly, the canonical URL is:

  https://anongit.gentoo.org/git/repo/sync/gentoo.git
  (https://gitweb.gentoo.org/repo/sync/gentoo.git)

Secondly, the merge commits (i.e. top commits that are verified
by Portage) are signed by dedicated key that is part of the infra key
set.  In other words, it works out of the box.

-- 
Best regards,
Michał Górny

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

Re: [gentoo-dev] News Item: Portage rsync hardlink support

[Zac Medico]

[-- Attachment #1.1: Type: text/plain, Size: 2124 bytes --]

On 07/08/2018 11:42 AM, Michał Górny wrote:
> W dniu nie, 08.07.2018 o godzinie 11∶04 -0700, użytkownik Zac Medico
> napisał:
>> On 07/08/2018 06:56 AM, Michał Górny wrote:
>>> W dniu nie, 08.07.2018 o godzinie 15∶02 +0200, użytkownik Kristian
>>> Fiskerstrand napisał:
>>>> On 07/08/2018 08:53 AM, Michał Górny wrote:
>>>>> Is safe git syncing implemented already? If not, maybe finish it first and cover both with a single news item. Git is going to be more efficient here, so people may want to learn they have an alternative.
>>>>
>>>> Why complicate things, and increase wait for something that benefits
>>>> most users, just to give alternatives to a few using non-default sync
>>>> mechanism. Securing git distribution is a whole different ballpark.
>>>>
>>>
>>> Let me rephrase.  Let's say I'm using rsync.  This new feature is
>>> something positive but it breaks my use case (for one of the listed
>>> reasons -- overlayfs, inode use, small fs cache).  After reading this
>>> news item, I learn that my only option is to disable the new feature.
>>>
>>> Now, I would appreciate being told that there's an alternate sync method
>>> that handles secure updates without having all those drawbacks.
>>
>> The thing is, the normal git tree doesn't even provide pre-generated
>> metadata, and I see then gentoo-mirror repo that provides metadata does
>> not have commits signed with an release key:
>>
>> https://github.com/gentoo-mirror/gentoo/commits/stable
>>
>> So I'm really not comfortable recommending git to anyone at this point.
> 
> Wrong twice.
> 
> Firstly, the canonical URL is:
> 
>   https://anongit.gentoo.org/git/repo/sync/gentoo.git
>   (https://gitweb.gentoo.org/repo/sync/gentoo.git)
> 
> Secondly, the merge commits (i.e. top commits that are verified
> by Portage) are signed by dedicated key that is part of the infra key
> set.  In other words, it works out of the box.

Is there any documentation that shows users how to migrate to git, and
what the pros and cons might be? Maybe its worthy of its own news item.
-- 
Thanks,
Zac


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: [gentoo-dev] News Item: Portage rsync hardlink support

[Rich Freeman]

On Sun, Jul 8, 2018 at 2:31 PM Kristian Fiskerstrand <k_f@gentoo.org> wrote:
>
> On 07/08/2018 08:10 PM, Rich Freeman wrote:
> > Again, the current portage support for git verification doesn't check
> > any developer keys.
>
> right, so why would it be material for a news item improving the status
> quo for those synching using the official rsync method?
>

I never claimed it should be, though mgorny made a reasonable argument
for giving users an alternative if they can't use the hard links.

-- 
Rich

Re: [gentoo-dev] News Item: Portage rsync hardlink support

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 2514 bytes --]

W dniu nie, 08.07.2018 o godzinie 11∶57 -0700, użytkownik Zac Medico
napisał:
> On 07/08/2018 11:42 AM, Michał Górny wrote:
> > W dniu nie, 08.07.2018 o godzinie 11∶04 -0700, użytkownik Zac Medico
> > napisał:
> > > On 07/08/2018 06:56 AM, Michał Górny wrote:
> > > > W dniu nie, 08.07.2018 o godzinie 15∶02 +0200, użytkownik Kristian
> > > > Fiskerstrand napisał:
> > > > > On 07/08/2018 08:53 AM, Michał Górny wrote:
> > > > > > Is safe git syncing implemented already? If not, maybe finish it first and cover both with a single news item. Git is going to be more efficient here, so people may want to learn they have an alternative.
> > > > > 
> > > > > Why complicate things, and increase wait for something that benefits
> > > > > most users, just to give alternatives to a few using non-default sync
> > > > > mechanism. Securing git distribution is a whole different ballpark.
> > > > > 
> > > > 
> > > > Let me rephrase.  Let's say I'm using rsync.  This new feature is
> > > > something positive but it breaks my use case (for one of the listed
> > > > reasons -- overlayfs, inode use, small fs cache).  After reading this
> > > > news item, I learn that my only option is to disable the new feature.
> > > > 
> > > > Now, I would appreciate being told that there's an alternate sync method
> > > > that handles secure updates without having all those drawbacks.
> > > 
> > > The thing is, the normal git tree doesn't even provide pre-generated
> > > metadata, and I see then gentoo-mirror repo that provides metadata does
> > > not have commits signed with an release key:
> > > 
> > > https://github.com/gentoo-mirror/gentoo/commits/stable
> > > 
> > > So I'm really not comfortable recommending git to anyone at this point.
> > 
> > Wrong twice.
> > 
> > Firstly, the canonical URL is:
> > 
> >   https://anongit.gentoo.org/git/repo/sync/gentoo.git
> >   (https://gitweb.gentoo.org/repo/sync/gentoo.git)
> > 
> > Secondly, the merge commits (i.e. top commits that are verified
> > by Portage) are signed by dedicated key that is part of the infra key
> > set.  In other words, it works out of the box.
> 
> Is there any documentation that shows users how to migrate to git, and
> what the pros and cons might be? Maybe its worthy of its own news item.

Maybe.  I don't really know, and don't think it's a good idea to show 30
news item of things users might like on every new Gentoo install.

-- 
Best regards,
Michał Górny

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

Re: [gentoo-dev] News Item: Portage rsync hardlink support

[Zac Medico]

[-- Attachment #1.1: Type: text/plain, Size: 2635 bytes --]

On 07/08/2018 01:08 PM, Michał Górny wrote:
> W dniu nie, 08.07.2018 o godzinie 11∶57 -0700, użytkownik Zac Medico
> napisał:
>> On 07/08/2018 11:42 AM, Michał Górny wrote:
>>> W dniu nie, 08.07.2018 o godzinie 11∶04 -0700, użytkownik Zac Medico
>>> napisał:
>>>> On 07/08/2018 06:56 AM, Michał Górny wrote:
>>>>> W dniu nie, 08.07.2018 o godzinie 15∶02 +0200, użytkownik Kristian
>>>>> Fiskerstrand napisał:
>>>>>> On 07/08/2018 08:53 AM, Michał Górny wrote:
>>>>>>> Is safe git syncing implemented already? If not, maybe finish it first and cover both with a single news item. Git is going to be more efficient here, so people may want to learn they have an alternative.
>>>>>>
>>>>>> Why complicate things, and increase wait for something that benefits
>>>>>> most users, just to give alternatives to a few using non-default sync
>>>>>> mechanism. Securing git distribution is a whole different ballpark.
>>>>>>
>>>>>
>>>>> Let me rephrase.  Let's say I'm using rsync.  This new feature is
>>>>> something positive but it breaks my use case (for one of the listed
>>>>> reasons -- overlayfs, inode use, small fs cache).  After reading this
>>>>> news item, I learn that my only option is to disable the new feature.
>>>>>
>>>>> Now, I would appreciate being told that there's an alternate sync method
>>>>> that handles secure updates without having all those drawbacks.
>>>>
>>>> The thing is, the normal git tree doesn't even provide pre-generated
>>>> metadata, and I see then gentoo-mirror repo that provides metadata does
>>>> not have commits signed with an release key:
>>>>
>>>> https://github.com/gentoo-mirror/gentoo/commits/stable
>>>>
>>>> So I'm really not comfortable recommending git to anyone at this point.
>>>
>>> Wrong twice.
>>>
>>> Firstly, the canonical URL is:
>>>
>>>   https://anongit.gentoo.org/git/repo/sync/gentoo.git
>>>   (https://gitweb.gentoo.org/repo/sync/gentoo.git)
>>>
>>> Secondly, the merge commits (i.e. top commits that are verified
>>> by Portage) are signed by dedicated key that is part of the infra key
>>> set.  In other words, it works out of the box.
>>
>> Is there any documentation that shows users how to migrate to git, and
>> what the pros and cons might be? Maybe its worthy of its own news item.
> 
> Maybe.  I don't really know, and don't think it's a good idea to show 30
> news item of things users might like on every new Gentoo install.

Well if instructions for setting up git sync and associated pros/cons
are not documented anywhere then I won't advise anyone to use it.
-- 
Thanks,
Zac


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: [gentoo-dev] News Item: Portage rsync hardlink support

[Zac Medico]

[-- Attachment #1.1: Type: text/plain, Size: 3048 bytes --]

On 07/08/2018 01:18 PM, Zac Medico wrote:
> On 07/08/2018 01:08 PM, Michał Górny wrote:
>> W dniu nie, 08.07.2018 o godzinie 11∶57 -0700, użytkownik Zac Medico
>> napisał:
>>> On 07/08/2018 11:42 AM, Michał Górny wrote:
>>>> W dniu nie, 08.07.2018 o godzinie 11∶04 -0700, użytkownik Zac Medico
>>>> napisał:
>>>>> On 07/08/2018 06:56 AM, Michał Górny wrote:
>>>>>> W dniu nie, 08.07.2018 o godzinie 15∶02 +0200, użytkownik Kristian
>>>>>> Fiskerstrand napisał:
>>>>>>> On 07/08/2018 08:53 AM, Michał Górny wrote:
>>>>>>>> Is safe git syncing implemented already? If not, maybe finish it first and cover both with a single news item. Git is going to be more efficient here, so people may want to learn they have an alternative.
>>>>>>>
>>>>>>> Why complicate things, and increase wait for something that benefits
>>>>>>> most users, just to give alternatives to a few using non-default sync
>>>>>>> mechanism. Securing git distribution is a whole different ballpark.
>>>>>>>
>>>>>>
>>>>>> Let me rephrase.  Let's say I'm using rsync.  This new feature is
>>>>>> something positive but it breaks my use case (for one of the listed
>>>>>> reasons -- overlayfs, inode use, small fs cache).  After reading this
>>>>>> news item, I learn that my only option is to disable the new feature.
>>>>>>
>>>>>> Now, I would appreciate being told that there's an alternate sync method
>>>>>> that handles secure updates without having all those drawbacks.
>>>>>
>>>>> The thing is, the normal git tree doesn't even provide pre-generated
>>>>> metadata, and I see then gentoo-mirror repo that provides metadata does
>>>>> not have commits signed with an release key:
>>>>>
>>>>> https://github.com/gentoo-mirror/gentoo/commits/stable
>>>>>
>>>>> So I'm really not comfortable recommending git to anyone at this point.
>>>>
>>>> Wrong twice.
>>>>
>>>> Firstly, the canonical URL is:
>>>>
>>>>   https://anongit.gentoo.org/git/repo/sync/gentoo.git
>>>>   (https://gitweb.gentoo.org/repo/sync/gentoo.git)
>>>>
>>>> Secondly, the merge commits (i.e. top commits that are verified
>>>> by Portage) are signed by dedicated key that is part of the infra key
>>>> set.  In other words, it works out of the box.
>>>
>>> Is there any documentation that shows users how to migrate to git, and
>>> what the pros and cons might be? Maybe its worthy of its own news item.
>>
>> Maybe.  I don't really know, and don't think it's a good idea to show 30
>> news item of things users might like on every new Gentoo install.
> 
> Well if instructions for setting up git sync and associated pros/cons
> are not documented anywhere then I won't advise anyone to use it.

I've attempted to configure it for myself, and this is what it does:

 * Using keys from /usr/share/openpgp-keys/gentoo-release.asc
 * Refreshing keys from keyserver ...
                                                       [ ok ]
 * No valid signature found: unable to verify signature (missing key?)


-- 
Thanks,
Zac


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: [gentoo-dev] News Item: Portage rsync hardlink support

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 3547 bytes --]

W dniu nie, 08.07.2018 o godzinie 14∶11 -0700, użytkownik Zac Medico
napisał:
> On 07/08/2018 01:18 PM, Zac Medico wrote:
> > On 07/08/2018 01:08 PM, Michał Górny wrote:
> > > W dniu nie, 08.07.2018 o godzinie 11∶57 -0700, użytkownik Zac Medico
> > > napisał:
> > > > On 07/08/2018 11:42 AM, Michał Górny wrote:
> > > > > W dniu nie, 08.07.2018 o godzinie 11∶04 -0700, użytkownik Zac Medico
> > > > > napisał:
> > > > > > On 07/08/2018 06:56 AM, Michał Górny wrote:
> > > > > > > W dniu nie, 08.07.2018 o godzinie 15∶02 +0200, użytkownik Kristian
> > > > > > > Fiskerstrand napisał:
> > > > > > > > On 07/08/2018 08:53 AM, Michał Górny wrote:
> > > > > > > > > Is safe git syncing implemented already? If not, maybe finish it first and cover both with a single news item. Git is going to be more efficient here, so people may want to learn they have an alternative.
> > > > > > > > 
> > > > > > > > Why complicate things, and increase wait for something that benefits
> > > > > > > > most users, just to give alternatives to a few using non-default sync
> > > > > > > > mechanism. Securing git distribution is a whole different ballpark.
> > > > > > > > 
> > > > > > > 
> > > > > > > Let me rephrase.  Let's say I'm using rsync.  This new feature is
> > > > > > > something positive but it breaks my use case (for one of the listed
> > > > > > > reasons -- overlayfs, inode use, small fs cache).  After reading this
> > > > > > > news item, I learn that my only option is to disable the new feature.
> > > > > > > 
> > > > > > > Now, I would appreciate being told that there's an alternate sync method
> > > > > > > that handles secure updates without having all those drawbacks.
> > > > > > 
> > > > > > The thing is, the normal git tree doesn't even provide pre-generated
> > > > > > metadata, and I see then gentoo-mirror repo that provides metadata does
> > > > > > not have commits signed with an release key:
> > > > > > 
> > > > > > https://github.com/gentoo-mirror/gentoo/commits/stable
> > > > > > 
> > > > > > So I'm really not comfortable recommending git to anyone at this point.
> > > > > 
> > > > > Wrong twice.
> > > > > 
> > > > > Firstly, the canonical URL is:
> > > > > 
> > > > >   https://anongit.gentoo.org/git/repo/sync/gentoo.git
> > > > >   (https://gitweb.gentoo.org/repo/sync/gentoo.git)
> > > > > 
> > > > > Secondly, the merge commits (i.e. top commits that are verified
> > > > > by Portage) are signed by dedicated key that is part of the infra key
> > > > > set.  In other words, it works out of the box.
> > > > 
> > > > Is there any documentation that shows users how to migrate to git, and
> > > > what the pros and cons might be? Maybe its worthy of its own news item.
> > > 
> > > Maybe.  I don't really know, and don't think it's a good idea to show 30
> > > news item of things users might like on every new Gentoo install.
> > 
> > Well if instructions for setting up git sync and associated pros/cons
> > are not documented anywhere then I won't advise anyone to use it.
> 
> I've attempted to configure it for myself, and this is what it does:
> 
>  * Using keys from /usr/share/openpgp-keys/gentoo-release.asc
>  * Refreshing keys from keyserver ...
>                                                        [ ok ]
>  * No valid signature found: unable to verify signature (missing key?)
> 

Please report a bug and attach your configuration along with keyring
version.

-- 
Best regards,
Michał Górny

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

Re: [gentoo-dev] News Item: Portage rsync hardlink support

[Zac Medico]

[-- Attachment #1.1: Type: text/plain, Size: 3510 bytes --]

On 07/08/2018 02:18 PM, Michał Górny wrote:
> W dniu nie, 08.07.2018 o godzinie 14∶11 -0700, użytkownik Zac Medico
> napisał:
>> On 07/08/2018 01:18 PM, Zac Medico wrote:
>>> On 07/08/2018 01:08 PM, Michał Górny wrote:
>>>> W dniu nie, 08.07.2018 o godzinie 11∶57 -0700, użytkownik Zac Medico
>>>> napisał:
>>>>> On 07/08/2018 11:42 AM, Michał Górny wrote:
>>>>>> W dniu nie, 08.07.2018 o godzinie 11∶04 -0700, użytkownik Zac Medico
>>>>>> napisał:
>>>>>>> On 07/08/2018 06:56 AM, Michał Górny wrote:
>>>>>>>> W dniu nie, 08.07.2018 o godzinie 15∶02 +0200, użytkownik Kristian
>>>>>>>> Fiskerstrand napisał:
>>>>>>>>> On 07/08/2018 08:53 AM, Michał Górny wrote:
>>>>>>>>>> Is safe git syncing implemented already? If not, maybe finish it first and cover both with a single news item. Git is going to be more efficient here, so people may want to learn they have an alternative.
>>>>>>>>>
>>>>>>>>> Why complicate things, and increase wait for something that benefits
>>>>>>>>> most users, just to give alternatives to a few using non-default sync
>>>>>>>>> mechanism. Securing git distribution is a whole different ballpark.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Let me rephrase.  Let's say I'm using rsync.  This new feature is
>>>>>>>> something positive but it breaks my use case (for one of the listed
>>>>>>>> reasons -- overlayfs, inode use, small fs cache).  After reading this
>>>>>>>> news item, I learn that my only option is to disable the new feature.
>>>>>>>>
>>>>>>>> Now, I would appreciate being told that there's an alternate sync method
>>>>>>>> that handles secure updates without having all those drawbacks.
>>>>>>>
>>>>>>> The thing is, the normal git tree doesn't even provide pre-generated
>>>>>>> metadata, and I see then gentoo-mirror repo that provides metadata does
>>>>>>> not have commits signed with an release key:
>>>>>>>
>>>>>>> https://github.com/gentoo-mirror/gentoo/commits/stable
>>>>>>>
>>>>>>> So I'm really not comfortable recommending git to anyone at this point.
>>>>>>
>>>>>> Wrong twice.
>>>>>>
>>>>>> Firstly, the canonical URL is:
>>>>>>
>>>>>>   https://anongit.gentoo.org/git/repo/sync/gentoo.git
>>>>>>   (https://gitweb.gentoo.org/repo/sync/gentoo.git)
>>>>>>
>>>>>> Secondly, the merge commits (i.e. top commits that are verified
>>>>>> by Portage) are signed by dedicated key that is part of the infra key
>>>>>> set.  In other words, it works out of the box.
>>>>>
>>>>> Is there any documentation that shows users how to migrate to git, and
>>>>> what the pros and cons might be? Maybe its worthy of its own news item.
>>>>
>>>> Maybe.  I don't really know, and don't think it's a good idea to show 30
>>>> news item of things users might like on every new Gentoo install.
>>>
>>> Well if instructions for setting up git sync and associated pros/cons
>>> are not documented anywhere then I won't advise anyone to use it.
>>
>> I've attempted to configure it for myself, and this is what it does:
>>
>>  * Using keys from /usr/share/openpgp-keys/gentoo-release.asc
>>  * Refreshing keys from keyserver ...
>>                                                        [ ok ]
>>  * No valid signature found: unable to verify signature (missing key?)
>>
> 
> Please report a bug and attach your configuration along with keyring
> version.

It works after upgrading to openpgp-keys-gentoo-release-20180706 from
openpgp-keys-gentoo-release-20180323.
-- 
Thanks,
Zac


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: [gentoo-dev] News Item: Portage rsync hardlink support

[Aaron W. Swenson]

[-- Attachment #1.1: Type: text/plain, Size: 4009 bytes --]

On July 8, 2018 5:38:48 PM EDT, Zac Medico <zmedico@gentoo.org> wrote:
>On 07/08/2018 02:18 PM, Michał Górny wrote:
>> W dniu nie, 08.07.2018 o godzinie 14∶11 -0700, użytkownik Zac Medico
>> napisał:
>>> On 07/08/2018 01:18 PM, Zac Medico wrote:
>>>> On 07/08/2018 01:08 PM, Michał Górny wrote:
>>>>> W dniu nie, 08.07.2018 o godzinie 11∶57 -0700, użytkownik Zac
>Medico
>>>>> napisał:
>>>>>> On 07/08/2018 11:42 AM, Michał Górny wrote:
>>>>>>> W dniu nie, 08.07.2018 o godzinie 11∶04 -0700, użytkownik Zac
>Medico
>>>>>>> napisał:
>>>>>>>> On 07/08/2018 06:56 AM, Michał Górny wrote:
>>>>>>>>> W dniu nie, 08.07.2018 o godzinie 15∶02 +0200, użytkownik
>Kristian
>>>>>>>>> Fiskerstrand napisał:
>>>>>>>>>> On 07/08/2018 08:53 AM, Michał Górny wrote:
>>>>>>>>>>> Is safe git syncing implemented already? If not, maybe
>finish it first and cover both with a single news item. Git is going to
>be more efficient here, so people may want to learn they have an
>alternative.
>>>>>>>>>>
>>>>>>>>>> Why complicate things, and increase wait for something that
>benefits
>>>>>>>>>> most users, just to give alternatives to a few using
>non-default sync
>>>>>>>>>> mechanism. Securing git distribution is a whole different
>ballpark.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Let me rephrase.  Let's say I'm using rsync.  This new feature
>is
>>>>>>>>> something positive but it breaks my use case (for one of the
>listed
>>>>>>>>> reasons -- overlayfs, inode use, small fs cache).  After
>reading this
>>>>>>>>> news item, I learn that my only option is to disable the new
>feature.
>>>>>>>>>
>>>>>>>>> Now, I would appreciate being told that there's an alternate
>sync method
>>>>>>>>> that handles secure updates without having all those
>drawbacks.
>>>>>>>>
>>>>>>>> The thing is, the normal git tree doesn't even provide
>pre-generated
>>>>>>>> metadata, and I see then gentoo-mirror repo that provides
>metadata does
>>>>>>>> not have commits signed with an release key:
>>>>>>>>
>>>>>>>> https://github.com/gentoo-mirror/gentoo/commits/stable
>>>>>>>>
>>>>>>>> So I'm really not comfortable recommending git to anyone at
>this point.
>>>>>>>
>>>>>>> Wrong twice.
>>>>>>>
>>>>>>> Firstly, the canonical URL is:
>>>>>>>
>>>>>>>   https://anongit.gentoo.org/git/repo/sync/gentoo.git
>>>>>>>   (https://gitweb.gentoo.org/repo/sync/gentoo.git)
>>>>>>>
>>>>>>> Secondly, the merge commits (i.e. top commits that are verified
>>>>>>> by Portage) are signed by dedicated key that is part of the
>infra key
>>>>>>> set.  In other words, it works out of the box.
>>>>>>
>>>>>> Is there any documentation that shows users how to migrate to
>git, and
>>>>>> what the pros and cons might be? Maybe its worthy of its own news
>item.
>>>>>
>>>>> Maybe.  I don't really know, and don't think it's a good idea to
>show 30
>>>>> news item of things users might like on every new Gentoo install.
>>>>
>>>> Well if instructions for setting up git sync and associated
>pros/cons
>>>> are not documented anywhere then I won't advise anyone to use it.
>>>
>>> I've attempted to configure it for myself, and this is what it does:
>>>
>>>  * Using keys from /usr/share/openpgp-keys/gentoo-release.asc
>>>  * Refreshing keys from keyserver ...
>>>                                                        [ ok ]
>>>  * No valid signature found: unable to verify signature (missing
>key?)
>>>
>> 
>> Please report a bug and attach your configuration along with keyring
>> version.
>
>It works after upgrading to openpgp-keys-gentoo-release-20180706 from
>openpgp-keys-gentoo-release-20180323.
>-- 
>Thanks,
>Zac

Does Portage not call attention to critical updates?

It used to make a special statement for a new stable Portage and strongly recommended that it be emerged first. It should probably do the same for openpgp-keys-gentoo-release.
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[-- Attachment #1.2: Type: text/html, Size: 5403 bytes --]

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 281 bytes --]

Re: [gentoo-dev] News Item: Portage rsync hardlink support

[Zac Medico]

[-- Attachment #1.1: Type: text/plain, Size: 6553 bytes --]

On 07/08/2018 02:50 PM, Aaron W. Swenson wrote:
> On July 8, 2018 5:38:48 PM EDT, Zac Medico <zmedico@gentoo.org> wrote:
> 
>     On 07/08/2018 02:18 PM, Michał Górny wrote:
> 
>         W dniu nie, 08.07.2018 o godzinie 14∶11 -0700, użytkownik Zac Medico
>         napisał:
> 
>             On 07/08/2018 01:18 PM, Zac Medico wrote:
> 
>                 On 07/08/2018 01:08 PM, Michał Górny wrote:
> 
>                     W dniu nie, 08.07.2018 o godzinie 11∶57 -0700,
>                     użytkownik Zac Medico
>                     napisał:
> 
>                         On 07/08/2018 11:42 AM, Michał Górny wrote:
> 
>                             W dniu nie, 08.07.2018 o godzinie 11∶04 
>                             -0700, użytkownik Zac Medico
>                             napisał:
> 
>                                 On 07/08/2018 06:56 AM, Michał Górny wrote:
> 
>                                     W dniu nie, 08.07.2018 o godzinie
>                                     15∶02 +0200, użytkownik Kristian
>                                     Fiskerstrand napisał:
> 
>                                         On 07/08/2018 08:53 AM, Michał
>                                         Górny wrote:
> 
>                                             Is safe git syncing
>                                             implemented already? If not,
>                                             maybe finish it first and
>                                             cover both with a single
>                                             news item. Git is going to
>                                             be more efficient here, so
>                                             people may want to learn
>                                             they have an alternative.
> 
> 
>                                         Why complicate things, and
>                                         increase wait for something that
>                                         benefits
>                                         most users, just to give
>                                         alternatives to a few using
>                                         non-default sync
>                                         mechanism. Securing git
>                                         distribution is a whole
>                                         different ballpark.
> 
> 
> 
>                                     Let me rephrase. Let's say I'm using
>                                     rsync. This new feature is
>                                     something positive but it breaks my
>                                     use case (for one of the listed
>                                     reasons -- overlayfs, inode use,
>                                     small fs cache). After reading this
>                                     news item, I learn that my only
>                                     option is to disable the new feature.
> 
>                                     Now, I would appreciate being told
>                                     that there's an alternate sync method
>                                     that handles secure updates without
>                                     having all those drawbacks.
> 
> 
>                                 The thing is, the normal git tree
>                                 doesn't even provide pre-generated
>                                 metadata, and I see then gentoo-mirror
>                                 repo that provides metadata does
>                                 not have commits signed with an release key:
> 
>                                 https://github.com/gentoo-mirror/gentoo/commits/stable
> 
>                                 So I'm really not comfortable
>                                 recommending git to anyone at this point.
> 
> 
>                             Wrong twice.
> 
>                             Firstly, the canonical URL is:
> 
>                             https://anongit.gentoo.org/git/repo/sync/gentoo.git
>                             (https://gitweb.gentoo.org/repo/sync/gentoo.git)
> 
>                             Secondly, the merge commits (i.e. top
>                             commits that are verified
>                             by Portage) are signed by dedicated key that
>                             is part of the infra key
>                             set. In other words, it works out of the box.
> 
> 
>                         Is there any documentation that shows users how
>                         to migrate to git, and
>                         what the pros and cons might be? Maybe its
>                         worthy of its own news item.
> 
> 
>                     Maybe. I don't really know, and don't think it's a
>                     good idea to show 30
>                     news item of things users might like on every new
>                     Gentoo install.
> 
> 
>                 Well if instructions for setting up git sync and
>                 associated pros/cons
>                 are not documented anywhere then I won't advise anyone
>                 to use it.
> 
> 
>             I've attempted to configure it for myself, and this is what
>             it does:
> 
>             * Using keys from /usr/share/openpgp-keys/gentoo-release.asc
>             * Refreshing keys from keyserver ...
>             [ ok ]
>             * No valid signature found: unable to verify signature
>             (missing key?)
> 
> 
> 
>         Please report a bug and attach your configuration along with keyring
>         version.
> 
> 
>     It works after upgrading to openpgp-keys-gentoo-release-20180706 from
>     openpgp-keys-gentoo-release-20180323.
> 
> 
> Does Portage not call attention to critical updates?

No, but that might be a nice feature. We'd have to introduce some kind
of standard mechanism via PMS or a GLEP.

> It used to make a special statement for a new stable Portage and
> strongly recommended that it be emerged first. It should probably do the
> same for openpgp-keys-gentoo-release.

Sure, but it this case we have a chicken-and-egg problem, because I
needed the latest openpgp-keys-gentoo-release installed but in order to
do that I had to sync, but then verification failed because I didn't
have the latest openpgp-keys-gentoo-release.
-- 
Thanks,
Zac


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: [gentoo-dev] News Item: Portage rsync hardlink support

[Rich Freeman]

On Sun, Jul 8, 2018 at 5:50 PM Aaron W. Swenson <titanofold@gentoo.org> wrote:
>
> Does Portage not call attention to critical updates?
>
> It used to make a special statement for a new stable Portage and strongly recommended that it be emerged first. It should probably do the same for openpgp-keys-gentoo-release.

If that is only needed for syncing, then I'm not sure there is really
a need.  If you have the ebuild for the latest keys, then you can
install them in any order at any time.  If you don't have the ebuild
but need it, then portage won't know, and syncing will simply fail
unless overridden somehow.

-- 
Rich

[gentoo-dev] News Item: Portage rsync hardlink support [v2]

[Zac Medico]

[-- Attachment #1.1: Type: text/plain, Size: 1543 bytes --]

In v2 there's a suggestion to use git for better efficiency than rsync,
as requested by Michał Górny:

Title: Portage rsync hardlink support
Author: Zac Medico <zmedico@gentoo.org>
Posted: 2018-07-11
Revision: 1
News-Item-Format: 2.0
Display-If-Installed: sys-apps/portage

For users of the rsync tree, beginning with sys-apps/portage-2.3.42,
the default behavior for sync operations will use hardlinks in order
to ensure that a repository remains in a valid state if something
goes wrong [1]. For example, if signature verification fails during a
sync operation, the new hardlink behavior will preserve the previous
state of the repository.

The new behavior may conflict with configurations that restrict the
use of hardlinks, such as overlay filesystems. Therefore, users will
have to set "sync-allow-hardlinks = no" in repos.conf if they have
a configuration that restricts the use of hardlinks, but this should
not be very common:

[DEFAULT]
sync-allow-hardlinks = no

Note that it possible to sync more efficiently using git [2] instead
of rsync, though git consumes an increasing amount of disk space over
time because shallow pull is currently not supported [3].

[1] https://bugs.gentoo.org/660410 sys-apps/portage: use rsync
    --link-dest to implement atomic repository updates (and abort if
    signature verification fails)
[2] https://wiki.gentoo.org/wiki/Portage_Security#git-mirror_repo
[3] https://bugs.gentoo.org/552814 sync using git pull --depth=1 by
    default
-- 
Thanks,
Zac


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: [gentoo-dev] News Item: Portage rsync hardlink support

[Zac Medico]

[-- Attachment #1.1: Type: text/plain, Size: 6392 bytes --]

On 07/08/2018 02:59 PM, Zac Medico wrote:
> On 07/08/2018 02:50 PM, Aaron W. Swenson wrote:
>> On July 8, 2018 5:38:48 PM EDT, Zac Medico <zmedico@gentoo.org> wrote:
>>
>>     On 07/08/2018 02:18 PM, Michał Górny wrote:
>>
>>         W dniu nie, 08.07.2018 o godzinie 14∶11 -0700, użytkownik Zac Medico
>>         napisał:
>>
>>             On 07/08/2018 01:18 PM, Zac Medico wrote:
>>
>>                 On 07/08/2018 01:08 PM, Michał Górny wrote:
>>
>>                     W dniu nie, 08.07.2018 o godzinie 11∶57 -0700,
>>                     użytkownik Zac Medico
>>                     napisał:
>>
>>                         On 07/08/2018 11:42 AM, Michał Górny wrote:
>>
>>                             W dniu nie, 08.07.2018 o godzinie 11∶04 
>>                             -0700, użytkownik Zac Medico
>>                             napisał:
>>
>>                                 On 07/08/2018 06:56 AM, Michał Górny wrote:
>>
>>                                     W dniu nie, 08.07.2018 o godzinie
>>                                     15∶02 +0200, użytkownik Kristian
>>                                     Fiskerstrand napisał:
>>
>>                                         On 07/08/2018 08:53 AM, Michał
>>                                         Górny wrote:
>>
>>                                             Is safe git syncing
>>                                             implemented already? If not,
>>                                             maybe finish it first and
>>                                             cover both with a single
>>                                             news item. Git is going to
>>                                             be more efficient here, so
>>                                             people may want to learn
>>                                             they have an alternative.
>>
>>
>>                                         Why complicate things, and
>>                                         increase wait for something that
>>                                         benefits
>>                                         most users, just to give
>>                                         alternatives to a few using
>>                                         non-default sync
>>                                         mechanism. Securing git
>>                                         distribution is a whole
>>                                         different ballpark.
>>
>>
>>
>>                                     Let me rephrase. Let's say I'm using
>>                                     rsync. This new feature is
>>                                     something positive but it breaks my
>>                                     use case (for one of the listed
>>                                     reasons -- overlayfs, inode use,
>>                                     small fs cache). After reading this
>>                                     news item, I learn that my only
>>                                     option is to disable the new feature.
>>
>>                                     Now, I would appreciate being told
>>                                     that there's an alternate sync method
>>                                     that handles secure updates without
>>                                     having all those drawbacks.
>>
>>
>>                                 The thing is, the normal git tree
>>                                 doesn't even provide pre-generated
>>                                 metadata, and I see then gentoo-mirror
>>                                 repo that provides metadata does
>>                                 not have commits signed with an release key:
>>
>>                                 https://github.com/gentoo-mirror/gentoo/commits/stable
>>
>>                                 So I'm really not comfortable
>>                                 recommending git to anyone at this point.
>>
>>
>>                             Wrong twice.
>>
>>                             Firstly, the canonical URL is:
>>
>>                             https://anongit.gentoo.org/git/repo/sync/gentoo.git
>>                             (https://gitweb.gentoo.org/repo/sync/gentoo.git)
>>
>>                             Secondly, the merge commits (i.e. top
>>                             commits that are verified
>>                             by Portage) are signed by dedicated key that
>>                             is part of the infra key
>>                             set. In other words, it works out of the box.
>>
>>
>>                         Is there any documentation that shows users how
>>                         to migrate to git, and
>>                         what the pros and cons might be? Maybe its
>>                         worthy of its own news item.
>>
>>
>>                     Maybe. I don't really know, and don't think it's a
>>                     good idea to show 30
>>                     news item of things users might like on every new
>>                     Gentoo install.
>>
>>
>>                 Well if instructions for setting up git sync and
>>                 associated pros/cons
>>                 are not documented anywhere then I won't advise anyone
>>                 to use it.
>>
>>
>>             I've attempted to configure it for myself, and this is what
>>             it does:
>>
>>             * Using keys from /usr/share/openpgp-keys/gentoo-release.asc
>>             * Refreshing keys from keyserver ...
>>             [ ok ]
>>             * No valid signature found: unable to verify signature
>>             (missing key?)
>>
>>
>>
>>         Please report a bug and attach your configuration along with keyring
>>         version.
>>
>>
>>     It works after upgrading to openpgp-keys-gentoo-release-20180706 from
>>     openpgp-keys-gentoo-release-20180323.
>>
>>
>> Does Portage not call attention to critical updates?
> 
> No, but that might be a nice feature. We'd have to introduce some kind
> of standard mechanism via PMS or a GLEP.

Actually GLEP 42 news items can be used for this, with a header like:

Display-If-Installed: <app-crypt/openpgp-keys-gentoo-release-20180706
-- 
Thanks,
Zac


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: [gentoo-dev] News Item: Portage rsync hardlink support

[Kristian Fiskerstrand]

On 07/08/2018 11:59 PM, Zac Medico wrote:
>> It used to make a special statement for a new stable Portage and
>> strongly recommended that it be emerged first. It should probably do the
>> same for openpgp-keys-gentoo-release.
> Sure, but it this case we have a chicken-and-egg problem, because I
> needed the latest openpgp-keys-gentoo-release installed but in order to
> do that I had to sync, but then verification failed because I didn't
> have the latest openpgp-keys-gentoo-release.

We're hopefully attributing that to a startup problem. Obviously as we
go along we need to have proper key rollover procedures so this should
never happen including backup keys.

-- 
Kristian Fiskerstrand
OpenPGP keyblock reachable at hkp://pool.sks-keyservers.net
fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3