[gentoo-dev] News item review: SquashDelta syncing support

[gentoo-dev] News item review: SquashDelta syncing support

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1759 bytes --]

Title: SquashDelta syncing support
Author: Michał Górny <mgorny@gentoo.org>
Content-Type: text/plain
Posted: 2015-05-xx
Revision: 1
News-Item-Format: 1.0
Display-If-Installed: >=sys-apps/portage-2.2.19

Starting with Portage 2.2.19, a new SquashDelta syncing method has been
introduced. It is meant to provide lightweight and efficient solution
for stable systems. The whole repository is contained within a single
pre-generated SquashFS image file. The daily snapshot of the repository
is first fetched from the mirrors, and afterwards updated in-place using
deltas (without repacking).

In order to enable SquashDelta syncing, please install
dev-util/squashmerge utility first:

  $ emerge -v dev-util/squashmerge

Afterwards, you can use a repos.conf entry similar to the following:

  [gentoo]
  location = /var/db/repos/gentoo
  sync-type = squashdelta
  sync-uri = mirror://gentoo/../snapshots/squashfs

During the initial sync, Portage will fetch the current Gentoo SquashFS
snapshot from the mirrors (~105M). For the synces following, it will
only fetch a single delta, and use squashmerge to quickly update
the local copy.

If possible, Portage will automatically mount (or remount) the SquashFS
after syncing. However, you may want to add an explicit /etc/fstab entry
for the filesystem to make it available before invoking 'emerge --sync':

  /var/cache/portage/squashfs/gentoo-current.sqfs /var/db/repos/gentoo \
    squashfs defaults 0 0

Please note that the current syncing code does not verify the OpenPGP
signature to confirm the authenticity of fetched snapshots and deltas.
This feature will be added as soon as gentoo-keys support in Portage is
available.

-- 
Best regards,
Michał Górny

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] News item review: SquashDelta syncing support

[Dirkjan Ochtman]

On Fri, May 15, 2015 at 7:51 AM, Michał Górny <mgorny@gentoo.org> wrote:
> Starting with Portage 2.2.19, a new SquashDelta syncing method has been
> introduced. It is meant to provide lightweight and efficient solution
> for stable systems. The whole repository is contained within a single
> pre-generated SquashFS image file. The daily snapshot of the repository
> is first fetched from the mirrors, and afterwards updated in-place using
> deltas (without repacking).

This sounds nice, but the news item currently leaves me wondering what
sort of improvements I should expect. It says the new method is
"lightweight and efficient", but it would be nice to quantify this a
little bit, or add a link to a page with more details. I think the
default sync method in the handbook up to now has always been rsync? A
comparison (both in terms of upside and in terms of downside) would be
nice. Also, whether we want to make this the new default at some
point, and if so, when.

Cheers,

Dirkjan

Re: [gentoo-dev] News item review: SquashDelta syncing support

[Diamond]

On Fri, 15 May 2015 08:23:27 -0700
Dirkjan Ochtman <djc@gentoo.org> wrote:

> On Fri, May 15, 2015 at 7:51 AM, Michał Górny <mgorny@gentoo.org>
> wrote:
> > Starting with Portage 2.2.19, a new SquashDelta syncing method has
> > been introduced. It is meant to provide lightweight and efficient
> > solution for stable systems. The whole repository is contained
> > within a single pre-generated SquashFS image file. The daily
> > snapshot of the repository is first fetched from the mirrors, and
> > afterwards updated in-place using deltas (without repacking).
> 
> This sounds nice, but the news item currently leaves me wondering what
> sort of improvements I should expect. It says the new method is
> "lightweight and efficient", but it would be nice to quantify this a
> little bit, or add a link to a page with more details. I think the
> default sync method in the handbook up to now has always been rsync? A
> comparison (both in terms of upside and in terms of downside) would be
> nice. Also, whether we want to make this the new default at some
> point, and if so, when.
> 
> Cheers,
> 
> Dirkjan
> 
> 

I've read the pdf article of Michał Górny and from my expirience with
emerge-delta-webrsync and app-portage/getdelta in the past this good old
new feature looks mostly useful for bad Internet connections (too slow
or too expensive ones) and looks mostly useless for syncing
relative to rsync method from local mirror like I use
http://mirror.yandex.ru/gentoo-distfiles/
 from my local region.
eix-sync gave me the following statistics (before introducing new
portage sync with repos.conf wich has stopped upgrade in the middle atm
because >=app-portage/layman-2.3.0 haven't been stabilised yet):
 * Time statistics:
    19 seconds for syncing
    17 seconds for eix-update
     1 seconds for eix-diff
    51 seconds total
or this one the other day:
* Time statistics:
    37 seconds for syncing
    11 seconds for eix-update
     1 seconds for eix-diff
    67 seconds total
So it takes usually 15-40 seconds for syncing using usual rsync method.
This deltas have their own drawbacks like "delta is under generation,
please wait half an hour or even more" or "your state is not the same
what was while generating delta on the host and lets do additional work
with more deltas". ))

Although, nice try with experimenting and trying to improve sync
mechanism. )

Re: [gentoo-dev] News item review: SquashDelta syncing support

[Ian Stakenvicius]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 15/05/15 12:15 PM, Diamond wrote:
> On Fri, 15 May 2015 08:23:27 -0700 Dirkjan Ochtman <djc@gentoo.org>
> wrote:
> 
>> On Fri, May 15, 2015 at 7:51 AM, Michał Górny
>> <mgorny@gentoo.org> wrote:
>>> Starting with Portage 2.2.19, a new SquashDelta syncing method
>>> has been introduced. It is meant to provide lightweight and
>>> efficient solution for stable systems. The whole repository is
>>> contained within a single pre-generated SquashFS image file.
>>> The daily snapshot of the repository is first fetched from the
>>> mirrors, and afterwards updated in-place using deltas (without
>>> repacking).
>> 
>> This sounds nice, but the news item currently leaves me wondering
>> what sort of improvements I should expect. [...]
> 
> I've read the pdf article of Michał Górny and from my expirience
> with emerge-delta-webrsync and app-portage/getdelta in the past
> this good old new feature looks mostly useful for bad Internet
> connections (too slow or too expensive ones) and looks mostly
> useless for syncing relative to rsync method from local mirror like
> I use [...]

Although this thread should be a review of the news item rather than a
review of the feature, I think both of these guys have a point.

The main benefit to this new feature is that it allows users to use a
squashfs image for their gentoo repo (portage tree) without having to
(re)generate it themselves locally every time they --sync, AND without
having to re-download an entire image from gentoo mirrors each time
either.

The new item doesn't really cover this much -- that the feature is for
supporting storage and synchronization of the gentoo repo on squashfs
rather than on a regular filesystem.  Perhaps it would be enough to
link to an article describing the benefits of using a squashfs'ed
portage tree, so users could chose whether they want this or not based
on that?  Similarly, it would probably be good to mention that this
new feature deprecates squash_portage and the other tools/methods out
there for doing the same thing locally.


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iF4EAREIAAYFAlVWO9kACgkQ2ugaI38ACPA1vwD9ELIdOgSSTfly3rT5zU6dzhGb
62LtN8loiRFhKfyAe/8A/24xw95j7qav/himVRA5OOjM3qTE++iBY/2yXPgLWpI5
=1+2r
-----END PGP SIGNATURE-----

Re: [gentoo-dev] News item review: SquashDelta syncing support

[Rich Freeman]

On Fri, May 15, 2015 at 2:32 PM, Ian Stakenvicius <axs@gentoo.org> wrote:
>
> The new item doesn't really cover this much -- that the feature is for
> supporting storage and synchronization of the gentoo repo on squashfs
> rather than on a regular filesystem.  Perhaps it would be enough to
> link to an article describing the benefits of using a squashfs'ed
> portage tree, so users could chose whether they want this or not based
> on that?  Similarly, it would probably be good to mention that this
> new feature deprecates squash_portage and the other tools/methods out
> there for doing the same thing locally.
>

That makes sense to me.  Some of the likely benefits would be:

1.  Less disk space use.
2.  Vastly less inode use.
3.  Much less CPU/IO to update.
4.  I suspect much less fragmentation/write/etc for storage on flash.
Then again, on filesystems like btrfs fragmentation might be worse due
to all the internal writes.
5.  Probably better read performance (less disk IO, more CPU).

Downsides include:
1.  No way to sync more frequently than whatever the update cycle is.
It would be more like emerge-webrsync and less like emerge --sync.
2.  Impossible to tweak ebuilds without setting up an overlay.  This
might be annoying for devs/etc.

-- 
Rich

Re: [gentoo-dev] News item review: SquashDelta syncing support

[Ian Stakenvicius]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 15/05/15 03:33 PM, Rich Freeman wrote:
> On Fri, May 15, 2015 at 2:32 PM, Ian Stakenvicius <axs@gentoo.org>
> wrote:
>> 
>> The new item doesn't really cover this much -- that the feature
>> is for supporting storage and synchronization of the gentoo repo
>> on squashfs rather than on a regular filesystem.  Perhaps it
>> would be enough to link to an article describing the benefits of
>> using a squashfs'ed portage tree, so users could chose whether
>> they want this or not based on that?  Similarly, it would
>> probably be good to mention that this new feature deprecates
>> squash_portage and the other tools/methods out there for doing
>> the same thing locally.
>> 
> 
> That makes sense to me.  Some of the likely benefits would be:
> 
> 1.  Less disk space use. 2.  Vastly less inode use. 3.  Much less
> CPU/IO to update. 4.  I suspect much less fragmentation/write/etc
> for storage on flash. Then again, on filesystems like btrfs
> fragmentation might be worse due to all the internal writes. 5.
> Probably better read performance (less disk IO, more CPU).
> 
> Downsides include: 1.  No way to sync more frequently than whatever
> the update cycle is. It would be more like emerge-webrsync and less
> like emerge --sync. 2.  Impossible to tweak ebuilds without setting
> up an overlay.  This might be annoying for devs/etc.
> 


Given the importance of this is to me more about the squashfs storage
than the sync method, it may even be pertinent to change the title of
the news item to something like:  "SquashFS repo, SquashDelta syncing
support"


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iF4EAREIAAYFAlVWSygACgkQ2ugaI38ACPAXdgEApXmmfrFJB1b4L0B4hKnNAuLs
Njl9rWczgmR4SjMgvBwA/AwIOujrtoiQd1iT4j9oqQAjYJ9S8O/vVJe/9yWJXpj/
=WI2a
-----END PGP SIGNATURE-----

Re: [gentoo-dev] News item review: SquashDelta syncing support

[Michael Orlitzky]

On 05/15/2015 10:51 AM, Michał Górny wrote:
> Title: SquashDelta syncing support

Do we need a news item for this at all? Everything is backward
compatible and most people don't need to do anything at all in response
to the news.

Re: [gentoo-dev] News item review: SquashDelta syncing support

[Alexis Ballier]

On Sat, 16 May 2015 10:59:20 -0400
Michael Orlitzky <mjo@gentoo.org> wrote:

> On 05/15/2015 10:51 AM, Michał Górny wrote:
> > Title: SquashDelta syncing support
> 
> Do we need a news item for this at all? Everything is backward
> compatible and most people don't need to do anything at all in
> response to the news.

I tend to agree here; while the feature looks cool, I'm not sure news
items purpose is to announce new features.

Re: [gentoo-dev] News item review: SquashDelta syncing support

[Pacho Ramos]

El sáb, 16-05-2015 a las 20:38 +0200, Alexis Ballier escribió:
> On Sat, 16 May 2015 10:59:20 -0400
> Michael Orlitzky <mjo@gentoo.org> wrote:
> 
> > On 05/15/2015 10:51 AM, Michał Górny wrote:
> > > Title: SquashDelta syncing support
> > 
> > Do we need a news item for this at all? Everything is backward
> > compatible and most people don't need to do anything at all in
> > response to the news.
> 
> I tend to agree here; while the feature looks cool, I'm not sure news
> items purpose is to announce new features.
> 

Maybe this could be announced in planet.gentoo.org or even main
gentoo.org news ;)

Re: [gentoo-dev] News item review: SquashDelta syncing support

[Michael Orlitzky]

On 05/16/2015 03:18 PM, Pacho Ramos wrote:
> 
> Maybe this could be announced in planet.gentoo.org or even main
> gentoo.org news ;)
> 
> 

We have gentoo-announce@g.o and gentoo-user@g.o too!

Re: [gentoo-dev] News item review: SquashDelta syncing support

[Brian Dolbec]

On Sat, 16 May 2015 15:23:24 -0400
Michael Orlitzky <mjo@gentoo.org> wrote:

> On 05/16/2015 03:18 PM, Pacho Ramos wrote:
> > 
> > Maybe this could be announced in planet.gentoo.org or even main
> > gentoo.org news ;)
> > 
> > 
> 
> We have gentoo-announce@g.o and gentoo-user@g.o too!
> 
> 

But it will have to wait.  There is a few bugs in the commit made to
portage for the release.  It doesn't run.  It is fixed in portage-9999
though.

We'll see if we can get it gpg verifying with gkeys for another release.

-- 
Brian Dolbec <dolsen>

Re: [gentoo-dev] News item review: SquashDelta syncing support

[Alon Bar-Lev]

On 15 May 2015 at 17:51, Michał Górny <mgorny@gentoo.org> wrote:
> Please note that the current syncing code does not verify the OpenPGP
> signature to confirm the authenticity of fetched snapshots and deltas.
> This feature will be added as soon as gentoo-keys support in Portage is
> available.

These are great news!
We can retire the webrsync.
Why not sign it similar to the portage snapshot are signed for now?
The webrsync signature validation is quite simple.

Just a reminder: please note the rollback prevention mechanism in
webrsync, it is not enough to check signature, but also prevent older
snapshot to be used.

Regards,
Alon

Re: [gentoo-dev] News item review: SquashDelta syncing support

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1237 bytes --]

Dnia 2015-05-15, o godz. 08:23:27
Dirkjan Ochtman <djc@gentoo.org> napisał(a):

> On Fri, May 15, 2015 at 7:51 AM, Michał Górny <mgorny@gentoo.org> wrote:
> > Starting with Portage 2.2.19, a new SquashDelta syncing method has been
> > introduced. It is meant to provide lightweight and efficient solution
> > for stable systems. The whole repository is contained within a single
> > pre-generated SquashFS image file. The daily snapshot of the repository
> > is first fetched from the mirrors, and afterwards updated in-place using
> > deltas (without repacking).
> 
> This sounds nice, but the news item currently leaves me wondering what
> sort of improvements I should expect. It says the new method is
> "lightweight and efficient", but it would be nice to quantify this a
> little bit, or add a link to a page with more details. I think the
> default sync method in the handbook up to now has always been rsync? A
> comparison (both in terms of upside and in terms of downside) would be
> nice. Also, whether we want to make this the new default at some
> point, and if so, when.

I'll rewrite the beginning to be more clear on advantages when I have
some time for it.

-- 
Best regards,
Michał Górny

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] News item review: SquashDelta syncing support

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 602 bytes --]

Dnia 2015-05-16, o godz. 10:59:20
Michael Orlitzky <mjo@gentoo.org> napisał(a):

> On 05/15/2015 10:51 AM, Michał Górny wrote:
> > Title: SquashDelta syncing support
> 
> Do we need a news item for this at all? Everything is backward
> compatible and most people don't need to do anything at all in response
> to the news.

No, we don't. But news items are good way to tell people that something
is happening. Look at Funtoo. Their users are happy because developers
announce stuff rather than expecting them to watch for random things
happening.

-- 
Best regards,
Michał Górny

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] News item review: SquashDelta syncing support

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 444 bytes --]

Dnia 2015-05-16, o godz. 15:23:24
Michael Orlitzky <mjo@gentoo.org> napisał(a):

> On 05/16/2015 03:18 PM, Pacho Ramos wrote:
> > 
> > Maybe this could be announced in planet.gentoo.org or even main
> > gentoo.org news ;)
> > 
> > 
> 
> We have gentoo-announce@g.o and gentoo-user@g.o too!

That's gentoo-dev-announce. 'dev' is the key part. And gentoo-user@ is
doubtedly used by sysadmins.

-- 
Best regards,
Michał Górny

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] News item review: SquashDelta syncing support

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1168 bytes --]

Dnia 2015-05-16, o godz. 23:48:01
Alon Bar-Lev <alonbl@gentoo.org> napisał(a):

> On 15 May 2015 at 17:51, Michał Górny <mgorny@gentoo.org> wrote:
> > Please note that the current syncing code does not verify the OpenPGP
> > signature to confirm the authenticity of fetched snapshots and deltas.
> > This feature will be added as soon as gentoo-keys support in Portage is
> > available.
> 
> These are great news!
> We can retire the webrsync.
> Why not sign it similar to the portage snapshot are signed for now?
> The webrsync signature validation is quite simple.

All signing is in place already for a long time. Just the verification
code is missing, and it wasn't added because I was told to wait for
gentoo-keys.

> Just a reminder: please note the rollback prevention mechanism in
> webrsync, it is not enough to check signature, but also prevent older
> snapshot to be used.

Truth be told, the squashdelta syncing wasn't really made with rollback
prevention in mind. I can't think immediately of any solution that
would prevent accidental rollback while preserving the intended
flexibility.

-- 
Best regards,
Michał Górny

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] News item review: SquashDelta syncing support

[Michael Orlitzky]

On 05/16/2015 06:01 PM, Michał Górny wrote:
>>
>> We have gentoo-announce@g.o and gentoo-user@g.o too!
> 
> That's gentoo-dev-announce. 'dev' is the key part. And gentoo-user@ is
> doubtedly used by sysadmins.
> 

This one:

https://archives.gentoo.org/gentoo-announce/

Re: [gentoo-dev] News item review: SquashDelta syncing support

[Michael Orlitzky]

On 05/16/2015 06:00 PM, Michał Górny wrote:
>>
>> Do we need a news item for this at all? Everything is backward
>> compatible and most people don't need to do anything at all in response
>> to the news.
> 
> No, we don't. But news items are good way to tell people that something
> is happening. Look at Funtoo. Their users are happy because developers
> announce stuff rather than expecting them to watch for random things
> happening.
> 

Ok, just keep in mind that you force thousands of people to type
`eselect news read new` on all of their machines when you post a feature
announcement.

Re: [gentoo-dev] News item review: SquashDelta syncing support

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 535 bytes --]

Dnia 2015-05-16, o godz. 20:38:36
Michael Orlitzky <mjo@gentoo.org> napisał(a):

> On 05/16/2015 06:01 PM, Michał Górny wrote:
> >>
> >> We have gentoo-announce@g.o and gentoo-user@g.o too!
> > 
> > That's gentoo-dev-announce. 'dev' is the key part. And gentoo-user@ is
> > doubtedly used by sysadmins.
> > 
> 
> This one:
> 
> https://archives.gentoo.org/gentoo-announce/

First time I hear about it. Looks to be used primarily for GLSAs.
I wonder if anyone actually uses it.

-- 
Best regards,
Michał Górny

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

[gentoo-dev] Re: News item review: SquashDelta syncing support

[Duncan]

Michael Orlitzky posted on Sat, 16 May 2015 20:45:08 -0400 as excerpted:

> On 05/16/2015 06:00 PM, Michał Górny wrote:
>>>
>>> Do we need a news item for this at all? Everything is backward
>>> compatible and most people don't need to do anything at all in
>>> response to the news.
>> 
>> No, we don't. But news items are good way to tell people that something
>> is happening. Look at Funtoo. Their users are happy because developers
>> announce stuff rather than expecting them to watch for random things
>> happening.
>> 
>> 
> Ok, just keep in mind that you force thousands of people to type
> `eselect news read new` on all of their machines when you post a feature
> announcement.

That's a good point, and a rather persuasive argument to include the "the 
benefits/down-sides are" sections already suggested elsewhere. =:^)

IOW, without a benefits/negatives section, you are absolutely correct, 
there's no obvious reason to post this as a news item as it simply forces 
a bunch of people to read news and wonder why they should care.

But with that benefits/negatives section added, it should become a useful 
news item, that even people who don't choose to switch to squashdelta 
should appreciate, as it keeps them informed of changes and allows them 
to make informed decisions based on them. =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

[gentoo-dev] Re: News item review: SquashDelta syncing support

[Duncan]

Michał Górny posted on Sun, 17 May 2015 05:47:15 +0200 as excerpted:

> Dnia 2015-05-16, o godz. 20:38:36 Michael Orlitzky <mjo@gentoo.org>
> napisał(a):
> 
>> On 05/16/2015 06:01 PM, Michał Górny wrote:
>> >>
>> >> We have gentoo-announce@g.o and gentoo-user@g.o too!
>> > 
>> > That's gentoo-dev-announce. 'dev' is the key part. And gentoo-user@
>> > is doubtedly used by sysadmins.
>> > 
>> > 
>> This one:
>> 
>> https://archives.gentoo.org/gentoo-announce/
> 
> First time I hear about it. Looks to be used primarily for GLSAs.
> I wonder if anyone actually uses it.

Answering the "does anyone actually use it" bit...

I'm subscribed (via gmane) and scan new message subjects, at least, 
reading messages when they (might) pertain.

Tho the GLSAs are of questionable usefulness here, since I'm on ~arch and 
update regularly enough that announced affected versions are usually 
ancient history on my boxen.  I think I've actually seen one or two that 
applied to me... in over a decade on gentoo.

But I still subscribe and at least scan titles, as for me it's part of 
being a good admin, just in case, both for the GLSAs, and in case there's 
anything else of importance posted.

Of course if the announce list was actually used for announcements other 
than GLSAs, it'd definitely enhance its general usefulness.  So please do 
start announcing stuff there -- IMO, anything fit for the front page is 
fit for the general announce list as well! =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] News item review: SquashDelta syncing support

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 546 bytes --]

On Sun, 17 May 2015 00:00:22 +0200
Michał Górny <mgorny@gentoo.org> wrote:
> No, we don't. But news items are good way to tell people that
> something is happening. Look at Funtoo. Their users are happy because
> developers announce stuff rather than expecting them to watch for
> random things happening.

When we designed news items, we deliberately said they weren't for
doing this kind of thing. News items are for high impact changes that
require user action. Use the regular PR channels for showing off.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-dev] News item review: SquashDelta syncing support

[Dale]

Michał Górny wrote:
> Dnia 2015-05-16, o godz. 20:38:36
> Michael Orlitzky <mjo@gentoo.org> napisał(a):
>
>> On 05/16/2015 06:01 PM, Michał Górny wrote:
>>>> We have gentoo-announce@g.o and gentoo-user@g.o too!
>>> That's gentoo-dev-announce. 'dev' is the key part. And gentoo-user@ is
>>> doubtedly used by sysadmins.
>>>
>> This one:
>>
>> https://archives.gentoo.org/gentoo-announce/
> First time I hear about it. Looks to be used primarily for GLSAs.
> I wonder if anyone actually uses it.
>



I subscribe to it and wish more things, new features for example, would
be posted there.  Sometimes portage has a new feature and unless one
reads the man page, one has no clue.  Of course, my wish is not limited
to portage's new features.  Other new Gentoo things would be nice too.

Dale

:-)  :-)

Re: [gentoo-dev] News item review: SquashDelta syncing support

[Michael Orlitzky]

On 05/16/2015 11:47 PM, Michał Górny wrote:
>>
>> This one:
>>
>> https://archives.gentoo.org/gentoo-announce/
> 
> First time I hear about it. Looks to be used primarily for GLSAs.
> I wonder if anyone actually uses it.
> 

For what it's worth, it's one of the three lists mentioned specifically
on our new "mailing lists" page:

  https://www.gentoo.org/get-involved/mailing-lists/

[gentoo-dev] Re: News item review: SquashDelta syncing support

[Martin Vaeth]

Rich Freeman <rich0@gentoo.org> wrote:
> Downsides include:
> 2.  Impossible to tweak ebuilds without setting up an overlay.  This
> might be annoying for devs/etc.

It is still possible to setup a read-writable portage tree
(using overlayfs/aufs/unionfs-fuse/... e.g. using the squashmount
tool from the mv overlay).

However, currently this does not play nicely with squashdelta:
You have to "undo" the mounting of squashdelta and have to use
different command (e.g. squashmount) afterwards.
Although this can probably be done e.g. in eix-sync with hooks,
I hope that in the near future there will be a possibility to
combine these methods more conveniently.

Currently, I made only some remarks in comment #3 of bug 549716,
because it seems that the sync module mechanism is currently
lacking the infrastructure for adding custom data (like hooks) to
a module.

Re: [gentoo-dev] Re: News item review: SquashDelta syncing support

[Brian Dolbec]

On Mon, 18 May 2015 05:48:59 +0000 (UTC)
Martin Vaeth <martin@mvath.de> wrote:

> Rich Freeman <rich0@gentoo.org> wrote:
> > Downsides include:
> > 2.  Impossible to tweak ebuilds without setting up an overlay.  This
> > might be annoying for devs/etc.
> 
> It is still possible to setup a read-writable portage tree
> (using overlayfs/aufs/unionfs-fuse/... e.g. using the squashmount
> tool from the mv overlay).
> 
> However, currently this does not play nicely with squashdelta:
> You have to "undo" the mounting of squashdelta and have to use
> different command (e.g. squashmount) afterwards.
> Although this can probably be done e.g. in eix-sync with hooks,
> I hope that in the near future there will be a possibility to
> combine these methods more conveniently.
> 
> Currently, I made only some remarks in comment #3 of bug 549716,
> because it seems that the sync module mechanism is currently
> lacking the infrastructure for adding custom data (like hooks) to
> a module.
> 
> 

No, that is not correct, the new sync system also introduced a new
native portage postsync hook system.

1) /etc/portage/postsync.d/...  runs each script once at the completion
of all repos that were synced. (no arguments passed in)

2) /etc/portage/repo.postsync.d  runs each script at the end of each
repo that is synced.  Each script can determine if it needs to run or
simply exit.  Each script is called with 3 arguments  the repo name,
the url link it was synced to and the path to the repo.

Using that information you could easily add a postsync hook there
looking for the gentoo repo argument. And if not exit.  If it is the
gentoo repo, call your squashmount script to remount it your way.

See the example script in that directory.

-- 
Brian Dolbec <dolsen>

[gentoo-dev] Re: News item review: SquashDelta syncing support

[Martin Vaeth]

Brian Dolbec <dolsen@gentoo.org> wrote:
> Martin Vaeth <martin@mvath.de> wrote:
>>
>> However, currently this does not play nicely with squashdelta:
>> You have to "undo" the mounting of squashdelta and have to use
>> different command (e.g. squashmount) afterwards.
>
> No, that is not correct [...]
> 2) /etc/portage/repo.postsync.d

I know about this hook, but this is not what I meant.

What I meant is the possibility to *replace* the automatic
mounting of portage by a different command (for instance,
a possibility to *avoid* that portage mounts/umounts
automatically but expects this to happens in this hook).

I give reasons for this below.

(This discussion belongs actually to portage-devel mailing
list or to some bug, but now I feel the necessity to clarify
the misunderstanding.)

It is not only inefficient and hackish (with possible problems
in case of unexpected situations like a SIGINT or other signal
at a "bad" time) if two programs/scripts "fight" about mounts
and undo each others' mounts. It also causes severe difficulties
in connection with overlayfs/aufs/...:

With these filesystems, you must create two (in case of
overlayfs even three) auxiliary directories, and in this
situation, it might be natural to choose these as
temporary direcories (e.g. generated in /tmp with unique names);
to understand the following explanation note also that two
of these directories (the squash filesystem and the overlay
filesystem) should "normally" always be mounted/umounted
together.

Now if portage umounts only the "overlay" directory,
the information about the temporary directory names is lost
(leaving possibly quite a bit data in /tmp undeleted
forever), and the second necessary "umount" does not
happen and is hard to do later on: It prevents the
space of the old squash file from actually being freed
(and this mount is hard to find later on: It is a mount
of a deleted file on an unknown temporary directory.
Of course, one could try to find this mount by some heuristic,
but this is extremely hackish, and the heuristic might find
some other squash file which the user has mounted similarly
for some other reason.)

In case of the mentioned "squashmount" tool, the situation
is better, because "squashmount" is rather complex and e.g.
stores/manages the names of temporary directories independently.
However, users might also want to use less sophisticated tools
than "squashmount" (and also "squashmount" has no easy solution
for "random" remounts of the mount points it manages, because
it is almost impossible to write a "generic" such solution.)

In any case, it is rather hackisch to write a lot of additional
code to "undo" an actually undesired umounting+mounting:
The "clean" solution appears to be to not do the "undesired"
(in this situation) umounting+mounting in the first place but
to execute *only* the actually desired (u)mount command(s)
instead.

Re: [gentoo-dev] News item review: SquashDelta syncing support

[Marc Schiffbauer]

[-- Attachment #1: Type: text/plain, Size: 698 bytes --]

* Michał Górny schrieb am 17.05.15 um 05:47 Uhr:
>Dnia 2015-05-16, o godz. 20:38:36
>Michael Orlitzky <mjo@gentoo.org> napisał(a):
>
>> On 05/16/2015 06:01 PM, Michał Górny wrote:
>> >>
>> >> We have gentoo-announce@g.o and gentoo-user@g.o too!
>> >
>> > That's gentoo-dev-announce. 'dev' is the key part. And gentoo-user@ is
>> > doubtedly used by sysadmins.
>> >
>>
>> This one:
>>
>> https://archives.gentoo.org/gentoo-announce/
>
>First time I hear about it. Looks to be used primarily for GLSAs.
>I wonder if anyone actually uses it.

We use it to watch GLSA announcements.

-Marc

-- 
0x35A64134 - 8AAC 5F46 83B4 DB70 8317
             3723 296C 6CCA 35A6 4134

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 173 bytes --]