[gentoo-dev] RFC: extension of etc-update

[gentoo-dev] RFC: extension of etc-update

[Joerg Hoh]

[-- Attachment #1: Type: text/plain, Size: 2032 bytes --]


Hi

I'm unhappy with the etc-update script. In my eyes it lacks at least one
necessary feature:

Detection of unmodified config-files

When I recently updated my sytem, it gave me more than 200 files to look
at. That is impossible, since in most cases I never heard of such a file
(and therefor I don't undestand what that file means). So I did a quick
look over the list, found nothing relevant to me and and did "-5". Well,
after that I wonderd why my mails were rejected. Oh, I've overseen
/etc/ssmtp.conf! Bad thing, with the default config it doesn't work.

My proposal is that we keep a list of configfiles and their md5 hashes.
etc-update maintains this list. We now have several cases:

* etc-update detects that we emerged a package which wasn't present and we
  have new config files. So etc-update does the md5-hashing and puts the
  hash into the hash-file.
* etc-update want's to install a new config-file and an 
  config-file is already present. Then it hashes the present config-file
  and compares the hash to the hash already present in the hash-file.
  - If the hashes differs, it asks the user what to do (as it now happens
    with every config-file). If the user wants to replace the old,
    user-modified config-file with the new one, the hash for the new
    config-file is written to the hash-file.
  - If the hashes are identical, it replaces the old config-file with the
    new one and updates the hash-file.

I've looked at the etc-update script, but since I'm not good at
shell-scripting, I decided to rewrite it in python. Not all is done yet,
and I don't wanto to do unneeded work if someone wants to enhance the
shell-script.

You find my python version at http://www.devone.org/linux/gentoo.html

Another extension would be versioning of configuration files via cvs/rcs,
subversion or arch. But I had to do some more work on that ...

Joerg, mostly happy gentoo user


-- 
Fachbegriffe der Informatik (Nr 152): PORN
- Poster Ohne Richtigen Namen 


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] RFC: extension of etc-update

[Guy Martin]

[-- Attachment #1: Type: text/plain, Size: 2852 bytes --]

Hi,

I've wrote a similar thing.
Btw it's more a quick hack than a complete end-user solution but it
works fine.

http://dev.gentoo.org/~gmsoft/tools/gm-etc-update

To use it you _MUST_ first have a clean system and run it once.
That means no file have to be updated via etc-update when you run it for
the first time because it have to build the MD5 database.

Once this is done, simply run it but beware :
-It will update unmodified files without asking
-It will update "trivial updates" withour asking
-It may not be bugless but I use it on ~10 box here and it always worked
as expected

Enjoy :)


On Tue, 10 Aug 2004 12:43:26 +0200
"Joerg Hoh" <joerg@devone.org> wrote:

> 
> Hi
> 
> I'm unhappy with the etc-update script. In my eyes it lacks at least
> one necessary feature:
> 
> Detection of unmodified config-files
> 
> When I recently updated my sytem, it gave me more than 200 files to
> look at. That is impossible, since in most cases I never heard of such
> a file(and therefor I don't undestand what that file means). So I did
> a quick look over the list, found nothing relevant to me and and did
> "-5". Well, after that I wonderd why my mails were rejected. Oh, I've
> overseen/etc/ssmtp.conf! Bad thing, with the default config it doesn't
> work.
> 
> My proposal is that we keep a list of configfiles and their md5
> hashes. etc-update maintains this list. We now have several cases:
> 
> * etc-update detects that we emerged a package which wasn't present
> and we
>   have new config files. So etc-update does the md5-hashing and puts
>   the hash into the hash-file.
> * etc-update want's to install a new config-file and an 
>   config-file is already present. Then it hashes the present
>   config-file and compares the hash to the hash already present in the
>   hash-file.- If the hashes differs, it asks the user what to do (as
>   it now happens
>     with every config-file). If the user wants to replace the old,
>     user-modified config-file with the new one, the hash for the new
>     config-file is written to the hash-file.
>   - If the hashes are identical, it replaces the old config-file with
>   the
>     new one and updates the hash-file.
> 
> I've looked at the etc-update script, but since I'm not good at
> shell-scripting, I decided to rewrite it in python. Not all is done
> yet, and I don't wanto to do unneeded work if someone wants to enhance
> the shell-script.
> 
> You find my python version at http://www.devone.org/linux/gentoo.html
> 
> Another extension would be versioning of configuration files via
> cvs/rcs, subversion or arch. But I had to do some more work on that
> ...
> 
> Joerg, mostly happy gentoo user
> 
> 
> -- 
> Fachbegriffe der Informatik (Nr 152): PORN
> - Poster Ohne Richtigen Namen 
> 
> 


-- 
Guy Martin
Gentoo Linux - HPPA port Lead / IPv6 team
Lug Charleroi (Belgium)

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] RFC: extension of etc-update

[Mike Williams]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Tuesday 10 August 2004 11:43, Joerg Hoh wrote:
> Hi
>
> I'm unhappy with the etc-update script. In my eyes it lacks at least one
> necessary feature:
>
> Detection of unmodified config-files

I think this should be a feature of portage, and I believe it's been discussed 
before.

Basic premise, as I remember, is that when portage merges a new version it 
purposely doesn't overwrite files CONFIG_PROTECT'd, but really it should look 
for the file in /var/db/pkg, check the md5sums, and if the same overwrite it.

> Another extension would be versioning of configuration files via cvs/rcs,
> subversion or arch. But I had to do some more work on that ...

dispatch-conf does basic backing up before overwriting.

- -- 
Mike Williams
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFBGKnQInuLMrk7bIwRAhHbAJwOhQJm17hbjULqWkDI3l/npmsQ2wCeLMIE
0XvdqnSdac1S4NyWXOuEB0o=
=7BO5
-----END PGP SIGNATURE-----

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] RFC: extension of etc-update

[Joerg Hoh]

[-- Attachment #1: Type: text/plain, Size: 1198 bytes --]

On Tue, Aug 10, 2004 at 11:56:16AM +0100, Mike Williams wrote:
> 
> Basic premise, as I remember, is that when portage merges a new version it 
> purposely doesn't overwrite files CONFIG_PROTECT'd, but really it should look 
> for the file in /var/db/pkg, check the md5sums, and if the same overwrite it.

the problem is the large number of files CONFIG_PROTECT'd ... 

> > Another extension would be versioning of configuration files via cvs/rcs,
> > subversion or arch. But I had to do some more work on that ...
> 
> dispatch-conf does basic backing up before overwriting.

That seems to be the right thing (according to the manpage), but I haven't
heard of it yet. The documentation mentions only etc-update. Does
dispatch-conf also handles the update of unchanged configs?

Joerg

-- 
Fachbegriffe der Informatik (Nr 366): Powerpoint
- Transportmittel für mindestens 50% der E-Mails eines Unternehmens. Die
Inhalte sind überwiegend mit humoristischem Anspruch. Ein kleiner Teil ist
ernst gemeint und erinnert an die letzten Zusammentreffen innerhalb der
Firma/des Konzerns, da wo man so schön Kaffee trinken und über Gott und die
Welt erzählen kann. 
	Holger Marzen

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] RFC: extension of etc-update

[Paul de Vrieze]

[-- Attachment #1: Type: text/plain, Size: 517 bytes --]

On Tuesday 10 August 2004 13:23, Joerg Hoh wrote:
> That seems to be the right thing (according to the manpage), but I haven't
> heard of it yet. The documentation mentions only etc-update. Does
> dispatch-conf also handles the update of unchanged configs?

Yes, it even has a configuration file that determines it's behaviour. It is a 
timesaver compared to etc-update. It also does auto-merges.

Paul

-- 
Paul de Vrieze
Gentoo Developer
Mail: pauldv@gentoo.org
Homepage: http://www.devrieze.net



[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] RFC: extension of etc-update

[Joerg Hoh]

[-- Attachment #1: Type: text/plain, Size: 884 bytes --]

On Tue, Aug 10, 2004 at 01:52:30PM +0200, Radoslaw Stachowiak wrote:
> 
> it doesnt in proper way, because dispatch-conf does not store md5sums,
> but uses /var/db/pkg data for its comparision. But using it (db) after
> installation results in md5sum lost (only md5 sums of NEW version of
> files are stored).
> 
> So there are two ways to do it:
> 1. store sums regardless of portage, and compare it after installation.
> 2. maybe, instead of md5sum, simple comparision if mtime < ctime than
> update is allowed would be fine?
> 
> please comment about 2) because it would be trivial to implement it in
> dispatch-conf.

ctime vs mtime would be acceptable; if it is combined with the
trivial-merge-approach, it's probably enough for most of the mentioned
issues.

Joerg


-- 
Fachbegriffe der Informatik (Nr 64): Network Computer
- grafikfähiges Terminal 


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] RFC: extension of etc-update

[Radoslaw Stachowiak]

On Tue, 10 Aug 2004 14:04:04 +0200, Joerg Hoh <joerg@devone.org> wrote:
> ctime vs mtime would be acceptable; if it is combined with the
> trivial-merge-approach, it's probably enough for most of the mentioned
> issues.

Ok, i implement it today and publish it for my dev.gentoo.org/~radek/
page for review with announcement here. It will also contain my
modification to allow batch-mode work. I use it as apart of my system
for semi-automatic management of a few gentoo boxes.

-- 
radoslaw.

--
gentoo-dev@gentoo.org mailing list

[gentoo-dev] small dispatch-conf patch, WAS: extension of etc-update

[Radoslaw Stachowiak]

Hello

I posted my patch to dispatch-conf at http://dev.gentoo.org/~radek/ 

This patch adds two important (for me :) things:

* Batch mode support, when dispatch updates all possible files (non
changed etc), and then prints remaining ones and exits.
* Merge unmodified files, using mtime < ctime comparision. This makes
unnecessary any md5sum storage/computing, while being (almost?)
equally safe.

Both things have supporting config file options for better integration.

Looking forward Your comments.

-- 
radoslaw.

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] small dispatch-conf patch, WAS: extension of etc-update

[Paul de Vrieze]

[-- Attachment #1: Type: text/plain, Size: 743 bytes --]

On Wednesday 11 August 2004 22:55, Radoslaw Stachowiak wrote:
> Hello
>
> I posted my patch to dispatch-conf at http://dev.gentoo.org/~radek/
>
> This patch adds two important (for me :) things:
>
> * Batch mode support, when dispatch updates all possible files (non
> changed etc), and then prints remaining ones and exits.
> * Merge unmodified files, using mtime < ctime comparision. This makes
> unnecessary any md5sum storage/computing, while being (almost?)
> equally safe.

Config files are normally small enough so that md5sum comparison is cheap 
enough. Why then rely on something fragile like mtime < ctime ?

Paul

-- 
Paul de Vrieze
Gentoo Developer
Mail: pauldv@gentoo.org
Homepage: http://www.devrieze.net

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] small dispatch-conf patch, WAS: extension of etc-update

[Radoslaw Stachowiak]

> > Because You have to calculate them for all files, and _before_
> > emerge. Also such database has to be regularly updated (sth like
> > eupdate/esearch).
>
> Hmm, I thought that things were only calculated/compared for those
> configuration files which might involve updating (those that get ._cfg etc)

yes, you are right. i was talking only about CONFIG_PROTECT files.
the biggest problem (as I see it) is that it requires special
maintenance and overhead to gather this data constantly. of course
computational overhead (due to use of md5) is negligible.

> > Not that Im sure of time approach, but Could You show one real word
> > scenario when ctime/mtime comparison would fail /while md5sum does
> > not/ ?
> 
> In cases of clock skew or toying with touch. Esp. the clock skew thing is not
> that uncommon.

Hmm, good point, but skew must be backwards, and not only it have to
be big enough (which is hard to achieve because every second after
package installation time, this necessary-for-failure time gap is
growing), but it has to be present at the time you hand modify
config-file-to-be-lost. i think its almost impossible.

Of course ntp time synchronisation would solve this.

As to touch. normal touch usage don't put You at risk because file
will have mtime > ctime so its not problem. special touch usage
(modify mtime to value other than current) is so rare that have to be
use for some purpose (eg for mark file to be overwritten by newer
update, even ifits modified) which is justified use for me.

Files overwritten are backed up by dispatch, so even if sth was
screwed up, you can get previous file from dispatch backup.

Im going to use it on some  machines this month, and will let You know
what happened :)

-- 
radek@gentoo.org.

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] RFC: extension of etc-update

[Donnie Berkholz]

Joerg Hoh (joerg@devone.org) wrote:
>
> On Tue, Aug 10, 2004 at 01:52:30PM +0200, Radoslaw Stachowiak wrote:
> >
> > it doesnt in proper way, because dispatch-conf does not store md5sums,
> > but uses /var/db/pkg data for its comparision. But using it (db) after
> > installation results in md5sum lost (only md5 sums of NEW version of
> > files are stored).
> >
> > So there are two ways to do it:
> > 1. store sums regardless of portage, and compare it after installation.
> > 2. maybe, instead of md5sum, simple comparision if mtime < ctime than
> > update is allowed would be fine?
> >
> > please comment about 2) because it would be trivial to implement it in
> > dispatch-conf.
>
> ctime vs mtime would be acceptable; if it is combined with the
> trivial-merge-approach, it's probably enough for most of the mentioned
> issues.

This still appears to have this problem, unless I missed it: What if the
default is exactly what a user wants in one version, then the default changes
in the next version? You assume an unmodified config file means the user
doesn't care what's in it.

But in this scenario, the file's MD5 would be unchanged from the initial
snapshot because the file is distributed as the user desires -- yet an update
wrongly overwrites this with a change that causes broken or unwanted results.

There should be at least an option to NOT do the MD5 checking you propose.

Thanks,
Donnie

-- 
Donnie Berkholz
Gentoo Linux



--
gentoo-dev@gentoo.org mailing list

[gentoo-dev] Re: RFC: extension of etc-update

[Duncan]

Mike Williams posted <200408101156.17481.mike@gaima.co.uk>, excerpted
below,  on Tue, 10 Aug 2004 11:56:16 +0100:

> On Tuesday 10 August 2004 11:43, Joerg Hoh wrote:
> 
>> Another extension would be versioning of configuration files via cvs/rcs,
>> subversion or arch. But I had to do some more work on that ...
> 
> dispatch-conf does basic backing up before overwriting.

Right.. Unfortunately, when I try to use it, I get file move errors, as if
it's attempting to hard-link across partitions.  I like the idea.. if it
worked!  (I haven't bug-checked this, as I expected it was designed that
way and therefore known behavior.  I just went back to etc-update, which
/still/ doesn't page right with the default config due to a parse error
because it wants quotes.  See bug.. 56785.. still marked new nearly a
month later, and even tho etc-update is documented core portage
functionality, even with a proposed patch in the form of an improved
sed command posted to the bug.)

-- 
Duncan - List replies preferred.   No HTML msgs.
"They that can give up essential liberty to obtain a little
temporary safety, deserve neither liberty nor safety." --
Benjamin Franklin



--
gentoo-dev@gentoo.org mailing list