[gentoo-dev] PostgreSQL Status

[gentoo-dev] PostgreSQL Status

[Tiziano Müller]

Since my blog doesn't get syndicated on planet.gentoo.org I'm posting the
entry here again, but it's probably a good idea anyway:

First I want to apologize for the current situation. I know we're lagging
behind and I know that bugs are piling up.
As a small excuse I can only say that my time is limited and PostgreSQL
isn't the only thing in Gentoo I (have to) maintain.

But there are good news:
Thanks to the help of Michael Krelin (also known as polyonymous on IRC) I
was able to commit a whole new set of ebuilds yesterday.
Now, you're probably going to ask why we just didn't go on with the current
ebuilds in the tree.

There are a couple of reasons:
a) dev-db/libpq is slotted but with collisions. Fixing this wasn't really an
option since slotting this was wrong from the beginning.
b) The split-up in two packages dev-db/{libpq,postgresql} was wrong too:
There are a couple of packages which do not only need libpq, but also one
of the other PostgreSQL libraries (like libecpg, libpgtypes, libecpg_compat
or libpgport).
c) Upgrading between major versions of PostgreSQL requires the DB admin to
bump the database using the old version, moving the database away and to
reload the dump into a new database cluster using the new version of
PostgreSQL. Having to take down the old server and purging the old version
of PostgreSQL before being able to try out the new one is more than
cumbersome. Therefore a slotted postgresql-server is needed to make the
upgrade easier.
d) A lot of things were broken in the old postgresql ebuilds as you can see
when going through the bug list and we needed to give them a general
overhaul.

What do the new ebuilds offer:
a) A split into dev-db/postgresql-{base,server,docs}. Now, I know that
splitting up packages isn't the Gentoo way. I know we could have done it
using USE flags but this approach gives more flexibility due to the current
way how binary packages are being generated and distributed.
b) New virtuals: virtual/postgresql-{base,server} to be able to install
pgcluster as your postgresql-base in a next step.
c) Slotting: It is now possible to have more than one major version of
PostgreSQL installed and running on the same machine.
d) A lot of other improvements, in detail, the following bugs will be fixed:
42894 Suggestion of use of slots for PostgreSQL
49864 dev-db/postgresql - pkg_config should allow passing optio...
55073 PostgreSQL client versus server installation
154620 PostgreSQL 8.1 server lacks instrumentation functions for...
158509 dev-db/libpq-8.0.9 fails configure with segmentation faul...
159223 postgresql threads USE flag required for ecpg
160178 dev-db/libpq : using (almost deprecated) gnuconfig_update
160181 dev-db/postgresql : using (almost deprecated) gnuconfig_u...
161803 dev-db/postgresql - /etc/conf.d/postgresql PG_OPTS variab...
161980 QA Notice: USE Flag 'kernel_linux' not in IUSE for dev-db...
177555 Postgresql/libpg upgrade from 8.1.8 to libpg-8.2.4 and po...
194350 >=dev-db/libpq-8.2.4 creates broken symlinks
194701 dev-db/postgresql - wrong elog into about configuration f...
198434 dev-db/postgresql - add ldap use flag
206725 dev-db/{libpq,postgresql} 8.2.7/8.3.1 version bump
206820 dev-db/postgresql default conf.d should not override max_...
209826 dev-db/postgresql - 2 issues with the current init script
210938 dev-db/postgresql - disable strict permission check on ss...
213699 dev-db/libpq-8.2.6 failed to configure w/ USE="threads"
214438 The dev-db/postgresql-8.2.6 ebuild does not respect confi...
215940 dev-db/postgresql provides bad init script for baselayout...


What you have to do to switch:
I'll put together more documentation as soon as the ebuilds get unmasked (in
1-2 weeks).
In general the only thing you have to then do is to uninstall dev-db/libpq
and dev-db/postgresql and install the same version of postgresql-base and
postgresql-server. No revdep-rebuild is needed.
For early adopters: It's best to wait until we changed the dependencies,
afterwards you can unmask the dev-db/postgresql-{docs,base,server}
packages...

What is needed from the ebuild developers side:
We have to change every dependency on dev-db/libpq to
virtual/postgresql-base and dev-db/postgresql to virtual/postgresql-server.
As soon as the old ebuilds are gone, we have to go through the ebuilds
depending on virtual/postgresql-server whether they can be built with
dev-db/postgresql-base only. Do not switch from dev-db/postgresql to
virtual/postgresql-base directly as long as you can't assure that your
package builds with libpq only.

Well, that's it basically. Don't hesitate to contact me in case of problems
or suggestions. You can answer to this post, leave comments on
http://dev-zero.ch/blog/ or send me a mail of course :-).
(But don't expect me to read forum entries since I simply do not have the
time to do that regularly.)

Again, many thanks to Michael Krelin and all the others who helped with
testing and sending patches.

Cheers,
Tiziano


-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] PostgreSQL Status

[Donnie Berkholz]

On 09:55 Wed 16 Apr     , Tiziano Müller wrote:
> What do the new ebuilds offer:
> a) A split into dev-db/postgresql-{base,server,docs}. Now, I know that
> splitting up packages isn't the Gentoo way. I know we could have done it
> using USE flags but this approach gives more flexibility due to the current
> way how binary packages are being generated and distributed.

I'd like to hear some more info on this point.

> In general the only thing you have to then do is to uninstall dev-db/libpq
> and dev-db/postgresql and install the same version of postgresql-base and
> postgresql-server. No revdep-rebuild is needed.
> For early adopters: It's best to wait until we changed the dependencies,
> afterwards you can unmask the dev-db/postgresql-{docs,base,server}
> packages...

People want `emerge postgresql` to do something. Otherwise it's not 
always obvious which random hyphenated packages you're supposed to 
install, and it's just like you're digging around some huge subpackage 
list in Ubuntu or Fedora.

Thanks,
Donnie
-- 
gentoo-dev@lists.gentoo.org mailing list

[gentoo-dev] Re: PostgreSQL Status

[Tiziano Müller]

Donnie Berkholz wrote:

> On 09:55 Wed 16 Apr     , Tiziano Müller wrote:
>> What do the new ebuilds offer:
>> a) A split into dev-db/postgresql-{base,server,docs}. Now, I know that
>> splitting up packages isn't the Gentoo way. I know we could have done it
>> using USE flags but this approach gives more flexibility due to the
>> current way how binary packages are being generated and distributed.
> 
> I'd like to hear some more info on this point.

Consider this use case:
30 machines with one staging machine and binary deployment.
On 28 machines you want the libraries only, on one you also need the server
and on one you want the docs.
Easy done with (sanely) splitted packages.
Please note that the only additional package compared to the split
dev-db/{libpq,postgresql} is dev-db/postgresql-docs.

> 
>> In general the only thing you have to then do is to uninstall
>> dev-db/libpq and dev-db/postgresql and install the same version of
>> postgresql-base and postgresql-server. No revdep-rebuild is needed.
>> For early adopters: It's best to wait until we changed the dependencies,
>> afterwards you can unmask the dev-db/postgresql-{docs,base,server}
>> packages...
> 
> People want `emerge postgresql` to do something. Otherwise it's not
> always obvious which random hyphenated packages you're supposed to
> install, and it's just like you're digging around some huge subpackage
> list in Ubuntu or Fedora.

Well, we can re-introduce a virtual/postgresql (or dev-db/postgresql) after
the old ebuilds are gone.

Cheers,
Tiziano

-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] PostgreSQL Status

[Carsten Lohrke]

[-- Attachment #1: Type: text/plain, Size: 1601 bytes --]

> c) Upgrading between major versions of PostgreSQL requires the DB admin to
> bump the database using the old version, moving the database away and to
> reload the dump into a new database cluster using the new version of
> PostgreSQL. Having to take down the old server and purging the old version
> of PostgreSQL before being able to try out the new one is more than
> cumbersome. Therefore a slotted postgresql-server is needed to make the
> upgrade easier.

As I read upstream's documentation¹, this is incorrect:

# It is recommended that you use the pg_dump and pg_dumpall programs from the
# newer version of PostgreSQL, to take advantage of any enhancements that may
# have been made in these programs. Current releases of the dump programs can
# read data from any server version back to 7.0. 

This has also been pointed out in a bug report, duped by some overzealous bug 
wrangler a few months ago.

> What do the new ebuilds offer:
> a) A split into dev-db/postgresql-{base,server,docs}. Now, I know that
> splitting up packages isn't the Gentoo way. I know we could have done it
> using USE flags but this approach gives more flexibility due to the current
> way how binary packages are being generated and distributed.

I wouldn't say so. Needlessly growing the forest of odd named, badly 
documented and sometimes needlessly added use flags is something I'm not fond 
of, looking at the development of Gentoo.


Thanks to everyone taking care of our PostgreSQL packages.


Carsten


[1] http://www.postgresql.org/docs/8.2/interactive/migration.html

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[gentoo-dev] Re: PostgreSQL Status

[Tiziano Müller]

Carsten Lohrke wrote:

>> c) Upgrading between major versions of PostgreSQL requires the DB admin
>> to bump the database using the old version, moving the database away and
>> to reload the dump into a new database cluster using the new version of
>> PostgreSQL. Having to take down the old server and purging the old
>> version of PostgreSQL before being able to try out the new one is more
>> than cumbersome. Therefore a slotted postgresql-server is needed to make
>> the upgrade easier.
> 
> As I read upstream's documentation¹, this is incorrect:
> 
> # It is recommended that you use the pg_dump and pg_dumpall programs from
> # the newer version of PostgreSQL, to take advantage of any enhancements
> # that may have been made in these programs. Current releases of the dump
> # programs can read data from any server version back to 7.0.

While the dump command can read clusters created by an older version it is
still necessary to dump and reload your data on version bumps between major
versions as written in
http://www.postgresql.org/docs/8.3/static/install-upgrading.html:
[...]
The internal data storage format typically changes in every major release of
PostgreSQL. Therefore, if you are upgrading an existing installation that
does not have a version number of "8.3.x", you must back up and restore
your data. If you are upgrading from PostgreSQL "8.3.x", the new version
can use your current data files so you should skip the backup and restore
steps below because they are unnecessary.
[...]

Cheers,
Tiziano


-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] Re: PostgreSQL Status

[Enrico Weigelt]

* Tiziano Müller <dev-zero@gentoo.org> schrieb:

> While the dump command can read clusters created by an older version it is
> still necessary to dump and reload your data on version bumps between major
> versions as written in

That's the point where MVCC is needed, at least for critical 
applications and large databases: each db has to be tested carefully
on the new server before moving to production and downtimes have 
to be minmal. An single-version system cannot provide this.

I'm doing this by assigning multiple sockets to the server (one 
per DB) and so migrating the DB's one by one w/o having to touch
the clients.


cu
-- 
---------------------------------------------------------------------
 Enrico Weigelt    ==   metux IT service - http://www.metux.de/
---------------------------------------------------------------------
 Please visit the OpenSource QM Taskforce:
 	http://wiki.metux.de/public/OpenSource_QM_Taskforce
 Patches / Fixes for a lot dozens of packages in dozens of versions:
	http://patches.metux.de/
---------------------------------------------------------------------
--
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] Re: PostgreSQL Status

[Carsten Lohrke]

[-- Attachment #1: Type: text/plain, Size: 554 bytes --]

On Mittwoch, 16. April 2008, Tiziano Müller wrote:
> While the dump command can read clusters created by an older version it is
> still necessary to dump and reload your data on version bumps between major
> versions [...

Of course. I didn't question the dump and reload cycle. Just saying you have 
to use the dump utility of the old version is not correct and goes against 
the recommendations of the PostgreSQL developers. Also the annoying 
pkg_setup() error && die stuff in the existing ebuilds has always been 
superfluous.


Carsten

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] PostgreSQL Status

[Luca Barbato]

Tiziano Müller wrote:
> What do the new ebuilds offer:
> a) A split into dev-db/postgresql-{base,server,docs}.

WRONG we aren't debian.

we have a nice use for documentation...

> Now, I know that
> splitting up packages isn't the Gentoo way. I know we could have done it
> using USE flags but this approach gives more flexibility due to the current
> way how binary packages are being generated and distributed.

It gives an annoyance please reconsider.

> b) New virtuals: virtual/postgresql-{base,server} to be able to install
> pgcluster as your postgresql-base in a next step.

see before.

> c) Slotting: It is now possible to have more than one major version of
> PostgreSQL installed and running on the same machine.

Great =)

> d) A lot of other improvements, in detail, the following bugs will be fixed:

Wonderful =)

lu

-- 

Luca Barbato
Gentoo Council Member
Gentoo/linux Gentoo/PPC
http://dev.gentoo.org/~lu_zero

-- 
gentoo-dev@lists.gentoo.org mailing list

[gentoo-dev] Re: PostgreSQL Status

[Tiziano Müller]

Luca Barbato wrote:

> Tiziano Müller wrote:
>> What do the new ebuilds offer:
>> a) A split into dev-db/postgresql-{base,server,docs}.
> 
> WRONG we aren't debian.
This is why we decided not to split out headers, clients and contrib.

> 
>> Now, I know that
>> splitting up packages isn't the Gentoo way. I know we could have done it
>> using USE flags but this approach gives more flexibility due to the
>> current way how binary packages are being generated and distributed.
> 
> It gives an annoyance please reconsider.
Done that. Won't change. See my answer to dberkholz's message.


-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] Re: PostgreSQL Status

[Enrico Weigelt]

* Tiziano Müller <dev-zero@gentoo.org> schrieb:

> > WRONG we aren't debian.
> This is why we decided not to split out headers, clients and contrib.

Actually, I'd like to see them all split out. But this sooner or
later requires the upstream (or an intermediate layer, like OSS-QM)
to support this - individual distros shouldn't do this by themselves
(too much unnecessary duplicate work).


cu
-- 
---------------------------------------------------------------------
 Enrico Weigelt    ==   metux IT service - http://www.metux.de/
---------------------------------------------------------------------
 Please visit the OpenSource QM Taskforce:
 	http://wiki.metux.de/public/OpenSource_QM_Taskforce
 Patches / Fixes for a lot dozens of packages in dozens of versions:
	http://patches.metux.de/
---------------------------------------------------------------------
--
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] Re: PostgreSQL Status

[Luca Barbato]

Tiziano Müller wrote:
> Luca Barbato wrote:
>> It gives an annoyance please reconsider.
> Done that. Won't change. See my answer to dberkholz's message.
> 

As long you keep a meta package, as you told in the reply I read just 
now, seems a good plan in the end.

lu

-- 

Luca Barbato
Gentoo Council Member
Gentoo/linux Gentoo/PPC
http://dev.gentoo.org/~lu_zero

-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] PostgreSQL Status

[Enrico Weigelt]

* Luca Barbato <lu_zero@gentoo.org> schrieb:
> Tiziano Müller wrote:
> >What do the new ebuilds offer:
> >a) A split into dev-db/postgresql-{base,server,docs}.
> 
> WRONG we aren't debian.

It's bad, just because Debian does it ?!
Sounds quite religions to me. I don't like religions interfering with
technical designs.


cu
-- 
---------------------------------------------------------------------
 Enrico Weigelt    ==   metux IT service - http://www.metux.de/
---------------------------------------------------------------------
 Please visit the OpenSource QM Taskforce:
 	http://wiki.metux.de/public/OpenSource_QM_Taskforce
 Patches / Fixes for a lot dozens of packages in dozens of versions:
	http://patches.metux.de/
---------------------------------------------------------------------
--
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] PostgreSQL Status

[Nirbheek Chauhan]

On Thu, Apr 17, 2008 at 10:10 PM, Enrico Weigelt <weigelt@metux.de> wrote:
> * Luca Barbato <lu_zero@gentoo.org> schrieb:
>
> > Tiziano Müller wrote:
>  > >What do the new ebuilds offer:
>  > >a) A split into dev-db/postgresql-{base,server,docs}.
>  >
>  > WRONG we aren't debian.
>
>  It's bad, just because Debian does it ?!

Yes, because their technical design is different from Gentoo's.
Debian/$binary_distro have to split up packages because they have no
other way of mapping configure flags to what the user wants to
install. Gentoo does, and they're called USE flags.

>  Sounds quite religions to me. I don't like religions interfering with
>  technical designs.

See above.

-- 
~Nirbheek Chauhan
--
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] PostgreSQL Status

[Enrico Weigelt]

* Nirbheek Chauhan <nirbheek.chauhan@gmail.com> schrieb:

> >  It's bad, just because Debian does it ?!
> 
> Yes, because their technical design is different from Gentoo's.

Still too religious ;-P

Would be different if you'd said "we don't need to split since we 
have useflags" or "we want to stay as near to upstream as possible"
instead of "we're not debian". Being different, just to be 
different is really too pubertal for me ;-o

> Debian/$binary_distro have to split up packages because they have no
> other way of mapping configure flags to what the user wants to
> install. Gentoo does, and they're called USE flags.

Right, binary distros have a much bigger presure on that, but this
doesn't mean that splitting is always bad.
Think of all these useflag-deps, which often wouldn't be necessary
if the packages would be split in finer granularity.
Of course, this is mostly the upstream's fault.

cu
-- 
---------------------------------------------------------------------
 Enrico Weigelt    ==   metux IT service - http://www.metux.de/
---------------------------------------------------------------------
 Please visit the OpenSource QM Taskforce:
 	http://wiki.metux.de/public/OpenSource_QM_Taskforce
 Patches / Fixes for a lot dozens of packages in dozens of versions:
	http://patches.metux.de/
---------------------------------------------------------------------
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] PostgreSQL Status

[Nirbheek Chauhan]

On Thu, Apr 17, 2008 at 10:37 PM, Enrico Weigelt <weigelt@metux.de> wrote:
> * Nirbheek Chauhan <nirbheek.chauhan@gmail.com> schrieb:
>
>
>  > >  It's bad, just because Debian does it ?!
>  >
>  > Yes, because their technical design is different from Gentoo's.
>
>  Still too religious ;-P

Of course, that's your opinion to have and keep.

>
>  Would be different if you'd said "we don't need to split since we
>  have useflags" or "we want to stay as near to upstream as possible"
>  instead of "we're not debian". Being different, just to be
>  different is really too pubertal for me ;-o
>

The person you were replying to is a Gentoo Council member, elected by
the very devs that run Gentoo. I believe one should try and look
deeper in the meaning of such a man's words before labelling them as
"pubertal".

The words of a veteran usually aren't written in blind emotion or with
prejudice.

>
>  > Debian/$binary_distro have to split up packages because they have no
>  > other way of mapping configure flags to what the user wants to
>  > install. Gentoo does, and they're called USE flags.
>
>  Right, binary distros have a much bigger presure on that, but this
>  doesn't mean that splitting is always bad.

There are specific use-cases where splitting is good, such as with
gstreamer, gtk-sharp, gnome-python{,-desktop,-extras} which are
essentially dependencies of other packages, and where built_with_use
checks are horrid to use.

>  Think of all these useflag-deps, which often wouldn't be necessary
>  if the packages would be split in finer granularity.
>  Of course, this is mostly the upstream's fault.

I personally have no opinion about the -base and -server split, since
I do not know enough about it. But I am firmly against the -docs split
since the doc USE flag is for this use-case, and I see no reason why
not to use it.

Just stick a USE=doc on -base and be done with it

-- 
~Nirbheek Chauhan
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] PostgreSQL Status

[Enrico Weigelt]

* Nirbheek Chauhan <nirbheek.chauhan@gmail.com> schrieb:

> >  Would be different if you'd said "we don't need to split since we
> >  have useflags" or "we want to stay as near to upstream as possible"
> >  instead of "we're not debian". Being different, just to be
> >  different is really too pubertal for me ;-o
> >
> 
> The person you were replying to is a Gentoo Council member, elected by
> the very devs that run Gentoo. I believe one should try and look
> deeper in the meaning of such a man's words before labelling them as
> "pubertal".

Well, this is my personal feeling, and this is totally independent from 
his position.

I've just got headaches with this "we're not debian" argumentation.
Surely there are technical reasons to do certain things different than
other distros, and so we should talk about exactly about these points.
(problem -> analyis -> solution). 

Just insisting on being different isn't a technical reason, just an 
pubertal behaviour (well, there are good reasons for this behaviour in 
that that age, eg. becoming independent from the parent generation).
This reminds me on the "anti-fascist" folks here in Germany, who tend to 
define themselves on being "against fascists" and declaring everyone with 
different opponions to be one (note that Germany never had noticable fascist 
movements - we had national socialists, but that's very different ;-O)

> The words of a veteran usually aren't written in blind emotion or with
> prejudice.

Might be. But in this case, I really feel different.

Please let's talk about concepts and practises of other distros objectively,
leaving out personal antipathies. Every one has different views and needs,
and technical decisions should derive them them, not from personal taste.

So for example the splitting issue has to be decided for each package. 

As we're currently talking about PostgreSQL, we have to look at the 
possible ways to do (or not to do) so with it and weight the pros and
cons of the different options.

This decision process should be individual to each package - there is
no (good) universal answer. If you try to declare an universal answer
to everyone, you get religious ;-P

> >  Right, binary distros have a much bigger presure on that, but this
> >  doesn't mean that splitting is always bad.
> 
> There are specific use-cases where splitting is good, such as with
> gstreamer, gtk-sharp, gnome-python{,-desktop,-extras} which are
> essentially dependencies of other packages, and where built_with_use
> checks are horrid to use.

Yes, and there a lots of them. 

Classical example: language bindings for certain libs. They really 
should be different packages. But, of course, the fault mostly comes
from the upstream and individual distros aren't the right place to
fix this (again, one of the reasons why I founded the OSS-QM project).

> I personally have no opinion about the -base and -server split, since
> I do not know enough about it. But I am firmly against the -docs split
> since the doc USE flag is for this use-case, and I see no reason why
> not to use it.

Historically, the manuals (actually, electronic books - printed out 
about 1k pages) have been an separate package from upstream. And this 
for a good reason: they an different entitiy (even maintained by 
different people), quite large and (un)related to the rest of PQ just
like an programming book to an invidiual compiler (note that it's also
contains of the most complete posix-SQL references in the OSS world).

> Just stick a USE=doc on -base and be done with it

This has an major drawback: requires to do an complete rebuild/reinstall
of the whole package if you just need the manual. When setting up an
new server, you normally don't need the complete manual installed 
(assuming you're already confident w/ PQ), but you need it someday
later when you have to look up something and other media (web access
or printed out) are not convenient/available.

I, personally, don't *need* it at all, but having an separate package
makes it more convenient. And I don't see any reasons against that 
split as long as people are willing to maintain it.


cu
-- 
---------------------------------------------------------------------
 Enrico Weigelt    ==   metux IT service - http://www.metux.de/
---------------------------------------------------------------------
 Please visit the OpenSource QM Taskforce:
 	http://wiki.metux.de/public/OpenSource_QM_Taskforce
 Patches / Fixes for a lot dozens of packages in dozens of versions:
	http://patches.metux.de/
---------------------------------------------------------------------
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] PostgreSQL Status

[Nirbheek Chauhan]

On Fri, Apr 18, 2008 at 12:17 AM, Enrico Weigelt <weigelt@metux.de> wrote:
> * Nirbheek Chauhan <nirbheek.chauhan@gmail.com> schrieb:
>  > I personally have no opinion about the -base and -server split, since
>  > I do not know enough about it. But I am firmly against the -docs split
>  > since the doc USE flag is for this use-case, and I see no reason why
>  > not to use it.
>
>  Historically, the manuals (actually, electronic books - printed out
>  about 1k pages) have been an separate package from upstream. And this
>  for a good reason: they an different entitiy (even maintained by
>  different people), quite large and (un)related to the rest of PQ just
>  like an programming book to an invidiual compiler (note that it's also
>  contains of the most complete posix-SQL references in the OSS world).

Manuals, yes, documentation, not necessarily. Documentation is often
*built* from the source code, or it's source is included with the
package source code and docs are generated at compile-time. And as the
Django and Python philosophy says, "No special case is special
enough". Hence, USE=doc gives documentation. Whether generated from
source, generated with it, or downloaded and installed.

>
>
>  > Just stick a USE=doc on -base and be done with it
>
>  This has an major drawback: requires to do an complete rebuild/reinstall
>  of the whole package if you just need the manual. When setting up an
>  new server, you normally don't need the complete manual installed
>  (assuming you're already confident w/ PQ), but you need it someday
>  later when you have to look up something and other media (web access
>  or printed out) are not convenient/available.

Now, there are lots of similar examples where USE flags exist purely
to pull in run-time deps that don't require the package to be
recompiled for usage, and not being able to specify such deps in an
ebuild is a deficiency on the package manager side. The proper
solution is to fix the package manager, not to split out such things
into separate packages. It generates chaos and inconsistency in the
tree, and only delays the fixing of the package manager by giving
half-baked solutions.

>
>  I, personally, don't *need* it at all, but having an separate package
>  makes it more convenient. And I don't see any reasons against that
>  split as long as people are willing to maintain it.

One reason I can think of is that people expect USE=doc to give them
documentation. They expect that a "doc" USE flag on a package will
give them the documentation. That's how Gentoo works, and so that's
how people *expect* it to work. Having a special case for a package
just to save a little build time is (imho) not worth the
inconsistency.

-- 
~Nirbheek Chauhan
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] PostgreSQL Status

[Enrico Weigelt]

* Nirbheek Chauhan <nirbheek.chauhan@gmail.com> schrieb:

> Manuals, yes, documentation, not necessarily. Documentation is often
> *built* from the source code, or it's source is included with the
> package source code and docs are generated at compile-time. 

ACK. But we first should define what's "just documentation" 
(things like readme's, changelogs, etc) and what's and separate
manual (book). There's often no general answer for that. Individual
package maintainers have to decide this.

> >  This has an major drawback: requires to do an complete rebuild/reinstall
> >  of the whole package if you just need the manual. When setting up an
> >  new server, you normally don't need the complete manual installed
> >  (assuming you're already confident w/ PQ), but you need it someday
> >  later when you have to look up something and other media (web access
> >  or printed out) are not convenient/available.
> 
> Now, there are lots of similar examples where USE flags exist purely
> to pull in run-time deps that don't require the package to be
> recompiled for usage, and not being able to specify such deps in an
> ebuild is a deficiency on the package manager side. The proper
> solution is to fix the package manager, not to split out such things
> into separate packages. It generates chaos and inconsistency in the
> tree, and only delays the fixing of the package manager by giving
> half-baked solutions.

We have virtuals. Properly used they can be a great mechanism for 
solving this.

One example, where it (IMHO) isn't done properly, is the X-server, where
adding another driver (which *IS* an separate package, already from upstream)
causes the whole server to be rebuild and also produces ugly circular deps
(yes, I know of the PDEPEND workaround). 
x11-base/xorg-server SHOULD be an virtual. 
(I already sucessfully did this quite some time ago, but meanwhile its outdated) 

> >  I, personally, don't *need* it at all, but having an separate package
> >  makes it more convenient. And I don't see any reasons against that
> >  split as long as people are willing to maintain it.
> 
> One reason I can think of is that people expect USE=doc to give them
> documentation. They expect that a "doc" USE flag on a package will
> give them the documentation. 

Right, but again we're at the question, what really belongs to the docs.
According your argumentation, the GCC programming guide should be also
pulled in by the doc usflag. Do you really want this ?


cu
-- 
---------------------------------------------------------------------
 Enrico Weigelt    ==   metux IT service - http://www.metux.de/
---------------------------------------------------------------------
 Please visit the OpenSource QM Taskforce:
 	http://wiki.metux.de/public/OpenSource_QM_Taskforce
 Patches / Fixes for a lot dozens of packages in dozens of versions:
	http://patches.metux.de/
---------------------------------------------------------------------
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] PostgreSQL Status

[Nirbheek Chauhan]

On Fri, Apr 18, 2008 at 1:15 AM, Enrico Weigelt <weigelt@metux.de> wrote:
> * Nirbheek Chauhan <nirbheek.chauhan@gmail.com> schrieb:
> > Manuals, yes, documentation, not necessarily. Documentation is often
>  > *built* from the source code, or it's source is included with the
>  > package source code and docs are generated at compile-time.
>
>  ACK. But we first should define what's "just documentation"
>  (things like readme's, changelogs, etc) and what's and separate
>  manual (book). There's often no general answer for that. Individual
>  package maintainers have to decide this.

I agree with you here, it's the maintainer's call.

>  > Now, there are lots of similar examples where USE flags exist purely
>  > to pull in run-time deps that don't require the package to be
>  > recompiled for usage, and not being able to specify such deps in an
>  > ebuild is a deficiency on the package manager side. The proper
>  > solution is to fix the package manager, not to split out such things
>  > into separate packages. It generates chaos and inconsistency in the
>  > tree, and only delays the fixing of the package manager by giving
>  > half-baked solutions.
>
>  We have virtuals. Properly used they can be a great mechanism for
>  solving this.
>
>  One example, where it (IMHO) isn't done properly, is the X-server, where
>  adding another driver (which *IS* an separate package, already from upstream)
>  causes the whole server to be rebuild and also produces ugly circular deps
>  (yes, I know of the PDEPEND workaround).
>  x11-base/xorg-server SHOULD be an virtual.
>  (I already sucessfully did this quite some time ago, but meanwhile its outdated)

This too, just like split ebuilds, is (imo) a hack, and not the
solution. A hack that's used in the tree in several places (-meta
ebuilds, xorg-server), and seems more "Gentoo-ish", but a hack
nonetheless, and hence a half-baked solution that delays the package
manager's fixing.

Maybe I'm just pushing too hard here, but the state of Gentoo's
package managers really irks me, and these workarounds just give
temporary relief to the problem.

>  > One reason I can think of is that people expect USE=doc to give them
>  > documentation. They expect that a "doc" USE flag on a package will
>  > give them the documentation.
>
>  Right, but again we're at the question, what really belongs to the docs.
>  According your argumentation, the GCC programming guide should be also
>  pulled in by the doc usflag. Do you really want this ?

Well, I think you answered this in your first paragraph above, that
it's entirely upto the maintainer.

If it were upto me, I'd go with pulling in the following with the doc USE flag:

- Documentation generated from the source code
- Documentation generated *with* the source code
- Documentation distributed with the source code (inside the tarball)
- Documentation distributed/advertised alongside the package/source
code (for instance GIMP help files)

-- 
~Nirbheek Chauhan
-- 
gentoo-dev@lists.gentoo.org mailing list

[gentoo-dev] Re: PostgreSQL Status

[Tiziano Müller]

Nirbheek Chauhan wrote:
> I personally have no opinion about the -base and -server split, since
> I do not know enough about it. But I am firmly against the -docs split
> since the doc USE flag is for this use-case, and I see no reason why
> not to use it.
> 
> Just stick a USE=doc on -base and be done with it

Well, this I can answer: The documentation installed by postgresql-docs
consists of both server and base documentation. There's also a 'doc'
USE-flag for postgresql-base which pulls in the docs.
Besides this, the use case I wrote in one of my last messages for
postgresql-base/-server also applies to postgresql-base/-docs or
postgresql-server/-docs.


-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] Re: PostgreSQL Status

[Nirbheek Chauhan]

On Fri, Apr 18, 2008 at 3:06 AM, Tiziano Müller <dev-zero@gentoo.org> wrote:
> Nirbheek Chauhan wrote:
>  > I personally have no opinion about the -base and -server split, since
>  > I do not know enough about it. But I am firmly against the -docs split
>  > since the doc USE flag is for this use-case, and I see no reason why
>  > not to use it.
>  >
>  > Just stick a USE=doc on -base and be done with it
>
>  Well, this I can answer: The documentation installed by postgresql-docs
>  consists of both server and base documentation. There's also a 'doc'
>  USE-flag for postgresql-base which pulls in the docs.
>  Besides this, the use case I wrote in one of my last messages for
>  postgresql-base/-server also applies to postgresql-base/-docs or
>  postgresql-server/-docs.
>

Ah, so -{base,server} have doc USE flags which pull in -docs? That
crosses out that worry of mine I suppose, thanks for the answer :)

-- 
~Nirbheek Chauhan
--
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] PostgreSQL Status

[Luca Barbato]

Enrico Weigelt wrote:
> * Luca Barbato <lu_zero@gentoo.org> schrieb:
>> Tiziano Müller wrote:
>>> What do the new ebuilds offer:
>>> a) A split into dev-db/postgresql-{base,server,docs}.
>> WRONG we aren't debian.
> 
> It's bad, just because Debian does it ?!
> Sounds quite religions to me. I don't like religions interfering with
> technical designs.

Other way round.

Gentoo has useflags to provide what binary distribution (debian pointed 
as its one of the best in their field) do by butchering the packages.

The only good reason to split a package is if takes too much to build 
and you have a clean way to do that (e.g qemu and kde) (then we provide 
meta packages to give back what people expect from emerge foo)

If upstream package its stuff in a way it's better to work with them to 
accomodate different needs, butchering leads to annoyance on our side 
and their.

lu

-- 

Luca Barbato
Gentoo Council Member
Gentoo/linux Gentoo/PPC
http://dev.gentoo.org/~lu_zero

-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] PostgreSQL Status

[Enrico Weigelt]

* Tiziano Müller <dev-zero@gentoo.org> schrieb:

Hi,

> Since my blog doesn't get syndicated on planet.gentoo.org I'm posting the
> entry here again, but it's probably a good idea anyway:

Actually, I'd clearly prefer lists like this one for those discussions.
Blogs are okay for writing journal articles and let people comment them,
but bad for real discussions, IMHO.

> First I want to apologize for the current situation. I know we're lagging
> behind and I know that bugs are piling up.
> As a small excuse I can only say that my time is limited and PostgreSQL
> isn't the only thing in Gentoo I (have to) maintain.

Well, my time is limited too (hmm, an contract for Gentoo development 
would be fine ;-)), otherwise I'd already reworked the pg ebuilds ;-o

> There are a couple of reasons:
> a) dev-db/libpq is slotted but with collisions. Fixing this wasn't really an
> option since slotting this was wrong from the beginning.

ACK. It should be done by *real* MVCC. But I doubt that portage is 
actually capable of this yet. For the vast majority of the cases I 
see slots just as a dirty hack ;-p

The main problem for MVCC (and also what often makes revdep-rebuild 
necessary) is the point that source and binary packages have to be 
completely different: the mapping from source to binary happens at
only build time and all the rest of the dependency handling derives
from that. I'm currently implementing this approach in my own build/
packaging system, but it's going to tricky.

> b) The split-up in two packages dev-db/{libpq,postgresql} was wrong too:
> There are a couple of packages which do not only need libpq, but also one
> of the other PostgreSQL libraries (like libecpg, libpgtypes, libecpg_compat
> or libpgport).

NAK! Each of these libraries are their own entities. Some clients even may
depend on one of them, but not libpq. So they clearly belong to their
own packages.

I do *NOT* want *ANY* unneeded stuff in my systems. You'll force me to fork.

> c) Upgrading between major versions of PostgreSQL requires the DB admin to
> bump the database using the old version, moving the database away and to
> reload the dump into a new database cluster using the new version of
> PostgreSQL. Having to take down the old server and purging the old version
> of PostgreSQL before being able to try out the new one is more than
> cumbersome. Therefore a slotted postgresql-server is needed to make the
> upgrade easier.

That's the point where we need *real* MVCC. An virtual package could 
coordinate the update process (eg. pulling in new versions and providing
update utils, maybe with some additional refcounting and automatic cleanup
based on that)

Yes, MVCC is not trivial ;-P

> What do the new ebuilds offer:
> a) A split into dev-db/postgresql-{base,server,docs}. Now, I know that
> splitting up packages isn't the Gentoo way. I know we could have done it
> using USE flags but this approach gives more flexibility due to the current
> way how binary packages are being generated and distributed.

I actually think packages should be split whenever suitable, useflags should 
only be used for *conditional* building (where features are switched that 
do *not* reflect separate modules) or for virtual (frontend) packages.

> b) New virtuals: virtual/postgresql-{base,server} to be able to install
> pgcluster as your postgresql-base in a next step.

Yes, frontend virtuals for convenience are an good idea for many users.
I, personally, probably won't use them, since my setups would be too 
complex for them. Other folks with simpler setups might be perfectly 
fine with them.

> c) Slotting: It is now possible to have more than one major version of
> PostgreSQL installed and running on the same machine.

Great. This makes smooth update processes easier (reduces the need of 
custom ebuilds). But I, pesonally, prefer *separate* packages instead 
of slots. 

> 159223 postgresql threads USE flag required for ecpg

BTW: does portage meanwhile handle feature dependencies ?
It's always a big mess when an whole install/update fails in the middle
just because some package wants some other built with certain useflag :(

> 161980 QA Notice: USE Flag 'kernel_linux' not in IUSE for dev-db...

Naive question: why does this useflag appear in pg ?


cu
-- 
---------------------------------------------------------------------
 Enrico Weigelt    ==   metux IT service - http://www.metux.de/
---------------------------------------------------------------------
 Please visit the OpenSource QM Taskforce:
 	http://wiki.metux.de/public/OpenSource_QM_Taskforce
 Patches / Fixes for a lot dozens of packages in dozens of versions:
	http://patches.metux.de/
---------------------------------------------------------------------
--
gentoo-dev@lists.gentoo.org mailing list