[gentoo-dev] Resolving HAL vs. pciutils/usbutils

[gentoo-dev] Resolving HAL vs. pciutils/usbutils

[Robin H. Johnson]

[-- Attachment #1.1: Type: text/plain, Size: 3178 bytes --]

Heya,

So now this is not a flamewar.

Jakub was originally going to complain at me for the upstream usbutils
adding support for gzipped usb.ids files, but a group of us (myself,
dsd, jakub, leio, steev) had a discussion about it, and came up with a
solution that both ends the breakage for direct users (HAL and others),
and provides forward momentum.

So firstly, what's the real problem? The original complaint came up
because HAL expected the uncompressed file to exist as pci.ids, and
wasn't ready to look at pci.ids.gz. While this caused breakage, it was
only a warning sign that there was a deeper problem.

libpci exists so that every application that wants to read the pci.ids
file does not need to know the format, and so that the pciutils
maintainers can change the file format (paired with changing their
parsing code), without causing breakage for any other packages.

HAL is not the only user of pci.ids/usb.ids:
http://www.google.com/codesearch?q=%22pci.ids%22
http://www.google.com/codesearch?q=%22usb.ids%22

Here's DSD's original thread on the HAL lists:
http://lists.freedesktop.org/archives/hal/2007-June/008834.html
And he also noted this, but I don't have a link for it.
<dsd_> upstream PCI are going to change the format too, they gave 
       like a years notice
Also:
<robbat2> dsd_, thanks. so really, once the libpci is installed as 
          a shared lib, there's no reason that HAL shouldn't use it?
<dsd_> robbat2: indeed - and nobody actually raised that as an 
       objection, parts of hal already statically link against 
       libpci...

So at the moment, HAL needs the changes, but the patches have been
blocked because pciutils needs move to a shared library and not cause
ABI breakages instead of the present static-lib-only situation.

usbutils is in very similar (but not quite identical) position, they
have not yet introduced any library, but will be doing so, to enable
similar format changes.

The solution that the group of us agreed on, was the following:
For USE=zlib, pciutils and usbutils should use the .gz version,
but ALSO maintain the uncompressed version until all applications 
have migrated to using the libraries to handle the pci.ids.

This uses the upstream-preferred formats for download, and stores
suitably so that both {pci,usb}utils work, and HAL works as well
(without the note about installing pciutils with USE=-zlib). The only
download until this transition is finished is using an extra 180Kb on
disk for the compressed versions (120Kb for the pci.id.gz and 60Kb for
usb.ids.gz).

Attached is my patch that has pci.ids stored in both styles (also cleans
up old .gz files if compression is not being used). I've got it ready to
commit to the tree with pciutils-2.2.8, and similarly for usb.ids
(pending an ongoing discussion with upstream usbutils about their
server-side usb.ids).

If there is no major dissent when I get up tomorrow, I'll commit this,
as the end to the breakage.

-- 
Robin Hugh Johnson
Gentoo Linux Developer & Infra Guy
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

[-- Attachment #1.2: pciutils-2.2.7-update-pciids-both-forms.patch --]
[-- Type: text/plain, Size: 865 bytes --]

diff -Nuar pciutils-2.2.7.orig/update-pciids.sh pciutils-2.2.7/update-pciids.sh
--- pciutils-2.2.7.orig/update-pciids.sh	2007-10-30 16:17:03.253630000 -0800
+++ pciutils-2.2.7/update-pciids.sh	2007-10-30 16:39:46.810463056 -0800
@@ -72,4 +72,18 @@
 	rm -f ${DEST%.gz} ${DEST%.gz}.old
 fi
 
+# if the gzip version exists, but we are not using compressed mode, 
+# erase the gzip version, as it is old and crufty
+if [ -z "$PCI_COMPRESSED_IDS" ] && [ -f $DEST.gz ]; then
+	rm -f ${DEST}.gz
+fi
+
+# Lastly, because HAL still reads the plaintext version only and 
+# has not yet migrated to using a library we leave the plaintext 
+# version in addition to the gzip version.
+# They need to migrate because the pci.ids format will be changing.
+if [ ${DEST%.gz} != ${DEST} ] ; then
+	gzip -d <${DEST} >${DEST%.gz}
+fi
+
 ${quiet} || echo "Done."

[-- Attachment #2: Type: application/pgp-signature, Size: 321 bytes --]

Re: [gentoo-dev] Resolving HAL vs. pciutils/usbutils

[Daniel Drake]

[-- Attachment #1: Type: text/plain, Size: 1633 bytes --]

Robin H. Johnson wrote:
> Heya,
> 
> So now this is not a flamewar.
> 
> Jakub was originally going to complain at me for the upstream usbutils
> adding support for gzipped usb.ids files, but a group of us (myself,
> dsd, jakub, leio, steev) had a discussion about it, and came up with a
> solution that both ends the breakage for direct users (HAL and others),
> and provides forward momentum.
> 
> So firstly, what's the real problem? The original complaint came up
> because HAL expected the uncompressed file to exist as pci.ids, and
> wasn't ready to look at pci.ids.gz. While this caused breakage, it was
> only a warning sign that there was a deeper problem.

I don't feel strongly enough to make an objection to your commit, but I 
think pciutils is doing the right thing, and despite me and Mike putting 
a hours into getting a decent HAL patch together the response I got was 
that as upstream they are simply "not interested" (no technical or 
logical objections provided), so I don't feel you should be putting 
workarounds in pciutils just to make HAL happy.

Especially because HAL really doesn't use pci.ids for anything useful. I 
am attaching a HAL ebuild patch which is the approach I'm in favour of 
and first mentioned several months ago. It does not require any HAL 
patches or pciutils modifications. It stems from the fact that really 
HAL doesn't really do anything useful with the ID-to-name mappings 
provided in pci.ids. It makes "that HAL bug" disappear with the click of 
the fingers. I didn't really get any proper answer why our HAL 
maintainers weren't keen on this when I first mentioned it.

Daniel

[-- Attachment #2: hal-ebuild.patch --]
[-- Type: text/plain, Size: 1578 bytes --]

--- hal-0.5.9-r1.ebuild.orig	2007-10-31 10:34:34.000000000 +0000
+++ hal-0.5.9-r1.ebuild	2007-10-31 10:46:15.000000000 +0000
@@ -80,13 +80,6 @@ function notify_inotify() {
 }
 
 pkg_setup() {
-	if ! built_with_use --missing false sys-apps/pciutils hal ; then
-		if built_with_use --missing false sys-apps/pciutils zlib ; then
-			eerror "You MUST build sys-apps/pciutils without the zlib USE flag"
-			die "You MUST build sys-apps/pciutils without the zlib USE flag"
-		fi
-	fi
-
 	if use kernel_linux; then
 		kernel_is ge 2 6 17 || ewarn "HAL requires a kernel version 2.6.17 or newer"
 
@@ -147,6 +140,7 @@ src_unpack() {
 src_compile() {
 	local backend=""
 	local acpi=""
+	local myconf=""
 
 	# TODO :: policykit should have a pam useflag
 	append-flags -rdynamic
@@ -164,6 +158,15 @@ src_compile() {
 		acpi="--disable-acpi-proc --disable-acpi-acpid"
 	fi
 
+	if [[ ! -e "${ROOT}"/usr/share/misc/pci.ids ]]; then
+		myconf="--disable-pci-ids"
+		elog "It looks like you've built pciutils with the zlib USE flag"
+		elog "meaning that your /usr/share/misc/pci.ids file is compressed"
+		elog "and incompatible with HAL. You almost certainly won't notice "
+		elog "any feature loss here, but if you do, just re-emerge pciutils "
+		elog "without the zlib flag, then re-emerge hal."
+	fi
+
 	econf --disable-policy-kit \
 		  --docdir=/usr/share/doc/${PF} \
 		  --with-os-type=gentoo \
@@ -182,6 +185,7 @@ src_compile() {
 		  $(use_enable selinux) \
 		  --disable-console-kit \
 		  ${acpi} \
+		  $myconf \
 	|| die "configure failed"
 #$(use_enable pam console-kit)
 

[gentoo-dev] Re: Resolving HAL vs. pciutils/usbutils

[Guilherme Amadio]

 Hi guys,

 I totally agree with the patch sent by Daniel in the previous message.
 I think it clarifies to the user (me, for example), why it's not possible
 to build pciutils with the zlib USE flag. This is just my humble opinion,
 though, as a user.

 And sorry to a bit off this thread, but I also would like to help with
 some translations of official docs and development. I've been using Gentoo
 since 1.4, but never really had time to help. Now I feel I'll have more
 time and, if you can point me to some Brazilian dev, if there are any, I
 can take some stuff to do.

 Cheers,

 Guilherme

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Re: Resolving HAL vs. pciutils/usbutils

[Jan Kundrát]

[-- Attachment #1: Type: text/plain, Size: 673 bytes --]

Guilherme Amadio wrote:
>  And sorry to a bit off this thread, but I also would like to help with
>  some translations of official docs and development. I've been using Gentoo
>  since 1.4, but never really had time to help. Now I feel I'll have more
>  time and, if you can point me to some Brazilian dev, if there are any, I
>  can take some stuff to do.

Hi Guilherme,
thanks for your interest, but this isn't really appropriate for this
list. Please see [1] and [2].

[1] http://www.gentoo.org/proj/en/gdp/doc/translators-howto.xml
[2] http://www.gentoo.org/proj/en/gdp/doc/doc-policy.xml

Cheers,
-jkt

-- 
cd /local/pub && more beer > /dev/mouth


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

[gentoo-dev] Re: Resolving HAL vs. pciutils/usbutils

[Ryan Hill]

Daniel Drake wrote:

> +	if [[ ! -e "${ROOT}"/usr/share/misc/pci.ids ]]; then
> +		myconf="--disable-pci-ids"

- don't use ${ROOT} outside of pkg_*
- en/disabling functionality based on existence of files is kinda
gross, especially when based on what exists on the compiling system,
which could be different than the system the package is ultimately
installed on.  yes it doesn't make much difference right now, but
what happens if hal does start making good use of pci.ids someday?

having pciutils install both a compressed and uncompressed file when
zlib is enabled is a much cleaner solution IMO.

-- 
                  fonts / wxWindows / gcc-porting / treecleaners
  EFFD 380E 047A 4B51 D2BD  C64F 8AA8 8346 F9A4 0662 (0xF9A40662)

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Resolving HAL vs. pciutils/usbutils

[Daniel Drake]

Wulf C. Krueger wrote:
> The question is not if some software is doing the right thing or not but 
> if our packages behave like they should for our users.

There is also value in satisfying and not deviating away from upstream, 
as well as respecting values of upstream decisions (such as offering 
compressed IDs to save bandwidth and disk space). But yes, the correct 
software behaviour is useful too, and I wouldn't be pushing a solution 
that caused a degradation in user experience.

> Neither is the question if any of us are happy but if our *users* are 
> happy when their installation process breaks because of "that HAL bug". 
> We don't make HAL, its upstream or anyone but our users happy. Our 
> obligation is primarily to them.

pciutils has an upstream too.

>> I am attaching a HAL ebuild patch which is the approach
> 
> ... that still potentially allows things to break because of animosities 
> among ourselves.

HAL handles the missing file condition at runtime if it was compiled 
with support for it. So, there will be no breakage under circumstances 
where the package was built for a different box. No issue here.

> We have a good solution, namely robbat2's, which will make sure things 
> don't break for our users. IMO, that's the way to go because the other 
> approaches make us look bad and are unworthy of a distribution that 
> wants to be taken seriously.

Things already work for the users with the hal useflag for pciutils, and 
things will also work with my patch in a slightly nicer/more robust way, 
without having to extend the HAL issue to the pciutils package.

I'm going to drop out of this discussion here, just wanted to point out 
that there is both technical reasoning behind my suggestion, no 
technical flaws that I know of, and no degradation in user experience. 
Only in distant corner cases would someone notice a difference, and the 
"fix" is easy and documented by the ebuild messages.

Daniel
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Resolving HAL vs. pciutils/usbutils

[Wulf C. Krueger]

[-- Attachment #1: Type: text/plain, Size: 1273 bytes --]

Hello Daniel!

> I don't feel strongly enough to make an objection to your commit,  
> but I think pciutils is doing the right thing,

The question is not if some software is doing the right thing or not  
but if our packages behave like they should for our users.

> and despite me and Mike putting a hours into getting a decent HAL  
> patch together the response I got was that as upstream they are  
> simply "not interested" (no technical or logical objections  
> provided), so I don't feel you should be putting workarounds in  
> pciutils just to make HAL happy.

Neither is the question if any of us are happy but if our *users* are  
happy when their installation process breaks because of "that HAL  
bug". We don't make HAL, its upstream or anyone but our users happy.  
Our obligation is primarily to them.

> I am attaching a HAL ebuild patch which is the approach

... that still potentially allows things to break because of  
animosities among ourselves.

We have a good solution, namely robbat2's, which will make sure things  
don't break for our users. IMO, that's the way to go because the other  
approaches make us look bad and are unworthy of a distribution that  
wants to be taken seriously.

-- 
Best regards, Wulf

[-- Attachment #2: PGP Digital Signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Resolving HAL vs. pciutils/usbutils

[Doug Goldstein]

Daniel Drake wrote:
> Robin H. Johnson wrote:
>> Heya,
>>
>> So now this is not a flamewar.
>>
>> Jakub was originally going to complain at me for the upstream usbutils
>> adding support for gzipped usb.ids files, but a group of us (myself,
>> dsd, jakub, leio, steev) had a discussion about it, and came up with a
>> solution that both ends the breakage for direct users (HAL and others),
>> and provides forward momentum.
>>
>> So firstly, what's the real problem? The original complaint came up
>> because HAL expected the uncompressed file to exist as pci.ids, and
>> wasn't ready to look at pci.ids.gz. While this caused breakage, it was
>> only a warning sign that there was a deeper problem.
>
> I don't feel strongly enough to make an objection to your commit, but
> I think pciutils is doing the right thing, and despite me and Mike
> putting a hours into getting a decent HAL patch together the response
> I got was that as upstream they are simply "not interested" (no
> technical or logical objections provided), so I don't feel you should
> be putting workarounds in pciutils just to make HAL happy.
>
>
>
> Daniel
Daniel,

That's a LOAD of garbage and you know it. You're just straight away
making up stuff, essentially lying. You know damn well what the reasons
were since they were explained to you on numerous occasions.

When HAL evaluated the usage of  libpci the following issues were
identified:
 1) increased memory usage, to the point that HAL was not usable on the
OLPC project
 2) ABI breakage between patch revisions (i.e. x.y.z and x.y.z+1 were
not ABI compatible)
 3) no shared library
 4) the library calls exit() when it encounters an error in parsing it's
own pci.ids file which would kill the whole app using it.

There might have been more. I don't remember. Refer to ML discussions
and refer to IRC logs with me.

Now Mike (vapier) rectified #4 several pciutils releases ago by
providing a callback function that we could define which would override
the default exit() behavior. I still think it's sub-par to have an
utility library call exit() by default but whatever.

You were told by me and the HAL ML that once #2 and #3 are rectified and
if you could provide some basic memory usage information along with your
patch (i.e. show #1 isn't true anymore) that we would happily accept
your patch.

Now #2 and #3 are still not true in the latest release. There is no
guarantee by the pciutils maintainers that they will maintain ABI
compatibility while keeping the same .so version number. And there is
still no shared library built.

You addressed #1 on the mailing list with a simple statement, which I
will paraphrase. "It doesn't use more memory on my machine". To which
Danny K asked if you could provide some basic data behind that and you
never did.

As a result, 3 out of 4 concerns with your patch and pciutils were never
addressed and the issue was dropped on the HAL ML pending more feedback.
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Resolving HAL vs. pciutils/usbutils

[Roy Marples]

On Wed, 2007-10-31 at 11:40 -0400, Doug Goldstein wrote:
> When HAL evaluated the usage of  libpci the following issues were
> identified:
>  1) increased memory usage, to the point that HAL was not usable on the
> OLPC project
>  2) ABI breakage between patch revisions (i.e. x.y.z and x.y.z+1 were
> not ABI compatible)
>  3) no shared library
>  4) the library calls exit() when it encounters an error in parsing it's
> own pci.ids file which would kill the whole app using it.
> 
> There might have been more. I don't remember. Refer to ML discussions
> and refer to IRC logs with me.

Begs the question why does HAL use libpci in the first place.

Thanks

Roy

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Resolving HAL vs. pciutils/usbutils

[Daniel Drake]

Doug Goldstein wrote:
> When HAL evaluated the usage of  libpci the following issues were
> identified:
>  1) increased memory usage, to the point that HAL was not usable on the
> OLPC project

I was only ever aware of concerns that memory usage might be high, but 
wasn't aware it caused specific problems.

I went through the first 3 pages of google results for
	"pciutils inurl:hal site:lists.freedesktop.org"
	"libpci inurl:hal site:lists.freedesktop.org"
and didn't see anything. Maybe it was discussed elsewhere.

Anyway, if this did happen once, it doesn't seem to happen any more, see 
below.

>  2) ABI breakage between patch revisions (i.e. x.y.z and x.y.z+1 were
> not ABI compatible)

This doesn't matter when you statically link against the library, as 
long as the API doesn't change. The API that is used in Mike's patch 
does not seem to have changed for a long time. (Nevertheless, see my 
notes under the following item -- this will be solved when the next one 
is solved.)

>  3) no shared library

This is a fair point, but I thought it was never raised as an objection, 
I didn't think it was actually a blocker for acceptance. Especially 
given that parts of HAL already statically link against libpci.

I just looked over the threads again and I now see this:
http://lists.freedesktop.org/archives/hal/2007-June/008836.html

I apologise, I must have missed that before.
OK, so having a dynamic libpci is an outstanding requirement for the 
patch. I will follow up with pciutils upstream about the current state 
of that.

>  4) the library calls exit() when it encounters an error in parsing it's
> own pci.ids file which would kill the whole app using it.
> 
> There might have been more. I don't remember. Refer to ML discussions
> and refer to IRC logs with me.

I looked over them, I don't see any others.

> Now Mike (vapier) rectified #4 several pciutils releases ago by
> providing a callback function that we could define which would override
> the default exit() behavior. I still think it's sub-par to have an
> utility library call exit() by default but whatever.

Yeah.

> You were told by me and the HAL ML that once #2 and #3 are rectified and
> if you could provide some basic memory usage information along with your
> patch (i.e. show #1 isn't true anymore) that we would happily accept
> your patch.

> You addressed #1 on the mailing list with a simple statement, which I
> will paraphrase. "It doesn't use more memory on my machine". To which
> Danny K asked if you could provide some basic data behind that and you
> never did.

I did:
http://lists.freedesktop.org/archives/hal/2007-June/008852.html
http://lists.freedesktop.org/archives/hal/2007-June/008861.html


Anyway, apologies for the oversight on the shared library thing -- it 
appears it wasn't total silent rejection after all. I'll let you know 
where that leads.

Thanks,
Daniel
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Resolving HAL vs. pciutils/usbutils

[Jan Kundrát]

[-- Attachment #1: Type: text/plain, Size: 396 bytes --]

Daniel Drake wrote:
> OK, so having a dynamic libpci is an outstanding requirement for the
> patch. I will follow up with pciutils upstream about the current state
> of that.

If you had any issues with Martin Mares, I can talk to him as he's my
teacher in one course at the university. He looks like a reasonable person.

Cheers,
-jkt

-- 
cd /local/pub && more beer > /dev/mouth


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: [gentoo-dev] Resolving HAL vs. pciutils/usbutils

[Rémi Cardona]

Roy Marples a écrit :
> Begs the question why does HAL use libpci in the first place.

2 reasons (that I know of) :

1) to make things pretty in lshal
2) to make writing FDI files somewhat less cryptic

http://gitweb.freedesktop.org/?p=hal.git;a=blob;f=hald/linux/device.c#l1554

Rémi
-- 
gentoo-dev@gentoo.org mailing list