[gentoo-dev] RFC: lzma tarball usage

[gentoo-dev] RFC: lzma tarball usage

[Mart Raudsepp]

[-- Attachment #1: Type: text/plain, Size: 1567 bytes --]

Hello,

Over the course of this year, a lzma-utils buildtime dependency has been
added to a few system packages, to handle .tar.lzma tarballs.
This has huge implications on the requirement of the system toolchain,
which is highly disturbing from a minimal (lets say embedded) systems
concern - lzma-utils depends on the C++ compiler and the libstdc++
beast, while a minimal system would like to avoid this at all cost.

I do realize one would remove build-time dependencies and the toolchain
on an embedded system on deployment anyway, but this means gcc USE=nocxx
USE flag is pretty much useless, while it would be nice to use it to
ensure that nothing sneaks in during development that depends on the C++
standard library easily instead of finding things break later.

This is a plea and also a request for comments on the matter of
using .tar.lzma tarballs or not, and for what packages this is
acceptable and for what not.

I'd be happy if some other unpacker is used than lzma-utils - one that
does not depend on libstdc++ - I'm sure it can be done, heck it's done
in integrated form in some other projects in less than a couple
kilobytes of code for the unpacking from a VFS. Meanwhile please
consider using the upstream provided .tar.gz tarballs instead and not
roll patchsets in .lzma just cause you can.

coreutils and linux-headers come to my mind out of system packages right
now. I'm sure more dragons await me.


-- 
Mart Raudsepp
Gentoo Developer
Mail: leio@gentoo.org
Weblog: http://planet.gentoo.org/developers/leio

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] RFC: lzma tarball usage

[Fabian Groffen]

On 07-05-2008 16:23:12 +0300, Mart Raudsepp wrote:
> This is a plea and also a request for comments on the matter of
> using .tar.lzma tarballs or not, and for what packages this is
> acceptable and for what not.

Just as a little background:
GNU chose to switch from bzip2 to lzma, for it produces smaller files
(less bandwith) and decompresses faster.

They no longer provide the bzip2 versions of archives for newer releases
IIRC, so it's either tar.gz or tar.lzma.

> I'd be happy if some other unpacker is used than lzma-utils - one that
> does not depend on libstdc++ - I'm sure it can be done, heck it's done
> in integrated form in some other projects in less than a couple
> kilobytes of code for the unpacking from a VFS. Meanwhile please
> consider using the upstream provided .tar.gz tarballs instead and not
> roll patchsets in .lzma just cause you can.

See above why it might not just be "'cause you can".

> coreutils and linux-headers come to my mind out of system packages right
> now. I'm sure more dragons await me.

m4, that one gave me some headaches, because lzma-utils required some
eautoreconf, which introduced a nice cycle.


-- 
Fabian Groffen
Gentoo on a different level
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] RFC: lzma tarball usage

[Mart Raudsepp]

[-- Attachment #1: Type: text/plain, Size: 2409 bytes --]

On K, 2008-05-07 at 15:34 +0200, Fabian Groffen wrote:
> On 07-05-2008 16:23:12 +0300, Mart Raudsepp wrote:
> > This is a plea and also a request for comments on the matter of
> > using .tar.lzma tarballs or not, and for what packages this is
> > acceptable and for what not.
> 
> Just as a little background:
> GNU chose to switch from bzip2 to lzma, for it produces smaller files
> (less bandwith) and decompresses faster.
> 
> They no longer provide the bzip2 versions of archives for newer releases
> IIRC, so it's either tar.gz or tar.lzma.
> 
> > I'd be happy if some other unpacker is used than lzma-utils - one that
> > does not depend on libstdc++ - I'm sure it can be done, heck it's done
> > in integrated form in some other projects in less than a couple
> > kilobytes of code for the unpacking from a VFS. Meanwhile please
> > consider using the upstream provided .tar.gz tarballs instead and not
> > roll patchsets in .lzma just cause you can.
> 
> See above why it might not just be "'cause you can".

"and not roll patchsets in .lzma just cause you can". Cause you can
applies to patchsets mostly. But using .tar.lzma instead of .tar.gz is
also a "because they are available and therefore I can use it"
neglecting the issues of

a) on-disk format is supposedly not even finalized; high potential
breakage of packages in existing ebuilds once lzma-utils gets updated
b) The currently used decompressor package links to libstdc++ (and
portage uses lzma, not lzmadec) unconditionally for most components
c) Potential security issues; details needed, but for other reasons it
makes sense to ban .tar.lzma's until a new C only rewritten lzma-utils
comes along anyway
d) too early adoption in critical system packages - once above issues
are solved, higher levels should be using it first, before critical
system packages (for example shows in the circular dep hell with m4)
e) It has been suggested the support should have been added with new
EAPI instead of local build deps (some of which are missing, for
instance in the hand-rolled for-no-reason-whatsoever .tar.lzma format
net-tools doesn't have a dep in addition to using lzma for no good
reason)

Probably some more.
Base-system, please stop using .tar.lzma for now, thank you.


-- 
Mart Raudsepp
Gentoo Developer
Mail: leio@gentoo.org
Weblog: http://planet.gentoo.org/developers/leio

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] RFC: lzma tarball usage

[Fabian Groffen]

On 08-05-2008 21:45:00 +0300, Mart Raudsepp wrote:
> d) too early adoption in critical system packages - once above issues
> are solved, higher levels should be using it first, before critical
> system packages (for example shows in the circular dep hell with m4)

been there, done that.

> e) It has been suggested the support should have been added with new
> EAPI instead of local build deps (some of which are missing, for
> instance in the hand-rolled for-no-reason-whatsoever .tar.lzma format
> net-tools doesn't have a dep in addition to using lzma for no good
> reason)

Chill, relax and cool down.  Instead, just ask those who decided to
follow upstream why and if they have even thought about the issues you
brought up.


-- 
Fabian Groffen
Gentoo on a different level
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] RFC: lzma tarball usage

[Santiago M. Mola]

On Thu, May 8, 2008 at 9:09 PM, Fabian Groffen <grobian@gentoo.org> wrote:
>
>  > e) It has been suggested the support should have been added with new
>  > EAPI instead of local build deps (some of which are missing, for
>  > instance in the hand-rolled for-no-reason-whatsoever .tar.lzma format
>  > net-tools doesn't have a dep in addition to using lzma for no good
>  > reason)
>
>  Chill, relax and cool down.  Instead, just ask those who decided to
>  follow upstream why and if they have even thought about the issues you
>  brought up.
>

Note that we're also speaking about downstream lzma archives. Like in
sys-apps/net-tools, where lzma hasn't been adopted even by upstream.

Regards,
-- 
Santiago M. Mola
Jabber ID: cooldwind@gmail.com
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] RFC: lzma tarball usage

[Mart Raudsepp]

[-- Attachment #1: Type: text/plain, Size: 1253 bytes --]

On N, 2008-05-08 at 21:09 +0200, Fabian Groffen wrote:
> > e) It has been suggested the support should have been added with new
> > EAPI instead of local build deps (some of which are missing, for
> > instance in the hand-rolled for-no-reason-whatsoever .tar.lzma format
> > net-tools doesn't have a dep in addition to using lzma for no good
> > reason)
> 
> Chill, relax and cool down.

Well, I said how it is. I don't see anything in it that indicates I am
so upset and angry that I need to do these things. I did however loose
hours of work time, but that's life.

> Instead, just ask those who decided to
> follow upstream why and if they have even thought about the issues you
> brought up.

This is what I am doing with this as well, in addition to the bug
reports. But as this is widespread to at least 4-6 system packages, I
brought it up here as well to ensure this is not something I have to
fight against in overlays and time wastes continuously in the future.
Oh and net-tools has not distributed anything in .tar.lzma, so this has
nothing to do with following upstream in any shape or form in this case.

-- 
Mart Raudsepp
Gentoo Developer
Mail: leio@gentoo.org
Weblog: http://planet.gentoo.org/developers/leio

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] RFC: lzma tarball usage

[Enrico Weigelt]

* Mart Raudsepp <leio@gentoo.org> schrieb:

<big_snip />

IMHO, lzma is far from being mature enough from being suited as 
packaging format for production systems. And actually, I don't 
see the benefit over well-approved tar+(gz|bz2). 

So my vote is to NOT use it for gentoo source packages.


cu
-- 
---------------------------------------------------------------------
 Enrico Weigelt    ==   metux IT service - http://www.metux.de/
---------------------------------------------------------------------
 Please visit the OpenSource QM Taskforce:
 	http://wiki.metux.de/public/OpenSource_QM_Taskforce
 Patches / Fixes for a lot dozens of packages in dozens of versions:
	http://patches.metux.de/
---------------------------------------------------------------------
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] RFC: lzma tarball usage

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 243 bytes --]

On Wednesday 07 May 2008, Fabian Groffen wrote:
> m4, that one gave me some headaches, because lzma-utils required some
> eautoreconf, which introduced a nice cycle.

must have been a prefix-only bug as the version in the tree never did
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 827 bytes --]

Re: [gentoo-dev] RFC: lzma tarball usage

[Fabian Groffen]

On 10-05-2008 03:32:08 -0400, Mike Frysinger wrote:
> On Wednesday 07 May 2008, Fabian Groffen wrote:
> > m4, that one gave me some headaches, because lzma-utils required some
> > eautoreconf, which introduced a nice cycle.
> 
> must have been a prefix-only bug as the version in the tree never did

Ehmm... you're right.  Sorry about that.


-- 
Fabian Groffen
Gentoo on a different level
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] RFC: lzma tarball usage

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 527 bytes --]

On Saturday 10 May 2008, Fabian Groffen wrote:
> On 10-05-2008 03:32:08 -0400, Mike Frysinger wrote:
> > On Wednesday 07 May 2008, Fabian Groffen wrote:
> > > m4, that one gave me some headaches, because lzma-utils required some
> > > eautoreconf, which introduced a nice cycle.
> >
> > must have been a prefix-only bug as the version in the tree never did
>
> Ehmm... you're right.  Sorry about that.

ive added a comment to the ebuild so as to future proof me or anyone else 
forgetting about this gotcha.
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 827 bytes --]

Re: [gentoo-dev] RFC: lzma tarball usage

[Natanael Copa]

On Wed, 2008-05-07 at 16:23 +0300, Mart Raudsepp wrote:

> I'd be happy if some other unpacker is used than lzma-utils - one that
> does not depend on libstdc++ - I'm sure it can be done, heck it's done
> in integrated form in some other projects in less than a couple
> kilobytes of code for the unpacking from a VFS. Meanwhile please
> consider using the upstream provided .tar.gz tarballs instead and not
> roll patchsets in .lzma just cause you can.

busybox has unlzma and seems to be a part of "system".

Should also be easy to create a really tiny unlzma from the busybox
source and ship with portage, or create a patch for tar or something.

-nc


-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] RFC: lzma tarball usage

[Ulrich Mueller]

>>>>> On Wed, 07 May 2008, Natanael Copa wrote:

> busybox has unlzma and seems to be a part of "system".

> Should also be easy to create a really tiny unlzma from the busybox
> source and ship with portage, or create a patch for tar or something.

The decoder of lzma-utils is also written in C only.

So it would also be possible to compile "lzmadec" without any need
for C++. Just call "make" in subdirs liblzmadec and lzmadec.

Ulrich
-- 
gentoo-dev@lists.gentoo.org mailing list

[gentoo-dev] Re: RFC: lzma tarball usage

[Duncan]

Ulrich Mueller <ulm@gentoo.org> posted
18465.49899.138685.587639@a1i15.kph.uni-mainz.de, excerpted below, on 
Wed, 07 May 2008 16:55:39 +0200:

> The decoder of lzma-utils is also written in C only.
> 
> So it would also be possible to compile "lzmadec" without any need for
> C++. Just call "make" in subdirs liblzmadec and lzmadec.

What about USE=decode-only or something similar for lzma-utils, then?  If 
desired, it could even be masked on "normal" profiles, but would then be 
there for the embedded and releng folks.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

-- 
gentoo-dev@lists.gentoo.org mailing list

[gentoo-dev] Re: RFC: lzma tarball usage

[Diego 'Flameeyes' Pettenò]

[-- Attachment #1: Type: text/plain, Size: 833 bytes --]

Duncan <1i5t5.duncan@cox.net> writes:

>> So it would also be possible to compile "lzmadec" without any need for
>> C++. Just call "make" in subdirs liblzmadec and lzmadec.
>
> What about USE=decode-only or something similar for lzma-utils, then?  If 
> desired, it could even be masked on "normal" profiles, but would then be 
> there for the embedded and releng folks.

USE=cxx should do just fine, it will disable the C++-related parts,
whatever they are. Sincerely I'd quite like to enable it on my vserver's
build chroots too.

(BTW I considered using lzma for backup compression, but I didn't get
much different results from bzip2 in term of size, but took quite longer
in case of compression... I still have some doubts whether lzma is worth
it).

-- 
Diego "Flameeyes" Pettenò
http://blog.flameeyes.eu/

[-- Attachment #2: Type: application/pgp-signature, Size: 196 bytes --]

Re: [gentoo-dev] Re: RFC: lzma tarball usage

[Graham Murray]

flameeyes@gmail.com (Diego 'Flameeyes' Pettenò) writes:

> USE=cxx should do just fine, it will disable the C++-related parts,
> whatever they are. Sincerely I'd quite like to enable it on my vserver's
> build chroots too.

Should that be USE=-cxx? The help for USE=cxx says that this builds
support for C++.
--
gentoo-dev@lists.gentoo.org mailing list

[gentoo-dev] Re: RFC: lzma tarball usage

[Diego 'Flameeyes' Pettenò]

[-- Attachment #1: Type: text/plain, Size: 331 bytes --]

Graham Murray <graham@gmurray.org.uk> writes:

> Should that be USE=-cxx? The help for USE=cxx says that this builds
> support for C++.

It was meant as setting a cxx USE on the ebuild, I wasn't certainly
meaning to disable the C++ parts with USE=cxx enabled ;)

-- 
Diego "Flameeyes" Pettenò
http://blog.flameeyes.eu/

[-- Attachment #2: Type: application/pgp-signature, Size: 196 bytes --]

[gentoo-dev] Re: RFC: lzma tarball usage

[Ulrich Mueller]

>>>>> On Thu, 08 May 2008, Diego 'Flameeyes' Pettenò wrote:

>>> So it would also be possible to compile "lzmadec" without any need
>>> for C++. Just call "make" in subdirs liblzmadec and lzmadec.
>> 
>> What about USE=decode-only or something similar for lzma-utils,
>> then? If desired, it could even be masked on "normal" profiles, but
>> would then be there for the embedded and releng folks.

> USE=cxx should do just fine, it will disable the C++-related parts,
> whatever they are. Sincerely I'd quite like to enable it on my
> vserver's build chroots too.

See <https://bugs.gentoo.org/show_bug.cgi?id=220899> for a first
attempt of an ebuild.

Ulrich
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] RFC: lzma tarball usage

[Benedikt Morbach]

On Wed, May 7, 2008 at 3:23 PM, Mart Raudsepp <leio@gentoo.org> wrote:
>  I'd be happy if some other unpacker is used than lzma-utils - one that
>  does not depend on libstdc++ - I'm sure it can be done, heck it's done
>  in integrated form in some other projects in less than a couple
>  kilobytes of code for the unpacking from a VFS. Meanwhile please
>  consider using the upstream provided .tar.gz tarballs instead and not
>  roll patchsets in .lzma just cause you can.

tar-1.20 has lzma support, so maybe it could handle this too, once it
goes into stable
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] RFC: lzma tarball usage

[Ulrich Mueller]

>>>>> On Wed, 7 May 2008, Benedikt Morbach wrote:

> tar-1.20 has lzma support, so maybe it could handle this too, once it
> goes into stable

This doesn't help, since it needs the lzma binary as a filter.

Ulrich
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] RFC: lzma tarball usage

[Benedikt Morbach]

Hi,
I sent this to -dev to, but I think as an ordinary user I can't write there...

On Wed, May 7, 2008 at 3:23 PM, Mart Raudsepp <leio@gentoo.org> wrote:
>  I'd be happy if some other unpacker is used than lzma-utils - one that
>  does not depend on libstdc++ - I'm sure it can be done, heck it's done
>  in integrated form in some other projects in less than a couple
>  kilobytes of code for the unpacking from a VFS. Meanwhile please
>  consider using the upstream provided .tar.gz tarballs instead and not
>  roll patchsets in .lzma just cause you can.

tar-1.20 has lzma support, so maybe it could handle this too, once it
goes into stable.
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] RFC: lzma tarball usage

[Chris Gianelloni]

[-- Attachment #1: Type: text/plain, Size: 974 bytes --]

On Wed, 2008-05-07 at 16:23 +0300, Mart Raudsepp wrote:
> I do realize one would remove build-time dependencies and the toolchain
> on an embedded system on deployment anyway, but this means gcc USE=nocxx
> USE flag is pretty much useless, while it would be nice to use it to
> ensure that nothing sneaks in during development that depends on the C++
> standard library easily instead of finding things break later.

It's a pain in the ass for Release Engineering, too.  At this point,
we're looking into how we need to modify the bootstrap sequence to
accommodate people using lzma for system (and lower) packages.

http://bugs.gentoo.org/show_bug.cgi?id=220074

We're already getting reports of this due to someone deciding that it'd
be a good idea to use lzma for our daily portage snapshots without any
discussion here.  Luckily, we still have the other tarballs to use, too.

-- 
Chris Gianelloni
Release Engineering Strategic Lead
Games Developer

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] RFC: lzma tarball usage

[Enrico Weigelt]

Hi,


I think, as long as there is no really minimal lzmadec available
yet (as standalone package), we should more standard compressors
like gzip or bzip2. Adding that whole bunch of deps just to 
save a few bytes IMHO isn't worth it.


cu
-- 
---------------------------------------------------------------------
 Enrico Weigelt    ==   metux IT service - http://www.metux.de/
---------------------------------------------------------------------
 Please visit the OpenSource QM Taskforce:
 	http://wiki.metux.de/public/OpenSource_QM_Taskforce
 Patches / Fixes for a lot dozens of packages in dozens of versions:
	http://patches.metux.de/
---------------------------------------------------------------------
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] RFC: lzma tarball usage

[Richard Freeman]

Enrico Weigelt wrote:
> I think, as long as there is no really minimal lzmadec available
> yet (as standalone package), we should more standard compressors
> like gzip or bzip2. Adding that whole bunch of deps just to 
> save a few bytes IMHO isn't worth it.

Keep in mind that this might mean doing our own repackaging of upstream 
if they don't have a supported option.  I think the only other option 
would be to create an "lzmalite" package or something like that which 
simply contains the decompressor in ordinary C.  You could really turn 
that into a separate package like gentoolkit or whatever - I wouldn't 
actually embed the code into portage since that isn't the unix way and 
it just forced other package managers (and other distros) to do the same 
thing.  An lzmalite package could have a life of its own and as a result 
benefit from fewer bugs/etc.

But, I'm not going to be the one writing the thing, so feel free to not 
listen to any of this...  :)
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] RFC: lzma tarball usage

[Doug Goldstein]

Richard Freeman wrote:
> Enrico Weigelt wrote:
>> I think, as long as there is no really minimal lzmadec available
>> yet (as standalone package), we should more standard compressors
>> like gzip or bzip2. Adding that whole bunch of deps just to save a 
>> few bytes IMHO isn't worth it.
>
> Keep in mind that this might mean doing our own repackaging of 
> upstream if they don't have a supported option.  I think the only 
> other option would be to create an "lzmalite" package or something 
> like that which simply contains the decompressor in ordinary C.  You 
> could really turn that into a separate package like gentoolkit or 
> whatever - I wouldn't actually embed the code into portage since that 
> isn't the unix way and it just forced other package managers (and 
> other distros) to do the same thing.  An lzmalite package could have a 
> life of its own and as a result benefit from fewer bugs/etc.
>
> But, I'm not going to be the one writing the thing, so feel free to 
> not listen to any of this...  :)
All upstreams in question still use gzip, they have only dropped bzip2 
support in favor of lzma.
-- 
gentoo-dev@lists.gentoo.org mailing list

[gentoo-dev] Re: RFC: lzma tarball usage

[Ryan Hill]

[-- Attachment #1: Type: text/plain, Size: 1099 bytes --]

On Wed, 07 May 2008 16:23:12 +0300
Mart Raudsepp <leio@gentoo.org> wrote:

> Hello,
> 
> Over the course of this year, a lzma-utils buildtime dependency has
> been added to a few system packages, to handle .tar.lzma tarballs.
> This has huge implications on the requirement of the system toolchain,
> which is highly disturbing from a minimal (lets say embedded) systems
> concern - lzma-utils depends on the C++ compiler and the libstdc++
> beast, while a minimal system would like to avoid this at all cost.

The new lzma-utils codebase uses liblzma, written in C.  It's at the
alpha stage but supposedly supports encoding/decoding the current lzma
format "well enough" (;P).  It probably has some fun bugs to find
and squish.

http://sf.net/mailarchive/forum.php?thread_name=200804251652.58484.lasse.collin%40tukaani.org&forum_name=lzmautils-announce

-- 
fonts, gcc-porting,                               by design, by neglect
mips, treecleaner,                        for a fact or just for effect
wxwidgets @ gentoo     EFFD 380E 047A 4B51 D2BD C64F 8AA8 8346 F9A4 0662

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Re: RFC: lzma tarball usage

[Doug Goldstein]

Ryan Hill wrote:
> On Wed, 07 May 2008 16:23:12 +0300
> Mart Raudsepp <leio@gentoo.org> wrote:
>
>   
>> Hello,
>>
>> Over the course of this year, a lzma-utils buildtime dependency has
>> been added to a few system packages, to handle .tar.lzma tarballs.
>> This has huge implications on the requirement of the system toolchain,
>> which is highly disturbing from a minimal (lets say embedded) systems
>> concern - lzma-utils depends on the C++ compiler and the libstdc++
>> beast, while a minimal system would like to avoid this at all cost.
>>     
>
> The new lzma-utils codebase uses liblzma, written in C.  It's at the
> alpha stage but supposedly supports encoding/decoding the current lzma
> format "well enough" (;P).  It probably has some fun bugs to find
> and squish.
>
> http://sf.net/mailarchive/forum.php?thread_name=200804251652.58484.lasse.collin%40tukaani.org&forum_name=lzmautils-announce
>
>   
According to the mailing list this change was done to fix security holes 
in the format and also resulted in a slightly different format that's 
incompatible with the previous verion. So lzma 5.x and higher will be a 
different on disk format. It's troubling to me that projects are using 
lzma when it's on disk format isn't even final and the project has 
security issues.
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] Re: RFC: lzma tarball usage

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 275 bytes --]

On Thu, 08 May 2008 09:17:08 -0400
Doug Goldstein <cardoe@gentoo.org> wrote:
> It's troubling to me that projects are using lzma when it's on disk
> format isn't even final and the project has security issues.

You mean projects like 'GNU tar'?

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Re: RFC: lzma tarball usage

[Doug Goldstein]

Ciaran McCreesh wrote:
> On Thu, 08 May 2008 09:17:08 -0400
> Doug Goldstein <cardoe@gentoo.org> wrote:
>   
>> It's troubling to me that projects are using lzma when it's on disk
>> format isn't even final and the project has security issues.
>>     
>
> You mean projects like 'GNU tar'?
>
>   
As far as I know Ciaran, all GNU projects have switched or are in the 
process of switching to lzma over bzip2. I believe the issue in question 
which prompted this original e-mail was due to coreutils. But I could be 
wrong.
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] Re: RFC: lzma tarball usage

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 799 bytes --]

On Thu, 08 May 2008 09:32:34 -0400
Doug Goldstein <cardoe@gentoo.org> wrote:
> Ciaran McCreesh wrote:
> > On Thu, 08 May 2008 09:17:08 -0400
> > Doug Goldstein <cardoe@gentoo.org> wrote:
> >> It's troubling to me that projects are using lzma when it's on disk
> >> format isn't even final and the project has security issues.
> >
> > You mean projects like 'GNU tar'?
> >   
> As far as I know Ciaran, all GNU projects have switched or are in the 
> process of switching to lzma over bzip2. I believe the issue in
> question which prompted this original e-mail was due to coreutils.
> But I could be wrong.

You miss my point. GNU tar sometimes changes its on disk format (and
will be doing so again at some point for xattrs), and it's had security
issues.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Re: RFC: lzma tarball usage

[Doug Goldstein]

Ciaran McCreesh wrote:
> On Thu, 08 May 2008 09:32:34 -0400
> Doug Goldstein <cardoe@gentoo.org> wrote:
>   
>> Ciaran McCreesh wrote:
>>     
>>> On Thu, 08 May 2008 09:17:08 -0400
>>> Doug Goldstein <cardoe@gentoo.org> wrote:
>>>       
>>>> It's troubling to me that projects are using lzma when it's on disk
>>>> format isn't even final and the project has security issues.
>>>>         
>>> You mean projects like 'GNU tar'?
>>>   
>>>       
>> As far as I know Ciaran, all GNU projects have switched or are in the 
>> process of switching to lzma over bzip2. I believe the issue in
>> question which prompted this original e-mail was due to coreutils.
>> But I could be wrong.
>>     
>
> You miss my point. GNU tar sometimes changes its on disk format (and
> will be doing so again at some point for xattrs), and it's had security
> issues.
>
>   
Fair enough. However, newer GNU tar's are able to untar the older 
formats. If you read the lzma changelogs, it appears to imply that newer 
ones won't be able to read older formats. The changelog specifically 
states if a user they are handling the issue "gracefully" by telling the 
user to upgrade or downgrade their lzma.
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] Re: RFC: lzma tarball usage

[James Cloos]

>>>>> "Doug" == Doug Goldstein <cardoe@gentoo.org> writes:

Doug> If you read the lzma changelogs, it appears to imply that newer
Doug> ones won't be able to read older formats. The changelog
Doug> specifically states if a user they are handling the issue
Doug> "gracefully" by telling the user to upgrade or downgrade their
Doug> lzma.

My understanding is that the new utils will be able to uncompress
current archives, but will not be able to create them and also will
be unable to convert current archives to the new format w/o a full
(and time-consuming) uncompress/compress cycle.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6
-- 
gentoo-dev@lists.gentoo.org mailing list

[gentoo-dev] Re: RFC: lzma tarball usage

[Diego 'Flameeyes' Pettenò]

[-- Attachment #1: Type: text/plain, Size: 953 bytes --]

Ciaran McCreesh <ciaran.mccreesh@googlemail.com> writes:

> You miss my point. GNU tar sometimes changes its on disk format (and
> will be doing so again at some point for xattrs)

It's not really important to the discussion, but...

The TAR format is designed as such that on disk formats can be extended
without breaking entirely backward compatibility. For what it's worth,
extended attributes and ACLs are already supported by star (Schilling's)
and bsdtar (libarchive). The fact that GNU tar doesn't support them at
the moment is a different problem. On the other hand, even if the GNU
tar does not support reading those attributes, it does handle them
gracefully, warning the user of unknown extended headers, and then
proceeding to unpack the data without preserving the extended
attributes.

So what Doug said stands perfectly and does not interest GNU tar at all.

-- 
Diego "Flameeyes" Pettenò
http://blog.flameeyes.eu/

[-- Attachment #2: Type: application/pgp-signature, Size: 196 bytes --]

Re: [gentoo-dev] Re: RFC: lzma tarball usage

[Doug Goldstein]

Doug Goldstein wrote:
> Ciaran McCreesh wrote:
>> On Thu, 08 May 2008 09:17:08 -0400
>> Doug Goldstein <cardoe@gentoo.org> wrote:
>>  
>>> It's troubling to me that projects are using lzma when it's on disk
>>> format isn't even final and the project has security issues.
>>>     
>>
>> You mean projects like 'GNU tar'?
>>
>>   
> As far as I know Ciaran, all GNU projects have switched or are in the 
> process of switching to lzma over bzip2. I believe the issue in 
> question which prompted this original e-mail was due to coreutils. But 
> I could be wrong.
Additionally to follow myself up, I believe one of the security issues 
was execution of arbitrary data either when untarred or just 
decompressed (assuming a  specially crafted lzma file).

Some of the other fun bits are lzma requires autotools but autotools are 
going to be compressed with lzma. So if we ever need to autoreconf, we 
have a chicken/egg issue.
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] Re: RFC: lzma tarball usage

[Robert Buchholz]

[-- Attachment #1: Type: text/plain, Size: 518 bytes --]

On Thursday 08 May 2008, Doug Goldstein wrote:
> Additionally to follow myself up, I believe one of the security
> issues was execution of arbitrary data either when untarred or just
> decompressed (assuming a  specially crafted lzma file).

Can you please point me to the location where this is mentioned. I read 
through the lzma git log, and I all I could find was data corruption 
(which usually is not a security issue) and the mention of the 
word "security" inside the announcement.

Thanks,
Robert

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

[gentoo-dev] Re: RFC: lzma tarball usage

[Ryan Hill]

[-- Attachment #1: Type: text/plain, Size: 1365 bytes --]

On Thu, 08 May 2008 09:17:08 -0400
Doug Goldstein <cardoe@gentoo.org> wrote:

> Ryan Hill wrote:
> > The new lzma-utils codebase uses liblzma, written in C.  It's at the
> > alpha stage but supposedly supports encoding/decoding the current
> > lzma format "well enough" (;P).  It probably has some fun bugs to
> > find and squish.
> >
> > http://sf.net/mailarchive/forum.php?thread_name=200804251652.58484.lasse.collin%40tukaani.org&forum_name=lzmautils-announce

> According to the mailing list this change was done to fix security
> holes in the format and also resulted in a slightly different format
> that's incompatible with the previous verion. So lzma 5.x and higher
> will be a different on disk format. It's troubling to me that
> projects are using lzma when it's on disk format isn't even final and
> the project has security issues.

The current format is fine.  It's the new format that has
design/security issues.  Yes the formats are incompatible, but so
are .tar.lzma and .7z, which are both lzma.  Either way I was just
offering it as a data point.  I have no real opinion one way or the
other.


-- 
fonts, gcc-porting,                               by design, by neglect
mips, treecleaner,                        for a fact or just for effect
wxwidgets @ gentoo     EFFD 380E 047A 4B51 D2BD C64F 8AA8 8346 F9A4 0662

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] RFC: lzma tarball usage

[Luca Barbato]

Mart Raudsepp wrote:
> Hello,
> 
> Over the course of this year, a lzma-utils buildtime dependency has been
> added to a few system packages, to handle .tar.lzma tarballs.
> This has huge implications on the requirement of the system toolchain,
> which is highly disturbing from a minimal (lets say embedded) systems
> concern - lzma-utils depends on the C++ compiler and the libstdc++
> beast, while a minimal system would like to avoid this at all cost.

I'd rewrite the C++ code in plain C if isn't that complex...

lu

-- 

Luca Barbato
Gentoo Council Member
Gentoo/linux Gentoo/PPC
http://dev.gentoo.org/~lu_zero

-- 
gentoo-dev@lists.gentoo.org mailing list