[gentoo-dev] [RFC] unpacker.eclass extensions

[gentoo-dev] [RFC] unpacker.eclass extensions

[Vadim A. Misbakh-Soloviov]

[-- Attachment #1: Type: text/plain, Size: 1571 bytes --]

As gamerlay maintainer, I'd be glad to introduce some changes to
unpacker.eclass:
1) merging unpacker-nixstaller (Makeself  subspecies) from gamerlay:


# @FUNCTION: unpack_nixstaller
# @USAGE: <files to unpack>
# @DESCRIPTION:
# Unpack nixstaller generated files
# They're shell scripts with the blob package tagged onto
# the end of the archive. In the blob placed tarballs with
# actual content.
#
# Please note, if you need additional dependecies make sure to unpack
subarch
# archive as first argument.
#
nixstaller_unpack() {
        unpack_makeself

        local unpack_files="$@"

        for i in $unpack_files ; do
                unpack_banner "$i"
                # Make sure that file exists
                [[ -f "./$i" ]] && (
                        local type=$(file -b ${i})
                        case ${type} in
                                data)
                                        tar -xJf "./$i"
                                        ;;
                                gzip*)
                                        tar -xzf "./$i"
                                        ;;
                        esac
                ) || die "Failed to unpack $i"
        done
}


Original author is Azamat Hackimov aka winterheart (in CC).

2) It'd be also nice to detect mojo installers (most of the time,
they're "*-bin" or "*Installer" ELF32 sfx-archives) and pass them to
unpack_zip by default.
Unfortunately, I've no free time ATM, but if it is needed, I can write
prototype for Mojo unpacking.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 901 bytes --]

Re: [gentoo-dev] [RFC] unpacker.eclass extensions

[Markos Chandras]

Hi Vadim,

On 15 June 2013 09:39, Vadim A. Misbakh-Soloviov <mva@mva.name> wrote:
>                 # Make sure that file exists
>                 [[ -f "./$i" ]] && (
>                         local type=$(file -b ${i})
>                         case ${type} in
>                                 data)
>                                         tar -xJf "./$i"
>                                         ;;
>                                 gzip*)
>                                         tar -xzf "./$i"
>                                         ;;
>                         esac
>                 ) || die "Failed to unpack $i"
>         done
> }
>
I have a couple of questions. Why sub-shell here? My understanding is
that first you need to make sure the
file exists and then unpack it. Also || die does not work in subshells.

http://devmanual.gentoo.org/ebuild-writing/error-handling/index.html

> 2) It'd be also nice to detect mojo installers (most of the time,
> they're "*-bin" or "*Installer" ELF32 sfx-archives) and pass them to
> unpack_zip by default.
> Unfortunately, I've no free time ATM, but if it is needed, I can write
> prototype for Mojo unpacking.
>

We can only merge what you show us for review :)

--
Regards,
Markos Chandras - Gentoo Linux Developer
http://dev.gentoo.org/~hwoarang

Re: [gentoo-dev] [RFC] unpacker.eclass extensions

[Markos Chandras]

On 15 June 2013 15:33, Markos Chandras <hwoarang@gentoo.org> wrote:
>Also || die does not work in subshells.
>
> http://devmanual.gentoo.org/ebuild-writing/error-handling/index.html
>

Sorry scratch that. I just realized || die is not inside the subshell

--
Regards,
Markos Chandras - Gentoo Linux Developer
http://dev.gentoo.org/~hwoarang

Re: [gentoo-dev] [RFC] unpacker.eclass extensions

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 1699 bytes --]

On Saturday 15 June 2013 04:39:10 Vadim A. Misbakh-Soloviov wrote:
> # @DESCRIPTION:
> # Unpack nixstaller generated files

needs a period at the end.  content in @DESCRIPTION is normalized.

> # They're shell scripts with the blob package tagged onto
> # the end of the archive. In the blob placed tarballs with
> # actual content.

"They're shell scripts with a tarball appended to them."

> # Please note, if you need additional dependecies make sure to unpack
> subarch
> # archive as first argument.

no idea what this means

> nixstaller_unpack() {

this does not follow the API naming convention ("unpack" comes first)

>         local unpack_files="$@"

this doesn't work.  you normalized the input into a string.  just inline the 
"$@" below.  or don't specify it at all ... this does the same thing (albeit, 
correctly):
	local src
	for src ; do

>                 unpack_banner "$i"
>                 # Make sure that file exists
>                 [[ -f "./$i" ]] && (
>                         local type=$(file -b ${i})
>                         case ${type} in
>                                 data)
>                                         tar -xJf "./$i"

why doesn't the bzip2 detect as bzip2 ?

>                                         ;;
>                                 gzip*)
>                                         tar -xzf "./$i"
>                                         ;;
>                         esac
>                 ) || die "Failed to unpack $i"

the subshell should go away -- you don't even need it:
	[[ $? -eq 0 ]] || die ...

i wish we could merge with the file detection in unpack_makeself somehow
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] [RFC] unpacker.eclass extensions

[Diego Elio Pettenò]

[-- Attachment #1: Type: text/plain, Size: 358 bytes --]

On 17/06/2013 06:55, Mike Frysinger wrote:
>> >                                         tar -xJf "./$i"
> why doesn't the bzip2 detect as bzip2 ?
> 

Capital J is XZ not bzip2.. still, it makes me think it needs a newer
version of file to recognize it.

-- 
Diego Elio Pettenò — Flameeyes
flameeyes@flameeyes.eu — http://blog.flameeyes.eu/


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 555 bytes --]

Re: [gentoo-dev] [RFC] unpacker.eclass extensions

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 499 bytes --]

On Monday 17 June 2013 03:15:53 Diego Elio Pettenò wrote:
> On 17/06/2013 06:55, Mike Frysinger wrote:
> >> >                                         tar -xJf "./$i"
> > 
> > why doesn't the bzip2 detect as bzip2 ?
> 
> Capital J is XZ not bzip2.. still, it makes me think it needs a newer
> version of file to recognize it.

the stupid default font doesn't clearly render that.  guess that's the older 
LZMA format rather than the newer XZ as the latter can be detected by file.
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] [RFC] unpacker.eclass extensions

[Rick "Zero_Chaos" Farina]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 06/17/2013 12:08 PM, Mike Frysinger wrote:
> On Monday 17 June 2013 03:15:53 Diego Elio Pettenò wrote:
>> On 17/06/2013 06:55, Mike Frysinger wrote:
>>>>>                                         tar -xJf "./$i"
>>>
>>> why doesn't the bzip2 detect as bzip2 ?
>>
>> Capital J is XZ not bzip2.. still, it makes me think it needs a newer
>> version of file to recognize it.
> 
> the stupid default font doesn't clearly render that.  guess that's the older 
> LZMA format rather than the newer XZ as the latter can be detected by file.

I make all my files with "tar cJf"

zero@ozzie ~ % file /usr/portage/distfiles/gr-osmosdr-0.0.2.tar.xz
/usr/portage/distfiles/gr-osmosdr-0.0.2.tar.xz: XZ compressed data

worksforme

- -Zero

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJRvz9dAAoJEKXdFCfdEflKLIsQAJ+AeXP5CIrlZSefzJCFgP4G
6Fd3XJPsTKr58KBl5KLmAVPOtRwac0vXty4F9lHMHECvnfNOh0UiajvNV/Jyz1yz
zPZ/91Vv0vXc5ZlNNLqmq+rtIt3wuEXCg0TNwxTMbKX0MWW8S7ssOzibiYcLLoqT
IHFKLXhtU3zIxQ4kLiMZ7e1QRBnrVvA4/VlcOKAs8R6cdsp/1e3fzhgZgMx9ft03
hmHLW9kMMC06/Bz8ZKsvnfuyWhvQrlvsOUH8JWC9pbyZO58fiCgwSFWRyma04d8k
ulJ2y8K4B/WkIwRZzL47ROZQ91FcEnv9Ei9GzXLK6u4gIIrepX0NPcea441F1uVa
4i/9sjyk/zm6M3dAZyGlsCxZX96pAuFuE/H2OFcsaj9PRcdD2nR0Q4uvKwUqn+BM
Mhcr2wQlBsLDHiQr/3g/WepZ7VGcKrfmWymjxcCZoj9VGMhMR6O/h3uGEAdLm3YK
1gLdvyQz+koUFdkmQIQHXFjd8q+165Lbky5CYqs6xekM1aSeLpskdZpVSI7GRp+j
NvbB7gtEw2iz2pS8OSQ834K7KNnWI7OydVsv4cYnCySg3md+BAL1S5LLsc7FdmHz
cRLjH8Yw5mH4EfeGFnIebWYsVMeRn8M0KJxVuz7GH1gjtIvOjfIOcjdWMBwOhryP
LgmjPDvPM7NGJ3mneTLM
=UOVj
-----END PGP SIGNATURE-----

Re: [gentoo-dev] [RFC] unpacker.eclass extensions

[Diego Elio Pettenò]

[-- Attachment #1: Type: text/plain, Size: 514 bytes --]

On 17/06/2013 17:54, Rick "Zero_Chaos" Farina wrote:
> I make all my files with "tar cJf"
> 
> zero@ozzie ~ % file /usr/portage/distfiles/gr-osmosdr-0.0.2.tar.xz
> /usr/portage/distfiles/gr-osmosdr-0.0.2.tar.xz: XZ compressed data

cJ with _current_ tar will generate XZ
cJ with _past_ tar could generate lzma
xJ with _current_ tar will extract both XZ and lzma
zJ with _past_ tar will only extract lzma


-- 
Diego Elio Pettenò — Flameeyes
flameeyes@flameeyes.eu — http://blog.flameeyes.eu/


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 555 bytes --]

Re: [gentoo-dev] [RFC] unpacker.eclass extensions

[Rick "Zero_Chaos" Farina]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 06/17/2013 04:19 PM, Diego Elio Pettenò wrote:
> On 17/06/2013 17:54, Rick "Zero_Chaos" Farina wrote:
>> I make all my files with "tar cJf"
>>
>> zero@ozzie ~ % file /usr/portage/distfiles/gr-osmosdr-0.0.2.tar.xz
>> /usr/portage/distfiles/gr-osmosdr-0.0.2.tar.xz: XZ compressed data
> 
> cJ with _current_ tar will generate XZ
> cJ with _past_ tar could generate lzma
> xJ with _current_ tar will extract both XZ and lzma
> zJ with _past_ tar will only extract lzma
> 
> 
tar for the last several years properly and automatically detects the
format of the input using just 'x'.  This is true for gnu and bsd tar
which should cover all of gentoo.  We should be passing "tar xf" to
extract things unless we want to hijack the decompression command like
is possible in portage with BZIP2_COMMAND=lbzip2 in which case we want
to pass "tar -I $BZIP2_COMMAND -xf" or (not yet in portage) "tar -I pixz
- -xf".  Some good examples of how this are handled are in catalyst but
I'll let dolson talk about his own work as he is much more qualified
than I am.

- -Zero
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJRv3NyAAoJEKXdFCfdEflKTYIP/ifbrOuhtEW2W3NRnUNImOmn
YfRuRt+50OpEeHbniAp32zOhg0l6nYCgZ4CqzoUDTUm106EyAbyQSNheBC1d6DL6
eRFbuV41+B1cPDFB5FJMHrhapvKQmciGxyfgpPLrVnIHXQhNUvM+3h7qPq2SOnog
tF8gQ3J3umfcJa3NMEqRRrWNkiytTniKZ537fLT0vxVvtOXSXswe3o3TJPNLpNVO
BAyHNvZz3TKPy3gsHtvedo1iz37yUxUxAPnI0R0ccLRnPEGlnnNcseaQvCLGBvZP
a7xqPMnM1gRv4xH9deWvh//mCSvIBjGoaeQLEP5ztjinrWalkFDikUk8xx8YZRev
YYlQbj1n3NgKk7Io/fyXjaNmZGXHkdT0kYosT2w+uzwHMQVSbwIsdR24dUIxXrcp
SG37KNB/wtPzOWpUYkV13wJgkLc1mVmEj92F47dSgGt4EKtSUng3sudDs2bPz4ds
7Z0lV4xLiG36ITZidBIIpjIWAKIQbQWIW+mIC25g40WGJ+qJtSz5bivwJa2a9PAx
Zqhja3pQFbYz+YhIUidM/rkcy1XINicaL1lUbpaHZqzy8z/ivWAh6iXMLmulqdfa
BjfPHVj0JcJs+ySbHMP/W3DN823BCzxOdc8cvHhLOG/Vhdysr+t5ga0yAQOawJsz
fWqUGz1tNiKLU3uBs+20
=2YDM
-----END PGP SIGNATURE-----

Re: [gentoo-dev] [RFC] unpacker.eclass extensions

[Brian Dolbec]

[-- Attachment #1: Type: text/plain, Size: 3279 bytes --]

On Mon, 2013-06-17 at 16:37 -0400, Rick "Zero_Chaos" Farina wrote:
> On 06/17/2013 04:19 PM, Diego Elio Pettenò wrote:
> > On 17/06/2013 17:54, Rick "Zero_Chaos" Farina wrote:
> >> I make all my files with "tar cJf"
> >>
> >> zero@ozzie ~ % file /usr/portage/distfiles/gr-osmosdr-0.0.2.tar.xz
> >> /usr/portage/distfiles/gr-osmosdr-0.0.2.tar.xz: XZ compressed data
> > 
> > cJ with _current_ tar will generate XZ
> > cJ with _past_ tar could generate lzma
> > xJ with _current_ tar will extract both XZ and lzma
> > zJ with _past_ tar will only extract lzma
> > 
> > 
> tar for the last several years properly and automatically detects the
> format of the input using just 'x'.  This is true for gnu and bsd tar
> which should cover all of gentoo.  We should be passing "tar xf" to
> extract things unless we want to hijack the decompression command like
> is possible in portage with BZIP2_COMMAND=lbzip2 in which case we want
> to pass "tar -I $BZIP2_COMMAND -xf" or (not yet in portage) "tar -I pixz
> -xf".  Some good examples of how this are handled are in catalyst but
> I'll let dolson talk about his own work as he is much more qualified
> than I am.
> 
> -Zero

Well, I don't code bash, but...

In catalyst, I had a request to add more types of
compression/decompression capabilities.  Plus I am also trying to clean
up the messy, poorly structured code.  What I have come up with is a
separate class that can be loaded with either compression or
decompression commands that can be configured, overridden, added to,...
It will provide a python interface to perform the actions requested,
keeping the detailed coding central and internal.  It will provide
automatic detection of the command to use via it's extension as a
combination of a pre-determined configured preference (if it is
compatible) or a suitable command determined by that extension.  There
can also be several predefined ways to do the same action, in which case
the pre-configured preference will take priority.

I hadn't gotten into reading this thread much, but did note the
similarity of what I was currently doing and what it was discussing.

What is directly relevant to the tar commands is listed in the
compression_definitions and decompression_definitions at the top of the
file attached.

This python lib can easily be made into a standalone pkg with a cli
interface to perform many common actions without the need of additional
coding in eclasses, package managers, etc..  It could provide gentoo
with a common, central, easily extended and updated method of performing
normal compression/decompression.

It is a work in progress, with only preliminary testing done inside my
catalyst rewrite.  I have not yet added configuration options and
override preferences.

It is available in my catalyst git repo in my dev space.  It is in the
compress branch.  Note there is no master branch there, so a basic clone
will error when it tries to checkout a master working copy on completion
of the clone.  Just "git checkout compress" after.

http://dev.gentoo.org/~dolsen/catalyst/

I've attached the current compress.py file so you can peruse and decide
if making this a standalone gentoo project is desired.  If not, it will
get merged into catalyst code base.
-- 
Brian Dolbec <dolsen@gentoo.org>

[-- Attachment #2: compress.py --]
[-- Type: text/x-python, Size: 10170 bytes --]


# Maintained in full by:
# Catalyst Team <catalyst@gentoo.org>
# Release Engineering Team <releng@gentoo.org>

'''
compress.py

Utility class to hold and handle all possible compression
and de-compression of files.  Including rsync transfers.

'''


import os
from collections import namedtuple

from support import cmd


definition_fields = ["func", "cmd", "args", "id", "extension"]
defintition_types = [ str,    str,   list,   str,  str]

'''The definition entries are to follow the the definition_types
with the exception of the first entry "Type" which is a mode identifier
for use in the class as a type ID and printable string.'''
compress_definitions = {
	"Type": ["Compression", "Compression definitions loaded"],
	"rsync"		:["rsync", "rsync", ["-a", "--delete", "%(source)s",  "%(destination)s"], "RSYNC", None],
	"lbzip2"		:["_common", "tar", ["-I", "lbzip2", "-cf", "%(filename)s", "-C", "%(basedir)s", "%(source)s"], "LBZIP2", "tbz2"],
	"tbz2"		:["_common", "tar", ["-I", "lbzip2", "-cf", "%(filename)s", "-C", "%(basedir)s", "%(source)s"], "LBZIP2", "tbz2"],
	"bz2"		:["_common", "tar", ["-cpjf", "%(filename)s", "-C", "%(basedir)s", "%(source)s"], "BZIP2", "tar.bz2"],
	"tar"		:["_common", "tar", ["-cpf", "%(filename)s", "-C", "%(basedir)s", "%(source)s"], "TAR", "tar"],
	"xz"		:["_common", "tar", ["-cpJf", "%(filename)s", "-C", "%(basedir)s", "%(source)s"], "XZ", "tar.xz"],
	"pixz"		:["_common", "tar", ["-I", "pixz", "-cpf", "%(filename)s", "-C", "%(basedir)s", "%(source)s"], "PIXZ", "xz"],
	"zip"		:["_common", "tar", ["-cpzf", "%(filename)s", "-C", "%(basedir)s", "%(source)s"], "GZIP", "zip"],
	}


decompress_definitions = {
	"Type": ["Decompression", "Decompression definitions loaded"],
	"rsync"		:["rsync", "rsync", ["-a", "--delete", "%(source)s", "%(destination)s"], "RSYNC", None],
	"lbzip2"		:["_common", "tar", ["-I", "lbzip2", "-xpf", "%(source)s", "-C", "%(destination)s"], "LBZIP2", "bz2"],
	"tbz2"		:["_common", "tar", ["-I", "lbzip2", "-xpf", "%(source)s", "-C", "%(destination)s"], "LBZIP2", "tbz2"],
	"bz2"		:["_common", "tar", ["-xpf", "%(source)s", "-C", "%(destination)s"], "BZIP2", "bz2"],
	"tar"		:["_common", "tar", ["-xpf", "%(source)s", "-C", "%(destination)s"], "TAR", "tar"],
	"xz"		:["_common", "tar", ["-xpf", "%(source)s", "-C", "%(destination)s"], "XZ", "xz"],
	"pixz"		:["_common", "tar", ["-I", "pixz", "-xpf", "%(source)s", "-C", "%(destination)s"], "PIXZ", "xz"],
	"zip"		:["_common", "tar", ["-xpzf", "%(source)s", "-C", "%(destination)s"], "GZIP", "zip"],
	"gz"		:["_common", "tar", ["-xpzf", "%(source)s", "-C", "%(destination)s"], "GZIP", "zip"],
	}


'''Configure this here in case it is ever changed.
This is the only edit point required then.'''
extension_separator = '.'


class CompressMap(object):
	'''Class for handling
	Catalyst's compression & decompression of archives'''

	'''fields: list of ordered field names for the (de)compression functions'''
	fields = definition_fields[:]


	def __init__(self, definitions=None, env=None,
			default_mode=None, separator=extension_separator):
		'''Class init

		@param compress_mode: boolean, defaults to True
			describes compression or de-compression definitions loaded
		@param definitions: dictionary of
			Key:[function, cmd, cmd_args, Print/id string, extension]
		@param env: environment to pass to the cmd subprocess
		'''
		if definitions is None:
			definitions = {}
			self.loaded_type = ["None", "No definitions loaded"]
		else:
			self.loaded_type = definitions.pop('Type')
		self.env = env or {}
		self.mode_error = self.loaded_type[0] + \
			" Error: No mode was passed in or automatically detected"
		self._map = {}
		self.extension_separator = separator
		# set some defaults depending on what is being loaded
		if self.loaded_type[0] in ['Compression']:
			self.mode = default_mode or 'tbz2'
			self.compress = self._compress
			self.extract = None
		else:
			self.mode = default_mode or 'auto'
			self.compress = None
			self.extract = self._extract

		# create the (de)compression definition namedtuple classes
		for name in list(definitions):
			obj = namedtuple(name, self.fields)
			obj.__slots__ = ()
			self._map[name] = obj._make(definitions[name])
		del obj


	def _compress(self, infodict=None, filename='', source=None,
			basedir='.', mode=None, auto_extension=False, fatal=True):
		'''Compression function

		@param infodict: optional dictionary of the next 4 parameters.
		@param filename: optional string, name ot the file to make
		@param source: optional string, path to a directory
		@param destination: optional string, path a directory
		@param mode: string, optional mode to use to (de)compress with
		@return boolean
		'''
		if not infodict:
			infodict = self.create_infodict(source, None,
				basedir, filename, mode or self.mode, auto_extension)
		if not infodict['mode']:
			print self.mode_error
			return False
		if auto_extension:
			infodict['auto-ext'] = True
		return self._run(infodict, fatal=fatal)


	def _extract(self, infodict=None, source=None, destination=None,
			mode=None, fatal=True):
		'''De-compression function

		@param infodict: optional dictionary of the next 3 parameters.
		@param source: optional string, path to a directory
		@param destination: optional string, path a directory
		@param mode: string, optional mode to use to (de)compress with
		@return boolean
		'''
		if self.loaded_type[0] not in ["Decompression"]:
			return False
		if not infodict:
			infodict = self.create_infodict(source, destination, mode=mode)
		if infodict['mode'] in [None]:
			infodict['mode'] = self.mode or 'auto'
		if infodict['mode'] in ['auto']:
			infodict['mode'] = self.get_extension(infodict['source'])
			if not infodict['mode']:
				print self.mode_error
				return False
		return self._run(infodict, fatal=fatal)


	def _run(self, infodict, fatal=True):
		'''Internal function that runs the designated function

		@param source: string, path to a directory or file
		@param destination: string, path a directoy or file
		@param mode; string, desired method to perform the
			compression or transfer
		@return boolean
		'''
		if not self.is_supported(infodict['mode']):
			print "mode: %s is not supported in the current %s definitions" \
				% (infodict['mode'], self.loaded_type[1])
			return False
		try:
			func = getattr(self, self._map[infodict['mode']].func)
			success = func(infodict, fatal)
		except AttributeError:
			print "FAILED to find function '%s'" % str(self._map[infodict['mode']].func)
			return False
		#except Exception as e:
			#msg = "Error performing %s %s, " % (mode, self.loaded_type[0]) + \
				#"is the appropriate utility installed on your system?"
			#print msg
			#print "Exception:", e
			#return False
		return success


	def get_extension(self, source):
		'''Extracts the file extension string from the source file

		@param source: string, file path of the file to determine
		@return string: file type extension of the source file
		'''
		#if self.extension_separator not in source:
			#return None
		#return source.rsplit(self.extension_separator, 1)[1]
		return os.path.splitext(source)[1]


	def rsync(self, infodict=None, source=None, destination=None,
			mode=None, fatal=True):
		'''Convienience function. Performs an rsync transfer

		@param infodict: optional dictionary of the next 3 parameters.
		@param source: optional string, path to a directory
		@param destination: optional string, path a directory
		@param mode: string, optional mode to use to (de)compress with
		@return boolean
		'''
		if not infodict:
			if not mode:
				mode = 'rsync'
			infodict = self.create_infodict(source, destination, mode=mode)
		return self._run(infodict, fatal=fatal)


	def _common(self, infodict, fatal=True):
		'''Internal function.  Performs commonly supported
		compression or decompression.

		@param files: dict as returned by this class's pair_files()
		@param mode: string, mode to use to (de)compress with
		@return boolean
		'''
		if not infodict['mode'] or not self.is_supported(infodict['mode']):
			print "ERROR: CompressMap; %s mode: %s not correctly set!" \
				% (self.loaded_type[0], infodict['mode'])
			return False

		#Avoid modifying the source dictionary
		cmdinfo = infodict.copy()

		cmdlist = self._map[cmdinfo['mode']]

		if cmdinfo['auto-ext']:
			cmdinfo['filename'] += self.extension_separator + \
				cmdlist.extension

		# Do the string substitution
		opts = ' '.join(cmdlist.args) %(cmdinfo)
		args = ' '.join([cmdlist.cmd, opts])

		return cmd(args, cmdlist.id, env=self.env, fatal=fatal)


	def create_infodict(self, source, destination=None, basedir=None,
			filename='', mode=None, auto_extension=False):
		'''Puts the source and destination paths into a dictionary
		for use in string substitution in the defintions
		%(source) and %(destination) fields embedded into the commands

		@param source: string, path to a directory
		@param destination: string, path a directory
		@param filename: optional string
		@return dict:
		'''
		return {
			'source': source,
			'destination': destination,
			'basedir': basedir,
			'filename': filename,
			'mode': mode or self.mode,
			'auto-ext': auto_extension,
			}


	def is_supported(self, mode):
		'''Truth function to test the mode desired is supported
		in the definitions loaded

		@param mode: string, mode to use to (de)compress with
		@return boolean
		'''
		return mode in list(self._map)


	@property
	def available_modes(self):
		'''Convienence function to return the available modes'''
		return list(self._map)


	def best_mode(self, prefered_mode, source):
		'''Compare the prefered_mode's extension with the source extension
		and returns the best choice

		@param prefered_mode: string
		@param source: string, path the the source file
		@return string: best mode to use for the extraction
		'''
		if source.endswith(self._map[prefered_mode].extension):
			return prefered_mode
		return self.get_extension(source)


	def extension(self, mode):
		'''Returns the predetermined extension auto-ext added
		to the filename for compression.

		@param mode: string
		@return string
		'''
		if self.is_supported(mode):
			return self._map[mode].extension
		return ''

Re: [gentoo-dev] [RFC] unpacker.eclass extensions

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 1032 bytes --]

On Monday 17 June 2013 16:37:06 Rick "Zero_Chaos" Farina wrote:
> On 06/17/2013 04:19 PM, Diego Elio Pettenò wrote:
> > On 17/06/2013 17:54, Rick "Zero_Chaos" Farina wrote:
> >> I make all my files with "tar cJf"
> >> 
> >> zero@ozzie ~ % file /usr/portage/distfiles/gr-osmosdr-0.0.2.tar.xz
> >> /usr/portage/distfiles/gr-osmosdr-0.0.2.tar.xz: XZ compressed data
> > 
> > cJ with _current_ tar will generate XZ
> > cJ with _past_ tar could generate lzma
> > xJ with _current_ tar will extract both XZ and lzma
> > zJ with _past_ tar will only extract lzma
> 
> tar for the last several years properly and automatically detects the
> format of the input using just 'x'.  This is true for gnu and bsd tar
> which should cover all of gentoo.

and that relies on the magic being in the output format which, as we pointed 
out, wasn't the case.  it's one of the reasons why the XZ format came to be.

build lzma-utils and compress some files and see how well tar & file handle it:
	http://tukaani.org/lzma/
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

[gentoo-dev] unpacker.eclass: add decompress probe helper

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 6471 bytes --]

On Monday 17 June 2013 01:55:09 Mike Frysinger wrote:
> i wish we could merge with the file detection in unpack_makeself somehow

this patch should unify that aspect
-mike

--- unpacker.eclass	10 Apr 2013 14:47:49 -0000	1.13
+++ unpacker.eclass	22 Jun 2013 17:54:09 -0000
@@ -22,7 +22,14 @@ ___ECLASS_ONCE_UNPACKER="recur -_+^+_- s
 # @DEFAULT_UNSET
 # @DESCRIPTION:
 # Utility to use to decompress bzip2 files.  Will dynamically pick between
-# `pbzip2` and `bzip2`.  Make sure your choice accepts the "-c" option.
+# `pbzip2` and `bzip2`.  Make sure your choice accepts the "-dc" options.
+# Note: this is meant for users to set, not ebuilds.
+
+# @ECLASS-VARIABLE: UNPACKER_GZIP
+# @DEFAULT_UNSET
+# @DESCRIPTION:
+# Utility to use to decompress gzip files.  Will dynamically pick between
+# `gzip` and `pigz`.  Make sure your choice accepts the "-dc" options.
 # Note: this is meant for users to set, not ebuilds.
 
 # for internal use only (unpack_pdv and unpack_makeself)
@@ -74,9 +81,9 @@ unpack_banner() {
 # parameter.  Here is an example:
 #
 # @CODE
-# 	vapier@vapier 0 pdv_unpack # strings hldsupdatetool.bin | grep lseek
+# 	$ strings hldsupdatetool.bin | grep lseek
 # 	lseek
-# 	vapier@vapier 0 pdv_unpack # strace -elseek ./hldsupdatetool.bin
+# 	$ strace -elseek ./hldsupdatetool.bin
 # 	lseek(3, -4, SEEK_END)					= 2981250
 # @CODE
 #
@@ -105,19 +112,17 @@ unpack_pdv() {
 	local tmpfile="${T}/${FUNCNAME}"
 	tail -c +$((${tailskip}+1)) ${src} 2>/dev/null | head -c 512 > "${tmpfile}"
 
-	local iscompressed=$(file -b "${tmpfile}")
-	if [[ ${iscompressed:0:8} == "compress" ]] ; then
-		iscompressed=1
+	local comp=$(decompress_prog "${tmpfile}")
+	if [[ -n ${comp} ]] ; then
+		comp="| ${comp}"
 		mv "${tmpfile}"{,.Z}
 		gunzip "${tmpfile}"
-	else
-		iscompressed=0
 	fi
-	local istar=$(file -b "${tmpfile}")
-	if [[ ${istar:0:9} == "POSIX tar" ]] ; then
-		istar=1
+	local istar filetype=$(file -b "${tmpfile}")
+	if [[ ${filetype} == "POSIX tar"* ]] ; then
+		istar=true
 	else
-		istar=0
+		istar=false
 	fi
 
 	#for some reason gzip dies with this ... dd cant provide buffer fast enough ?
@@ -125,29 +130,18 @@ unpack_pdv() {
 	#	| dd ibs=${tailskip} skip=1 \
 	#	| gzip -dc \
 	#	> ${datafile}
-	if [ ${iscompressed} -eq 1 ] ; then
-		if [ ${istar} -eq 1 ] ; then
-			tail -c +$((${tailskip}+1)) "${src}" 2>/dev/null \
-				| head -c $((${metaskip}-${tailskip})) \
-				| tar -xzf -
-		else
-			tail -c +$((${tailskip}+1)) "${src}" 2>/dev/null \
-				| head -c $((${metaskip}-${tailskip})) \
-				| gzip -dc \
-				> ${datafile}
-		fi
+	(
+	th() {
+		tail -c +$(( tailskip + 1 )) "${src}" 2>/dev/null | \
+			head -c $(( metaskip - tailskip ))
+	}
+	if ${istar} ; then
+		eval th ${comp} | tar --no-same-owner -xf -
 	else
-		if [ ${istar} -eq 1 ] ; then
-			tail -c +$((${tailskip}+1)) "${src}" 2>/dev/null \
-				| head -c $((${metaskip}-${tailskip})) \
-				| tar --no-same-owner -xf -
-		else
-			tail -c +$((${tailskip}+1)) "${src}" 2>/dev/null \
-				| head -c $((${metaskip}-${tailskip})) \
-				> ${datafile}
-		fi
+		eval th ${comp} > "${datafile}"
 	fi
-	true
+	exit 0
+	)
 	#[ -s "${datafile}" ] || die "failure unpacking pdv ('${metaskip}' '${tailskip}' '${datafile}')"
 	#assert "failure unpacking pdv ('${metaskip}' '${tailskip}' '${datafile}')"
 }
@@ -218,31 +212,11 @@ unpack_makeself() {
 	esac
 
 	# lets grab the first few bytes of the file to figure out what kind of archive it is
-	local filetype tmpfile="${T}/${FUNCNAME}"
+	local tmpfile="${T}/${FUNCNAME}"
 	eval ${exe} 2>/dev/null | head -c 512 > "${tmpfile}"
-	filetype=$(file -b "${tmpfile}") || die
-	case ${filetype} in
-		*tar\ archive*)
-			eval ${exe} | tar --no-same-owner -xf -
-			;;
-		bzip2*)
-			eval ${exe} | bzip2 -dc | tar --no-same-owner -xf -
-			;;
-		gzip*)
-			eval ${exe} | tar --no-same-owner -xzf -
-			;;
-		compress*)
-			eval ${exe} | gunzip | tar --no-same-owner -xf -
-			;;
-		XZ*)
-			eval ${exe} | unxz | tar --no-same-owner -xf -
-			;;
-		*)
-			eerror "Unknown filetype \"${filetype}\" ?"
-			false
-			;;
-	esac
-	assert "failure unpacking (${filetype}) makeself ${src##*/} ('${ver}' +${skip})"
+	local comp=$(decompress_prog "${tmpfile}")
+	eval ${exe} | eval "${comp:+${comp} |}" tar --no-same-owner -xf -
+	assert "failure unpacking makeself ${src##*/} ('${ver}' +${skip})"
 }
 
 # @FUNCTION: unpack_deb
@@ -319,6 +293,54 @@ unpack_zip() {
 	[[ $? -le 1 ]] || die "unpacking ${zip} failed (arch=unpack_zip)"
 }
 
+# @FUNCTION: decompress_prog
+# @USAGE: <mime|format|filename>
+# @DESCRIPTION:
+# Get the program name (and args) needed to decompress things.
+# You can give this a mime type, or a format name (bz2/gz/xz/etc...),
+# or you can give it a filename.  If the type can be detected based on
+# the suffix alone, we'll use that, otherwise we'll rely on `file` to
+# probe the type.
+decompress_prog() {
+	[[ $# -eq 1 ]] || die "Usage: ${FUNCNAME} <mime|format|filename>"
+
+	local comp
+
+	case $1 in
+	application/x-bzip2\;*|\
+	bzip2|bz2|\
+	*.bz2|*.tbz|*.tbz2)
+		local bzcmd=${PORTAGE_BZIP2_COMMAND:-$(type -P pbzip2 || type -P bzip2)}
+		local bzuncmd=${PORTAGE_BUNZIP2_COMMAND:-${bzcmd} -d}
+		: ${UNPACKER_BZ2:=${bzuncmd}}
+		comp="${UNPACKER_BZ2} -dc"
+		;;
+
+	application/x-gzip\;*|\
+	application/x-compress\;*|\
+	gzip|gz|\
+	*.z|*.gz|*.tgz)
+		: ${UNPACKER_GZIP:=$(type -P pigz || type -P gzip)}
+		comp="${UNPACKER_GZIP} -dc"
+		;;
+
+	application/x-xz\;*|\
+	lzma|xz|\
+	*.lzma|*.xz|*.txz)
+		comp="xz -dc"
+		;;
+	esac
+
+	if [[ -n ${comp} ]] ; then
+		echo "${comp}"
+		return 0
+	fi
+
+	if [[ -s $1 ]] ; then
+		decompress_prog "$(file -ib "$1")"
+	fi
+}
+
 # @FUNCTION: _unpacker
 # @USAGE: <one archive to unpack>
 # @INTERNAL
@@ -333,19 +355,7 @@ _unpacker() {
 	a=$(find_unpackable_file "${a}")
 
 	# first figure out the decompression method
-	case ${m} in
-	*.bz2|*.tbz|*.tbz2)
-		local bzcmd=${PORTAGE_BZIP2_COMMAND:-$(type -P pbzip2 || type -P bzip2)}
-		local bzuncmd=${PORTAGE_BUNZIP2_COMMAND:-${bzcmd} -d}
-		: ${UNPACKER_BZ2:=${bzuncmd}}
-		comp="${UNPACKER_BZ2} -c"
-		;;
-	*.z|*.gz|*.tgz)
-		comp="gzip -dc" ;;
-	*.lzma|*.xz|*.txz)
-		comp="xz -dc" ;;
-	*)	comp="" ;;
-	esac
+	comp=$(decompress_prog "${n}")
 
 	# then figure out if there are any archiving aspects
 	arch=""

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]