* [gentoo-dev] [RFC] unpacker.eclass extensions @ 2013-06-15 8:39 Vadim A. Misbakh-Soloviov 2013-06-15 14:33 ` Markos Chandras 2013-06-17 5:55 ` Mike Frysinger 0 siblings, 2 replies; 12+ messages in thread From: Vadim A. Misbakh-Soloviov @ 2013-06-15 8:39 UTC (permalink / raw To: gentoo-dev; +Cc: azamat.hackimov [-- Attachment #1: Type: text/plain, Size: 1571 bytes --] As gamerlay maintainer, I'd be glad to introduce some changes to unpacker.eclass: 1) merging unpacker-nixstaller (Makeself subspecies) from gamerlay: # @FUNCTION: unpack_nixstaller # @USAGE: <files to unpack> # @DESCRIPTION: # Unpack nixstaller generated files # They're shell scripts with the blob package tagged onto # the end of the archive. In the blob placed tarballs with # actual content. # # Please note, if you need additional dependecies make sure to unpack subarch # archive as first argument. # nixstaller_unpack() { unpack_makeself local unpack_files="$@" for i in $unpack_files ; do unpack_banner "$i" # Make sure that file exists [[ -f "./$i" ]] && ( local type=$(file -b ${i}) case ${type} in data) tar -xJf "./$i" ;; gzip*) tar -xzf "./$i" ;; esac ) || die "Failed to unpack $i" done } Original author is Azamat Hackimov aka winterheart (in CC). 2) It'd be also nice to detect mojo installers (most of the time, they're "*-bin" or "*Installer" ELF32 sfx-archives) and pass them to unpack_zip by default. Unfortunately, I've no free time ATM, but if it is needed, I can write prototype for Mojo unpacking. [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 901 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [gentoo-dev] [RFC] unpacker.eclass extensions 2013-06-15 8:39 [gentoo-dev] [RFC] unpacker.eclass extensions Vadim A. Misbakh-Soloviov @ 2013-06-15 14:33 ` Markos Chandras 2013-06-15 14:37 ` Markos Chandras 2013-06-17 5:55 ` Mike Frysinger 1 sibling, 1 reply; 12+ messages in thread From: Markos Chandras @ 2013-06-15 14:33 UTC (permalink / raw To: gentoo-dev; +Cc: azamat.hackimov Hi Vadim, On 15 June 2013 09:39, Vadim A. Misbakh-Soloviov <mva@mva.name> wrote: > # Make sure that file exists > [[ -f "./$i" ]] && ( > local type=$(file -b ${i}) > case ${type} in > data) > tar -xJf "./$i" > ;; > gzip*) > tar -xzf "./$i" > ;; > esac > ) || die "Failed to unpack $i" > done > } > I have a couple of questions. Why sub-shell here? My understanding is that first you need to make sure the file exists and then unpack it. Also || die does not work in subshells. http://devmanual.gentoo.org/ebuild-writing/error-handling/index.html > 2) It'd be also nice to detect mojo installers (most of the time, > they're "*-bin" or "*Installer" ELF32 sfx-archives) and pass them to > unpack_zip by default. > Unfortunately, I've no free time ATM, but if it is needed, I can write > prototype for Mojo unpacking. > We can only merge what you show us for review :) -- Regards, Markos Chandras - Gentoo Linux Developer http://dev.gentoo.org/~hwoarang ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [gentoo-dev] [RFC] unpacker.eclass extensions 2013-06-15 14:33 ` Markos Chandras @ 2013-06-15 14:37 ` Markos Chandras 0 siblings, 0 replies; 12+ messages in thread From: Markos Chandras @ 2013-06-15 14:37 UTC (permalink / raw To: gentoo-dev; +Cc: azamat.hackimov On 15 June 2013 15:33, Markos Chandras <hwoarang@gentoo.org> wrote: >Also || die does not work in subshells. > > http://devmanual.gentoo.org/ebuild-writing/error-handling/index.html > Sorry scratch that. I just realized || die is not inside the subshell -- Regards, Markos Chandras - Gentoo Linux Developer http://dev.gentoo.org/~hwoarang ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [gentoo-dev] [RFC] unpacker.eclass extensions 2013-06-15 8:39 [gentoo-dev] [RFC] unpacker.eclass extensions Vadim A. Misbakh-Soloviov 2013-06-15 14:33 ` Markos Chandras @ 2013-06-17 5:55 ` Mike Frysinger 2013-06-17 7:15 ` Diego Elio Pettenò 2013-06-22 17:55 ` [gentoo-dev] unpacker.eclass: add decompress probe helper Mike Frysinger 1 sibling, 2 replies; 12+ messages in thread From: Mike Frysinger @ 2013-06-17 5:55 UTC (permalink / raw To: gentoo-dev; +Cc: Vadim A. Misbakh-Soloviov, azamat.hackimov [-- Attachment #1: Type: Text/Plain, Size: 1699 bytes --] On Saturday 15 June 2013 04:39:10 Vadim A. Misbakh-Soloviov wrote: > # @DESCRIPTION: > # Unpack nixstaller generated files needs a period at the end. content in @DESCRIPTION is normalized. > # They're shell scripts with the blob package tagged onto > # the end of the archive. In the blob placed tarballs with > # actual content. "They're shell scripts with a tarball appended to them." > # Please note, if you need additional dependecies make sure to unpack > subarch > # archive as first argument. no idea what this means > nixstaller_unpack() { this does not follow the API naming convention ("unpack" comes first) > local unpack_files="$@" this doesn't work. you normalized the input into a string. just inline the "$@" below. or don't specify it at all ... this does the same thing (albeit, correctly): local src for src ; do > unpack_banner "$i" > # Make sure that file exists > [[ -f "./$i" ]] && ( > local type=$(file -b ${i}) > case ${type} in > data) > tar -xJf "./$i" why doesn't the bzip2 detect as bzip2 ? > ;; > gzip*) > tar -xzf "./$i" > ;; > esac > ) || die "Failed to unpack $i" the subshell should go away -- you don't even need it: [[ $? -eq 0 ]] || die ... i wish we could merge with the file detection in unpack_makeself somehow -mike [-- Attachment #2: This is a digitally signed message part. --] [-- Type: application/pgp-signature, Size: 836 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [gentoo-dev] [RFC] unpacker.eclass extensions 2013-06-17 5:55 ` Mike Frysinger @ 2013-06-17 7:15 ` Diego Elio Pettenò 2013-06-17 16:08 ` Mike Frysinger 2013-06-22 17:55 ` [gentoo-dev] unpacker.eclass: add decompress probe helper Mike Frysinger 1 sibling, 1 reply; 12+ messages in thread From: Diego Elio Pettenò @ 2013-06-17 7:15 UTC (permalink / raw To: gentoo-dev [-- Attachment #1: Type: text/plain, Size: 358 bytes --] On 17/06/2013 06:55, Mike Frysinger wrote: >> > tar -xJf "./$i" > why doesn't the bzip2 detect as bzip2 ? > Capital J is XZ not bzip2.. still, it makes me think it needs a newer version of file to recognize it. -- Diego Elio Pettenò — Flameeyes flameeyes@flameeyes.eu — http://blog.flameeyes.eu/ [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 555 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [gentoo-dev] [RFC] unpacker.eclass extensions 2013-06-17 7:15 ` Diego Elio Pettenò @ 2013-06-17 16:08 ` Mike Frysinger 2013-06-17 16:54 ` Rick "Zero_Chaos" Farina 0 siblings, 1 reply; 12+ messages in thread From: Mike Frysinger @ 2013-06-17 16:08 UTC (permalink / raw To: gentoo-dev [-- Attachment #1: Type: Text/Plain, Size: 499 bytes --] On Monday 17 June 2013 03:15:53 Diego Elio Pettenò wrote: > On 17/06/2013 06:55, Mike Frysinger wrote: > >> > tar -xJf "./$i" > > > > why doesn't the bzip2 detect as bzip2 ? > > Capital J is XZ not bzip2.. still, it makes me think it needs a newer > version of file to recognize it. the stupid default font doesn't clearly render that. guess that's the older LZMA format rather than the newer XZ as the latter can be detected by file. -mike [-- Attachment #2: This is a digitally signed message part. --] [-- Type: application/pgp-signature, Size: 836 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [gentoo-dev] [RFC] unpacker.eclass extensions 2013-06-17 16:08 ` Mike Frysinger @ 2013-06-17 16:54 ` Rick "Zero_Chaos" Farina 2013-06-17 20:19 ` Diego Elio Pettenò 0 siblings, 1 reply; 12+ messages in thread From: Rick "Zero_Chaos" Farina @ 2013-06-17 16:54 UTC (permalink / raw To: gentoo-dev -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 06/17/2013 12:08 PM, Mike Frysinger wrote: > On Monday 17 June 2013 03:15:53 Diego Elio Pettenò wrote: >> On 17/06/2013 06:55, Mike Frysinger wrote: >>>>> tar -xJf "./$i" >>> >>> why doesn't the bzip2 detect as bzip2 ? >> >> Capital J is XZ not bzip2.. still, it makes me think it needs a newer >> version of file to recognize it. > > the stupid default font doesn't clearly render that. guess that's the older > LZMA format rather than the newer XZ as the latter can be detected by file. I make all my files with "tar cJf" zero@ozzie ~ % file /usr/portage/distfiles/gr-osmosdr-0.0.2.tar.xz /usr/portage/distfiles/gr-osmosdr-0.0.2.tar.xz: XZ compressed data worksforme - -Zero -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBAgAGBQJRvz9dAAoJEKXdFCfdEflKLIsQAJ+AeXP5CIrlZSefzJCFgP4G 6Fd3XJPsTKr58KBl5KLmAVPOtRwac0vXty4F9lHMHECvnfNOh0UiajvNV/Jyz1yz zPZ/91Vv0vXc5ZlNNLqmq+rtIt3wuEXCg0TNwxTMbKX0MWW8S7ssOzibiYcLLoqT IHFKLXhtU3zIxQ4kLiMZ7e1QRBnrVvA4/VlcOKAs8R6cdsp/1e3fzhgZgMx9ft03 hmHLW9kMMC06/Bz8ZKsvnfuyWhvQrlvsOUH8JWC9pbyZO58fiCgwSFWRyma04d8k ulJ2y8K4B/WkIwRZzL47ROZQ91FcEnv9Ei9GzXLK6u4gIIrepX0NPcea441F1uVa 4i/9sjyk/zm6M3dAZyGlsCxZX96pAuFuE/H2OFcsaj9PRcdD2nR0Q4uvKwUqn+BM Mhcr2wQlBsLDHiQr/3g/WepZ7VGcKrfmWymjxcCZoj9VGMhMR6O/h3uGEAdLm3YK 1gLdvyQz+koUFdkmQIQHXFjd8q+165Lbky5CYqs6xekM1aSeLpskdZpVSI7GRp+j NvbB7gtEw2iz2pS8OSQ834K7KNnWI7OydVsv4cYnCySg3md+BAL1S5LLsc7FdmHz cRLjH8Yw5mH4EfeGFnIebWYsVMeRn8M0KJxVuz7GH1gjtIvOjfIOcjdWMBwOhryP LgmjPDvPM7NGJ3mneTLM =UOVj -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [gentoo-dev] [RFC] unpacker.eclass extensions 2013-06-17 16:54 ` Rick "Zero_Chaos" Farina @ 2013-06-17 20:19 ` Diego Elio Pettenò 2013-06-17 20:37 ` Rick "Zero_Chaos" Farina 0 siblings, 1 reply; 12+ messages in thread From: Diego Elio Pettenò @ 2013-06-17 20:19 UTC (permalink / raw To: gentoo-dev [-- Attachment #1: Type: text/plain, Size: 514 bytes --] On 17/06/2013 17:54, Rick "Zero_Chaos" Farina wrote: > I make all my files with "tar cJf" > > zero@ozzie ~ % file /usr/portage/distfiles/gr-osmosdr-0.0.2.tar.xz > /usr/portage/distfiles/gr-osmosdr-0.0.2.tar.xz: XZ compressed data cJ with _current_ tar will generate XZ cJ with _past_ tar could generate lzma xJ with _current_ tar will extract both XZ and lzma zJ with _past_ tar will only extract lzma -- Diego Elio Pettenò — Flameeyes flameeyes@flameeyes.eu — http://blog.flameeyes.eu/ [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 555 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [gentoo-dev] [RFC] unpacker.eclass extensions 2013-06-17 20:19 ` Diego Elio Pettenò @ 2013-06-17 20:37 ` Rick "Zero_Chaos" Farina 2013-06-17 21:41 ` Brian Dolbec 2013-06-19 21:15 ` Mike Frysinger 0 siblings, 2 replies; 12+ messages in thread From: Rick "Zero_Chaos" Farina @ 2013-06-17 20:37 UTC (permalink / raw To: gentoo-dev -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 06/17/2013 04:19 PM, Diego Elio Pettenò wrote: > On 17/06/2013 17:54, Rick "Zero_Chaos" Farina wrote: >> I make all my files with "tar cJf" >> >> zero@ozzie ~ % file /usr/portage/distfiles/gr-osmosdr-0.0.2.tar.xz >> /usr/portage/distfiles/gr-osmosdr-0.0.2.tar.xz: XZ compressed data > > cJ with _current_ tar will generate XZ > cJ with _past_ tar could generate lzma > xJ with _current_ tar will extract both XZ and lzma > zJ with _past_ tar will only extract lzma > > tar for the last several years properly and automatically detects the format of the input using just 'x'. This is true for gnu and bsd tar which should cover all of gentoo. We should be passing "tar xf" to extract things unless we want to hijack the decompression command like is possible in portage with BZIP2_COMMAND=lbzip2 in which case we want to pass "tar -I $BZIP2_COMMAND -xf" or (not yet in portage) "tar -I pixz - -xf". Some good examples of how this are handled are in catalyst but I'll let dolson talk about his own work as he is much more qualified than I am. - -Zero -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBAgAGBQJRv3NyAAoJEKXdFCfdEflKTYIP/ifbrOuhtEW2W3NRnUNImOmn YfRuRt+50OpEeHbniAp32zOhg0l6nYCgZ4CqzoUDTUm106EyAbyQSNheBC1d6DL6 eRFbuV41+B1cPDFB5FJMHrhapvKQmciGxyfgpPLrVnIHXQhNUvM+3h7qPq2SOnog tF8gQ3J3umfcJa3NMEqRRrWNkiytTniKZ537fLT0vxVvtOXSXswe3o3TJPNLpNVO BAyHNvZz3TKPy3gsHtvedo1iz37yUxUxAPnI0R0ccLRnPEGlnnNcseaQvCLGBvZP a7xqPMnM1gRv4xH9deWvh//mCSvIBjGoaeQLEP5ztjinrWalkFDikUk8xx8YZRev YYlQbj1n3NgKk7Io/fyXjaNmZGXHkdT0kYosT2w+uzwHMQVSbwIsdR24dUIxXrcp SG37KNB/wtPzOWpUYkV13wJgkLc1mVmEj92F47dSgGt4EKtSUng3sudDs2bPz4ds 7Z0lV4xLiG36ITZidBIIpjIWAKIQbQWIW+mIC25g40WGJ+qJtSz5bivwJa2a9PAx Zqhja3pQFbYz+YhIUidM/rkcy1XINicaL1lUbpaHZqzy8z/ivWAh6iXMLmulqdfa BjfPHVj0JcJs+ySbHMP/W3DN823BCzxOdc8cvHhLOG/Vhdysr+t5ga0yAQOawJsz fWqUGz1tNiKLU3uBs+20 =2YDM -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [gentoo-dev] [RFC] unpacker.eclass extensions 2013-06-17 20:37 ` Rick "Zero_Chaos" Farina @ 2013-06-17 21:41 ` Brian Dolbec 2013-06-19 21:15 ` Mike Frysinger 1 sibling, 0 replies; 12+ messages in thread From: Brian Dolbec @ 2013-06-17 21:41 UTC (permalink / raw To: gentoo-dev [-- Attachment #1: Type: text/plain, Size: 3279 bytes --] On Mon, 2013-06-17 at 16:37 -0400, Rick "Zero_Chaos" Farina wrote: > On 06/17/2013 04:19 PM, Diego Elio Pettenò wrote: > > On 17/06/2013 17:54, Rick "Zero_Chaos" Farina wrote: > >> I make all my files with "tar cJf" > >> > >> zero@ozzie ~ % file /usr/portage/distfiles/gr-osmosdr-0.0.2.tar.xz > >> /usr/portage/distfiles/gr-osmosdr-0.0.2.tar.xz: XZ compressed data > > > > cJ with _current_ tar will generate XZ > > cJ with _past_ tar could generate lzma > > xJ with _current_ tar will extract both XZ and lzma > > zJ with _past_ tar will only extract lzma > > > > > tar for the last several years properly and automatically detects the > format of the input using just 'x'. This is true for gnu and bsd tar > which should cover all of gentoo. We should be passing "tar xf" to > extract things unless we want to hijack the decompression command like > is possible in portage with BZIP2_COMMAND=lbzip2 in which case we want > to pass "tar -I $BZIP2_COMMAND -xf" or (not yet in portage) "tar -I pixz > -xf". Some good examples of how this are handled are in catalyst but > I'll let dolson talk about his own work as he is much more qualified > than I am. > > -Zero Well, I don't code bash, but... In catalyst, I had a request to add more types of compression/decompression capabilities. Plus I am also trying to clean up the messy, poorly structured code. What I have come up with is a separate class that can be loaded with either compression or decompression commands that can be configured, overridden, added to,... It will provide a python interface to perform the actions requested, keeping the detailed coding central and internal. It will provide automatic detection of the command to use via it's extension as a combination of a pre-determined configured preference (if it is compatible) or a suitable command determined by that extension. There can also be several predefined ways to do the same action, in which case the pre-configured preference will take priority. I hadn't gotten into reading this thread much, but did note the similarity of what I was currently doing and what it was discussing. What is directly relevant to the tar commands is listed in the compression_definitions and decompression_definitions at the top of the file attached. This python lib can easily be made into a standalone pkg with a cli interface to perform many common actions without the need of additional coding in eclasses, package managers, etc.. It could provide gentoo with a common, central, easily extended and updated method of performing normal compression/decompression. It is a work in progress, with only preliminary testing done inside my catalyst rewrite. I have not yet added configuration options and override preferences. It is available in my catalyst git repo in my dev space. It is in the compress branch. Note there is no master branch there, so a basic clone will error when it tries to checkout a master working copy on completion of the clone. Just "git checkout compress" after. http://dev.gentoo.org/~dolsen/catalyst/ I've attached the current compress.py file so you can peruse and decide if making this a standalone gentoo project is desired. If not, it will get merged into catalyst code base. -- Brian Dolbec <dolsen@gentoo.org> [-- Attachment #2: compress.py --] [-- Type: text/x-python, Size: 10170 bytes --] # Maintained in full by: # Catalyst Team <catalyst@gentoo.org> # Release Engineering Team <releng@gentoo.org> ''' compress.py Utility class to hold and handle all possible compression and de-compression of files. Including rsync transfers. ''' import os from collections import namedtuple from support import cmd definition_fields = ["func", "cmd", "args", "id", "extension"] defintition_types = [ str, str, list, str, str] '''The definition entries are to follow the the definition_types with the exception of the first entry "Type" which is a mode identifier for use in the class as a type ID and printable string.''' compress_definitions = { "Type": ["Compression", "Compression definitions loaded"], "rsync" :["rsync", "rsync", ["-a", "--delete", "%(source)s", "%(destination)s"], "RSYNC", None], "lbzip2" :["_common", "tar", ["-I", "lbzip2", "-cf", "%(filename)s", "-C", "%(basedir)s", "%(source)s"], "LBZIP2", "tbz2"], "tbz2" :["_common", "tar", ["-I", "lbzip2", "-cf", "%(filename)s", "-C", "%(basedir)s", "%(source)s"], "LBZIP2", "tbz2"], "bz2" :["_common", "tar", ["-cpjf", "%(filename)s", "-C", "%(basedir)s", "%(source)s"], "BZIP2", "tar.bz2"], "tar" :["_common", "tar", ["-cpf", "%(filename)s", "-C", "%(basedir)s", "%(source)s"], "TAR", "tar"], "xz" :["_common", "tar", ["-cpJf", "%(filename)s", "-C", "%(basedir)s", "%(source)s"], "XZ", "tar.xz"], "pixz" :["_common", "tar", ["-I", "pixz", "-cpf", "%(filename)s", "-C", "%(basedir)s", "%(source)s"], "PIXZ", "xz"], "zip" :["_common", "tar", ["-cpzf", "%(filename)s", "-C", "%(basedir)s", "%(source)s"], "GZIP", "zip"], } decompress_definitions = { "Type": ["Decompression", "Decompression definitions loaded"], "rsync" :["rsync", "rsync", ["-a", "--delete", "%(source)s", "%(destination)s"], "RSYNC", None], "lbzip2" :["_common", "tar", ["-I", "lbzip2", "-xpf", "%(source)s", "-C", "%(destination)s"], "LBZIP2", "bz2"], "tbz2" :["_common", "tar", ["-I", "lbzip2", "-xpf", "%(source)s", "-C", "%(destination)s"], "LBZIP2", "tbz2"], "bz2" :["_common", "tar", ["-xpf", "%(source)s", "-C", "%(destination)s"], "BZIP2", "bz2"], "tar" :["_common", "tar", ["-xpf", "%(source)s", "-C", "%(destination)s"], "TAR", "tar"], "xz" :["_common", "tar", ["-xpf", "%(source)s", "-C", "%(destination)s"], "XZ", "xz"], "pixz" :["_common", "tar", ["-I", "pixz", "-xpf", "%(source)s", "-C", "%(destination)s"], "PIXZ", "xz"], "zip" :["_common", "tar", ["-xpzf", "%(source)s", "-C", "%(destination)s"], "GZIP", "zip"], "gz" :["_common", "tar", ["-xpzf", "%(source)s", "-C", "%(destination)s"], "GZIP", "zip"], } '''Configure this here in case it is ever changed. This is the only edit point required then.''' extension_separator = '.' class CompressMap(object): '''Class for handling Catalyst's compression & decompression of archives''' '''fields: list of ordered field names for the (de)compression functions''' fields = definition_fields[:] def __init__(self, definitions=None, env=None, default_mode=None, separator=extension_separator): '''Class init @param compress_mode: boolean, defaults to True describes compression or de-compression definitions loaded @param definitions: dictionary of Key:[function, cmd, cmd_args, Print/id string, extension] @param env: environment to pass to the cmd subprocess ''' if definitions is None: definitions = {} self.loaded_type = ["None", "No definitions loaded"] else: self.loaded_type = definitions.pop('Type') self.env = env or {} self.mode_error = self.loaded_type[0] + \ " Error: No mode was passed in or automatically detected" self._map = {} self.extension_separator = separator # set some defaults depending on what is being loaded if self.loaded_type[0] in ['Compression']: self.mode = default_mode or 'tbz2' self.compress = self._compress self.extract = None else: self.mode = default_mode or 'auto' self.compress = None self.extract = self._extract # create the (de)compression definition namedtuple classes for name in list(definitions): obj = namedtuple(name, self.fields) obj.__slots__ = () self._map[name] = obj._make(definitions[name]) del obj def _compress(self, infodict=None, filename='', source=None, basedir='.', mode=None, auto_extension=False, fatal=True): '''Compression function @param infodict: optional dictionary of the next 4 parameters. @param filename: optional string, name ot the file to make @param source: optional string, path to a directory @param destination: optional string, path a directory @param mode: string, optional mode to use to (de)compress with @return boolean ''' if not infodict: infodict = self.create_infodict(source, None, basedir, filename, mode or self.mode, auto_extension) if not infodict['mode']: print self.mode_error return False if auto_extension: infodict['auto-ext'] = True return self._run(infodict, fatal=fatal) def _extract(self, infodict=None, source=None, destination=None, mode=None, fatal=True): '''De-compression function @param infodict: optional dictionary of the next 3 parameters. @param source: optional string, path to a directory @param destination: optional string, path a directory @param mode: string, optional mode to use to (de)compress with @return boolean ''' if self.loaded_type[0] not in ["Decompression"]: return False if not infodict: infodict = self.create_infodict(source, destination, mode=mode) if infodict['mode'] in [None]: infodict['mode'] = self.mode or 'auto' if infodict['mode'] in ['auto']: infodict['mode'] = self.get_extension(infodict['source']) if not infodict['mode']: print self.mode_error return False return self._run(infodict, fatal=fatal) def _run(self, infodict, fatal=True): '''Internal function that runs the designated function @param source: string, path to a directory or file @param destination: string, path a directoy or file @param mode; string, desired method to perform the compression or transfer @return boolean ''' if not self.is_supported(infodict['mode']): print "mode: %s is not supported in the current %s definitions" \ % (infodict['mode'], self.loaded_type[1]) return False try: func = getattr(self, self._map[infodict['mode']].func) success = func(infodict, fatal) except AttributeError: print "FAILED to find function '%s'" % str(self._map[infodict['mode']].func) return False #except Exception as e: #msg = "Error performing %s %s, " % (mode, self.loaded_type[0]) + \ #"is the appropriate utility installed on your system?" #print msg #print "Exception:", e #return False return success def get_extension(self, source): '''Extracts the file extension string from the source file @param source: string, file path of the file to determine @return string: file type extension of the source file ''' #if self.extension_separator not in source: #return None #return source.rsplit(self.extension_separator, 1)[1] return os.path.splitext(source)[1] def rsync(self, infodict=None, source=None, destination=None, mode=None, fatal=True): '''Convienience function. Performs an rsync transfer @param infodict: optional dictionary of the next 3 parameters. @param source: optional string, path to a directory @param destination: optional string, path a directory @param mode: string, optional mode to use to (de)compress with @return boolean ''' if not infodict: if not mode: mode = 'rsync' infodict = self.create_infodict(source, destination, mode=mode) return self._run(infodict, fatal=fatal) def _common(self, infodict, fatal=True): '''Internal function. Performs commonly supported compression or decompression. @param files: dict as returned by this class's pair_files() @param mode: string, mode to use to (de)compress with @return boolean ''' if not infodict['mode'] or not self.is_supported(infodict['mode']): print "ERROR: CompressMap; %s mode: %s not correctly set!" \ % (self.loaded_type[0], infodict['mode']) return False #Avoid modifying the source dictionary cmdinfo = infodict.copy() cmdlist = self._map[cmdinfo['mode']] if cmdinfo['auto-ext']: cmdinfo['filename'] += self.extension_separator + \ cmdlist.extension # Do the string substitution opts = ' '.join(cmdlist.args) %(cmdinfo) args = ' '.join([cmdlist.cmd, opts]) return cmd(args, cmdlist.id, env=self.env, fatal=fatal) def create_infodict(self, source, destination=None, basedir=None, filename='', mode=None, auto_extension=False): '''Puts the source and destination paths into a dictionary for use in string substitution in the defintions %(source) and %(destination) fields embedded into the commands @param source: string, path to a directory @param destination: string, path a directory @param filename: optional string @return dict: ''' return { 'source': source, 'destination': destination, 'basedir': basedir, 'filename': filename, 'mode': mode or self.mode, 'auto-ext': auto_extension, } def is_supported(self, mode): '''Truth function to test the mode desired is supported in the definitions loaded @param mode: string, mode to use to (de)compress with @return boolean ''' return mode in list(self._map) @property def available_modes(self): '''Convienence function to return the available modes''' return list(self._map) def best_mode(self, prefered_mode, source): '''Compare the prefered_mode's extension with the source extension and returns the best choice @param prefered_mode: string @param source: string, path the the source file @return string: best mode to use for the extraction ''' if source.endswith(self._map[prefered_mode].extension): return prefered_mode return self.get_extension(source) def extension(self, mode): '''Returns the predetermined extension auto-ext added to the filename for compression. @param mode: string @return string ''' if self.is_supported(mode): return self._map[mode].extension return '' ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [gentoo-dev] [RFC] unpacker.eclass extensions 2013-06-17 20:37 ` Rick "Zero_Chaos" Farina 2013-06-17 21:41 ` Brian Dolbec @ 2013-06-19 21:15 ` Mike Frysinger 1 sibling, 0 replies; 12+ messages in thread From: Mike Frysinger @ 2013-06-19 21:15 UTC (permalink / raw To: gentoo-dev [-- Attachment #1: Type: Text/Plain, Size: 1032 bytes --] On Monday 17 June 2013 16:37:06 Rick "Zero_Chaos" Farina wrote: > On 06/17/2013 04:19 PM, Diego Elio Pettenò wrote: > > On 17/06/2013 17:54, Rick "Zero_Chaos" Farina wrote: > >> I make all my files with "tar cJf" > >> > >> zero@ozzie ~ % file /usr/portage/distfiles/gr-osmosdr-0.0.2.tar.xz > >> /usr/portage/distfiles/gr-osmosdr-0.0.2.tar.xz: XZ compressed data > > > > cJ with _current_ tar will generate XZ > > cJ with _past_ tar could generate lzma > > xJ with _current_ tar will extract both XZ and lzma > > zJ with _past_ tar will only extract lzma > > tar for the last several years properly and automatically detects the > format of the input using just 'x'. This is true for gnu and bsd tar > which should cover all of gentoo. and that relies on the magic being in the output format which, as we pointed out, wasn't the case. it's one of the reasons why the XZ format came to be. build lzma-utils and compress some files and see how well tar & file handle it: http://tukaani.org/lzma/ -mike [-- Attachment #2: This is a digitally signed message part. --] [-- Type: application/pgp-signature, Size: 836 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
* [gentoo-dev] unpacker.eclass: add decompress probe helper 2013-06-17 5:55 ` Mike Frysinger 2013-06-17 7:15 ` Diego Elio Pettenò @ 2013-06-22 17:55 ` Mike Frysinger 1 sibling, 0 replies; 12+ messages in thread From: Mike Frysinger @ 2013-06-22 17:55 UTC (permalink / raw To: gentoo-dev; +Cc: Vadim A. Misbakh-Soloviov, azamat.hackimov [-- Attachment #1: Type: Text/Plain, Size: 6471 bytes --] On Monday 17 June 2013 01:55:09 Mike Frysinger wrote: > i wish we could merge with the file detection in unpack_makeself somehow this patch should unify that aspect -mike --- unpacker.eclass 10 Apr 2013 14:47:49 -0000 1.13 +++ unpacker.eclass 22 Jun 2013 17:54:09 -0000 @@ -22,7 +22,14 @@ ___ECLASS_ONCE_UNPACKER="recur -_+^+_- s # @DEFAULT_UNSET # @DESCRIPTION: # Utility to use to decompress bzip2 files. Will dynamically pick between -# `pbzip2` and `bzip2`. Make sure your choice accepts the "-c" option. +# `pbzip2` and `bzip2`. Make sure your choice accepts the "-dc" options. +# Note: this is meant for users to set, not ebuilds. + +# @ECLASS-VARIABLE: UNPACKER_GZIP +# @DEFAULT_UNSET +# @DESCRIPTION: +# Utility to use to decompress gzip files. Will dynamically pick between +# `gzip` and `pigz`. Make sure your choice accepts the "-dc" options. # Note: this is meant for users to set, not ebuilds. # for internal use only (unpack_pdv and unpack_makeself) @@ -74,9 +81,9 @@ unpack_banner() { # parameter. Here is an example: # # @CODE -# vapier@vapier 0 pdv_unpack # strings hldsupdatetool.bin | grep lseek +# $ strings hldsupdatetool.bin | grep lseek # lseek -# vapier@vapier 0 pdv_unpack # strace -elseek ./hldsupdatetool.bin +# $ strace -elseek ./hldsupdatetool.bin # lseek(3, -4, SEEK_END) = 2981250 # @CODE # @@ -105,19 +112,17 @@ unpack_pdv() { local tmpfile="${T}/${FUNCNAME}" tail -c +$((${tailskip}+1)) ${src} 2>/dev/null | head -c 512 > "${tmpfile}" - local iscompressed=$(file -b "${tmpfile}") - if [[ ${iscompressed:0:8} == "compress" ]] ; then - iscompressed=1 + local comp=$(decompress_prog "${tmpfile}") + if [[ -n ${comp} ]] ; then + comp="| ${comp}" mv "${tmpfile}"{,.Z} gunzip "${tmpfile}" - else - iscompressed=0 fi - local istar=$(file -b "${tmpfile}") - if [[ ${istar:0:9} == "POSIX tar" ]] ; then - istar=1 + local istar filetype=$(file -b "${tmpfile}") + if [[ ${filetype} == "POSIX tar"* ]] ; then + istar=true else - istar=0 + istar=false fi #for some reason gzip dies with this ... dd cant provide buffer fast enough ? @@ -125,29 +130,18 @@ unpack_pdv() { # | dd ibs=${tailskip} skip=1 \ # | gzip -dc \ # > ${datafile} - if [ ${iscompressed} -eq 1 ] ; then - if [ ${istar} -eq 1 ] ; then - tail -c +$((${tailskip}+1)) "${src}" 2>/dev/null \ - | head -c $((${metaskip}-${tailskip})) \ - | tar -xzf - - else - tail -c +$((${tailskip}+1)) "${src}" 2>/dev/null \ - | head -c $((${metaskip}-${tailskip})) \ - | gzip -dc \ - > ${datafile} - fi + ( + th() { + tail -c +$(( tailskip + 1 )) "${src}" 2>/dev/null | \ + head -c $(( metaskip - tailskip )) + } + if ${istar} ; then + eval th ${comp} | tar --no-same-owner -xf - else - if [ ${istar} -eq 1 ] ; then - tail -c +$((${tailskip}+1)) "${src}" 2>/dev/null \ - | head -c $((${metaskip}-${tailskip})) \ - | tar --no-same-owner -xf - - else - tail -c +$((${tailskip}+1)) "${src}" 2>/dev/null \ - | head -c $((${metaskip}-${tailskip})) \ - > ${datafile} - fi + eval th ${comp} > "${datafile}" fi - true + exit 0 + ) #[ -s "${datafile}" ] || die "failure unpacking pdv ('${metaskip}' '${tailskip}' '${datafile}')" #assert "failure unpacking pdv ('${metaskip}' '${tailskip}' '${datafile}')" } @@ -218,31 +212,11 @@ unpack_makeself() { esac # lets grab the first few bytes of the file to figure out what kind of archive it is - local filetype tmpfile="${T}/${FUNCNAME}" + local tmpfile="${T}/${FUNCNAME}" eval ${exe} 2>/dev/null | head -c 512 > "${tmpfile}" - filetype=$(file -b "${tmpfile}") || die - case ${filetype} in - *tar\ archive*) - eval ${exe} | tar --no-same-owner -xf - - ;; - bzip2*) - eval ${exe} | bzip2 -dc | tar --no-same-owner -xf - - ;; - gzip*) - eval ${exe} | tar --no-same-owner -xzf - - ;; - compress*) - eval ${exe} | gunzip | tar --no-same-owner -xf - - ;; - XZ*) - eval ${exe} | unxz | tar --no-same-owner -xf - - ;; - *) - eerror "Unknown filetype \"${filetype}\" ?" - false - ;; - esac - assert "failure unpacking (${filetype}) makeself ${src##*/} ('${ver}' +${skip})" + local comp=$(decompress_prog "${tmpfile}") + eval ${exe} | eval "${comp:+${comp} |}" tar --no-same-owner -xf - + assert "failure unpacking makeself ${src##*/} ('${ver}' +${skip})" } # @FUNCTION: unpack_deb @@ -319,6 +293,54 @@ unpack_zip() { [[ $? -le 1 ]] || die "unpacking ${zip} failed (arch=unpack_zip)" } +# @FUNCTION: decompress_prog +# @USAGE: <mime|format|filename> +# @DESCRIPTION: +# Get the program name (and args) needed to decompress things. +# You can give this a mime type, or a format name (bz2/gz/xz/etc...), +# or you can give it a filename. If the type can be detected based on +# the suffix alone, we'll use that, otherwise we'll rely on `file` to +# probe the type. +decompress_prog() { + [[ $# -eq 1 ]] || die "Usage: ${FUNCNAME} <mime|format|filename>" + + local comp + + case $1 in + application/x-bzip2\;*|\ + bzip2|bz2|\ + *.bz2|*.tbz|*.tbz2) + local bzcmd=${PORTAGE_BZIP2_COMMAND:-$(type -P pbzip2 || type -P bzip2)} + local bzuncmd=${PORTAGE_BUNZIP2_COMMAND:-${bzcmd} -d} + : ${UNPACKER_BZ2:=${bzuncmd}} + comp="${UNPACKER_BZ2} -dc" + ;; + + application/x-gzip\;*|\ + application/x-compress\;*|\ + gzip|gz|\ + *.z|*.gz|*.tgz) + : ${UNPACKER_GZIP:=$(type -P pigz || type -P gzip)} + comp="${UNPACKER_GZIP} -dc" + ;; + + application/x-xz\;*|\ + lzma|xz|\ + *.lzma|*.xz|*.txz) + comp="xz -dc" + ;; + esac + + if [[ -n ${comp} ]] ; then + echo "${comp}" + return 0 + fi + + if [[ -s $1 ]] ; then + decompress_prog "$(file -ib "$1")" + fi +} + # @FUNCTION: _unpacker # @USAGE: <one archive to unpack> # @INTERNAL @@ -333,19 +355,7 @@ _unpacker() { a=$(find_unpackable_file "${a}") # first figure out the decompression method - case ${m} in - *.bz2|*.tbz|*.tbz2) - local bzcmd=${PORTAGE_BZIP2_COMMAND:-$(type -P pbzip2 || type -P bzip2)} - local bzuncmd=${PORTAGE_BUNZIP2_COMMAND:-${bzcmd} -d} - : ${UNPACKER_BZ2:=${bzuncmd}} - comp="${UNPACKER_BZ2} -c" - ;; - *.z|*.gz|*.tgz) - comp="gzip -dc" ;; - *.lzma|*.xz|*.txz) - comp="xz -dc" ;; - *) comp="" ;; - esac + comp=$(decompress_prog "${n}") # then figure out if there are any archiving aspects arch="" [-- Attachment #2: This is a digitally signed message part. --] [-- Type: application/pgp-signature, Size: 836 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2013-06-22 17:55 UTC | newest] Thread overview: 12+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2013-06-15 8:39 [gentoo-dev] [RFC] unpacker.eclass extensions Vadim A. Misbakh-Soloviov 2013-06-15 14:33 ` Markos Chandras 2013-06-15 14:37 ` Markos Chandras 2013-06-17 5:55 ` Mike Frysinger 2013-06-17 7:15 ` Diego Elio Pettenò 2013-06-17 16:08 ` Mike Frysinger 2013-06-17 16:54 ` Rick "Zero_Chaos" Farina 2013-06-17 20:19 ` Diego Elio Pettenò 2013-06-17 20:37 ` Rick "Zero_Chaos" Farina 2013-06-17 21:41 ` Brian Dolbec 2013-06-19 21:15 ` Mike Frysinger 2013-06-22 17:55 ` [gentoo-dev] unpacker.eclass: add decompress probe helper Mike Frysinger
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox