[gentoo-pms] xpak documentation

[gentoo-pms] xpak documentation

[Lars Hartmann]

[-- Attachment #1.1: Type: text/plain, Size: 598 bytes --]

Hi,

i have stumbled across this document a few days ago and realized that
there is no documentation for the xpak format as used in gentoo binary
packages.

Attached is a technical documentation of the xpak format i have written
some time ago, that you could use as a source if you want, as i am the
only author of it, i could also give you permission to redistribute it
under whatever license you want.

Greetings Lars Hartmann
-- 
/"\    ASCII Ribbon Campaign
\ /    Respect for low technology.
 X     Keep e-mail messages readable by any computer system.
/ \    Keep it ASCII. 

[-- Attachment #1.2: xpak.txt --]
[-- Type: text/x-vhdl, Size: 7339 bytes --]

+-----------------------------------------------------------------------------+
| Copyright (c) 2008, Lars Hartmann                                           |
| All rights reserved.                                                        |
|                                                                             |
| Redistribution and use in source and binary forms, with or without          |
| modification, are permitted provided that the following conditions are met: |
|                                                                             |
|    1. Redistributions of source code must retain the above copyright        |
|       notice, this list of conditions and the following disclaimer.         |
|                                                                             |
|    2. Redistributions in binary form must reproduce the above copyright     |
|       notice, this list of conditions and the following disclaimer in the   |
|       documentation and/or other materials provided with the distribution.  |
|                                                                             |
|    3. Neither the name of the Original Author, nor the names of its         |
|       contributors may be used to endorse or promote products derived from  |
|       this software without specific prior written permission.              |
|                                                                             |
| THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" |
| AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE   |
| IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE  |
| ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE    |
| LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR         |
| CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF        |
| SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS    |
| INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN     |
| CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)     |
| ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE  |
| POSSIBILITY OF SUCH DAMAGE.                                                 |
+-----------------------------------------------------------------------------+

                             The XPAK Data Format
                             ====================

Notes
=====

 Data Types
 ==========

  Integer: every offset or length(len) value in this documentation will be an
           unsigned 32bit integer in big endian byte order(32bit unsigned big
           int or uint32(big) ).

  String:  All Strings, Mentioned in this Documentation are ASCII encoded, and
           not Null terminated

  Values:  The actual values of the individual xpak entries are stored as
           Strings.

 Vertical Bars
 =============

  The vertical bars '|' are not part of the file format, they are merely used
  to illustrate how the offset values apply to the data.

Synopsis
========

  <tarball>:      |<-xpak_offset->|
             <tar>|<    xpak     >|<xpak_offset>"STOP"

  <xpak>   : "XPAKPACK"<index_len><data_len><index><data>"XPAKSTOP"

  <index>  : |<-------------index_len------------->|
             |<index1><index2><index3><index4><...>|

  <indexN> :           |<-name_len->|
             <name_len>|<   name   >|<data_offset><data_len>

  <data>   : |<--------------data_len------------->|
             |<-dataN_offset->|<-dataN_len->|
             |<     data     >|<  data_N   >|<data>|

Detailed
========

 Every gentoo binary package has a xpak attached to it which contains build
 time information like the use flags it was built with, the ebuild it was
 built from, the environmental variables, CFLAGs, CXXFLAGs, ....

 lets have a look at this XPAK:

 If you look at a gentoo binary package (binpkg) with a hex-editor you'll
 notice the that after the data, which belongs to the tarball you find a
 binary blob - the <xpak>, an offset which holds the bytes from the start of
 the <xpak> to the end of the file - <xpak_offset> and finally the String
 "STOP":

                            |<-xpak_offset->|
                      <tbz2>|<     xpak    >|<xpak_offset>"STOP"|

 Here you see the tbz2 archive, and the attached xpak-blob, the xpak-offset
 and the string "STOP" at the end.

 If we read the offset and count <offset> bytes backwards from the start of
 xpak_offset, we have found the start of the XPAK Block which starts with the
 String "XPAKPACK". this xpak block consists of the string "XPAKPACK", the
 length of the index block - <index-len>, the length of the data block -
 <data-len>, a <index-len> bytes long binary blob which holds the index, a
 <data-len> bytes long binary blob which holds the data and the String
 "XPAKSTOP" at the end:

                                   |<-index_len->|<-data_len->|
    "XPAKPACK"<index_len><data_len>|<   index   >|<   data   >|"XPAKSTOP"

 To actually get the index and the data, we cut out <index_len> bytes, after
 the end of <data_len> and for the index block and cut out the next <data_len>
 bytes for the data block. If we have done everything right up to this point,
 the following bytes would be the ASCII formatted string "XPAKSTOP".

 The actual data is truncated to one big block - so if we want to read it we
 need the actual positions of each information in this big data block, this
 information can be obtained from the indices which are stored in the index
 block.

 Now we take a closer look at the index block:
 The index block consists of several truncated index blocks:

          |<-----------------------index_len---------------------->|
          |<index1><index2><index3><index4><index5><index6><index7>|

 The index block holds all information we need to find the data we want in the
 <data> block. It consists of truncated index elements with a length
 <index_len>. Each of those index elements stands for one information in the
 data block and consists of the length of its name - <name_len>, a <name_len>
 bytes long String - the Name of the data block, this index belongs to, the
 offset of the data block (counted from the beginning of <data>) -
 <data_offset>, and the length of that data block - <data_len>:

                         |<-name_len->|
               <name_len>|<   name   >|<dataN_offset><dataN_len>

 Now lets take a closer look at the data block: the data block contains
 truncated data blocks with a total length of <data_len>:

         |<------------------------data_len------------------------>|
         |<data1><data2><data3><data4><data5><data6><data7><data...>|

 This binary block with a length of <data_len> bytes, consists of truncated
 data.

 To select one data element, we need the <data_offset> and the <data_len> from
 the index, if we have those we can count <data_offset> bytes from the start
 of the data block, and then cut out the next <data_len> bytes. there we got
 our data Block:

                  |<-----dataN_offset----->|<--dataN_len->|
                  |<data1data2data3data...>|<data-we-want>|

[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-pms] xpak documentation

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 404 bytes --]

On Sun, 01 Feb 2009 11:48:41 +0100
Lars Hartmann <lars@chaotika.org> wrote:
> i have stumbled across this document a few days ago and realized that
> there is no documentation for the xpak format as used in gentoo binary
> packages.

Mm, that's because I'm not convinced the binary format Portage uses
just now is standard-worthy. I'd rather keep PMS restricted in scope.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-pms] xpak documentation

[Lars Hartmann]

[-- Attachment #1: Type: text/plain, Size: 850 bytes --]

I am wondering about the following two points:
     1. Is there an effort to define a standard for binary packages
        especially the way how the metadata are attached?
     2. Are there any other alternatives to xpak?
Am Sonntag, den 01.02.2009, 14:53 +0000 schrieb Ciaran McCreesh:
> On Sun, 01 Feb 2009 11:48:41 +0100
> Lars Hartmann <lars@chaotika.org> wrote:
> > i have stumbled across this document a few days ago and realized that
> > there is no documentation for the xpak format as used in gentoo binary
> > packages.
> 
> Mm, that's because I'm not convinced the binary format Portage uses
> just now is standard-worthy. I'd rather keep PMS restricted in scope.
> 
-- 
/"\    ASCII Ribbon Campaign
\ /    Respect for low technology.
 X     Keep e-mail messages readable by any computer system.
/ \    Keep it ASCII. 

[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]