[gentoo-portage-dev] Re: mkinitrd

[gentoo-portage-dev] Re: mkinitrd

[Ned Ludd]

[-- Attachment #1: Type: text/plain, Size: 3268 bytes --]

On Wed, 2004-10-27 at 15:20, Peter S. Mazinger wrote:
> On Wed, 27 Oct 2004, Ned Ludd wrote:
> 
> > I've been thinking about initrd and I'd like to move down that path at some point for my own use.
> > Perhaps kernel+busybox for the initd then another partition for packages.
> > I'd like to go right from building packages on my own system to dishing (http:// || ftp:// ) them up from my $PKGDIR and installing right to the device when it's mounted +rw.
> > Problem I see however with the idea now is that the native portage format is a tad bloated (noman|nodoc|noinfo) all run at the installing phase vs the before packages phase which would force me to put a strip on the runtime device to keep packages small enough.
> > So my first goal is to get the dev-portage guys talking with me about redefining/extending the portage binary format a little.
> > 
> > hrmm I guess I'll subscribe to gentoo-portage-dev-subscribe@gentoo.org
> > and spark up a conversation.
> 
> Are you thinking of adding subpackaging and/or foreign format support, 
> like ipkg?

I have very simple ways todo ipkg format now that I can share with you if you want to deploy it now. 
But the topic of binary format handling is a hot topic (but nobody is really talking about it)
Guess I'll CC:  this to gentoo-portage-dev as my first post.

Right now portage only handles the .tbz2 format. This current format poses a few problems for it to really be useful for us on embedded devices. 
The tarballs contain man files/docs/info pages (lots of bloat to us).  None of the ELF's are stripped in the package.
Normally on our development systems we don't see these files as they don't get installed to the runtime system because FEATURES="noman noinfo nodoc" are set and this is handled in the dyn_install phase. So in order to get the desired behaviors inside we want it's almost like we need to do.
emerge pkg ; quickpkg pkg vs emerge -B package

Another problem that exists for us is that embedded systems usually never have python/perl/(your fav scripting lang here) installed. Without python installed I see no easy way to get at the additional package metadata thats joined into the .tbz2
This metadata is what holds the basic info like (what arch is this package for. USE/IUSE etc..) but even half of that is sealed up in a file called environment.bz2 so we have to decompress & extract the entire .tbz2 then parse (not source) the decompressed 'environment' file just to find out if a package is for our ARCH or not.
This is by no means very efficient.
Also the INSTALL_MASK feature needs work ( http://bugs.gentoo.org/show_bug.cgi?id=67190 )

Now to answer your orig question.
ipkg format. This is sorta of a yes & no thing.  In order for portage to better support more formats it should have abstraction layers between diff formats.
So first we must decide what formats we want to support. 
I'd think out of the gate we should support the most primitive of these pkg formats which is the classic .tar.gz (.tgz) format for the actual file system data.

Note to self: 
Look at source code for /usr/bin/tbz2tool
(busybox emerge binary applet?)

> 
> Peter
-- 
Ned Ludd <solar@gentoo.org>
Gentoo (hardened,security,infrastructure,embedded) Developer

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-portage-dev] Re: mkinitrd

[Ned Ludd]

[-- Attachment #1.1: Type: text/plain, Size: 4670 bytes --]

On Wed, 2004-10-27 at 18:43, Ned Ludd wrote:
> On Wed, 2004-10-27 at 15:20, Peter S. Mazinger wrote:
> > On Wed, 27 Oct 2004, Ned Ludd wrote:
> > 
> > > I've been thinking about initrd and I'd like to move down that path at some point for my own use.
> > > Perhaps kernel+busybox for the initd then another partition for packages.
> > > I'd like to go right from building packages on my own system to dishing (http:// || ftp:// ) them up from my $PKGDIR and installing right to the device when it's mounted +rw.
> > > Problem I see however with the idea now is that the native portage format is a tad bloated (noman|nodoc|noinfo) all run at the installing phase vs the before packages phase which would force me to put a strip on the runtime device to keep packages small enough.
> > > So my first goal is to get the dev-portage guys talking with me about redefining/extending the portage binary format a little.
> > > 
> > > hrmm I guess I'll subscribe to gentoo-portage-dev-subscribe@gentoo.org
> > > and spark up a conversation.
> > 
> > Are you thinking of adding subpackaging and/or foreign format support, 
> > like ipkg?

Here is a mocked up version to do what I(we?) need for embedded devices.
while trying to cope with the existing portage binary format.

uCpie etc # cat make.conf 
PKGDIR=/dev/shm/portage/packages/
PORTAGE_TMPDIR=/dev/shm/portage/tmp
PORTAGE_BINHOST=http://oc12.net/~solar/uclibc/gentoo/x86/packages/world/
INDEX_DIR=$PKGDIR
FEATURES="noman noinfo nodoc distclean"
INSTALL_MASK="*.a *.o *.h *.pl *.sample /usr/include /etc/init.d /etc/conf.d /usr/share/locale/"
INSTALL_MASK="${INSTALL_MASK} /usr/share/gcc-data/"

As we can see there is no easy way to overcome the (pre|post)inst functions if we pay attention to the flow of the script.
If we had stripping and no{man,doc,info} and INSTALL_MASK handled by portage before the  dyn_package() and a logical way to parse the extra metadata that a tbz2 provides then we could be in much better shape. 
One major advantage to a new binary format for us to not have to incur thr extra overhead from having to provide a strip util at runtime. You know now that I think of it the whole stripping setup in prepall/prepstrip phase is wrong.. So this is a bug..

Anybody from the portage team have anything in mind with portage's binary handling anytime soon? 
Extended it? Expanding it? Roadblocking changes to it?

> 
> I have very simple ways todo ipkg format now that I can share with you if you want to deploy it now. 
> But the topic of binary format handling is a hot topic (but nobody is really talking about it)
> Guess I'll CC:  this to gentoo-portage-dev as my first post.
> 
> Right now portage only handles the .tbz2 format. This current format poses a few problems for it to really be useful for us on embedded devices. 
> The tarballs contain man files/docs/info pages (lots of bloat to us).  None of the ELF's are stripped in the package.
> Normally on our development systems we don't see these files as they don't get installed to the runtime system because FEATURES="noman noinfo nodoc" are set and this is handled in the dyn_install phase. So in order to get the desired behaviors inside we want it's almost like we need to do.
> emerge pkg ; quickpkg pkg vs emerge -B package
> 
> Another problem that exists for us is that embedded systems usually never have python/perl/(your fav scripting lang here) installed. Without python installed I see no easy way to get at the additional package metadata thats joined into the .tbz2
> This metadata is what holds the basic info like (what arch is this package for. USE/IUSE etc..) but even half of that is sealed up in a file called environment.bz2 so we have to decompress & extract the entire .tbz2 then parse (not source) the decompressed 'environment' file just to find out if a package is for our ARCH or not.
> This is by no means very efficient.
> Also the INSTALL_MASK feature needs work ( http://bugs.gentoo.org/show_bug.cgi?id=67190 )
> 
> Now to answer your orig question.
> ipkg format. This is sorta of a yes & no thing.  In order for portage to better support more formats it should have abstraction layers between diff formats.
> So first we must decide what formats we want to support. 
> I'd think out of the gate we should support the most primitive of these pkg formats which is the classic .tar.gz (.tgz) format for the actual file system data.
> 
> Note to self: 
> Look at source code for /usr/bin/tbz2tool
> (busybox emerge binary applet?)
> 
> > 
> > Peter
-- 
Ned Ludd <solar@gentoo.org>
Gentoo (hardened,security,infrastructure,embedded) Developer

[-- Attachment #1.2: Type: text/plain, Size: 5847 bytes --]

#!/bin/busybox sh

[ -r /etc/rc ] && . /etc/rc
[ -r /etc/make.conf ] && . /etc/make.conf
[ "$ROOT" = "" ] && ROOT=/
[ "$INDEX_DIR" = "" ] && INDEX_DIR=/dev/shm/

einfo() {
	echo ">>> ${*}"
}

eerror() {
	echo "!!! ${*}"
}

die() {
	eerror $*
	exit 1
}

hasq() {
	local x
	local me=$1
	shift
	# All the TTY checks really only help out depend. Which is nice.
	# Logging kills all this anyway. Everything becomes a pipe. --NJ
	for x in "$@"; do
		if [ "${x}" = "${me}" ]; then
			[ "$VERBOSE" = 1 ] && echo $me
			return 0
		fi
	done
	return 1
}

fetched_package() {
	echo "fetched_package(${PKGDIR}/All/$1)"
	[ ! -e "${PKGDIR}/All/$1" ]  && return 0 || return 1
}

usage() {
cat <<__EOF__
  -h|--help        : This help
  -d|--debug       : set -x
  -p|--pretend     : pretend only
  -v|--verbose     : set verbose flag
  -f|--fetch       : fetch binary files
  -K|--usepkgonly  : use binary package
  -C|--unmerge     : uninstall a binary package
  --info|info      : display info
  --sync           : preform $opt operation
  --list           : update index and list packages
  --update         : update index file
  --clean          : clean up tmpfiles
  --root=<dir>     : define ROOT=$ROOT
  --index=<dir>    : define INDEX_DIR=$INDEX_DIR
  --pkgdir=<dir>   : define PKGDIR=$PKGDIR
  --tmpdir=<dir>   : define PORTAGE_TMPDIR=$PORTAGE_TMPDIR
  --binhost=<url>  : define PORTAGE_BINHOST=$PORTAGE_BINHOST
__EOF__
}

self_setup() {
	local foo="Try adding it to your /etc/make.conf or exporting it as an environment variable."
	[ "$PKGDIR" = "" ] && \
		die "PKGDIR is not set. $foo"
	[ "$PORTAGE_BINHOST" = "" ] && \
		die "PORTAGE_BINHOST is not set. $foo"
	[ "$PORTAGE_TMPDIR" = "" ] && \
		die "PORTAGE_TMPDIR is not set. $foo"
	[ -e "${PKGDIR}/All" ] || mkdir -p "${PKGDIR}/All"
	[ ! -e ${INDEX_DIR}/INDEX ] && export UPDATE=1
}

fetch_package() {
	local pkg=$1
	if fetched_package $pkg ; then
		wget -c -P ${PKGDIR}/All/ ${PORTAGE_BINHOST}/${pkg}
		[ $? != 0 ] && die "Error fetching ${pkg}"
	fi
}

update_index() {
	if [ "${UPDATE}" = 1 ] ; then
		wget -O ${INDEX_DIR}/INDEX ${PORTAGE_BINHOST}/INDEX
		[ $? != 0 ] && \
			die "Failed to fetch ${PORTAGE_BINHOST}/INDEX to ${INDEX_DIR}/INDEX"
	fi
}

if [ "$1" = "" ];then
	usage
	exit 0
fi

for opt in "$@" ; do
	case "${opt}" in
		-h|--help)	usage;exit 0;;
		-d|--debug)	set -x;;
		-p|--pretend)	PRETEND=1;;
		-v|--verbose)	VERBOSE=1;;
		-f|--fetch)	FETCH=1;;
		-K|--usepkgonly) USEPKG=1; FETCH=1;;
		-C|--unmerge)	REMOVEPKG=1;;
		--info|info)	set;exit 0;;
		--list)		LIST=1;UPDATE=1;;
		--update)	UPDATE=1;;
		--clean)	CLEAN=1;;
		--root=*)	ROOT="$(echo $opt | cut -d = -f 2)";;
		--pkgdir=*)	PKGDIR="$(echo $opt | cut -d = -f 2)";;
		--binhost=*)	PORTAGE_BINHOST="$(echo $opt | cut -d = -f 2)";;
		--tmpdir=*)	PORTAGE_TMPDIR="$(echo $opt | cut -d = -f 2)";;
		--index=*)	INDEX_DIR="$(echo $opt | cut -d = -f 2)";;
		*) pkgs="${pkgs} $opt";;
	esac
done

self_setup

file_ext=".tbz2"

tar_flags="x"
[ "${VERBOSE}" = 1 ] && tar_flags="${tar_flags}v"
[ "$file_ext" = ".tbz2" ] && tar_flags="j${tar_flags}" 
[ "$file_ext" = ".tbz" ] && tar_flags="z${tar_flags}" 

update_index
[ -e "${INDEX_DIR}/INDEX" ] && INDEX_CACHE="$(sort -u ${INDEX_DIR}/INDEX)" || INDEX_CACHE=""

if [ "$LIST" = 1 ] ; then
	for pkg in ${INDEX_CACHE}; do basename $pkg ${file_ext} ; done
	exit 0
fi

if [ "$FETCH"  = 1 ]; then
	for pkg in ${INDEX_CACHE}; do
		for p in ${pkgs}; do
			ok=0
			this_pkg="$(basename ${pkg} ${file_ext})"
			if [ "${this_pkg}" = "${p}" ]; then
				ok=1
			else
				[ "$(echo $this_pkg | cut -d '-' -f 1)" = "${p}" ] && ok=1
			fi
			[ $ok = 1 ] && fetch_package ${pkg}
		done
	done
fi

[ "$USEPKG" = 1 ] && for p in $pkgs; do
	list="$(ls -1 ${PKGDIR}/All/${p}*${file_ext} 2>/dev/null)"
	if [ "$list" = "" ]; then
		eerror "package ${p} not found. try --fetch"
	else
		for pkg in ${list}; do
			P=$(basename $pkg ${file_ext})
			IMAGE=${PORTAGE_TMPDIR}/${P}
			mkdir -p ${IMAGE}
			einfo "Extracting ${P} to ${IMAGE}"
			tar ${tar_flags}f ${PKGDIR}/All/$(basename ${pkg}) -C ${IMAGE}

			# Package cleanup and sstrip executables if nostrip is not set in features
			hasq noman   $FEATURES && rm -fR ${IMAGE}/usr/share/man ${IMAGE}/usr/man
			hasq nodoc   $FEATURES && rm -fR ${IMAGE}/usr/share/doc ${IMAGE}/usr/doc
			hasq noinfo  $FEATURES && rm -fR ${IMAGE}/usr/share/info ${IMAGE}/usr/info

			for f in ${INSTALL_MASK} ; do
				# remove dirs
				if [ -d "${IMAGE}/${f}" ] ; then
					einfo "Removing dir $f"
					rm -rf ${IMAGE}/${f}
				else
					cruft=$(find ${IMAGE} -name ${f})
					if [ "${cruft}" != "" ]; then
						einfo "Removing [$(echo -n "${cruft}" | tr ' ' '\n' | wc -l)] files matching $f"
						[ "$VERBOSE" = 1 ] && einfo "rm -rf ${cruft}"
						rm -rf "${cruft}"
					fi
				fi
			done
			hasq nostrip $FEATURES || \
				([ -e /usr/bin/sstrip && find ${IMAGE}/ -type f | xargs -n1 /usr/bin/sstrip ) 2> /dev/null
				
			cd ${IMAGE}
			find | xargs -n1 ls -ld | gzip >> ${PORTAGE_TMPDIR}/CONTENTS-${P}.gz
			cd -
			einfo $([ "$PRETEND" = 1 ] && echo Pretend) Installing to ${pkg} ${ROOT}
			gzip -d < ${PORTAGE_TMPDIR}/CONTENTS-${P}.gz
			if [ "$PRETEND" != 1 ]; then
				[ "$COUNTDOWN" = "" ] && COUNTDOWN="5 4 3 2 1"
				for cnt in ${COUNTDOWN}; do
					echo -ne "[$cnt] "
					sleep $cnt
				done
				cp -a ${IMAGE}/* ${ROOT}/
			fi
			hasq keeptemp $FEATURES || rm -fR ${IMAGE}
			if hasq distclean $FEATURES; then
				CLEAN=1
				size="$(/bin/ls -lh  ${pkg} | /bin/awk '{print $5}')"
				einfo "Freeing up ${size} by removing ${P}${file_ext}"
				rm ${pkg}
			fi
		done
	fi
done

if [ "$CLEAN" = 1 ]; then
	einfo "Cleaning up."
	rm -rf ${PORTAGE_TMPDIR}/*
fi

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-portage-dev] Re: mkinitrd

[Ned Ludd]

[-- Attachment #1: Type: text/plain, Size: 3145 bytes --]

On Thu, 2004-10-28 at 02:30, Peter S. Mazinger wrote:
> On Thu, 28 Oct 2004, Ned Ludd wrote:
> 
> > On Wed, 2004-10-27 at 18:43, Ned Ludd wrote:
> > > On Wed, 2004-10-27 at 15:20, Peter S. Mazinger wrote:
> > > > On Wed, 27 Oct 2004, Ned Ludd wrote:
> > > > 
> > > > > I've been thinking about initrd and I'd like to move down that path at some point for my own use.
> > > > > Perhaps kernel+busybox for the initd then another partition for packages.
> > > > > I'd like to go right from building packages on my own system to dishing (http:// || ftp:// ) them up from my $PKGDIR and installing right to the device when it's mounted +rw.
> > > > > Problem I see however with the idea now is that the native portage format is a tad bloated (noman|nodoc|noinfo) all run at the installing phase vs the before packages phase which would force me to put a strip on the runtime device to keep packages small enough.
> > > > > So my first goal is to get the dev-portage guys talking with me about redefining/extending the portage binary format a little.
> > > > > 
> > > > > hrmm I guess I'll subscribe to gentoo-portage-dev-subscribe@gentoo.org
> > > > > and spark up a conversation.
> > > > 
> > > > Are you thinking of adding subpackaging and/or foreign format support, 
> > > > like ipkg?
> > 
> > Here is a mocked up version to do what I(we?) need for embedded devices.
> > while trying to cope with the existing portage binary format.
> > 
> > uCpie etc # cat make.conf 
> > PKGDIR=/dev/shm/portage/packages/
> > PORTAGE_TMPDIR=/dev/shm/portage/tmp
> > PORTAGE_BINHOST=http://oc12.net/~solar/uclibc/gentoo/x86/packages/world/
> > INDEX_DIR=$PKGDIR
> > FEATURES="noman noinfo nodoc distclean"
> > INSTALL_MASK="*.a *.o *.h *.pl *.sample /usr/include /etc/init.d /etc/conf.d /usr/share/locale/"
> > INSTALL_MASK="${INSTALL_MASK} /usr/share/gcc-data/"
> 
> Do we not need /etc/conf.d /etc/init.d? I am not sure about that. I 
> can't imagine that we will have a real baselayout-lite, that will work  
> w/o these. I would also add *.pyc, *.pyo.
> 
> What does distclean?

In the context of the script it does 
if hasq distclean $FEATURES; then
	CLEAN=1
	size="$(/bin/ls -lh  ${pkg} | /bin/awk '{print $5}')"
	einfo "Freeing up ${size} by removing ${P}${file_ext}"
	rm ${pkg}
fi

which would remove the package after it's done installing it. 
Being that we are only using a tmpfs this all sits in memory. So no need to keep a large distfiles tre hanging around chewing up memory.
I also implanted this feature in an /etc/portage/bashrc I posted to gentoo-dev titled "portage toys"

> 
> Your busybox "applet" uses /dev/shm. This is not supported/needed for 
> uClibc, so it is not sure to have it set up, and I do not know, if the 
> PKGDIR[_TMPDIR] dirs do not need dev? What happens if /dev/shm is mounted 
> w/ nodev, or similar options discussed on the hardened list and proposed 
> as default on ml?

yeah? i defined it to use /dev/shm which I happen to mount rw on my
devs..


> 
> Peter
-- 
Ned Ludd <solar@gentoo.org>
Gentoo (hardened,security,infrastructure,embedded) Developer

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]