[gentoo-embedded] Re: [gentoo-dev] Portage in Embedded Systems?

[gentoo-embedded] Re: [gentoo-dev] Portage in Embedded Systems?

[Ned Ludd]

[-- Attachment #1: Type: text/plain, Size: 3576 bytes --]

On Mon, 2004-10-04 at 23:33, Chris White wrote:
> All,
> 
>     Maybe I've just not seen this, but what sort of footprint does 
> portage leave on embedded systems with low cpu/ram?  

I think what your asking about here is what we are starting to coin as a
gentoo-lite system. A lot of people are gaining an interest in this.
Running gentoo with portage on crappy old hardware. Or they just want
the performance boast and lower memory usage. For a system like this I'd
think you would want atleast a P75 with atleast 32M of of ram. 
Portage is not so big. But python itself is a beast.

In one experiment I've managed to get pythons runtime down to about 2
Megs of HD space. And in another experiment with the portage tree itself
I managed to get it down to 14 Megs with the use of squashfs and
excluding a few things from the tree which I know are not needed to do
emerge system. But a full portage-rsync tree compressed was about 17
Megs

Now if we are talking embedded systems in the way I like to think of
them (ie firmware) then the min requirements are about 4M of Ram and 3M
of flash space using a semi default setup, give or take depending on the
device your building for.

> I ask this mainly 
> because it's a Good Thing To Know (tm) considering the last LWE 
> conference was full of people asking about using Gentoo for embedded 
> devices. 
portage needs work and a fair bit of it.
Other than myself and mike more people need to propose ideas to the
portage team to make things more flexible.

>  Something like higher end Palms may be able to dish it out, 
> but what happens when you get to lower end palms or even cell phones?

What about them?
Most cell phones are ARM based.
Mike Frysinger is currently working on generic uclibc arm little endian
stages. When he has those complete (and most of the bugs worked out)
I'll start on generic uclibc arm big endian stages. When I have those
complete and I'm happy with it I'm going to ship the device off to OSU
so we can continue to support the arch from a (le||be) perspective. The
unit I will be developing with is a nslu2 that was a donation to the
gentoo embedded project thanks to the guys over at the nslu2-linux
project (http://www.nslu2-linux.org/) who had a fund raiser in order to
get me one. They ended up getting 9x the amount in donations needed to
send me a unit and were able to send them to a number of other embedded
projects.

Unfortunately there are a few drawbacks to our embedded support right
now. 
1) Lack of skilled (wo)manpower.
2) Lack of proper cross-toolchain handling by portage. So everything is
considered native-* vs cross-* (this means you must use the same host
arch as your target arch) or use a binfmt_elf kernel module to emulate
your target arch.
3) package management for embedded devices. (no all devices are
read-only)
- ipkg format seems ideal here but I/we have not enough input from the
community to tell what will be ideal in the long run.

If anybody that has a decent level of cross compiling experience and
thinks that they would be interested in gentoo supporting better cross-*
support please contact me. (seriously motivated people only)

>  
> Thanks ahead of time for any/all comments and hold on (*ChrisWhite 
> prepares fireproof suit)... and flames.

hrmm flames.. none right now but as soon as I can think of something or
get a blowtorch I'll be sure to direct it your way.

-- 
Ned Ludd <solar@gentoo.org>
Gentoo (hardened,security,infrastructure,embedded,toolchain) Developer

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[gentoo-embedded] Re: [gentoo-dev] Portage in Embedded Systems?

[Ned Ludd]

[-- Attachment #1: Type: text/plain, Size: 4097 bytes --]

Oh yeah one more sexy thing that we are rather proud of. 
Our stage(3)
http://gentoo.osuosl.org/experimental/x86/embedded/stages/
stage3-x86-uclibc-20040927.tar.bz2     27-Sep-2004 23:07   34M

Standard stage(3)
http://gentoo.osuosl.org/releases/x86/2004.2/stages/x86/
stage3-x86-2004.2.tar.bz2     22-Jul-2004 08:41   89M

On Tue, 2004-10-05 at 19:09, Ned Ludd wrote:
> On Mon, 2004-10-04 at 23:33, Chris White wrote:
> > All,
> > 
> >     Maybe I've just not seen this, but what sort of footprint does 
> > portage leave on embedded systems with low cpu/ram?  
> 
> I think what your asking about here is what we are starting to coin as a
> gentoo-lite system. A lot of people are gaining an interest in this.
> Running gentoo with portage on crappy old hardware. Or they just want
> the performance boast and lower memory usage. For a system like this I'd
> think you would want atleast a P75 with atleast 32M of of ram. 
> Portage is not so big. But python itself is a beast.
> 
> In one experiment I've managed to get pythons runtime down to about 2
> Megs of HD space. And in another experiment with the portage tree itself
> I managed to get it down to 14 Megs with the use of squashfs and
> excluding a few things from the tree which I know are not needed to do
> emerge system. But a full portage-rsync tree compressed was about 17
> Megs
> 
> Now if we are talking embedded systems in the way I like to think of
> them (ie firmware) then the min requirements are about 4M of Ram and 3M
> of flash space using a semi default setup, give or take depending on the
> device your building for.
> 
> > I ask this mainly 
> > because it's a Good Thing To Know (tm) considering the last LWE 
> > conference was full of people asking about using Gentoo for embedded 
> > devices. 
> portage needs work and a fair bit of it.
> Other than myself and mike more people need to propose ideas to the
> portage team to make things more flexible.
> 
> >  Something like higher end Palms may be able to dish it out, 
> > but what happens when you get to lower end palms or even cell phones?
> 
> What about them?
> Most cell phones are ARM based.
> Mike Frysinger is currently working on generic uclibc arm little endian
> stages. When he has those complete (and most of the bugs worked out)
> I'll start on generic uclibc arm big endian stages. When I have those
> complete and I'm happy with it I'm going to ship the device off to OSU
> so we can continue to support the arch from a (le||be) perspective. The
> unit I will be developing with is a nslu2 that was a donation to the
> gentoo embedded project thanks to the guys over at the nslu2-linux
> project (http://www.nslu2-linux.org/) who had a fund raiser in order to
> get me one. They ended up getting 9x the amount in donations needed to
> send me a unit and were able to send them to a number of other embedded
> projects.
> 
> Unfortunately there are a few drawbacks to our embedded support right
> now. 
> 1) Lack of skilled (wo)manpower.
> 2) Lack of proper cross-toolchain handling by portage. So everything is
> considered native-* vs cross-* (this means you must use the same host
> arch as your target arch) or use a binfmt_elf kernel module to emulate
> your target arch.
> 3) package management for embedded devices. (no all devices are
> read-only)
> - ipkg format seems ideal here but I/we have not enough input from the
> community to tell what will be ideal in the long run.
> 
> If anybody that has a decent level of cross compiling experience and
> thinks that they would be interested in gentoo supporting better cross-*
> support please contact me. (seriously motivated people only)
> 
> >  
> > Thanks ahead of time for any/all comments and hold on (*ChrisWhite 
> > prepares fireproof suit)... and flames.
> 
> hrmm flames.. none right now but as soon as I can think of something or
> get a blowtorch I'll be sure to direct it your way.
-- 
Ned Ludd <solar@gentoo.org>
Gentoo (hardened,security,infrastructure,embedded,toolchain) Developer

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-embedded] Re: [gentoo-dev] Portage in Embedded Systems?

[Marius Schäfer]

On Wednesday 06 October 2004 01:09, Ned Ludd wrote:
> On Mon, 2004-10-04 at 23:33, Chris White wrote:
> > All,
> >
> >     Maybe I've just not seen this, but what sort of footprint does
> > portage leave on embedded systems with low cpu/ram?
>
> I think what your asking about here is what we are starting to coin as a
> gentoo-lite system. A lot of people are gaining an interest in this.
> Running gentoo with portage on crappy old hardware. Or they just want
> the performance boast and lower memory usage. For a system like this I'd
> think you would want atleast a P75 with atleast 32M of of ram.
> Portage is not so big. But python itself is a beast.
>
> In one experiment I've managed to get pythons runtime down to about 2
> Megs of HD space. And in another experiment with the portage tree itself

How did you do that? I would be interessted very much because of an other 
project of mine.

Thanks
Marius

> I managed to get it down to 14 Megs with the use of squashfs and
> excluding a few things from the tree which I know are not needed to do
> emerge system. But a full portage-rsync tree compressed was about 17
> Megs
>
> Now if we are talking embedded systems in the way I like to think of
> them (ie firmware) then the min requirements are about 4M of Ram and 3M
> of flash space using a semi default setup, give or take depending on the
> device your building for.
>
> > I ask this mainly
> > because it's a Good Thing To Know (tm) considering the last LWE
> > conference was full of people asking about using Gentoo for embedded
> > devices.
>
> portage needs work and a fair bit of it.
> Other than myself and mike more people need to propose ideas to the
> portage team to make things more flexible.
>
> >  Something like higher end Palms may be able to dish it out,
> > but what happens when you get to lower end palms or even cell phones?
>
> What about them?
> Most cell phones are ARM based.
> Mike Frysinger is currently working on generic uclibc arm little endian
> stages. When he has those complete (and most of the bugs worked out)
> I'll start on generic uclibc arm big endian stages. When I have those
> complete and I'm happy with it I'm going to ship the device off to OSU
> so we can continue to support the arch from a (le||be) perspective. The
> unit I will be developing with is a nslu2 that was a donation to the
> gentoo embedded project thanks to the guys over at the nslu2-linux
> project (http://www.nslu2-linux.org/) who had a fund raiser in order to
> get me one. They ended up getting 9x the amount in donations needed to
> send me a unit and were able to send them to a number of other embedded
> projects.
>
> Unfortunately there are a few drawbacks to our embedded support right
> now.
> 1) Lack of skilled (wo)manpower.
> 2) Lack of proper cross-toolchain handling by portage. So everything is
> considered native-* vs cross-* (this means you must use the same host
> arch as your target arch) or use a binfmt_elf kernel module to emulate
> your target arch.
> 3) package management for embedded devices. (no all devices are
> read-only)
> - ipkg format seems ideal here but I/we have not enough input from the
> community to tell what will be ideal in the long run.
>
> If anybody that has a decent level of cross compiling experience and
> thinks that they would be interested in gentoo supporting better cross-*
> support please contact me. (seriously motivated people only)
>
> > Thanks ahead of time for any/all comments and hold on (*ChrisWhite
> > prepares fireproof suit)... and flames.
>
> hrmm flames.. none right now but as soon as I can think of something or
> get a blowtorch I'll be sure to direct it your way.

--
gentoo-embedded@gentoo.org mailing list

Re: [gentoo-embedded] Re: [gentoo-dev] Portage in Embedded Systems?

[Ned Ludd]

[-- Attachment #1: Type: text/plain, Size: 5075 bytes --]

On Thu, 2004-10-07 at 12:08, Marius Schäfer wrote:
> On Wednesday 06 October 2004 01:09, Ned Ludd wrote:
> > On Mon, 2004-10-04 at 23:33, Chris White wrote:
> > > All,
> > >
> > >     Maybe I've just not seen this, but what sort of footprint does
> > > portage leave on embedded systems with low cpu/ram?
> >
> > I think what your asking about here is what we are starting to coin as a
> > gentoo-lite system. A lot of people are gaining an interest in this.
> > Running gentoo with portage on crappy old hardware. Or they just want
> > the performance boast and lower memory usage. For a system like this I'd
> > think you would want atleast a P75 with atleast 32M of of ram.
> > Portage is not so big. But python itself is a beast.
> >
> > In one experiment I've managed to get pythons runtime down to about 2
> > Megs of HD space. And in another experiment with the portage tree itself
> 
> How did you do that? I would be interessted very much because of an other 
> project of mine.

compile with -Os
delete the crap modules that you will never need. 
test/lib-tk/curses and a few others that you don't really need.
rm all the .pyo/.pyc files.
export PYTHON_DONTCOMPILE=1 ; so they never get compiled again. This is
gentoo patch only. Naturally it wont be as fast cuz your using non
bytecoded versions.
kill off the /usr/include/python2.*/*.h files
About 2M of fat can be kill to nuking the config dir
/usr/lib/python2.*/config/


If I were you and your developing an app that needs python I'd do
something like this.

strace -eopen -o python-foo.txt
cat python-foo.txt | grep -v "No such file or directory"  | grep python
|cut -d \" -f 2 | grep -v pyo$ | grep -v pyc$


Oh and if you wanted to make it nicer you could patch python itself to
read compressed .py files using gzopen/gzread or the likes. Or make your
/usr/lib/python2.* a compressed read-only file which you mount in a
loopback. Probably get the whole mess to <=1M that way.

-good luck.



> 
> Thanks
> Marius
> 
> > I managed to get it down to 14 Megs with the use of squashfs and
> > excluding a few things from the tree which I know are not needed to do
> > emerge system. But a full portage-rsync tree compressed was about 17
> > Megs
> >
> > Now if we are talking embedded systems in the way I like to think of
> > them (ie firmware) then the min requirements are about 4M of Ram and 3M
> > of flash space using a semi default setup, give or take depending on the
> > device your building for.
> >
> > > I ask this mainly
> > > because it's a Good Thing To Know (tm) considering the last LWE
> > > conference was full of people asking about using Gentoo for embedded
> > > devices.
> >
> > portage needs work and a fair bit of it.
> > Other than myself and mike more people need to propose ideas to the
> > portage team to make things more flexible.
> >
> > >  Something like higher end Palms may be able to dish it out,
> > > but what happens when you get to lower end palms or even cell phones?
> >
> > What about them?
> > Most cell phones are ARM based.
> > Mike Frysinger is currently working on generic uclibc arm little endian
> > stages. When he has those complete (and most of the bugs worked out)
> > I'll start on generic uclibc arm big endian stages. When I have those
> > complete and I'm happy with it I'm going to ship the device off to OSU
> > so we can continue to support the arch from a (le||be) perspective. The
> > unit I will be developing with is a nslu2 that was a donation to the
> > gentoo embedded project thanks to the guys over at the nslu2-linux
> > project (http://www.nslu2-linux.org/) who had a fund raiser in order to
> > get me one. They ended up getting 9x the amount in donations needed to
> > send me a unit and were able to send them to a number of other embedded
> > projects.
> >
> > Unfortunately there are a few drawbacks to our embedded support right
> > now.
> > 1) Lack of skilled (wo)manpower.
> > 2) Lack of proper cross-toolchain handling by portage. So everything is
> > considered native-* vs cross-* (this means you must use the same host
> > arch as your target arch) or use a binfmt_elf kernel module to emulate
> > your target arch.
> > 3) package management for embedded devices. (no all devices are
> > read-only)
> > - ipkg format seems ideal here but I/we have not enough input from the
> > community to tell what will be ideal in the long run.
> >
> > If anybody that has a decent level of cross compiling experience and
> > thinks that they would be interested in gentoo supporting better cross-*
> > support please contact me. (seriously motivated people only)
> >
> > > Thanks ahead of time for any/all comments and hold on (*ChrisWhite
> > > prepares fireproof suit)... and flames.
> >
> > hrmm flames.. none right now but as soon as I can think of something or
> > get a blowtorch I'll be sure to direct it your way.
-- 
Ned Ludd <solar@gentoo.org>
Gentoo (hardened,security,infrastructure,embedded,toolchain) Developer

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-embedded] Re: [gentoo-dev] Portage in Embedded Systems?

[Mike Frysinger]

On Thursday 07 October 2004 12:37 pm, Ned Ludd wrote:
> On Thu, 2004-10-07 at 12:08, Marius Schäfer wrote:
> > On Wednesday 06 October 2004 01:09, Ned Ludd wrote:
> > > On Mon, 2004-10-04 at 23:33, Chris White wrote:
> > > > All,
> > > >
> > > >     Maybe I've just not seen this, but what sort of footprint does
> > > > portage leave on embedded systems with low cpu/ram?
> > >
> > > I think what your asking about here is what we are starting to coin as
> > > a gentoo-lite system. A lot of people are gaining an interest in this.
> > > Running gentoo with portage on crappy old hardware. Or they just want
> > > the performance boast and lower memory usage. For a system like this
> > > I'd think you would want atleast a P75 with atleast 32M of of ram.
> > > Portage is not so big. But python itself is a beast.
> > >
> > > In one experiment I've managed to get pythons runtime down to about 2
> > > Megs of HD space. And in another experiment with the portage tree
> > > itself
> >
> > How did you do that? I would be interessted very much because of an other
> > project of mine.
>
> compile with -Os
> delete the crap modules that you will never need.
> test/lib-tk/curses and a few others that you don't really need.
> rm all the .pyo/.pyc files.
> export PYTHON_DONTCOMPILE=1 ; so they never get compiled again. This is
> gentoo patch only. Naturally it wont be as fast cuz your using non
> bytecoded versions.
> kill off the /usr/include/python2.*/*.h files
> About 2M of fat can be kill to nuking the config dir
> /usr/lib/python2.*/config/
>
>
> If I were you and your developing an app that needs python I'd do
> something like this.
>
> strace -eopen -o python-foo.txt
> cat python-foo.txt | grep -v "No such file or directory"  | grep python
>
> |cut -d \" -f 2 | grep -v pyo$ | grep -v pyc$
>
> Oh and if you wanted to make it nicer you could patch python itself to
> read compressed .py files using gzopen/gzread or the likes. Or make your
> /usr/lib/python2.* a compressed read-only file which you mount in a
> loopback. Probably get the whole mess to <=1M that way.
>
> -good luck.
>
> > Thanks
> > Marius
> >
> > > I managed to get it down to 14 Megs with the use of squashfs and
> > > excluding a few things from the tree which I know are not needed to do
> > > emerge system. But a full portage-rsync tree compressed was about 17
> > > Megs
> > >
> > > Now if we are talking embedded systems in the way I like to think of
> > > them (ie firmware) then the min requirements are about 4M of Ram and 3M
> > > of flash space using a semi default setup, give or take depending on
> > > the device your building for.
> > >
> > > > I ask this mainly
> > > > because it's a Good Thing To Know (tm) considering the last LWE
> > > > conference was full of people asking about using Gentoo for embedded
> > > > devices.
> > >
> > > portage needs work and a fair bit of it.
> > > Other than myself and mike more people need to propose ideas to the
> > > portage team to make things more flexible.
> > >
> > > >  Something like higher end Palms may be able to dish it out,
> > > > but what happens when you get to lower end palms or even cell phones?
> > >
> > > What about them?
> > > Most cell phones are ARM based.
> > > Mike Frysinger is currently working on generic uclibc arm little endian
> > > stages. When he has those complete (and most of the bugs worked out)
> > > I'll start on generic uclibc arm big endian stages. When I have those
> > > complete and I'm happy with it I'm going to ship the device off to OSU
> > > so we can continue to support the arch from a (le||be) perspective. The
> > > unit I will be developing with is a nslu2 that was a donation to the
> > > gentoo embedded project thanks to the guys over at the nslu2-linux
> > > project (http://www.nslu2-linux.org/) who had a fund raiser in order to
> > > get me one. They ended up getting 9x the amount in donations needed to
> > > send me a unit and were able to send them to a number of other embedded
> > > projects.
> > >
> > > Unfortunately there are a few drawbacks to our embedded support right
> > > now.
> > > 1) Lack of skilled (wo)manpower.
> > > 2) Lack of proper cross-toolchain handling by portage. So everything is
> > > considered native-* vs cross-* (this means you must use the same host
> > > arch as your target arch) or use a binfmt_elf kernel module to emulate
> > > your target arch.
> > > 3) package management for embedded devices. (no all devices are
> > > read-only)
> > > - ipkg format seems ideal here but I/we have not enough input from the
> > > community to tell what will be ideal in the long run.
> > >
> > > If anybody that has a decent level of cross compiling experience and
> > > thinks that they would be interested in gentoo supporting better
> > > cross-* support please contact me. (seriously motivated people only)
> > >
> > > > Thanks ahead of time for any/all comments and hold on (*ChrisWhite
> > > > prepares fireproof suit)... and flames.
> > >
> > > hrmm flames.. none right now but as soon as I can think of something or
> > > get a blowtorch I'll be sure to direct it your way.

--
gentoo-embedded@gentoo.org mailing list

Re: [gentoo-embedded] Re: [gentoo-dev] Portage in Embedded Systems?

[Mike Frysinger]

On Thursday 07 October 2004 12:37 pm, Ned Ludd wrote:
> Oh and if you wanted to make it nicer you could patch python itself to
> read compressed .py files using gzopen/gzread or the likes. Or make your
> /usr/lib/python2.* a compressed read-only file which you mount in a
> loopback. Probably get the whole mess to <=1M that way.

anyone know if this has been researched ?
-mike

--
gentoo-embedded@gentoo.org mailing list

Re: [gentoo-embedded] Re: [gentoo-dev] Portage in Embedded Systems?

[stephane ancelot]

> > In one experiment I've managed to get pythons runtime down to about 2
> > Megs of HD space. And in another experiment with the portage tree itself
>
> How did you do that? I would be interessted very much because of an other
> project of mine.

May be you will be able to set a cloop filesystem ?

>
> Thanks
> Marius

--
gentoo-embedded@gentoo.org mailing list