[gentoo-user] jffs2 on gentoo

[gentoo-user] jffs2 on gentoo

[James]

Hello,

I have a firewall that is built pretty minimally on a P3 and
and old 4 gig ide disk:

/dev/hda3              2068348   1668104    400244  81% /
/dev/hda1               100728     40452     60276  41% /boot

I have a 4 gig Cf card (sandisk) and a ide-cf card that should
make the CF card look like an ide hard drive.

I've been searching for a wiki or something that describes the general
sequence of events to migrate the existing gentoo system to the
CF/ide disk, with no luck.

I did find this page:
http://gentoo-wiki.com/Mounting_a_block_device_with_JFFS2


But it seems vague(outdated) and missing many steps. Or
am I confused? I'm just looking for some outline or verbose
steps to replace an ide drive on a system with a CF/ide drive
and jffs2, as I have many systems that I'd like to do this with, 
for core reliability on minimalistic gentoo servers. I plan
on having additional space on these systems (when needed)
via NFS.


Any help or ideas are greatly appreciated.


James

-- 
gentoo-user@lists.gentoo.org mailing list

Re: [gentoo-user] jffs2 on gentoo

[Dan Farrell]

On Fri, 14 Mar 2008 18:27:22 +0000 (UTC)
James <wireless@tampabay.rr.com> wrote:

> Hello,
> 
> I have a firewall that is built pretty minimally on a P3 and
> and old 4 gig ide disk:
> 
> /dev/hda3              2068348   1668104    400244  81% /
> /dev/hda1               100728     40452     60276  41% /boot
> 
> I have a 4 gig Cf card (sandisk) and a ide-cf card that should
> make the CF card look like an ide hard drive.
> 
> I've been searching for a wiki or something that describes the general
> sequence of events to migrate the existing gentoo system to the
> CF/ide disk, with no luck.
> 
> I did find this page:
> http://gentoo-wiki.com/Mounting_a_block_device_with_JFFS2
> 
> 
> But it seems vague(outdated) and missing many steps. Or
> am I confused? I'm just looking for some outline or verbose
> steps to replace an ide drive on a system with a CF/ide drive
> and jffs2, as I have many systems that I'd like to do this with, 
> for core reliability on minimalistic gentoo servers. I plan
> on having additional space on these systems (when needed)
> via NFS.
> 
> 
> Any help or ideas are greatly appreciated.
> 
> 
> James
> 

You shouldn't need to do anything special - just copy all files over
exactly, and then set up GRUB on the CF card.  

Your biggest problem is you need to cut 75% from your installation.
Delete portage, logs, and so on to do so.  You can mount portage from
NFS later.  You might also have to uninstall X and all desktop
environments to get it that small -- assuming you can do so, it's a
good idea.  
-- 
gentoo-user@lists.gentoo.org mailing list

[gentoo-user] Re: jffs2 on gentoo

[James]

Dan Farrell <dan <at> spore.ath.cx> writes:


> > I have a 4 gig Cf card (sandisk) and a ide-cf card that should
> > make the CF card look like an ide hard drive.

> > http://gentoo-wiki.com/Mounting_a_block_device_with_JFFS2

> > But it seems vague(outdated) and missing many steps. Or
> > am I confused? I'm just looking for some outline or verbose
> > steps to replace an ide drive on a system with a CF/ide drive
> > and jffs2, as I have many systems that I'd like to do this with, 
> > for core reliability on minimalistic gentoo servers. I plan
> > on having additional space on these systems (when needed)
> > via NFS.


> You shouldn't need to do anything special - just copy all files over
> exactly, and then set up GRUB on the CF card.  

Well, I've heard otherwise. Use jffs2 or the CF card will wear out 
prematurely.......

I should have been more specific on my questions. When using 'fdisk'
to format the CF card: In section 4d of the handbook, I do not see
a 'mkjffs2fs' command. So I'm confused as to how you put the jffs2
file system on a partition.




-- 
gentoo-user@lists.gentoo.org mailing list

Re: [gentoo-user] Re: jffs2 on gentoo

[Stroller]

On 15 Mar 2008, at 20:08, James wrote:
>> You shouldn't need to do anything special - just copy all files over
>> exactly, and then set up GRUB on the CF card.
>
> Well, I've heard otherwise. Use jffs2 or the CF card will wear out
> prematurely.......

I've "heard" lots about using flashdrives for filesystems, but I've  
never read on a mailing list anything actually definitive on the  
subject. I find many posts to be confused.

This comes up regularly on the mythtv-users list <http://tinyurl.com/ 
2z7s7h>, and conclusions are vague.

_As I understand it_, previous posters are right though - either use  
a device with built in wear-levelling (and a file-system it's aware  
of), OR use a wear-levelling filesystem. It's not clear what happens  
if you use a wear-levelling filesystem on a device with built in wear- 
levelling.

Anecdotally, "it won't work, your flash drive will die in no  
time" (repeated often) and "we used these all the time in my last job  
and never had one die on us" (IIRC the latter guy worked on till  
systems, basically a PC with a touchscreen, and used SanDisk cards;  
it's unclear how much writing was done to the drive).

Stroller's law: ask about using compact-flash camera memory in Linux  
PC applications, and you will receive many anecdotal & hearsay replies.
Corollary: just try it. No-one else frikkin' is!

> ... When using 'fdisk'
> to format the CF card: In section 4d of the handbook, I do not see
> a 'mkjffs2fs' command.

$ eix -S jffs
* sys-fs/mtd
      Available versions:  20040825 20050519
      Homepage:            http://sources.redhat.com/jffs2/
      Description:         JFFS2 is a log-structured file system  
designed for use on flash devices in embedded systems.
$

if this doesn't provide the userspace tools, then the homepage prolly  
does.

Stroller.
-- 
gentoo-user@lists.gentoo.org mailing list

Re: [gentoo-user] Re: jffs2 on gentoo

[Jan Seeger]

[-- Attachment #1: Type: text/plain, Size: 1015 bytes --]

On Sun, 16. Mar, Stroller spammed my inbox with 
>>
>> Well, I've heard otherwise. Use jffs2 or the CF card will wear out
>> prematurely.......
>
> I've "heard" lots about using flashdrives for filesystems, but I've never 
> read on a mailing list anything actually definitive on the subject. I find 
> many posts to be confused.

Yeah, it's the same here. I read an article in the german computer magazine c't,
and they said that they have tried to break USB sticks with repeated writes, but
have never succeeded (I think they ran 10000 writes, but I could be wrong).

So why not just buy a cheap USB stick for 10 € (or whatever), mount it sync and
write a little script which writes to a file, deletes it and begins again. Have
it record the number of times the file was written and check the consistency
after every write (md5sum perhaps?). Leave it running for a (long?) time and
then you will probably encounter errors (if there are...).

Actually, this sounds interesting^^
Regards,
Jan

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-user] Re: jffs2 on gentoo

[Alan McKinnon]

On Sunday 16 March 2008, Jan Seeger wrote:
> Yeah, it's the same here. I read an article in the german computer
> magazine c't, and they said that they have tried to break USB sticks
> with repeated writes, but have never succeeded (I think they ran
> 10000 writes, but I could be wrong).

That test is probably insufficient. Somebody actually did this test on 
lkml some time ago, and found that the better devices were rated to 
100,000 writes to the same cell and the el-cheapo jobs were somewhere 
around 10,000.

IOW, the manufacturer of a cheapie says it should cope with 10,000 
writes *at least*, so in practice you could expect more.

I'd start to believe a test that does >1,000,000 writes to the same cell 
before drawing any conclusions.

-- 
Alan McKinnon
alan dot mckinnon at gmail dot com

-- 
gentoo-user@lists.gentoo.org mailing list

Re: [gentoo-user] Re: jffs2 on gentoo

[Jan Seeger]

[-- Attachment #1: Type: text/plain, Size: 1578 bytes --]

As a followup, I have actually written said script (in perl), and would welcome
any improvement comments. File size of the test file shouldn't matter, since
without wear leveling, the same cells should get written over and over again.

Only thing I need to do now is run it for a long time... Unfortunately, I need
linux for that since we need to mount the drive sync. And I have no live CD in
the house... Ts. What have I come to?

Anyway, here's the script: 


use strict;
use warnings; 

use Digest::MD5;
use Getopt::Std;
use File::Basename;
use File::Spec;
use File::Copy;

my %opts;
getopts("d:i:",\%opts);

if (! $opts{d} || ! $opts{i}) {
	die <<"EOF"
-d: Mountpoint of drive to be tested. Should be mounted with "sync" mount option.
-i: Input file. Will be copied to mountpoint to test integrity.

Leave running for a long time to "test" your USB stick.
EOF
}

my $counter = 0;
my $originaldigest;
my $outfilename = File::Spec->catfile($opts{d},(fileparse($opts{i}))[0]);
my $digester = Digest::MD5->new();

open my $handle,"<$opts{i}";
binmode($handle);

$digester->addfile($handle);
close($handle);

$originaldigest = $digester->digest();

while (1) {
	print "Running test $counter.\n";
	copy($opts{i},$outfilename);
	open my $outhandle,"<$outfilename";
	binmode($outhandle);
	$digester->addfile($outhandle);
	if ($digester->digest() ne $originaldigest) {
		die "Failed write at read $counter.\n";
	}
	close($outhandle);
	unlink($outfilename);
	$counter++;
}


-- 
thenybble.de/blog/ -- four bits at a time

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-user] Re: jffs2 on gentoo

[W.Kenworthy]

I have been following this thread intermittantly and have not seen a
comment on the following:

I believe that writing a file to a single location is not the way to do
this: you need to write a byte to the usb key in the same location, but
need to ensure it continually changes: perhaps rotating 1's/0's.
Alternatively, the concern is that the FAT/inode table or the like is
where the most wear will occur - perhaps concentrate there? (i.e., do a
journelled FS like reiserfs with a fast update?

Do any USB keys do some kind of write minimisation in the controller? -
no change in the data/no write? - seems a logical way to extend the
life?

BillK


On Sun, 2008-03-16 at 19:03 +0100, Jan Seeger wrote:
> As a followup, I have actually written said script (in perl), and would welcome
-- 
gentoo-user@lists.gentoo.org mailing list

Re: [gentoo-user] Re: jffs2 on gentoo

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 972 bytes --]

On Mon, 17 Mar 2008 06:43:28 +0900, W.Kenworthy wrote:

> I believe that writing a file to a single location is not the way to do
> this: you need to write a byte to the usb key in the same location, but
> need to ensure it continually changes: perhaps rotating 1's/0's.
> Alternatively, the concern is that the FAT/inode table or the like is
> where the most wear will occur - perhaps concentrate there? (i.e., do a
> journelled FS like reiserfs with a fast update?

It used to be that writing a large file to a USB key mounted with the
sync option would update the FAT for each block written, so writing a
large file several times would soon kill it. I destroyed a 1GB key like
this by continually writing modified KNOPPIX images to it. That was a
couple of years ago, I've no idea if the kernel still writes FAT like
this because I've mounted flash devices with nosync ever since.


-- 
Neil Bothwick

Madness takes its toll. Exact change, please.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-user] Re: jffs2 on gentoo

[Jan Seeger]

[-- Attachment #1: Type: text/plain, Size: 923 bytes --]

On Sun, 16. Mar, W.Kenworthy spammed my inbox with 
> I believe that writing a file to a single location is not the way to do
> this: you need to write a byte to the usb key in the same location, but
> need to ensure it continually changes: perhaps rotating 1's/0's.
> Alternatively, the concern is that the FAT/inode table or the like is
> where the most wear will occur - perhaps concentrate there?

Yeah, if I have the stick mounted sync and always copy and delete a file, some
bytes should get flipped around regularly. *If* there is no internal wear
leveling, that is. On USB sticks with internal wear leveling, you will, from a
size of about 1 GB upwards, never (Well, perhaps after 10 years...)  see a
failure due to media wear.

For the record: My USB stick has now gone through 78560 read/write cycles and is
still happily copying.

Regards,
Jan

-- 
thenybble.de/blog/ -- four bits at a time

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-user] jffs2 on gentoo

[Florian Philipp]

[-- Attachment #1: Type: text/plain, Size: 1575 bytes --]


On Fri, 2008-03-14 at 18:27 +0000, James wrote:
> Hello,
> 
> I have a firewall that is built pretty minimally on a P3 and
> and old 4 gig ide disk:
> 
> /dev/hda3              2068348   1668104    400244  81% /
> /dev/hda1               100728     40452     60276  41% /boot
> 
> I have a 4 gig Cf card (sandisk) and a ide-cf card that should
> make the CF card look like an ide hard drive.
> 
> I've been searching for a wiki or something that describes the general
> sequence of events to migrate the existing gentoo system to the
> CF/ide disk, with no luck.
> 
> I did find this page:
> http://gentoo-wiki.com/Mounting_a_block_device_with_JFFS2
> 
> 
> But it seems vague(outdated) and missing many steps. Or
> am I confused? I'm just looking for some outline or verbose
> steps to replace an ide drive on a system with a CF/ide drive
> and jffs2, as I have many systems that I'd like to do this with, 
> for core reliability on minimalistic gentoo servers. I plan
> on having additional space on these systems (when needed)
> via NFS.
> 
> 
> Any help or ideas are greatly appreciated.
> 
> 
> James
> 

As far as I know you won't need jffs2 (or any other fs for flash
memory). It is meant to be used on embedded devices that directly access
the flash memory. In your case, the CF-disk takes care of wear leveling.
Just use ext2. However, you could still get problems because, as far as
I know, wear leveling needs to be tuned for the FS and most probably no
one tuned the CF-disk for ext2. Maybe you could use fat instead...

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-user] jffs2 on gentoo

[dexters84]

[-- Attachment #1: Type: text/plain, Size: 2108 bytes --]

Florian Philipp pisze:
> On Fri, 2008-03-14 at 18:27 +0000, James wrote:
>   
>> Hello,
>>
>> I have a firewall that is built pretty minimally on a P3 and
>> and old 4 gig ide disk:
>>
>> /dev/hda3              2068348   1668104    400244  81% /
>> /dev/hda1               100728     40452     60276  41% /boot
>>
>> I have a 4 gig Cf card (sandisk) and a ide-cf card that should
>> make the CF card look like an ide hard drive.
>>
>> I've been searching for a wiki or something that describes the general
>> sequence of events to migrate the existing gentoo system to the
>> CF/ide disk, with no luck.
>>
>> I did find this page:
>> http://gentoo-wiki.com/Mounting_a_block_device_with_JFFS2
>>
>>
>> But it seems vague(outdated) and missing many steps. Or
>> am I confused? I'm just looking for some outline or verbose
>> steps to replace an ide drive on a system with a CF/ide drive
>> and jffs2, as I have many systems that I'd like to do this with, 
>> for core reliability on minimalistic gentoo servers. I plan
>> on having additional space on these systems (when needed)
>> via NFS.
>>
>>
>> Any help or ideas are greatly appreciated.
>>
>>
>> James
>>
>>     
>
> As far as I know you won't need jffs2 (or any other fs for flash
> memory). It is meant to be used on embedded devices that directly access
> the flash memory. In your case, the CF-disk takes care of wear leveling.
> Just use ext2. However, you could still get problems because, as far as
> I know, wear leveling needs to be tuned for the FS and most probably no
> one tuned the CF-disk for ext2. Maybe you could use fat instead...
>   
I did this sort of system a while ago. I've used 1GB card with gentoo 
and cf-ide adapter. There are some tricky parts that nobody mentions.
One of them is that I wasn't able to boot from my 1GB "hard drive" when 
it was connected via 80 pin ide cable, I've dig up some old 40 pin ata 
cable and it worked.
Other things you have to remeber concern file system usage, you musn't 
create swap partition, disable local syslog, log rotation, turn 
everything except desired daemons etc.

regards
dexter

[-- Attachment #2: Type: text/html, Size: 2570 bytes --]

[gentoo-user] Re: jffs2 on gentoo

[James]

dexters84 <dexters84 <at> gmail.com> writes:




 I did this sort of system a while ago. I've used 1GB card with gentoo
 and cf-ide adapter. There are some tricky parts that nobody mentions.
 One of them is that I wasn't able to boot from my 1GB "hard drive" when
 it was connected via 80 pin ide cable, I've dig up some old 40 pin ata
 cable and it worked.

  Other things you have to remeber concern file system usage, you musn't
  create swap partition, disable local syslog, log rotation, turn
  everything except desired daemons etc.



Where did you get the idea not to use swap?

What file system did you use?


James




-- 
gentoo-user@lists.gentoo.org mailing list

Re: [gentoo-user] Re: jffs2 on gentoo

[Stroller]

On 15 Mar 2008, at 20:17, James wrote:
> dexters84 <dexters84 <at> gmail.com> writes:
>
>>   Other things you have to remeber concern file system usage, you  
>> musn't
>>   create swap partition, disable local syslog, log rotation, turn
>>   everything except desired daemons etc.
>
>
> Where did you get the idea not to use swap?

Too many writes.
(dexters writes "disable local syslog, log rotation" for the same  
reason)

I believe the size of the writes can be relevant as well.
In any case, swapfs was not designed for flash memory
(see also Windows Vista's ReadyBoost <http://en.wikipedia.org/wiki/ 
ReadyBoost>).

Stroller.
-- 
gentoo-user@lists.gentoo.org mailing list

Re: [gentoo-user] Re: jffs2 on gentoo

[dexters84]

Stroller pisze:
>
> On 15 Mar 2008, at 20:17, James wrote:
>> dexters84 <dexters84 <at> gmail.com> writes:
>>
>>>   Other things you have to remeber concern file system usage, you 
>>> musn't
>>>   create swap partition, disable local syslog, log rotation, turn
>>>   everything except desired daemons etc.
>>
>>
>> Where did you get the idea not to use swap?
>
> Too many writes.
> (dexters writes "disable local syslog, log rotation" for the same reason)
>
> I believe the size of the writes can be relevant as well.
> In any case, swapfs was not designed for flash memory
> (see also Windows Vista's ReadyBoost 
> <http://en.wikipedia.org/wiki/ReadyBoost>).
>
> Stroller.
That was exactly my point. Systems based on cf card as hard drive are 
usually small - one function focused devices, hence there is no need for 
swap partition. To extend lifetime of cf card you have to minimize all 
possible read/writes to card.
-- 
gentoo-user@lists.gentoo.org mailing list

[gentoo-user] Re: jffs2 on gentoo

[James]

dexters84 <dexters84 <at> gmail.com> writes:


> > I believe the size of the writes can be relevant as well.

> > Stroller.

> That was exactly my point. Systems based on cf card as hard drive are 
> usually small - one function focused devices, hence there is no need for 
> swap partition. To extend lifetime of cf card you have to minimize all 
> possible read/writes to card.


OK,


I agree.
minimize the size of the system and 
minimize writes to extend the
life of the CF.

Wear leveling is *probably* built into the IDE to CF converter
carrier board?

So no need to work about which File System to use or tuning the
file system (EXT2).

Is that it?


James



-- 
gentoo-user@lists.gentoo.org mailing list

Re: [gentoo-user] Re: jffs2 on gentoo

[Stroller]

On 17 Mar 2008, at 18:10, James wrote:
> ...
> Wear leveling is *probably* built into the IDE to CF converter
> carrier board?

Almost certainly not, I'd have thought. Aren't those boards just dumb  
pin-convertors? CF cards "talk" IDE.

Stroller.
-- 
gentoo-user@lists.gentoo.org mailing list

Re: [gentoo-user] Re: jffs2 on gentoo

[Florian Philipp]

[-- Attachment #1: Type: text/plain, Size: 650 bytes --]


On Tue, 2008-03-18 at 01:47 +0000, Stroller wrote:
> On 17 Mar 2008, at 18:10, James wrote:
> > ...
> > Wear leveling is *probably* built into the IDE to CF converter
> > carrier board?
> 
> Almost certainly not, I'd have thought. Aren't those boards just dumb  
> pin-convertors? CF cards "talk" IDE.
> 
> Stroller.

Yes  they are.

Another thought crossed my mind today: Does wear leveling work if I
create loopback devices (ext2-formatted) on FAT32?

By the way: Why is wear leveling filesystem-dependent anyway? I would
have thought it were working on blocks (like device mapper, cryptsetup,
lvm and so on) and not on files.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-user] Re: jffs2 on gentoo

[Stroller]

On 18 Mar 2008, at 10:33, Florian Philipp wrote:
> On Tue, 2008-03-18 at 01:47 +0000, Stroller wrote:
>> On 17 Mar 2008, at 18:10, James wrote:
>>> ...
>>> Wear leveling is *probably* built into the IDE to CF converter
>>> carrier board?
>>
>> Almost certainly not, I'd have thought. Aren't those boards just dumb
>> pin-convertors? CF cards "talk" IDE.
>
> Yes  they are.
>
> Another thought crossed my mind today: Does wear leveling work if I
> create loopback devices (ext2-formatted) on FAT32?

Surely so. In this case you would be writing to the flash device's  
FAT32 filessystem. It doesn't matter if you're writing a .RAW picture  
file, an .iso or your loopback fs.

> By the way: Why is wear leveling filesystem-dependent anyway?

No idea. Please note that in this thread I have stated that I  
_understand_ wear-levelling to be filesystem-dependent - it is others  
who have made replies stating this more confidently.

> I would
> have thought it were working on blocks (like device mapper,  
> cryptsetup,
> lvm and so on) and not on files.

Ah! But here we come back to the problem of recording how many times  
a given block has been written upon, in order not to kill that block.  
Most filesystems don't have to do that.

Stroller.

-- 
gentoo-user@lists.gentoo.org mailing list

Re: [gentoo-user] Re: jffs2 on gentoo

[dexters84]

Hi

In my system I didn't bother with any of embedded file systems - I've 
created 1 GB ext2 partition (journalising in ext3 increases read/write 
count), and it worked just like any other hard drive. Bios detected 
correct capacity - I was lucky with that, but in case where BIOS doesn't 
detect CF card properly google is Your friend.
I don't have all doc I've used during setup but I remember reading this one
http://silent.gumph.org/content/4/1/011-linux-on-cf.html

regards

Stroller pisze:
>
> On 18 Mar 2008, at 10:33, Florian Philipp wrote:
>> On Tue, 2008-03-18 at 01:47 +0000, Stroller wrote:
>>> On 17 Mar 2008, at 18:10, James wrote:
>>>> ...
>>>> Wear leveling is *probably* built into the IDE to CF converter
>>>> carrier board?
>>>
>>> Almost certainly not, I'd have thought. Aren't those boards just dumb
>>> pin-convertors? CF cards "talk" IDE.
>>
>> Yes  they are.
>>
>> Another thought crossed my mind today: Does wear leveling work if I
>> create loopback devices (ext2-formatted) on FAT32?
>
> Surely so. In this case you would be writing to the flash device's 
> FAT32 filessystem. It doesn't matter if you're writing a .RAW picture 
> file, an .iso or your loopback fs.
>
>> By the way: Why is wear leveling filesystem-dependent anyway?
>
> No idea. Please note that in this thread I have stated that I 
> _understand_ wear-levelling to be filesystem-dependent - it is others 
> who have made replies stating this more confidently.
>
>> I would
>> have thought it were working on blocks (like device mapper, cryptsetup,
>> lvm and so on) and not on files.
>
> Ah! But here we come back to the problem of recording how many times a 
> given block has been written upon, in order not to kill that block. 
> Most filesystems don't have to do that.
>
> Stroller.
>

-- 
gentoo-user@lists.gentoo.org mailing list