[gentoo-user] How to find out to what file(...) writes goes on a idle system...

[gentoo-user] How to find out to what file(...) writes goes on a idle system...

[meino.cramer]

Hi,

on different systems I see the write stats (/proc/dikstats) to
physical existing disks steadily increasing. 
Looking at the output of lsof I cannot find any file suspicous
for receiving those writes.

In the context of preserving the live of flash media by minimizing
the count of unessary writes I want to know which
application/daemon/etc is continous writing to that media and which
"""entity""" (file/pipe/fifo...) is receiving those writes...

How can I find that information?

Thank you very much in advance for any help!
Best regards,
Meino

Re: [gentoo-user] How to find out to what file(...) writes goes on a idle system...

[Johannes Altmanninger]

[-- Attachment #1: Type: text/plain, Size: 619 bytes --]

Hi,

meino.cramer@gmx.de writes:

> In the context of preserving the live of flash media by minimizing
> the count of unessary writes I want to know which
> application/daemon/etc is continous writing to that media and which
> """entity""" (file/pipe/fifo...) is receiving those writes...

You could use this:

# echo 1 > /proc/sys/vm/block_dump

then every read and write operation on block devices shows up in dmesg
with the PID, process name and the block id. (This can be a lot of
lines, so dmesg -c might be useful) I'm not exactly sure how to identify
which files belong to which block, though.

Regards
Johannes

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 472 bytes --]

Re: [gentoo-user] How to find out to what file(...) writes goes on a idle system...

[Andrew Savchenko]

[-- Attachment #1: Type: text/plain, Size: 885 bytes --]

On Sat, 06 Dec 2014 12:01:16 +0100 Johannes Altmanninger wrote:
> Hi,
> 
> meino.cramer@gmx.de writes:
> 
> > In the context of preserving the live of flash media by minimizing
> > the count of unessary writes I want to know which
> > application/daemon/etc is continous writing to that media and which
> > """entity""" (file/pipe/fifo...) is receiving those writes...
> 
> You could use this:
> 
> # echo 1 > /proc/sys/vm/block_dump
> 
> then every read and write operation on block devices shows up in dmesg
> with the PID, process name and the block id. (This can be a lot of
> lines, so dmesg -c might be useful) I'm not exactly sure how to identify
> which files belong to which block, though.

This depends on filesystem being used. For ext* family debugfs may
be used:
# debugfs /dev/<your_dev>
ncheck inode1 inode2 ...

Best regards,
Andrew Savchenko

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-user] How to find out to what file(...) writes goes on a idle system...

[lee]

meino.cramer@gmx.de writes:

> Hi,
>
> on different systems I see the write stats (/proc/dikstats) to
> physical existing disks steadily increasing. 
> Looking at the output of lsof I cannot find any file suspicous
> for receiving those writes.
>
> In the context of preserving the live of flash media by minimizing
> the count of unessary writes I want to know which
> application/daemon/etc is continous writing to that media and which
> """entity""" (file/pipe/fifo...) is receiving those writes...
>
> How can I find that information?

iotop might tell you.

Since you don't see anything in lsof, I'd assume that the file (if it's
a file) is opened and closed rather than being kept open.  Or could it
be a swap partition which is used?


-- 
Again we must be afraid of speaking of daemons for fear that daemons
might swallow us.  Finally, this fear has become reasonable.

Re: [gentoo-user] How to find out to what file(...) writes goes on a idle system...

[Frank Steinmetzger]

[-- Attachment #1: Type: text/plain, Size: 661 bytes --]

On Sat, Dec 06, 2014 at 12:27:06PM +0100, lee wrote:
> meino.cramer@gmx.de writes:
> 
> > Hi,
> >
> > on different systems I see the write stats (/proc/dikstats) to
> > physical existing disks steadily increasing. 
> > Looking at the output of lsof I cannot find any file suspicous
> > for receiving those writes.
> > […]
> > How can I find that information?
>
> iotop might tell you.

I frequently use iotop -o. Another possibility might be ftop.
-- 
Gruß | Greetings | Qapla’
Please do not share anything from, with or about me on any social network.

“Your code is shit.. your argument is shit.” – Linus Torvalds, linux.kernel

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[gentoo-user] Re: How to find out to what file(...) writes goes on a idle system...

[James]

 <meino.cramer <at> gmx.de> writes:


> on different systems I see the write stats (/proc/dikstats) to
> physical existing disks steadily increasing. 
> Looking at the output of lsof I cannot find any file suspicous
> for receiving those writes.

Ok so in my experiences you need a (2) pronged approach.

(1) Then pursue quantifying with tools just what is causing the 
writes, strategies for minimization and monitoring as needed.

So folks are going down path (1) with you, that is fine.

(2) First minimize those write to your non-mechanical memory.

Path (2)
on any and all minimized gentoo or embedded gentoo systems,
I start out with USE="-*" to keep things minimum. Yea that tweaks the
devs now, but minimal system are just that, minimized, imho, so that
is a firm standard I always operation on. Set the minimum
number of global flags and the thinest  profile that will work for
your system. Every flag invokes more code and hence more processes,
more files, more writing to media.

Also, all log files should be written off the embeded system via
NFS or other similar mechanisms.

If you want further help, put up a document where folks can spend
$20 and get a similar board up and running embedded gentoo. Then 
they can see exactly what you see have and you can work as a team, or not,
your call.

I have dozens of tricks to minimize a gentoo system. But it is quite
a bit of work, just so you know. It's not a do this and it great. It
more like, try this, study the result and then alter the strategy.

hth,
James

Re: [gentoo-user] Re: How to find out to what file(...) writes goes on a idle system...

[meino.cramer]

James <wireless@tampabay.rr.com> [14-12-06 18:16]:
>  <meino.cramer <at> gmx.de> writes:
> 
> 
> > on different systems I see the write stats (/proc/dikstats) to
> > physical existing disks steadily increasing. 
> > Looking at the output of lsof I cannot find any file suspicous
> > for receiving those writes.
> 
> Ok so in my experiences you need a (2) pronged approach.
> 
> (1) Then pursue quantifying with tools just what is causing the 
> writes, strategies for minimization and monitoring as needed.
> 
> So folks are going down path (1) with you, that is fine.
> 
> (2) First minimize those write to your non-mechanical memory.
> 
> Path (2)
> on any and all minimized gentoo or embedded gentoo systems,
> I start out with USE="-*" to keep things minimum. Yea that tweaks the
> devs now, but minimal system are just that, minimized, imho, so that
> is a firm standard I always operation on. Set the minimum
> number of global flags and the thinest  profile that will work for
> your system. Every flag invokes more code and hence more processes,
> more files, more writing to media.
> 
> Also, all log files should be written off the embeded system via
> NFS or other similar mechanisms.
> 
> If you want further help, put up a document where folks can spend
> $20 and get a similar board up and running embedded gentoo. Then 
> they can see exactly what you see have and you can work as a team, or not,
> your call.
> 
> I have dozens of tricks to minimize a gentoo system. But it is quite
> a bit of work, just so you know. It's not a do this and it great. It
> more like, try this, study the result and then alter the strategy.
> 
> hth,
> James
>

Hi,

thank you very for all help I received regarding my question.

The system is already down to a limit. The by default running
processes are:


root         1     0  0 15:36 ?        00:00:00 init [3]  
root         2     0  0 15:36 ?        00:00:00 [kthreadd]
root         3     2  0 15:36 ?        00:00:01 [ksoftirqd/0]
root         5     2  0 15:36 ?        00:00:00 [kworker/0:0H]
root         7     2  0 15:36 ?        00:00:00 [khelper]
root         8     2  0 15:36 ?        00:00:00 [kdevtmpfs]
root       160     2  0 15:36 ?        00:00:00 [writeback]
root       162     2  0 15:36 ?        00:00:00 [crypto]
root       164     2  0 15:36 ?        00:00:00 [bioset]
root       166     2  0 15:36 ?        00:00:00 [kblockd]
root       168     2  0 15:36 ?        00:00:00 [cfg80211]
root       169     2  0 15:36 ?        00:00:00 [kworker/0:1]
root       280     2  0 15:36 ?        00:00:00 [kswapd0]
root       296     2  0 15:36 ?        00:00:00 [fsnotify_mark]
root       372     2  0 15:36 ?        00:00:00 [ipv6_addrconf]
root       398     2  0 15:36 ?        00:00:00 [deferwq]
root       406     2  0 15:36 ?        00:00:12 [mmcqd/0]
root       412     2  0 15:36 ?        00:00:00 [kworker/0:2]
root       415     2  0 15:36 ?        00:00:00 [jbd2/mmcblk0p2-]
root       416     2  0 15:36 ?        00:00:00 [ext4-rsv-conver]
root       563     1  0 15:37 ?        00:00:00 /lib/systemd/systemd-udevd --daemon
root       952     1  0 15:37 ?        00:00:00 /usr/sbin/ifplugd --iface=usb0
root      1380     1  0 15:37 ?        00:00:00 /usr/sbin/sshd
root      1399     1  0 15:37 tty1     00:00:00 /sbin/agetty 38400 tty1 linux
root      1400     1  0 15:37 tty2     00:00:00 /sbin/agetty 38400 tty2 linux
root      1401     1  0 15:37 tty3     00:00:00 /sbin/agetty 38400 tty3 linux
root      1402     1  0 15:37 tty4     00:00:00 /sbin/agetty 38400 tty4 linux
root      1403     1  0 15:37 tty5     00:00:00 /sbin/agetty 38400 tty5 linux
root      1404     1  0 15:37 tty6     00:00:00 /sbin/agetty 38400 tty6 linux
root      1405     1  0 15:37 ttyS0    00:00:00 /sbin/agetty -L 9600 ttyS0 vt100
root      1406  1380  0 15:37 ?        00:00:02 sshd: root@pts/0 
root      1412  1406  0 15:37 pts/0    00:00:00 screen -R -d
root      1414  1412  0 15:37 ?        00:00:01 SCREEN -R -d
root      1415  1414  0 15:37 pts/1    00:00:05 -/bin/zsh
root      1434     2  0 15:38 ?        00:00:00 [kworker/0:1H]
root      1866     2  0 15:43 ?        00:00:00 [kworker/u2:0]
root      8556     2  0 16:49 ?        00:00:00 [kworker/u2:2]

The count of getty processes may be decreaseable...but the rest is ok,
I think.

When I do a ftop I get no process, which have an open file handle for
writes...sometimes screen writes to utmp but thats it.

I suspect the swapfile I mounted as swapdevice for being guilty. 
I will deactivate that and we will see then.

When looking at /proc/diskstats: Will I see writes to FIFOs on the
disk as writes to the disk???
If YES...it would explain it...

Best regards,
Meino

[gentoo-user] Re: How to find out to what file(...) writes goes on a idle system...

[James]

 <meino.cramer <at> gmx.de> writes:


> > (1) Then pursue quantifying with tools just what is causing the 
> > writes, strategies for minimization and monitoring as needed.

> > So folks are going down path (1) with you, that is fine.

Prong (1) includes all issues related to systemd. Probably embedded
experience with systemd is rare, just guessing. Certainly I have none
of that experience. So post to those iotop responses and remind
folks you are using systemd on an embedded (gentoo) micro.

root  563  1  0 15:37 ?   00:00:00 /lib/systemd/systemd-udevd --daemon




> > (2) First minimize those write to your non-mechanical memory.

> > I have dozens of tricks to minimize a gentoo system. But it is quite
> > a bit of work, just so you know. It's not a do this and it great. It
> > more like, try this, study the result and then alter the strategy.
> > 
> > hth,
> > James
> >
> 
> Hi,
> 
> thank you very for all help I received regarding my question.
> 
> The system is already down to a limit. The by default running
> processes are:
> 
> root         1     0  0 15:36 ?        00:00:00 init [3]  
> root         2     0  0 15:36 ?        00:00:00 [kthreadd]
> root         3     2  0 15:36 ?        00:00:01 [ksoftirqd/0]
> root         5     2  0 15:36 ?        00:00:00 [kworker/0:0H]
> root         7     2  0 15:36 ?        00:00:00 [khelper]
> root         8     2  0 15:36 ?        00:00:00 [kdevtmpfs]
> root       160     2  0 15:36 ?        00:00:00 [writeback]
> root       162     2  0 15:36 ?        00:00:00 [crypto]
> root       164     2  0 15:36 ?        00:00:00 [bioset]
> root       166     2  0 15:36 ?        00:00:00 [kblockd]
> root       168     2  0 15:36 ?        00:00:00 [cfg80211]
> root       169     2  0 15:36 ?        00:00:00 [kworker/0:1]
> root       280     2  0 15:36 ?        00:00:00 [kswapd0]
> root       296     2  0 15:36 ?        00:00:00 [fsnotify_mark]
> root       372     2  0 15:36 ?        00:00:00 [ipv6_addrconf]
> root       398     2  0 15:36 ?        00:00:00 [deferwq]
> root       406     2  0 15:36 ?        00:00:12 [mmcqd/0]
> root       412     2  0 15:36 ?        00:00:00 [kworker/0:2]
> root       415     2  0 15:36 ?        00:00:00 [jbd2/mmcblk0p2-]
> root       416     2  0 15:36 ?        00:00:00 [ext4-rsv-conver]
> root       563     1  0 15:37 ?        00:00:00 /lib/systemd/systemd-udevd
--daemon
> root       952     1  0 15:37 ?        00:00:00 /usr/sbin/ifplugd --iface=usb0
> root      1380     1  0 15:37 ?        00:00:00 /usr/sbin/sshd
> root      1399     1  0 15:37 tty1     00:00:00 /sbin/agetty 38400 tty1 linux
> root      1400     1  0 15:37 tty2     00:00:00 /sbin/agetty 38400 tty2 linux
> root      1401     1  0 15:37 tty3     00:00:00 /sbin/agetty 38400 tty3 linux
> root      1402     1  0 15:37 tty4     00:00:00 /sbin/agetty 38400 tty4 linux
> root      1403     1  0 15:37 tty5     00:00:00 /sbin/agetty 38400 tty5 linux
> root      1404     1  0 15:37 tty6     00:00:00 /sbin/agetty 38400 tty6 linux
> root      1405     1  0 15:37 ttyS0    00:00:00 /sbin/agetty -L 9600 ttyS0
vt100
> root      1406  1380  0 15:37 ?        00:00:02 sshd: root <at> pts/0 
> root      1412  1406  0 15:37 pts/0    00:00:00 screen -R -d
> root      1414  1412  0 15:37 ?        00:00:01 SCREEN -R -d
> root      1415  1414  0 15:37 pts/1    00:00:05 -/bin/zsh
> root      1434     2  0 15:38 ?        00:00:00 [kworker/0:1H]
> root      1866     2  0 15:43 ?        00:00:00 [kworker/u2:0]
> root      8556     2  0 16:49 ?        00:00:00 [kworker/u2:2]

I'd research "kworker"

http://askubuntu.com/questions/33640/kworker-what-is-it-and-why-is-it-hogging-so-much-cpu

 
> The count of getty processes may be decreaseable...but the rest is ok,
> I think.

Those are static and just sitting incase you need a getty, so not a problem

> When I do a ftop I get no process, which have an open file handle for
> writes...sometimes screen writes to utmp but thats it.

With a traditional (non systemd) approach, init scripts just fire
up things at boot time and such. With systemd, I have no idea
what's going on. It's a curious situation and maybe systemd has
no issue in your excessive writes; pure speculation on my part.
But an embedded system just sitting idle should use very little
resource and sit quietly, in my experiences.


> I suspect the swapfile I mounted as swapdevice for being guilty. 
> I will deactivate that and we will see then.

good thing to examine.

> When looking at /proc/diskstats: Will I see writes to FIFOs on the
> disk as writes to the disk???
> If YES...it would explain it...


Also good to look at.

I usually use ext2 or one of the newer files systems, just for solid state
memory. Here is a good link to get your research your fs options.

http://free-electrons.com/blog/managing-flash-storage-with-linux/

Since you have (2) boards, have you considered installing the second one
differently (different file system, no systemd  etc etc to compare the
2 results?  If you can put different install-varients on different usb/sd/?
media, then you can just power down and change them out while comparing
resource utilization; just a thought, but something I do routinely.


Also, you need to get a custom/stripped kernel building methodology 
establish so you can try to reduce the kernel size by eliminating things
you do not use, testing scheduler options and a host of things.

Your entire toolchain needs to be documented and keep track of what 
you change and the results of which test you run, when and such.

Also,  when compiling a kernel for a minimize system I find that using
(CFLAGS="-Os""  yields faster and small executables) and is an excellent
way to reduce resource loading on a minimized or embedded system.

It's going to be a long journey, so keep meticulous records of what you
try, the results and your conclusions for the best path forward.


I'm out for a while.
hth,
James

Re: [gentoo-user] Re: How to find out to what file(...) writes goes on a idle system...

[meino.cramer]

Hi James,

...my board does not use systemd as far as I know...the
whole mimic is original gentoo stage3 stuff and Gentoo
defaults to openrc/udev and not systemd (or am I wrong?)

Cheers
Meino


James <wireless@tampabay.rr.com> [14-12-06 21:16]:
>  <meino.cramer <at> gmx.de> writes:
> 
> 
> > > (1) Then pursue quantifying with tools just what is causing the 
> > > writes, strategies for minimization and monitoring as needed.
> 
> > > So folks are going down path (1) with you, that is fine.
> 
> Prong (1) includes all issues related to systemd. Probably embedded
> experience with systemd is rare, just guessing. Certainly I have none
> of that experience. So post to those iotop responses and remind
> folks you are using systemd on an embedded (gentoo) micro.
> 
> root  563  1  0 15:37 ?   00:00:00 /lib/systemd/systemd-udevd --daemon
> 
> 
> 
> 
> > > (2) First minimize those write to your non-mechanical memory.
> 
> > > I have dozens of tricks to minimize a gentoo system. But it is quite
> > > a bit of work, just so you know. It's not a do this and it great. It
> > > more like, try this, study the result and then alter the strategy.
> > > 
> > > hth,
> > > James
> > >
> > 
> > Hi,
> > 
> > thank you very for all help I received regarding my question.
> > 
> > The system is already down to a limit. The by default running
> > processes are:
> > 
> > root         1     0  0 15:36 ?        00:00:00 init [3]  
> > root         2     0  0 15:36 ?        00:00:00 [kthreadd]
> > root         3     2  0 15:36 ?        00:00:01 [ksoftirqd/0]
> > root         5     2  0 15:36 ?        00:00:00 [kworker/0:0H]
> > root         7     2  0 15:36 ?        00:00:00 [khelper]
> > root         8     2  0 15:36 ?        00:00:00 [kdevtmpfs]
> > root       160     2  0 15:36 ?        00:00:00 [writeback]
> > root       162     2  0 15:36 ?        00:00:00 [crypto]
> > root       164     2  0 15:36 ?        00:00:00 [bioset]
> > root       166     2  0 15:36 ?        00:00:00 [kblockd]
> > root       168     2  0 15:36 ?        00:00:00 [cfg80211]
> > root       169     2  0 15:36 ?        00:00:00 [kworker/0:1]
> > root       280     2  0 15:36 ?        00:00:00 [kswapd0]
> > root       296     2  0 15:36 ?        00:00:00 [fsnotify_mark]
> > root       372     2  0 15:36 ?        00:00:00 [ipv6_addrconf]
> > root       398     2  0 15:36 ?        00:00:00 [deferwq]
> > root       406     2  0 15:36 ?        00:00:12 [mmcqd/0]
> > root       412     2  0 15:36 ?        00:00:00 [kworker/0:2]
> > root       415     2  0 15:36 ?        00:00:00 [jbd2/mmcblk0p2-]
> > root       416     2  0 15:36 ?        00:00:00 [ext4-rsv-conver]
> > root       563     1  0 15:37 ?        00:00:00 /lib/systemd/systemd-udevd
> --daemon
> > root       952     1  0 15:37 ?        00:00:00 /usr/sbin/ifplugd --iface=usb0
> > root      1380     1  0 15:37 ?        00:00:00 /usr/sbin/sshd
> > root      1399     1  0 15:37 tty1     00:00:00 /sbin/agetty 38400 tty1 linux
> > root      1400     1  0 15:37 tty2     00:00:00 /sbin/agetty 38400 tty2 linux
> > root      1401     1  0 15:37 tty3     00:00:00 /sbin/agetty 38400 tty3 linux
> > root      1402     1  0 15:37 tty4     00:00:00 /sbin/agetty 38400 tty4 linux
> > root      1403     1  0 15:37 tty5     00:00:00 /sbin/agetty 38400 tty5 linux
> > root      1404     1  0 15:37 tty6     00:00:00 /sbin/agetty 38400 tty6 linux
> > root      1405     1  0 15:37 ttyS0    00:00:00 /sbin/agetty -L 9600 ttyS0
> vt100
> > root      1406  1380  0 15:37 ?        00:00:02 sshd: root <at> pts/0 
> > root      1412  1406  0 15:37 pts/0    00:00:00 screen -R -d
> > root      1414  1412  0 15:37 ?        00:00:01 SCREEN -R -d
> > root      1415  1414  0 15:37 pts/1    00:00:05 -/bin/zsh
> > root      1434     2  0 15:38 ?        00:00:00 [kworker/0:1H]
> > root      1866     2  0 15:43 ?        00:00:00 [kworker/u2:0]
> > root      8556     2  0 16:49 ?        00:00:00 [kworker/u2:2]
> 
> I'd research "kworker"
> 
> http://askubuntu.com/questions/33640/kworker-what-is-it-and-why-is-it-hogging-so-much-cpu
> 
>  
> > The count of getty processes may be decreaseable...but the rest is ok,
> > I think.
> 
> Those are static and just sitting incase you need a getty, so not a problem
> 
> > When I do a ftop I get no process, which have an open file handle for
> > writes...sometimes screen writes to utmp but thats it.
> 
> With a traditional (non systemd) approach, init scripts just fire
> up things at boot time and such. With systemd, I have no idea
> what's going on. It's a curious situation and maybe systemd has
> no issue in your excessive writes; pure speculation on my part.
> But an embedded system just sitting idle should use very little
> resource and sit quietly, in my experiences.
> 
> 
> > I suspect the swapfile I mounted as swapdevice for being guilty. 
> > I will deactivate that and we will see then.
> 
> good thing to examine.
> 
> > When looking at /proc/diskstats: Will I see writes to FIFOs on the
> > disk as writes to the disk???
> > If YES...it would explain it...
> 
> 
> Also good to look at.
> 
> I usually use ext2 or one of the newer files systems, just for solid state
> memory. Here is a good link to get your research your fs options.
> 
> http://free-electrons.com/blog/managing-flash-storage-with-linux/
> 
> Since you have (2) boards, have you considered installing the second one
> differently (different file system, no systemd  etc etc to compare the
> 2 results?  If you can put different install-varients on different usb/sd/?
> media, then you can just power down and change them out while comparing
> resource utilization; just a thought, but something I do routinely.
> 
> 
> Also, you need to get a custom/stripped kernel building methodology 
> establish so you can try to reduce the kernel size by eliminating things
> you do not use, testing scheduler options and a host of things.
> 
> Your entire toolchain needs to be documented and keep track of what 
> you change and the results of which test you run, when and such.
> 
> Also,  when compiling a kernel for a minimize system I find that using
> (CFLAGS="-Os""  yields faster and small executables) and is an excellent
> way to reduce resource loading on a minimized or embedded system.
> 
> It's going to be a long journey, so keep meticulous records of what you
> try, the results and your conclusions for the best path forward.
> 
> 
> I'm out for a while.
> hth,
> James
> 
> 
> 
> 
> 
> 
> 
> 
> 

Re: [gentoo-user] Re: How to find out to what file(...) writes goes on a idle system...

[Tom H]

On Sat, Dec 6, 2014 at 11:05 PM, <meino.cramer@gmx.de> wrote:
> James <wireless@tampabay.rr.com> [14-12-06 21:16]:
>>
>> Prong (1) includes all issues related to systemd. Probably embedded
>> experience with systemd is rare, just guessing. Certainly I have none
>> of that experience. So post to those iotop responses and remind
>> folks you are using systemd on an embedded (gentoo) micro.
>>
>> root  563  1  0 15:37 ?   00:00:00 /lib/systemd/systemd-udevd --daemon
>
> ...my board does not use systemd as far as I know...the
> whole mimic is original gentoo stage3 stuff and Gentoo
> defaults to openrc/udev and not systemd (or am I wrong?)

Please bottom-post.

udevd has been installed as systemd-udevd since 208 or 210.