From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) by finch.gentoo.org (Postfix) with ESMTP id 99ACA1381F4 for ; Tue, 14 Aug 2012 15:12:15 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id EE40721C053; Tue, 14 Aug 2012 15:11:54 +0000 (UTC) Received: from out5-smtp.messagingengine.com (out5-smtp.messagingengine.com [66.111.4.29]) by pigeon.gentoo.org (Postfix) with ESMTP id F25A021C02A for ; Tue, 14 Aug 2012 15:09:54 +0000 (UTC) Received: from compute3.internal (compute3.nyi.mail.srv.osa [10.202.2.43]) by gateway1.nyi.mail.srv.osa (Postfix) with ESMTP id AC96320C39 for ; Tue, 14 Aug 2012 11:09:54 -0400 (EDT) Received: from frontend1.nyi.mail.srv.osa ([10.202.2.160]) by compute3.internal (MEProxy); Tue, 14 Aug 2012 11:09:54 -0400 DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=binarywings.net; h=message-id:date:from:mime-version:to:subject:references :in-reply-to:content-type; s=mesmtp; bh=8x4Krj3Ttea3ZIb0OIOO3qQs hhY=; b=jAsHTwAnab9MFEsyJT9QQnDXUUhPdHcympsz0jJXGUQG2MLdguyGwesD kSVOzQYpr8KAnlKTp8gd3yd4/+XARus+bEfd4rjWQkZBKfdU3Tr5+9lSu0rbXzF0 t9zCHbgFWZ2M18qAfOaeDJen5QKKAMc0+vx3g+CH/ZeFYi+eswI= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d= messagingengine.com; h=message-id:date:from:mime-version:to :subject:references:in-reply-to:content-type; s=smtpout; bh=8x4K rj3Ttea3ZIb0OIOO3qQshhY=; b=Kikv54RlcV2KymZbhr9Q4a7pEsNmZX0kG6mO LjnXedwPWffC7J8mrtjUFfO8SQRniwYfQHRmmGPqIMTaEhK0R5ZtfR4V9NTF8zxw ZEtesRNe5QCB+QewhV7OcPM0JvlS//HylbXmbrDSISnhm/1W18Y/idlQk8bgEfYl fupM5Ds= X-Sasl-enc: nHpy5u0o3f7Rxkuaj6rW83ZJXI87gw4bY+5JyNqdTn4A 1344956993 Received: from [10.197.236.44] (unknown [46.115.35.236]) by mail.messagingengine.com (Postfix) with ESMTPA id 1FA218E01FC for ; Tue, 14 Aug 2012 11:09:52 -0400 (EDT) Message-ID: <502A6A39.4000800@binarywings.net> Date: Tue, 14 Aug 2012 17:09:45 +0200 From: Florian Philipp User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0.6esrpre) Gecko/20120804 Thunderbird/10.0.6 Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-user@lists.gentoo.org Reply-to: gentoo-user@lists.gentoo.org MIME-Version: 1.0 To: gentoo-user@lists.gentoo.org Subject: Re: [gentoo-user] Fast file system for cache directory with lot's of files References: <5029114E.8070402@admin-box.com> <502A0AA2.9030607@admin-box.com> <20120814104631.00228994@hactar.digimed.co.uk> <502A5889.8070506@admin-box.com> In-Reply-To: <502A5889.8070506@admin-box.com> X-Enigmail-Version: 1.3.5 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enigBB3303D4A9302FF3F5D16ED9" X-Archives-Salt: 304d958a-0df0-43fe-bb3c-c55aa1de2d8f X-Archives-Hash: e1573b93a0ace6e13046f5ed210bfc0c This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigBB3303D4A9302FF3F5D16ED9 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Am 14.08.2012 15:54, schrieb Daniel Troeder: > On 14.08.2012 11:46, Neil Bothwick wrote: >> On Tue, 14 Aug 2012 10:21:54 +0200, Daniel Troeder wrote: >> >>> There is also the possibility to write a really small daemon (less th= an >>> 50 lines of C) that registers with inotify for the entire fs and >>> journals the file activity to a sqlite-db. >> >> sys-process/incron ? > Uh... didn't know that one! ... very interesting :) >=20 > Have you used it? > How does it perform if there are lots of modifications going on? > Does it have a throttle against fork bombing? > must-read-myself-a-little..... >=20 > A incron line > # sqlite3 /file.sql 'INSERT filename, date INTO table' > would be inefficient, because it spawn lots of processes, but it would > be very nice to simply test out the idea. Then a > # sqlite3 /file.sql 'SELECT filename FROM table SORTBY date < date-30da= ys' > or something to get the files older than 30 days, and voil=E1 :) >=20 >=20 Maybe inotifywait is better for this kind of batch job. Collecting events: inotifywait -rm -e CREATE,DELETE --timefmt '%s' --format \ "$(printf '%%T\t%%e\t%%w%%f')" /tmp > events.tbl # the printf is there because inotifywait's format does not # recognize common escapes like \t # Output format: # Seconds since epoch \t CREATE/DELETE \t file name \n Filtering events: sort --stable -k3 events.tbl | awk ' function update() { line=3D$0; exists=3D $2=3D=3D"DELETE" ? 0 : 1; file=3D$3 } NR=3D=3D1{ update(); next } { if($3!=3Dfile && exists=3D=3D1){ print line } update() }' # Sorts by file name while preserving temporal order. # Uses awk to suppress output of files that have been deleted. # Output: Last CREATE event for each existing file Retrieving files created 30+ days ago: awk -v newest=3D$(date -d -5seconds +%s) ' $1>newest{ nextfile } { print $3 }' Remarks: The awk scripts need some improvement if you have to handle whitespaces in filenames but with the input format, it should be able to work with everything except newlines. Inotifywait itself is utterly useless when dealing with newlines in file names unless you want to put some serious effort into sanitizing the outp= ut. Regards, Florian Philipp --------------enigBB3303D4A9302FF3F5D16ED9 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAlAqaj0ACgkQqs4uOUlOuU9jSQCfY1YdvE9SyhE0EBKMx+KdAOep 8hUAnA7we0vgUyUe4MI9q/vw3vZZT1fB =vB+E -----END PGP SIGNATURE----- --------------enigBB3303D4A9302FF3F5D16ED9--