From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from lists.gentoo.org ([140.105.134.102] helo=robin.gentoo.org) by nuthatch.gentoo.org with esmtp (Exim 4.60) (envelope-from ) id 1FqYjj-0008Sg-NS for garchives@archives.gentoo.org; Wed, 14 Jun 2006 17:01:00 +0000 Received: from robin.gentoo.org (localhost [127.0.0.1]) by robin.gentoo.org (8.13.6/8.13.6) with SMTP id k5EGvVYc017667; Wed, 14 Jun 2006 16:57:31 GMT Received: from ender.volumehost.net (adsl-69-154-123-202.dsl.fyvlar.swbell.net [69.154.123.202]) by robin.gentoo.org (8.13.6/8.13.6) with ESMTP id k5EGd1tp004657 for ; Wed, 14 Jun 2006 16:39:01 GMT Received: from localhost (localhost [127.0.0.1]) by ender.volumehost.net (Postfix) with ESMTP id 96EB516808 for ; Wed, 14 Jun 2006 16:38:22 +0000 (UTC) Received: from ender.volumehost.net ([127.0.0.1]) by localhost (ender.volumehost.net [127.0.0.1]) (amavisd-new, port 10024) with LMTP id 03869-05 for ; Wed, 14 Jun 2006 16:38:21 +0000 (UTC) Received: from [192.168.1.65] (ip70-178-219-238.ks.ks.cox.net [70.178.219.238]) (using SSLv3 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ender.volumehost.net (Postfix) with ESMTP id E26FA147FA for ; Wed, 14 Jun 2006 16:38:20 +0000 (UTC) From: "Boyd Stephen Smith Jr." To: gentoo-user@lists.gentoo.org Subject: Re: [gentoo-user] how does a pipe work? Which process wait for which one, or they don't actually wait each other? Date: Wed, 14 Jun 2006 11:38:53 -0500 User-Agent: KMail/1.9.3 References: <1150290735.8524.12.camel@joe.realss> In-Reply-To: <1150290735.8524.12.camel@joe.realss> X-Eric-Conspiracy: There is no conspiracy Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-user@gentoo.org Reply-to: gentoo-user@lists.gentoo.org MIME-Version: 1.0 Content-Type: multipart/signed; boundary="nextPart1258254.XXRLUtjfmJ"; protocol="application/pgp-signature"; micalg=pgp-sha1 Content-Transfer-Encoding: 7bit Message-Id: <200606141138.57986.bss03@volumehost.net> X-Virus-Scanned: amavisd-new at volumehost.net X-Archives-Salt: 55fa804e-9351-43ff-9758-a3eb080bcfda X-Archives-Hash: c316a2e8b30ecbb406fc64c8f47237c5 --nextPart1258254.XXRLUtjfmJ Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline On Wednesday 14 June 2006 08:12, =E5=BC=A0=E9=9F=A1=E6=AD=A6 wrote=20 about '[gentoo-user] how does a pipe work? Which process wait for which=20 one, or they don't actually wait each other?': > How does pipe actually work? I mean, when there is a pipe like this: > $ appA | appB > What happen if appA produced output when appB is still busy processing > the data and did not require any data from input? > > possibility 1) as long as appA can get the resource (CPU?), it simply > keep outputing data, and this data is cached in memory, as long as there > is enough memory, and will finally feed to appB when appB finished his > business and begin to accept more data; > > possibility 2) as long as appB stop requiring data, appA is suspended, > the resource goes to appB. appA is only given resource (CPU?) when appB > finished his business and begin to accept more data; > > Which one is true? I know usually 1) and 2) makes no difference to most > users, that's why the detail explanation of "1) or 2)" is so hard to > google out. Neither! Both! The implementation of pipes varies from *NIX to *NIX, and possibly within=20 the same *NIX, since a shell might 'enhance' the pipes provided by the=20 kernel/libc. (The shell binary is ultimately responsible for implementing= =20 the pipe, so it may arbitrarily 'decorate' a standard pipe.) In any case, you can't depend on any particular behavior if you want your=20 shell script to be portable. In linux/bash I believe it works like this: Each, pipe has a small (~1=20 page) FIFO buffer in memory. (Not sure if this is kernel or userspace.) =20 Both processes are started and compete for CPU time in the standard way. =20 Either process may block on I/O when it performs standard, blocking I/O on= =20 the pipe. appA will block if the FIFO gets full; appB will block if the=20 =46IFO gets empty. If you really must know: Use the Source, Luke. > In my case appA gets the data from another host who have very short > timeout settings, appB is used to compress the data obtained from appA. > the compression is very difficult, usually at 30Kbps, the network is > very fast, around 10Mbps. appB compress the data tunck by tunck, if > Linux actually works in mode 2), the network connection is dropped when > the interval of two tuncks of appB compressing data is longer then the > network timeout setting. appA acutally don't know how to restart > connection from where it was dropped, thus understanding this difference > makes sense to me. This also depends a lot on the application. appA can use asynchronous I/O,= =20 provide a larger buffer (perhaps even a temporary file), and/or send=20 keepalives through the network. Also, appB's compression my be=20 interrupted while more data is written to the buffer. > I made several experiements and my appA and appB both works fine, but I > don't dare to share this appA/B to others unless I get the mechnism > understood. With the speeds you mention, the timeout would have to be ~8s or less for=20 the socket to be dropped.[1] Once a socket is established, they are=20 amazingly stable; timeout for an established socket is usually more like=20 5-10 minutes or even an hour. Heck, I think OBSD 3.8 defaulted to a 1 DAY= =20 timeout before the OS reaped an established socket. Also, you generally want to compress stuff before putting it on the wire,=20 not after... =2D-=20 "If there's one thing we've established over the years, it's that the vast majority of our users don't have the slightest clue what's best for them in terms of package stability." =2D- Gentoo Developer Ciaran McCreesh [1] That's assuming 15Kbps compression rate and the ability to send=20 full-size 16KB ip packets. Most likely, ~1s would suffice, since packets=20 are generally 1500B ~=3D 15Kb in size. --nextPart1258254.XXRLUtjfmJ Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (GNU/Linux) iD8DBQBEkDuhq72nDbhDXToRAo8CAJ0bZL1CPgJG/yfJuaaYgQtczhwuhACfSnJd gTGUCRm7QaUDiAAY9e9wU68= =G/9z -----END PGP SIGNATURE----- --nextPart1258254.XXRLUtjfmJ-- -- gentoo-user@gentoo.org mailing list