From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from pigeon.gentoo.org ([208.92.234.80] helo=lists.gentoo.org) by finch.gentoo.org with esmtp (Exim 4.60) (envelope-from ) id 1Ou65X-0002J4-RJ for garchives@archives.gentoo.org; Fri, 10 Sep 2010 16:04:47 +0000 Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 0235BE0822 for ; Fri, 10 Sep 2010 16:04:29 +0000 (UTC) Received: from mail-yw0-f53.google.com (mail-yw0-f53.google.com [209.85.213.53]) by pigeon.gentoo.org (Postfix) with ESMTP id 7677BE087F for ; Fri, 10 Sep 2010 15:10:08 +0000 (UTC) Received: by ywf7 with SMTP id 7so1583749ywf.40 for ; Fri, 10 Sep 2010 08:10:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:sender:received :in-reply-to:references:date:x-google-sender-auth:message-id:subject :from:to:content-type; bh=6Lfzh0P1HQQTWdlUcPqvXzC/zpI2DbQu5na5yzK2NSU=; b=phH2daBKOuLsHVIoik68M/Y5UznPzdaUBsbECrsesov8yozHNZCVToBvxmnZDO9/Cc vG8yPWRqUesf7X8IOB/OuVf5406RFmhet/Y8FyedKYdcMrUBmxpjrOex/0FT7FF0dMOo DddiiRI/d8a0SAMFB+47zee+ZEzSz55rPWs+U= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:content-type; b=Rg0fhmCwtE4WuRhwp42e1E+1qIohxAR+a/7IBwoMIGPeZ+od5/0XEhBAH+AJIgvwQK d+lP5JxQgjiwrch3if2ZegnHt3YC6XL27y0wx+ja60uJECyoBKNp4BfYEvIyW3U9nbO5 06Zj/R8I6G8sNeRh6NbSELm4e/mD6TVkTsY/U= Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-user@lists.gentoo.org Reply-to: gentoo-user@lists.gentoo.org MIME-Version: 1.0 Received: by 10.239.137.74 with SMTP id k10mr43384hbk.202.1284131407345; Fri, 10 Sep 2010 08:10:07 -0700 (PDT) Sender: paul.hartman@gmail.com Received: by 10.239.137.78 with HTTP; Fri, 10 Sep 2010 08:10:07 -0700 (PDT) In-Reply-To: <4C8947AD.6010906@admin-box.com> References: <4C8947AD.6010906@admin-box.com> Date: Fri, 10 Sep 2010 10:10:07 -0500 X-Google-Sender-Auth: hnJl_JMmYCnOKCnNhRRkpbk4goc Message-ID: Subject: Re: [gentoo-user] Pipe Lines - A really basic question From: Paul Hartman To: gentoo-user@lists.gentoo.org Content-Type: text/plain; charset=ISO-8859-1 X-Archives-Salt: ef39fb9c-0164-4ce5-8ea6-034f49c2b41c X-Archives-Hash: 85e1dae4119bd0a6fab28cc8543a0657 On Thu, Sep 9, 2010 at 3:46 PM, Daniel Troeder wrote: > On 09/09/2010 07:24 PM, Matt Neimeyer wrote: >> My generic question is: When I'm using a pipe line series of commands >> do I use up more/less space than doing things in sequence? >> >> For example, I have a development Gentoo VM that has a hard drive that >> is too small... I wanted to move a database off of that onto another >> machine but when I tried the following I filled my partition and 'evil >> things' happened... >> >> mysqldump blah... >> gzip blah... >> >> In this specific case I added another virtual drive, mounted that and >> went on with life but I'm curious if I could have gotten away with the >> pipe line instead. Will doing something like this still use "twice" >> the space? >> >> mysqldump | gzip > file.sql.gz >> >> OR going back to my generic question if I pipe line like "type | sort >> | unique > output" does that only use 1x or 3x the disk space? >> >> Thanks in advance! >> >> Matt >> >> P.S. If the answer is "it depends" how do know what it depends on? >> > Everyone already answered the disk space question. I want to add just > this: It also saves you lots of i/o-bandwidth: only the compressed data > gets written to disk. As i/o is the most common bottleneck, it is often > an imperative to do as much as possible in a pipe. If you're lucky it > can also mean, that multiple programs run at the same time, resulting in > higher throughput. Lucky is, when consumer and producer (right and left > of pipe) can work simultaneously because the buffer is big enough. You > can see this every time you (un)pack a tar.gz. And if you have a huge amount of data where compression causes CPU to become the bottleneck you can use something like pbzip2 which uses all CPUs/cores in parallel to speed up [de]compression. :)