[gentoo-user] copy a bunch of files...

[gentoo-user] copy a bunch of files...

[Mark Knecht]

Hi,
   Looking for a simple way to do a big copy at the command line. I
have a bunch of files (maybe 100 right now, but it will grow) that I
can find with locate and grep:

c2stable ~ # locate Correlation | grep Builder | grep csv
/home/mark/Builder/TF/TF.D-17M-2009_06-2010_11/Correlation/TF.D-17M-2009_06-2010_11-V1.csv
/home/mark/Builder/TF/TF.D-17M-2009_06-2010_11/Correlation/TF.D-17M-2009_06-2010_11-V2.csv
/home/mark/Builder/TF/TF.D-17M-2009_06-2010_11/Correlation/TF.D-17M-2009_06-2010_11-V3.csv
<SNIP>
/home/mark/Builder/TF/TF.D-31M-2009_06-2010_11/Correlation/TF.D-31M-2009_06-2010_11-V4.csv
/home/mark/Builder/TF/TF.D-31M-2009_06-2010_11/Correlation/TF.D-31M-2009_06-2010_11-V5.csv
c2stable ~ #

   I need to copy these files to a new directory
(~mark/CorrelationTests) where I will modify what's in them before
running correlation tests on the contents.

   How do I feed the output of the command above to cp at the command
line to get this done?

   I've been playing with things like while & read  but I can't get it right.

Thanks,
Mark

Re: [gentoo-user] copy a bunch of files...

[Florian Philipp]

[-- Attachment #1: Type: text/plain, Size: 1800 bytes --]

Am 08.02.2011 19:27, schrieb Mark Knecht:
> Hi,
>    Looking for a simple way to do a big copy at the command line. I
> have a bunch of files (maybe 100 right now, but it will grow) that I
> can find with locate and grep:
> 
> c2stable ~ # locate Correlation | grep Builder | grep csv
> /home/mark/Builder/TF/TF.D-17M-2009_06-2010_11/Correlation/TF.D-17M-2009_06-2010_11-V1.csv
> /home/mark/Builder/TF/TF.D-17M-2009_06-2010_11/Correlation/TF.D-17M-2009_06-2010_11-V2.csv
> /home/mark/Builder/TF/TF.D-17M-2009_06-2010_11/Correlation/TF.D-17M-2009_06-2010_11-V3.csv
> <SNIP>
> /home/mark/Builder/TF/TF.D-31M-2009_06-2010_11/Correlation/TF.D-31M-2009_06-2010_11-V4.csv
> /home/mark/Builder/TF/TF.D-31M-2009_06-2010_11/Correlation/TF.D-31M-2009_06-2010_11-V5.csv
> c2stable ~ #
> 
>    I need to copy these files to a new directory
> (~mark/CorrelationTests) where I will modify what's in them before
> running correlation tests on the contents.
> 
>    How do I feed the output of the command above to cp at the command
> line to get this done?
> 
>    I've been playing with things like while & read  but I can't get it right.
> 
> Thanks,
> Mark
> 

locate Correlation | grep Builder | grep csv | xargs -IARG cp ARG
~mark/CorrelationTests

-IARG tells xargs to replace the occurrence of ARG in the parameters
with the actual parameters read from stdin.

or

locate Correlation | grep Builder | grep csv | while read file; do
cp "$file" ~mark/CorrelationTests; done

BTW: Wouldn't grep 'Builder/.*\.csv' match better (some intermediate
directory Builder, ending on .csv)?

Even easier:
locate ~mark/'*/Builder/*.csv' | xargs -IARG cp ARG ~mark/CorrelationTests

Warning: I've not tested every line. Use with caution.

Hope this helps,
Florian Philipp


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [gentoo-user] copy a bunch of files...

[Mark Knecht]

On Tue, Feb 8, 2011 at 11:02 AM, Florian Philipp <lists@binarywings.net> wrote:
> Am 08.02.2011 19:27, schrieb Mark Knecht:
>> Hi,
>>    Looking for a simple way to do a big copy at the command line. I
>> have a bunch of files (maybe 100 right now, but it will grow) that I
>> can find with locate and grep:
>>
>> c2stable ~ # locate Correlation | grep Builder | grep csv
>> /home/mark/Builder/TF/TF.D-17M-2009_06-2010_11/Correlation/TF.D-17M-2009_06-2010_11-V1.csv
>> /home/mark/Builder/TF/TF.D-17M-2009_06-2010_11/Correlation/TF.D-17M-2009_06-2010_11-V2.csv
>> /home/mark/Builder/TF/TF.D-17M-2009_06-2010_11/Correlation/TF.D-17M-2009_06-2010_11-V3.csv
>> <SNIP>
>> /home/mark/Builder/TF/TF.D-31M-2009_06-2010_11/Correlation/TF.D-31M-2009_06-2010_11-V4.csv
>> /home/mark/Builder/TF/TF.D-31M-2009_06-2010_11/Correlation/TF.D-31M-2009_06-2010_11-V5.csv
>> c2stable ~ #
>>
>>    I need to copy these files to a new directory
>> (~mark/CorrelationTests) where I will modify what's in them before
>> running correlation tests on the contents.
>>
>>    How do I feed the output of the command above to cp at the command
>> line to get this done?
>>
>>    I've been playing with things like while & read  but I can't get it right.
>>
>> Thanks,
>> Mark
>>
>
> locate Correlation | grep Builder | grep csv | xargs -IARG cp ARG
> ~mark/CorrelationTests
>
> -IARG tells xargs to replace the occurrence of ARG in the parameters
> with the actual parameters read from stdin.
>

Thanks! This worked nicely and is relatively easy to remember.

> or
>
> locate Correlation | grep Builder | grep csv | while read file; do
> cp "$file" ~mark/CorrelationTests; done
>

This is what I was trying to do but was unsuccessful.

> BTW: Wouldn't grep 'Builder/.*\.csv' match better (some intermediate
> directory Builder, ending on .csv)?
>

Yes, that does seem to work. I guess that's grepping for the path of a
file starting with Builder and ending with CSV?

Good one.

> Even easier:
> locate ~mark/'*/Builder/*.csv' | xargs -IARG cp ARG ~mark/CorrelationTests
>
> Warning: I've not tested every line. Use with caution.
>
> Hope this helps,
> Florian Philipp

It did very much. Thanks!

Now to work on modifying the files, again with a loop for all the
files in ~mark/CorrelationTests

- Mark

[gentoo-user] Re: copy a bunch of files...

[Harry Putnam]

Mark Knecht <markknecht@gmail.com> writes:

>> locate Correlation | grep Builder | grep csv | while read file; do
>> cp "$file" ~mark/CorrelationTests; done

Just a minor point that would simplify the cmd by one cmd call.

You could use awk instead of 2 calls to grep.  It might be a tiny bit
slower... but I doubt it would be noticeable at the proposed scale.

awk is basic tool on linux for cmd line work... its quite a bit more
powerful than grep, but of course more complicated.

In this case though it would be a simple awk cmd.

 
Instead of 

   locate Correlation | grep Builder | grep csv | [...]

You could use:
   locate Correlation | awk '/Builder/ && /cvs/'| [...]
   

That should weed out the same files as the double call to grep

Re: [gentoo-user] copy a bunch of files...

[Kfir Lavi]

[-- Attachment #1: Type: text/plain, Size: 1473 bytes --]

On Tue, Feb 8, 2011 at 8:27 PM, Mark Knecht <markknecht@gmail.com> wrote:

> Hi,
>   Looking for a simple way to do a big copy at the command line. I
> have a bunch of files (maybe 100 right now, but it will grow) that I
> can find with locate and grep:
>
> c2stable ~ # locate Correlation | grep Builder | grep csv
>
> /home/mark/Builder/TF/TF.D-17M-2009_06-2010_11/Correlation/TF.D-17M-2009_06-2010_11-V1.csv
>
> /home/mark/Builder/TF/TF.D-17M-2009_06-2010_11/Correlation/TF.D-17M-2009_06-2010_11-V2.csv
>
> /home/mark/Builder/TF/TF.D-17M-2009_06-2010_11/Correlation/TF.D-17M-2009_06-2010_11-V3.csv
> <SNIP>
>
> /home/mark/Builder/TF/TF.D-31M-2009_06-2010_11/Correlation/TF.D-31M-2009_06-2010_11-V4.csv
>
> /home/mark/Builder/TF/TF.D-31M-2009_06-2010_11/Correlation/TF.D-31M-2009_06-2010_11-V5.csv
> c2stable ~ #
>
>   I need to copy these files to a new directory
> (~mark/CorrelationTests) where I will modify what's in them before
> running correlation tests on the contents.
>
>   How do I feed the output of the command above to cp at the command
> line to get this done?
>
>   I've been playing with things like while & read  but I can't get it
> right.
>
> Thanks,
> Mark
>
>
Another way to do it is with find:
find /home/mark/Builder -type f -iname '*csv' -exec cp {}
~mark/CorrelationTests \;

If catching the *csv is not enough, you can use -ipath instead of -iname and
do something like this:
-ipath '*Builder*csv'
this will match all the path.

Regards,
Kfir

[-- Attachment #2: Type: text/html, Size: 1937 bytes --]

Re: [gentoo-user] copy a bunch of files...

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 459 bytes --]

On Wed, 9 Feb 2011 13:38:37 +0200, Kfir Lavi wrote:

> Another way to do it is with find:
> find /home/mark/Builder -type f -iname '*csv' -exec cp {}
> ~mark/CorrelationTests \;

Replace \; with + for a faster process, as Mark said there are hundreds
of these files.

Or, if you use zsh instead of bash, it can be as simple as

cp Builder/**/*.csv CorrelationTests


-- 
Neil Bothwick

Learn from your parents' mistakes - use birth control!

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] copy a bunch of files...

[Helmut Jarausch]

On 02/09/2011 12:56:09 PM, Neil Bothwick wrote:
> On Wed, 9 Feb 2011 13:38:37 +0200, Kfir Lavi wrote:
> 
> > Another way to do it is with find:
> > find /home/mark/Builder -type f -iname '*csv' -exec cp {}
> > ~mark/CorrelationTests \;
> 
> Replace \; with + for a faster process, as Mark said there are
> hundreds
> of these files.
> 
> Or, if you use zsh instead of bash, it can be as simple as
> 
> cp Builder/**/*.csv CorrelationTests
> 
There is a problem with this approach, though.
It can easily give "command line too long".

Helmut.

Re: [gentoo-user] copy a bunch of files...

[Mark Knecht]

On Wed, Feb 9, 2011 at 4:09 AM, Helmut Jarausch
<jarausch@igpm.rwth-aachen.de> wrote:
> On 02/09/2011 12:56:09 PM, Neil Bothwick wrote:
>> On Wed, 9 Feb 2011 13:38:37 +0200, Kfir Lavi wrote:
>>
>> > Another way to do it is with find:
>> > find /home/mark/Builder -type f -iname '*csv' -exec cp {}
>> > ~mark/CorrelationTests \;
>>
>> Replace \; with + for a faster process, as Mark said there are
>> hundreds
>> of these files.
>>
>> Or, if you use zsh instead of bash, it can be as simple as
>>
>> cp Builder/**/*.csv CorrelationTests
>>
> There is a problem with this approach, though.
> It can easily give "command line too long".
>
> Helmut.

Lots of interesting ideas. I use apps so often I've never become very
strong at the command line and yet people built this whole Linux
empire using it. It's very powerful.

One thing I didn't make clear in my original post - it didn't seem
important to confuse my real question which was the copy itself and
not locating the files - but which likely changes how well some of
these commands would work in my specific case was that the Builder
directory actually has _many_ CSV files ut specifically I needed only
the ones in the Correlation directories. Additionally, being that this
is stock & futures trading data, generally at a given time I need the
CSV files for a specific symbol, for instance in the original post:

c2stable ~ # locate Correlation | grep Builder | grep csv
/home/mark/Builder/TF/TF.D-17M-2009_06-2010_11/Correlation/TF.D-17M-2009_06-2010_11-V1.csv
/home/mark/Builder/TF/TF.D-17M-2009_06-2010_11/Correlation/TF.D-17M-2009_06-2010_11-V2.csv
/home/mark/Builder/TF/TF.D-17M-2009_06-2010_11/Correlation/TF.D-17M-2009_06-2010_11-V3.csv
<SNIP>
/home/mark/Builder/TF/TF.D-31M-2009_06-2010_11/Correlation/TF.D-31M-2009_06-2010_11-V4.csv
/home/mark/Builder/TF/TF.D-31M-2009_06-2010_11/Correlation/TF.D-31M-2009_06-2010_11-V5.csv
c2stable ~ #

I knew I wanted the Correlation directory but it turned out I had
other directories with that name on the system, so I added the grep
Builder to get me into the right tree and CSV to find only the CSV
files. However at that point I only had Russell futures data (TF.D) so
I didn't have to go further. Now, however, as I bring in Dow futures
(YM, YM.D) , S&P 500 futures (ES, ES.D), and NASDAQ futures (NQ, ND.D)
I just an an extra grep in and I'm there in terms of finding the files
I need for a certain test.

Additionally I have test results for other date ranges that will show
up soon, (2001-2005, or 2003-2011, etc. Additional greps are an easy
way to just winnow it down to a point where I'm finding what I need to
find.

This thread has given me a lot of new commands to go look at.

Thanks,
Mark

Re: [gentoo-user] copy a bunch of files...

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 629 bytes --]

On Wed, 9 Feb 2011 06:46:34 -0800, Mark Knecht wrote:

> One thing I didn't make clear in my original post - it didn't seem
> important to confuse my real question which was the copy itself and
> not locating the files - but which likely changes how well some of
> these commands would work in my specific case was that the Builder
> directory actually has _many_ CSV files ut specifically I needed
> only the ones in the Correlation directories.

Bear in mind that locate only returns files in its database, not any
created since the last time its cron job was run. Find seems a more
appropriate tool for this task.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] copy a bunch of files...

[Mark Knecht]

On Wed, Feb 9, 2011 at 12:10 PM, Neil Bothwick <neil@digimed.co.uk> wrote:
> On Wed, 9 Feb 2011 06:46:34 -0800, Mark Knecht wrote:
>
>> One thing I didn't make clear in my original post - it didn't seem
>> important to confuse my real question which was the copy itself and
>> not locating the files - but which likely changes how well some of
>> these commands would work in my specific case was that the Builder
>> directory actually has _many_ CSV files ut specifically I needed
>> only the ones in the Correlation directories.
>
> Bear in mind that locate only returns files in its database, not any
> created since the last time its cron job was run. Find seems a more
> appropriate tool for this task.
>

Good point. As I bring these files down from a bunch of Windows VMs I
typically run updatedb before doing this, but find would probably be
safer.

Thanks,
Mark