[gentoo-user] xargs and rm funkiness

[gentoo-user] xargs and rm funkiness

[Daniel D Jones]

Running the command:

find -name *.ext | xargs -0 rm

I get the result:

rm: cannot remove `Long File Name One.ext\nLong File Name Two.ext\nLong File 
Name Three.ext\n': File name too long.

(The actual list is much longer than this, of course, or I wouldn't be using 
xargs.)  For some reason, the \n isn't being recognized as a separator but 
rather as a part of a single long file name.  Don't think $IFS would affect a 
command like rm but it doesn't appear to be the issue:

ddjones@merlin ~ $ set | grep IFS
IFS=$' \t\n'

I don't see any other ser variable which looks like a likely candidate to 
cause the behavior.  Anyone have a clue what's going on?

-- 
"Why assume so glibly that the God who created the universe is still running 
it? It is certainly conceivable that He may have finished it and then turned it 
over to lesser gods to operate." - H.L. Mencken

Re: [gentoo-user] xargs and rm funkiness

[covici]

Daniel D Jones <ddjones@riddlemaster.org> wrote:

> Running the command:
> 
> find -name *.ext | xargs -0 rm
> 
> I get the result:
> 
> rm: cannot remove `Long File Name One.ext\nLong File Name Two.ext\nLong File 
> Name Three.ext\n': File name too long.
> 
> (The actual list is much longer than this, of course, or I wouldn't be using 
> xargs.)  For some reason, the \n isn't being recognized as a separator but 
> rather as a part of a single long file name.  Don't think $IFS would affect a 
> command like rm but it doesn't appear to be the issue:
> 
> ddjones@merlin ~ $ set | grep IFS
> IFS=$' \t\n'
> 
> I don't see any other ser variable which looks like a likely candidate to 
> cause the behavior.  Anyone have a clue what's going on?
Why do you have -0 -- this replaces the \n's with a null character -- is
that what you want?

-- 
Your life is like a penny.  You're going to lose it.  The question is:
How do
you spend it?

         John Covici
         covici@ccs.covici.com

Re: [gentoo-user] xargs and rm funkiness

[Daniel D Jones]

On Friday 21 May 2010 22:11:49 covici@ccs.covici.com wrote:
> Daniel D Jones <ddjones@riddlemaster.org> wrote:
> > Running the command:
> >
> > find -name *.ext | xargs -0 rm
> >
> > I get the result:
> >
> > rm: cannot remove `Long File Name One.ext\nLong File Name Two.ext\nLong
> > File Name Three.ext\n': File name too long.
> >
> > (The actual list is much longer than this, of course, or I wouldn't be
> > using xargs.)  For some reason, the \n isn't being recognized as a
> > separator but rather as a part of a single long file name.  Don't think
> > $IFS would affect a command like rm but it doesn't appear to be the
> > issue:
> >
> > ddjones@merlin ~ $ set | grep IFS
> > IFS=$' \t\n'
> >
> > I don't see any other ser variable which looks like a likely candidate to
> > cause the behavior.  Anyone have a clue what's going on?
> 
> Why do you have -0 -- this replaces the \n's with a null character -- is
> that what you want?

Not exactly.  It doesn't replace anything.  It tells xargs to look for a null 
to separate fields and to ignore the normal field separation characters.  This 
is required if you have spaces in the field name, otherwise xargs sees the 
spaces as a field separator.

xargs was doing exactly what I told it to do.  Unfortunately, I didn't read 
the man pages and was relying on some poorly written web pages which indicated 
that -0 told xargs to skip spaces but didn't mention that it also told it to 
ignore \n.  

The solution, as Patrick Holthaus pointed out, was to use the -print0 argument 
with find, which instructs find to use the null character as a field separator in 
its output.

-- 
"The road of excess leads to the palace of wisdom." - William Blake

Re: [gentoo-user] xargs and rm funkiness

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 561 bytes --]

On Fri, 21 May 2010 21:49:49 -0400, Daniel D Jones wrote:

> find -name *.ext | xargs -0 rm
> 
> I get the result:
> 
> rm: cannot remove `Long File Name One.ext\nLong File Name Two.ext\nLong
> File Name Three.ext\n': File name too long.

xargs can suck with anything but plain ASCII-without-spaces filenames.,
and it quite unnecessary here.

find -name *.ext -exe rm "{}" \;

or maybe even

find -name *.ext -exe rm "{}" +


-- 
Neil Bothwick

He who asks a question is a fool for a minute,
He who doesn't ask is a fool for a lifetime.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] xargs and rm funkiness

[Jan Engelhardt]

On Saturday 2010-05-22 09:46, Neil Bothwick wrote:

>On Fri, 21 May 2010 21:49:49 -0400, Daniel D Jones wrote:
>
>> find -name *.ext | xargs -0 rm
>> 
>> I get the result:
>> 
>> rm: cannot remove `Long File Name One.ext\nLong File Name Two.ext\nLong
>> File Name Three.ext\n': File name too long.
>
>xargs can suck with anything but plain ASCII-without-spaces filenames.,

That's why you use NUL terminators. Simple as that.

Also find's -exe calls rm *for each* file when used with \;.

>find -name *.ext -exe rm "{}" \;
>
>or maybe even
>
>find -name *.ext -exe rm "{}" +

Re: [gentoo-user] xargs and rm funkiness

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 500 bytes --]

On Sat, 22 May 2010 20:38:53 +0200 (CEST), Jan Engelhardt wrote:

> >xargs can suck with anything but plain ASCII-without-spaces
> >filenames.,  
> 
> That's why you use NUL terminators. Simple as that.

That only helps with spaces, what if filenames contain other reserved
characters?

> Also find's -exe calls rm *for each* file when used with \;.

Which is why I suggested +, but the -delete option may be even better.


-- 
Neil Bothwick

Echo > Speak: "Whale oil beef hooked"

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] xargs and rm funkiness

[Joerg Schilling]

Neil Bothwick <neil@digimed.co.uk> wrote:

> xargs can suck with anything but plain ASCII-without-spaces filenames.,
> and it quite unnecessary here.
>
> find -name *.ext -exe rm "{}" \;
>
> or maybe even
>
> find -name *.ext -exe rm "{}" +

Just avoid xargs as it is the source of the proplem.

find -name *.ext -exec some-command "{}" +

Is the preferred method since 1990.

Jörg

-- 
 EMail:joerg@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin
       js@cs.tu-berlin.de                (uni)  
       joerg.schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily

Re: [gentoo-user] xargs and rm funkiness

[Patrick Holthaus]

[-- Attachment #1: Type: Text/Plain, Size: 205 bytes --]

Hey!

On Saturday 22 May 2010 03:49:49 Daniel D Jones wrote:
> Running the command:
> 
> find -name *.ext | xargs -0 rm

You might try:

find -name *.ext -print0 | xargs -0 rm

Cheers
Patrick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-user] xargs and rm funkiness

[Joerg Schilling]

Patrick Holthaus <patrick.holthaus@uni-bielefeld.de> wrote:

> You might try:
>
> find -name *.ext -print0 | xargs -0 rm

But this is non-standard.

UNIX introduced -exec {} + 1990 (when David Korn rewrote find(1)
and it is in the POSIX standared since some time.

Jörg

-- 
 EMail:joerg@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin
       js@cs.tu-berlin.de                (uni)  
       joerg.schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily

Re: [gentoo-user] xargs and rm funkiness

[Daniel D Jones]

On Wednesday 26 May 2010 06:42:08 Joerg Schilling wrote:
> Patrick Holthaus <patrick.holthaus@uni-bielefeld.de> wrote:
> > You might try:
> >
> > find -name *.ext -print0 | xargs -0 rm
> 
> But this is non-standard.

In what way is this non-standard?  That is, what standard is it contrary to?  
TMTOWTDI (There's More Than One Way To Do It) applies just as strongly to *nix 
in general as it does to Perl.  When there are multiple ways to do something, 
it's often either a user preference issue or the method should be decided 
based upon the particular details of the desired result.  -exec may be a POSIX 
standard function, but that doesn't mean it must be used over other options or 
you're breaking the standard.
 
> UNIX introduced -exec {} + 1990 (when David Korn rewrote find(1)
> and it is in the POSIX standared since some time.

-exec (which potentially has problems with race conditions - -execdir should 
almost always be used instead) runs the command once for each file found.  
xargs will call the command once for as many files as it can fit on the command 
line.  For some instances, like rm, that probably isn't significant.  But if 
you're calling a complex process with lots of files, the overhead of starting 
the many extra processes may be significant.

-- 
"You have attributed conditions to villainy that simply result from 
stupidity." - Robert A. Heinlein

Re: [gentoo-user] xargs and rm funkiness

[Etaoin Shrdlu]

On Saturday 29 May 2010, Daniel D Jones wrote:
> On Wednesday 26 May 2010 06:42:08 Joerg Schilling wrote:
> > Patrick Holthaus <patrick.holthaus@uni-bielefeld.de> wrote:
> > > You might try:
> > >
> > > find -name *.ext -print0 | xargs -0 rm
> >
> > But this is non-standard.
> 
> In what way is this non-standard?  That is, what standard is it contrary
>  to?

SUS (aka POSIX), although some people are pushing to include -print0 | xargs 
-0 into the standard. What Joerg meant is that the above construct will only 
run when using GNU find and xargs. Of course, if you're running Linux, that is 
probably the case already anyway.

>  TMTOWTDI (There's More Than One Way To Do It) applies just as strongly
>  to *nix in general as it does to Perl.  When there are multiple ways to do
>  something, it's often either a user preference issue or the method should
>  be decided based upon the particular details of the desired result.  -exec
>  may be a POSIX standard function, but that doesn't mean it must be used
>  over other options or you're breaking the standard.
> 
> > UNIX introduced -exec {} + 1990 (when David Korn rewrote find(1)
> > and it is in the POSIX standared since some time.
> 
> -exec (which potentially has problems with race conditions - -execdir
>  should almost always be used instead) runs the command once for each file
>  found. 

If you use -exec {} + as he wrote, this is not true.

>  xargs will call the command once for as many files as it can fit on
>  the command line. 

And so does -exec {} +

>  For some instances, like rm, that probably isn't
>  significant.  But if you're calling a complex process with lots of files,
>  the overhead of starting the many extra processes may be significant.

See above.

Re: [gentoo-user] xargs and rm funkiness

[Joerg Schilling]

Etaoin Shrdlu <shrdlu@unlimitedmail.org> wrote:

> On Saturday 29 May 2010, Daniel D Jones wrote:
> > On Wednesday 26 May 2010 06:42:08 Joerg Schilling wrote:
> > > Patrick Holthaus <patrick.holthaus@uni-bielefeld.de> wrote:
> > > > You might try:
> > > >
> > > > find -name *.ext -print0 | xargs -0 rm
> > >
> > > But this is non-standard.
> > 
> > In what way is this non-standard?  That is, what standard is it contrary
> >  to?
>
> SUS (aka POSIX), although some people are pushing to include -print0 | xargs 
> -0 into the standard. What Joerg meant is that the above construct will only 
> run when using GNU find and xargs. Of course, if you're running Linux, that is 
> probably the case already anyway.
>

And there is a big cheavat against this proposal as xargs -0 was introduced 
long after -exec + exsists and as introducing xargs -0 would force us to 
change _many_ other utilities too in order to come to a consistent overall
behavior again. For this reason, there was even the proposal to instead remove 
"xargs" from the standard as -exec + does everything that is needed.

Jörg

-- 
 EMail:joerg@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin
       js@cs.tu-berlin.de                (uni)  
       joerg.schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily

Re: [gentoo-user] xargs and rm funkiness

[Alan McKinnon]

On Saturday 29 May 2010 17:05:34 Daniel D Jones wrote:
> On Wednesday 26 May 2010 06:42:08 Joerg Schilling wrote:
> > Patrick Holthaus <patrick.holthaus@uni-bielefeld.de> wrote:
> > > You might try:
> > > 
> > > find -name *.ext -print0 | xargs -0 rm
> > 
> > But this is non-standard.
> 
> In what way is this non-standard?  That is, what standard is it contrary
> to? TMTOWTDI (There's More Than One Way To Do It) applies just as strongly
> to *nix in general as it does to Perl.  When there are multiple ways to do
> something, it's often either a user preference issue or the method should
> be decided based upon the particular details of the desired result.  -exec
> may be a POSIX standard function, but that doesn't mean it must be used
> over other options or you're breaking the standard.
> 
> > UNIX introduced -exec {} + 1990 (when David Korn rewrote find(1)
> > and it is in the POSIX standared since some time.
> 
> -exec (which potentially has problems with race conditions - -execdir
> should almost always be used instead) runs the command once for each file
> found. xargs will call the command once for as many files as it can fit on
> the command line.  For some instances, like rm, that probably isn't
> significant.  But if you're calling a complex process with lots of files,
> the overhead of starting the many extra processes may be significant.

Perhaps you don't know Joerg yet. When dealing with the man, it's important to 
know where he's coming from - and that is not "how Linux does stuff"

He invariably refers to POSIX when mentioning standards. He uses this standard 
to ensure that his code will work on any *nix platform. This puts him at odds 
with the Linux crowd sometimes - two very different viewpoints.

It's not "-exec" that causes one processto be launched per item found, it is 
"-exec \;"

He referred to "-exec +" which has the same behaviour as you mention - use as 
many filenames as will fit on the command line.


-- 
alan dot mckinnon at gmail dot com

Re: [gentoo-user] xargs and rm funkiness

[Daniel D Jones]

On Saturday 29 May 2010 14:59:16 Alan McKinnon wrote:
> On Saturday 29 May 2010 17:05:34 Daniel D Jones wrote:
...
> > -exec (which potentially has problems with race conditions - -execdir
> > should almost always be used instead) runs the command once for each file
> > found. xargs will call the command once for as many files as it can fit
> > on the command line.  For some instances, like rm, that probably isn't
> > significant.  But if you're calling a complex process with lots of files,
> > the overhead of starting the many extra processes may be significant.
> 
> Perhaps you don't know Joerg yet. When dealing with the man, it's important
>  to know where he's coming from - and that is not "how Linux does stuff"
> 
> He invariably refers to POSIX when mentioning standards. He uses this
>  standard to ensure that his code will work on any *nix platform. This puts
>  him at odds with the Linux crowd sometimes - two very different
>  viewpoints.

I wasn't coming from a Linux perspective.  I'm a network engineer.  At work, I 
touch SSH servers running SunOS, file servers running BSD (don't recall what 
flavor off the top of my head - I'm not in them that often), terminals running 
HPUX and run Linux at home.  xargs is available on all of them. 
 
> It's not "-exec" that causes one processto be launched per item found, it
>  is "-exec \;"
> 
> He referred to "-exec +" which has the same behaviour as you mention - use
>  as many filenames as will fit on the command line.

You're correct, of course.  I missed that in the man pages.  Mea culpa.  (I'm 
a network engineer, not a sysadmin.)

-- 
"If "everybody knows" such-and-such, then it ain't so, by at least ten 
thousand to one." - Robert A. Heinlein

Re: [gentoo-user] xargs and rm funkiness

[David W Noon]

[-- Attachment #1: Type: text/plain, Size: 835 bytes --]

On Sat, 22 May 2010 10:10:02 +0200, Neil Bothwick wrote about Re:
[gentoo-user] xargs and rm funkiness:

>On Fri, 21 May 2010 21:49:49 -0400, Daniel D Jones wrote:
>
>> find -name *.ext | xargs -0 rm
>> 
>> I get the result:
>> 
>> rm: cannot remove `Long File Name One.ext\nLong File Name
>> Two.ext\nLong File Name Three.ext\n': File name too long.
>
>xargs can suck with anything but plain ASCII-without-spaces filenames.,
>and it quite unnecessary here.
>
>find -name *.ext -exe rm "{}" \;
>
>or maybe even
>
>find -name *.ext -exe rm "{}" +

Or simpler still:

find -name *.ext -delete

-- 
Regards,

Dave  [RLU #314465]
======================================================================
dwnoon@ntlworld.com (David W Noon)
======================================================================

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] xargs and rm funkiness

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 289 bytes --]

On Sat, 22 May 2010 12:35:10 +0100, David W Noon wrote:

> >find -name *.ext -exe rm "{}" +  
> 
> Or simpler still:
> 
> find -name *.ext -delete

Neat - I hadn't noticed that option.

Anyone for find / -delete ?


-- 
Neil Bothwick

Top Oxymorons Number 31: Small crowd

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] xargs and rm funkiness

[Daniel D Jones]

On Saturday 22 May 2010 08:25:55 Neil Bothwick wrote:
> On Sat, 22 May 2010 12:35:10 +0100, David W Noon wrote:
> > >find -name *.ext -exe rm "{}" +
> >
> > Or simpler still:
> >
> > find -name *.ext -delete
> 
> Neat - I hadn't noticed that option.
> 
> Anyone for find / -delete ?

If you use the -delete switch, just be careful.  From the man page:

<QUOTE>
Warnings: Don't forget that the find command line is evaluated as an 
expression, so putting -delete first will make find try to delete               
everything  below  the starting points you specified.  
</QUOTE>

That means that the command

find -name *.tmp -delete

will delete all your temp files while

find -delete -name *.tmp

will delete everything below your current directory.  If you're in the root 
directory, it's equivalent to running your suggested command above.  I just 
found this out the hard way, although luckily I wasn't in the root directory 
when I ran the command and so didn't trash my system.  I did lose the changes 
I'd made to the project I was working on but fortunately had a backup of the 
original files.
 
-- 
"If you give me six lines written by the most honest man, I will find something 
in them to hang him." - Cardinal Richelieu