[gentoo-user] is a nice "place" :-D

[gentoo-user] is a nice "place" :-D

[Felix Miata]

After attempting to install for the first time last week, I started 3 
different threads here looking for help. I'm pleased with the nature of the 
responses, and being able to succeed eventually using a mix of those 
responses and my own efforts digging into Google, gentoo.org and cranial 
cobwebs. So, thanks to all who replied, and even to those who showed interest 
without replying.

For http://fm.no-ip.com/Tmp/Linux/G/, newly created to use with those three 
threads, 'cat /var/log/apache2/access_log | grep "GET /Tmp/Linux/G" | grep -v 
<myip> | sort > outfile' generated 117 lines. That's a lot more hits than I 
can ever remember getting before when asking for help from a mailing list 
(even if it did take 5 days to accumulate so many).

I'm curious if anyone here would like to offer a better variant of my local 
query that would limit the hit count so that no more than one hit per IP is 
represented in the output? My skill with such things is very limited. I can't 
think of the the name of a command to cut the IP off the front of each line, 
much less how to compare if it's a non-first instance to be discarded. Or, 
maybe there's an Apache utility for doing this that I just don't know about?
-- 
"The wise are known for their understanding, and pleasant
words are persuasive." Proverbs 16:21 (New Living Translation)

  Team OS/2 ** Reg. Linux User #211409 ** a11y rocks!

Felix Miata  ***  http://fm.no-ip.com/

Re: [gentoo-user] is a nice "place" :-D

[Alan McKinnon]

Apparently, though unproven, at 01:10 on Tuesday 17 May 2011, Felix Miata did 
opine thusly:

> After attempting to install for the first time last week, I started 3
> different threads here looking for help. I'm pleased with the nature of the
> responses, and being able to succeed eventually using a mix of those
> responses and my own efforts digging into Google, gentoo.org and cranial
> cobwebs. So, thanks to all who replied, and even to those who showed
> interest without replying.
> 
> For http://fm.no-ip.com/Tmp/Linux/G/, newly created to use with those three
> threads, 'cat /var/log/apache2/access_log | grep "GET /Tmp/Linux/G" | grep
> -v <myip> | sort > outfile' generated 117 lines. That's a lot more hits
> than I can ever remember getting before when asking for help from a
> mailing list (even if it did take 5 days to accumulate so many).
> 
> I'm curious if anyone here would like to offer a better variant of my local
> query that would limit the hit count so that no more than one hit per IP is
> represented in the output? My skill with such things is very limited. I
> can't think of the the name of a command to cut the IP off the front of
> each line, much less how to compare if it's a non-first instance to be
> discarded. Or, maybe there's an Apache utility for doing this that I just
> don't know about?

There's always a million ways to skin a cat like this. At a high volume site 
you would of course not try and deal with this directly from the apache logs. 
You would send them to syslog which would parse them and write them to a 
database from where you could run sophisticated SQL.

There are also Apache analyser apps out there, google will find them.

But I think all that is overkill for what you want. Your command works fine 
except for needing to discard duplicate IPs. You don't seem to need to know 
the details of the GET, so just grab using awk the first field and sort | uniq 
the result. It will run a tad quicker (and reveal less n00bness to your 
audience) if you grep the file directly instead of cat | grep:

grep "GET /Tmp/Linux/G" | /var/log/apache2/access_log | grep-v <myip> | \
awk '{print $1}' | sort | uniq | wc

In true grand Unix tradition you cannot get quicker, dirtier or more effective 
than that


-- 
alan dot mckinnon at gmail dot com

Re: [gentoo-user] is a nice "place" :-D

[Willie Wong]

On Tue, May 17, 2011 at 01:33:39AM +0200, Alan McKinnon wrote:
> grep "GET /Tmp/Linux/G" | /var/log/apache2/access_log | grep-v <myip> | \
> awk '{print $1}' | sort | uniq | wc
> 
> In true grand Unix tradition you cannot get quicker, dirtier or more effective 
> than that
> 

You can replace "sort | uniq" by "sort -u"

And the "Grand Unix Tradition" probably would 'cut' instead of awk :)

While you are at it, an incantation that pipes grep to awk? Seriously?

W
-- 
Willie W. Wong                                     wwong@math.princeton.edu
Data aequatione quotcunque fluentes quantitae involvente fluxiones invenire 
         et vice versa   ~~~  I. Newton

Re: [gentoo-user] is a nice "place" :-D

[Felix Miata]

On 2011/05/17 01:33 (GMT+0200) Alan McKinnon composed:

> grep "GET /Tmp/Linux/G" | /var/log/apache2/access_log | grep-v<myip>  | \
> awk '{print $1}' | sort | uniq | wc

> In true grand Unix tradition you cannot get quicker, dirtier or more effective
> than that

It almost worked too. :-)

	grep "GET /Tmp/Linux/G" /var/log/apache2/access_log | grep -v <myip>  | \
	awk '{print $1}' | sort | uniq | wc -l

got me almost what I wanted, 20 unique IPs, but that's a lot of stuff to 
remember, which for me will never happen. So I tried converting to an alias.

	grep "GET $1" | /var/log/apache2/access_log | grep -v <myip>  | \
	awk '{print $1}' | sort | uniq | wc -l

sort of works, except I won't always be looking for GET as part of what to 
grep for, or might require more than one whitepsace instance, and am tripping 
over how to deal with the whitespace if I leave GET out of the alias and only 
put on cmdline if I actually want it as part of what to grep for.

	grep "GET $1 $2" | /var/log/apache2/access_log | grep -v <myip>  | \
	awk '{print $1}' | sort | uniq | wc -l

seems to work, but I'm not sure there aren't booby traps besides 2nd or more 
whitespace instances I'm not considering, even though it gets the same answer 
for this particular case.
-- 
"The wise are known for their understanding, and pleasant
words are persuasive." Proverbs 16:21 (New Living Translation)

  Team OS/2 ** Reg. Linux User #211409 ** a11y rocks!

Felix Miata  ***  http://fm.no-ip.com/

Re: [gentoo-user] is a nice "place" :-D

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 820 bytes --]

On Tue, 17 May 2011 01:33:39 +0200, Alan McKinnon wrote:

> grep "GET /Tmp/Linux/G" | /var/log/apache2/access_log | grep-v <myip> |
> \ awk '{print $1}' | sort | uniq | wc
> 
> In true grand Unix tradition you cannot get quicker, dirtier or more
> effective than that
> 

awk does pattern matching, o you can ditch the grep stage and use

 awk '! /myip/ {print $1}'

You could use awk to search for the GET patterns too, not only saving yet
another process, but making sure that no one else, including you next
month, can work out what the command is supposed to do.

sort -u would save having a separate process for uniq, but I've no idea
if it's faster. It's only worth using sort -u if you would use uniq with
no arguments.


-- 
Neil Bothwick

- We are but packets in the internet of Life-

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] is a nice "place" :-D

[Pandu Poluan]

On 2011-05-17, Neil Bothwick <neil@digimed.co.uk> wrote:
> On Tue, 17 May 2011 01:33:39 +0200, Alan McKinnon wrote:
>
>> grep "GET /Tmp/Linux/G" | /var/log/apache2/access_log | grep-v <myip> |
>> \ awk '{print $1}' | sort | uniq | wc
>>
>> In true grand Unix tradition you cannot get quicker, dirtier or more
>> effective than that
>>
>
> awk does pattern matching, o you can ditch the grep stage and use
>
>  awk '! /myip/ {print $1}'
>
> You could use awk to search for the GET patterns too, not only saving yet
> another process, but making sure that no one else, including you next
> month, can work out what the command is supposed to do.
>

Meh, me forgetting what an awk snippet do? Never!

sed ... now that's a wholly different story :-P

> sort -u would save having a separate process for uniq, but I've no idea
> if it's faster. It's only worth using sort -u if you would use uniq with
> no arguments.
>

And you can actually do the 'uniq' or '-u' function within awk. Quite
easily, in fact.

Here's a sample of awk doing uniq:

awk '!x[$1]++ { print $1 }'

Benefit? It doesn't care if the non-unique lines are one-after-another
or spread all over the text. The above snippet prints only the first
occurence. Combine that with a test for match:

awk '!x[$1]++ && $0 ~ /awesome_regex_pattern/ {print $1}'

then with a test for negated match

awk '!x[$1]++ && $0 ~ /awesome_regex_pattern/ && $0 !~
/more_awesome_regex/ {print $1}'

Rgds,
--
Pandu E Poluan - IT Optimizer
My website: http://pandu.poluan.info/

Re: [gentoo-user] is a nice "place" :-D

[Juan Diego Tascón]

On Tue, May 17, 2011 at 5:43 AM, Pandu Poluan <pandu@poluan.info> wrote:
> On 2011-05-17, Neil Bothwick <neil@digimed.co.uk> wrote:
>> On Tue, 17 May 2011 01:33:39 +0200, Alan McKinnon wrote:
>>
>>> grep "GET /Tmp/Linux/G" | /var/log/apache2/access_log | grep-v <myip> |
>>> \ awk '{print $1}' | sort | uniq | wc
>>>
>>> In true grand Unix tradition you cannot get quicker, dirtier or more
>>> effective than that
>>>
>>
>> awk does pattern matching, o you can ditch the grep stage and use
>>
>>  awk '! /myip/ {print $1}'
>>
>> You could use awk to search for the GET patterns too, not only saving yet
>> another process, but making sure that no one else, including you next
>> month, can work out what the command is supposed to do.
>>
>
> Meh, me forgetting what an awk snippet do? Never!
>
> sed ... now that's a wholly different story :-P
>
>> sort -u would save having a separate process for uniq, but I've no idea
>> if it's faster. It's only worth using sort -u if you would use uniq with
>> no arguments.
>>
>
> And you can actually do the 'uniq' or '-u' function within awk. Quite
> easily, in fact.
>
> Here's a sample of awk doing uniq:
>
> awk '!x[$1]++ { print $1 }'
>
> Benefit? It doesn't care if the non-unique lines are one-after-another
> or spread all over the text. The above snippet prints only the first
> occurence. Combine that with a test for match:
>
> awk '!x[$1]++ && $0 ~ /awesome_regex_pattern/ {print $1}'
>
> then with a test for negated match
>
> awk '!x[$1]++ && $0 ~ /awesome_regex_pattern/ && $0 !~
> /more_awesome_regex/ {print $1}'
>
> Rgds,
> --
> Pandu E Poluan - IT Optimizer
> My website: http://pandu.poluan.info/
>
>

I have always wondered if there is a way to do awk '{ print $1}' using
only builtin bash functions when you only have a one line string

Re: [gentoo-user] is a nice "place" :-D

[Alex Schuster]

Juan Diego Tascón writes:

> I have always wondered if there is a way to do awk '{ print $1}' using
> only builtin bash functions when you only have a one line string

str="one two five"

# remove all from the first blank on, but will not work with
# other whitespace
echo ${str%% *}

or

# set $1, $2, $3, ... to words of $str
set $str
echo $1

or

# create array holding one word per element
strarr=( $str )
echo $strarr  (or echo ${strarr[0]})

	Wonko

Re: [gentoo-user] is a nice "place" :-D

[Juan Diego Tascón]

On Tue, May 17, 2011 at 8:36 AM, Alex Schuster <wonko@wonkology.org> wrote:
> Juan Diego Tascón writes:
>
>> I have always wondered if there is a way to do awk '{ print $1}' using
>> only builtin bash functions when you only have a one line string
>
> str="one two five"
>
> # remove all from the first blank on, but will not work with
> # other whitespace
> echo ${str%% *}
>
> or
>
> # set $1, $2, $3, ... to words of $str
> set $str
> echo $1
>
> or
>
> # create array holding one word per element
> strarr=( $str )
> echo $strarr  (or echo ${strarr[0]})
>
>        Wonko
>
>

thanks for the info

Re: [gentoo-user] is a nice "place" :-D

[Pandu Poluan]

On 2011-05-17, Alex Schuster <wonko@wonkology.org> wrote:
> Juan Diego Tascón writes:
>
>> I have always wondered if there is a way to do awk '{ print $1}' using
>> only builtin bash functions when you only have a one line string
>
> str="one two five"
>
> # remove all from the first blank on, but will not work with
> # other whitespace
> echo ${str%% *}
>
> or
>
> # set $1, $2, $3, ... to words of $str
> set $str
> echo $1
>
> or
>
> # create array holding one word per element
> strarr=( $str )
> echo $strarr  (or echo ${strarr[0]})
>
> 	Wonko
>
>

How about this:

str="one two three"
read word etc <<< "$str"
echo $word

(not tested, though)

--
Pandu E Poluan - IT Optimizer
My website: http://pandu.poluan.info/

Re: [gentoo-user] is a nice "place" :-D

[Stroller]

On 17/5/2011, at 11:43am, Pandu Poluan wrote:
> On 2011-05-17, Neil Bothwick <neil@digimed.co.uk> wrote:
>> On Tue, 17 May 2011 01:33:39 +0200, Alan McKinnon wrote:
>> 
>>> grep "GET /Tmp/Linux/G" | /var/log/apache2/access_log | grep-v <myip> |
>>> \ awk '{print $1}' | sort | uniq | wc
>>> 
>> ...
>> awk does pattern matching, o you can ditch the grep stage and use
>> 
>> awk '! /myip/ {print $1}'
>> ...
> 
> Meh, me forgetting what an awk snippet do? Never!
> 
> sed ... now that's a wholly different story :-P

Not addressed at you, specifically, but it rather seems like sed & awk are much under-appreciated these days. I'd guess that this may be due to the changing nature of *nix users, but they seem to have "gone out of fashion". Aside from sed's simple replace, I have certainly never learned to do anything useful with them.

Stroller.

Re: [gentoo-user] is a nice "place" :-D

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 788 bytes --]

On Tue, 17 May 2011 18:38:33 +0100, Stroller wrote:

> Not addressed at you, specifically, but it rather seems like sed & awk
> are much under-appreciated these days. I'd guess that this may be due
> to the changing nature of *nix users, but they seem to have "gone out
> of fashion". Aside from sed's simple replace, I have certainly never
> learned to do anything useful with them.

They both have a steep initial learning curve, which leads to their
adoption being put off. I put awk in the same category as screen, one of
those programs that you hear people going on about for years, but always
manage to put off trying them. Once you do try them, you use them for
everything but slicing bread.


-- 
Neil Bothwick

Hors d'oeuvres: 3 sandwiches cut into 40 pieces.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] is a nice "place" :-D

[Alan McKinnon]

Apparently, though unproven, at 12:17 on Wednesday 18 May 2011, Neil Bothwick 
did opine thusly:

> On Tue, 17 May 2011 18:38:33 +0100, Stroller wrote:
> > Not addressed at you, specifically, but it rather seems like sed & awk
> > are much under-appreciated these days. I'd guess that this may be due
> > to the changing nature of *nix users, but they seem to have "gone out
> > of fashion". Aside from sed's simple replace, I have certainly never
> > learned to do anything useful with them.
> 
> They both have a steep initial learning curve, which leads to their
> adoption being put off. I put awk in the same category as screen, one of
> those programs that you hear people going on about for years, but always
> manage to put off trying them. Once you do try them, you use them for
> everything but slicing bread.


Add bash to that list.

Have you read the full man page for the bloody thing?

No wonder most folk stop at launching it after login



-- 
alan dot mckinnon at gmail dot com

Re: [gentoo-user] is a nice "place" :-D

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 757 bytes --]

On Wed, 18 May 2011 21:03:47 +0200, Alan McKinnon wrote:

> > They both have a steep initial learning curve, which leads to their
> > adoption being put off. I put awk in the same category as screen, one
> > of those programs that you hear people going on about for years, but
> > always manage to put off trying them. Once you do try them, you use
> > them for everything but slicing bread.  
 
> Add bash to that list.
> 
> Have you read the full man page for the bloody thing?

It's better than the zsh man page, which is split into several pages.
That sounds like a good idea, and it would have been if there had been
some sort of index so you knew which page to look on.


-- 
Neil Bothwick

Top Oxymorons Number 46: Found missing

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] is a nice "place" :-D

[Alan Mackenzie]

Hi, Alan.

On Wed, May 18, 2011 at 09:03:47PM +0200, Alan McKinnon wrote:
> Apparently, though unproven, at 12:17 on Wednesday 18 May 2011, Neil Bothwick 
> did opine thusly:

> > On Tue, 17 May 2011 18:38:33 +0100, Stroller wrote:
> > > Not addressed at you, specifically, but it rather seems like sed &
> > > awk are much under-appreciated these days. I'd guess that this may
> > > be due to the changing nature of *nix users, but they seem to have
> > > "gone out of fashion". Aside from sed's simple replace, I have
> > > certainly never learned to do anything useful with them.

> > They both have a steep initial learning curve, which leads to their
> > adoption being put off. I put awk in the same category as screen, one
> > of those programs that you hear people going on about for years, but
> > always manage to put off trying them. Once you do try them, you use
> > them for everything but slicing bread.


> Add bash to that list.

> Have you read the full man page for the bloody thing?

You're not meant to read that man page through from beginning to end.
Anybody who could learn bash that way would be superhuman.
Unfortunately, the info pages for bash are not well organised.  So
beginners have to learn from books, many of which are not good.

And bash is about the most disorganised, arbitrary language around, full
of crazy little quirks and odd sytaxes.  And I love it.  ;-)

> No wonder most folk stop at launching it after login

> -- 
> alan dot mckinnon at gmail dot com

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: [gentoo-user] is a nice "place" :-D

[Alan McKinnon]

Apparently, though unproven, at 22:15 on Wednesday 18 May 2011, Alan Mackenzie 
did opine thusly:

> Hi, Alan.
> 
> On Wed, May 18, 2011 at 09:03:47PM +0200, Alan McKinnon wrote:
> > Apparently, though unproven, at 12:17 on Wednesday 18 May 2011, Neil
> > Bothwick
> > 
> > did opine thusly:
> > > On Tue, 17 May 2011 18:38:33 +0100, Stroller wrote:
> > > > Not addressed at you, specifically, but it rather seems like sed &
> > > > awk are much under-appreciated these days. I'd guess that this may
> > > > be due to the changing nature of *nix users, but they seem to have
> > > > "gone out of fashion". Aside from sed's simple replace, I have
> > > > certainly never learned to do anything useful with them.
> > > 
> > > They both have a steep initial learning curve, which leads to their
> > > adoption being put off. I put awk in the same category as screen, one
> > > of those programs that you hear people going on about for years, but
> > > always manage to put off trying them. Once you do try them, you use
> > > them for everything but slicing bread.
> > 
> > Add bash to that list.
> > 
> > Have you read the full man page for the bloody thing?
> 
> You're not meant to read that man page through from beginning to end.

Um, I did ....

> Anybody who could learn bash that way would be superhuman.

I doubt I learned much though. I even took the effort to reformat it as an OOo 
doc so I could find stuff and give it to others.

It was an interesting exercise, not necessary an interesting *learning* 
exercise


> Unfortunately, the info pages for bash are not well organised.  So
> beginners have to learn from books, many of which are not good.
> 
> And bash is about the most disorganised, arbitrary language around, full
> of crazy little quirks and odd sytaxes.  And I love it.  ;-)

The difference between bash and perl?

Perl was inspired by a linguist, who at least puts his foot down at the truly 
crazy suggestions. Bash has no such thing.


-- 
alan dot mckinnon at gmail dot com

Re: [gentoo-user] is a nice "place" :-D

[Walter Dnes]

On Wed, May 18, 2011 at 10:28:55PM +0200, Alan McKinnon wrote

> The difference between bash and perl?
> 
> Perl was inspired by a linguist, who at least puts his foot down at
> the truly crazy suggestions. Bash has no such thing.

  Perl is a bloated mediocre operating system, complete with repo (CPAN).
But it lacks a lightweight extraction/reporting language.  BTW, the most
powerful little-known gem in bash is that you can open filehandles to do
sequential reads and writes, just like in BASIC. For examples, see...
http://blogmag.net/blog/read/48/File_descriptors_in_shell

-- 
Walter Dnes <waltdnes@waltdnes.org>

Re: [gentoo-user] is a nice "place" :-D

[David Haller]

Hello,

On Tue, 17 May 2011, Alan McKinnon wrote:
>grep "GET /Tmp/Linux/G" | /var/log/apache2/access_log | grep-v <myip> | \
>awk '{print $1}' | sort | uniq | wc

useless use of ...

awk '/GET \/Tmp\/Linux\/G/{ips[$1]++;}END{print length(ips);}' \
    /var/log/apache2/access_log

I add each access to ips[<IP>] in case you'd want to print that to,
e.g. by using

END {
    for( i in ips ) {
        print i ":" ips[i] " accesses";
    }
    print length(ips) " unique IPs total";
}

as the "END" block.

HTH,
-dnh

-- 
Any research done on how to efficiently use computers has been long lost
in the mad rush to upgrade systems to do things that aren't needed by
people who don't understand what they are really supposed to do with
them.                                             -- Graham Reed, in asr