[gentoo-user] Curious pattern in log files from ssh...

[gentoo-user] Curious pattern in log files from ssh...

[Steve]

I've recently discovered a curious pattern emerging in my system log
with failed login attempts via ssh.

Previously, I noticed dictionary attacks launched - which were easy to
detect... and I've a process to block the IP address of any host that
repeatedly fails to authenticate.

What I see now is quite different... I'm seeing a dictionary attack
originating from a wide range of IP addresses - testing user-names in
sequence... it has been in progress since 22nd November 2008 and has
tried 7195 user names in alphabetical order from 521 distinct hosts -
with no successive two attempts from the same host.

I'm not particularly concerned - since I'm confident that all my users
have strong passwords... but it strikes me that this data identifies a
bot-net that is clearly malicious attempting to break passwords.

Sure, I could use IPtables to block all these bad ports... or... I could
disable password authentication entirely... but I keep thinking that
there has to be something better I can do... any suggestions?  Is there
a simple way to integrate a block-list of known-compromised hosts into
IPtables - rather like my postfix is configured to drop connections from
known spam sources from the sbl-xbl.spamhaus.org DNS block list, for
example.

Break in attempts today (attempted username/IP address):
--
huck 190.60.41.82
huckleberry 81.196.122.2
huckleberry 58.39.145.213
huckleberry 60.230.184.143
hue 58.196.4.2
hue 83.228.92.228
huela 193.41.235.225
huela 193.41.235.225
huey 201.21.216.198
huey 81.149.101.27
hugh 200.123.174.145
hugh 83.228.92.228
hugh 212.46.24.146
hugo 195.234.169.138
hugo 193.86.111.6
hugo 201.224.199.201
hume 69.217.30.214
hume 80.118.132.88
hummer 71.166.159.177
hummer 200.126.119.91
hummer 61.4.210.33
humphrey 80.34.55.88
humphrey 213.163.19.158
humvee 85.222.53.48
humvee 80.24.4.23
hung 61.47.31.130
hung 70.46.140.187
hunter 67.40.86.204
hunter 83.228.92.228
hunter 200.60.156.90
huong 207.250.220.196
huong 125.63.77.3
huong 200.62.142.212
huslu 219.93.187.38
huslu 121.223.228.249
huslu 200.29.135.50
hussein 200.60.156.90
hussein 200.6.220.46
hussein 125.63.77.3
huy 60.191.111.234
huy 200.79.25.39
huyen 213.136.105.130
huyen 190.144.61.58
huyen 121.33.199.37
hy 121.33.199.37
hy 90.190.96.46
hyacinth 81.196.122.2
hyacinth 189.43.21.244
hyacinth 99.242.205.242
hyman 201.21.216.198
hypatia 218.28.143.246
hypatia 195.234.169.138
iain 200.118.119.48
iain 124.42.124.87
iain 194.224.118.61
ian 189.56.92.42
ian 201.28.119.60
ian 210.187.18.199
ianna 211.154.254.120
ianna 84.242.66.10
ianna 193.41.235.225
ianthe 81.246.26.179
ibtesam 87.30.163.87
ichabod 201.251.61.108
ida 62.61.141.93
ida 80.24.4.23
idalee 85.222.53.48
idalee 190.144.61.58
--

[gentoo-user] Re: Curious pattern in log files from ssh...

[Nikos Chantziaras]

Steve wrote:
> [...]
> Sure, I could use IPtables to block all these bad ports... or... I could
> disable password authentication entirely... but I keep thinking that
> there has to be something better I can do... any suggestions?

I'm using DenyHosts to battle this.  It adds the IPs to /etc/hosts.deny 
after a configurable amount of failed logins.  It even downloads an 
online list of IPs where attacks originate from and uploads attacks to 
your box to this list too (if you allow it in the configuration).

After I installed this, no more brute-forcing :)  I used to have 
thousands per day.

http://www.denyhosts.net

It's in portage.

Re: [gentoo-user] Re: Curious pattern in log files from ssh...

[Paul Hartman]

On Wed, Dec 3, 2008 at 2:16 PM, Nikos Chantziaras <realnc@arcor.de> wrote:
> Steve wrote:
>>
>> [...]
>> Sure, I could use IPtables to block all these bad ports... or... I could
>> disable password authentication entirely... but I keep thinking that
>> there has to be something better I can do... any suggestions?
>
> I'm using DenyHosts to battle this.  It adds the IPs to /etc/hosts.deny
> after a configurable amount of failed logins.  It even downloads an online
> list of IPs where attacks originate from and uploads attacks to your box to
> this list too (if you allow it in the configuration).
>
> After I installed this, no more brute-forcing :)  I used to have thousands
> per day.
>
> http://www.denyhosts.net
>
> It's in portage.

The big botnet attacks are doing no more than 2 login attempts per IP,
making stuff like denyhosts hard to use (unless you set it to ban
after 1 login attempt, but that'll catch real users who make a typo)

[gentoo-user] Re: Curious pattern in log files from ssh...

[Nikos Chantziaras]

Paul Hartman wrote:
> On Wed, Dec 3, 2008 at 2:16 PM, Nikos Chantziaras <realnc@arcor.de> wrote:
>> Steve wrote:
>>> [...]
>>> Sure, I could use IPtables to block all these bad ports... or... I could
>>> disable password authentication entirely... but I keep thinking that
>>> there has to be something better I can do... any suggestions?
>> I'm using DenyHosts to battle this.  It adds the IPs to /etc/hosts.deny
>> after a configurable amount of failed logins.  It even downloads an online
>> list of IPs where attacks originate from and uploads attacks to your box to
>> this list too (if you allow it in the configuration).
>>
>> After I installed this, no more brute-forcing :)  I used to have thousands
>> per day.
>>
>> http://www.denyhosts.net
>>
>> It's in portage.
> 
> The big botnet attacks are doing no more than 2 login attempts per IP,
> making stuff like denyhosts hard to use (unless you set it to ban
> after 1 login attempt, but that'll catch real users who make a typo)

In that case, changing the SSH port to something unlikely (I use one 
above 30000) should be sufficient :P

Re: [gentoo-user] Curious pattern in log files from ssh...

[Albert Hopkins]

You could try sshguard or denyhosts.

Re: [gentoo-user] Curious pattern in log files from ssh...

[Paul Hartman]

On Wed, Dec 3, 2008 at 2:02 PM, Steve <Gentoo_sjh@shic.co.uk> wrote:
> I've recently discovered a curious pattern emerging in my system log
> with failed login attempts via ssh.
>
> Previously, I noticed dictionary attacks launched - which were easy to
> detect... and I've a process to block the IP address of any host that
> repeatedly fails to authenticate.
>
> What I see now is quite different... I'm seeing a dictionary attack
> originating from a wide range of IP addresses - testing user-names in
> sequence... it has been in progress since 22nd November 2008 and has
> tried 7195 user names in alphabetical order from 521 distinct hosts -
> with no successive two attempts from the same host.

This has been going on all year, you're lucky if you just started
getting it. :)

AFAIK nobody has found any specific fingerprint or anything to block
it by. The "solution" seems to be: only allow SSH from specific IP
addresses, don't use port 22, don't use password auth, use some kind
of portknocking, etc. as you already alluded to. If you Google for
distributed ssh brute force attacks, there are some fairly detailed
articles out there from earlier in the year.

Good luck :)

Paul

Re: [gentoo-user] Curious pattern in log files from ssh...

[Simon]

I noticed the same thing on my host several weeks ago.

I strongly suggest removing root access to your ssh, root is probably being 
tried by more than 50% of all login attempts...  the other trials are 
semi-intelligent random usernames (ie, users that might really well exists, like 
'apache' etc... but other usernames which may not like 'albert').
If your username is not part of the list of attempts, then it won't be tried 
much, and I once found out that if your password is alphanumeric with lower and 
upper cases, the hacker as a worst chance of finding your password in 
(26*2+10)^8(chars long) = 62^8 = 2.18e14 steps or 218 millions of millions of 
steps.  This is assuming they try the correct username each time!

The other thing you should do is place ssh on another port, very high.  IIRC, 
port numbers are 16bits and can go as high as 65k... you could use 22xxx where 
xxx is a random favorite number for example.

Since it is very unlikely that the attacker is targeting you specifically, 
changing the port number (and removing root access) will very likely stop the 
attack forever.  Though, if the attacker did target you, then you will need some 
more security tools (intrusion detection, etc...).

Good luck!
Simon

Steve wrote:
> I've recently discovered a curious pattern emerging in my system log
> with failed login attempts via ssh.
> 
> Previously, I noticed dictionary attacks launched - which were easy to
> detect... and I've a process to block the IP address of any host that
> repeatedly fails to authenticate.
> 
> What I see now is quite different... I'm seeing a dictionary attack
> originating from a wide range of IP addresses - testing user-names in
> sequence... it has been in progress since 22nd November 2008 and has
> tried 7195 user names in alphabetical order from 521 distinct hosts -
> with no successive two attempts from the same host.
> 
> I'm not particularly concerned - since I'm confident that all my users
> have strong passwords... but it strikes me that this data identifies a
> bot-net that is clearly malicious attempting to break passwords.
> 
> Sure, I could use IPtables to block all these bad ports... or... I could
> disable password authentication entirely... but I keep thinking that
> there has to be something better I can do... any suggestions?  Is there
> a simple way to integrate a block-list of known-compromised hosts into
> IPtables - rather like my postfix is configured to drop connections from
> known spam sources from the sbl-xbl.spamhaus.org DNS block list, for
> example.
> 
> Break in attempts today (attempted username/IP address):
> --
> huck 190.60.41.82
> huckleberry 81.196.122.2
> huckleberry 58.39.145.213
> huckleberry 60.230.184.143
> hue 58.196.4.2
> hue 83.228.92.228
> huela 193.41.235.225
> huela 193.41.235.225
> huey 201.21.216.198
> huey 81.149.101.27
> hugh 200.123.174.145
> hugh 83.228.92.228
> hugh 212.46.24.146
> hugo 195.234.169.138
> hugo 193.86.111.6
> hugo 201.224.199.201
> hume 69.217.30.214
> hume 80.118.132.88
> hummer 71.166.159.177
> hummer 200.126.119.91
> hummer 61.4.210.33
> humphrey 80.34.55.88
> humphrey 213.163.19.158
> humvee 85.222.53.48
> humvee 80.24.4.23
> hung 61.47.31.130
> hung 70.46.140.187
> hunter 67.40.86.204
> hunter 83.228.92.228
> hunter 200.60.156.90
> huong 207.250.220.196
> huong 125.63.77.3
> huong 200.62.142.212
> huslu 219.93.187.38
> huslu 121.223.228.249
> huslu 200.29.135.50
> hussein 200.60.156.90
> hussein 200.6.220.46
> hussein 125.63.77.3
> huy 60.191.111.234
> huy 200.79.25.39
> huyen 213.136.105.130
> huyen 190.144.61.58
> huyen 121.33.199.37
> hy 121.33.199.37
> hy 90.190.96.46
> hyacinth 81.196.122.2
> hyacinth 189.43.21.244
> hyacinth 99.242.205.242
> hyman 201.21.216.198
> hypatia 218.28.143.246
> hypatia 195.234.169.138
> iain 200.118.119.48
> iain 124.42.124.87
> iain 194.224.118.61
> ian 189.56.92.42
> ian 201.28.119.60
> ian 210.187.18.199
> ianna 211.154.254.120
> ianna 84.242.66.10
> ianna 193.41.235.225
> ianthe 81.246.26.179
> ibtesam 87.30.163.87
> ichabod 201.251.61.108
> ida 62.61.141.93
> ida 80.24.4.23
> idalee 85.222.53.48
> idalee 190.144.61.58
> --
> 
> 
> 

Re: [gentoo-user] Curious pattern in log files from ssh...

[Steve]

Simon wrote:
> Since it is very unlikely that the attacker is targeting you
> specifically, changing the port number (and removing root access) will
> very likely stop the attack forever.  Though, if the attacker did
> target you, then you will need some more security tools (intrusion
> detection, etc...).

I recognise that this doesn't seem to be a targeted attack - but it is
still frustrating to find that someone has evaded my IP blocking
strategy... even though they pose only a slightly elevated risk by
having done so.  (Of course, I don't permit root login - that would be
madness... and, as far as I'm aware, no-one has guessed even a valid
user name... they're all obscure!)

The thing that strikes me is that, in evading my blocking strategy, they
clearly identified a bot-net of compromised hosts.  With this in mind,
ideally, I'd like to:

1. Automatically detect and block all future attacks on all ports from
all hosts which are involved in this coordinated attack.  These hosts
can't be trusted not to be malicious.
2. Somehow inform the administrator of the hosts attacking me (in a
respectful way) since, I presume, they are unaware that their host is
involved in the attack.
3. Ideally, share this kind of information so that myself and others are
better protected from bot-net attacks in future.

It's the sort of thing I imagine has already been done - and there's no
point in re-inventing the wheel.

Re: [gentoo-user] Curious pattern in log files from ssh...

[Mick]

[-- Attachment #1: Type: text/plain, Size: 2205 bytes --]

On Thursday 04 December 2008, Steve wrote:
> Simon wrote:
> > Since it is very unlikely that the attacker is targeting you
> > specifically, changing the port number (and removing root access) will
> > very likely stop the attack forever.  Though, if the attacker did
> > target you, then you will need some more security tools (intrusion
> > detection, etc...).
>
> I recognise that this doesn't seem to be a targeted attack - but it is
> still frustrating to find that someone has evaded my IP blocking
> strategy... even though they pose only a slightly elevated risk by
> having done so.  (Of course, I don't permit root login - that would be
> madness... and, as far as I'm aware, no-one has guessed even a valid
> user name... they're all obscure!)
>
> The thing that strikes me is that, in evading my blocking strategy, they
> clearly identified a bot-net of compromised hosts.  With this in mind,
> ideally, I'd like to:
>
> 1. Automatically detect and block all future attacks on all ports from
> all hosts which are involved in this coordinated attack.  These hosts
> can't be trusted not to be malicious.
> 2. Somehow inform the administrator of the hosts attacking me (in a
> respectful way) since, I presume, they are unaware that their host is
> involved in the attack.
> 3. Ideally, share this kind of information so that myself and others are
> better protected from bot-net attacks in future.
>
> It's the sort of thing I imagine has already been done - and there's no
> point in re-inventing the wheel.

I recall something similar whereby the attacked machines would automatically 
launch an attack on the botnet/spammer to effect a DoS.  Then the spammers 
complained and the guys who had written the software were forced by the 
police to recall it . . . sometimes I wonder.  Anyway, I'm a bit thin on 
details - this was all the rage about 4-5 years ago as a legit way to defend 
yourself against spam.

What I think is required is a script which will identify the compromised 
machine and promptly reformat its MSWindows OS - problem solved.  Of course 
how you keep tabs on this tool not being misused is another thing.
-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-user] Curious pattern in log files from ssh...

[Alan McKinnon]

On Wednesday 03 December 2008 22:02:43 Steve wrote:
> I've recently discovered a curious pattern emerging in my system log
> with failed login attempts via ssh.
>
> Previously, I noticed dictionary attacks launched - which were easy to
> detect... and I've a process to block the IP address of any host that
> repeatedly fails to authenticate.
>
> What I see now is quite different... I'm seeing a dictionary attack
> originating from a wide range of IP addresses - testing user-names in
> sequence... it has been in progress since 22nd November 2008 and has
> tried 7195 user names in alphabetical order from 521 distinct hosts -
> with no successive two attempts from the same host.

Slashdot yesterday, read the front page

It seems to be a co-ordinated and very well synchronized stealth bot-net. You 
are one of many that has noticed this. I am noticing scans on machines that 
have never been scanned before in all the time they have been up.

You should indeed be very concerned and take extra special due care with your 
security arrangements currently. In fact, if you admin machines that are in 
any way critical, you really *really* should be undertaking a thorough 
security audit and make very sure you have done everything and covered all 
your bases.



-- 
alan dot mckinnon at gmail dot com

Re: [gentoo-user] Curious pattern in log files from ssh...

[Dmitry S. Makovey]

[-- Attachment #1: Type: text/plain, Size: 926 bytes --]

On December 3, 2008, Steve wrote:
> Sure, I could use IPtables to block all these bad ports... or... I could
> disable password authentication entirely... but I keep thinking that
> there has to be something better I can do... any suggestions?  Is there
> a simple way to integrate a block-list of known-compromised hosts into
> IPtables - rather like my postfix is configured to drop connections from
> known spam sources from the sbl-xbl.spamhaus.org DNS block list, for
> example.

I went the path of paswordless entries (i.e. DSA/RSA keys) and I think it 
helped a lot, no botnet/worm/cracker is known to do selective key assembly so 
far and it's a labour-intensive process. I think applying keys is a very good 
step forward (well, and make sure every externally exposed service is 
properly patched and secured ;) ).

-- 
Dmitry Makovey
Web Systems Administrator
Athabasca University
(780) 675-6245

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-user] Curious pattern in log files from ssh...

[Steve]

Thanks for all the replies so far... I'll reply once to these... (Oh,
and when I said "ports" in my original post, I meant "addresses" - my
typing fingers just ignored my brain...)

I'm against a 'novel port' approach - as I am against port-knocking (for
my server) because these may prove challenging for the environments from
which I may want to log on.  I want to retain a 'standard' service to
make it easiest for me to connect to my server from a remote site
without requiring reconfiguration of firewalls etc.

I have, in the past, used DSA only keys - but this was frustrating on
several occasions when I wanted access to my server and didn't have my
SSH keys available to me... I almost always connect using a key pair
rather than a password - but the password option is very useful to allow
me to get hold of my SSH keys in the first place in some environments. 
If I found a distributed attack on a valid user name, for example, I'd
consider this a critical change - however inconvenient.

I previously used denyhosts - but (I can't remember why) it became
preferable to block with IPtables rather than with tcpwrappers... which
prompted me to dump it in favour of a bespoke script based upon
blacklist.py (http://blinkeye.ch/mediawiki/index.php/SSH_Blocking) -
though, now, I'm tempted by the more professional looking sshguard -
thanks for the tip.  Of course, this doesn't really address the problem
I posted about - because I'm now faced with a highly distributed
dictionary attack...

It strikes me that, given the conclusive nature of this attack (which,
by virtue of the fact that the usernames are attempted in alphabetical
order proves it to be a single coordinated attack) I can create a list
of a large number of IP addresses - which, likely, correspond to
compromised hosts.  It strikes me that this would be a perfect source of
information to set up a block list... and, if others' logs show similar
attacks, it should be easy enough to combine this data to provide
distributed protection from a distributed attack.  I don't think for one
second that this attack is targeted - neither my hardware or the
information on my server is particularly interesting to anyone but me. 
It would be extremely interesting to me, however, if it were to
transpire that my IP address originated login attempts such as these -
as this would clearly demonstrate it to be compromised...  I suspect,
too, the ISPs should be interested to inform their subscribers in the
interest of security... though, of course, I recognise that this is
being optimistic.

When I exposed my server to internet SSH logins, I carefully considered
security... though I also had to consider convenience - since that was
the only reason for doing so in the first place.  If I could block all
IPs suspected of being in a bot-net - then this would be an improvement
in security without a great cost in terms of lost convenience.  Right
now, in the context of this attack which circumvents my earlier blocking
strategy, I'm looking for a viable blacklist solution in order to avoid
white-listing.  A potential solution for me would be to have sshd be far
more choosy about source IPs when using password authentication... for
example, restricting it to hosts in the UK... but still allowing remote
access wherever I've propagated DSA keys... but I think this would be
tricky to set up.  A shared block-list, I suspect, would be the most
effective response to this attack... and the response most likely to
minimise others' exposure, too.

Steve

Re: [gentoo-user] Curious pattern in log files from ssh...

[Dmitry S. Makovey]

[-- Attachment #1: Type: text/plain, Size: 2291 bytes --]

On December 3, 2008, Steve wrote:
> I have, in the past, used DSA only keys - but this was frustrating on
> several occasions when I wanted access to my server and didn't have my
> SSH keys available to me... I almost always connect using a key pair
> rather than a password - but the password option is very useful to allow
> me to get hold of my SSH keys in the first place in some environments.
> If I found a distributed attack on a valid user name, for example, I'd
> consider this a critical change - however inconvenient.

get yourself some portable linux device capable of either USB, ethernet or 
wifi connection (OpenMoko, Nokia NXXX, etc.) plug your keys there - and 
voila, you've got yourelf both secure terminal and key storage in one box. I 
would be highly suspicious initiating SSH connection with my servers from 
untrusted box (which is any box not built and maintained by me ;) ) as there 
is a chance of keylogger (no matter how friendly owner of spoken box is - you 
don't know if he wasn't hacked and you have no time for even casual 
checking).

You can use variation of port-knocking and reverse your strategy based on the 
pattern:

1. drop first connection from specified IP and record it in "first_try" table
2. drop second connection from specified IP and record it in "second_try" 
table
3. if IP is in both first_try and second_try - allow it to attempt 
authentication but only with the keys. (removing it from *_try tables and 
possibly recording it in whitelist)
4. if IP fails X number of attempts within specified timeframe - remove from 
whitelist and record in blacklist

bit tricky logic, but fairly simple to implement (I use *BSD PF so no ready 
recipe for iptables here ;) ).

bit paranoid, but it covers your initial concern with distributed attack and 
single-attempts. You can further collect older entries from first_try into 
blacklist and do whatever you please with them. 

You can also collect high-frequency attempts into blacklist and have very big 
blacklist you can sell off on eBay :)

P.S.
I actually don't do any of the above. It was just a surge of creative paranoia 
in response to initial request :)

-- 
Dmitry Makovey
Web Systems Administrator
Athabasca University
(780) 675-6245

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-user] Curious pattern in log files from ssh...

[Steve]

Dmitry S. Makovey wrote:
> P.S. I actually don't do any of the above. It was just a surge of creative paranoia 
> in response to initial request :)
All good ideas - except selling the blacklist... I'd be happiest to
share my blacklist for free... my objective is to minimise exposure to
botnets - rather than to accept another level of complexity with
legitimate use.

Re: [gentoo-user] Curious pattern in log files from ssh...

[Paul Hartman]

On Wed, Dec 3, 2008 at 4:55 PM, Steve <Gentoo_sjh@shic.co.uk> wrote:
> Dmitry S. Makovey wrote:
>> P.S. I actually don't do any of the above. It was just a surge of creative paranoia
>> in response to initial request :)
> All good ideas - except selling the blacklist... I'd be happiest to
> share my blacklist for free... my objective is to minimise exposure to
> botnets - rather than to accept another level of complexity with
> legitimate use.

I think using Dmitry's idea of rejecting the first 2 connections, but
then allowing it as normal on the third attempt would satisfy your
requirements for being on the normal port, allowing all IPs and
requiring no special setup on the client end (other than knowing they
have to to retry twice).

Of course, this is assuming the botnet stops after rejected connections...

Re: [gentoo-user] Curious pattern in log files from ssh...

[Dmitry S. Makovey]

[-- Attachment #1: Type: text/plain, Size: 357 bytes --]

On December 3, 2008, Paul Hartman wrote:
> Of course, this is assuming the botnet stops after rejected connections...

oh no, not rejected - dropped ;) let them go through pains of timing out 
without knowing if anything is actually listening on the other side ;)

-- 
Dmitry Makovey
Web Systems Administrator
Athabasca University
(780) 675-6245

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-user] Curious pattern in log files from ssh...

[Steve]

Paul Hartman wrote:
> I think using Dmitry's idea of rejecting the first 2 connections, but
> then allowing it as normal on the third attempt would satisfy your
> requirements for being on the normal port, allowing all IPs and
> requiring no special setup on the client end (other than knowing they
> have to to retry twice).
>   
Erm - surely I either need to set up my client to port-knock... which is
a faff I'd rather avoid... in order to use the technique.  Port knocking
would be especially infuriating from trusted clients where I'd like to
use standard software like WinSCP; Putty; Symbian Putty - etc.

While I recognise port knocking as a valuable strategy in some
circumstances, it seems a very bad fit for my needs.

GEO-IP blocking would be fairly good... if I could limit this to
password authentication only - as would blacklisting known bot-net
participants.

While these exotic ideas are interesting - a better way to identify
malicious hosts is, by far, my preferred solution.

Re: [gentoo-user] Curious pattern in log files from ssh...

[Dmitry S. Makovey]

[-- Attachment #1: Type: text/plain, Size: 1001 bytes --]

On December 3, 2008, Steve wrote:
> Paul Hartman wrote:
> > I think using Dmitry's idea of rejecting the first 2 connections, but
> > then allowing it as normal on the third attempt would satisfy your
> > requirements for being on the normal port, allowing all IPs and
> > requiring no special setup on the client end (other than knowing they
> > have to to retry twice).
>
> Erm - surely I either need to set up my client to port-knock... which is
> a faff I'd rather avoid... in order to use the technique.  

nope. just start connection. wait a minute. cancel. start another one. wait a 
minute. cancel. start new one - voila! :)

> While I recognise port knocking as a valuable strategy in some
> circumstances, it seems a very bad fit for my needs.

well. Nobody but you knows your requiremens and specifics - we're just listing 
options. It's up to you to either take 'em or leave 'em ;)

-- 
Dmitry Makovey
Web Systems Administrator
Athabasca University
(780) 675-6245

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-user] Curious pattern in log files from ssh...

[Steve]

Dmitry S. Makovey wrote:
>> Erm - surely I either need to set up my client to port-knock... which
>> is a faff I'd rather avoid... in order to use the technique.
> nope. just start connection. wait a minute. cancel. start another one. wait a 
> minute. cancel. start new one - voila! :)
>   
Eeew... especially as this would apply to all connections - even the
ones where I have a DSA key.  I might be able to cope with this if it
only applied to my initial connection, from which I could grab a copy of
the DSA key.
> well. Nobody but you knows your requiremens and specifics - we're just listing 
> options. It's up to you to either take 'em or leave 'em ;)
Fair enough - but I've still not found an option for sharing/using
shared block lists for bot-nets.

Re: [gentoo-user] Curious pattern in log files from ssh...

[Dmitry S. Makovey]

[-- Attachment #1: Type: text/plain, Size: 1610 bytes --]

On December 3, 2008, Steve wrote:
> Dmitry S. Makovey wrote:
> >> Erm - surely I either need to set up my client to port-knock... which
> >> is a faff I'd rather avoid... in order to use the technique.
> >
> > nope. just start connection. wait a minute. cancel. start another one.
> > wait a minute. cancel. start new one - voila! :)
>
> Eeew... especially as this would apply to all connections - even the
> ones where I have a DSA key.  I might be able to cope with this if it
> only applied to my initial connection, from which I could grab a copy of
> the DSA key.

Ok, let's theoreticise some more. My paranoia feels particularly frisky today, 
so here it is:
remember, I've mentioned origianlly that once you authenticate successfully 
once with DSA key - your IP is whitelisted. So subsequent connections go 
right through.

> > well. Nobody but you knows your requiremens and specifics - we're just
> > listing options. It's up to you to either take 'em or leave 'em ;)
>
> Fair enough - but I've still not found an option for sharing/using
> shared block lists for bot-nets.

Open a Wiki page on Wikipedia, update it every so often and provide simple 
parser for it so others can recycle same IPs. Since it's a Wiki page - others 
can update it as well (including botnet owners, but then they'd have to 
reveal themselves - tricky situation) :)

P.S.
I think I'd better stop with my mad science projects here before I go too far 
and invent brand new theory on host protection ;)

-- 
Dmitry Makovey
Web Systems Administrator
Athabasca University
(780) 675-6245

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

RE: [gentoo-user] Curious pattern in log files from ssh...

[Adam Carter]

> Open a Wiki page on Wikipedia, update it every so often and
> provide simple
> parser for it so others can recycle same IPs. Since it's a
> Wiki page - others
> can update it as well (including botnet owners, but then
> they'd have to reveal themselves - tricky situation) :)

Reveal themselves in what way? If you're taking about source IP, they can just use one of their bots to make the page update...

Re: [gentoo-user] Curious pattern in log files from ssh...

[Dmitry S. Makovey]

[-- Attachment #1: Type: text/plain, Size: 571 bytes --]

On December 4, 2008, Adam Carter wrote:
> > Open a Wiki page on Wikipedia, update it every so often and
> > provide simple
> > parser for it so others can recycle same IPs. Since it's a
> > Wiki page - others
> > can update it as well (including botnet owners, but then
> > they'd have to reveal themselves - tricky situation) :)
>
> Reveal themselves in what way? If you're taking about source IP, they can
> just use one of their bots to make the page update...

true.

-- 
Dmitry Makovey
Web Systems Administrator
Athabasca University
(780) 675-6245

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-user] Curious pattern in log files from ssh...

[Shawn Haggett]

Dmitry S. Makovey wrote:
> On December 3, 2008, Steve wrote:
>> Dmitry S. Makovey wrote:
>>> well. Nobody but you knows your requiremens and specifics - we're just
>>> listing options. It's up to you to either take 'em or leave 'em ;)
>> Fair enough - but I've still not found an option for sharing/using
>> shared block lists for bot-nets.
> 
> Open a Wiki page on Wikipedia, update it every so often and provide simple 
> parser for it so others can recycle same IPs. Since it's a Wiki page - others 
> can update it as well (including botnet owners, but then they'd have to 
> reveal themselves - tricky situation) :)

I hear the botnet owners have 1 or 2 spare machine scattered around the world they can proxy through... :)

Shawn

RE: [gentoo-user] Curious pattern in log files from ssh...

[Adam Carter]

> I previously used denyhosts - but (I can't remember why) it became
> preferable to block with IPtables rather than with
> tcpwrappers... which
> prompted me to dump it in favour of a bespoke script based upon
> blacklist.py (http://blinkeye.ch/mediawiki/index.php/SSH_Blocking) -
> though, now, I'm tempted by the more professional looking sshguard -
> thanks for the tip.  Of course, this doesn't really address
> the problem
> I posted about - because I'm now faced with a highly distributed
> dictionary attack...

Fail2ban is iptables based. From the website it now appears to have a map feature so if say you notice most of the attacks coming from China, and none of you ssh useres are in China, you could perhaps block the entire country with http://people.netfilter.org/~peejix/geoip/howto/geoip-HOWTO.html

Re: [gentoo-user] Curious pattern in log files from ssh...

[Evgeniy Bushkov]

[-- Attachment #1: Type: text/plain, Size: 2528 bytes --]

Steve пишет:
> I've recently discovered a curious pattern emerging in my system log
> with failed login attempts via ssh.
>
> Previously, I noticed dictionary attacks launched - which were easy to
> detect... and I've a process to block the IP address of any host that
> repeatedly fails to authenticate.
>
> What I see now is quite different... I'm seeing a dictionary attack
> originating from a wide range of IP addresses - testing user-names in
> sequence... it has been in progress since 22nd November 2008 and has
> tried 7195 user names in alphabetical order from 521 distinct hosts -
> with no successive two attempts from the same host.
>
> I'm not particularly concerned - since I'm confident that all my users
> have strong passwords... but it strikes me that this data identifies a
> bot-net that is clearly malicious attempting to break passwords.
>
> Sure, I could use IPtables to block all these bad ports... or... I could
> disable password authentication entirely... but I keep thinking that
> there has to be something better I can do... any suggestions?  Is there
> a simple way to integrate a block-list of known-compromised hosts into
> IPtables - rather like my postfix is configured to drop connections from
> known spam sources from the sbl-xbl.spamhaus.org DNS block list, for
> example.
>
> Break in attempts today (attempted username/IP address):
> --
>
>   
Hi.
Best of all you can add iptables rules. It's better then use any script. 
Also take a note that there are no "known-compromised hosts" because ANY 
IP can be forged.
I've sometimes seen such rules in the internet. These I use in my firewall:

iptables -A INPUT -i eth0 -p tcp -m state --state NEW --dport 22 -m 
recent --name sshattack --set
iptables -A INPUT -i eth0 -p tcp -m state --state NEW --dport 22 -m 
recent --name sshattack --rcheck --seconds 60 --hitcount 4 -j LOG -m 
limit --limit 3/minute --limit-burst 3 --log-level 4 --log-prefix 'SSH 
REJECT: '
iptables -A INPUT -i eth0 -p tcp -m state --state NEW --dport 22 -m 
recent --name sshattack --rcheck --seconds 60 --hitcount 4 -j REJECT 
--reject-with tcp-reset

These rules give you possibility to use your ssh service from any IP but 
reject repeated login attempts from malicious users. You can tune 
parameter --hitcount to limit amount of
sequential login attempts per minute. Also you can look at 
/proc/net/ipt_recent/sshattack for malicious IPs and how often they were 
used.

Best regards,
Evgeniy B.



[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/x-pkcs7-signature, Size: 5581 bytes --]

RE: [gentoo-user] Curious pattern in log files from ssh...

[Adam Carter]

> Also take a note that there are no "known-compromised hosts"

What about hosts listed in RBLs? http://en.wikipedia.org/wiki/Comparison_of_DNS_blacklists. It would be interesting to see if how much correlation there is between ssh brute forcing bots and the contents of the various lists.

> because ANY IP can be forged.

Its easy enough to forge a SYN, but to setup a session so you can make a password guessing attempt requires that you also get the packets back from the server, which is an order of magnitude more difficult. Ever since OSes have implemented well chosen initial sequence numbers, spoofing of TCP sessions has become very difficult.

RE: [gentoo-user] Curious pattern in log files from ssh...

[Adam Carter]

> > Also take a note that there are no "known-compromised hosts"
>
> What about hosts listed in RBLs?
> http://en.wikipedia.org/wiki/Comparison_of_DNS_blacklists. It
> would be interesting to see if how much correlation there is
> between ssh brute forcing bots and the contents of the various lists.

Maybe http://wiki.duskglow.com/tiki-index.php?page=Packetbl "PacketBL is a program that uses DNS blocklists to determine whether to accept or reject packets"

Used with dnsbl.ahbl.org "Aggregate zone, contains UCE/bulk email senders, open proxies, open relays, trojaned/infected machines, comment/trackback spammers"

would be a good solution.

Re: [gentoo-user] Curious pattern in log files from ssh...

[Evgeniy Bushkov]

[-- Attachment #1: Type: text/plain, Size: 1204 bytes --]

Adam Carter пишет:
>> Also take a note that there are no "known-compromised hosts"
>>     
>
> What about hosts listed in RBLs? http://en.wikipedia.org/wiki/Comparison_of_DNS_blacklists. It would be interesting to see if how much correlation there is between ssh brute forcing bots and the contents of the various lists.
>   
It's just interesting. But I don't trust them enough. I don't know how 
these lists were composed. We've periodically seen viruses outbreaks, 
some computers IPs could get into lists because of trojans and so on. 
One day you won't reach your server from your own home computer...
>   
>> because ANY IP can be forged.
>>     
>
> Its easy enough to forge a SYN, but to setup a session so you can make a password guessing attempt requires that you also get the packets back from the server, which is an order of magnitude more difficult. Ever since OSes have implemented well chosen initial sequence numbers, spoofing of TCP sessions has become very difficult.
>
>   
I agree but as admin I prefer to think about many things worse than they 
really are. If something wrong is possible it's better to avoid it 
beforehand.

Best regards,
Evgeniy B.


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/x-pkcs7-signature, Size: 5581 bytes --]

Re: [gentoo-user] Curious pattern in log files from ssh...

[Joshua Murphy]

On Fri, Dec 5, 2008 at 10:05 AM, Evgeniy Bushkov <zhen@dotcomltd.ru> wrote:
> Adam Carter пишет:
>>>
>>> Also take a note that there are no "known-compromised hosts"
>>>
>>
>> What about hosts listed in RBLs?
>> http://en.wikipedia.org/wiki/Comparison_of_DNS_blacklists. It would be
>> interesting to see if how much correlation there is between ssh brute
>> forcing bots and the contents of the various lists.
>>
>
> It's just interesting. But I don't trust them enough. I don't know how these
> lists were composed. We've periodically seen viruses outbreaks, some
> computers IPs could get into lists because of trojans and so on. One day you
> won't reach your server from your own home computer...

The fact that a lot of 'compromised hosts' are home users with
providers like comcast, verizon, etc lends another trouble as well...
dynamic IPs mean that the next person with the luck of the draw in
getting that IP can't reach your servers either, and if *you* happen
to be that person, no reasonable whitelist will ever get you back in
from that location until you get another IP.

>>
>>
>>>
>>> because ANY IP can be forged.
>>>
>>
>> Its easy enough to forge a SYN, but to setup a session so you can make a
>> password guessing attempt requires that you also get the packets back from
>> the server, which is an order of magnitude more difficult. Ever since OSes
>> have implemented well chosen initial sequence numbers, spoofing of TCP
>> sessions has become very difficult.
>>
>>
>
> I agree but as admin I prefer to think about many things worse than they
> really are. If something wrong is possible it's better to avoid it
> beforehand.
>
> Best regards,
> Evgeniy B.

Careful with that line of thinking... you'll inevitably come to the
conclusion that there's no hope and you're better off just turning the
system off, unplugging it from the wall, and locking it into a very
sturdy vault deep beneath a very solid mountain! (until you ponder
yourself insane over the security risks that exist even then, let
alone the impact on usability)

-- 
Poison [BLX]
Joshua M. Murphy

Re: [gentoo-user] Curious pattern in log files from ssh...

[Christian Franke]

[-- Attachment #1: Type: text/plain, Size: 1391 bytes --]

On 12/03/2008 09:02 PM, Steve wrote:
> I've recently discovered a curious pattern emerging in my system log
> with failed login attempts via ssh.
> 
> I'm not particularly concerned - since I'm confident that all my users
> have strong passwords... but it strikes me that this data identifies a
> bot-net that is clearly malicious attempting to break passwords.
> 
> Sure, I could use IPtables to block all these bad ports... or... I could
> disable password authentication entirely... but I keep thinking that
> there has to be something better I can do... any suggestions?  Is there
> a simple way to integrate a block-list of known-compromised hosts into
> IPtables - rather like my postfix is configured to drop connections from
> known spam sources from the sbl-xbl.spamhaus.org DNS block list, for
> example.

I just don't see what blocking ssh-bruteforce attempts should be good
for, at least on a server where few _users_ are active.

The chance that security of a well configured system will be compromised
by that is next to zero, and on recent systems it is also impossible to
cause significant load with ssh-login-attempts.

Also, things like fail2ban add new attack-possibilities to a system, I
remember the old DoS for fail2ban, resulting from a wrong regex in log
file parsing, but I think at least this is fixed now.

Regards,
Christian Franke


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: [gentoo-user] Curious pattern in log files from ssh...

[Dmitry S. Makovey]

[-- Attachment #1: Type: text/plain, Size: 1149 bytes --]

On December 4, 2008, Christian Franke wrote:
> I just don't see what blocking ssh-bruteforce attempts should be good
> for, at least on a server where few _users_ are active.

Considering how much creative paranoia I've exposed in this thread it might 
come as a surprise, but I do agree with the above statement. Strong passwords 
(or key-only authentication) would prevent brute-force attacks from being 
successfull. The only thing that is semi-usefull side-effect is that you can 
identify compromised machines and deny ANY type of traffic from them 
preventing possible DoS launched against you. But then IPs are so easy to 
spoof :) Balance is what makes sysadmin comfortable enough and doesn't 
compromise usability of the server, so everybody decides for themselves. OP 
obviously wants that "extra" layer of protection and notification so with a 
bit of creativity and some external tools it's possible to achieve. As long 
as he doesn't forget about other aspects of security - he should do just fine 
with all those extra measures :)

-- 
Dmitry Makovey
Web Systems Administrator
Athabasca University
(780) 675-6245

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-user] Curious pattern in log files from ssh...

[Alan McKinnon]

On Thursday 04 December 2008 21:03:17 Christian Franke wrote:
> On 12/03/2008 09:02 PM, Steve wrote:
> > I've recently discovered a curious pattern emerging in my system log
> > with failed login attempts via ssh.
> >
> > I'm not particularly concerned - since I'm confident that all my users
> > have strong passwords... but it strikes me that this data identifies a
> > bot-net that is clearly malicious attempting to break passwords.
> >
> > Sure, I could use IPtables to block all these bad ports... or... I could
> > disable password authentication entirely... but I keep thinking that
> > there has to be something better I can do... any suggestions?  Is there
> > a simple way to integrate a block-list of known-compromised hosts into
> > IPtables - rather like my postfix is configured to drop connections from
> > known spam sources from the sbl-xbl.spamhaus.org DNS block list, for
> > example.
>
> I just don't see what blocking ssh-bruteforce attempts should be good
> for, at least on a server where few _users_ are active.

Two reasons:

a. Maybe, just maybe, you overlooked something. Belts, braces and a drawstring 
for good measure is not a bad thing.

b. You probably want to get all that crap out of your log files off into some 
other place where you can cope with it. Parsing auth log files that are 95% 
brute force attempts is no fun. I like to have the crap in place A and the 
real stuff in place B, makes my job so much easier
>
> The chance that security of a well configured system will be compromised
> by that is next to zero, and on recent systems it is also impossible to
> cause significant load with ssh-login-attempts.

Uh-huh. We all said that for many years. Then some bright spark actually 
looked at the patches the debian openssh maintainer was applying and we all 
had one of those special "oops..." moments

Did you have any idea of just how weak certs made on a debian box were before 
it hit the headlines? No-one I know did.

> Also, things like fail2ban add new attack-possibilities to a system, I
> remember the old DoS for fail2ban, resulting from a wrong regex in log
> file parsing, but I think at least this is fixed now.

Whereas that is true enough in itself, the actual risk of such is rather low 
in comparison to the gains. Hence it is not a valid reason to not use 
fail2ban and such-like apps.

If it were, we should all just stop using iptables and libwrap and openssl on 
the off-chance that maybe, just maybe, they open an attack vector. But that's 
silly reasoning right?


-- 
alan dot mckinnon at gmail dot com

Re: [gentoo-user] Curious pattern in log files from ssh...

[Steve]

Alan McKinnon wrote:
> On Thursday 04 December 2008 21:03:17 Christian Franke wrote:
>   
>> I just don't see what blocking ssh-bruteforce attempts should be good
>> for, at least on a server where few _users_ are active.
>>     
> Two reasons:
>
> a. Maybe, just maybe, you overlooked something. Belts, braces and a drawstring 
> for good measure is not a bad thing.
>
> b. You probably want to get all that crap out of your log files off into some 
> other place where you can cope with it. Parsing auth log files that are 95% 
> brute force attempts is no fun. I like to have the crap in place A and the 
> real stuff in place B, makes my job so much easier
>   
I agree 100% with the above - another issue is that I'd like to block
all traffic from malicious hosts - I realise that the traffic is low at
the moment, but that need not be the case in future.
>> Also, things like fail2ban add new attack-possibilities to a system, I
>> remember the old DoS for fail2ban, resulting from a wrong regex in log
>> file parsing, but I think at least this is fixed now.
>>     
> Whereas that is true enough in itself, the actual risk of such is rather low 
> in comparison to the gains. Hence it is not a valid reason to not use 
> fail2ban and such-like apps.
The issue for me is that the cost of a DOS is far, far lower than the
cost of a break-in.  The cost of a DOS that prevents access from new
hosts is orders of magnitude lower than the cost of a DOS.  Everyone's
risk profiles are different - but, for me, keeping out intruders is
critical (they may result in unrecoverable data loss) and my
accessibility objective is that it be the 'norm' that I can log in with
an unusual-username and complex password from a trustworthy PC whose IP
address can not be determined in advance... using only bog-standard
tools and no non-remembered personal data.

I'm coming around to the idea of port-knocking, but my gut instinct is
that it is a bit baroque and has potential for me to louse-up its
implementation... It definitely adversely affects usability - though, I
admit, less than I first suspected.  I'm still quite interested in the
idea of identifying botnets where used to subvert the tactics used by
fail2ban; blacklist.py, etc. and using these to, in turn, block access
to any service... including, for example, hosted web-services which are,
potentially, in spite of taking all the obvious precautions, more
vulnerable to attack - IMHO.

I'm definitely thinking that it would be a good idea if there were a way
to publish botnet lists... such that they could be collated and turned
into a DNSBL style resource.  If such a resource existed, I'd definitely
chose to use it (overridden by a few whitelist entries of my own -
just-in-case...) and I'd be very happy to report back to it in order to
help keeping this problem under control.  Incidentally, I'd also
consider it useful to monitor this block list for any occurrence of my
own IP address - since that would be an early indication that one of my
hosts may be compromised.