[gentoo-user] Hacked by association?

[gentoo-user] Hacked by association?

[Grant]

Last night my host sent out a message that their database had been
compromised.  I contacted them this morning and it turns out that all
of their trouble tickets were exposed.  I checked my records and
(stupidly) I had included my root password in an email to them about a
year ago.  I (stupidly) hadn't changed the password since.  I've
changed it now and rebooted the system, but what do you think?  Do I
need to start this thing over?

- Grant
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Hacked by association?

[Dan Farrell]

On Wed, 19 Sep 2007 11:09:30 -0700
Grant <emailgrant@gmail.com> wrote:

> Last night my host sent out a message that their database had been
> compromised.  I contacted them this morning and it turns out that all
> of their trouble tickets were exposed.  I checked my records and
> (stupidly) I had included my root password in an email to them about a
> year ago.  I (stupidly) hadn't changed the password since.  I've
> changed it now and rebooted the system, but what do you think?  Do I
> need to start this thing over?
> 
> - Grant

I think you should take a look at the programs that
are running, and netstat -l, and see if anything is fishy. 
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Hacked by association?

[Grant]

> > Last night my host sent out a message that their database had been
> > compromised.  I contacted them this morning and it turns out that all
> > of their trouble tickets were exposed.  I checked my records and
> > (stupidly) I had included my root password in an email to them about a
> > year ago.  I (stupidly) hadn't changed the password since.  I've
> > changed it now and rebooted the system, but what do you think?  Do I
> > need to start this thing over?
> >
> > - Grant
>
> I think you should take a look at the programs that
> are running, and netstat -l, and see if anything is fishy.

I recognize everything in 'ps -ef' I think, but I've never really used
netstat before.  Under "Active Internet connections" I don't
recognize:

tcp localhost:10030
tcp *:snpp

I don't recognize most of the paths under UNIX domain sockets.
Anything particular I should look for?

- Grant
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Hacked by association?

[Ryan Sims]

On 9/19/07, Grant <emailgrant@gmail.com> wrote:
> > > Last night my host sent out a message that their database had been
> > > compromised.  I contacted them this morning and it turns out that all
> > > of their trouble tickets were exposed.  I checked my records and
> > > (stupidly) I had included my root password in an email to them about a
> > > year ago.  I (stupidly) hadn't changed the password since.  I've
> > > changed it now and rebooted the system, but what do you think?  Do I
> > > need to start this thing over?
> > >
> > > - Grant
> >
> > I think you should take a look at the programs that
> > are running, and netstat -l, and see if anything is fishy.
>
> I recognize everything in 'ps -ef' I think, but I've never really used
> netstat before.  Under "Active Internet connections" I don't
> recognize:
>
> tcp localhost:10030
> tcp *:snpp
>
> I don't recognize most of the paths under UNIX domain sockets.
> Anything particular I should look for?

Try using the -p option to netstat to get the PID of those two
connections, see if its anything suspicious


-- 
Ryan W Sims
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Hacked by association?

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 757 bytes --]

On Wed, 19 Sep 2007 11:09:30 -0700, Grant wrote:

> Last night my host sent out a message that their database had been
> compromised.  I contacted them this morning and it turns out that all
> of their trouble tickets were exposed.  I checked my records and
> (stupidly) I had included my root password in an email to them about a
> year ago.  I (stupidly) hadn't changed the password since.  I've
> changed it now and rebooted the system, but what do you think?  Do I
> need to start this thing over?

equery check sys-process/procps
equery check sys-apps/coreutils

Make sure that none of the executable files have changed.

Also, emerge and run app-forensics/rkhunter


-- 
Neil Bothwick

Top Oxymorons Number 37: Sanitary landfill

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-user] Hacked by association?

[Mick]

[-- Attachment #1: Type: text/plain, Size: 1089 bytes --]

On Wednesday 19 September 2007, Grant wrote:

> I recognize everything in 'ps -ef' I think, but I've never really used
> netstat before.  Under "Active Internet connections" I don't
> recognize:
>
> tcp localhost:10030
> tcp *:snpp

Hmm, are you running postfix on this server (just a suspicion).

Also, snpp is for pagers: 
http://en.wikipedia.org/wiki/Simple_Network_Paging_Protocol

Run # netstat -anop which will show you the process owner.  Hopefully, if 
there is something running it will show up (clever scripts can mask 
themselves from netstat, ps auxf, etc.).

Then run lsof (check man lsof) to see if there is anything suspicious there, 
like another user logged in either as root or with a different name.  

Finally, ask your ISP to boot off a LiveCD and scan the machine with rkhunter 
and chrootkit.

Depending on how many thousands of tickets the database had the crackers may 
or may have not found out about your root passwd.  On the other hand, if you 
can't sleep at nights it is better to format and reinstall.

HTH.
-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-user] Hacked by association?

[Daniel da Veiga]

On 9/19/07, Neil Bothwick <neil@digimed.co.uk> wrote:
> On Wed, 19 Sep 2007 11:09:30 -0700, Grant wrote:
>
> > Last night my host sent out a message that their database had been
> > compromised.  I contacted them this morning and it turns out that all
> > of their trouble tickets were exposed.  I checked my records and
> > (stupidly) I had included my root password in an email to them about a
> > year ago.  I (stupidly) hadn't changed the password since.  I've
> > changed it now and rebooted the system, but what do you think?  Do I
> > need to start this thing over?
>
> equery check sys-process/procps
> equery check sys-apps/coreutils
>
> Make sure that none of the executable files have changed.
>
> Also, emerge and run app-forensics/rkhunter
>

I'm not a security expert, not even near. But, if I was in a possible
vulnerable position like a leaked root password, wouldn't an "emerge
-ef world" and a posterior offline "emerge -e world" replace any
possible binary changed by an intruder? That would minimize the risk,
and allied with rkhunter and other forensic tools and password change
could make you pretty sure that your environment is safe afain...

Just a thought...
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Hacked by association?

[Volker Armin Hemmann]

On Mittwoch, 19. September 2007, Grant wrote:

>   Do I
> need to start this thing over?

yes. No tool can tell you for certain, that no malware is rampage on your 
system. netstat, ps, emerge might be hacked already. As might be md5sum and 
other tools to generate and compare ckecksums. There is only one way to make 
sure your system is clean:

reinstallation

-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Hacked by association?

[Grant]

> > I recognize everything in 'ps -ef' I think, but I've never really used
> > netstat before.  Under "Active Internet connections" I don't
> > recognize:
> >
> > tcp localhost:10030
> > tcp *:snpp
>
> Also, snpp is for pagers:
> http://en.wikipedia.org/wiki/Simple_Network_Paging_Protocol

With netstat -lp it looks like *:snpp is associated with apache2 and
is using the same pid as *:http and *:https.  I've never set up
anything having to do with a pager.  I've never had a pager.  What can
I do to investigate that further?

> Then run lsof (check man lsof) to see if there is anything suspicious there,
> like another user logged in either as root or with a different name.

Any handy lsof commands?

- Grant
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Hacked by association?

[Jerry McBride]

On Wednesday 19 September 2007 07:16:09 pm Grant wrote:
> > > I recognize everything in 'ps -ef' I think, but I've never really used
> > > netstat before.  Under "Active Internet connections" I don't
> > > recognize:
> > >
> > > tcp localhost:10030
> > > tcp *:snpp
> >
> > Also, snpp is for pagers:
> > http://en.wikipedia.org/wiki/Simple_Network_Paging_Protocol
>
> With netstat -lp it looks like *:snpp is associated with apache2 and
> is using the same pid as *:http and *:https.  I've never set up
> anything having to do with a pager.  I've never had a pager.  What can
> I do to investigate that further?
>
> > Then run lsof (check man lsof) to see if there is anything suspicious
> > there, like another user logged in either as root or with a different
> > name.
>
> Any handy lsof commands?
>

Not sure about lsof... but something I did was to boot from a rescue disk, 
mounting the suspected partition and piped the outout from tree to a text 
file... A glance through the text file showed a lot of stuff from alien 
sources, explainging where some storage space had disappeared. The fix in 
that situation was a simple reformat and better inchains rules. Yeah, 
ipchains... this was a few years back.

Good luck Grant.



-- 


From the Desk of: Jerome D. McBride
--
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Hacked by association?

[Grant]

> > Last night my host sent out a message that their database had been
> > compromised.  I contacted them this morning and it turns out that all
> > of their trouble tickets were exposed.  I checked my records and
> > (stupidly) I had included my root password in an email to them about a
> > year ago.  I (stupidly) hadn't changed the password since.  I've
> > changed it now and rebooted the system, but what do you think?  Do I
> > need to start this thing over?
>
> equery check sys-process/procps
> equery check sys-apps/coreutils

These check out.

> Make sure that none of the executable files have changed.
>
> Also, emerge and run app-forensics/rkhunter

chkrootkit reports no problems whatsoever which is actually kind of
weird as I remember some things being reported last time I ran it, but
I looked into them then and they weren't a problem.

rkhunter reports no problems but it says it couldn't determine the OS
so MD5 checks were skipped.

- Grant
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Hacked by association?

[Grant]

> > > I recognize everything in 'ps -ef' I think, but I've never really used
> > > netstat before.  Under "Active Internet connections" I don't
> > > recognize:
> > >
> > > tcp localhost:10030
> > > tcp *:snpp
> >
> > Also, snpp is for pagers:
> > http://en.wikipedia.org/wiki/Simple_Network_Paging_Protocol
>
> With netstat -lp it looks like *:snpp is associated with apache2 and
> is using the same pid as *:http and *:https.  I've never set up
> anything having to do with a pager.  I've never had a pager.  What can
> I do to investigate that further?

This snpp pager thing is the weirdest thing I've found.  It sounds
like the kind of thing I would know if I set up.  Someone has some
kind of pager alert installed on my system?

- Grant
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Hacked by association?

[Dan Farrell]

On Wed, 19 Sep 2007 18:47:37 -0700
Grant <emailgrant@gmail.com> wrote:

> > > > I recognize everything in 'ps -ef' I think, but I've never
> > > > really used netstat before.  Under "Active Internet
> > > > connections" I don't recognize:
> > > >
> > > > tcp localhost:10030
> > > > tcp *:snpp
> > >
> > > Also, snpp is for pagers:
> > > http://en.wikipedia.org/wiki/Simple_Network_Paging_Protocol
> >
> > With netstat -lp it looks like *:snpp is associated with apache2 and
> > is using the same pid as *:http and *:https.  I've never set up
> > anything having to do with a pager.  I've never had a pager.  What
> > can I do to investigate that further?
> 
> This snpp pager thing is the weirdest thing I've found.  It sounds
> like the kind of thing I would know if I set up.  Someone has some
> kind of pager alert installed on my system?
> 
> - Grant
http://www.qpage.org/rfc1861.html

Network Working Group
Request for Comments: 1861 
October 1995 

...
...1. Introduction With all due apologies to the Glenayre engineers (who
take offense at the term "nerd") beepers are as much a part of computer
nerdom as X- terminals--perhaps, unfortunately, more. The intent of
Simple Network Paging Protocol is to provide a standard whereby pages
can be delivered to individual paging terminals...

I thought that was amusing.  Now I think the question is, if apache is
really serving that, isn't something going to show up in the logs
maybe?  

and BTw, have you done an external portmap?
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Hacked by association?

[Mark]

On 20/09/2007, Grant <emailgrant@gmail.com> wrote:
> > equery check sys-process/procps
> > equery check sys-apps/coreutils
>
> These check out.

Chances are you are fine then.

> chkrootkit reports no problems whatsoever which is actually kind of
> weird as I remember some things being reported last time I ran it, but
> I looked into them then and they weren't a problem.

The last time? Be careful, chkrootkit/rkhunter should always be used
on the fly, leaving them on a system could allow them to be
compromised and therefore negate the checks they run.

> rkhunter reports no problems but it says it couldn't determine the OS
> so MD5 checks were skipped.

Which doesn't matter as you checked out with the equery.

One other thing to check is to look for additional user (or root /
toor) accounts. A cracker may well have added one to allow them access
after the fact.

Still I would be of the opinion that you are safe.

Thanks
Mark
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Hacked by association?

[Mick]

[-- Attachment #1: Type: text/plain, Size: 1404 bytes --]

On Thursday 20 September 2007, Grant wrote:
> > > I recognize everything in 'ps -ef' I think, but I've never really used
> > > netstat before.  Under "Active Internet connections" I don't
> > > recognize:
> > >
> > > tcp localhost:10030
> > > tcp *:snpp
> >
> > Also, snpp is for pagers:
> > http://en.wikipedia.org/wiki/Simple_Network_Paging_Protocol
>
> With netstat -lp it looks like *:snpp is associated with apache2 and
> is using the same pid as *:http and *:https.  I've never set up
> anything having to do with a pager.  I've never had a pager.  What can
> I do to investigate that further?

I assume then that this is spawned by apache, but don't know why apache would 
spawn something like this.  What happens if you shut apache down?  Is it 
still there?  You could post in apache M/Ls in case they know or have seen 
this before.

> > Then run lsof (check man lsof) to see if there is anything suspicious
> > there, like another user logged in either as root or with a different
> > name.
>
> Any handy lsof commands?

I am not good with regex so I would just run it plain and work tediously my 
way down the list, or start from the known suspects:  check the port that 
snpp is using as well as 10030, e.g.

# lsof -i @your_host_name.com:10030  (you can use the IP address here too)

# lsof -i @your_host_name.com:snpp

etc.

HTH.
-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-user] Hacked by association?

[Hans-Werner Hilse]

Hi,

On Wed, 19 Sep 2007 16:16:09 -0700 Grant <emailgrant@gmail.com> wrote:

> With netstat -lp it looks like *:snpp is associated with apache2 and
> is using the same pid as *:http and *:https.  I've never set up
> anything having to do with a pager.  I've never had a pager.  What can
> I do to investigate that further?

Do you by chance run a PHP debugger or similar stuff, i.e. some
specialized apache modules with other interfaces than HTTP(S)?

-hwh
-- 
gentoo-user@gentoo.org mailing list

[gentoo-user] Re: Hacked by association?

[Alexander Skwar]

· Grant <emailgrant@gmail.com>:


>> > tcp localhost:10030
>> > tcp *:snpp
[...]
> With netstat -lp it looks like *:snpp is associated with apache2 and
> is using the same pid as *:http and *:https.

If that's so, then there should be a Listen directive in
httpd.conf or one of the included files. Do a

        grep -r 444 /etc/apache2

444 is the number associated with snpp.

Alexander Skwar
-- 
Ever get the feeling that the world's on tape and one of the reels is missing?
                -- Rich Little


-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Re: Hacked by association?

[Grant]

> >> > tcp localhost:10030
> >> > tcp *:snpp
> [...]
> > With netstat -lp it looks like *:snpp is associated with apache2 and
> > is using the same pid as *:http and *:https.
>
> If that's so, then there should be a Listen directive in
> httpd.conf or one of the included files. Do a
>
>         grep -r 444 /etc/apache2
>
> 444 is the number associated with snpp.

And you solved it.  I use that port (443 + 1) for a second https
<Location />.  Thank you, I didn't know snpp used 444.

- Grant
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Hacked by association?

[Grant]

> >   Do I
> > need to start this thing over?
>
> yes. No tool can tell you for certain, that no malware is rampage on your
> system. netstat, ps, emerge might be hacked already. As might be md5sum and
> other tools to generate and compare ckecksums. There is only one way to make
> sure your system is clean:
>
> reinstallation

Although I haven't found any evidence of intrusion, I've been urged
off-list to reinstall and since I'm about 4 hours early to rise this
morning I think I better.

Can we go over a good plan for the transition?  My main concerns are
backing up the right files and a good remote installation procedure as
it's been years since I did that.  Thanks.

- Grant
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Hacked by association?

[Volker Armin Hemmann]

On Freitag, 21. September 2007, Grant wrote:
> > >   Do I
> > > need to start this thing over?
> >
> > yes. No tool can tell you for certain, that no malware is rampage on your
> > system. netstat, ps, emerge might be hacked already. As might be md5sum
> > and other tools to generate and compare ckecksums. There is only one way
> > to make sure your system is clean:
> >
> > reinstallation
>
> Although I haven't found any evidence of intrusion, I've been urged
> off-list to reinstall and since I'm about 4 hours early to rise this
> morning I think I better.

If your intruder has at least some skills and don't want to leave evidence 
behind, you have nearly zero chance to find any signs. That is the evil part 
about being 'maybe hacked'.
Even with the best tools you can only say 'the hacker must be good' and 
not 'there was no hacker'. 

>
> Can we go over a good plan for the transition?  My main concerns are
> backing up the right files and a good remote installation procedure as
> it's been years since I did that.  Thanks.

I would tar everything up and copy the files back you really want - after 
checking them. Stuff from /etc, like the files in /etc/conf.d, make.conf, the 
files in /etc/portage and other stuff you edited, the /home tree, your 
database and website files, if there are any. But don't copy anything back 
without having a look first. Your world-file might be helpfull to spare some 
time. /usr/portage stuff should be nuked completly - it is so easy to replace 
it is not worth the risk of a hacked ebuild ... 
Don't forget to mkfs the partitions first before you start reinstallation.
About remote installation: never done that, hopefully someone else on the list 
can help you with that.



-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Hacked by association?

[Volker Armin Hemmann]

On Samstag, 22. September 2007, Grant wrote:
> > >   Do I
> > > need to start this thing over?
> >
> > yes. No tool can tell you for certain, that no malware is rampage on your
> > system. netstat, ps, emerge might be hacked already. As might be md5sum
> > and other tools to generate and compare ckecksums. There is only one way
> > to make sure your system is clean:
> >
> > reinstallation
>
> I had another idea.  Would it work to monitor my machine's traffic
> from another machine on the network and determine if I've been hacked
> that way?  Any ssh traffic other than mine would be a giveaway.
>
> - Grant

and who says that the hacker uses ssh in the future? or connects to the box in 
the next couple of weeks?
-- 
gentoo-user@gentoo.org mailing list