[gentoo-user] chkrootkit LKM trojan ?

[gentoo-user] chkrootkit LKM trojan ?

[Dave S]

HI, I have a potential security problem ... 

and err its not on gentoo, its on ubuntu but I am not getting any response 
there & you guys are the most tech bunch I know  - Thought I would lay it on 
the table :)

I just had an email from chkrootkit last night -

---

The following suspicious files and directories were found:

You have     3 process hidden for readdir command
You have     3 process hidden for ps command
chkproc: Warning: Possible LKM Trojan installed

---

Running chkrootkit now and all is OK

root@dave-comp:~#
root@dave-comp:~# chkrootkit | grep chkproc
Checking `lkm'... chkproc: nothing detected
root@dave-comp:~#   

I have even 'sudo install --reinstall chkrootkit' in case its binarys have 
been modified (paranoid)

Running rkhunter shows no problems

I am at a bit off a loss and would value some advice / opinions. I can see two 
possibilities

(a) I have a trojan, seems unlikely I am behind a netgear router firewall NAT 
with no incoming ports open. Running nothing more than samba, ssh and unison 
on the local network though I have to admit I have not hardened my system.

(b) Its a false alarm - it is called by /etc/cron.daily so a lot of different 
scripts are called at the same time - though I have no idea what could have 
caused it.

Any help / advice greatfully received

Dave
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] chkrootkit LKM trojan ?

[Hemmann, Volker Armin]

On Sunday 16 July 2006 20:25, Dave S wrote:
> HI, I have a potential security problem ...
>
> and err its not on gentoo, its on ubuntu but I am not getting any response
> there & you guys are the most tech bunch I know  - Thought I would lay it
> on the table :)
>
> I just had an email from chkrootkit last night -
>
> ---
>
> The following suspicious files and directories were found:
>
> You have     3 process hidden for readdir command
> You have     3 process hidden for ps command
> chkproc: Warning: Possible LKM Trojan installed
>
> ---
>
> Running chkrootkit now and all is OK
>
> root@dave-comp:~#
> root@dave-comp:~# chkrootkit | grep chkproc
> Checking `lkm'... chkproc: nothing detected
> root@dave-comp:~#
>
> I have even 'sudo install --reinstall chkrootkit' in case its binarys have
> been modified (paranoid)

if you installed using the tools of the system, it could be worthless, because 
compromised. Boot from a cd and check from the cd.
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] chkrootkit LKM trojan ?

[Dave S]

On Sunday 16 July 2006 19:54, Hemmann, Volker Armin wrote:
> On Sunday 16 July 2006 20:25, Dave S wrote:
> > HI, I have a potential security problem ...
> >
> > and err its not on gentoo, its on ubuntu but I am not getting any
> > response there & you guys are the most tech bunch I know  - Thought I
> > would lay it on the table :)
> >
> > I just had an email from chkrootkit last night -
> >
> > ---
> >
> > The following suspicious files and directories were found:
> >
> > You have     3 process hidden for readdir command
> > You have     3 process hidden for ps command
> > chkproc: Warning: Possible LKM Trojan installed
> >
> > ---
> >
> > Running chkrootkit now and all is OK
> >
> > root@dave-comp:~#
> > root@dave-comp:~# chkrootkit | grep chkproc
> > Checking `lkm'... chkproc: nothing detected
> > root@dave-comp:~#
> >
> > I have even 'sudo install --reinstall chkrootkit' in case its binarys
> > have been modified (paranoid)
>
> if you installed using the tools of the system, it could be worthless,
> because compromised. Boot from a cd and check from the cd.

I understand. Booted from knoppix 5.0.1, executed a

'chroot /mnt/hda1 chkrootkit' and a 
'chroot /mnt/hda1 rkhunter -c' 

- both scans brought back nothing. From what I have read the chkrootkit & 
rkhunter binarys would have been from the CD and therefore untainted ? Am I 
correct ?

Are there any other checks I can do - re-installing the system is not my 
preferred option :)

Dave




-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] chkrootkit LKM trojan ?

[Hemmann, Volker Armin]

On Sunday 16 July 2006 21:54, Dave S wrote:
> On Sunday 16 July 2006 19:54, Hemmann, Volker Armin wrote:
> > On Sunday 16 July 2006 20:25, Dave S wrote:
> > > HI, I have a potential security problem ...
> > >
> > > and err its not on gentoo, its on ubuntu but I am not getting any
> > > response there & you guys are the most tech bunch I know  - Thought I
> > > would lay it on the table :)
> > >
> > > I just had an email from chkrootkit last night -
> > >
> > > ---
> > >
> > > The following suspicious files and directories were found:
> > >
> > > You have     3 process hidden for readdir command
> > > You have     3 process hidden for ps command
> > > chkproc: Warning: Possible LKM Trojan installed
> > >
> > > ---
> > >
> > > Running chkrootkit now and all is OK
> > >
> > > root@dave-comp:~#
> > > root@dave-comp:~# chkrootkit | grep chkproc
> > > Checking `lkm'... chkproc: nothing detected
> > > root@dave-comp:~#
> > >
> > > I have even 'sudo install --reinstall chkrootkit' in case its binarys
> > > have been modified (paranoid)
> >
> > if you installed using the tools of the system, it could be worthless,
> > because compromised. Boot from a cd and check from the cd.
>
> I understand. Booted from knoppix 5.0.1, executed a
>
> 'chroot /mnt/hda1 chkrootkit' and a
> 'chroot /mnt/hda1 rkhunter -c'
>
> - both scans brought back nothing. From what I have read the chkrootkit &
> rkhunter binarys would have been from the CD and therefore untainted ? Am I
> correct ?
>

no, if you chroot, the binaries from the chroot are used.

use chkrootkit without chrooting - best with full path (/usr/sbin/chkrootkit)
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] chkrootkit LKM trojan ?

[Hemmann, Volker Armin]

oh, and read this:
http://www.chkrootkit.org/faq/
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] chkrootkit LKM trojan ?

[Dave S]

On Sunday 16 July 2006 21:36, Hemmann, Volker Armin wrote:
> oh, and read this:
> http://www.chkrootkit.org/faq/

Interesting ...

How accurate is chkproc? 
 If you run chkproc on a server that runs lots of short time processes it 
could report some false positives. chkproc compares the ps output with 
the /proc contents. If processes are created/killed during this operation 
chkproc could point out these PIDs as suspicious.


"no, if you chroot, the binaries from the chroot are used.

use chkrootkit without chrooting - best with full path (/usr/sbin/chkrootkit)"

The problem is if I do not chroot chkrootkit will scan the knoppix CD - tried 
it :). It needs to access the live proc etc on a running system.

Dave
-- 
gentoo-user@gentoo.org mailing list

[gentoo-user] Re: chkrootkit LKM trojan ?

[dnlt0hn5ntzhbqkv51]

On Sun, 16 Jul 2006 15:54:18 -0400, Dave S <gentoo@pusspaws.net> wrote:

> On Sunday 16 July 2006 19:54, Hemmann, Volker Armin wrote:
>> On Sunday 16 July 2006 20:25, Dave S wrote:
>> > HI, I have a potential security problem ...
>> >
>> > and err its not on gentoo, its on ubuntu but I am not getting any
>> > response there & you guys are the most tech bunch I know  - Thought I
>> > would lay it on the table :)
>> >
>> > I just had an email from chkrootkit last night -
>> >
>> > ---
>> >
>> > The following suspicious files and directories were found:
>> >
>> > You have     3 process hidden for readdir command
>> > You have     3 process hidden for ps command
>> > chkproc: Warning: Possible LKM Trojan installed
>> >
>> > ---
>> >
>> > Running chkrootkit now and all is OK
>> >
>> > root@dave-comp:~#
>> > root@dave-comp:~# chkrootkit | grep chkproc
>> > Checking `lkm'... chkproc: nothing detected
>> > root@dave-comp:~#
>> >
>> > I have even 'sudo install --reinstall chkrootkit' in case its binarys
>> > have been modified (paranoid)
>>
>> if you installed using the tools of the system, it could be worthless,
>> because compromised. Boot from a cd and check from the cd.
>
> I understand. Booted from knoppix 5.0.1, executed a
>
> 'chroot /mnt/hda1 chkrootkit' and a
> 'chroot /mnt/hda1 rkhunter -c'
>
> - both scans brought back nothing. From what I have read the chkrootkit &
> rkhunter binarys would have been from the CD and therefore untainted ?  
> Am I
> correct ?
>
> Are there any other checks I can do - re-installing the system is not my
> preferred option :)
>
> Dave

I'm a newbie, so discount this appropriately.

1. IIUC, running rkhunter/chkrootkit from knoppix simply checks the  
knoppix cd.
2. You want second/third opinions. IIWU,
       i. I'd scan the box with a Trojan signature scanner - e.g. fprotect,  
AntiVir, etc.
       from Knoppix - first assuring that you have current signatures.
       ii. I'd reemerge/recompile the kernel WITHOUT modules or module  
support, and clear out your usr/lib/modules         (though IIUC, this
        can be foiled).
       iii. I'd try zeppoo.
3. Try to figure out how you got it. e.g. you installed software from an  
unreliable source; your privileges are screwed up; you have an unpatched  
server(s) running; etc.

Maybe.... you could find the both the vector and the lkm  -  but  
understanding that the only real solution to a
rootkit is restoring from a clean backup, or rebuilding :-(


-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] chkrootkit LKM trojan ?

[Benno Schulenberg]

Dave S wrote:
> On Sunday 16 July 2006 21:36, Hemmann, Volker Armin wrote:
> > "no, if you chroot, the binaries from the chroot are used.
>
> The problem is if I do not chroot chkrootkit will scan the
> knoppix CD - tried it :). It needs to access the live proc etc on
> a running system.

Use -r.  Even when you chroot, still the /proc of your Knoppix 
system will be used, as it's the only system running.

Benno
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] chkrootkit LKM trojan ?

[Jerry McBride]

On Sunday 16 July 2006 15:54, Dave S wrote:
> On Sunday 16 July 2006 19:54, Hemmann, Volker Armin wrote:
> > On Sunday 16 July 2006 20:25, Dave S wrote:
> > > HI, I have a potential security problem ...
> > >
> > > and err its not on gentoo, its on ubuntu but I am not getting any
> > > response there & you guys are the most tech bunch I know  - Thought I
> > > would lay it on the table :)
> > >
> > > I just had an email from chkrootkit last night -
> > >
> > > ---
> > >
> > > The following suspicious files and directories were found:
> > >
> > > You have     3 process hidden for readdir command
> > > You have     3 process hidden for ps command
> > > chkproc: Warning: Possible LKM Trojan installed
> > >
> > > ---
> > >
> > > Running chkrootkit now and all is OK
> > >
> > > root@dave-comp:~#
> > > root@dave-comp:~# chkrootkit | grep chkproc
> > > Checking `lkm'... chkproc: nothing detected
> > > root@dave-comp:~#
> > >
> > > I have even 'sudo install --reinstall chkrootkit' in case its binarys
> > > have been modified (paranoid)
> >
> > if you installed using the tools of the system, it could be worthless,
> > because compromised. Boot from a cd and check from the cd.
>
> I understand. Booted from knoppix 5.0.1, executed a
>
> 'chroot /mnt/hda1 chkrootkit' and a
> 'chroot /mnt/hda1 rkhunter -c'
>
> - both scans brought back nothing. From what I have read the chkrootkit &
> rkhunter binarys would have been from the CD and therefore untainted ? Am I
> correct ?
>
> Are there any other checks I can do - re-installing the system is not my
> preferred option :)
>
> Dave

Hi Dave,

Just went through the same scare with an OLD linux server a few weeks ago.

This "could" be a false positive...

What you should do is run chkrootkit with verbose option turned on. Take the 
pids it show you and compare them to what's listed in /proc. 

Each running process has a pid and it's listed under /proc. In each pid listed 
under proc there's a /exe link that gives you the path to the program owning 
the pid. There a /status file that will give you the name of the program. 
There's other info there also. If there's any discrepancies between what's 
list in /proc and what ps tells you, you've been infected with LKM for sure. 

Naturally, you have to be there when chkrootkit complains...

But don't stop here...

You can also try running rootkit-hunter and compare the output.

You can cp known good tools (in your case, ps) from a backup to your infected 
box and run it to get "true" information. 

I knew a co-worker that ran "tree" across a suspected infected box and found a 
number of  hidden directories on it. It was indeed infected.

Also, if this machine was running a firewall, look in the logs. If you've kept 
a running archive, hopefully spanning a week or two, you may be able to 
figure out when and where the attack came from.

Hope that helps.

Jerry



-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] chkrootkit LKM trojan ?

[Dave S]

On Sunday 16 July 2006 22:12, Benno Schulenberg wrote:
> Dave S wrote:
> > On Sunday 16 July 2006 21:36, Hemmann, Volker Armin wrote:
> > > "no, if you chroot, the binaries from the chroot are used.
> >
> > The problem is if I do not chroot chkrootkit will scan the
> > knoppix CD - tried it :). It needs to access the live proc etc on
> > a running system.
>
> Use -r.  Even when you chroot, still the /proc of your Knoppix
> system will be used, as it's the only system running.
>
> Benno

Did a chrootlit -r /mnt/hda1 ... all clear.

Looks good - can never be 100% sure of cource :)

Dave

-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Re: chkrootkit LKM trojan ?

[Dave S]

On Sunday 16 July 2006 21:52, dnlt0hn5ntzhbqkv51 wrote:
> On Sun, 16 Jul 2006 15:54:18 -0400, Dave S <gentoo@pusspaws.net> wrote:
> > On Sunday 16 July 2006 19:54, Hemmann, Volker Armin wrote:
> >> On Sunday 16 July 2006 20:25, Dave S wrote:
> >> > HI, I have a potential security problem ...
> >> >
> >> > and err its not on gentoo, its on ubuntu but I am not getting any
> >> > response there & you guys are the most tech bunch I know  - Thought I
> >> > would lay it on the table :)
> >> >
> >> > I just had an email from chkrootkit last night -
> >> >
> >> > ---
> >> >
> >> > The following suspicious files and directories were found:
> >> >
> >> > You have     3 process hidden for readdir command
> >> > You have     3 process hidden for ps command
> >> > chkproc: Warning: Possible LKM Trojan installed
> >> >
> >> > ---
> >> >
> >> > Running chkrootkit now and all is OK
> >> >
> >> > root@dave-comp:~#
> >> > root@dave-comp:~# chkrootkit | grep chkproc
> >> > Checking `lkm'... chkproc: nothing detected
> >> > root@dave-comp:~#
> >> >
> >> > I have even 'sudo install --reinstall chkrootkit' in case its binarys
> >> > have been modified (paranoid)
> >>
> >> if you installed using the tools of the system, it could be worthless,
> >> because compromised. Boot from a cd and check from the cd.
> >
> > I understand. Booted from knoppix 5.0.1, executed a
> >
> > 'chroot /mnt/hda1 chkrootkit' and a
> > 'chroot /mnt/hda1 rkhunter -c'
> >
> > - both scans brought back nothing. From what I have read the chkrootkit &
> > rkhunter binarys would have been from the CD and therefore untainted ?
> > Am I
> > correct ?
> >
> > Are there any other checks I can do - re-installing the system is not my
> > preferred option :)
> >
> > Dave
>
> I'm a newbie, so discount this appropriately.
>
> 1. IIUC, running rkhunter/chkrootkit from knoppix simply checks the
> knoppix cd.
> 2. You want second/third opinions. IIWU,
>        i. I'd scan the box with a Trojan signature scanner - e.g. fprotect,
> AntiVir, etc.
>        from Knoppix - first assuring that you have current signatures.
>        ii. I'd reemerge/recompile the kernel WITHOUT modules or module
> support, and clear out your usr/lib/modules         (though IIUC, this
>         can be foiled).
>        iii. I'd try zeppoo.
> 3. Try to figure out how you got it. e.g. you installed software from an
> unreliable source; your privileges are screwed up; you have an unpatched
> server(s) running; etc.

I am pretty picky about my software - have not messed with permissions & its a 
desktop machine not running any external services.

>
> Maybe.... you could find the both the vector and the lkm  -  but
> understanding that the only real solution to a
> rootkit is restoring from a clean backup, or rebuilding :-(

... gulp ... On digging around and listening to you guys I am going to go with 
a false +ve. My clue came when I discovered how chkrootkit detected the 
problem ...

How accurate is chkproc? 
 If you run chkproc on a server that runs lots of short time processes it 
could report some false positives. chkproc compares the ps output with 
the /proc contents. If processes are created/killed during this operation 
chkproc could point out these PIDs as suspicious.

That fits in with the fact that chkrootkit & rkhunter now report clean (& also 
fits in with someone tinkering from the inside !)

I will keep a slightly suspicious eye on the box from now on :)

Cheers

Dave 




-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] chkrootkit LKM trojan ?

[Dave S]

On Sunday 16 July 2006 22:25, Jerry McBride wrote:
> On Sunday 16 July 2006 15:54, Dave S wrote:
> > On Sunday 16 July 2006 19:54, Hemmann, Volker Armin wrote:
> > > On Sunday 16 July 2006 20:25, Dave S wrote:
> > > > HI, I have a potential security problem ...
> > > >
> > > > and err its not on gentoo, its on ubuntu but I am not getting any
> > > > response there & you guys are the most tech bunch I know  - Thought I
> > > > would lay it on the table :)
> > > >
> > > > I just had an email from chkrootkit last night -
> > > >
> > > > ---
> > > >
> > > > The following suspicious files and directories were found:
> > > >
> > > > You have     3 process hidden for readdir command
> > > > You have     3 process hidden for ps command
> > > > chkproc: Warning: Possible LKM Trojan installed
> > > >
> > > > ---
> > > >
> > > > Running chkrootkit now and all is OK
> > > >
> > > > root@dave-comp:~#
> > > > root@dave-comp:~# chkrootkit | grep chkproc
> > > > Checking `lkm'... chkproc: nothing detected
> > > > root@dave-comp:~#
> > > >
> > > > I have even 'sudo install --reinstall chkrootkit' in case its binarys
> > > > have been modified (paranoid)
> > >
> > > if you installed using the tools of the system, it could be worthless,
> > > because compromised. Boot from a cd and check from the cd.
> >
> > I understand. Booted from knoppix 5.0.1, executed a
> >
> > 'chroot /mnt/hda1 chkrootkit' and a
> > 'chroot /mnt/hda1 rkhunter -c'
> >
> > - both scans brought back nothing. From what I have read the chkrootkit &
> > rkhunter binarys would have been from the CD and therefore untainted ? Am
> > I correct ?
> >
> > Are there any other checks I can do - re-installing the system is not my
> > preferred option :)
> >
> > Dave
>
> Hi Dave,
>
> Just went through the same scare with an OLD linux server a few weeks ago.
>
> This "could" be a false positive...
>
> What you should do is run chkrootkit with verbose option turned on. Take
> the pids it show you and compare them to what's listed in /proc.
>
> Each running process has a pid and it's listed under /proc. In each pid
> listed under proc there's a /exe link that gives you the path to the
> program owning the pid. There a /status file that will give you the name of
> the program. There's other info there also. If there's any discrepancies
> between what's list in /proc and what ps tells you, you've been infected
> with LKM for sure.
>
> Naturally, you have to be there when chkrootkit complains...

Thats the problem it was an automated email at midnight - all looks OK now - 
apart from my paranoia that is ...

>
> But don't stop here...
>
> You can also try running rootkit-hunter and compare the output.

Done it - it reports clean
>
> You can cp known good tools (in your case, ps) from a backup to your
> infected box and run it to get "true" information.
>
> I knew a co-worker that ran "tree" across a suspected infected box and
> found a number of  hidden directories on it. It was indeed infected.

I will look into it.

>
> Also, if this machine was running a firewall, look in the logs. If you've
> kept a running archive, hopefully spanning a week or two, you may be able
> to figure out when and where the attack came from.

Netgear firewall ADSL NAT, tea machine, router - I will have a look in the 
logs for anything suspicious - good idea.

>
> Hope that helps.
>
> Jerry

Cheers

Dave
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Re: chkrootkit LKM trojan ?

[Hans-Werner Hilse]

Hi,

On Mon, 17 Jul 2006 19:36:30 +0100
Dave S <gentoo@pusspaws.net> wrote:

> How accurate is chkproc? 
>  If you run chkproc on a server that runs lots of short time processes it 
> could report some false positives. chkproc compares the ps output with 
> the /proc contents. If processes are created/killed during this operation 
> chkproc could point out these PIDs as suspicious.
> 
> That fits in with the fact that chkrootkit & rkhunter now report clean (& also 
> fits in with someone tinkering from the inside !)

The problem I see here is that you can't expect chkrootkit to find
something when scanning from a clean base (Live-CD) when the only hint
you had was an alert from chkproc. You probably would have gotten the
alert from chkrootkit in the first place. chkproc inspects the
currently running system (and the /proc for the currently running
kernel). I.e. if it has no signature for the rootkit itself, it can't
find it again from that "clean" kernel.

Do you have the possibility to monitor internet connections on an
intermediary gateway? I think monitoring it for a few days would give
you a better hint if there might be something active.

And there are other things to think about. Do you have a webserver
running? CGI scripts? PHP applications? Do you have other network
reachable services? Were you running a firewall?

The past kernel bugs had very early exploit scripts. It is really a
no-brainer to insert a rootkit if something lets you, say, write a
script to /tmp and call it by exploitable buffer overflows, badly
written CGI...

And remember that there's (nearly) no possibility for a positive proof
of the non-existence of a root kit.

-hwh
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Re: chkrootkit LKM trojan ?

[Dave S]

On Monday 17 July 2006 21:35, Hans-Werner Hilse wrote:
> Hi,
>
> On Mon, 17 Jul 2006 19:36:30 +0100
>
> Dave S <gentoo@pusspaws.net> wrote:
> > How accurate is chkproc?
> >  If you run chkproc on a server that runs lots of short time processes it
> > could report some false positives. chkproc compares the ps output with
> > the /proc contents. If processes are created/killed during this operation
> > chkproc could point out these PIDs as suspicious.
> >
> > That fits in with the fact that chkrootkit & rkhunter now report clean (&
> > also fits in with someone tinkering from the inside !)
>
> The problem I see here is that you can't expect chkrootkit to find
> something when scanning from a clean base (Live-CD) when the only hint
> you had was an alert from chkproc. You probably would have gotten the
> alert from chkrootkit in the first place. chkproc inspects the
> currently running system (and the /proc for the currently running
> kernel). I.e. if it has no signature for the rootkit itself, it can't
> find it again from that "clean" kernel.
>
> Do you have the possibility to monitor internet connections on an
> intermediary gateway? I think monitoring it for a few days would give
> you a better hint if there might be something active.
>
> And there are other things to think about. Do you have a webserver
> running?
Nope

> CGI scripts?
Nope

> PHP applications?
Nope

> Do you have other network   
> reachable services?

Nope none outside of my LAN
> > Were you running a firewall?  

Yep - a netgear router firewall, NAT & state aware 
>
> The past kernel bugs had very early exploit scripts. It is really a
> no-brainer to insert a rootkit if something lets you, say, write a
> script to /tmp and call it by exploitable buffer overflows, badly
> written CGI...
>
> And remember that there's (nearly) no possibility for a positive proof
> of the non-existence of a root kit.

I am now seriously considering installing tripwire - To be sure of a clean 
tripwire database I know it means a clean install ... gulp ...

>
> -hwh
-- 
gentoo-user@gentoo.org mailing list