[gentoo-user] Re: Rooted/compromised Gentoo, seeking advice [Solved?]

[gentoo-user] Re: Rooted/compromised Gentoo, seeking advice [Solved?]

[Paul Hartman]

On Mon, Aug 9, 2010 at 11:25 AM, Paul Hartman
<paul.hartman+gentoo@gmail.com> wrote:
> Hi, today when working remotely I ran nethogs and noticed suspicious
> network traffic coming from my home gentoo box. It was very low
> traffic (less than 1KB/sec bandwidth usage) but according to nethogs
> it was between a root user process and various suspicious-looking
> ports on outside hosts in other countries that I have no business
> with. netstat didn't show anything, however, but when I ran chkrootkit
> told me that netstat was INFECTED. I immediately issued "shutdown -h
> now" and now I won't be able to take a further look at it until I get
> home and have physical access to the box. System uptime was a few
> months. It was last updated for installation of a 2.6.33 kernel
> (2.6.35 is out now).

Well, so far everything I'm seeing points to a false alarm. :) It
seems I may have overreacted due to my lack of understanding.

First, when I got home and inspected router settings I realized the
strange activity I saw earlier was happening on a port I had opened
for Vuze (the bittorrent client). Nethogs output was like this:

NetHogs version 0.7.0
PID USER     PROGRAM                      DEV        SENT      RECEIVED
0     root     ..7423-213.138.94.110:49971             0.032       0.038 KB/sec
0     root     ..7423-72.191.172.228:54861             0.000       0.000 KB/sec
0     root     ..00:17423-82.52.3.94:57635             0.000       0.000 KB/sec
0     root     unknown TCP                             0.000
0.000 KB/sec
TOTAL                                                0.032       0.038 KB/sec

Based on my Googling tonight, it seems this may simply be how it
displays incoming connection attempts. I found a post on the Ubuntu
Launchpad site that is basically asking the same question:
https://answers.launchpad.net/ubuntu/+source/nethogs/+question/113880

I changed my designated port setting in Vuze, opened that port on my
firewall, and then waited a few minutes and sure enough this same kind
of "mystery traffic" started to appear on that port. So it would seem
to be innocent bittorrent traffic. Egg on my face.

Second, the problem of chkrootkit telling me "find" and "netstat" were
INFECTED, in big scary upper-case letters. The files appear to be
genuine, I checked and double-checked and they appear to be
legitimate. I re-emerged them and the files match and still fail the
test. After looking into how chkroot does its tests, it's simply
grepping the strings from the file. I have debugging info compiled
into everything on my system and perhaps that means the files are
quite a bit more chatty than usual when it comes to strings. The
damning strings that caused it to give me an INFECTED warning? (using
the pattern from chkrootkit's test)

/usr/bin/find: sharefile.h
/bin/netstat: sockaddr.h

To further test this false-positive theory, I stripped those two
binaries of debugging data and now they do not appear as INFECTED by
the test. If anyone else wants to compile net-tools or findutils with
debugging data and nostrip and then run chkrootkit to see what results
you get on these files, that would be quite helpful in confirming
this.

I then tried rkhunter. It gave me numerous warnings, but after
checking the log for details they all appear to be harmless (For
example, it warns that /usr/bin/ldd is a script, not a binary... as
far as I can tell, that is how it's supposed to be)

Next I ran app-forensics/lynis, which is a more general system
settings audit. Everything looked normal there, too.

I've audited all of my logs, bash history, etc and everything looks
fine. The logs are complete. I use metalog so I've got duplicate log
data in most cases, split up into different files and directories, and
they all match. I've checked the other computers/devices in the house
and don't see any signs of any funny business.

The router settings and activity all look normal as well. I already
had non-default password, telnet disabled, external admin interface
disabled, web interface disabled, etc. and the firmware is the latest
version, supposedly not vulnerable to the milw0rm attack so I think it
is secure as can be expected.

I've checked all servers & online services that allow me to view my
login history and I don't see any unusual activity.

At this point I feel pretty good that my box was not compromised and
it was only ignorance and panic on my part. To play it safe, I'm going
to leave it disconnected for tonight and do some monitoring tomorrow
with wireshark just to be absolutely sure there's nothing going on.
Wish me luck! :)

I am grateful to everyone for their ideas and suggestions, and I'm
definitely going to change my sudoers privileges and more importantly
my habits and assumptions. The grace period that William alluded to
(timestamp_timeout is what Google tells me) may help to relieve a bit
of the "pain" of having to type my password so often.

Thanks,
Paul

Re: [gentoo-user] Re: Rooted/compromised Gentoo, seeking advice [Solved?]

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 755 bytes --]

On Tue, 10 Aug 2010 01:10:37 -0500, Paul Hartman wrote:

> Second, the problem of chkrootkit telling me "find" and "netstat" were
> INFECTED, in big scary upper-case letters. The files appear to be
> genuine,

chkrootkit hasn't been updated in over a year, a bit scary for a malware
scanner.

> I then tried rkhunter. It gave me numerous warnings, but after
> checking the log for details they all appear to be harmless (For
> example, it warns that /usr/bin/ldd is a script, not a binary... as
> far as I can tell, that is how it's supposed to be)

You can tweak the rkhunter config to skip specific tests on specific
files (or patterns) to avoid these false positives.


-- 
Neil Bothwick

Top Oxymorons Number 3: Working vacation

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[gentoo-user] Increasing security [WAS: Rooted/compromised Gentoo, seeking advice [Solved?]

[Enrico Weigelt]

* Paul Hartman <paul.hartman+gentoo@gmail.com> wrote:

<snip>

Apropos cracked machines:

In recent years I often got trouble w/ cracked customer's boxes
(one eg. was abused for SIP-calling people around the world and
asking them for their debit card codes ;-o). So thought about
protection against those scenarios. The solution:

Put all remotely available services into containers and make the 
host system only accessible via special channels (eg. serial console). 
You can run automatic sanity tests and security alerts from the hosts
system, which cannot be highjacked (as long as there's no kernel
bug which allows escaping a container ;-o).

This also brings several other benefits, eg. easier backups, quick
migration to other machines, etc.


cu
-- 
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 phone:  +49 36207 519931  email: weigelt@metux.de
 mobile: +49 151 27565287  icq:   210169427         skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

Re: [gentoo-user] Increasing security [WAS: Rooted/compromised Gentoo, seeking advice [Solved?]

[Mark Knecht]

On Fri, Aug 13, 2010 at 8:25 AM, Enrico Weigelt <weigelt@metux.de> wrote:
> * Paul Hartman <paul.hartman+gentoo@gmail.com> wrote:
>
> <snip>
>
> Apropos cracked machines:
>
> In recent years I often got trouble w/ cracked customer's boxes
> (one eg. was abused for SIP-calling people around the world and
> asking them for their debit card codes ;-o). So thought about
> protection against those scenarios. The solution:
>
> Put all remotely available services into containers and make the
> host system only accessible via special channels (eg. serial console).
> You can run automatic sanity tests and security alerts from the hosts
> system, which cannot be highjacked (as long as there's no kernel
> bug which allows escaping a container ;-o).
>
> This also brings several other benefits, eg. easier backups, quick
> migration to other machines, etc.
>
>
> cu

Hi Enrico,
   Since I'm not an IT guy could you please explain this just a bit
more? What is 'a container'? Is it a chroot running on the same
machine? A different machine? Something completely different?

   In the OP's case (I believe) he thought a personal machine at home
was compromised. If that's the case then without doubling my
electrical bill (2 computers) how would I implement your containers?

Thanks,
Mark

Re: [gentoo-user] Increasing security [WAS: Rooted/compromised Gentoo, seeking advice [Solved?]

[Bill Longman]

On 08/13/2010 09:25 AM, Mark Knecht wrote:
> On Fri, Aug 13, 2010 at 8:25 AM, Enrico Weigelt <weigelt@metux.de> wrote:
>> * Paul Hartman <paul.hartman+gentoo@gmail.com> wrote:
>>
>> <snip>
>>
>> Apropos cracked machines:
>>
>> In recent years I often got trouble w/ cracked customer's boxes
>> (one eg. was abused for SIP-calling people around the world and
>> asking them for their debit card codes ;-o). So thought about
>> protection against those scenarios. The solution:
>>
>> Put all remotely available services into containers and make the
>> host system only accessible via special channels (eg. serial console).
>> You can run automatic sanity tests and security alerts from the hosts
>> system, which cannot be highjacked (as long as there's no kernel
>> bug which allows escaping a container ;-o).
>>
>> This also brings several other benefits, eg. easier backups, quick
>> migration to other machines, etc.
>>
>>
>> cu
> 
> Hi Enrico,
>    Since I'm not an IT guy could you please explain this just a bit
> more? What is 'a container'? Is it a chroot running on the same
> machine? A different machine? Something completely different?
> 
>    In the OP's case (I believe) he thought a personal machine at home
> was compromised. If that's the case then without doubling my
> electrical bill (2 computers) how would I implement your containers?

Basically just run VMWare/Virtualbox etc and put the services in there.

That's why I force my kids to use IE in a VM....

No, chroots are NOT the same. They run on the same system.

Re: [gentoo-user] Increasing security [WAS: Rooted/compromised Gentoo, seeking advice [Solved?]

[Enrico Weigelt]

* Mark Knecht <markknecht@gmail.com> wrote:

Hi,

>    Since I'm not an IT guy could you please explain this just a bit
> more? What is 'a container'? Is it a chroot running on the same
> machine? A different machine? Something completely different?

http://lxc.sourceforge.net/
http://wiki.openvz.org/Main_Page

Unlike VM solutions like kvm, vmware, etc, these (OS-side) 
container implementations split off the operating system 
resources (filesystem, network interfaces, process-IDs, ...)
into namespaces, so each container only sees its own resources,
not those of the host system or other containers.

That's essentially what's behind the "virtual private server"
solutions offered by various ISPs.

>    In the OP's case (I believe) he thought a personal machine at home
> was compromised. If that's the case then without doubling my
> electrical bill (2 computers) how would I implement your containers?

He would have several virtual servers running on just one metal.
If the host system is not accessible from the outside world, just
the virtual servers - an attacker could probably highjack what's
inside the virtual servers, but cant get to the host system.


cu
-- 
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 phone:  +49 36207 519931  email: weigelt@metux.de
 mobile: +49 151 27565287  icq:   210169427         skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

Re: [gentoo-user] Increasing security [WAS: Rooted/compromised Gentoo, seeking advice [Solved?]

[Enrico Weigelt]

* Bill Longman <bill.longman@gmail.com> wrote:

> Basically just run VMWare/Virtualbox etc and put the services in there.

well, these solutions are way "bigger" (iow: more resource
intensive), since they run a complete operation system instance
within the virtual machine.

> No, chroots are NOT the same. They run on the same system.

well, chroots have not much to do with containers (even contains
could be said to include chroot as a building block) - they just
run certain processes with a different root directory (iow: these
processes see just see a subdirectory as it would be the whole
filesystem). that's nice for testing porposes or to isolate
different kind of isolate programs/libraries (eg. use different
libc's, ABIs or calling conventions, 32bit subsystems on an 
native 64bit host, etc, etc), but don't really add security.


cu
-- 
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 phone:  +49 36207 519931  email: weigelt@metux.de
 mobile: +49 151 27565287  icq:   210169427         skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

Re: [gentoo-user] Increasing security [WAS: Rooted/compromised Gentoo, seeking advice [Solved?]

[Mark Knecht]

On Fri, Aug 13, 2010 at 11:58 AM, Enrico Weigelt <weigelt@metux.de> wrote:
> * Mark Knecht <markknecht@gmail.com> wrote:
>
> Hi,
>
>>    Since I'm not an IT guy could you please explain this just a bit
>> more? What is 'a container'? Is it a chroot running on the same
>> machine? A different machine? Something completely different?
>
> http://lxc.sourceforge.net/
> http://wiki.openvz.org/Main_Page
>
> Unlike VM solutions like kvm, vmware, etc, these (OS-side)
> container implementations split off the operating system
> resources (filesystem, network interfaces, process-IDs, ...)
> into namespaces, so each container only sees its own resources,
> not those of the host system or other containers.
>
> That's essentially what's behind the "virtual private server"
> solutions offered by various ISPs.
>
>>    In the OP's case (I believe) he thought a personal machine at home
>> was compromised. If that's the case then without doubling my
>> electrical bill (2 computers) how would I implement your containers?
>
> He would have several virtual servers running on just one metal.
> If the host system is not accessible from the outside world, just
> the virtual servers - an attacker could probably highjack what's
> inside the virtual servers, but cant get to the host system.
>
>
> cu

Thank you Enrico. I'll have to learn about this.

Cheers,
Mark

Re: [gentoo-user] Increasing security [WAS: Rooted/compromised Gentoo, seeking advice [Solved?]

[Jarry]

On 13. 8. 2010 21:05, Enrico Weigelt wrote:
> * Bill Longman<bill.longman@gmail.com>  wrote:
>
>> Basically just run VMWare/Virtualbox etc and put the services in there.
>
> well, these solutions are way "bigger" (iow: more resource
> intensive), since they run a complete operation system instance
> within the virtual machine.

That is why I picked up Linux-VServer (actually, first I tried
OpenVZ but could not make it run). It is a kind of compromise,
where all guests share the same kernel. This brings certain
security implications, but on the other side, I can run dozens
of guest on a moderate machine, with 4-cores and 8GB memory
(i.e. a guest running bind takes just about 20MB of memory)...

The only service running on my "host" (main system) is sshd,
which I secured as much as I could. Everything else (web, mail,
dns, ftp, syslog, X, and plenty of users' services) runs on its
own guest-system, chrooted in addition (where it was possible).

Jarry

-- 
_______________________________________________________________
This mailbox accepts e-mails only from selected mailing-lists!
Everything else is considered to be spam and therefore deleted.

Re: [gentoo-user] Increasing security [WAS: Rooted/compromised Gentoo, seeking advice [Solved?]

[Bill Longman]

On 08/14/2010 12:32 PM, Jarry wrote:
> On 13. 8. 2010 21:05, Enrico Weigelt wrote:
>> * Bill Longman<bill.longman@gmail.com>  wrote:
>>
>>> Basically just run VMWare/Virtualbox etc and put the services in there.
>>
>> well, these solutions are way "bigger" (iow: more resource
>> intensive), since they run a complete operation system instance
>> within the virtual machine.
> 
> That is why I picked up Linux-VServer (actually, first I tried
> OpenVZ but could not make it run). It is a kind of compromise,
> where all guests share the same kernel. This brings certain
> security implications, but on the other side, I can run dozens
> of guest on a moderate machine, with 4-cores and 8GB memory
> (i.e. a guest running bind takes just about 20MB of memory)...

This looks rather interesting, Jarry. Is it simply a matter of compiling
the vserver-sources and util-vserver? Did it take much time to set up
the kernel for your box? Or is it pretty much a typical kernel setup?
Any good tools in the util-vserver package?

> The only service running on my "host" (main system) is sshd,
> which I secured as much as I could. Everything else (web, mail,
> dns, ftp, syslog, X, and plenty of users' services) runs on its
> own guest-system, chrooted in addition (where it was possible).

Sounds very efficient.

TIA,

Bill

Re: [gentoo-user] Increasing security [WAS: Rooted/compromised Gentoo, seeking advice [Solved?]

[Mark Knecht]

On Mon, Aug 16, 2010 at 7:16 AM, Bill Longman <bill.longman@gmail.com> wrote:
> On 08/14/2010 12:32 PM, Jarry wrote:
>> On 13. 8. 2010 21:05, Enrico Weigelt wrote:
>>> * Bill Longman<bill.longman@gmail.com>  wrote:
>>>
>>>> Basically just run VMWare/Virtualbox etc and put the services in there.
>>>
>>> well, these solutions are way "bigger" (iow: more resource
>>> intensive), since they run a complete operation system instance
>>> within the virtual machine.
>>
>> That is why I picked up Linux-VServer (actually, first I tried
>> OpenVZ but could not make it run). It is a kind of compromise,
>> where all guests share the same kernel. This brings certain
>> security implications, but on the other side, I can run dozens
>> of guest on a moderate machine, with 4-cores and 8GB memory
>> (i.e. a guest running bind takes just about 20MB of memory)...
>
> This looks rather interesting, Jarry. Is it simply a matter of compiling
> the vserver-sources and util-vserver? Did it take much time to set up
> the kernel for your box? Or is it pretty much a typical kernel setup?
> Any good tools in the util-vserver package?
>
>> The only service running on my "host" (main system) is sshd,
>> which I secured as much as I could. Everything else (web, mail,
>> dns, ftp, syslog, X, and plenty of users' services) runs on its
>> own guest-system, chrooted in addition (where it was possible).
>
> Sounds very efficient.
>
> TIA,
>
> Bill

Certainly looks interesting.

I guess the baselayout-vserver packages is somehow for setting up each
of the guests?

QUESTION: Where does X run? In the host or separate copies in each guest?

For a long time I've wanted to set up a single piece of hardware for
my parents, but with two screens, two keyboards, two mice. Each user
would have what they expect in front of them physically but it's
really a single computer. Can that be done using this software?

Thanks,
Mark

Re: [gentoo-user] Increasing security [WAS: Rooted/compromised Gentoo, seeking advice [Solved?]

[Jarry]

On 16. 8. 2010 17:29, Mark Knecht wrote:
> On Mon, Aug 16, 2010 at 7:16 AM, Bill Longman<bill.longman@gmail.com>:
>>>
>>> That is why I picked up Linux-VServer (actually, first I tried
>>> OpenVZ but could not make it run). It is a kind of compromise,
>>> where all guests share the same kernel. This brings certain
>>> security implications, but on the other side, I can run dozens
>>> of guest on a moderate machine, with 4-cores and 8GB memory
>>> (i.e. a guest running bind takes just about 20MB of memory)...
>>
>> This looks rather interesting, Jarry. Is it simply a matter of compiling
>> the vserver-sources and util-vserver? Did it take much time to set up
>> the kernel for your box? Or is it pretty much a typical kernel setup?
>> Any good tools in the util-vserver package?

vserver-sources and util-vserver was all I needed. Kernel is
pretty much like common, with ~10 additional options. util-vserver
contains handy tools, like "v*" (* being emerge, esync, kill,
limit, mount, ps, sched, etc.). Updating all gentoo-guests can be
done with one command executed in host...

>> Sounds very efficient.

Really is. Now I'm running 27 guests, mostly gentoo but also
some ubuntu and opensuse. Actually, it is possible to run any
linux-based system (as I said all systems share the same kernel).
There is also pretty good control over resources allocated
to individual guests (disk, memory, cpu).

Administration is very comfortable. Tasks like clonning,
backup/restore, moving, migration, etc, are very easy to...

> I guess the baselayout-vserver packages is somehow for setting up each
> of the guests?

Guests are installed using customised stage3 (baselayout2-based).
After that, you work with them as with normal gentoo-system.

> QUESTION: Where does X run? In the host or separate copies in each guest?

If you need X, you can create a special guest for it, and run X
there. The only thing which must run in host are kernel-modules
(i.e. nvidia driver). I tested this only as an experiment, but
it works. I've heard of someone running X+Wine in vserver-guest.
It is also possible to run X+VMware+Windows in vserver-guest...

> For a long time I've wanted to set up a single piece of hardware for
> my parents, but with two screens, two keyboards, two mice. Each user
> would have what they expect in front of them physically but it's
> really a single computer. Can that be done using this software?

Frankly, I do not know. But for each guest you can setup different
tty and IP, so maybe it would be possible. Though I think maybe
some kind of terminal server would be more suitable...

Jarry

-- 
_______________________________________________________________
This mailbox accepts e-mails only from selected mailing-lists!
Everything else is considered to be spam and therefore deleted.

Re: [gentoo-user] Increasing security [WAS: Rooted/compromised Gentoo, seeking advice [Solved?]

[Bill Longman]

On 08/16/2010 09:07 AM, Jarry wrote:
> On 16. 8. 2010 17:29, Mark Knecht wrote:
>> On Mon, Aug 16, 2010 at 7:16 AM, Bill Longman<bill.longman@gmail.com>:
>>>>
>>>> That is why I picked up Linux-VServer (actually, first I tried
>>>> OpenVZ but could not make it run). It is a kind of compromise,
>>>> where all guests share the same kernel. This brings certain
>>>> security implications, but on the other side, I can run dozens
>>>> of guest on a moderate machine, with 4-cores and 8GB memory
>>>> (i.e. a guest running bind takes just about 20MB of memory)...
>>>
>>> This looks rather interesting, Jarry. Is it simply a matter of compiling
>>> the vserver-sources and util-vserver? Did it take much time to set up
>>> the kernel for your box? Or is it pretty much a typical kernel setup?
>>> Any good tools in the util-vserver package?
> 
> vserver-sources and util-vserver was all I needed. Kernel is
> pretty much like common, with ~10 additional options. util-vserver
> contains handy tools, like "v*" (* being emerge, esync, kill,
> limit, mount, ps, sched, etc.). Updating all gentoo-guests can be
> done with one command executed in host...
> 
>>> Sounds very efficient.
> 
> Really is. Now I'm running 27 guests, mostly gentoo but also
> some ubuntu and opensuse. Actually, it is possible to run any
> linux-based system (as I said all systems share the same kernel).
> There is also pretty good control over resources allocated
> to individual guests (disk, memory, cpu).
> 
> Administration is very comfortable. Tasks like clonning,
> backup/restore, moving, migration, etc, are very easy to...
> 
>> I guess the baselayout-vserver packages is somehow for setting up each
>> of the guests?
> 
> Guests are installed using customised stage3 (baselayout2-based).
> After that, you work with them as with normal gentoo-system.

The Gentoo version of Solaris Zones! w00t!

Re: [gentoo-user] Increasing security [WAS: Rooted/compromised Gentoo, seeking advice [Solved?]

[Enrico Weigelt]

* Jarry <mr.jarry@gmail.com> wrote:

> The only service running on my "host" (main system) is sshd,
> which I secured as much as I could.

If you have some physical access (eg. serial console), you 
could even drop sshd (or only bind it to some local interface)
to get around possible ssh attacks. That's what I'm doing on
several machines.

> Everything else (web, mail, dns, ftp, syslog, X, and plenty of
> users' services) runs on its own guest-system, chrooted in
> addition (where it was possible).

Yes, that's also my approach. 

BTW: I'm currently trying to convice one of my customers - an
major German ISP - to provide a generic solution for such kind
of environments: customers can allocate and configure containers 
at will (also via robot interfaces), and the ISP takes care of
the cluster of host machines ... maybe I get the leading product
managers convinced some day ;-)


cu
-- 
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 phone:  +49 36207 519931  email: weigelt@metux.de
 mobile: +49 151 27565287  icq:   210169427         skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------