[gentoo-user] How to record memory usage & bandwidth usage?

[gentoo-user] How to record memory usage & bandwidth usage?

[Pandu Poluan]

The head honcho of my company just asked me to "plan for migration of
X into the cloud" (where "X" is the online trading server that our
investors used).

Now, I need to monitor how much RAM is used throughout the day by X,
also how much bandwidth gets eaten by X throughout the day.

What tools do you recommend?

Remember: The data will be used for 'post-mortem' analysis, so I don't
need any fancy schmancy presentation. Just raw data, taken every N
seconds.

Rgds,
-- 
FdS Pandu E Poluan
~ IT Optimizer ~

 • LOPSA Member #15248
 • Blog : http://pepoluan.tumblr.com
 • Linked-In : http://id.linkedin.com/in/pepoluan

Re: [gentoo-user] How to record memory usage & bandwidth usage?

[Mick]

[-- Attachment #1: Type: Text/Plain, Size: 832 bytes --]

On Tuesday 11 Oct 2011 10:48:31 Pandu Poluan wrote:
> The head honcho of my company just asked me to "plan for migration of
> X into the cloud" (where "X" is the online trading server that our
> investors used).
> 
> Now, I need to monitor how much RAM is used throughout the day by X,
> also how much bandwidth gets eaten by X throughout the day.
> 
> What tools do you recommend?
> 
> Remember: The data will be used for 'post-mortem' analysis, so I don't
> need any fancy schmancy presentation. Just raw data, taken every N
> seconds.

I have used mrtg and nagios to capture and monitor both, but you'll have to 
install and configure them.

If you're good with perl or python, then some simple script should be able to 
capture such values and record on a flat file, or even a database. 
-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] How to record memory usage & bandwidth usage?

[Matthew Marlowe]

Pandu,

Any  modern monitoring framework/server with a web interface will have
tools to select metrics to retrieve and store into a database and
display/graph/alert as needed using whatever reasonable collection
interval you define.
If your metrics are relatively simple, you should be able to get a
solution implemented rather quickly without having to write any of
your own code and the overhead/resources needed on your server would
just be proportional to the number of metrics collected and their
frequency.

My current monitoring tool of choice is zabbix, but there are many options.

Matt

On Tue, Oct 11, 2011 at 3:21 AM, Mick <michaelkintzios@gmail.com> wrote:
> On Tuesday 11 Oct 2011 10:48:31 Pandu Poluan wrote:
>> The head honcho of my company just asked me to "plan for migration of
>> X into the cloud" (where "X" is the online trading server that our
>> investors used).
>>
>> Now, I need to monitor how much RAM is used throughout the day by X,
>> also how much bandwidth gets eaten by X throughout the day.
>>
>> What tools do you recommend?
>>
>> Remember: The data will be used for 'post-mortem' analysis, so I don't
>> need any fancy schmancy presentation. Just raw data, taken every N
>> seconds.
>
> I have used mrtg and nagios to capture and monitor both, but you'll have to
> install and configure them.
>
> If you're good with perl or python, then some simple script should be able to
> capture such values and record on a flat file, or even a database.
> --
> Regards,
> Mick
>



-- 
Matthew Marlowe
matt@professionalsysadmin.com
Senior Internet Infrastructure Consultant         DevOps/VMware/SysAdmin
https://www.twitter.com/deploylinux                       Gentoo Linux Dev

           "Courage is not simply one of the virtues, but the form
              of every virtue at the testing point."  -- C.S. Lewis

[gentoo-user] Re: How to record memory usage & bandwidth usage?

[James]

Pandu Poluan <pandu <at> poluan.info> writes:


> The head honcho of my company just asked me to "plan for migration of
> X into the cloud" (where "X" is the online trading server that our
> investors used).

This is a single server or many at different locations.
If a WAN monitoring is what you are after, along with individual
server resources, you have many choices.

> Now, I need to monitor how much RAM is used throughout the day by X,
> also how much bandwidth gets eaten by X throughout the day.

Most of the packages monitor ram as well as other resource utilization
of the servers, firewall, routers and other SNMP devices in your network.
some experimentation may be warranted to find what your team likes best.

> What tools do you recommend?

OH boy. I like JFFNMS very very much. It has a very old version in portage
(masked) but a very new version out there for Debian and Ubuntu. It 
runs on all nix, if you want to driectly compile and install.

I'll be putting together a new ebuild, as soon as I get it working
with the latest postgresql. Mysql works out of the box. Postgresql-9
has many new and very cool features.

> Remember: The data will be used for 'post-mortem' analysis, so I don't
> need any fancy schmancy presentation. Just raw data, taken every N
> seconds.

Personally, I have some large, high risk design work going on. JFFNMS
and pg9 are the best choices from my research. A whiz like yourself
could easily look at the old JFFNMS ebuild and create a new one.
PG-9 (please no flame wars on mysql vs pg9) is very cool and what
my work is migrating too, once I get some breathing room.

Craig at jffnms.org is very cool and responsive. He also works closely
with those that submit patches. Nagios is a large, disorder array that
had many devs fork off since the project leader (was/is an a_ole)
is quite difficult to work with.

JFFNMS rules and is very cool for managing cisco and other routers,
not to mention a myriad of snmp(1,2.3) devices and all types
of servers. The original guy, Javier, was snapped  up by someone
worth billions, to manage and extend his financial network, but, Craig
is probably stronger coder, and extraordinarily nice human being.
It's mostly php. Lots of folks extend JFFNMS, Craig keeps it clean
and well written and documented code.

http://www.jffnms.org/

hth,
James

Re: [gentoo-user] Re: How to record memory usage & bandwidth usage?

[Pandu Poluan]

[-- Attachment #1: Type: text/plain, Size: 3557 bytes --]

(Sorry for the late reply; somehow this thread got lost in the mess)

On Oct 12, 2011 2:03 AM, "James" <wireless@tampabay.rr.com> wrote:
>
> Pandu Poluan <pandu <at> poluan.info> writes:
>
>
> > The head honcho of my company just asked me to "plan for migration of
> > X into the cloud" (where "X" is the online trading server that our
> > investors used).
>
> This is a single server or many at different locations.
> If a WAN monitoring is what you are after, along with individual
> server resources, you have many choices.
>

It's a single server that's part of a three-server system. The server needs
to communicate with its 2 cohorts continuously, so I have to provision
enough backhaul bandwidth from the cloud to my data center.

In addition to provisioning enough RAM and CPU, of course.

> > Now, I need to monitor how much RAM is used throughout the day by X,
> > also how much bandwidth gets eaten by X throughout the day.
>
> Most of the packages monitor ram as well as other resource utilization
> of the servers, firewall, routers and other SNMP devices in your network.
> some experimentation may be warranted to find what your team likes best.
>

Currently I've settled on a simple solution: run dstat[1] with nohup 30
minutes before 1st trading session, stop it 30 minutes after 2nd trading
session, and send the CSV record via email. Less intrusion into the system
(which the Systems guys rightly have reservations of).

> > What tools do you recommend?
>
> OH boy. I like JFFNMS very very much. It has a very old version in portage
> (masked) but a very new version out there for Debian and Ubuntu. It
> runs on all nix, if you want to driectly compile and install.
>
> I'll be putting together a new ebuild, as soon as I get it working
> with the latest postgresql. Mysql works out of the box. Postgresql-9
> has many new and very cool features.
>

Cool! I *love* Postgresql! Update me when the ebuild's done?

> > Remember: The data will be used for 'post-mortem' analysis, so I don't
> > need any fancy schmancy presentation. Just raw data, taken every N
> > seconds.
>
> Personally, I have some large, high risk design work going on. JFFNMS
> and pg9 are the best choices from my research. A whiz like yourself
> could easily look at the old JFFNMS ebuild and create a new one.

Naaah, I'm going to wait for your ebuild. I'm sometimes lazy, you know ;-)

> PG-9 (please no flame wars on mysql vs pg9) is very cool and what
> my work is migrating too, once I get some breathing room.
>
> Craig at jffnms.org is very cool and responsive. He also works closely
> with those that submit patches. Nagios is a large, disorder array that
> had many devs fork off since the project leader (was/is an a_ole)
> is quite difficult to work with.
>

That sounds really cool. I've been hesitant to go the Nagios route because
of the mess. I'll sure to be checking out JFFNMS.

> JFFNMS rules and is very cool for managing cisco and other routers,
> not to mention a myriad of snmp(1,2.3) devices and all types
> of servers. The original guy, Javier, was snapped  up by someone
> worth billions, to manage and extend his financial network, but, Craig
> is probably stronger coder, and extraordinarily nice human being.
> It's mostly php. Lots of folks extend JFFNMS, Craig keeps it clean
> and well written and documented code.
>
> http://www.jffnms.org/
>
> hth,
> James
>

Thanks for the heads-up! Although the original problem is solved already
(granted, in a somewhat kludgy way), your post is a great write-opener! Much
appreciated :-)

Rgds,

[-- Attachment #2: Type: text/html, Size: 4382 bytes --]

Re: [gentoo-user] Re: How to record memory usage & bandwidth usage?

[Pandu Poluan]

[-- Attachment #1: Type: text/plain, Size: 3923 bytes --]

(My age surely is catching up with me, I forgot to include the URL for
dstat)

[1] http://dag.wieers.com/home-made/dstat/

Rgds,
 On Oct 26, 2011 8:27 AM, "Pandu Poluan" <pandu@poluan.info> wrote:

> (Sorry for the late reply; somehow this thread got lost in the mess)
>
> On Oct 12, 2011 2:03 AM, "James" <wireless@tampabay.rr.com> wrote:
> >
> > Pandu Poluan <pandu <at> poluan.info> writes:
> >
> >
> > > The head honcho of my company just asked me to "plan for migration of
> > > X into the cloud" (where "X" is the online trading server that our
> > > investors used).
> >
> > This is a single server or many at different locations.
> > If a WAN monitoring is what you are after, along with individual
> > server resources, you have many choices.
> >
>
> It's a single server that's part of a three-server system. The server needs
> to communicate with its 2 cohorts continuously, so I have to provision
> enough backhaul bandwidth from the cloud to my data center.
>
> In addition to provisioning enough RAM and CPU, of course.
>
> > > Now, I need to monitor how much RAM is used throughout the day by X,
> > > also how much bandwidth gets eaten by X throughout the day.
> >
> > Most of the packages monitor ram as well as other resource utilization
> > of the servers, firewall, routers and other SNMP devices in your network.
> > some experimentation may be warranted to find what your team likes best.
> >
>
> Currently I've settled on a simple solution: run dstat[1] with nohup 30
> minutes before 1st trading session, stop it 30 minutes after 2nd trading
> session, and send the CSV record via email. Less intrusion into the system
> (which the Systems guys rightly have reservations of).
>
> > > What tools do you recommend?
> >
> > OH boy. I like JFFNMS very very much. It has a very old version in
> portage
> > (masked) but a very new version out there for Debian and Ubuntu. It
> > runs on all nix, if you want to driectly compile and install.
> >
> > I'll be putting together a new ebuild, as soon as I get it working
> > with the latest postgresql. Mysql works out of the box. Postgresql-9
> > has many new and very cool features.
> >
>
> Cool! I *love* Postgresql! Update me when the ebuild's done?
>
> > > Remember: The data will be used for 'post-mortem' analysis, so I don't
> > > need any fancy schmancy presentation. Just raw data, taken every N
> > > seconds.
> >
> > Personally, I have some large, high risk design work going on. JFFNMS
> > and pg9 are the best choices from my research. A whiz like yourself
> > could easily look at the old JFFNMS ebuild and create a new one.
>
> Naaah, I'm going to wait for your ebuild. I'm sometimes lazy, you know ;-)
>
> > PG-9 (please no flame wars on mysql vs pg9) is very cool and what
> > my work is migrating too, once I get some breathing room.
> >
> > Craig at jffnms.org is very cool and responsive. He also works closely
> > with those that submit patches. Nagios is a large, disorder array that
> > had many devs fork off since the project leader (was/is an a_ole)
> > is quite difficult to work with.
> >
>
> That sounds really cool. I've been hesitant to go the Nagios route because
> of the mess. I'll sure to be checking out JFFNMS.
>
> > JFFNMS rules and is very cool for managing cisco and other routers,
> > not to mention a myriad of snmp(1,2.3) devices and all types
> > of servers. The original guy, Javier, was snapped  up by someone
> > worth billions, to manage and extend his financial network, but, Craig
> > is probably stronger coder, and extraordinarily nice human being.
> > It's mostly php. Lots of folks extend JFFNMS, Craig keeps it clean
> > and well written and documented code.
> >
> > http://www.jffnms.org/
> >
> > hth,
> > James
> >
>
> Thanks for the heads-up! Although the original problem is solved already
> (granted, in a somewhat kludgy way), your post is a great write-opener! Much
> appreciated :-)
>
> Rgds,
>

[-- Attachment #2: Type: text/html, Size: 4954 bytes --]

Re: [gentoo-user] Re: How to record memory usage & bandwidth usage?

[kashani]

On 10/25/2011 6:27 PM, Pandu Poluan wrote:
> (Sorry for the late reply; somehow this thread got lost in the mess)
>
> On Oct 12, 2011 2:03 AM, "James" <wireless@tampabay.rr.com
> <mailto:wireless@tampabay.rr.com>> wrote:
>  >
>  > Pandu Poluan <pandu <at> poluan.info <http://poluan.info>> writes:
>  >
>  >
>  > > The head honcho of my company just asked me to "plan for migration of
>  > > X into the cloud" (where "X" is the online trading server that our
>  > > investors used).
>  >
>  > This is a single server or many at different locations.
>  > If a WAN monitoring is what you are after, along with individual
>  > server resources, you have many choices.
>  >
>
> It's a single server that's part of a three-server system. The server
> needs to communicate with its 2 cohorts continuously, so I have to
> provision enough backhaul bandwidth from the cloud to my data center.
>
> In addition to provisioning enough RAM and CPU, of course.
>
>  > > Now, I need to monitor how much RAM is used throughout the day by X,
>  > > also how much bandwidth gets eaten by X throughout the day.
>  >
>  > Most of the packages monitor ram as well as other resource utilization
>  > of the servers, firewall, routers and other SNMP devices in your network.
>  > some experimentation may be warranted to find what your team likes best.
>  >
>
> Currently I've settled on a simple solution: run dstat[1] with nohup 30
> minutes before 1st trading session, stop it 30 minutes after 2nd trading
> session, and send the CSV record via email. Less intrusion into the
> system (which the Systems guys rightly have reservations of).
>

	You're not going to be happy with this design for a couple of reasons.

1. It's more expensive that your current setup. If the two servers at 
your datacenter are down I assume the server is the cloud is useless and 
vice versa. You already have to maintain infrastructure for those two 
servers so you're realizing no savings by eliminating on server from 
your infrastructure. Buying a $1500 rack server amortized over three 
years is a better deal than paying for equivalent power in the cloud.

2. Latency. You're increasing it.

3. Cloud performance varies. Networks split, machines run slow, it 
happens. You'll have more consistent performance on your own machines. 
It's getting better, but it's still something with which to be aware.

	Migrating to virtual servers makes some sense, but you need to look at 
it on a case by case basis.

kashani

Re: [gentoo-user] Re: How to record memory usage & bandwidth usage?

[Pandu Poluan]

[-- Attachment #1: Type: text/plain, Size: 3274 bytes --]

On Oct 26, 2011 8:55 AM, "kashani" <kashani-list@badapple.net> wrote:
>
> On 10/25/2011 6:27 PM, Pandu Poluan wrote:
>>
>> (Sorry for the late reply; somehow this thread got lost in the mess)
>>
>> On Oct 12, 2011 2:03 AM, "James" <wireless@tampabay.rr.com
>> <mailto:wireless@tampabay.rr.com>> wrote:
>>
>>  >
>>  > Pandu Poluan <pandu <at> poluan.info <http://poluan.info>> writes:
>>  >
>>  >
>>  > > The head honcho of my company just asked me to "plan for migration
of
>>  > > X into the cloud" (where "X" is the online trading server that our
>>  > > investors used).
>>  >
>>  > This is a single server or many at different locations.
>>  > If a WAN monitoring is what you are after, along with individual
>>  > server resources, you have many choices.
>>  >
>>
>> It's a single server that's part of a three-server system. The server
>> needs to communicate with its 2 cohorts continuously, so I have to
>> provision enough backhaul bandwidth from the cloud to my data center.
>>
>> In addition to provisioning enough RAM and CPU, of course.
>>
>>  > > Now, I need to monitor how much RAM is used throughout the day by X,
>>  > > also how much bandwidth gets eaten by X throughout the day.
>>  >
>>  > Most of the packages monitor ram as well as other resource utilization
>>  > of the servers, firewall, routers and other SNMP devices in your
network.
>>  > some experimentation may be warranted to find what your team likes
best.
>>  >
>>
>> Currently I've settled on a simple solution: run dstat[1] with nohup 30
>> minutes before 1st trading session, stop it 30 minutes after 2nd trading
>> session, and send the CSV record via email. Less intrusion into the
>> system (which the Systems guys rightly have reservations of).
>>
>
>        You're not going to be happy with this design for a couple of
reasons.
>
> 1. It's more expensive that your current setup. If the two servers at your
datacenter are down I assume the server is the cloud is useless and vice
versa. You already have to maintain infrastructure for those two servers so
you're realizing no savings by eliminating on server from your
infrastructure. Buying a $1500 rack server amortized over three years is a
better deal than paying for equivalent power in the cloud.
>
> 2. Latency. You're increasing it.
>
> 3. Cloud performance varies. Networks split, machines run slow, it
happens. You'll have more consistent performance on your own machines. It's
getting better, but it's still something with which to be aware.
>
>        Migrating to virtual servers makes some sense, but you need to look
at it on a case by case basis.
>
> kashani
>

Indeed.

The fact is that the server-to-be-clouded is a trading server used by
approx. 3000+ clients (investors). Our 20 Mbps line is barely able to cater
them all, and my company has to pay through the nose for each additional
Mbps.

How big a nose? I'm looking at the quotation on my desk that says my.company
needs to shell out 200 USD (excl tax) per additional Mbps per month...

The traffic between that server and the other 2 is less than 20 Mbps, but
how much less, that's what I'm trying to find out.

As to latency... the "powers that be" had decided that, "... sucks to be
trading on your own; better to go through our dealers... " :-P

Rgds,

[-- Attachment #2: Type: text/html, Size: 4333 bytes --]