[gentoo-user] Cross system dependencies

[gentoo-user] Cross system dependencies

[J. Roeleveld]

Hi all,

This is something I have been looking for for a while now, but not found 
anything easily usable yet.

I need a way to add dependencies to services which are provided by different 
servers. For instance, my mail server uses DNS to locate my LDAP server which 
contains the mail aliases. All these are running on different machines.
Currently, I manually ensure these are all started in the correct sequence, I 
would like to automate this to the point where I can start all 3 servers at 
the same time and have the different services wait for the dependency services 
to be available even though they are on different systems.

All the dependency systems in the init-systems I could find are all based on 
dependencies on the same server. Does anyone know of something that can 
already provide this type of dependencies? Or do I need to write something 
myself?
I have been unable to locate anything via Google, but I might be using the 
wrong search words.

Many thanks,

Joost

Re: [gentoo-user] Cross system dependencies

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 1169 bytes --]

On Sat, 28 Jun 2014 11:36:11 +0200, J. Roeleveld wrote:

> I need a way to add dependencies to services which are provided by
> different servers. For instance, my mail server uses DNS to locate my
> LDAP server which contains the mail aliases. All these are running on
> different machines. Currently, I manually ensure these are all started
> in the correct sequence, I would like to automate this to the point
> where I can start all 3 servers at the same time and have the different
> services wait for the dependency services to be available even though
> they are on different systems.
> 
> All the dependency systems in the init-systems I could find are all
> based on dependencies on the same server. Does anyone know of something
> that can already provide this type of dependencies? Or do I need to
> write something myself?

With systemd you can add ExecStartPre=/some/script to the service's unit
file where /some/script waits for the remote services to become available,
and possibly return an error if the service does not become available
within a set time.


-- 
Neil Bothwick

I have a mind like a steel...uh...thingamajig...

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-user] Cross system dependencies

[J. Roeleveld]

On Saturday, June 28, 2014 01:39:41 PM Neil Bothwick wrote:
> On Sat, 28 Jun 2014 11:36:11 +0200, J. Roeleveld wrote:
> > I need a way to add dependencies to services which are provided by
> > different servers. For instance, my mail server uses DNS to locate my
> > LDAP server which contains the mail aliases. All these are running on
> > different machines. Currently, I manually ensure these are all started
> > in the correct sequence, I would like to automate this to the point
> > where I can start all 3 servers at the same time and have the different
> > services wait for the dependency services to be available even though
> > they are on different systems.
> > 
> > All the dependency systems in the init-systems I could find are all
> > based on dependencies on the same server. Does anyone know of something
> > that can already provide this type of dependencies? Or do I need to
> > write something myself?
> 
> With systemd you can add ExecStartPre=/some/script to the service's unit
> file where /some/script waits for the remote services to become available,
> and possibly return an error if the service does not become available
> within a set time.

That method works for any init-system and writing a script to check and if 
necessary fail is my temporary fall-back plan. I was actually hoping for a 
method that can be used to monitor availability and, if necessary, stop 
services when the dependencies disappear.

--
Joost

Re: [gentoo-user] Cross system dependencies

[thegeezer]

On 06/28/2014 07:06 PM, J. Roeleveld wrote:
> On Saturday, June 28, 2014 01:39:41 PM Neil Bothwick wrote:
>> On Sat, 28 Jun 2014 11:36:11 +0200, J. Roeleveld wrote:
>>> I need a way to add dependencies to services which are provided by
>>> different servers. For instance, my mail server uses DNS to locate my
>>> LDAP server which contains the mail aliases. All these are running on
>>> different machines. Currently, I manually ensure these are all started
>>> in the correct sequence, I would like to automate this to the point
>>> where I can start all 3 servers at the same time and have the different
>>> services wait for the dependency services to be available even though
>>> they are on different systems.
>>>
>>> All the dependency systems in the init-systems I could find are all
>>> based on dependencies on the same server. Does anyone know of something
>>> that can already provide this type of dependencies? Or do I need to
>>> write something myself?
>> With systemd you can add ExecStartPre=/some/script to the service's unit
>> file where /some/script waits for the remote services to become available,
>> and possibly return an error if the service does not become available
>> within a set time.
> That method works for any init-system and writing a script to check and if 
> necessary fail is my temporary fall-back plan. I was actually hoping for a 
> method that can be used to monitor availability and, if necessary, stop 
> services when the dependencies disappear.
>
> --
> Joost
>

the difficulty is in identifying failed services.
local network issue / load issue could mean your services start bouncing.
the best way is to have redundancy so it doesn't matter as much

having said all of that::

systemd will start servers and buffer network activity - how this works
for non local services would be interesting to see.

with openrc :
you could on the DNS server have a service which is just a batch script
that uses watches for pid / program path in "ps" which outputs ACK or
NAK to a file in an NFS share  say /nfs/monitoring/dns

then on the mail server you could have a service that polls
/nfs/monitoring/dns for NAK or ACK
you can then choose to have this service directly start your dependent
services, or if you adjust /etc/init.d/postfix to have depends =
"mymonitorDNS" which is an empty shell of a service. your watchdog
service could stop / start the empty shell of a script mymonitorDNS, and
then postfix depends on mymonitorDNS
this would save you from "i've just stopped the mail server for
maintenance and my watchdogservice has just restarted it due to a
NAK>ACK event"

or...
you could have a central master machine which has it's own services,
watchdog and monitor... i.e. /etc/init.d/thepostfixserver start  /
depends on thednsserver which just runs
# ssh postfixserver '/etc/init.d/postfix start' 

or...
puppet and it's kin

Re: [gentoo-user] Cross system dependencies

[J. Roeleveld]

On Saturday, June 28, 2014 09:23:17 PM thegeezer wrote:
> On 06/28/2014 07:06 PM, J. Roeleveld wrote:
> > On Saturday, June 28, 2014 01:39:41 PM Neil Bothwick wrote:
> >> On Sat, 28 Jun 2014 11:36:11 +0200, J. Roeleveld wrote:
> >>> I need a way to add dependencies to services which are provided by
> >>> different servers. For instance, my mail server uses DNS to locate my
> >>> LDAP server which contains the mail aliases. All these are running on
> >>> different machines. Currently, I manually ensure these are all started
> >>> in the correct sequence, I would like to automate this to the point
> >>> where I can start all 3 servers at the same time and have the different
> >>> services wait for the dependency services to be available even though
> >>> they are on different systems.
> >>> 
> >>> All the dependency systems in the init-systems I could find are all
> >>> based on dependencies on the same server. Does anyone know of something
> >>> that can already provide this type of dependencies? Or do I need to
> >>> write something myself?
> >> 
> >> With systemd you can add ExecStartPre=/some/script to the service's unit
> >> file where /some/script waits for the remote services to become
> >> available,
> >> and possibly return an error if the service does not become available
> >> within a set time.
> > 
> > That method works for any init-system and writing a script to check and if
> > necessary fail is my temporary fall-back plan. I was actually hoping for a
> > method that can be used to monitor availability and, if necessary, stop
> > services when the dependencies disappear.
> > 
> > --
> > Joost
> 
> the difficulty is in identifying failed services.
> local network issue / load issue could mean your services start bouncing.
> the best way is to have redundancy so it doesn't matter as much

I know that. A proper system for this would have a configurable amount of 
retries with a wait-time in between.

> having said all of that::
> 
> systemd will start servers and buffer network activity - how this works
> for non local services would be interesting to see.

It would, but I am not going to migrate my servers to something like systemd 
without a clear and proven advantage. For me, that currently does not exist.
It also would not work as not all the software I run will happily wait while 
the rest of the stack starts.
I would end up in a bigger mess thanks to timeout issues during startup.

> with openrc :
> you could on the DNS server have a service which is just a batch script
> that uses watches for pid / program path in "ps" which outputs ACK or
> NAK to a file in an NFS share  say /nfs/monitoring/dns

Yes, but in order to access the NFS share, I need DNS to be running. Chicken-
egg problem.

> then on the mail server you could have a service that polls
> /nfs/monitoring/dns for NAK or ACK
> you can then choose to have this service directly start your dependent
> services, or if you adjust /etc/init.d/postfix to have depends =
> "mymonitorDNS" which is an empty shell of a service. your watchdog
> service could stop / start the empty shell of a script mymonitorDNS, and
> then postfix depends on mymonitorDNS
> this would save you from "i've just stopped the mail server for
> maintenance and my watchdogservice has just restarted it due to a
> NAK>ACK event"

That is the problem I have with these watchdog services. During boot, I want 
it to wait. But it needs to understand not to start a service when I stopped 
it during runtime.
Otherwise it could prevent a clean shutdown as well...

> or...
> you could have a central master machine which has it's own services,
> watchdog and monitor... i.e. /etc/init.d/thepostfixserver start  /
> depends on thednsserver which just runs
> # ssh postfixserver '/etc/init.d/postfix start'
> 
> or...
> puppet and it's kin

Last time I looked at puppet, it seemed too complex for what I need.
I will recheck it again.

Thanks,

Joost

Re: [gentoo-user] Cross system dependencies

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 342 bytes --]

On Sun, 29 Jun 2014 08:55:41 +0200, J. Roeleveld wrote:

> > or...
> > puppet and it's kin  
> 
> Last time I looked at puppet, it seemed too complex for what I need.
> I will recheck it again.

What about something like monit?


-- 
Neil Bothwick

Bug: (n.) any program feature not yet described to the marketing
department.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-user] Cross system dependencies

[J. Roeleveld]

On Sunday, June 29, 2014 09:35:33 AM Neil Bothwick wrote:
> On Sun, 29 Jun 2014 08:55:41 +0200, J. Roeleveld wrote:
> > > or...
> > > puppet and it's kin
> > 
> > Last time I looked at puppet, it seemed too complex for what I need.
> > I will recheck it again.
> 
> What about something like monit?

Hmm... I looked into that before, don't recall why I didn't look into it 
properly before.

Just had a look on the website, it looks usable, will need to check this.
Will also replace nagios at the same time, which I find ok, but don't really 
like it.

I might open a new thread at a later stage when I get round to trying it.

Thanks,

Joost