[gentoo-user] Postgres suddenly can't access files in its /etc directory

[gentoo-user] Postgres suddenly can't access files in its /etc directory

[napalm]

[-- Attachment #1: Type: text/plain, Size: 1204 bytes --]

I'm unsure if I should be posting this to the -hardened mailing list as
I'm using the hardened profile but all of a sudden I'm getting a rather
strange error when trying to start postgres.

# /etc/init.d/postgresql-9.1 start
 * Caching service dependencies ...                                      [ ok ]
 * The following file(s) are not readable by 'postgres':
 *     /etc/postgresql-9.1/postgresql.conf
 *     /etc/postgresql-9.1/pg_ident.conf
 *     /etc/postgresql-9.1/pg_hba.conf
 * HINT: Try: 'chmod 644 /etc/postgresql-9.1/*.conf'
 * ERROR: postgresql-9.1 failed to start

That's what I'm getting when I attempt to start it and I don't seem to have modified anything.

Looking into the init script I can see it's doing su postgres -c "test -r /etc/postgresql-9.1/pg_hba.conf" and the like but the output of:
	su postgres -c "test -r /etc/postgresql-9.1/pg_hba.conf" || echo "fail"	
is fail... so I'm quite at a loss as to what could be going on here. All of the files are owned by postgres, have the correct permissions (I ran chmod 644 as it hinted) and it should be able to traverse to the directory as everything has the execute bit from /etc onwards.

Any tips?

Cheers!

[-- Attachment #2: Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-user] Postgres suddenly can't access files in its /etc directory

[Michael Mol]

On Mon, Apr 23, 2012 at 9:21 AM,  <napalm@squareownz.org> wrote:
> I'm unsure if I should be posting this to the -hardened mailing list as
> I'm using the hardened profile but all of a sudden I'm getting a rather
> strange error when trying to start postgres.
>
> # /etc/init.d/postgresql-9.1 start
>  * Caching service dependencies ...                                      [ ok ]
>  * The following file(s) are not readable by 'postgres':
>  *     /etc/postgresql-9.1/postgresql.conf
>  *     /etc/postgresql-9.1/pg_ident.conf
>  *     /etc/postgresql-9.1/pg_hba.conf
>  * HINT: Try: 'chmod 644 /etc/postgresql-9.1/*.conf'
>  * ERROR: postgresql-9.1 failed to start
>
> That's what I'm getting when I attempt to start it and I don't seem to have modified anything.
>
> Looking into the init script I can see it's doing su postgres -c "test -r /etc/postgresql-9.1/pg_hba.conf" and the like but the output of:
>        su postgres -c "test -r /etc/postgresql-9.1/pg_hba.conf" || echo "fail"
> is fail... so I'm quite at a loss as to what could be going on here. All of the files are owned by postgres, have the correct permissions (I ran chmod 644 as it hinted) and it should be able to traverse to the directory as everything has the execute bit from /etc onwards.
>
> Any tips?

I tend to go at these things with a bludgeon. Capture and grep the
strace output for syscalls accessing those files, and you can see
exactly what error message the system gives. Or you might see that
it's not looking in the right place.

-- 
:wq

Re: [gentoo-user] Postgres suddenly can't access files in its /etc directory

[Michael Orlitzky]

On 04/23/12 09:21, napalm@squareownz.org wrote:
> I'm unsure if I should be posting this to the -hardened mailing list as
> I'm using the hardened profile but all of a sudden I'm getting a rather
> strange error when trying to start postgres.
> 
> # /etc/init.d/postgresql-9.1 start
>  * Caching service dependencies ...                                      [ ok ]
>  * The following file(s) are not readable by 'postgres':
>  *     /etc/postgresql-9.1/postgresql.conf
>  *     /etc/postgresql-9.1/pg_ident.conf
>  *     /etc/postgresql-9.1/pg_hba.conf
>  * HINT: Try: 'chmod 644 /etc/postgresql-9.1/*.conf'
>  * ERROR: postgresql-9.1 failed to start
> 
> That's what I'm getting when I attempt to start it and I don't seem to have modified anything.
> 
> Looking into the init script I can see it's doing su postgres -c "test -r /etc/postgresql-9.1/pg_hba.conf" and the like but the output of:
> 	su postgres -c "test -r /etc/postgresql-9.1/pg_hba.conf" || echo "fail"	
> is fail... so I'm quite at a loss as to what could be going on here. All of the files are owned by postgres, have the correct permissions (I ran chmod 644 as it hinted) and it should be able to traverse to the directory as everything has the execute bit from /etc onwards.
> 

You can `su postgres` and then try to read the files yourself. That
should reveal the problem.

Re: [gentoo-user] Postgres suddenly can't access files in its /etc directory

[napalm]

[-- Attachment #1: Type: text/plain, Size: 1607 bytes --]

On Mon, Apr 23, 2012 at 10:43:18AM -0400, Michael Orlitzky wrote:
> On 04/23/12 09:21, napalm@squareownz.org wrote:
> > I'm unsure if I should be posting this to the -hardened mailing list as
> > I'm using the hardened profile but all of a sudden I'm getting a rather
> > strange error when trying to start postgres.
> > 
> > # /etc/init.d/postgresql-9.1 start
> >  * Caching service dependencies ...                                      [ ok ]
> >  * The following file(s) are not readable by 'postgres':
> >  *     /etc/postgresql-9.1/postgresql.conf
> >  *     /etc/postgresql-9.1/pg_ident.conf
> >  *     /etc/postgresql-9.1/pg_hba.conf
> >  * HINT: Try: 'chmod 644 /etc/postgresql-9.1/*.conf'
> >  * ERROR: postgresql-9.1 failed to start
> > 
> > That's what I'm getting when I attempt to start it and I don't seem to have modified anything.
> > 
> > Looking into the init script I can see it's doing su postgres -c "test -r /etc/postgresql-9.1/pg_hba.conf" and the like but the output of:
> > 	su postgres -c "test -r /etc/postgresql-9.1/pg_hba.conf" || echo "fail"	
> > is fail... so I'm quite at a loss as to what could be going on here. All of the files are owned by postgres, have the correct permissions (I ran chmod 644 as it hinted) and it should be able to traverse to the directory as everything has the execute bit from /etc onwards.
> > 
> 
> You can `su postgres` and then try to read the files yourself. That
> should reveal the problem.
> 

Interestingly I can't `su postgres` even though I have set it a shell,
thanks for the tip though, I'm getting closer!

[-- Attachment #2: Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-user] Postgres suddenly can't access files in its /etc directory

[Michael Orlitzky]

On 04/23/12 11:44, napalm@squareownz.org wrote:
>>
>> You can `su postgres` and then try to read the files yourself. That
>> should reveal the problem.
>>
> 
> Interestingly I can't `su postgres` even though I have set it a shell,
> thanks for the tip though, I'm getting closer!

You can also try `su - postgres` which will attempt to switch to
postgres's home directory. Is that readable/executable?

Re: [gentoo-user] Postgres suddenly can't access files in its /etc directory

[napalm]

[-- Attachment #1: Type: text/plain, Size: 830 bytes --]

On Mon, Apr 23, 2012 at 11:56:47AM -0400, Michael Orlitzky wrote:
> On 04/23/12 11:44, napalm@squareownz.org wrote:
> >>
> >> You can `su postgres` and then try to read the files yourself. That
> >> should reveal the problem.
> >>
> > 
> > Interestingly I can't `su postgres` even though I have set it a shell,
> > thanks for the tip though, I'm getting closer!
> 
> You can also try `su - postgres` which will attempt to switch to
> postgres's home directory. Is that readable/executable?
> 

Postgres doesn't have a home directory and if I create one and chown it
postgres:postgres I still can't do anything. I'm totally at a loss here.

Here's the strace output from `su - postgres`:
http://pastie.org/private/dilrgts7xqrafxu15widga

I'm so confused at the moment!
Thanks for the help so far though guys.

[-- Attachment #2: Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-user] Postgres suddenly can't access files in its /etc directory

[Graham Murray]

napalm@squareownz.org writes:

> Postgres doesn't have a home directory and if I create one and chown it
> postgres:postgres I still can't do anything. I'm totally at a loss here.

Postgres should have a home directory - /var/lib/postgresql

If you run su - postgres, this is the directory you should be in.

Re: [gentoo-user] Postgres suddenly can't access files in its /etc directory

[napalm]

[-- Attachment #1: Type: text/plain, Size: 1202 bytes --]

On Tue, Apr 24, 2012 at 06:27:22AM +0100, Graham Murray wrote:
> napalm@squareownz.org writes:
> 
> > Postgres doesn't have a home directory and if I create one and chown it
> > postgres:postgres I still can't do anything. I'm totally at a loss here.
> 
> Postgres should have a home directory - /var/lib/postgresql
> 
> If you run su - postgres, this is the directory you should be in.
> 
Ah okay, I've changed it back to this then. I do indeed enter there now.
I've emerged it without threads or pam as I suspected it may have been
either of them but that hasn't seemed to solve much of anything.

The current error I'm getting is:
# /etc/init.d/postgresql-9.1 start
 * Starting PostgreSQL ...
 * start-stop-daemon: did not create a valid pid in
 * `/var/lib/postgresql/9.1/data/postmaster.pid'
 * Check the PostgreSQL 9.1 log for a detailed explanation of the
 * above error.                [ !! ]

Which is what happens when start-stop-daemon fails to execute its
command. I'm not entirely sure what start-stop-daemon is or what
permissions it may need or be missing so I'm about to look into that.

I seem to be getting somewhere at least.

Thanks again for the help!

[-- Attachment #2: Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-user] Postgres suddenly can't access files in its /etc directory

[Michael Orlitzky]

On 04/24/12 05:31, napalm@squareownz.org wrote:
> On Tue, Apr 24, 2012 at 06:27:22AM +0100, Graham Murray wrote:
>> napalm@squareownz.org writes:
>>
>>> Postgres doesn't have a home directory and if I create one and chown it
>>> postgres:postgres I still can't do anything. I'm totally at a loss here.
>>
>> Postgres should have a home directory - /var/lib/postgresql
>>
>> If you run su - postgres, this is the directory you should be in.
>>
> Ah okay, I've changed it back to this then. I do indeed enter there now.
> I've emerged it without threads or pam as I suspected it may have been
> either of them but that hasn't seemed to solve much of anything.
> 
> The current error I'm getting is:
> # /etc/init.d/postgresql-9.1 start
>  * Starting PostgreSQL ...
>  * start-stop-daemon: did not create a valid pid in
>  * `/var/lib/postgresql/9.1/data/postmaster.pid'
>  * Check the PostgreSQL 9.1 log for a detailed explanation of the
>  * above error.                [ !! ]
> 
> Which is what happens when start-stop-daemon fails to execute its
> command. I'm not entirely sure what start-stop-daemon is or what
> permissions it may need or be missing so I'm about to look into that.
> 

Is everything under (and including) /var/lib/postgresql owned by
postgres:postgres?

  ~ # ls /var/lib/postgresql/9.1
  total 4.0K
  drwx------ 13 postgres postgres 4.0K 2012-04-23 18:58 data

  ~ # ls /var/lib/postgresql/9.1/data/
  total 1.2M
  -rw------- 1 postgres postgres    4 2012-02-14 00:14 PG_VERSION
  drwx------ 7 postgres postgres 4.0K 2012-04-23 10:31 base
  drwx------ 2 postgres postgres 4.0K 2012-04-23 18:59 global
  drwx------ 2 postgres postgres 4.0K 2012-02-14 00:14 pg_clog
  drwx------ 4 postgres postgres 4.0K 2012-02-14 00:14 pg_multixact
  drwx------ 2 postgres postgres 4.0K 2012-04-23 18:58 pg_notify
  drwx------ 2 postgres postgres 4.0K 2012-02-14 00:14 pg_serial
  drwx------ 2 postgres postgres 4.0K 2012-04-24 09:57 pg_stat_tmp
  drwx------ 2 postgres postgres 4.0K 2012-04-20 13:42 pg_subtrans
  drwx------ 2 postgres postgres 4.0K 2012-02-14 00:14 pg_tblspc
  drwx------ 2 postgres postgres 4.0K 2012-02-14 00:14 pg_twophase
  drwx------ 3 postgres postgres 4.0K 2012-04-23 11:36 pg_xlog
  -rw------- 1 postgres postgres 1.1M 2012-04-24 09:55 postmaster.log
  -rw------- 1 postgres postgres  134 2012-04-23 18:58 postmaster.opts
  -rw------- 1 postgres postgres   92 2012-04-23 18:58 postmaster.pid

Re: [gentoo-user] Postgres suddenly can't access files in its /etc directory

[Stroller]

On 23 April 2012, at 22:18, napalm@squareownz.org wrote:
> … 
>  and if I create one and chown it
> postgres:postgres I still can't do anything. I'm totally at a loss here.

If you want to say something like "Postgres doesn't have a home directory", please post the output of `grep -i postgres /etc/passwd`. Maybe also the output of `echo ~postgres`.

Right now I assume you're looking in /home and not seeing /home/postgres. I assume you're running `mkdir /home/postgres`. This is not how home directories are created and allocated. 

Stroller.

Re: [gentoo-user] Postgres suddenly can't access files in its /etc directory

[J. Roeleveld]

On Mon, April 23, 2012 3:21 pm, napalm@squareownz.org wrote:
> I'm unsure if I should be posting this to the -hardened mailing list as
> I'm using the hardened profile but all of a sudden I'm getting a rather
> strange error when trying to start postgres.
>
> # /etc/init.d/postgresql-9.1 start
>  * Caching service dependencies ...                                      [
> ok ]
>  * The following file(s) are not readable by 'postgres':
>  *     /etc/postgresql-9.1/postgresql.conf
>  *     /etc/postgresql-9.1/pg_ident.conf
>  *     /etc/postgresql-9.1/pg_hba.conf
>  * HINT: Try: 'chmod 644 /etc/postgresql-9.1/*.conf'
>  * ERROR: postgresql-9.1 failed to start
>
> That's what I'm getting when I attempt to start it and I don't seem to
> have modified anything.
>
> Looking into the init script I can see it's doing su postgres -c "test -r
> /etc/postgresql-9.1/pg_hba.conf" and the like but the output of:
> 	su postgres -c "test -r /etc/postgresql-9.1/pg_hba.conf" || echo "fail"
> is fail... so I'm quite at a loss as to what could be going on here. All
> of the files are owned by postgres, have the correct permissions (I ran
> chmod 644 as it hinted) and it should be able to traverse to the directory
> as everything has the execute bit from /etc onwards.
>
> Any tips?

I don't have much experience with Hardenened, but are you certain that any
permissions (including ACLs) are set correctly for PostgreSQL to access
all its files?

Do you have "sec-policy/selinux-postgresql" installed? And did you
re-emerge this after the update?

--
Joost

Re: [gentoo-user] Postgres suddenly can't access files in its /etc directory

[napalm]

[-- Attachment #1: Type: text/plain, Size: 2916 bytes --]

On Thu, Apr 26, 2012 at 07:46:10AM +0200, J. Roeleveld wrote:
> On Mon, April 23, 2012 3:21 pm, napalm@squareownz.org wrote:
> > I'm unsure if I should be posting this to the -hardened mailing list as
> > I'm using the hardened profile but all of a sudden I'm getting a rather
> > strange error when trying to start postgres.
> >
> > # /etc/init.d/postgresql-9.1 start
> >  * Caching service dependencies ...                                      [
> > ok ]
> >  * The following file(s) are not readable by 'postgres':
> >  *     /etc/postgresql-9.1/postgresql.conf
> >  *     /etc/postgresql-9.1/pg_ident.conf
> >  *     /etc/postgresql-9.1/pg_hba.conf
> >  * HINT: Try: 'chmod 644 /etc/postgresql-9.1/*.conf'
> >  * ERROR: postgresql-9.1 failed to start
> >
> > That's what I'm getting when I attempt to start it and I don't seem to
> > have modified anything.
> >
> > Looking into the init script I can see it's doing su postgres -c "test -r
> > /etc/postgresql-9.1/pg_hba.conf" and the like but the output of:
> > 	su postgres -c "test -r /etc/postgresql-9.1/pg_hba.conf" || echo "fail"
> > is fail... so I'm quite at a loss as to what could be going on here. All
> > of the files are owned by postgres, have the correct permissions (I ran
> > chmod 644 as it hinted) and it should be able to traverse to the directory
> > as everything has the execute bit from /etc onwards.
> >
> > Any tips?
> 
> I don't have much experience with Hardenened, but are you certain that any
> permissions (including ACLs) are set correctly for PostgreSQL to access
> all its files?
> 
> Do you have "sec-policy/selinux-postgresql" installed? And did you
> re-emerge this after the update?
> 
> --
> Joost
> 
I got things working in the end by deleting everything to do with
postgres, re-emerging and then restoring from a backup (it's fine
because the database is only updated a few times a day).

Still totally confused as to what the issue was. I hadn't been fiddling
with permissions or anything at all, didn't even go near the postgres
config files and there was no update to postgres so I'm just at a loss.

I don't have sec-policy/selinux-postgresql installed, more using PaX and
GRSecurity than selinux on my current installation, doubt that would
have helped.

I'm a bit annoyed that I couldn't solve the issue without doing the sort
of "turn it off and on" approach but it has done the trick so I guess
that's that.

I must have messed something up somewhere. Any guess as to if PAM or a
glibc update could have broken it? I wouldn't have thought glibc but I'm
a little clueless when it comes to PAM, then again I tried emerging
(without deleting everything) with USE="-pam" to no avail.

Anyway thanks for the help everyone, sorry I can't give a better
diagnosis. I did check strace logs and everything, couldn't locate the
error. Blargh!

Cheers,
David

[-- Attachment #2: Type: application/pgp-signature, Size: 316 bytes --]