[gentoo-user] All of a sudden, no apache2. No clue why.

[gentoo-user] All of a sudden, no apache2. No clue why.

[Kevin O'Gorman]

[-- Attachment #1: Type: text/plain, Size: 2406 bytes --]

As of today, my apache2 web server seems to refuse to start.  I've tried a
system reboot, to no avail --
connections are refused on port 80.

In /etc/init.d it looks like this: if I try to start it, it says it's
already started.  netstat says there's no listener
on port 80.  If I try to restart it, it cannot start a listener.  I'm really
bummed.

Any ideas how to get this going again?

Here's a short look at what I was doing in /etc/init.d

treat init.d # ./apache2 start
 * WARNING:  apache2 has already been started.
treat init.d # ./apache2 restart
 * Stopping apache2 ...
httpd (no pid file) not
running                                                       [ ok ]
 * Starting apache2 ...
(98)Address already in use: make_sock: could not bind to address
64.166.164.49:80
no listening sockets available, shutting down
Unable to open
logs                                                                   [ ok
]
treat init.d # ./apache2 start
 * WARNING:  apache2 has already been started.
treat init.d # netstat -l --inet
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address
State
tcp        0      0 *:printer               *:*
LISTEN
tcp        0      0 *:2222                  *:*
LISTEN
tcp        0      0 t:domain                *:*
LISTEN
tcp        0      0 treat.kosmanor.c:domain *:*
LISTEN
tcp        0      0 localhost:domain        *:*
LISTEN
tcp        0      0 *:ipp                   *:*
LISTEN
tcp        0      0 *:smtp                  *:*
LISTEN
tcp        0      0 localhost:rndc          *:*
LISTEN
netstat: no support for `AF INET (sctp)' on this system.
treat init.d # netstat -l --inet -n
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address
State
tcp        0      0 0.0.0.0:515             0.0.0.0:*
LISTEN
tcp        0      0 0.0.0.0:2222            0.0.0.0:*
LISTEN
tcp        0      0 192.168.1.149:53        0.0.0.0:*
LISTEN
tcp        0      0 64.166.164.49:53        0.0.0.0:*
LISTEN
tcp        0      0 127.0.0.1:53            0.0.0.0:*
LISTEN
tcp        0      0 0.0.0.0:631             0.0.0.0:*
LISTEN
tcp        0      0 0.0.0.0:25              0.0.0.0:*
LISTEN
tcp        0      0 127.0.0.1:953           0.0.0.0:*
LISTEN
netstat: no support for `AF INET (sctp)' on this system.
treat init.d #


Any ideas?  What else could I look at?

-- 
Kevin O'Gorman, PhD

[-- Attachment #2: Type: text/html, Size: 6940 bytes --]

Re: [gentoo-user] All of a sudden, no apache2. No clue why.

[Tomas Krasnican]

Kevin O'Gorman wrote:
> As of today, my apache2 web server seems to refuse to start.  I've tried
> a system reboot, to no avail --
> connections are refused on port 80.

I think that apache will try to create listener on address:port, which have already created (because it is possibly defined that).
After this fail apache will die and you can't see any active listener on this port in netstat -l, thats correct.

Try to check for twice definition of the same listener. 

Or, example, if you have listener for 0.0.0.0:80 and you trying to create listener for 1.2.3.4.80. 

...or dubble include of the same configuration file? I don't know, if is it possible...

Regards,
Tomas Krasnican

Re: [gentoo-user] All of a sudden, no apache2. No clue why.

[Kyle Bader]

>  * Starting apache2 ...
> (98)Address already in use: make_sock: could not bind to address
> 64.166.164.49:80
> no listening sockets available, shutting down
> Unable to open
> logs                                                                   [ ok
> ]

Make sure an interface is listening on that address:

ip a |grep 64.166.164.49

Check for bound processes:

lsof -i @64.166.164.49

If that fails I'd strace the startup manually.

-- 

Kyle

Re: [gentoo-user] All of a sudden, no apache2. No clue why.

[William Kenworthy]

There is an "Unable to open logs" in there ... are you doing some fancy
remote logging that cant start? - I cant see anything that says its
specifically port 80 thats causing the problem, just no listening
sockets available.

syslog is udp port 514, and then there is ssl on 443.

BillK



On Thu, 2010-07-29 at 14:24 -0700, Kevin O'Gorman wrote:
> As of today, my apache2 web server seems to refuse to start.  I've
> tried a system reboot, to no avail --
> connections are refused on port 80.
> 
> In /etc/init.d it looks like this: if I try to start it, it says it's
> already started.  netstat says there's no listener
> on port 80.  If I try to restart it, it cannot start a listener.  I'm
> really bummed.
> 
> Any ideas how to get this going again?
> 
> Here's a short look at what I was doing in /etc/init.d
> 
> treat init.d # ./apache2 start
>  * WARNING:  apache2 has already been started.
> treat init.d # ./apache2 restart
>  * Stopping apache2 ...
> httpd (no pid file) not running
> [ ok ]
>  * Starting apache2 ...
> (98)Address already in use: make_sock: could not bind to address
> 64.166.164.49:80
> no listening sockets available, shutting down
> Unable to open logs
> [ ok ]

Re: [gentoo-user] All of a sudden, no apache2. No clue why.

[Kevin O'Gorman]

[-- Attachment #1: Type: text/plain, Size: 1011 bytes --]

On Thu, Jul 29, 2010 at 3:12 PM, Tomas Krasnican <krasko@krasko.sk> wrote:

> Kevin O'Gorman wrote:
> > As of today, my apache2 web server seems to refuse to start.  I've tried
> > a system reboot, to no avail --
> > connections are refused on port 80.
>
> I think that apache will try to create listener on address:port, which have
> already created (because it is possibly defined that).
> After this fail apache will die and you can't see any active listener on
> this port in netstat -l, thats correct.
>
> Try to check for twice definition of the same listener.
>
> Or, example, if you have listener for 0.0.0.0:80 and you trying to create
> listener for 1.2.3.4.80.
>
> ...or dubble include of the same configuration file? I don't know, if is it
> possible...
>
> Regards,
> Tomas Krasnican
>

Thanks, but...

Grepping for Listen, all I see is

hexDirs.conf:Listen 64.166.164.49:80
hexDirs.conf:Listen localhost:80

And grepping for Includes, I find nothing suspicious or cyclic.

-- 
Kevin O'Gorman, PhD

[-- Attachment #2: Type: text/html, Size: 1527 bytes --]

Re: [gentoo-user] All of a sudden, no apache2. No clue why.

[Kevin O'Gorman]

[-- Attachment #1: Type: text/plain, Size: 2012 bytes --]

On Thu, Jul 29, 2010 at 3:14 PM, Kyle Bader <kyle.bader@gmail.com> wrote:

> >  * Starting apache2 ...
> > (98)Address already in use: make_sock: could not bind to address
> > 64.166.164.49:80
> > no listening sockets available, shutting down
> > Unable to open
> > logs                                                                   [
> ok
> > ]
>
> Make sure an interface is listening on that address:
>
> ip a |grep 64.166.164.49
>
> treat apache2 # ip a | grep 64.166.164.49
    inet 64.166.164.49/29 brd 64.166.164.55 scope global eth0
treat apache2 #
That's no listener.

Check for bound processes:
>
> lsof -i @64.166.164.49
>

treat apache2 # lsof -i @64.166.164.49
COMMAND  PID  USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
named   4725 named   21u  IPv4   8406      0t0  TCP treat.kosmanor.com:domain
(LISTEN)
named   4725 named  513u  IPv4   8405      0t0  UDP treat.kosmanor.com:domain

ntpd    5117   ntp   18u  IPv4   9045      0t0  UDP treat.kosmanor.com:ntp
firefox 7832 kevin   27u  IPv4  29522      0t0  TCP treat.kosmanor.com:57043
->nuq04s01-in-f83.1e100.net:https (ESTABLISHED)
firefox 7832 kevin   56u  IPv4  29352      0t0  TCP treat.kosmanor.com:57034
->nuq04s01-in-f83.1e100.net:https (ESTABLISHED)
firefox 7832 kevin   58u  IPv4  29453      0t0  TCP treat.kosmanor.com:54324
->nuq04s01-in-f18.1e100.net:https (ESTABLISHED)
firefox 7832 kevin   63u  IPv4  29536      0t0  TCP treat.kosmanor.com:59436
->nuq04s01-in-f102.1e100.net:http (ESTABLISHED)
firefox 7832 kevin   66u  IPv4  29538      0t0  TCP treat.kosmanor.com:56773
->74.125.164.30:http (ESTABLISHED)
firefox 7832 kevin   72u  IPv4  29475      0t0  TCP treat.kosmanor.com:37415
->mg201a.mail.vip.mud.yahoo.com:http (CLOSE_WAIT)
treat apache2 #

And that's a DNS listener, an NTP listener, and firefox as a client, not a
listener.  Though it makes me want to track down 1e100.net and find out who
they are.

I'll see about strace.


> If that fails I'd strace the startup manually.
>
> --
>
> Kyle
>



-- 
Kevin O'Gorman, PhD

[-- Attachment #2: Type: text/html, Size: 3073 bytes --]

Re: [gentoo-user] All of a sudden, no apache2. No clue why.

[Kevin O'Gorman]

[-- Attachment #1.1: Type: text/plain, Size: 3995 bytes --]

> On Thu, 2010-07-29 at 14:24 -0700, Kevin O'Gorman wrote:
> > As of today, my apache2 web server seems to refuse to start.  I've
> > tried a system reboot, to no avail --
> > connections are refused on port 80.
> >
> > In /etc/init.d it looks like this: if I try to start it, it says it's
> > already started.  netstat says there's no listener
> > on port 80.  If I try to restart it, it cannot start a listener.  I'm
> > really bummed.
> >
> > Any ideas how to get this going again?
> >
> > Here's a short look at what I was doing in /etc/init.d
> >
> > treat init.d # ./apache2 start
> >  * WARNING:  apache2 has already been started.
> > treat init.d # ./apache2 restart
> >  * Stopping apache2 ...
> > httpd (no pid file) not running
> > [ ok ]
> >  * Starting apache2 ...
> > (98)Address already in use: make_sock: could not bind to address
> > 64.166.164.49:80
> > no listening sockets available, shutting down
> > Unable to open logs
> > [ ok ]
>
>
>
> On Thu, Jul 29, 2010 at 4:27 PM, William Kenworthy <billk@iinet.net.au>wrote:
There is an "Unable to open logs" in there ... are you doing some fancy
remote logging that cant start? - I cant see anything that says its
specifically port 80 thats causing the problem, just no listening
sockets available.

syslog is udp port 514, and then there is ssl on 443.

BillK

I'm doing nothing fancy, but I did have a nearly full root directory.  I
flushed out some
old portage stuff an I'm back to 19 GB free.  I still get the same result
messages during
a reboot.

It is true I modified the local apache config
/etc/apache2/kosmanor/hexDirs.conf slightly just before.  I added a /HexData
alias, like
a few others that I have.  I've attached the config file in case you can
find any unintended
change.

Looking at the logs, there does not seem to be logging going on since July
16, and things
were working then, I thought.  List of log directory follows.  I do seem to
have two different styles of log rotation going on simultaneously, but that
seems to have been the case for
quite a while now; I'll find that later.

treat apache2 # ls -l
total 4368
-rw-rw-rw- 1 root root      0 Jul 16 03:10 access_log
-rw-rw-rw- 1 root root 191438 Jun 15  2009 access_log.1.gz
-rw-rw-rw- 1 root root 678879 Jun 25 03:10 access_log-20100625.gz
-rw-rw-rw- 1 root root 693424 Jul  2 03:10 access_log-20100702.gz
-rw-rw-rw- 1 root root 701307 Jul  9 03:10 access_log-20100709.gz
-rw-rw-rw- 1 root root 840730 Jul 16 03:10 access_log-20100716.gz
-rw-rw-rw- 1 root root 234663 Jun  8  2009 access_log.2.gz
-rw-rw-rw- 1 root root 270349 Jun  1  2009 access_log.3.gz
-rw-rw-rw- 1 root root 277761 May 25  2009 access_log.4.gz
-rw-rw-rw- 1 root root      0 Jul 16 03:10 error_log
-rw-rw-rw- 1 root root 117611 Jun 15  2009 error_log.1.gz.out
-rw-rw-rw- 1 root root  61608 Jun 25 03:10 error_log-20100625.gz
-rw-rw-rw- 1 root root  69397 Jul  2 03:10 error_log-20100702.gz
-rw-rw-rw- 1 root root 118085 Jul  9 03:10 error_log-20100709.gz
-rw-rw-rw- 1 root root 114433 Jul 16 03:10 error_log-20100716.gz
-rw-rw-rw- 1 root root   5706 Jun  8  2009 error_log.2.gz
-rw-rw-rw- 1 root root   5628 Jun  1  2009 error_log.3.gz
-rw-rw-rw- 1 root root   6344 May 25  2009 error_log.4.gz
-rw-rw-rw- 1 root root      0 Feb  3  2008 ssl_access_log
-rw-rw-rw- 1 root root      0 Feb  3  2008 ssl_error_log
-rw-rw-rw- 1 root root      0 Feb  3  2008 ssl_request_log
treat apache2 # pwd
/var/log/apache2
treat apache2 #


The ownership may seem odd, but agrees with backups.  That July 16 log ends
with
something familiar:

[Fri Jul 16 03:10:07 2010] [notice] SIGUSR1 received.  Doing graceful
restart
(98)Address already in use: make_sock: could not bind to address 0.0.0.0:80
no listening sockets available, shutting down
Unable to open logs
treat apache2 #

So I guess my page has been down for a while.  A nuisance at most because
it's just my
personal stuff, which a handful of people care about.  But I want it back
up, not to mention the logging.



-- 
Kevin O'Gorman, PhD

[-- Attachment #1.2: Type: text/html, Size: 8051 bytes --]

[-- Attachment #2: hexDirs.conf --]
[-- Type: text/plain, Size: 7151 bytes --]

#
# Listen: Allows you to bind Apache to specific IP addresses and/or
# ports, instead of the default. See also the <VirtualHost>
# directive.
#
# Change this to Listen on specific IP addresses as shown below to 
# prevent Apache from glomming onto all bound IP addresses (0.0.0.0)
#
#Listen 12.34.56.78:80
# Listen 80
# no need to listen promiscuously (I think)
Listen 64.166.164.49:80
Listen localhost:80

#
# If you wish httpd to run as a different user or group, you must run
# httpd as root initially and it will switch.  
#
# User/Group: The name (or #number) of the user/group to run httpd as.
#  . On SCO (ODT 3) use "User nouser" and "Group nogroup".
#  . On HPUX you may not be able to use shared memory as nobody, and the
#    suggested workaround is to create a user www and use that user.
#  NOTE that some kernels refuse to setgid(Group) or semctl(IPC_SET)
#  when the value of (unsigned)Group is above 60000; 
#  don't use Group #-1 on these systems!
#
User apache
Group apache

#
# ServerAdmin: Your address, where problems with the server should be
# e-mailed.  This address appears on some server-generated pages, such
# as error documents.  e.g. admin@your-domain.com
#
ServerAdmin webmaster@kosmanor.com

#
# ServerName gives the name and port that the server uses to identify itself.
# This can often be determined automatically, but we recommend you specify
# it explicitly to prevent problems during startup.
#
# If this is not set to valid DNS name for your host, server-generated
# redirections will not work.  See also the UseCanonicalName directive.
#
# If your host doesn't have a registered DNS name, enter its IP address here.
# You will have to access it by its address anyway, and this will make 
# redirections work in a sensible way.
#
ServerName www.kosmanor.com

#
# UseCanonicalName: Determines how Apache constructs self-referencing 
# URLs and the SERVER_NAME and SERVER_PORT variables.
# When set "Off", Apache will use the Hostname and Port supplied
# by the client.  When set "On", Apache will use the value of the
# ServerName directive.
#
UseCanonicalName Off


#
# Each directory to which Apache has access can be configured with respect
# to which services and features are allowed and/or disabled in that
# directory (and its subdirectories). 
#
# First, we configure the "default" to be a very restrictive set of 
# features.  
#
<Directory />
    Options FollowSymLinks
    AllowOverride None
</Directory>

#
# Note that from this point forward you must specifically allow
# particular features to be enabled - so if something's not working as
# you might expect, make sure that you have specifically enabled it
# below.
#

#
# UserDir: The name of the directory that is appended onto a user's home
# directory if a ~user request is received.
# enable by adding -D USERDIR to /etc/conf.d/apache2
#
<IfModule mod_userdir.c>
    UserDir public_html

#
# Control access to UserDir directories.  The following is an example
# for a site where these directories are restricted to read-only.
#
    <Directory /home/*/public_html>
        AllowOverride FileInfo AuthConfig Limit Indexes
        Options MultiViews Indexes SymLinksIfOwnerMatch IncludesNoExec
        <Limit GET POST OPTIONS PROPFIND>
            Order allow,deny
            Allow from all
       </Limit>
       <LimitExcept GET POST OPTIONS PROPFIND>
            Order deny,allow
            Deny from all
       </LimitExcept>
    </Directory>


# Enable this additional section if you would like to make use of a
# suexec-enabled cgi-bin directory on a per-user basis.
#
#<Directory /home/*/public_html/cgi-bin>
#    Options ExecCGI
#    SetHandler cgi-script
#</Directory>

</IfModule>


#
# DirectoryIndex: sets the file that Apache will serve if a directory
# is requested.
#
# The index.html.var file (a type-map) is used to deliver content-
# negotiated documents.  The MultiViews Option can be used for the 
# same purpose, but it is much slower.
#
DirectoryIndex index.html index.html.var

#
# AccessFileName: The name of the file to look for in each directory
# for additional configuration directives.  See also the AllowOverride 
# directive.
#
AccessFileName .htaccess

#
# The following lines prevent .htaccess and .htpasswd files from being 
# viewed by Web clients. 
#
<FilesMatch "^\.ht">
    Order allow,deny
    Deny from all
</FilesMatch>


# override
ServerSignature Off

#
# Aliases: Add here as many aliases as you need (with no limit). The format is 
# Alias fakename realname
#
# Note that if you include a trailing / on fakename then the server will
# require it to be present in the URL.  So "/icons" isn't aliased in this
# example, only "/icons/".  If the fakename is slash-terminated, then the 
# realname must also be slash terminated, and if the fakename omits the 
# trailing slash, the realname must also omit it.
#
# We include the /icons/ alias for FancyIndexed directory listings.  If you
# do not use FancyIndexing, you may comment this out.
#

<Location />
    Order allow,deny
    Allow from all
</Location>
	
Alias /hex  /hex/web
<Directory /hex/web>
    Options FollowSymLinks
    # charset most important for non 8859-1 sets.
    AddCharset iso-8859-1 .da .en_US .C
    AddCharset iso-8859-2 .pl
    Order allow,deny
    Allow from all
</Directory>

Alias /Oggie /Dogs/Oggie
<Directory /Oggie>
    Order allow,deny
    Allow from all
</Directory>

Alias /Gleni /Dogs/Gleni
<Directory /Gleni>
    Order allow,deny
    Allow from all
</Directory>

Alias /HexData /golem/golem
<Directory /HexData>
    Order allow,deny
    Allow from all
</Directory>


Alias /Cully /Dogs/Cully
<Directory /Cully>
    Order allow,deny
    Allow from all
</Directory>

Alias /theory /hex/theory
<Directory /hex/theory>
    Options FollowSymLinks
    AddCharset iso-8859-1 .da .en_US .C
    AddCharset iso-8859-2 .pl
    AuthUserFile /etc/apache2/kosmanor/passwords
    AuthGroupFile /dev/null
    AuthName "Hex Theory"
    AuthType Basic
    Require valid-user
    Order allow,deny
    Allow from all
</Directory>

ScriptAlias /hex-bin /hex/bin
<Directory /hex/bin>
    Options FollowSymLinks
#    AddHandler mod_python .py
#    PythonHandler hexscript
#    PythonDebug On
    Order allow,deny
    Allow from all
</Directory>


ScriptAlias /my-bin  /hex/hexTest
<Directory /hex/hexTest>
    AuthUserFile /etc/apache2/kosmanor/passwords
    AuthGroupFile /dev/null
    AuthName "OHex Advanced"
    AuthType Basic
    Require valid-user
    Options FollowSymLinks
</Directory>


#
# ScriptAlias: This controls which directories contain server scripts.
# ScriptAliases are essentially the same as Aliases, except that
# documents in the realname directory are treated as applications and
# run by the server when requested rather than as documents sent to the client.
# The same rules about trailing "/" apply to ScriptAlias directives as to
# Alias.

#
# Redirect allows you to tell clients about documents which used to exist in
# your server's namespace, but do not anymore. This allows you to tell the
# clients where to look for the relocated document.
# Example:
# Redirect permanent /foo http://www.example.com/bar

Re: [gentoo-user] All of a sudden, no apache2. No clue why.

[Kyle Bader]

>> >  * Starting apache2 ...
>> > (98)Address already in use: make_sock: could not bind to address
>> > 64.166.164.49:80
>> > no listening sockets available, shutting down
>> > Unable to open
>> > logs                                                                   [

Strace will probably reveal which log file can't be opened, something
like this will probably do the trick:

strace /path/to/apache2 -D <module list> -d /path/to/apache2dir

> And that's a DNS listener, an NTP listener, and firefox as a client, not a
> listener.  Though it makes me want to track down 1e100.net and find out who
> they are.

Google:

   Domain Name: 1E100.NET
   Registrar: MARKMONITOR INC.
   Whois Server: whois.markmonitor.com
   Referral URL: http://www.markmonitor.com
   Name Server: NS1.GOOGLE.COM
   Name Server: NS2.GOOGLE.COM
   Name Server: NS3.GOOGLE.COM
   Name Server: NS4.GOOGLE.COM

Registrant:
        DNS Admin
        Google Inc.
        1600 Amphitheatre Parkway
         Mountain View CA 94043
        US
        dns-admin@google.com +1.6502530000 Fax: +1.6506188571

-- 

Kyle

Re: [gentoo-user] All of a sudden, no apache2. No clue why.

[Kevin O'Gorman]

[-- Attachment #1: Type: text/plain, Size: 1010 bytes --]

On Fri, Jul 30, 2010 at 9:12 AM, Kyle Bader <kyle.bader@gmail.com> wrote:

> >> >  * Starting apache2 ...
> >> > (98)Address already in use: make_sock: could not bind to address
> >> > 64.166.164.49:80
> >> > no listening sockets available, shutting down
> >> > Unable to open
> >> > logs
> [
>
> Strace will probably reveal which log file can't be opened, something
> like this will probably do the trick:
>
> strace /path/to/apache2 -D <module list> -d /path/to/apache2dir
>
> > And that's a DNS listener, an NTP listener, and firefox as a client, not
> a
> > listener.  Though it makes me want to track down 1e100.net and find out
> who
> > they are.
>
> Google:
>
>
I should have known that.  1e100 is a mathematical/programming synonym for
google.
Amusing.

I do most of this in gmail, so those connections look normal.

I don't know what the usual module list is, so I guess I have to go trolling
throught the
init.d scripts to figure it out, unless somebody knows a better way.

-- 
Kevin O'Gorman, PhD

[-- Attachment #2: Type: text/html, Size: 1676 bytes --]