[gentoo-dev] bugs.gentoo.org status report, 2009/03/14 06h00 UTC

[gentoo-dev] bugs.gentoo.org status report, 2009/03/14 06h00 UTC

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 1467 bytes --]

Yes guys, I know that Bugzilla is down.

Last night, while I was sleeping, we got a slew of IPs hitting the
dependency graph generation. This wouldn't have been a problem normally,
but they seemed to hit graphs that took an inordinate amount of memory
to generate with GraphViz (collectively 8GiB of RAM and 32GiB of swap).
This morning, I got into the box, saw some OOMs of the GraphViz
processes, killed off the remaining ones, banned the IPs, and then had
to rush off to a work meeting.

I came back this evening, to find the box not responding again, and my
last SSH shell was painfully slow then just hung - not died, but hung,
the TCP connection is still alive, but the shell isn't responding
(shortly after I had seen a loadavg exceeding 1k).

Whomever attacked it came back I think. And I can't get in to block them
right now.  I've contacted the sponsor so that they can hard reboot the
box for me, but I don't expect any action from them for the next 5-6
hours at least.

Meantime, I'm enacting a plan B, to at least get us some slow Bugzilla
functionality, via the second bugzilla box that normally runs the
background computations (duplicates etc). I do however fully expect
whomever the attacker is to come right back at it - so I'm turning off
the dependency graphs.

-- 
Robin Hugh Johnson
Gentoo Linux Developer & Infra Guy
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

[-- Attachment #2: Type: application/pgp-signature, Size: 330 bytes --]

[gentoo-dev] bugs.gentoo.org status report, 2009/03/19 10h00 UTC

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 1228 bytes --]

The primary Bugzilla webserver is now back in operation.

Additionally, for the moment, I've re-enabled the load-balancing, but
note that it comes with a warning...
Load balanced bugzilla webservers:
http://bugs-web-lb.gentoo.org/
(HTTPS supported as well, but the SSL certificate won't match).

Visiting either specific side of the webserver nodes:
http://bugs-web1.gentoo.org/
http://bugs-web2.gentoo.org/
(The web node you're on is listed on the frontpage only).

Caveat:
- Why can't we just always use the load-balancer?
Unfortunately bugzilla writes a number of files to the local disk and
then gives you a URL to them. If the file was written to disk on web1,
but your request was delivered to web2, then you would get a 404 error.
- What pages are most affected by this?
The major ones I know about so far are the graph outputs from reports,
and the bug dependency graphs (which are disabled at the moment due to
the recent abuse). If you're using the load-balanced version of the
site, and you get a 404, please file a bug to bugzilla@

-- 
Robin Hugh Johnson
Gentoo Linux Developer & Infra Guy
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

[-- Attachment #2: Type: application/pgp-signature, Size: 330 bytes --]

Re: [gentoo-dev] bugs.gentoo.org status report, 2009/03/19 10h00 UTC

[Ramon van Alteren]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Robin H. Johnson wrote:
> The primary Bugzilla webserver is now back in operation.
> 
> Additionally, for the moment, I've re-enabled the load-balancing, but
> note that it comes with a warning...
> Load balanced bugzilla webservers:
> http://bugs-web-lb.gentoo.org/
> (HTTPS supported as well, but the SSL certificate won't match).
> 
> Visiting either specific side of the webserver nodes:
> http://bugs-web1.gentoo.org/
> http://bugs-web2.gentoo.org/
> (The web node you're on is listed on the frontpage only).
> 
> Caveat:
> - Why can't we just always use the load-balancer?
> Unfortunately bugzilla writes a number of files to the local disk and
> then gives you a URL to them. If the file was written to disk on web1,
> but your request was delivered to web2, then you would get a 404 error.

Robbat, would persistency on loadbalancer level solve this problem ?
In that case a tcp-connect that has been build stays with that
real-server instance in the loadbalancer, provided that data from the
same ip is coming in below a specified timeout.

We've used this in the past when we still used disk-based sessions in
our webapp. It works well, but can create hotspots in your webfarm if a
large percentage of your userbase is behind a single NATed gateway.

It would also limit your attacker to a single host.

Ramon
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (Darwin)

iEYEARECAAYFAknQnNoACgkQwiVM6CtDHQ1zwgCfZfEXwjZ9a0y7mHjq7A5MAxTo
HPIAn17SCBu0M71j6UBH8uW+7bVpMUnD
=gzHX
-----END PGP SIGNATURE-----