Last modified: 2014-03-20 21:27:16 UTC
For three days now, the database (tools-db) has been extremely slow. The past two days, I and other have really noticed the slowness between 6z-12z. Web pages would return with unable to connect to the database. I've been watching it more closely the past 12 hours. Every time it gets real slow and when I remember to look, Icinga has been reporting load average above 2.5. The higher the load average, the slower it gets. It has also been reporting that it is out of memory all day. With programs using memory as a "cache" and releasing it when needed (ie NFS), not sure how accurate that is. It is not a problem with the webserver. Non database web page work just fine. Programs, with database access on the queue, are also extremely slow. Programs that normally take three hours are taking eight hours.
Indeed, tools-db was meant as a temporary measure (it is a VM, which sucks for a database) and is a bit underpowered for the current load. The good news is that the Labs move to Ashburn also means dedicated hardware for tools-db, which should help a great deal.
Assuming this has been fixed by the move to eqiad.
