Last modified: 2014-10-08 02:50:17 UTC
Cannot save any edit on Betalabs, the following error is appearing on the save dialog: Exception Caught: DB connection error: Can't connect to MySQL server on '10.68.16.193' (4) (10.68.16.193)
Cannot open VE for any page as well, showing the error: Error loading data from server: readonly: The wiki is currently in read-only mode. Would you like to retry?
https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/4588/ https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/label=deployment-bastion-eqiad,wikidb=enwiki/4588/console > DB connection error: Can't connect to MySQL server on '10.68.16.193' (4) (10.68.16.193)
All browser tests will fail, all users/developers will complain. Marc/Andrew/Yuvi: Anything going on that might explain this? Any changes from production that might have caused havoc? Sean: adding you because the database update script is failing on beta: any changes you made recently that might effect this?
(due to a mid-air collision, re-adding Sean. Sean, see above)
So... There was a 'rogue' mysql server on deployment-db1, which wasn't being recognized by the init script for... some reason. I killed it manually, and started mysql again, and all seems well... This *might* have been because of the virt1005 failure related restart earlier (that is pure speculation). Should investigate more deeply if this happens again...
Thanks Yuvi! (Sorry for the pings Marc/Andrew/Sean :) )
