Last modified: 2014-10-16 20:09:57 UTC
Since updating vagrant I get this on a periodic basis. No idea why. Have to vagrant halt / vagrant up to fix.
I experience this problem, too.
Any particular workflow or steps to reproduce? If you can reproduce reliably, I'd be interested to know if `vagrant enable-role zend` (switching wiki runtime from hhvm to php5) makes the problem go away or not.
It's happening every hour or so for me. I've switched to zend, will report back my experience later.
After switching to zend I have not had this problem all day.
I am on zend and I think I get it when I reload a page multiple times in a short period of time.
Also occurs on Labs-Vagrant, apparently. Is there a way to resolve this there other than waiting for a few hours? As far as I know, no vagrant halt/vagrant up exists for Labs-Vagrant.
(In reply to Vogone from comment #6) > Also occurs on Labs-Vagrant, apparently. Is there a way to resolve this > there other than waiting for a few hours? As far as I know, no vagrant > halt/vagrant up exists for Labs-Vagrant. At worst in MediaWiki-Vagrant or Labs you should be able to clear any such errors by restarting the hhvm service(s): sudo service hhvm restart sudo service jobrunner restart I can't think of anything that would hold a db connection open otherwise. Getting to the root of the problem, someone needs to determine if this is an hhvm bug, a general MediaWiki bug or a tuning problem for the default MySQL install we provide. It may very well be that our use-cases have somehow outgrown the default database settings.
I get it. I'll investigate. It has something to do with hhvm keeping a massive number of open files.
OK! I smashed my instance (again again) and rebuilt it and it still fails just like this. The trouble is that the hhvm service hangs on to stuff forever. Just never lets go. I commented out 10-CirrusSearch.php and still got the errors so I don't believe Cirrus is itself responsible.
Looks like enabling the zend role fixes the problem for me. Sadly, I might have to give up on debugging this for a while and use zend.
Tagging hhvm and setting priority to high. I can no longer develop against hhvm which makes me less confident we'll be able to cut over to it. Since I have no idea why this happens in development I have no idea why it doesn't happen in production.
Since I sunk a day into figuring things out I should make it clear what I found before I put this aside. The issue is that hhvm was holding on to all open files and sockets after the request ended. You get about 40 page loads before you run out of file handles. Cirrus's test suite has 1000s of loads so that is why I hit it.
I have the following vagrant roles enabled: apisandbox betafeatures cirrussearch geodata And that triggers it. Zend stops it. You can reproduce it simply by reloading the main page about 40-50 times. It'll start blowing up every time really fast. You can also reproduce it by running the Cirrus test suite against it.
I'm running into this as well. FWICT, it's not necessarily hitting an open files limit but the mysql max_connections limit (defaults to 151). To reproduce and verify: 1) Disable zend role if enabled and provision 2) Restart mysql or hhvm (or both to be safe) 3) Open mysql cli and `show processlist` (you should see the single mysql process) (19:31) root@localhost:[wiki]> show processlist; +----+------+-----------+------+---------+------+-------+------------------+ | Id | User | Host | db | Command | Time | State | Info | +----+------+-----------+------+---------+------+-------+------------------+ | 40 | root | localhost | wiki | Query | 0 | NULL | show processlist | +----+------+-----------+------+---------+------+-------+------------------+ 1 row in set (0.01 sec) 4) Load http://localhost:8080/wiki 5) Rinse and repeat 3 and 4 to view growing number of persistent connections. Once > 151 is reached, you should get the "too many connections" error
I believe I have a fix filed upstream. At least, when I apply the fix on my machine my problem goes away. I'm not 100% sure its causing Dan's max_connections issue but it very well could. It was certainly causing some file-like things to stay open when they shouldn't and the MySQL connection falls into that category.
