Last modified: 2014-09-24 18:40:19 UTC

Wikimedia Bugzilla is closed!

Wikimedia migrated from Bugzilla to Phabricator. Bug reports are handled in Wikimedia Phabricator.
This static website is read-only and for historical purposes. It is not possible to log in and except for displaying bug reports and their history, links might be broken. See T73212, the corresponding Phabricator task for complete and up-to-date bug report information.
Bug 71212 - Make it possible to quickly and programmaticly pool and depool application servers
Make it possible to quickly and programmaticly pool and depool application se...
Status: NEW
Product: Wikimedia
Classification: Unclassified
Apache configuration (Other open bugs)
wmf-deployment
All All
: Normal normal (vote)
: ---
Assigned To: Nobody - You can work on this!
: hhvm
: 71211 (view as bug list)
Depends on:
Blocks: 70418
  Show dependency treegraph
 
Reported: 2014-09-24 00:18 UTC by Ori Livneh
Modified: 2014-09-24 18:40 UTC (History)
8 users (show)

See Also:
Web browser: ---
Mobile Platform: ---
Assignee Huggle Beta Tester: ---


Attachments

Description Ori Livneh 2014-09-24 00:18:42 UTC
HHVM takes more time to initialize than PHP5, because it needs to translate PHP to byte code, analyze the types flowing through the code, and then compile the byte code down to machine code. This process takes a few minutes, but during that time the application server is too slow to serve user requests.

Because this is the case, it is essential to avoid a cold restart of the entire production cluster. We need to have a staggered deployment process that allows us to take servers offline while they are warming up and then bring them back online once they're ready.

To meet these requirements, we need better automation. The current process for adding and removing servers from the pool serving user requests is too slow and too manual.
Comment 1 Ori Livneh 2014-09-24 00:20:25 UTC
*** Bug 71211 has been marked as a duplicate of this bug. ***
Comment 2 Bryan Davis 2014-09-24 00:36:42 UTC
Adding hhvm keyword. This is not directly related to the HHVM code base but it is an important component for full cluster HHVM deployment.
Comment 3 Giuseppe Lavagetto 2014-09-24 06:00:54 UTC
This is a long-standing issue we encounter for a number of reasons: we need an orchestration tool different from "vi".

So to achieve this specific goal we have two ways, the quick one and the correct one.

The quick one:
- Create a file on the local disk of each machine with a predefined content ("OK"), and make apache serve it via some url like "/is-this-pooled"; then modify this file when we want to actually depool a server

The correct one:
- Set up a distributed, consistent configuration database like Etcd or Zookeeper (my preference goes strongly to the first one, and I think there is some consensus in ops about this).
- Make pybal fetch its config from this database, and watch the database for changes
- Make scap/puppet/whatever work with it (for puppet, the best option is to use etcd as a secondary hiera backend just for orchestrated functionality)

I think the effort for this is really different, and while the latter solution will be implemented, I'm not sure it's advisable to do it now if it's going to be an hhvm blocker.

I'll add to this bug people that I think are relevant.

Note You need to log in before you can comment on or make changes to this bug.


Navigation
Links