Last modified: 2014-11-13 15:56:01 UTC
Currently, the generators functions use yield, which is not tread-safe. PWB should offer a thread-safe version using one of the many interesting suggestions from http://www.dabeaz.com/generators/Generators.pdf (or any other method :P)
What is the goal you want to achieve by this? Remember that threads in Python are useless for computations, due to the GIL.
As I understand it, I/O happens outside of GIL. As the API requests are the most time-consuming part of many of my robots (and more precisely the connection to the servers), being able to do requests from several threads should somewhat improve performance (as long as the throttling is not too aggressive). I've noticed that the preloading limit is not only 50 pages, making this problem even more stringent for many small pages. It's probably also a good idea for things like image upload/download. If it helps, we can do some tests to see if performance is increased for a simple file downloader?
I see. There are already some features in place, but we are maybe not using asynchronous requests at all points where it might be useful. First of all, connections should be re-used - this is already a feature in the httplib2 library. The next layer, comms.threadedhttp, supports asynchronous requests ('features' would be a closer term - basically, you create a request and then wait for a lock to be released). However, I don't think we use this feature anywhere, as it's not exposed in the higher-up layers. For saving pages, which (I think) is the most relevant place for async request, we already have support, where requests that do not return a reply that has to be handled can be handled asynchronously - see Page.put_async. For pagegenerators, we might be able to win a bit by requesting the (i+1)th page before returning the i-th page (or, for the PreloadingGenerator, by requesting the (i+1)th batch before all pages from the i-th batch have been returned).
(In reply to comment #3) > The next layer, comms.threadedhttp, supports asynchronous requests. [...] I don't think we use this feature anywhere, as > it's not exposed in the higher-up layers. I've noticed that while writing the answer to Gerard's questions today :) > For saving pages, which (I think) is the most relevant place for async > request, > we already have support, where requests that do not return a reply that has > to > be handled can be handled asynchronously - see Page.put_async. I've experimented with put_async with mixed results. When the upload works, it's mostly OK, however when one request hits an error (like a 504 from the server) it just keeps trying again and again, keeping the thread blocked. Instead, the request should probably be de-queued, processed and, if a callback has been registered, the callback should be called in order to allow the bot to re-queue the request. This, however, could cause trouble if the order of the requests is important. The bot can receive a callback, but AFAIK it cannot remove already queued requests. Also, what happens if no callback has been registered? Should we simply re-queue the request? I don't have a perfect solution at this time, but this is a point that should be considered. Another possible issue, that PWB can't really do much about, is that one can get a 504 even if the save is successful, making the re-queueing useless. I don't have a good solution for that either, but we could consult with the Wikimedia developers. > For pagegenerators, we might be able to win a bit by requesting the (i+1)th > page before returning the i-th page (or, for the PreloadingGenerator, by > requesting the (i+1)th batch before all pages from the i-th batch have been > returned). This should be especially useful if it can be controlled by the user. Do you have any ideas on how to do this? I think there were some good ideas brought up on this bug. Should we start a thread on the mailing list so we can gather more input on this?
Change 172023 had a related patch set uploaded by John Vandenberg: Asynchronous HTTP requests https://gerrit.wikimedia.org/r/172023
