Last modified: 2012-10-30 14:14:30 UTC

Wikimedia Bugzilla is closed!

Wikimedia migrated from Bugzilla to Phabricator. Bug reports are handled in Wikimedia Phabricator.
This static website is read-only and for historical purposes. It is not possible to log in and except for displaying bug reports and their history, links might be broken. See T42428, the corresponding Phabricator task for complete and up-to-date bug report information.
Bug 40428 - many API requests failing
many API requests failing
Status: RESOLVED WORKSFORME
Product: Wikimedia
Classification: Unclassified
General/Unknown (Other open bugs)
wmf-deployment
All All
: High blocker (vote)
: ---
Assigned To: Nobody - You can work on this!
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2012-09-21 14:52 UTC by merl
Modified: 2012-10-30 14:14 UTC (History)
5 users (show)

See Also:
Web browser: ---
Mobile Platform: ---
Assignee Huggle Beta Tester: ---


Attachments

Description merl 2012-09-21 14:52:42 UTC
Since some time my interwiki bot has much of errors when doing api requests. Many requests pass without any problems but some fail.

There are two kind of errors which i think are related:

1) Some http requests are not answered, so i am getting a timeout exception (after 30 seconds as is set on my config)

2) When requesting action=query&meta=siteinfo&siprop=dbrepllag&sishowalldb&format=xml i am getting an error response (/api/error):
 siincludeAllDenied:Cannot view all servers info unless $wgShowHostnames is true

headers of an example for ja.wikipedia.org:
Date: Fri, 21 Sep 2012 14:38:44 GMT
Server: Apache/2.2.14 (Ubuntu)
X-Powered-By: PHP/5.3.2-2wm1
X-Content-Type-Options: nosniff
MediaWiki-API-Error: siincludeAllDenied
X-Frame-Options: DENY
Cache-Control: private
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 132
Keep-Alive: timeout=15, max=99
Connection: Keep-Alive
Content-Type: text/xml; charset=utf-8

3) Also some api edit requests using maxlag are failing although all replags are low.

Especially because of the api error message i think there are some internal connection problems at wmf. My bot is using the pmtpa load balancer.

Also reusing a connection because of a keep-alive header isn't working, too.
Comment 1 Sumana Harihareswara 2012-09-25 15:03:04 UTC
merl: when did this problem start? What proportion of requests are failing?
Comment 2 Damian Z 2012-09-25 15:15:28 UTC
Are you using ssl?
Do you have ipv6 support? (If not, does your code force ipv4 and not rely on a timeout).
When you say edit requests are failing, what error code/headers are you getting back?
Is this on any specific wiki/wikis?
Comment 3 merl 2012-09-25 15:24:37 UTC
I noticed this problem on friday when i was having a quick look at my recent logs. My bot did not report an error to me because after the request fails the script resends the same requests again. This is mostly succesful as far i can see. But of course the increased number of requests needed slow down my bot and increases load at wmf.

The error is happening if i send queries that need replag information like the two examples above (siprop=dbrepllag&sishowalldb or maxlag on edits). Most of these queries are successful, so i would guess that there are only few misconfigured servers or having a broken connection to database servers.

The returned error for edits is the same as with too high replag. I am using maxlag=12 (my code also has functions to slow down edit speed much earlier and this part is not showing a high replag).   

Above i posted the full http response header. As you can see it does not contain any information about the server having processed the query with a failure.

I am using ipv6 (can fall back zu ipv4) without ssl (java, latest apache httpcomponents api).
Comment 4 merl 2012-09-25 15:25:57 UTC
I got errors for wikis on different clusters.
Comment 5 Sam Reed (reedy) 2012-09-25 18:54:25 UTC
It would be useful if you could tell us what servers are serving you the various errors.

Some of these sound like there are servers out of sync.

For example, wgShowHostnames is always set to true on the WMF cluster, though is set to false as the mediawiki default

On some errors (not all, depending on the nature) you should be given a servername

<?xml version="1.0"?>
<api servedby="srv291">
  <error code="help" info="" xml:space="preserve">
Comment 6 db [inactive,noenotif] 2012-09-26 18:00:19 UTC
(In reply to comment #5)
> On some errors (not all, depending on the nature) you should be given a
> servername
> <?xml version="1.0"?>
> <api servedby="srv291">
>   <error code="help" info="" xml:space="preserve">

You will not get the server name, when $wgShowHostnames is false ...
Comment 7 Sam Reed (reedy) 2012-09-26 18:27:34 UTC
(In reply to comment #6)
> (In reply to comment #5)
> > On some errors (not all, depending on the nature) you should be given a
> > servername
> > <?xml version="1.0"?>
> > <api servedby="srv291">
> >   <error code="help" info="" xml:space="preserve">
> 
> You will not get the server name, when $wgShowHostnames is false ...

reedy@fenari:/home/wikipedia/common$ mwscript eval.php enwiki
> var_dump( $wgShowHostnames );
bool(true)
Comment 8 merl 2012-10-09 14:00:59 UTC
Since thursday my bot log does not contain any siincludeAllDenied error anymore.
How have you fixed it?
Comment 9 Andre Klapper 2012-10-21 02:06:19 UTC
merl: Still no new siincludeAllDenied errors anymore? If so this ticket should probably be closed as RESOLVED WORKSFORME?
Comment 10 merl 2012-10-21 10:37:54 UTC
Problem seems to be solved since mw 1.21 update. So maybe is a wmf1.20 only bug.
Comment 11 Andre Klapper 2012-10-30 14:14:30 UTC
Closing as per last comment, but please do reopen this report if this happens again.

Note You need to log in before you can comment on or make changes to this bug.


Navigation
Links