Last modified: 2012-02-20 15:15:39 UTC

Wikimedia Bugzilla is closed!

Wikimedia migrated from Bugzilla to Phabricator. Bug reports are handled in Wikimedia Phabricator.
This static website is read-only and for historical purposes. It is not possible to log in and except for displaying bug reports and their history, links might be broken. See T33603, the corresponding Phabricator task for complete and up-to-date bug report information.
First Last Prev Next    This bug is not in your last search results.
Bug 31603 - Regression: API categorymembers and login occasionally fails to gzip/compress responses
Regression: API categorymembers and login occasionally fails to gzip/compress...
Status: RESOLVED FIXED
Product: Wikimedia
Classification: Unclassified
General/Unknown (Other open bugs)
unspecified
All All
: High normal (vote)
: ---
Assigned To: Nobody - You can work on this!
:
Depends on:
Blocks:
  Show dependency tree / graph
 
Reported: 2011-10-11 09:12 UTC by MER-C
Modified: 2012-02-20 15:15 UTC (History)
9 users (show)

See Also:
Web browser: ---
Mobile Platform: ---
Assignee Huggle Beta Tester: ---

Attachments
Output of test program (11.24 KB, text/plain)
2011-10-11 09:15 UTC, MER-C 		Details
Add an attachment (proposed patch, testcase, etc.)

Description MER-C 2011-10-11 09:12:13 UTC 
While repeatedly fetching the URL

http://en.wikipedia.org/w/api.php?format=xml&action=query&list=categorymembers&cmtitle=Category:Candidates_for_speedy_deletion

with "Accept-encoding: gzip" set, occasionally (about 10% probability) I will receive a response that is uncompressed. I've received reports this also happens when using API login (see [[User talk:MER-C#Wiki.java...again]]).

Test code:

import java.io.*;
import java.net.*;

public class Blah
{
    public static void main(String[] args) throws IOException
    {
        for (int i = 0; i < 25; i++)
        {
            URLConnection connection = new URL("http://en.wikipedia.org/w/api.php?format=xml&action=query&list=categorymembers&cmtitle=Category:Candidates_for_speedy_deletion").openConnection();
            connection.setRequestProperty("Accept-encoding", "gzip");
            connection.setRequestProperty("User-Agent", "Test script");
            connection.connect();
            System.out.println(connection.getHeaderFields());
        }
    }
}

This merely dumps the HTTP headers. Sample output:

{null=[HTTP/1.0 200 OK], Date=[Tue, 11 Oct 2011 08:48:51 GMT], Vary=[Accept-Encoding,X-Forwarded-Proto], Content-Length=[332], Content-Encoding=[gzip], Content-Type=[text/xml; charset=utf-8], X-Cache-Lookup=[MISS from sq38.wikimedia.org:80, MISS from sq34.wikimedia.org:3128], Connection=[keep-alive], Server=[Apache], X-Cache=[MISS from sq38.wikimedia.org, MISS from sq34.wikimedia.org], X-Content-Type-Options=[nosniff], Cache-Control=[private]}
{null=[HTTP/1.0 200 OK], Date=[Tue, 11 Oct 2011 08:48:52 GMT], Vary=[Accept-Encoding,X-Forwarded-Proto], Content-Length=[332], Content-Encoding=[gzip], Content-Type=[text/xml; charset=utf-8], X-Cache-Lookup=[MISS from sq64.wikimedia.org:80, MISS from sq34.wikimedia.org:3128], Connection=[keep-alive], Server=[Apache], X-Cache=[MISS from sq64.wikimedia.org, MISS from sq34.wikimedia.org], X-Content-Type-Options=[nosniff], Cache-Control=[private]}
{null=[HTTP/1.0 200 OK], Date=[Tue, 11 Oct 2011 08:48:52 GMT], Vary=[Accept-Encoding,X-Forwarded-Proto], Content-Length=[332], Content-Encoding=[gzip], Content-Type=[text/xml; charset=utf-8], X-Cache-Lookup=[MISS from sq64.wikimedia.org:80, MISS from sq34.wikimedia.org:3128], Connection=[keep-alive], Server=[Apache], X-Cache=[MISS from sq64.wikimedia.org, MISS from sq34.wikimedia.org], X-Content-Type-Options=[nosniff], Cache-Control=[private]}
{null=[HTTP/1.0 200 OK], Date=[Tue, 11 Oct 2011 08:48:54 GMT], Vary=[Accept-Encoding,X-Forwarded-Proto], Content-Length=[332], Content-Encoding=[gzip], Content-Type=[text/xml; charset=utf-8], X-Cache-Lookup=[MISS from sq65.wikimedia.org:80, MISS from sq34.wikimedia.org:3128], Connection=[keep-alive], Server=[Apache], X-Cache=[MISS from sq65.wikimedia.org, MISS from sq34.wikimedia.org], X-Content-Type-Options=[nosniff], Cache-Control=[private]}
{null=[HTTP/1.0 200 OK], Date=[Tue, 11 Oct 2011 08:48:57 GMT], Vary=[Accept-Encoding,X-Forwarded-Proto], Content-Length=[332], Content-Encoding=[gzip], Content-Type=[text/xml; charset=utf-8], X-Cache-Lookup=[MISS from sq64.wikimedia.org:80, MISS from sq34.wikimedia.org:3128], Connection=[keep-alive], Server=[Apache], X-Cache=[MISS from sq64.wikimedia.org, MISS from sq34.wikimedia.org], X-Content-Type-Options=[nosniff], Cache-Control=[private]}
{null=[HTTP/1.0 200 OK], Date=[Tue, 11 Oct 2011 08:48:57 GMT], Vary=[Accept-Encoding,X-Forwarded-Proto], Content-Length=[332], Content-Encoding=[gzip], Content-Type=[text/xml; charset=utf-8], X-Cache-Lookup=[MISS from sq74.wikimedia.org:80, MISS from sq34.wikimedia.org:3128], Connection=[keep-alive], Server=[Apache], X-Cache=[MISS from sq74.wikimedia.org, MISS from sq34.wikimedia.org], X-Content-Type-Options=[nosniff], Cache-Control=[private]}
{null=[HTTP/1.0 200 OK], Date=[Tue, 11 Oct 2011 08:48:58 GMT], Vary=[Accept-Encoding,X-Forwarded-Proto], Content-Length=[332], Content-Encoding=[gzip], Content-Type=[text/xml; charset=utf-8], X-Cache-Lookup=[MISS from sq60.wikimedia.org:80, MISS from sq34.wikimedia.org:3128], Connection=[keep-alive], Server=[Apache], X-Cache=[MISS from sq60.wikimedia.org, MISS from sq34.wikimedia.org], X-Content-Type-Options=[nosniff], Cache-Control=[private]}
{null=[HTTP/1.0 200 OK], Date=[Tue, 11 Oct 2011 08:48:58 GMT], Vary=[Accept-Encoding,X-Forwarded-Proto], Content-Length=[332], Content-Encoding=[gzip], Content-Type=[text/xml; charset=utf-8], X-Cache-Lookup=[MISS from sq37.wikimedia.org:80, MISS from sq34.wikimedia.org:3128], Connection=[keep-alive], Server=[Apache], X-Cache=[MISS from sq37.wikimedia.org, MISS from sq34.wikimedia.org], X-Content-Type-Options=[nosniff], Cache-Control=[private]}
{null=[HTTP/1.0 200 OK], Date=[Tue, 11 Oct 2011 08:48:59 GMT], Vary=[Accept-Encoding,X-Forwarded-Proto], Content-Length=[332], Content-Encoding=[gzip], Content-Type=[text/xml; charset=utf-8], X-Cache-Lookup=[MISS from sq66.wikimedia.org:80, MISS from sq34.wikimedia.org:3128], Connection=[keep-alive], Server=[Apache], X-Cache=[MISS from sq66.wikimedia.org, MISS from sq34.wikimedia.org], X-Content-Type-Options=[nosniff], Cache-Control=[private]}
{null=[HTTP/1.0 200 OK], Date=[Tue, 11 Oct 2011 08:49:00 GMT], Vary=[Accept-Encoding,X-Forwarded-Proto], Content-Length=[332], Content-Encoding=[gzip], Content-Type=[text/xml; charset=utf-8], X-Cache-Lookup=[MISS from sq65.wikimedia.org:80, MISS from sq34.wikimedia.org:3128], Connection=[keep-alive], Server=[Apache], X-Cache=[MISS from sq65.wikimedia.org, MISS from sq34.wikimedia.org], X-Content-Type-Options=[nosniff], Cache-Control=[private]}
{null=[HTTP/1.0 200 OK], Date=[Tue, 11 Oct 2011 08:49:00 GMT], Vary=[Accept-Encoding,X-Forwarded-Proto], Content-Length=[332], Content-Encoding=[gzip], Content-Type=[text/xml; charset=utf-8], X-Cache-Lookup=[MISS from sq66.wikimedia.org:80, MISS from sq34.wikimedia.org:3128], Connection=[keep-alive], Server=[Apache], X-Cache=[MISS from sq66.wikimedia.org, MISS from sq34.wikimedia.org], X-Content-Type-Options=[nosniff], Cache-Control=[private]}
{null=[HTTP/1.0 200 OK], Date=[Tue, 11 Oct 2011 08:49:02 GMT], Vary=[Accept-Encoding,X-Forwarded-Proto], Content-Length=[332], Content-Encoding=[gzip], Content-Type=[text/xml; charset=utf-8], X-Cache-Lookup=[MISS from sq77.wikimedia.org:80, MISS from sq34.wikimedia.org:3128], Connection=[keep-alive], Server=[Apache], X-Cache=[MISS from sq77.wikimedia.org, MISS from sq34.wikimedia.org], X-Content-Type-Options=[nosniff], Cache-Control=[private]}
{null=[HTTP/1.0 200 OK], Date=[Tue, 11 Oct 2011 08:49:04 GMT], Vary=[Accept-Encoding,X-Forwarded-Proto], Content-Length=[332], Content-Encoding=[gzip], Content-Type=[text/xml; charset=utf-8], X-Cache-Lookup=[MISS from sq37.wikimedia.org:80, MISS from sq34.wikimedia.org:3128], Connection=[keep-alive], Server=[Apache], X-Cache=[MISS from sq37.wikimedia.org, MISS from sq34.wikimedia.org], X-Content-Type-Options=[nosniff], Cache-Control=[private]}
{null=[HTTP/1.0 200 OK], Date=[Tue, 11 Oct 2011 08:49:05 GMT], Vary=[Accept-Encoding,X-Forwarded-Proto], Content-Length=[332], Content-Encoding=[gzip], Content-Type=[text/xml; charset=utf-8], X-Cache-Lookup=[MISS from sq39.wikimedia.org:80, MISS from sq34.wikimedia.org:3128], Connection=[keep-alive], Server=[Apache], X-Cache=[MISS from sq39.wikimedia.org, MISS from sq34.wikimedia.org], X-Content-Type-Options=[nosniff], Cache-Control=[private]}
{null=[HTTP/1.0 200 OK], Date=[Tue, 11 Oct 2011 08:49:06 GMT], Vary=[Accept-Encoding,X-Forwarded-Proto], Content-Length=[332], Content-Encoding=[gzip], Content-Type=[text/xml; charset=utf-8], X-Cache-Lookup=[MISS from sq40.wikimedia.org:80, MISS from sq34.wikimedia.org:3128], Connection=[keep-alive], Server=[Apache], X-Cache=[MISS from sq40.wikimedia.org, MISS from sq34.wikimedia.org], X-Content-Type-Options=[nosniff], Cache-Control=[private]}
{null=[HTTP/1.0 200 OK], Date=[Tue, 11 Oct 2011 08:49:06 GMT], Vary=[Accept-Encoding,X-Forwarded-Proto], Content-Length=[332], Content-Encoding=[gzip], Content-Type=[text/xml; charset=utf-8], X-Cache-Lookup=[MISS from sq62.wikimedia.org:80, MISS from sq34.wikimedia.org:3128], Connection=[keep-alive], Server=[Apache], X-Cache=[MISS from sq62.wikimedia.org, MISS from sq34.wikimedia.org], X-Content-Type-Options=[nosniff], Cache-Control=[private]}
{null=[HTTP/1.0 200 OK], Date=[Tue, 11 Oct 2011 08:49:07 GMT], Vary=[Accept-Encoding,X-Forwarded-Proto], Content-Length=[759], Content-Type=[text/xml; charset=utf-8], X-Cache-Lookup=[MISS from sq72.wikimedia.org:80, MISS from sq34.wikimedia.org:3128], Connection=[keep-alive], Server=[Apache], X-Cache=[MISS from sq72.wikimedia.org, MISS from sq34.wikimedia.org], X-Content-Type-Options=[nosniff], Cache-Control=[private]}
{null=[HTTP/1.0 200 OK], Date=[Tue, 11 Oct 2011 08:49:07 GMT], Vary=[Accept-Encoding,X-Forwarded-Proto], Content-Length=[332], Content-Encoding=[gzip], Content-Type=[text/xml; charset=utf-8], X-Cache-Lookup=[MISS from sq75.wikimedia.org:80, MISS from sq34.wikimedia.org:3128], Connection=[keep-alive], Server=[Apache], X-Cache=[MISS from sq75.wikimedia.org, MISS from sq34.wikimedia.org], X-Content-Type-Options=[nosniff], Cache-Control=[private]}
{null=[HTTP/1.0 200 OK], Date=[Tue, 11 Oct 2011 08:49:08 GMT], Vary=[Accept-Encoding,X-Forwarded-Proto], Content-Length=[332], Content-Encoding=[gzip], Content-Type=[text/xml; charset=utf-8], X-Cache-Lookup=[MISS from sq72.wikimedia.org:80, MISS from sq34.wikimedia.org:3128], Connection=[keep-alive], Server=[Apache], X-Cache=[MISS from sq72.wikimedia.org, MISS from sq34.wikimedia.org], X-Content-Type-Options=[nosniff], Cache-Control=[private]}
{null=[HTTP/1.0 200 OK], Date=[Tue, 11 Oct 2011 08:49:09 GMT], Vary=[Accept-Encoding,X-Forwarded-Proto], Content-Length=[332], Content-Encoding=[gzip], Content-Type=[text/xml; charset=utf-8], X-Cache-Lookup=[MISS from sq66.wikimedia.org:80, MISS from sq34.wikimedia.org:3128], Connection=[keep-alive], Server=[Apache], X-Cache=[MISS from sq66.wikimedia.org, MISS from sq34.wikimedia.org], X-Content-Type-Options=[nosniff], Cache-Control=[private]}
{null=[HTTP/1.0 200 OK], Date=[Tue, 11 Oct 2011 08:49:11 GMT], Vary=[Accept-Encoding,X-Forwarded-Proto], Content-Length=[332], Content-Encoding=[gzip], Content-Type=[text/xml; charset=utf-8], X-Cache-Lookup=[MISS from sq72.wikimedia.org:80, MISS from sq34.wikimedia.org:3128], Connection=[keep-alive], Server=[Apache], X-Cache=[MISS from sq72.wikimedia.org, MISS from sq34.wikimedia.org], X-Content-Type-Options=[nosniff], Cache-Control=[private]}
{null=[HTTP/1.0 200 OK], Date=[Tue, 11 Oct 2011 08:49:13 GMT], Vary=[Accept-Encoding,X-Forwarded-Proto], Content-Length=[759], Content-Type=[text/xml; charset=utf-8], X-Cache-Lookup=[MISS from sq60.wikimedia.org:80, MISS from sq34.wikimedia.org:3128], Connection=[keep-alive], Server=[Apache], X-Cache=[MISS from sq60.wikimedia.org, MISS from sq34.wikimedia.org], X-Content-Type-Options=[nosniff], Cache-Control=[private]}
{null=[HTTP/1.0 200 OK], Date=[Tue, 11 Oct 2011 08:49:14 GMT], Vary=[Accept-Encoding,X-Forwarded-Proto], Content-Length=[759], Content-Type=[text/xml; charset=utf-8], X-Cache-Lookup=[MISS from sq60.wikimedia.org:80, MISS from sq34.wikimedia.org:3128], Connection=[keep-alive], Server=[Apache], X-Cache=[MISS from sq60.wikimedia.org, MISS from sq34.wikimedia.org], X-Content-Type-Options=[nosniff], Cache-Control=[private]}
{null=[HTTP/1.0 200 OK], Date=[Tue, 11 Oct 2011 08:49:14 GMT], Vary=[Accept-Encoding,X-Forwarded-Proto], Content-Length=[332], Content-Encoding=[gzip], Content-Type=[text/xml; charset=utf-8], X-Cache-Lookup=[MISS from sq62.wikimedia.org:80, MISS from sq34.wikimedia.org:3128], Connection=[keep-alive], Server=[Apache], X-Cache=[MISS from sq62.wikimedia.org, MISS from sq34.wikimedia.org], X-Content-Type-Options=[nosniff], Cache-Control=[private]}
{null=[HTTP/1.0 200 OK], Date=[Tue, 11 Oct 2011 08:49:15 GMT], Vary=[Accept-Encoding,X-Forwarded-Proto], Content-Length=[332], Content-Encoding=[gzip], Content-Type=[text/xml; charset=utf-8], X-Cache-Lookup=[MISS from sq73.wikimedia.org:80, MISS from sq34.wikimedia.org:3128], Connection=[keep-alive], Server=[Apache], X-Cache=[MISS from sq73.wikimedia.org, MISS from sq34.wikimedia.org], X-Content-Type-Options=[nosniff], Cache-Control=[private]}

This wasn't an issue prior to the 1.18 upgrade on Wikimedia.
Comment 1 MER-C 2011-10-11 09:15:54 UTC 
Created attachment 9214 [details]
Output of test program

Attached output, I forgot that bugzilla wraps lines. (And I can't edit it out.)
Comment 2 Sam Reed (reedy) 2011-10-11 16:30:36 UTC 
How occasionally is occasionally?

Seems it might be more likely we've got a bad API Apache
Comment 3 tedder 2011-10-11 18:06:55 UTC 
@Reedy: in MER-C's example, three of 25 requests fail. I was seeing some level of errors, now when I run that script all 25 succeed.

Keep in mind this isn't/wasn't a problem with 'categorymembers', it is/was a general problem.
Comment 4 db [inactive,noenotif] 2011-10-11 18:08:48 UTC 
Accept-Encoding=gzip is optional for the server. You should not insist on that, because the server can you also send an uncompress result.

Check with

"gzip".equals( connection.getHeaderField( "Content-Encoding" ) )

if the server send you an gzipped result and you have to wrap your InputStream into a GZIPInputStream or not.
Comment 5 db [inactive,noenotif] 2011-10-11 18:17:32 UTC 
(In reply to comment #4)
> Accept-Encoding=gzip is optional for the server. You should not insist on that,
> because the server can you also send an uncompress result.

Sorry, that is wrong, but makes the world easy.

http://www.ietf.org/rfc/rfc2068.txt - 14.3 Accept-Encoding
If an Accept-Encoding header is present, and if the server cannot send a response which is acceptable according to the Accept-Encoding header, then the server SHOULD send an error response with the 406 (Not Acceptable) status code.
Comment 6 Fastily 2011-10-12 04:25:49 UTC 
I experienced this bug as well yesterday, but the issues appear to have gone away. All the test cases I ran just now succeeded.  Despite this, it may still be worth looking into; bugs do not just magically appear and then disappear.
Comment 7 Sam Reed (reedy) 2011-10-12 04:43:40 UTC 
Even more sounds like a bad server
Comment 8 Sam Reed (reedy) 2011-12-11 16:57:06 UTC 
Any more recent replications of this bug?

I'm moving it to site related stuff also
Comment 9 MER-C 2011-12-12 01:29:37 UTC 
I ran a 5000+ request program on the Saturday after Thanksgiving without encountering this problem.
Comment 10 Antoine "hashar" Musso (WMF) 2012-02-20 15:15:39 UTC 
Issue seems to have been fixed per comment 9. So I am closing this bug.
Note You need to log in before you can comment on or make changes to this bug.
First Last Prev Next    This bug is not in your last search results.

Navigation
Links