Last modified: 2013-07-08 15:35:20 UTC

Wikimedia Bugzilla is closed!

Wikimedia migrated from Bugzilla to Phabricator. Bug reports are handled in Wikimedia Phabricator.
This static website is read-only and for historical purposes. It is not possible to log in and except for displaying bug reports and their history, links might be broken. See T33680, the corresponding Phabricator task for complete and up-to-date bug report information.
Bug 31680 - [Regression] Thumbnail cache should be invalidated on re-upload
[Regression] Thumbnail cache should be invalidated on re-upload
Status: RESOLVED WORKSFORME
Product: Wikimedia
Classification: Unclassified
Media storage (Other open bugs)
unspecified
All All
: High major with 4 votes (vote)
: ---
Assigned To: Nobody - You can work on this!
https://commons.wikimedia.org/wiki/Co...
: code-update-regression, need-integration-test
: 32388 32430 (view as bug list)
Depends on:
Blocks: 41371
  Show dependency treegraph
 
Reported: 2011-10-13 22:10 UTC by Krinkle
Modified: 2013-07-08 15:35 UTC (History)
25 users (show)

See Also:
Web browser: ---
Mobile Platform: ---
Assignee Huggle Beta Tester: ---


Attachments
180px thumb (12.06 KB, image/jpeg)
2012-11-07 08:46 UTC, Nemo
Details

Description Krinkle 2011-10-13 22:10:04 UTC
I'm not entirely sure if it's a regression. Marking as such for now.

https://commons.wikimedia.org/wiki/File:Station-Leeuwarden-WLM.jpg has had a new upload, but the 800px thumbnail cache is not being invalided/purged/cleared.

A recent reupload took place removing the watermark, but the 800px thumbnail still has it. Any other random size that I come up with that wasn't cached before does show the correct latest version.

800px: https://upload.wikimedia.org/wikipedia/commons/thumb/0/0e/Station-Leeuwarden-WLM.jpg/800px-Station-Leeuwarden-WLM.jpg

651px: https://upload.wikimedia.org/wikipedia/commons/thumb/0/0e/Station-Leeuwarden-WLM.jpg/651px-Station-Leeuwarden-WLM.jpg

action=purge or null-edit on the description page didn't fix it. Neither did clearing complete browser cache.

Even if action=purge did work, imho cache should automatically be marked invalid/cleared/whatever when a file has been (re)uploaded, just like is done for wikitext cache when an article is edited.
Comment 1 Mark A. Hershberger 2011-10-15 20:38:16 UTC
I'm not seeing the watermark now.  Can you give us an example?
Comment 2 Mark A. Hershberger 2011-10-15 22:03:04 UTC
tagging bugs for Marcus to look at
Comment 3 Emile 2011-11-15 14:39:40 UTC
See discussion on Wikimedia Commons on this page (Pissed off):
http://commons.wikimedia.org/wiki/Commons:Village_pump
Comment 4 Saibo 2011-11-15 16:13:52 UTC
Merged to here from (my) Bug 32430

Current thumbs fail to update on new file version. The most current thumb shows
an older version.

Bug 28613 apparently reoccurs

Example:
https://commons.wikimedia.org/wiki/File:Introduction_to_the_RepRap_Self_Replicating_3D_Printer.ogv
You should not see the Google Earth screenshot - it is only in the oldest
version. Both newer versions are blanked out. This behaviour did also occur
before I had hidden the oldest file version's content.

More examples: [[commons:Commons:Village_pump#Pissed_off]]
Comment 5 Saibo 2011-11-15 16:14:01 UTC
*** Bug 32430 has been marked as a duplicate of this bug. ***
Comment 6 Bawolff (Brian Wolff) 2011-11-16 00:34:14 UTC
bug 32388 is borderline a dupe of this too. There have been intermittent examples of this bug for several months now...
Comment 7 Saibo 2011-11-16 13:26:37 UTC
This bug messes up file histories if users are not aware of this bug. E.g. when using the Rotatebot to reupload a rotated version of a image (needed due to the sloppy implementation of the EXIF rotation at MediaWiki). Please do not include too much bugs in MediaWiki - otherwise it gets usuable (even with workarounds like Rotatebot). ;-)
Comment 8 Bawolff (Brian Wolff) 2011-11-16 13:44:57 UTC
(In reply to comment #7)
> This bug messes up file histories if users are not aware of this bug. E.g. when
> using the Rotatebot to reupload a rotated version of a image (needed due to the
> sloppy implementation of the EXIF rotation at MediaWiki). Please do not include
> too much bugs in MediaWiki - otherwise it gets usuable (even with workarounds
> like Rotatebot). ;-)

How prevalent is old thumbnails not updating in your opinion (like very rough estimate - is it 1 out of 100 000 images, or 1 out of every 10 type deal).


>Please do not include too much bugs in MediaWiki

lol, we try to avoid that.
Comment 9 p858snake 2011-11-16 13:49:08 UTC
(In reply to comment #7)
> This bug messes up file histories if users are not aware of this bug. E.g. when
> using the Rotatebot to reupload a rotated version of a image (needed due to the
> sloppy implementation of the EXIF rotation at MediaWiki).

(In regards to the comment about the "sloppy" implementation of rotations)
If there are issues, Please report them in bugzilla.

But the issue I believe you are talking about presenting is images being rotated in third party image editing programs (for example: paint.net, gimp, photoshop, etc) and the program subsequently not updating the images metadata ([[Exif]] data) to reflect the rotational change in the file, which is what MediaWiki reads to detect how the image should be displayed.

Although this is a issue, There isn't really anything much more on the MediaWiki end that can be done to combat this I believe.
Comment 10 Saibo 2011-11-16 14:25:07 UTC
(In reply to comment #8)
> How prevalent is old thumbnails not updating in your opinion (like very rough
> estimate - is it 1 out of 100 000 images, or 1 out of every 10 type deal).
Well, you can click through https://commons.wikimedia.org/wiki/User:Rotatebot/Log
30 checked - nonupdating 4 (but just the 800px thumbs):
* https://commons.wikimedia.org/wiki/File:Manastirea_Neamtului_-_July_2008_-_7.jpg
* https://commons.wikimedia.org/wiki/File:Manastirea_Neamtului_-_July_2008_%2813%29.jpg
* https://commons.wikimedia.org/wiki/File:Manastirea_Neamtului_-_July_2008_%2812%29.jpg
* https://commons.wikimedia.org/wiki/File:Manastirea_Rameti_vara_4.jpg


> >Please do not include too much bugs in MediaWiki
> lol, we try to avoid that.
Just wanted to warn just in case... ;-)


(In reply to comment #9)
> (In regards to the comment about the "sloppy" implementation of rotations)
> If there are issues, Please report them in bugzilla.
there: Bug 31504
 
> But the issue I believe you are talking about presenting is images being
> rotated in third party image editing programs (for example: paint.net, gimp,
> photoshop, etc) and the program subsequently not updating the images metadata
> ([[Exif]] data) to reflect the rotational change in the file, which is what
> MediaWiki reads to detect how the image should be displayed.
Yes, that is a problem too. Anyway: Off topic here.
Comment 11 Bawolff (Brian Wolff) 2011-11-16 14:54:11 UTC
(In reply to comment #10)
> (In reply to comment #8)
> > How prevalent is old thumbnails not updating in your opinion (like very rough
> > estimate - is it 1 out of 100 000 images, or 1 out of every 10 type deal).
> Well, you can click through
> https://commons.wikimedia.org/wiki/User:Rotatebot/Log
> 30 checked - nonupdating 4 (but just the 800px thumbs):
> *
> https://commons.wikimedia.org/wiki/File:Manastirea_Neamtului_-_July_2008_-_7.jpg
> *
> https://commons.wikimedia.org/wiki/File:Manastirea_Neamtului_-_July_2008_%2813%29.jpg
> *
> https://commons.wikimedia.org/wiki/File:Manastirea_Neamtului_-_July_2008_%2812%29.jpg
> * https://commons.wikimedia.org/wiki/File:Manastirea_Rameti_vara_4.jpg
> 
> 

I just had a conversation with Saibo on irc. Apparently these thumbs are only wrong if you're browsing from europe. Us north american folks can simulate by putting 
 91.198.174.234 upload.wikimedia.org
in our etc/hosts.

So my first guess is that HTCP purge packets aren't getting to european squids, but it gets weirder. If you mangle the url  of the thumb so that it should be a cache miss (aka put ?foo at the end of the thumb url) I still get the wrong thumb. My understanding is that an image squid cache miss should come from some server full of image thumbs, so this should be the same regardless from where you're accessing. (But I could be mis-interpreting something, I'm not super familar with how wmf works its servers in different places of the world set up).

Anyways, here is the HTTP headers of the request that returned the wrong thumb.

Server	nginx/0.7.65
Date	Wed, 16 Nov 2011 14:43:00 GMT
Content-Type	image/jpeg
Connection	keep-alive
Content-Length	105454
Accept-Ranges	bytes
X-Cache	MISS from amssq51.esams.wikimedia.org, MISS from amssq55.esams.wikimedia.org
X-Cache-Lookup	MISS from amssq51.esams.wikimedia.org:3128, MISS from amssq55.esams.wikimedia.org:80
Via	1.1 amssq51.esams.wikimedia.org:3128 (squid/2.7.STABLE9), 1.0 amssq55.esams.wikimedia.org:80 (squid/2.7.STABLE9)
Comment 12 Rob Lanphier 2011-11-16 21:33:47 UTC
Aaron Schulz and I did a fair amount of work trying to repro and investigate this last night.  We didn't think to do the /etc/hosts trick, but one of our theories was a Europe-only problem, so that makes sense.  We *did* repro a thumbnailing problem, though it was only temporary and easily fixed with ?action=purge on the image description page.  I'm not sure we actually reproduced the problem that's causing the most grief, but we did at least prove it's possible (if not as easy) to see the problem on one of the North American squid servers.

One theory we had: maybe it's a race condition, where the thumb is requested after the image is updated, but before the thumb is generated, since the "Age:" http header for the image was clearly younger than the upstream thumbnail should have been.
Comment 13 Rob Lanphier 2011-11-16 22:25:40 UTC
Mark Bergsma is reporting that the HTCP purge daemon hasn't been running for a while, and will only restart manually, which is the likely cause.  He's manually restarted this (so the symptoms should go away...please test), but let's not close this until he's confirmed that the daemon can be restarted automatically.
Comment 14 Bawolff (Brian Wolff) 2011-11-16 23:42:58 UTC
I just tested on the djvu file listed at bug 32388. Earlier that file had a broken thumb (When viewed from europe) that action=purging on the image desc page wasn't fixing. I just viewed it again, and thumb was still broken, but i purged, and this time the thumb got purged.

Additionally Saibo said on irc he hasn't seen any new instances recently

Therefore I'm calling this fixed.

p.s.
One thing I am rather confused about is how the wrong thumb was being shown on squid cache *misses* even if it was just htcp that was broken. (Perhaps there's just more caching layers that work in complicated ways I'm not familiar with)
Comment 15 Bawolff (Brian Wolff) 2011-11-16 23:46:15 UTC
*** Bug 32388 has been marked as a duplicate of this bug. ***
Comment 16 Saibo 2011-11-16 23:55:54 UTC
[[:File:White Storks Migrating Northwards Over Bental Mountain DSC00695.JPG]] and [[:File:Introduction_to_the_RepRap_Self_Replicating_3D_Printer.ogv]] could be fixed now by a purge (which didn't help before - at least at the video and I am sure someone tried to purge the Storks, too).
Comment 17 Mark A. Hershberger 2011-11-17 14:47:07 UTC
(In reply to comment #16)
> [[:File:White Storks Migrating Northwards Over Bental Mountain DSC00695.JPG]]
> and [[:File:Introduction_to_the_RepRap_Self_Replicating_3D_Printer.ogv]] could
> be fixed now by a purge (which didn't help before - at least at the video and I
> am sure someone tried to purge the Storks, too).

Could you clarify why you re-opened this?
Comment 18 Saibo 2011-11-17 16:39:57 UTC
(In reply to comment #17)
> Could you clarify why you re-opened this?
I did not know that it was closed. Blame bugzilla. Maybe I didn't reload the page before typing my comment and it was closed in the meantime?
Comment 19 Saibo 2011-11-21 18:32:38 UTC
It seems to reoccur:
75px-NYCS-bull-trans-3.svg.png and 76px-NYCS-bull-trans-3.svg.png  differ in color since a new version with slightly different color was uploaded.
Purge https://commons.wikimedia.org/w/index.php?title=File:NYCS-bull-trans-3.svg&action=purge doesn't help.


wget https://upload.wikimedia.org/wikipedia/commons/thumb/
2/25/NYCS-bull-trans-3.svg/75px-NYCS-bull-trans-3.svg.png
--2011-11-21 19:24:43--  https://upload.wikimedia.org/wikipedia/commons/thumb/2/
25/NYCS-bull-trans-3.svg/75px-NYCS-bull-trans-3.svg.png
Auflösen des Hostnamen »upload.wikimedia.org«.... 91.198.174.234
Verbindungsaufbau zu upload.wikimedia.org|91.198.174.234|:443... verbunden.
HTTP Anforderung gesendet, warte auf Antwort... 200 OK
Länge: 2172 (2,1K) [image/png]
In »75px-NYCS-bull-trans-3.svg.png« speichern.

100%[======================================>] 2.172       --.-K/s   in 0s

2011-11-21 19:24:44 (148 MB/s) - »75px-NYCS-bull-trans-3.svg.png« gespeichert [2                                                           172/2172]



wget https://upload.wikimedia.org/wikipedia/commons/thumb/                                                           2/25/NYCS-bull-trans-3.svg/76px-NYCS-bull-trans-3.svg.png
--2011-11-21 19:24:48--  https://upload.wikimedia.org/wikipedia/commons/thumb/2/                                                           25/NYCS-bull-trans-3.svg/76px-NYCS-bull-trans-3.svg.png
Auflösen des Hostnamen »upload.wikimedia.org«.... 91.198.174.234
Verbindungsaufbau zu upload.wikimedia.org|91.198.174.234|:443... verbunden.
HTTP Anforderung gesendet, warte auf Antwort... 200 OK
Länge: nicht spezifiziert [image/png]
In »76px-NYCS-bull-trans-3.svg.png« speichern.

    [ <=>                                   ] 2.297       --.-K/s   in 0s

2011-11-21 19:24:58 (356 MB/s) - »76px-NYCS-bull-trans-3.svg.png« gespeichert [2                                                           297]

md5sum *NYC*
46ec1ad2dac00fc86aa5f670ef363ba2  75px-NYCS-bull-trans-3.svg.png
37d2adff21668b9089f303cd39e91e9f  76px-NYCS-bull-trans-3.svg.png
Comment 20 Bawolff (Brian Wolff) 2011-11-22 14:45:12 UTC
>md5sum *NYC*
>46ec1ad2dac00fc86aa5f670ef363ba2  75px-NYCS-bull-trans-3.svg.png
>37d2adff21668b9089f303cd39e91e9f  76px-NYCS-bull-trans-3.svg.png

Umm, comparing md5sum's of different size thumbnails isn't going to tell you very much. Even if they were both correct thumbnails, the md5sum would be different. (However, I can confirm by visual inspection that the wrong thumb was returned for the 75px size).


This seems to have a different root cause then the other issues reported here, as it doesn't differ between europe/non-europe.
Comment 21 Emile 2011-11-22 18:14:36 UTC
I am certainly not a computer-guru but I use the Wikipedia-pages and the Wikimedia Commons-page quite often.

And I have experienced the problem described here. I have also experienced something else. 

After I have uploaded a picture from which I have removed a watermark, I want to edit the page to remove the template that is on the page. 

This problem occurs when I push the edit-button too early (milliseconds of course) to make a quick edit of the page. 

Solution: upload the file and then wait till the thumbnail is fully updated. Then edit the page.

Maybe (and I repeat maybe) this is the solution.
Comment 23 Saibo 2011-12-12 20:46:33 UTC
(In reply to comment #20)
> >md5sum *NYC*
> >46ec1ad2dac00fc86aa5f670ef363ba2  75px-NYCS-bull-trans-3.svg.png
> >37d2adff21668b9089f303cd39e91e9f  76px-NYCS-bull-trans-3.svg.png
> 
> Umm, comparing md5sum's of different size thumbnails isn't going to tell you
> very much. 
I wasn't comparing them. It was just to enable others to see which file I get delivered. ;-)
Comment 25 TMg 2011-12-19 04:38:11 UTC
PS: There must be a much bigger problem. All thumbnails are loading very, very slow. Some are not loading at all (timeout). For example, when going to http://commons.wikimedia.org/wiki/Special:NewFiles it takes about a minute to load all thumbnails. A few thumbnails load and are displayed, then all transfers are blocked, then a few more thumbnails load, transfer is stopped again and so on. This does not happen on other web pages, so it's not my network.
Comment 26 Saibo 2011-12-19 18:11:01 UTC
(In reply to comment #25)
> PS: There must be a much bigger problem.

Yes, but it is another problem → Already at [[:bugzilla:32432]].
Comment 27 Mark A. Hershberger 2012-04-19 16:38:02 UTC
also see bug 36048 which may be a dupe.
Comment 29 TMg 2012-05-16 21:53:03 UTC
PS: I had an idea. Maybe the "invalidate all thumbnails" process is not triggered because of the upload method used or because of some broken JavaScript in one of the upload methods? I'm using the basic upload form.

This is an other bug but I have to explain it a bit. When I click "re-upload this image" an way to complicated and therefor useless upload form shows up, made of a dozen input elements. This doesn't make any sense because most of the information entered there will be lost. The comment will be cropped. Sometimes the basic upload form shows up when I try to re-upload an image but most of the time I have to add "&uploadformstyle=basic" to the URL to force it.

I'm using Opera 11.64.
Comment 30 Aaron Schulz 2012-05-16 21:53:09 UTC
None of the bad thumbsnails are on ms5, I assume they are at least on squid.
Comment 31 Ben Hartshorne 2012-05-16 22:01:05 UTC
just for the record, the four resolutions in swift are 142, 220, 320, and 800. Neither of the two broken files exist on swift (120px or 640px).  (neither in the container listing nor on disk.)
Comment 32 Marco 2012-06-13 11:17:30 UTC
(In reply to comment #10)
> (In reply to comment #8)
> > How prevalent is old thumbnails not updating in your opinion (like very rough
> > estimate - is it 1 out of 100 000 images, or 1 out of every 10 type deal).
> Well, you can click through
> https://commons.wikimedia.org/wiki/User:Rotatebot/Log
> 30 checked - nonupdating 4 (but just the 800px thumbs):
> *
> https://commons.wikimedia.org/wiki/File:Manastirea_Neamtului_-_July_2008_-_7.jpg
> *
> https://commons.wikimedia.org/wiki/File:Manastirea_Neamtului_-_July_2008_%2813%29.jpg
> *
> https://commons.wikimedia.org/wiki/File:Manastirea_Neamtului_-_July_2008_%2812%29.jpg
> * https://commons.wikimedia.org/wiki/File:Manastirea_Rameti_vara_4.jpg
> 

The 800px thumbs of the last three files are still not updated right now.

Furthermore https://commons.wikimedia.org/wiki/File:Grau.jpg is affected (95px-thumb)!
and https://commons.wikimedia.org/wiki/File:Kkerepresentant.jpg and https://commons.wikimedia.org/wiki/File:PHILIPPE_SUEUR.jpg were affected. (I circumvent by reverting twice to the old version...)
Comment 33 TMg 2012-07-31 16:33:50 UTC
I do not upload many files to commons but every time I do it I came across this issue. Today this file (out of four) does not refresh after I uploaded a new version (the old version contains thin white lines between the colors). I tried to add "?action=purge" to both the description page and the thumbnail.

http://upload.wikimedia.org/wikipedia/commons/thumb/7/74/Rgb-colorwheel.svg/480px-Rgb-colorwheel.svg.png

As usual other thumbnail sizes show the new version:

http://upload.wikimedia.org/wikipedia/commons/thumb/7/74/Rgb-colorwheel.svg/481px-Rgb-colorwheel.svg.png
Comment 34 Bawolff (Brian Wolff) 2012-07-31 16:38:33 UTC
(In reply to comment #33)
> I do not upload many files to commons but every time I do it I came across this
> issue. Today this file (out of four) does not refresh after I uploaded a new
> version (the old version contains thin white lines between the colors). I tried
> to add "?action=purge" to both the description page and the thumbnail.
> 
> http://upload.wikimedia.org/wikipedia/commons/thumb/7/74/Rgb-colorwheel.svg/480px-Rgb-colorwheel.svg.png
> 
> As usual other thumbnail sizes show the new version:
> 
> http://upload.wikimedia.org/wikipedia/commons/thumb/7/74/Rgb-colorwheel.svg/481px-Rgb-colorwheel.svg.png

Look fine to me....
Comment 35 TMg 2012-08-01 14:44:54 UTC
(In reply to comment #34)
> Look fine to me....

Yes, because it was finally refreshed and displays the latest version now. Yesterday it showed the old version for hours.

I uploaded several new versions and most (but nor all) of the thumbnails are refreshed immediately. Some showed the old version but I was able to force a refresh by adding "?action=purge" to the thumb URL and the description page. This single thumb always showed the old version no matter what I tried. I consider this a bug. As said, this happens every few uploads. It's not a rare bug.
Comment 36 Bawolff (Brian Wolff) 2012-08-02 13:29:09 UTC
>This single thumb always showed the old version no matter what I tried. I
>consider this a bug.

So just to clarify, the bug is not that it never gets purged, the bug is that in some cases it takes several hours to get purged?
Comment 37 TMg 2012-08-04 20:17:38 UTC
(In reply to comment #36)
> So just to clarify, the bug is not that it never gets purged, the bug is that
> in some cases it takes several hours to get purged?

Seems so. Does not make a big difference (I know, it may be an important difference for the developers). This bug is wasting our time for many, many months now. It makes people re-upload files again and again. Nobody knows why a thumb is not refreshed and why it is "magically" refreshed the next day.

Also this is no new information. I don't see anybody complaining about a thumb that is still wrong after weeks. It seems in all cases it is refreshed after a while. No idea how long it took in average. I had better things to do.
Comment 38 TMg 2012-10-10 12:29:33 UTC
Almost every time I re-upload an image I get old garbage for some thumbnail sizes. At least this is how it feels. I think I posted dozens of examples for over a year now. See bug #28613. Purge never does anything. Some hours or days later the bad thumbnails are "magically" fixed.

Old garbage:

http://upload.wikimedia.org/wikipedia/commons/thumb/7/73/Mediawiki-versionsvergleich.png/640px-Mediawiki-versionsvergleich.png

Correct:

http://upload.wikimedia.org/wikipedia/commons/thumb/7/73/Mediawiki-versionsvergleich.png/641px-Mediawiki-versionsvergleich.png
Comment 39 TMg 2012-10-16 17:50:04 UTC
The story continues. This returns a 404 since yesterday. What the ...?

http://upload.wikimedia.org/wikipedia/commons/thumb/2/2c/Broom_icon.svg/22px-Broom_icon.svg.png
Comment 40 Aaron Schulz 2012-10-16 17:54:07 UTC
The "magic" fixing might just be the files getting rotated out of squid/varnish via LRU or expiry (cache time depends on last-modified date). I doubt the actual purge actions were just delayed for that long.

Either this is systemic (certain file purge actions always give bogus purge urls to cache) or there is some intermittent network problem.
Comment 41 TMg 2012-10-16 19:46:19 UTC
I'm spending $100 to the foundation if somebody please, please fix the broken squid or whatever causes this issue. It's more than a year now!
Comment 42 Andre Klapper 2012-10-16 20:47:57 UTC
(In reply to comment #39)
> The story continues. This returns a 404 since yesterday. What the ...?
> http://upload.wikimedia.org/wikipedia/commons/thumb/2/2c/Broom_icon.svg/22px-Broom_icon.svg.png

I get the suspicion that bug 40980 is the same issue.
Comment 43 Platonides 2012-10-16 22:07:30 UTC
(In reply to comment #40)
> Either this is systemic (certain file purge actions always give bogus purge
> urls to cache) or there is some intermittent network problem.

If it was an intermittent network problem then after purging several times one of them would succeed. Maybe an squid ignores all purges (under certain load conditions?) and when a thumbnail gets assigned to it, it never gets out? (until expiry)

Or maybe if the network is overloaded all the purges gets lost in some hour frame...
Comment 44 TMg 2012-10-17 20:10:07 UTC
I'm not sure but I think we reached a new record. The file

http://upload.wikimedia.org/wikipedia/commons/thumb/7/73/Mediawiki-versionsvergleich.png/640px-Mediawiki-versionsvergleich.png
http://commons.wikimedia.org/wiki/File:Mediawiki-versionsvergleich.png

is showing old garbage for a total of 8 days now! Time to call Guinness? Did somebody turned the "purge" feature off finally? Is the foundation out of money and stopped the thumbnail servers? Are the "Wiki Love Monuments" mass uploads to blame?
Comment 45 Andre Klapper 2012-10-23 01:00:27 UTC
(In reply to comment #40)
> The "magic" fixing might just be the files getting rotated out of squid/varnish
> via LRU or expiry (cache time depends on last-modified date). I doubt the
> actual purge actions were just delayed for that long.
> 
> Either this is systemic (certain file purge actions always give bogus purge
> urls to cache) or there is some intermittent network problem.

Aaron: Any idea how to investigate further in order to track this down?
Comment 46 Aaron Schulz 2012-10-26 00:19:55 UTC
(In reply to comment #43)
> (In reply to comment #40)
> > Either this is systemic (certain file purge actions always give bogus purge
> > urls to cache) or there is some intermittent network problem.
> 
> If it was an intermittent network problem then after purging several times one
> of them would succeed. Maybe an squid ignores all purges (under certain load
> conditions?) and when a thumbnail gets assigned to it, it never gets out?
> (until expiry)
> 
> Or maybe if the network is overloaded all the purges gets lost in some hour
> frame...

An intermittent problem could become permanent. See bug 41130.

(In reply to comment #45)
> Aaron: Any idea how to investigate further in order to track this down?

The cause is likely the same as 41130.
Comment 48 Nemo 2012-11-07 08:46:03 UTC
Created attachment 11323 [details]
180px thumb

LOL
To clarify, the 180px thumb is from the previous image, which got deleted https://commons.wikimedia.org/wiki/Commons:Deletion_requests/File:Faedo.jpg
Is cache invalidation on deletion also covered by this bug?
Comment 49 Derk-Jan Hartman 2012-11-08 11:25:09 UTC
@Nemo Hard to tell, since bug 41130 in theory affects each possible cache invalidation, this can also be the case for invalidations after a deletion.

However, if every deletion shows this problem all the time (instead of irregularly) then it's more likely to be a separate issue then bug 41130.

Note I just fixed this particular case, by applying a workaround for bug 41130.
Comment 50 Marco 2012-11-08 13:35:22 UTC
(In reply to comment #49)
> Note I just fixed this particular case, by applying a workaround for bug 41130.
the 180px thumb I mentioned above is now fixed.
Comment 51 Nemo 2013-01-23 13:01:46 UTC
https://commons.wikimedia.org/wiki/File:InterfaceMessageGroup_inherit_graph.png after reupload showed a wrong "thumb" with |frame (it's actually the original).

Compare:
https://upload.wikimedia.org/wikipedia/commons/c/c9/InterfaceMessageGroup_inherit_graph.png (old)
https://upload.wikimedia.org/wikipedia/commons/c/c9/InterfaceMessageGroup_inherit_graph.png?foo=bar (correct)

wget -S https://upload.wikimedia.org/wikipedia/commons/c/c9/InterfaceMessageGroup_inherit_graph.png
--2013-01-23 14:01:12--  https://upload.wikimedia.org/wikipedia/commons/c/c9/InterfaceMessageGroup_inherit_graph.png
Risoluzione di upload.wikimedia.org (upload.wikimedia.org)... 91.198.174.234, 2620:0:862:ed1a::b
Connessione a upload.wikimedia.org (upload.wikimedia.org)|91.198.174.234|:443... connesso.
Richiesta HTTP inviata, in attesa di risposta... 
  HTTP/1.1 200 OK
  Server: nginx/1.1.19
  Date: Wed, 23 Jan 2013 13:01:13 GMT
  Content-Type: image/png
  Content-Length: 49315
  Connection: keep-alive
  X-Object-Meta-Sha1base36: nodvjwj8x4qo47cmfuoaze2mf34ynbo
  Last-Modified: Tue, 02 Oct 2012 11:16:26 GMT
  ETag: efa32df0a11e6b3866ef4648faae69fd
  X-Timestamp: 1349176586.61961
  Accept-Ranges: bytes
  Access-Control-Allow-Origin: *
  X-Cache: HIT from sq44.wikimedia.org
  X-Cache-Lookup: HIT from sq44.wikimedia.org:3128
  Age: 533511
  X-Cache: HIT from knsq20.knams.wikimedia.org
  X-Cache-Lookup: HIT from knsq20.knams.wikimedia.org:3128
  X-Cache: MISS from knsq20.knams.wikimedia.org
  X-Cache-Lookup: MISS from knsq20.knams.wikimedia.org:80
  Via: 1.1 sq44.wikimedia.org:3128 (squid/2.7.STABLE9), 1.0 knsq20.knams.wikimedia.org:3128 (squid/2.7.STABLE9), 1.0 knsq20.knams.wikimedia.org:80 (squid/2.7.STABLE9)
Lunghezza: 49315 (48K) [image/png]
Comment 52 TMg 2013-01-25 20:47:52 UTC
This bug is like a virus. Also local file uploads in the German Wikipedia are infected.

File description page:
http://de.wikipedia.org/wiki/Datei:Radiobuttons.gif

Broken garbage:
http://upload.wikimedia.org/wikipedia/de/d/dc/Radiobuttons.gif
http://upload.wikimedia.org/wikipedia/de/thumb/d/dc/Radiobuttons.gif/120px-Radiobuttons.gif

How it should look:
http://upload.wikimedia.org/wikipedia/de/d/dc/Radiobuttons.gif?dummy

Tracert:
Routenverfolgung zu upload-lb.esams.wikimedia.org [91.198.174.234] über maximal 30 Abschnitte:
[I removed the first few steps]
  8    53 ms    52 ms    51 ms  ge0-1-0-cr0.ixf.de.as6908.net [80.81.192.244]
  9    58 ms    57 ms    56 ms  te3-4-3502-cr0.nik.nl.as6908.net [78.41.154.17]
 10    56 ms    54 ms    56 ms  ge-2-2.br1-knams.wikimedia.org [78.41.155.38]
 11    56 ms    56 ms    56 ms  ve7.te-8-1.csw1-esams.wikimedia.org [91.198.174.250]
 12    56 ms    58 ms    64 ms  upload-lb.esams.wikimedia.org [91.198.174.234]
Comment 53 Marco 2013-01-28 18:24:31 UTC
(In reply to comment #51)

(In reply to comment #52)

Both occureces seem to be fixed. Thank goes to Leslie I assume?
Comment 55 Marco 2013-01-30 13:16:52 UTC
(In reply to comment #54)
>  And I'm in Europe.
Me too.
Still working for me since 28.1.
Comment 56 Andre Klapper 2013-01-30 13:36:46 UTC
(In reply to comment #55)
> Still working for me since 28.1.

Today it also works for me, finally.
Comment 57 Bawolff (Brian Wolff) 2013-07-08 15:35:20 UTC
Resolving worksforme. Seems to work now (presumably due to various varnish htcp fixes).

If this acts up again, probably best to open a new bug instead of reopening this one.

Note You need to log in before you can comment on or make changes to this bug.


Navigation
Links