Last modified: 2012-11-12 22:05:36 UTC
Created attachment 10960 [details] firefox/firebug/net screenshot showing the problem Sometimes it does load but it then needs seveal minutes. use case : i load a page in wikipedia from chromium, either directly or using search. The page stays blank and load forever. My understanding is that bits.wikimedia.org doesn't answer, or takes a very long time to answer. With firefox, the behaviour is different, but the same underlying problem can be observed. The difference is that firefox manages to display the whole page even if it still waits for data from bits.wikimedia.org, while chromium can't. None of this happens with ipv4. When using ipv4, everything works flawlessly. My configuration is the following : linux/gentoo with 3.5.1 kernel, www-client/chromium-21.0.1180.57, www-client/firefox-bin-14.0.1. But i'm quite confident this doesn't matter here. I did few tests from this computer: * bits.wikimedia.org resolves as ipv6, very quickly (2620:0:862:ed1a::a) * it pings as ipv6 (very quickly, too) * i can see ports 22/80/443 opened with "nmap -6 bits.wikimedia.org" * if i copy/paste the bits.wikimedia.org url and use either curl or wget with "-6" option to force ipv6, the file is downloaded without problem. I'm really confused on what to conclude from that. I confirm that the file is NOT loading both from firefox and chromium (unless using ipv4 of course). I used firefox/firebug to make the attached snapshot using the "net tab". Unfortunately, bits.wikimedia.org never answered to this very query. You will also notice that some of the images took more than 10S even before the download could be started. But this is a different issue : at least those finally came in. I'm very often connected on freenode as orzel, if you want some realtime tets.
I am using Wikipedia over IPv6 with self-compiled chromium under FreeBSD without any major problems. Can you sniff the traffic using tcpdump? Something like: tcpdump -w /tmp/file -n -s 0 -i <yourinterface> ip6 would do. Feel free to send the dump privately to me as this might contain some private information. saper@freenode
The command doesn't work, i dont understand why. It says "tcpdump : /tmp/file : no such file or directory", although my understanding of the man page is that tcpdump is supposed to create/fill this file, not read. Anyway, it seems that for the last few days, it works a lot better. It's not as fast as ipv4 from here, but the page now actually load, which is a huge improvement. Nothing has changed from there... so i can't really say. My guess is a change on your side (infrastructure).
That will not have last long : the problem is back, and the page is stuck again on "bits.wikimedia.org" :-(
Probably a problem with the C library or network configuration, not enough information to actually reproduce. Please reopen if more information (i.e. sniffed network traffic) is available.
You could also provide the output of the command line: traceroute6 bits.wikimedia.org
This last one is easy, here it is. I already said that bits.wikimedia.org pings from here, does traceroute really provide any more information ? Concerning the sniff information, as said, the command you provided doesn't work and i can't understand how it works, i'll need a lot more time to find out this. traceroute to bits-lb.esams.wikimedia.org (2620:0:862:ed1a::a) from 2002:52e1:9a02::1, 30 hops max, 24 byte packets 1 2002:c058:6301::1 (2002:c058:6301::1) 147.447 ms 106.458 ms 105.622 ms 2 gigabitethernet3-7.core1.ash1.he.net (2001:470:0:136::1) 149.968 ms 127.011 ms 153.828 ms 3 10gigabitethernet1-2.core1.nyc4.he.net (2001:470:0:36::2) 153.325 ms 110.646 ms 106.384 ms 4 10gigabitethernet1-2.core1.lon1.he.net (2001:470:0:128::2) 164.971 ms 151.515 ms 106.396 ms 5 10gigabitethernet1-1.core1.ams1.he.net (2001:470:0:3f::2) 209.332 ms 114.69 ms 122.879 ms 6 xe-1-1-0.cr2-knams.wikimedia.org (2001:7f8:1::a504:3821:1) 167.918 ms 152.421 ms 134.21 ms 7 vl400-ve7.csw1-esams.wikimedia.org (2620:0:862:fe00::1) 131.065 ms 142.933 ms 270.966 ms 8 bits-lb.esams.wikimedia.org (2620:0:862:ed1a::a) 164.769 ms 132.346 ms 130.732 ms
(In reply to comment #2) > The command doesn't work, i dont understand why. It says "tcpdump : /tmp/file : > no such file or directory" Maybe /tmp doesn't exist ? although that seems nearly impossible. You could try an alternate file name e.g. tcpdump -w ~/dump0.pcap -n -s 0 -i interfacename ip6 (and obviously replace interfacename with your interface's name)
If you get the curl utility, you might want to try out: curl -6 -i http://bits.wikimedia.org/ Will report the response header as well as the response. repoening bug
Here is what I have from nightshade.toolserver.org towards you: $ traceroute6 2002:52e1:9a02::1 traceroute to 2002:52e1:9a02::1 (2002:52e1:9a02::1), 30 hops max, 80 byte packets 1 2620:0:862:101::2 (2620:0:862:101::2) 0.254 ms 0.296 ms 0.370 ms 2 vl400-ve7.br1-knams.wikimedia.org (2620:0:862:fe00::2) 0.859 ms 0.854 ms 0.872 ms 3 ge5-1.cr1.AMS2.NL.content-core.net (2001:7f8:1::a501:5598:1) 1.548 ms 1.659 ms 1.762 ms 4 ge2-5-59.cr4.FRA3.content-core.net (2a01:138:0:106::13) 7.968 ms 7.992 ms 8.020 ms 5 Tenge1-1-42.cr2.FRA3.content-core.net (2a01:138:0:116::6) 8.114 ms 7.988 ms 7.999 ms 6 freehackers.org (2002:52e1:9a02::1) 134.865 ms 131.797 ms 132.341 ms
Yes, i have curl of course. Here is the result (seems ok to me) : verdi ~ # curl -6 -i http://bits.wikimedia.org/ HTTP/1.1 200 OK Server: Apache Last-Modified: Thu, 12 Aug 2010 16:12:20 GMT ETag: "b2-48da2a1772100" Content-Type: text/html X-Varnish: 1456902389 Via: 1.1 varnish Content-Length: 178 Accept-Ranges: bytes Date: Thu, 20 Sep 2012 13:47:11 GMT X-Varnish: 1781001547 Age: 0 Via: 1.1 varnish Connection: keep-alive X-Cache: cp3001 miss (0) <html> <head><title>bits and pieces</title> <meta http-equiv="refresh" content="1;url=http://www.wikimedia.org/" /> </head> <body> bits and pieces live here! </body> </html>
(In reply to comment #7) > (In reply to comment #2) > > The command doesn't work, i dont understand why. It says "tcpdump : /tmp/file : > > no such file or directory" > > Maybe /tmp doesn't exist ? although that seems nearly impossible. > > You could try an alternate file name e.g. > tcpdump -w ~/dump0.pcap -n -s 0 -i interfacename ip6 > > (and obviously replace interfacename with your interface's name) I really dont understand the error. Of course /tmp exists, and i had already tried some other places for the file. I've updated tcpdump from version 4.2.1 to 4.3.0 and still have the same problem. Exemple : berlioz tmp # tcpdump -w ~/dump0.pcap -n -s 0 -i eth0 ip6 tcpdump: /home/root/dump0.pcap: No such file or directory berlioz tmp # tcpdump -w /tmp/dump0.pcap -n -s 0 -i eth0 ip6 tcpdump: /tmp/dump0.pcap: No such file or directory Maybe i could do something around tcpdump -n -s 0 -i eth0 ip6 > /tmp/dump.txt That would not be pcap... but still useful ? or not ?
(In reply to comment #11) > Maybe i could do something around > tcpdump -n -s 0 -i eth0 ip6 > /tmp/dump.txt > > That would not be pcap... but still useful ? or not ? Maybe. or you could try -w - >/tmp/file ? tcpdump(8) says: > -w Write the raw packets to file rather than parsing and > printing them out. They can later be printed with the -r > option. Standard output is used if file is ``-''. See > pcap-savefile(5) for a description of the file format.
Something must be pretty broken with your tcpdump/tcap... Nevertheless I see that having delays around ~170ms is not pretty. I would say that 6to4 tunneling is at fault here. Do you have the same issues when trying to download something from upload.wikimedia.org? Looks like you have a static IPv4 address so getting a tunnel from HE or SixXS should be easy to get rid of 6to4...
Regarding tcpdump: your wonderful distribution might chroot() your tcpdump by default. You need to figure out how to disable this and/or figure out your chroot directory :)
Gee, you right, tcpdump is chrooted. And this is a recent change on my "wonderful distribution" (gentoo:). So here's a capture. I've removed "-s0" as its useless and added 'port http' to prevent other stuff to be captured. This gives: # tcpdump -w /chromium.wikipedia.pcap -n -i eth0 "ip6 and port http" tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes 225 packets captured 225 packets received by filter 0 packets dropped by kernel 4294967244 packets dropped by interface i had this typed in chromium url: http://en.wikipedia.org/wiki/Special:Search?search=wikipedia Then hitted enter on the text console with the tcpdump entry, then(and only then) hitted enter in the chromium url. It got stuck to the traditional "waiting for data from bits.wikimedia.org" And then the pcap file size would not change. I waited a little bit, then stopped sniffing and this is the file.
Created attachment 11131 [details] packet sniffing while trying to reach wikipedia
Regarding the question about upload.wikimedia.org, yes, it seems i have the same problem. Pasting this in chromium wont load, while it does load with my (ipv4 only) firefox. http://upload.wikimedia.org/wikipedia/commons/3/3f/Ubuntu_family_tree_11-06.png
(In reply to comment #15) > I've removed "-s0" as its useless How do you figure?
(In reply to comment #18) > (In reply to comment #15) > > I've removed "-s0" as its useless > > How do you figure? the man page says "-s0" is the default, which is confirmed by the command output, "capture size" has the same value : # tcpdump -w /chromium.wikipedia.pcap -n -i eth0 "ip6 and port http" tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes # tcpdump -w /chromium.wikipedia.pcap -s 0 -n -i eth0 "ip6 and port http" tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
Looks to me that your only problem is delay on the link. I am afraid 6to4 is not a good solution in this case.
Well, anyway, the conclusion seems to be that my tunnel/broker is at fault, so i should just try to use something else i'm afraid.
Possibly related wikitech-l thread: http://thread.gmane.org/gmane.science.linguistics.wikipedia.technical/64555
There was a tele2 issue, it is now fixed.
The Tele2 issue was with pmtpa/eqiad prefixes, not with esams, so this shouldn't apply. Also, n the tele2 issue prefixes were completely unreachable, while traceroutes pasted above look normal too. (I guess this answers the question if traceroutes are important :) From the looks of the issue it smells like an MTU issue to me. curl works because the / in bits is a very small page ("just bits and pieces") while Firefox/Chrome fetch the actual bits which are larger and go over the actual MTU. Could you run ping6 -s 1500 and see if it gets through? If it does, could you play with "ping6 -M do -s $N" where N a value 1300-1500 until you don't get back "packet too big"? Also, do note that we run our own 6to4 gateways to avoid such issues, so the path is asymmetric: the user goes through their closest 6to4 gateway (closest path to 192.88.99.1) but we're going back to 2002::/16 directly over IPv4.
unfortunately, as a result of previous comments, i've switched away from 6to4. I can confirm that it indeed solved the problem, but i'm not able to test 6to4 anymore..
