Last modified: 2014-10-17 09:14:30 UTC
deployment-logstash1.eqiad.wmflabs ended up with a filled /var/ /var/log/apache2/other_vhosts_access.log was filled with lines such as: logstash.beta.wmflabs.org:80 127.0.0.1 - - [17/Oct/2014:08:16:52 +0000] "GET /server-status HTTP/1.1" 301 592 "-" "Python-urllib/2.7" logstash.beta.wmflabs.org:80 127.0.0.1 - - [17/Oct/2014:08:16:52 +0000] "GET /server-status HTTP/1.1" 301 592 "-" "Python-urllib/2.7" logstash.beta.wmflabs.org:80 127.0.0.1 - - [17/Oct/2014:08:16:52 +0000] "\x16\x03\x01" 301 308 "-" "-" logstash.beta.wmflabs.org:80 127.0.0.1 - - [17/Oct/2014:08:16:52 +0000] "\x16\x03\x01" 301 308 "-" "-" /var/log/diamond/diamond.log had a lot of: [2014-10-14 19:33:37,250] [Thread-1] Error retrieving HTTPD stats for host 127.0.0.1:80, url '/server-status?auto': [Errno 99] Cannot assign requested address I guess something is (was?) wrong in the Diamond collector used to monitor logstash. I have deleted the access.log (freeing up 850MB) file and restarted diamond.
+ Yuvi Panda: seems the diamond collector for logstash spams the apache access log with garbage requests such as ""\x16\x03\x01" :D
That might be due to monitor fatal using an HTTPS connection: modules/beta/files/monitor_fatals.rb:27:https://logstash-beta.wmflabs.org/#/dashboard/elasticsearch/fatalmonitor And the sequence "\x16\x03\x01" would mean an attempt to establish an SSL connection on port 80 which does not have mod_ssl.
