Last modified: 2014-11-01 19:06:09 UTC

Wikimedia Bugzilla is closed!

Wikimedia migrated from Bugzilla to Phabricator. Bug reports are handled in Wikimedia Phabricator.
This static website is read-only and for historical purposes. It is not possible to log in and except for displaying bug reports and their history, links might be broken. See T74867, the corresponding Phabricator task for complete and up-to-date bug report information.
Bug 72867 - Multiple queue runners on tools-mail
Multiple queue runners on tools-mail
Status: NEW
Product: Wikimedia Labs
Classification: Unclassified
tools (Other open bugs)
unspecified
All All
: Unprioritized normal
: ---
Assigned To: Marc A. Pelletier
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2014-11-01 19:06 UTC by Tim Landscheidt
Modified: 2014-11-01 19:06 UTC (History)
4 users (show)

See Also:
Web browser: ---
Mobile Platform: ---
Assignee Huggle Beta Tester: ---


Attachments

Description Tim Landscheidt 2014-11-01 19:06:09 UTC
On tools-mail, there are often errors in exim's paniclog:

| root@tools-mail:~# cat /var/log/exim4/paniclog 
| 2014-10-30 16:29:09 1Xjsaz-0004TE-Hw re-exec of exim (/usr/sbin/exim4) with -Mc failed: No such file or directory
| 2014-10-30 21:09:04 1Xjwxs-0002Vi-A9 re-exec of exim (/usr/sbin/exim4) with -Mc failed: No such file or directory
| 2014-10-30 23:49:05 1XjzSj-0006Kv-Ej re-exec of exim (/usr/sbin/exim4) with -Mc failed: No such file or directory
| 2014-11-01 07:48:50 1XkTQY-00047u-0m re-exec of exim (/usr/sbin/exim4) with -Mc failed: No such file or directory
| 2014-11-01 17:50:07 1XkcoR-0007T8-1y re-exec of exim (/usr/sbin/exim4) with -Mc failed: No such file or directory
| root@tools-mail:~#

These seem to stem from multiple exim queue runners running at the same time:

| root@tools-mail:~# ps aux | fgrep exim
| 107       1215  0.0  0.0 108852  1600 ?        Ss   18:48   0:00 /usr/sbin/exim4 -bd -q10m
| root      1653  0.0  0.0   8028   612 pts/1    S+   18:53   0:00 fgrep --color=auto exim
| 107       9782  0.0  0.0 109876  1700 ?        S    Oct30   0:00 /usr/sbin/exim4 -bd -q10m
| 107      15924  0.0  0.0 109880  1708 ?        S    07:48   0:00 /usr/sbin/exim4 -bd -q10m
| 107      17323  0.0  0.0 109880  1720 ?        S    Oct30   0:00 /usr/sbin/exim4 -bd -q10m
| 107      24421  0.0  0.0 109876  1700 ?        S    Oct30   0:00 /usr/sbin/exim4 -bd -q10m
| 107      28751  0.0  0.0 109880  1704 ?        S    17:50   0:00 /usr/sbin/exim4 -bd -q10m
| root@tools-mail:~#

These extra queue runners could be spawned from a cron job or Puppet runs; /etc/cron.daily/exim4-base looks innocent and would only account for one extra queue runner per day.  However:

| root@tools-mail:~# find /proc -maxdepth 1 -user Debian-exim -ls
| 33257803    0 dr-xr-xr-x   8 Debian-exim Debian-exim        0 Nov  1 18:48 /proc/1215
| 31720118    0 dr-xr-xr-x   8 Debian-exim Debian-exim        0 Oct 30 21:09 /proc/9782
| 32898356    0 dr-xr-xr-x   8 Debian-exim Debian-exim        0 Nov  1 07:49 /proc/15924
| 31558318    0 dr-xr-xr-x   8 Debian-exim Debian-exim        0 Oct 30 16:29 /proc/17323
| 31816466    0 dr-xr-xr-x   8 Debian-exim Debian-exim        0 Oct 30 23:49 /proc/24421
| 33226702    0 dr-xr-xr-x   8 Debian-exim Debian-exim        0 Nov  1 17:50 /proc/28751
| root@tools-mail:~#

shows that there are multiple instances per day and the time (:09/:29/:49) correlates nicely with the Puppet runs that happen on this VM at :08/:28/:48.

But the Puppet logs (that do cover the period above) mention only the exim4-daemon-light/exim4-daemon-heavy fight:

| root@tools-mail:~# zgrep -i exim /var/log/puppet.log /var/log/puppet.log.?.gz | sort -u
| /var/log/puppet.log.1.gz:Notice: /Stage[main]/Exim4/Package[exim4-daemon-light]/ensure: created
| /var/log/puppet.log.1.gz:Notice: /Stage[main]/Toollabs::Mailrelay/Package[exim4-daemon-heavy]/ensure: created
| /var/log/puppet.log.2.gz:Notice: /Stage[main]/Exim4/Package[exim4-daemon-light]/ensure: created
| /var/log/puppet.log.2.gz:Notice: /Stage[main]/Toollabs::Mailrelay/Package[exim4-daemon-heavy]/ensure: created
| /var/log/puppet.log.3.gz:Notice: /Stage[main]/Exim4/Package[exim4-daemon-light]/ensure: created
| /var/log/puppet.log.3.gz:Notice: /Stage[main]/Toollabs::Mailrelay/Package[exim4-daemon-heavy]/ensure: created
| /var/log/puppet.log.4.gz:Notice: /Stage[main]/Exim4/Package[exim4-daemon-light]/ensure: created
| /var/log/puppet.log.4.gz:Notice: /Stage[main]/Toollabs::Mailrelay/Package[exim4-daemon-heavy]/ensure: created
| /var/log/puppet.log.5.gz:Notice: /Stage[main]/Exim4/Package[exim4-daemon-light]/ensure: created
| /var/log/puppet.log.5.gz:Notice: /Stage[main]/Toollabs::Mailrelay/Package[exim4-daemon-heavy]/ensure: created
| /var/log/puppet.log.6.gz:Notice: /Stage[main]/Exim4/Package[exim4-daemon-light]/ensure: created
| /var/log/puppet.log.6.gz:Notice: /Stage[main]/Toollabs::Mailrelay/Package[exim4-daemon-heavy]/ensure: created
| /var/log/puppet.log.7.gz:Notice: /Stage[main]/Exim4/Package[exim4-daemon-light]/ensure: created
| /var/log/puppet.log.7.gz:Notice: /Stage[main]/Toollabs::Mailrelay/Package[exim4-daemon-heavy]/ensure: created
| /var/log/puppet.log:Notice: /Stage[main]/Exim4/Package[exim4-daemon-light]/ensure: created
| /var/log/puppet.log:Notice: /Stage[main]/Toollabs::Mailrelay/Package[exim4-daemon-heavy]/ensure: created
| root@tools-mail:~#

but not any "service restarted" or something similar.

Note You need to log in before you can comment on or make changes to this bug.


Navigation
Links