Wikimedia Bugzilla is closed!

Prioritization and scheduling of this bug is tracked on Mingle card https://mingle.corp.wikimedia.org/projects/analytics/cards/1260

The cron job is running

  python /a/limn-mobile-data/generate.py /a/limn-mobile-data/mobile/ && /usr/bin/rsync [...]

. As the first part of the conjunction (prints [1] to stderr and) has exit code 1, rsync is not started.

Looking at the last few commits of /a/limn-mobile-data, it seems other
people are maintaining those scripts.
Do we know with whom to escalate?



[1]
/a/limn-mobile-data/generate.py:81: Warning: Truncated incorrect DOUBLE value: 'stable'
  cursor.execute(sql)
Traceback (most recent call last):
  File "/a/limn-mobile-data/generate.py", line 133, in <module>
    dg.execute()
  File "/a/limn-mobile-data/generate.py", line 108, in execute
    headers, rows = self.execute_sql(file_path, db_name)
  File "/a/limn-mobile-data/generate.py", line 81, in execute_sql
    cursor.execute(sql)
  File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 174, in execute
    self.errorhandler(self, exc, value)
  File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue
_mysql_exceptions.OperationalError: (2013, 'Lost connection to MySQL server during query')

Yes, I'm cc-ing Julius, Yuvi, and Jon and letting them fight for who gets it :)

Thanks very much Christian.  It seems that nobody was aware of this issue nor how to read the logs.  How did you access them?

Change 95202 had a related patch set uploaded by JGonera:
Fix error in edits_web table

https://gerrit.wikimedia.org/r/95202

Change 95202 merged by JGonera:
Fix error in edits_web table

https://gerrit.wikimedia.org/r/95202

(In reply to comment #3)
> It seems that nobody was aware of this issue nor
> how to read the logs.  How did you access them?

Oh ... there were logs already?
Interesting. I did neither know that, nor did I find that.
Logs would have made it easier.

Anyways ... Here's what I did:
* find a graph that looks wrong
  (http://mobile-reportcard.wmflabs.org/graphs/successful-edits-main)
* find the datafile that looks wrong
  (http://stat1001.wikimedia.org/limn-public-data/mobile/datafiles/successful-edits-main.csv)
* find the relevant apache config
  (/etc/apache2/sites-enabled/000-default on stat1001)
* find the file in the corresponding DocumentRoot
  (/var/www/limn-public-data/mobile/datafiles/successful-edits-main.csv)
* find out how the file gets there (puppet repo,
  cron job in misc::statistics::limn::mobile_data_sync in
  manifests/misc/statistics.pp)
* Look at the code, and decide if it does harm to run once.
* Run the code and capture output.

No, there are no logs, I assumed based on your first comment.  I think, then, that we should set up monitoring on this process somehow.  Juliusz fixed generate.py and it seems to be working now.

resolution was:

* fix generate.py
* wait a bit for cron to run it (30 min - 60 min)
* clear cache and try the limn graphs again

(In reply to comment #7)
> I think, then, that we should set up monitoring on this process somehow.

Corresponding bug report number? :)