Last modified: 2014-11-13 19:13:21 UTC
The pagecounts files for 2014-11-12T21/1H did not get automatically generated [1]. What happened? [1] _________________________________________________________________ qchris@stat1002 // jobs: 0 // time: 14:44:14 // exit code: 0 cwd: ~ ~/cluster-scripts/dump_pagecounts_all_sites_status.sh +------------------+------------------+---------------+---------------+ | Hour | Date in filename | pagecounts | projectcounts | +------------------+------------------+---------------+---------------+ [...] | 2014-11-12T19/1H | 20141112-200000 | . | . | | 2014-11-12T20/1H | 20141112-210000 | . | . | | 2014-11-12T21/1H | 20141112-220000 | X | X | | 2014-11-12T22/1H | 20141112-230000 | . | . | | 2014-11-12T23/1H | 20141113-000000 | . | . | [...] +------------------+------------------+---------------+---------------+ Statuses: . --> File is there X --> File is missing
The corresponding partitions did not show errors, but an Oozie job (that is responsible for marking partitions as successful) was in status KILLED [1]. The root cause being that the webrequest status file was not reported as having 0 file size. When checking manually, the file exists and has size 0. I restarted the job, and now it passed just fine, and the pagecounts/projectcounts file got created. [1] Job ID : 0037441-140725140105408-oozie-oozi-C ------------------------------------------------------------------------------------------------------------------------------------ Job Name : hive_add_partition-webrequest-mobile App Path : hdfs://analytics-hadoop/wmf/refinery/current/oozie/webrequest/partition/add/coordinator.xml Status : RUNNING Start Time : 2014-10-10 16:00 GMT End Time : 3000-01-01 00:00 GMT Pause Time : - [...] 0037441-140725140105408-oozie-oozi-C@798 798 - - - 0059645-140725140105408-oozie-oozi-W - 0037441-140725140105408-oozie-oozi-C - 2014-11-12 21:04:39 GMT 2014-11-12 21:00:00 GMTKILLED 2014-11-12 23:10:26 GMT -
