Last modified: 2014-08-06 23:56:26 UTC
videoscalers run "jobs-loop.sh ... webVideoTranscode" to process video transcoding jobs. there are unprocessed jobs in the queue that never run.
Seems to be jobs failing rather that not ever getting run.
Jan, can you clarify what you think the next steps on this one should be?
http://commons.wikimedia.org/wiki/File:Turtle_at_Mississippi_River_Park_and_Museum_Tunica_Resorts_MS.oggtheora.ogv is another case where its happening, trying to get more date on it now to see what might cause it.
(In reply to comment #0 by jgerber) > videoscalers run "jobs-loop.sh ... webVideoTranscode" to process video > transcoding jobs. there are unprocessed jobs in the queue that never run. (In reply to comment #1 by aschulz4587) > Seems to be jobs failing rather that not ever getting run. Is this still a problem, and how to find out? (In reply to comment #4 by jgerber) > trying to get more date on it now to see what might cause it. jgerber: Did this ever happen?
(In reply to Andre Klapper from comment #6) Currently we have 6723 queued transcodes as per https://commons.wikimedia.org/wiki/Special:TimedMediaHandler It seems most (/all) of them were "Added to Job queue 37 days, .. hours, .. minutes, .. seconds ago" as part of the 160p-ogg-transcode batch. Though the jobs seem not to fail when resubmitted. Thus I don't know if it is related to this bug report specifically. Do you think I should open another bug report requesting to re-queue all queued transcodes?
(In reply to Marco from comment #7) > (In reply to Andre Klapper from comment #6) > Currently we have 6723 queued transcodes as per > https://commons.wikimedia.org/wiki/Special:TimedMediaHandler > It seems most (/all) of them were "Added to Job queue 37 days, .. hours, .. > minutes, .. seconds ago" as part of the 160p-ogg-transcode batch. > > Though the jobs seem not to fail when resubmitted. Thus I don't know if it > is related to this bug report specifically. Do you think I should open > another bug report requesting to re-queue all queued transcodes? Its separate (I think its related to how we temporarily stopped making 160p files. When re-renabled it I don't think jobs got re-added despite what that page says). Anyhow, see bug 61690
>(I think its related to how we temporarily stopped making 160p > files. When re-renabled it I don't think jobs got re-added despite what that > page says). Err, actually we didn't do that. So never mind.... Anyways, it looks like a bunch of jobs disappeared, and then there's an inconsistent state with TimedMediaHandler thinking they are just pending, not gone. Which may or may not be the same bug as this one. I'm not sure.
(In reply to Bawolff (Brian Wolff) from comment #9) > >(I think its related to how we temporarily stopped making 160p > > files. When re-renabled it I don't think jobs got re-added despite what that > > page says). > > Err, actually we didn't do that. So never mind.... > > Anyways, it looks like a bunch of jobs disappeared, and then there's an > inconsistent state with TimedMediaHandler thinking they are just pending, > not gone. Which may or may not be the same bug as this one. I'm not sure. Ugh, sorry. Cannot read. Thought this was bug 61401. Ignore everything I said. This is quite likely the right bug. (In reply to Andre Klapper from comment #6) > (In reply to comment #0 by jgerber) > > videoscalers run "jobs-loop.sh ... webVideoTranscode" to process video > > transcoding jobs. there are unprocessed jobs in the queue that never run. > > (In reply to comment #1 by aschulz4587) > > Seems to be jobs failing rather that not ever getting run. > > > Is this still a problem, and how to find out? Someone with access to job queue log (I believe that's basically the folks with shell access) can find out if there are failing jobs that do not have their "failed" status reflected in the transcode table. It would be interesting to see if those 6723 queued transcodes ever ran and failed, or if they just never ran. ----- It might be good to do something like - If there are transcodes that have been pending for more than 10 days, and if the job queue for webVideoTranscode is empty, then automatically re-add the jobs for those videos
> (In reply to comment #10) > Someone with access to job queue log (I believe that's basically the folks > with shell access) can find out if there are failing jobs that do not have > their "failed" status reflected in the transcode table. > > It would be interesting to see if those 6723 queued transcodes ever ran and > failed, or if they just never ran. Is there any progress in making the analysis? > It might be good to do something like - If there are transcodes that have > been pending for more than 10 days, and if the job queue for > webVideoTranscode is empty, then automatically re-add the jobs for those > videos Is there any progress in implementing this? If not, I may code a bot to fulfill the request to reset those transcodes made @ https://commons.wikimedia.org/w/index.php?title=Commons%3ABots%2FWork_requests&diff=122148976&oldid=122026535 which
Change 133994 had a related patch set uploaded by Brian Wolff: Automatically re-add transcode jobs if transcode pending for 72h https://gerrit.wikimedia.org/r/133994
(In reply to Gerrit Notification Bot from comment #12) > Change 133994 had a related patch set uploaded by Brian Wolff: > Automatically re-add transcode jobs if transcode pending for 72h > > https://gerrit.wikimedia.org/r/133994 This would obviously be a band aid solution. We should figure out why they arent working in the first place.
(In reply to Bawolff (Brian Wolff) from comment #13) > > https://gerrit.wikimedia.org/r/133994 > > This would obviously be a band aid solution. We should figure out why they > arent working in the first place. Workaround patch has two -1s. What's the plan here?
(In reply to Andre Klapper from comment #14) > (In reply to Bawolff (Brian Wolff) from comment #13) > > > https://gerrit.wikimedia.org/r/133994 > > > > This would obviously be a band aid solution. We should figure out why they > > arent working in the first place. > > Workaround patch has two -1s. What's the plan here? Good question. I need to figure out a new plan/investigate further or talk it over with Arron. Gilles' -1 is just cosmetic so that -1 is trivial.
