Last modified: 2014-05-20 18:01:03 UTC
On queue machines: Can't exec "/usr/bin/jsub": No such file or directory. Can't exec "/usr/bin/qsub": No such file or directory. With the new cron, I have to send programs to the queue. Programs determine if a dump file for a particular wiki is available or runs daily scans of particular wikis. If dump file is available or daily needs to be run, these programs send the checkwiki scans to the queue for the appropriate wiki language. Only other option is to write a 100+ line crontab.
This affects me to. I run python script in crontab that constructs and submits SGE array jobs. That's no longer possible. Did some research and it seems the exec instances are no longer permitted access the SGE master: error: denied: host "tools-exec-05.eqiad.wmflabs" is neither submit nor admin host
The exec instances are not "no longer" permitted to submit jobs, but never were :-). But until the crontab change this wasn't relevant.
Is there a reason for this? I use a python script that constructs a SGE array job. With this new rule the python script must run on the exec instances and won't be able to submit the array job. Before the python script ran on tools-dev.
(In reply to Johan from comment #3) > Is there a reason for this? I use a python script that constructs a SGE > array job. With this new rule the python script must run on the exec > instances and won't be able to submit the array job. Before the python > script ran on tools-dev. Johan, while you and I were doing simple scripts to send jobs to the queue, others were not. They were doing bigger jobs and the amount of these bigger jobs caused the cron machine to slow down or not function at times. Before the cron switch, around ~15% of my cron jobs never ran. So, having cron jobs sent to the queue is a good thing.™ Having jobs on the queue not being able to submit jobs is a different story. Currently, it is penalizing those of us who doing the right thing before the switch. This problem has also been known for a week now.
Originally, it was not intended for jobs to be able to spawn jobs (the potential for a runaway "fork bomb" that would severely impact other users was too great). Rather than turn that feature on, which has its own set of problems, I've made a workaround available: rather than use jsub in a cron job, you can use 'jlocal' which explicitly uses a local shell (and from which, therefore, you can run a shell script that does submit jobs). I'm going to document it shortly, but it's already available for use.
(In reply to Marc A. Pelletier from comment #5) Any status update on documentation? I can't use 'jlocal' as I keep getting: "/usr/bin/jlocal": No such file or directory
Where are you using it? on your crontab or your submitted jobs?
On queue machines and tools-login. On crontab and submitted jobs.
thats your problem, jlocal should only exist on the -submit host. you basically have a wrapper script you invoke that submits other jobs. if you use jlocal in your crontab to start that script, it should be able to then submit jobs to the grid
That still doesn't solve my original question. jlocal is not found anywhere. I can't use it on any host as it is not found. I can't use it on the submit host as it is not found. Where is jlocal?
jlocal is on the submit host, I used it daily and just got a email from a cron using it ~2 minutes ago
(In reply to Betacommand from comment #11) > jlocal is on the submit host, I used it daily and just got a email from a > cron using it ~2 minutes ago Could you provide an example of how you are doing this. Then, my brain might have an 'ah ha' moment (doubtful).
0 1 * * * jlocal python /data/project/betacommand-dev/svn_copy/email_logs.py
