Last modified: 2014-03-31 15:40:25 UTC
Hi, could you please migrate or create a new folder for math in the eqiad project. Otherwise new instances in eqiad can not be created. I deleted all custom puppet roles... but that did not help. Thanks. The puppet error is: notice: /Stage[first]/Apt::Update/Exec[/usr/bin/apt-get update]/returns: executed successfully notice: /Stage[main]/Role::Labs::Instance/Mount[/home]/ensure: ensure changed 'unmounted' to 'mounted' err: /Stage[main]/Role::Labs::Instance/Mount[/home]: Could not evaluate: Execution of '/bin/mount -o rw,vers=4,bg,hard,intr,sec=sys,proto=tcp,port=0,noatime,nofsc /home' returned 32: mount.nfs: mounting labstore.svc.eqiad.wmnet:/project/math/home failed, reason given by server: No such file or directory notice: /Stage[main]/Base::Puppet/Exec[puppet snmp trap]/returns: executed successfully notice: instanceproject: math notice: /Stage[main]/Base::Puppet/Notify[instanceproject: math]/message: defined 'message' as 'instanceproject: math' notice: /Stage[main]/Role::Labs::Instance/Mount[/data/project]/ensure: ensure changed 'unmounted' to 'mounted' err: /Stage[main]/Role::Labs::Instance/Mount[/data/project]: Could not evaluate: Execution of '/bin/mount -o rw,vers=4,bg,hard,intr,sec=sys,proto=tcp,port=0,noatime,nofsc /data/project' returned 32: mount.nfs: mounting labstore.svc.eqiad.wmnet:/project/math/project failed, reason given by server: No such file or directory
Ah, this is because shared volumes (both home and project) were turned off in the project settings. We're going to have shared volumes for everyone in eqiad but that setting is still observed by some of the scripts. I'm not sure what the proper general fix is, but in the meantime your project should be all set.
This seems to break compatibilty with labsvagrant err: /Stage[main]/Labs_vagrant/User[vagrant]/ensure: change from absent to present failed: Could not create user vagrant: Execution of '/usr/sbin/useradd -m vagrant' returned 12: useradd: cannot create directory /home/vagrant
My feeling is that this is more a bug in the labsvagrant role.
If the labsvagrant class is broken then this should maybe have its own bug, or needs to be taken up with the people who wrote that class (yuvi and/or matanya). This doesn't look like a failure of eqiad, although it may have to do with the way the nfs /home share is set up.
Change 118053 had a related patch set uploaded by Physikerwelt: Change home directory of vagrant user https://gerrit.wikimedia.org/r/118053
(In reply to physikerwelt from comment #2) > err: /Stage[main]/Labs_vagrant/User[vagrant]/ensure: change from absent to > present failed: Could not create user vagrant: Execution of > '/usr/sbin/useradd -m vagrant' returned 12: useradd: cannot create directory > /home/vagrant My best guess is that this was caused by the intermittent problem of the NFS server making shared read-only due to a race in application of ACLs. Coren knows about this problem and is trying to track down it's root cause. It seems to be cache related and will typically clear after a reboot or a 10-15 minute wait. There was a related problem that the NFSv4 server in eqiad is enforcing idmapd identities and the NFS server didn't know about the locally created vagrant user. Coren has fixed this by adding a vagrant user to the eqiad LDAP. You may have to create the /home/vagrant directory manually at this point. This may be something that needs to be fixed in the labs_vagrant role.
In fact I had to create the /home/vagrant directory manually and fix permissions. Thus, I do not see the bug as fixed.
Change 118053 merged by Ori.livneh: Change home directory of vagrant user https://gerrit.wikimedia.org/r/118053
