Last modified: 2014-04-01 20:05:32 UTC

Wikimedia Bugzilla is closed!

Wikimedia migrated from Bugzilla to Phabricator. Bug reports are handled in Wikimedia Phabricator.
This static website is read-only and for historical purposes. It is not possible to log in and except for displaying bug reports and their history, links might be broken. See T63899, the corresponding Phabricator task for complete and up-to-date bug report information.
Bug 61899 - Automatic home directory creation is apparently broken
Automatic home directory creation is apparently broken
Status: RESOLVED FIXED
Product: Wikimedia Labs
Classification: Unclassified
tools (Other open bugs)
unspecified
All All
: Unprioritized blocker
: ---
Assigned To: Marc A. Pelletier
:
Depends on:
Blocks: 62213
  Show dependency treegraph
 
Reported: 2014-02-25 10:29 UTC by Tim Landscheidt
Modified: 2014-04-01 20:05 UTC (History)
4 users (show)

See Also:
Web browser: ---
Mobile Platform: ---
Assignee Huggle Beta Tester: ---


Attachments

Description Tim Landscheidt 2014-02-25 10:29:42 UTC
Nikerabbit had the problem yesterday that his home directory wasn't created on his initial login.  This problem reoccured just now with bjelleklang:

| scfc@tools-login:~$ sudo ls -alR /home/bjelleklang
| /home/bjelleklang:
| insgesamt 16
| drwx------   2 root root    20 Feb 25 08:58 .
| drwxr-xr-x 420 root root 12288 Feb 25 08:58 ..
| -rw-------   1 root root     0 Feb 25 08:58 .bashrc
| scfc@tools-login:~$

With Nikerabbit, removing the directory and asking the user to log in again fixed this.  With bjelleklang it didn't work:

| scfc@tools-login:~$ sudo rm -Rf /home/bjelleklang
| scfc@tools-login:~$ sudo ls -alR /home/bjelleklang
| ls: Zugriff auf /home/bjelleklang nicht möglich: Datei oder Verzeichnis nicht gefunden
| [wait for the user to log in]
| scfc@tools-login:~$ sudo ls -alR /home/bjelleklang
| /home/bjelleklang:
| insgesamt 16
| drwx------   2 root root    28 Feb 25 10:06 .
| drwxr-xr-x 420 root root 12288 Feb 25 10:06 ..
| -rw-------   1 root root     0 Feb 25 10:06 .bashrc
| scfc@tools-login:~$

I fixed this for bjelleklang with "cp -R /etc/skel ~bjelleklang && chown -R bjelleklang.wikidev ~bjelleklang".

What's odd is that a) the directory does get created and b) one file (.bashrc) gets created.
Comment 1 Tim Landscheidt 2014-02-25 10:32:26 UTC
For testing, I've deleted /home/scfc-test, logged in as that and the directory was created just fine.  Next up creating a new test account to see if there's something wrong with newly created users in LDAP.
Comment 2 Tim Landscheidt 2014-02-25 10:46:37 UTC
Created [[wikitech:User:Tim Landscheidt (Test 2)]], added to Tools, uploaded ssh key, logged into tools.wmflabs.org, no problem at all.  I'm out of ideas.
Comment 3 Marc A. Pelletier 2014-02-25 16:32:37 UTC
I'm wondering if that's a case of "logged in before the credentials were actually propagated to the NFS server" timing.
Comment 4 Tim Landscheidt 2014-02-25 17:06:45 UTC
With scfc-test2, I tried logging in directly after adding to Tools and before /public/keys/scfc-test2 was created, and it just didn't accept the key, but it didn't half-create /home/scfc-test2.

Also, shouldn't the NFS server a) use LDAP, so any changes are instantaneous and site-wide, and b) not look up user names?  I assumed that NFS servers accept any uids/gids and (if so) only handle 0 differently.
Comment 5 Marc A. Pelletier 2014-02-25 17:20:54 UTC
It does, but it's not instantaneous.

But also, we are using NFSv4; that doesn't just trust user IDs. :-)

So yeah, I'm not sure what happened in those cases either.  Odd.
Comment 6 Tim Landscheidt 2014-03-03 19:41:38 UTC
Apparently mga73 was another instance of this (only .bashrc, everything owned by root).
Comment 7 MZMcBride 2014-03-03 19:47:43 UTC
I think jorm was having this issue on unicorn.wmflabs.org.
Comment 8 Brandon Harris 2014-03-03 20:01:06 UTC
It's possibly the same issue.  Here's what happens when I try to log in:


zombieland:~ bharris$ ssh unicorn.wmflabs.org

If you are having access problems, please see:https://labsconsole.wikimedia.org/wiki/Access#Accessing_public_and_private_instances
Creating directory '/home/bharris'.
Unable to create and initialize directory '/home/bharris'.
Welcome to Ubuntu 12.04.2 LTS (GNU/Linux 3.2.0-57-virtual x86_64)

75 packages can be updated.
3 updates are security updates.

The last Puppet run was at Mon Mar  3 19:53:03 UTC 2014 (7 minutes ago).
*** System restart required ***

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

Last login: Mon Mar  3 19:31:09 2014 from 216.38.130.164
Connection to unicorn.wmflabs.org closed.
Comment 9 Christopher Johnson 2014-03-13 09:12:44 UTC
chown reports "Invalid argument" on new labs instances.
Comment 10 Christopher Johnson 2014-03-13 09:30:47 UTC
(In reply to christopher.johnson from comment #9)
> chown reports "Invalid argument" on new labs instances.

Also, if the "Share home directories across instances" is disabled, the home directory cannot be initialized on login.  This problem is specific to /home.
Comment 11 Marc A. Pelletier 2014-03-13 13:02:10 UTC
Yes, if there is no shared home, then it won't be able to mount a /home share.  The puppet noise is annoying, but we haven't found a clean way to prevent that yet (whether or not the project requested shared homes is not visible from puppet).  The warning is annoying but harmless.

Christopher, can you give me a bit more detail?  chown with what values report invalid argument, and on which share?
Comment 12 Christopher Johnson 2014-03-13 15:29:26 UTC
Marc,

for example, adduser foo will fail if it is run as root, because the script chown's /home/foo.  It is specific to /home only.  NFSv.4 (idmapd) related?
Comment 13 Marc A. Pelletier 2014-03-18 22:51:53 UTC
Ah, that is normal behavior for NFS4, which deals in usernames and not user IDs.  There are two ways around this: either make sure the username that needs to own shared files is known to the file server (in practice, "is in LDAP") or have local users' homes not be on NFS (/usr/lib is a reasonable place for 'system' users, for instance).

NFS4 does change the semantics of file ownership; rather than use user IDs and hope that they happen to match the same users on different clients, it relies on a central list of users for that.

What I have done, in cases where it's reasonable to do so, is add global system users to LDAP; this makes everything work as expected and is considerably more reliable and rely on user creation from puppet.
Comment 14 Marc A. Pelletier 2014-03-25 17:35:30 UTC
Is there still something I can do for you?

Note You need to log in before you can comment on or make changes to this bug.


Navigation
Links