Last modified: 2014-08-27 01:27:55 UTC
Around 2014-08-21 20:00:00 all WMF master database servers experienced a permanent increase in read traffic, between 50% and 400%. Write-load has not changed significantly. 1.24wmf17 rolled out to wikipedias around 2014-08-21 19:55:00. There is an obvious outlier in query logs SELECT /* User::loadPasswords ... */, which, on masters with many wikis/users, is now executing more frequently than all other queries combined. Last week it hardly registered. Possible culprit is https://gerrit.wikimedia.org/r/#/c/151649/ , the bug 69102 fix. Can we use a slave for such non-trivial read traffic?
Worth noting that SELECT /* User::loadFromDatabase ... */ has been in the top 10 queries on masters for a while, but it is barely 1/10th the volume of SELECT /* User::loadPasswords ... */. Passwords being reloaded multiple times per request, perhaps?
I doubt Gerrit change #151649 is the cause, since it added $this->loadPasswords(); in setPassword and setNewpasswords, which should be called only when a user resets or sets a new password, unless we have many paranoid users resetting their passwords every hour or so :P Adding Parent5446 as cc since that was his changeset
Yeah, I see, that does sound odd. Possibly https://gerrit.wikimedia.org/r/#/c/151569/
(In reply to Sean Pringle from comment #3) > Possibly https://gerrit.wikimedia.org/r/#/c/151569/ Ugh, I wonder if this is due to bug 69381. That might explain it...
Gerrit change #149658 can be related: Changed password default to PBKDF2. Bug 28419 From release notes: The default password type for MediaWiki has been changed from MD5 to PBKDF2. Password hashes will automatically be updated as users log in. But on the other hand, it has been disabled on WMF by Gerrit change #153850, by making MD5 the current hash, so it shouldn't be affected, unless: A) the change to use MD5 on WMF isn't being applied somehow. B) the code is loading the password anyway to check if it's an old MD5 hash, even if it then realizes it shouldn't be changed.
> the code is loading the password anyway to check if it's an old MD5 hash, even if it then realizes it shouldn't be changed. Hmm, but LocalSettings (CommonSettings) overrides DefaultSettings settings completly, so for the code nothing has changed and if there wasn't a change in this routine we don't had this problem just now :/ I think like legoktm that this comes from the problem described in bug 69381 :/
(In reply to Jesús Martínez Novo (Ciencia Al Poder) from comment #5) > A) the change to use MD5 on WMF isn't being applied somehow. reedy@tin:~$ mwscript eval.php mediawikiwiki > var_dump( $wgPasswordDefault ); string(1) "B" > reedy@tin:~$ mwscript eval.php enwiki > var_dump( $wgPasswordDefault ); string(1) "B"
Any users with type :A: passwords will get their hash updated to type :B:, but that should be a pretty small segment of users. I can confirm normal users with type :B: passwords aren't getting reset on login. Oh wow, I missed that MassMessage was doing that in the UserGetRights hook. That hook can be called something like 10 times in a single page view, if you hit the wrong page. If Sean can confirm that the user being querried is always the same username ($wgMassMessageAccountUsername), then that will confirm it. If it's the logged in user's name, then we have another problem.
Yep, confirmed. Looking only at enwiki sampled traffic, user 20181147 'MediaWiki message delivery' accounts for 98.6% of the User::loadPasswords queries.
The fix for bug 69381 will go out in today afternoon's SWAT window.
It actually went out around 11 UTC. Sean, could you check if there are less queries now?
Load returned to normal immediately when Reedy did: 18:53 logmsgbot: reedy Synchronized php-1.24wmf18/extensions/MassMessage: (no message) (duration: 00m 14s) Not 11 UTC, but presumably included the relevant fix. Thanks everyone, for such a prompt response on this bug!
