Wikimedia Bugzilla is closed!

From your paste it looks like you have some bad MW database setup code from somewhere: 

==> default: Notice: /Stage[main]/Role::Mobilefrontend/Mediawiki::Wiki[mobile]/Exec[mobilewiki_setup]/returns: A database query error has occurred.
==> default: Notice: /Stage[main]/Role::Mobilefrontend/Mediawiki::Wiki[mobile]/Exec[mobilewiki_setup]/returns: Query: INSERT INTO `updatelog`
==> default: Notice: /Stage[main]/Role::Mobilefrontend/Mediawiki::Wiki[mobile]/Exec[mobilewiki_setup]/returns:  SELECT 'filearchive-fa_major_mime-patch-fa_major_mime-chemical.sql' AS ul_key, null as ul_value
==> default: Notice: /Stage[main]/Role::Mobilefrontend/Mediawiki::Wiki[mobile]/Exec[mobilewiki_setup]/returns:  UNION SELECT 'image-img_major_mime-patch-img_major_mime-chemical.sql', null
==> default: Notice: /Stage[main]/Role::Mobilefrontend/Mediawiki::Wiki[mobile]/Exec[mobilewiki_setup]/returns:  UNION SELECT 'oldimage-oi_major_mime-patch-oi_major_mime-chemical.sql', null
==> default: Notice: /Stage[main]/Role::Mobilefrontend/Mediawiki::Wiki[mobile]/Exec[mobilewiki_setup]/returns:  UNION SELECT 'user_groups-ug_group-patch-ug_group-length-increase-255.sql', null
==> default: Notice: /Stage[main]/Role::Mobilefrontend/Mediawiki::Wiki[mobile]/Exec[mobilewiki_setup]/returns:  UNION SELECT 'user_former_groups-ufg_group-patch-ufg_group-length-increase-255.sql', null
==> default: Notice: /Stage[main]/Role::Mobilefrontend/Mediawiki::Wiki[mobile]/Exec[mobilewiki_setup]/returns:  UNION SELECT 'user_properties-up_property-patch-up_property.sql', null
==> default: Notice: /Stage[main]/Role::Mobilefrontend/Mediawiki::Wiki[mobile]/Exec[mobilewiki_setup]/returns:
==> default: Notice: /Stage[main]/Role::Mobilefrontend/Mediawiki::Wiki[mobile]/Exec[mobilewiki_setup]/returns: Function: DatabaseBase::sourceFile( /vagrant/mediawiki/maintenance/update-keys.sql )
==> default: Notice: /Stage[main]/Role::Mobilefrontend/Mediawiki::Wiki[mobile]/Exec[mobilewiki_setup]/returns: Error: 1062 Duplicate entry 'filearchive-fa_major_mime-patch-fa_major_mime-chemical.sql' for key 'PRIMARY' ()

http://pastebin.com/g47qTxAT

Fresh machine rebuild - still fails. I listed all the enabled roles at the end.

Tried disabling vector skin just in case - http://pastebin.com/2RmrnnC2

(In reply to Yuri Astrakhan from comment #2)
> http://pastebin.com/g47qTxAT
> 
> Fresh machine rebuild - still fails. I listed all the enabled roles at the
> end.

Your latest error output paste still has the errors from Exec[mobilewiki_setup] failing. I had a very similar problem recently on updating an existing VM and it turned out to be caused by a LocalSettings.php file from settings.d/wikis/*/ (in my case "wiki", in your case "mobile") having been deleted at some point. For me the cause for this was bug 50912. I had updated my mediawiki clone and not noticed that there was a merge conflict before I ran `vagrant provision`.

There are 3 ways to fix this:
* `vagrant destroy -f; vagrant up` to rebuild everything from scratch
* Drop the mobile database in the vm so that it is recreated cleanly.
* Copy the LocalSettings.php file from any other wiki into your settings.d/wikis/mobile directory. It turns out that the per-wiki LocalSettings.php file is not actually used for anything except controlling whether or not puppet tries to run the setup scripts to create the wiki or not. The LocalSettings.php file in your mediawiki checkout is the only one that is sourced at runtime and should be a copy of settings.d/wikis/wiki/LocalSettings.php.

Steps to reproduce:

* vagrant destroy -f
* vagrant reset-roles
* vagrant git-update
* vagrant up
* vagrant enable-role mobilefrontend
* vagrant provision

Failure is from the mobilewiki install itself. The installer fails claiming that the database already exists.

Immediately following puppet run that show creating the wiki as failing:

$ mysql -e 'show databases;'
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mobilewiki         |
| mysql              |
| performance_schema |
| wiki               |
+--------------------+

/vagrant/settings.d/wikis/mobilewiki/LocalSettings.php also exists and http://mobile.wiki.local.wmftest.net:8080/wiki/Special:Version responds with expected mobile skin content.

Is puppet failure a false negative somehow? Investigating further.

(In reply to Bryan Davis from comment #6)
> 
> Is puppet failure a false negative somehow? Investigating further.

Additional `vagrant provision` calls seem to complete with no errors and no failures for me. I'm going to redo my steps to reproduce with verbose vagrant and puppet logging to see if I can figure out what's happening on the initial provisioning attempt.

It looks to me like puppet is doing this:

* Run Exec['mobilewiki_check_settings'] which fails to connect to db and notifies Exec['mobile_setup']
* Run Exec['mobile_setup'] because the LocalSettings.php file in it's creates clause is missing
* Run Exec['mobile_setup'} again (wtfbbq?) and crash because of a db error for duplicate keys on an insert.

Log excerpts from `vagrant enable-role mobilefrontend && PUPPET_DEBUG=1 vagrant provision`:

# Check for existing db runs
==> default: Debug: Exec[mobilewiki_check_settings](provider=posix): Executing check 'mwscript sql.php --wikidb=mobilewiki </dev/null'
==> default: Debug: Executing 'mwscript sql.php --wikidb=mobilewiki </dev/null'
==> default: Debug: /Stage[main]/Role::Mobilefrontend/Mediawiki::Wiki[mobile]/Exec[mobilewiki_check_settings]/unless: DB connection error: Unknown database 'mobilewiki' ()
==> default: Debug: /Stage[main]/Role::Mobilefrontend/Mediawiki::Wiki[mobile]/Exec[mobilewiki_check_settings]/unless: Backtrace:
{...snip...}

# Check failed, so remove mobilewiki/LocalSettings.php
==> default: Debug: Exec[mobilewiki_check_settings](provider=posix): Executing 'rm -f /vagrant/settings.d/wikis/mobilewiki/LocalSettings.php'
{...snip...}

# Tell Exec[mobilewiki_setup] that it needs to run
==> default: Info: /Stage[main]/Role::Mobilefrontend/Mediawiki::Wiki[mobile]/Exec[mobilewiki_check_settings]: Scheduling refresh of Exec[mobilewiki_setup]
==> default: Debug: /Stage[main]/Role::Mobilefrontend/Mediawiki::Wiki[mobile]/Exec[mobilewiki_check_settings]: The container Mediawiki::Wiki[mobile] will propagate my refresh event

# Run Exec[mobilewiki_setup]
==> default: Debug: Exec[mobilewiki_setup](provider=posix): Executing '/usr/local/bin/multiversion-install mobilewiki admin --dbname mobilewiki --dbpass vagrant --dbuser root --pass vagrant --scriptpath /w --server http://mobile.wiki.local.wmftest.net:8888 --confpath /vagrant/settings.d/wikis/mobilewiki
{...snip...}

# Install worked, mobilewiki/LocalSettings.php and database created
==> default: Notice: /Stage[main]/Role::Mobilefrontend/Mediawiki::Wiki[mobile]/Exec[mobilewiki_setup]/returns: executed successfully
{...snip...}

# Here's the WTF. Puppet decides to run Exec[mobilewiki_setup] a second time
==> default: Debug: Exec[mobilewiki_setup](provider=posix): Executing '/usr/local/bin/multiversion-install mobilewiki admin --dbname mobilewiki --dbpass vagrant --dbuser root --pass vagrant --scriptpath /w --server http://mobile.wiki.local.wmftest.net:8888 --confpath /vagrant/settings.d/wikis/mobilewiki
{...snip...}
=> default: Notice: /Stage[main]/Role::Mobilefrontend/Mediawiki::Wiki[mobile]/Exec[mobilewiki_setup]/returns: A database query error has occurred.
{...snip...}
==> default: Error: /Stage[main]/Role::Mobilefrontend/Mediawiki::Wiki[mobile]/Exec[mobilewiki_setup]: Failed to call refresh: /usr/local/bin/multiversion-install mobilewiki admin --dbname mobilewiki --dbpass vagrant --dbuser root --pass vagrant --scriptpath /w --server http://mobile.wiki.local.wmftest.net:8888 --confpath /vagrant/settings.d/wikis/mobilewiki
==> default:  returned 1 instead of one of [0]

Dan has a pending patch at Ia2ff2ab53966ade7bbacd705dc43369345d1b0e0 that will probably fix this as an unintended but happy side effect. I'll test that patch and see if I can get it merged.

Change 159217 had a related patch set uploaded by BryanDavis:
Wiki installer now conditional on existence of database

https://gerrit.wikimedia.org/r/159217

Change 159217 merged by jenkins-bot:
Wiki installer now conditional on existence of database

https://gerrit.wikimedia.org/r/159217

The merged patch fixed the problem for me, but I'd like Yuri to confirm before closing.