Last modified: 2014-07-22 01:08:36 UTC

Wikimedia Bugzilla is closed!

Wikimedia migrated from Bugzilla to Phabricator. Bug reports are handled in Wikimedia Phabricator.
This static website is read-only and for historical purposes. It is not possible to log in and except for displaying bug reports and their history, links might be broken. See T70260, the corresponding Phabricator task for complete and up-to-date bug report information.
Bug 68260 - Jenkins: browserstests puppet manifest fails on "File[/var/lib/elasticsearch]:ensure"
Jenkins: browserstests puppet manifest fails on "File[/var/lib/elasticsearch]...
Status: RESOLVED FIXED
Product: Wikimedia
Classification: Unclassified
Continuous integration (Other open bugs)
wmf-deployment
All All
: Normal major (vote)
: ---
Assigned To: Nobody - You can work on this!
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2014-07-19 03:14 UTC by Krinkle
Modified: 2014-07-22 01:08 UTC (History)
4 users (show)

See Also:
Web browser: ---
Mobile Platform: ---
Assignee Huggle Beta Tester: ---


Attachments

Description Krinkle 2014-07-19 03:14:44 UTC
(note, this is from setting up a new ubuntu-precise, not ubuntu-trusty image and provisioning it with role::ci::slave::labs from the integration puppetmaster)


Notice: /Stage[main]/Role::Ci::Slave::Browsertests/File[/var/lib/elasticsearch]: Not removing directory; use 'force' to override
Notice: /Stage[main]/Role::Ci::Slave::Browsertests/File[/var/lib/elasticsearch]: Not removing directory; use 'force' to override
Error: Could not remove existing file
Error: /Stage[main]/Role::Ci::Slave::Browsertests/File[/var/lib/elasticsearch]/ensure: change from directory to link failed: Could not remove existing file
Notice: /Stage[main]/Elasticsearch/File[/var/log/elasticsearch/elasticsearch_index_search_slowlog.log]: Dependency File[/var/lib/elasticsearch] has failures: true
Warning: /Stage[main]/Elasticsearch/File[/var/log/elasticsearch/elasticsearch_index_search_slowlog.log]: Skipping because of failed dependencies
Notice: /Stage[main]/Elasticsearch/File[/var/log/elasticsearch/elasticsearch_index_indexing_slowlog.log]: Dependency File[/var/lib/elasticsearch] has failures: true
Warning: /Stage[main]/Elasticsearch/File[/var/log/elasticsearch/elasticsearch_index_indexing_slowlog.log]: Skipping because of failed dependencies
Notice: /Stage[main]/Elasticsearch/File[/etc/logrotate.d/elasticsearch]: Dependency File[/var/lib/elasticsearch] has failures: true
Warning: /Stage[main]/Elasticsearch/File[/etc/logrotate.d/elasticsearch]: Skipping because of failed dependencies
Notice: /Stage[main]/Elasticsearch/File[/etc/elasticsearch/elasticsearch.yml]: Dependency File[/var/lib/elasticsearch] has failures: true
Warning: /Stage[main]/Elasticsearch/File[/etc/elasticsearch/elasticsearch.yml]: Skipping because of failed dependencies
Notice: /Stage[main]/Elasticsearch/File[/var/log/elasticsearch/elasticsearch.log]: Dependency File[/var/lib/elasticsearch] has failures: true
Warning: /Stage[main]/Elasticsearch/File[/var/log/elasticsearch/elasticsearch.log]: Skipping because of failed dependencies
Notice: /Stage[main]/Elasticsearch/File[/etc/elasticsearch/logging.yml]: Dependency File[/var/lib/elasticsearch] has failures: true
Warning: /Stage[main]/Elasticsearch/File[/etc/elasticsearch/logging.yml]: Skipping because of failed dependencies
Notice: /Stage[main]/Elasticsearch/File[/etc/default/elasticsearch]: Dependency File[/var/lib/elasticsearch] has failures: true
Warning: /Stage[main]/Elasticsearch/File[/etc/default/elasticsearch]: Skipping because of failed dependencies
Notice: /Stage[main]/Elasticsearch/Service[elasticsearch]: Dependency File[/var/lib/elasticsearch] has failures: true
Warning: /Stage[main]/Elasticsearch/Service[elasticsearch]: Skipping because of failed dependencies
Comment 1 Gerrit Notification Bot 2014-07-19 03:21:37 UTC
Change 147713 had a related patch set uploaded by Krinkle:
contint: Remove browsertests class from role::ci::slave::labs

https://gerrit.wikimedia.org/r/147713
Comment 2 Antoine "hashar" Musso (WMF) 2014-07-21 18:17:16 UTC
The problem is that the elasticsearch package creates /var/lib/elasticsearch and populate it with some contents.  On labs,  /var/ is quite small, so we have to mount some extra disk space on /mnt/ and then link /var/lib/elasticsearch to /mnt/elasticsearch.

Since the elasticsearch package is installed before the mount and symlink are realized, puppet bails out because /var/lib/elasticsearch is already there.

The require => statements in puppet should not let that happens.  The manual workaround would be:

- remove elasticsearch package : dpkg --purge elastic search
- delete the directory:  rm -fR /var/lib/elasticsearch

Make sure /mnt is a lvm mount of the instance extra disk.

Re run puppet.



If that fails, I am willing to fix it up / figure out what is wrong in puppet definition. Would just need the instance name :)
Comment 3 Gerrit Notification Bot 2014-07-21 18:17:55 UTC
Change 147713 abandoned by Hashar:
contint: Remove browsertests class from role::ci::slave::labs

Reason:
Abandoning, I have posted some explanation and a possible manual workaround on the bug report : https://bugzilla.wikimedia.org/show_bug.cgi?id=68260#c2

https://gerrit.wikimedia.org/r/147713
Comment 4 Antoine "hashar" Musso (WMF) 2014-07-21 21:12:12 UTC
I have created the instance integration-slave1004-trusty.eqiad.wmflabs , switched it to use the project puppetmaster ( integration-puppetmaster ).

Applying role::ci::slave::labs on it.

Puppet run is logged in /var/log/puppet.log starting with:

 Info: Applying configuration version '1405740130'
Comment 5 Antoine "hashar" Musso (WMF) 2014-07-22 01:08:36 UTC
That works for me on integration-slave1004-trusty . I guess the slaves had an obsolete puppet manifest deployed or we had some transient error.

Note You need to log in before you can comment on or make changes to this bug.


Navigation
Links