Wikimedia Bugzilla is closed!

It will not be allowed in operations/puppet due to both composer and pulling from github.
This is what was tried before doing it the way it is currently done!

I understand that much was tried in that direction. And it probably didn't help that composer on its own was not accepted at that time.

composer: That is now solved in operations/puppet.git/modules/contint/manifests/slave-scripts.pp .

pulling from github/packagist: I think in general having such a dependency on an external service in our software build process is a bad idea. Having that directly in puppet would be even worse. So yes, to solve this bug the puppet code may not pull from github. What a jenkis or cron job does is another matter. (That still wouldn't be good, but one has to start somewhere.)

I updated the local git puppet repo (and labs/private) and now puppet agent -tv fails due to some syntax error:

sudo puppet agent -tv
info: Retrieving plugin
info: Loading facts in /etc/puppet/modules/base/lib/facter/default_gateway.rb
info: Loading facts in /etc/puppet/modules/base/lib/facter/physicalcorecount.rb
info: Loading facts in /etc/puppet/modules/base/lib/facter/ec2id.rb
info: Loading facts in /etc/puppet/modules/base/lib/facter/projectgid.rb
info: Loading facts in /etc/puppet/modules/vm/lib/facter/meminbytes.rb
info: Loading facts in /etc/puppet/modules/stdlib/lib/facter/puppet_vardir.rb
info: Loading facts in /etc/puppet/modules/stdlib/lib/facter/facter_dot_d.rb
info: Loading facts in /etc/puppet/modules/stdlib/lib/facter/pe_version.rb
info: Loading facts in /etc/puppet/modules/stdlib/lib/facter/root_home.rb
info: Loading facts in /etc/puppet/modules/puppet_statsd/lib/facter/puppet_config_dir.rb
info: Loading facts in /etc/puppet/modules/apt/lib/facter/apt.rb
info: Loading facts in /var/lib/puppet/lib/facter/puppet_config_dir.rb
info: Loading facts in /var/lib/puppet/lib/facter/default_gateway.rb
info: Loading facts in /var/lib/puppet/lib/facter/physicalcorecount.rb
info: Loading facts in /var/lib/puppet/lib/facter/puppet_vardir.rb
info: Loading facts in /var/lib/puppet/lib/facter/ec2id.rb
info: Loading facts in /var/lib/puppet/lib/facter/facter_dot_d.rb
info: Loading facts in /var/lib/puppet/lib/facter/pe_version.rb
info: Loading facts in /var/lib/puppet/lib/facter/meminbytes.rb
info: Loading facts in /var/lib/puppet/lib/facter/projectgid.rb
info: Loading facts in /var/lib/puppet/lib/facter/root_home.rb
info: Loading facts in /var/lib/puppet/lib/facter/apt.rb
err: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not parse for environment production: Syntax error at 'default'; expected '}' at /etc/puppet/manifests/role/phabricator.pp:89 on node wikidata-builder3.eqiad.wmflabs
warning: Not using cache on failed catalog
err: Could not retrieve catalog; skipping run

and just a side note, the wikidata-builder3 hasn't been making any builds the past week due to https://github.com/wmde/WikidataBuilder/issues/29

as a workaround and imho, better, easier to maintain solution, I proposed to add a grunt file to WikidataBuildResources and we might be able to just use that:

https://github.com/wmde/WikidataBuildResources/pull/9

if that's agreeable, then we need to either make a new instance or modify this one accordingly.

Replied to comment #3 in https://bugzilla.wikimedia.org/show_bug.cgi?id=71411#c4 .

Quickly looking at this again I don't see anything the wikidata slaves have that the normal ones don't. We should just change all the wikidata jobs to run on the normal jenkins slaves in labs. Then the jenkins slave instance from wikidata-build can be removed.

If composer is now solved in operations/puppet.git/modules/contint/manifests/slave-scripts.pp then yes the wd and regular slaves should be the same, even if composer is in a different place (easy to fix)

We don't need to add any stuff pulling from github into puppet, well unless I am missing something.

If we do make this move its definitely worth adding a few more slaves to the LabsContInt slaves group as we use a lot of resources (mainly time actually)!

Infact, there are a few more slaves than when I last looked so should be fine ;p

> Infact, there are a few more slaves than when I last looked so should be fine ;p

And the wikidata instances will be reclaimed. So that is a null sum for labs infra.


> If composer is now solved in operations/puppet.git/modules/contint/manifests/slave-scripts.pp then

Looking at integration/config.git , the wikidata jobs are using:

 timeout 300 /usr/local/bin/composer install


The composer validate job has:

  /srv/deployment/integration/composer/vendor/bin/composer 


We might want to update the integration/composer.git repo and switch the wikidata jobs the Wikimedia slaves.