Wikimedia Bugzilla is closed!

I have a bash script that seems to be working to automate this, but I'm not entirely sure of the best way to tie it into deployment-salt. We could do one of several things:

* Cron it on deployment-salt and email *someone* when it blows up
* Add it to operations/puppet as an add-on for role::puppet::self that anyone could use (deploy script, setup cron, allow configuration of warning emails)
* Make deployment-salt a Jenkins slave and set the script up to run as a post-merge Zuul task on operations/puppet

The script I have written is available on deployment-salt.eqiad.wmflabs as /home/bd808/git-sync-upstream

Note that deployment-prep is special in its use of role::puppet::self in that it always uses HEAD (AFAIK).  Typically, there will be local changes and/or bisects, so automatic updates shouldn't be the default.

(In reply to Tim Landscheidt from comment #2)
> Note that deployment-prep is special in its use of role::puppet::self in
> that it always uses HEAD (AFAIK).

We are actually using a model where we layer pending gerrit patches on top of origin/production via cherry-pick and update by rebasing those local commits on the origin branch. The nice thing about this is that when a commit is actually merged in gerrit the rebase will detect that and discard the local commit in favor of the merged version. The number of pending patches we have at any given time has varied between zero (doesn't happen very often) and nine or ten with four or five being normal.

> Typically, there will be local changes and/or bisects, so automatic
> updates shouldn't be the default.

The model of rebasing against an upstream tracking branch should work for almost anyone, but I agree that automatic updating should be a separate role that each project chooses to apply if we go in that direction.

I have setup a temporary cron job on deployment-salt as user bd808 to update the operations/puppet.git checkout using ~/bd808/git-sync-upstream. It runs hourly at 17 minutes after the hour and logs to /home/bd808/git-sync.log.

If this works reliably for a couple of days I'll look into setting up a more permanent solution.

Created attachment 15801 [details]
bash script to update puppet checkout in /var/lib/git/operations/puppet

Attaching the script that has been running on deployment-salt for a couple of weeks. 

The script generally runs with no issues but occasionally gets tripped up by the merge of a cherry-picked patch that has been amended in gerrit but not updated in deployment-salt to use the latest version. When this happens, I have manually corrected via:
* git rebase --abort
* git rebase --interactive
** Remove the merged patch causing the conflict by deleting its line

At the moment I'm leaning towards using Jenkins to manage the repo on deployment-salt as this could be hooked into the operations/puppet gerrit repo to update beta on each upstream commit and would also provide a means of notification in the case of rebase failures. The unfortunate drawback to this would be that it's a solution very specific to beta and not a general solution for anyone using a self-hosted puppetmaster in labs.

Change 151099 had a related patch set uploaded by BryanDavis:
beta: puppet rebase script

https://gerrit.wikimedia.org/r/151099

Patch applied in beta via cherry-pick (how meta). The cron that was running a version of this script from my home dir has been disabled.

Change 151099 merged by Filippo Giunchedi:
beta: puppet rebase script

https://gerrit.wikimedia.org/r/151099

Thanks Filippo!