Wikimedia Bugzilla is closed!

(In reply to Jon from comment #0)

> https://integration.wikimedia.org/ci/job/mwext-MobileFrontend-qunit/6228/

Since Modern is no longer in core, why is it in the skins directory?

https://gerrit.wikimedia.org/r/#/c/135429/ added a warning message whenever an old skin is found under Mediawiki /skins/ . The skins have been removed from core but are still present in Jenkins workspace.

The reason is that since mediawiki/core.git is so huge to clone, we are using git archive to grab a snapshot of MediaWiki in the Jenkins job workspace.  A side effect is that files removed from the repository are not removed from the workspace.


A potential way to solve it would be to have mediawiki core in a dedicated directory which we would be able to delete recursively before taking a snapshot with git archive.  From there we could symlink extensions from their dedicated directories under mediawiki /extensions/ .

I also have an ongoing project to clone multiple repositories which might let us solve the issue.

Meanwhile my recommendation is to revert the deprecation patch ( https://gerrit.wikimedia.org/r/#/c/135429/ ) until CI has catched up and ensure MediaWiki code base is clean in the Jenkins workspaces.

Change 138626 had a related patch set uploaded by Hashar:
Revert "Officially deprecate skin autodiscovery"

https://gerrit.wikimedia.org/r/138626

We could clean up the /skins/ directory in all workspaces potentially, but I can't remember whether the same job could use a REL1_*** codebase which would bring back the skins.

(In reply to Antoine "hashar" Musso from comment #2)
> The reason is that since mediawiki/core.git is so huge to clone, we are
> using git archive to grab a snapshot of MediaWiki in the Jenkins job
> workspace.  A side effect is that files removed from the repository are not
> removed from the workspace.

Uck :/

(In reply to Antoine "hashar" Musso from comment #4)
> We could clean up the /skins/ directory in all workspaces potentially, but I
> can't remember whether the same job could use a REL1_*** codebase which
> would bring back the skins.

Lets do that for now. People shouldn't be backporting stuff to REL branches in the near future, which gives us a little more time to fix jenkins to delete files properly.

Sorry Kunal :-(  And you have no idea how much it piss me off to have to revert such a patch from core :-(

Taking mwext-VisualEditor-qunit as a usecase there's a couple facts:

1) As being a repo not having the target git repo (VE) at its root, it has git:clean:false and git:wipe:true (defaults), see 'use-zuul-for-mw-ext' in jenkins-job-builder-config and confirmed on https://integration.wikimedia.org/ci/job/mwext-VisualEditor-qunit/configure (expand the Advanced sections)

* [ ] Clean after checkout [_]
      (runs `git clean`)
* [x] Wipe out workspace before build


2) Looking at an example build:
   https://integration.wikimedia.org/ci/job/mwext-VisualEditor-qunit/10021/consoleFull
   - Wiping out workspace first.
   - Cloning repository git://zuul.eqiad.wmnet/mediawiki/extensions/VisualEditor
   - /srv/deployment/integration/slave-scripts/bin/mw-core-get.sh master

   In that order.

3) Even if this wouldn't happen, I think it's fine to use full wipe instead of git-clean since all these run on the local system and should be able to use hardlinks during the cloning (for the extension repo), and for mwcore we use archive from a central repo /srv/ssd/gerrit/mediawiki/core.git so its' fine to wipe the workspace of the extension repo)

4) Confirmed by running mkdir ~/test; export WORKSPACE=~/test; /srv/ssd/gerrit/mediawiki/core.git master; that the central git repo is not dirty, the old files and skin directories are not in there.


So then why do we have RELEASE-NOTES-1.22 and skins/Modern.php in mwext-VisualEditor-qunit workspace still?

Rephrasing bug summary. For the record, this only affects files that have actually landed in master. We do clean out files from proposed patches properly, this bug is specifically due to and related to fetching of latest master within extensions.

I went ahead and merged Jon's revert so that mobile and VE development are not blocked.

Change 138626 merged by jenkins-bot:
Revert "Officially deprecate skin autodiscovery"

https://gerrit.wikimedia.org/r/138626

Let's get this issue fixed now so we can work towards merging Bartosz's patch again.

(In reply to Krinkle from comment #7)
> Taking mwext-VisualEditor-qunit as a usecase there's a couple facts:
> 
> 1) As being a repo not having the target git repo (VE) at its root, it has
> git:clean:false and git:wipe:true (defaults), see 'use-zuul-for-mw-ext' in
> jenkins-job-builder-config and confirmed on
> https://integration.wikimedia.org/ci/job/mwext-VisualEditor-qunit/configure
> (expand the Advanced sections)
> 
> * [ ] Clean after checkout [_]
>       (runs `git clean`)
> * [x] Wipe out workspace before build
> 
> 
> 2) Looking at an example build:
>   
> https://integration.wikimedia.org/ci/job/mwext-VisualEditor-qunit/10021/
> consoleFull
>    - Wiping out workspace first.
>    - Cloning repository
> git://zuul.eqiad.wmnet/mediawiki/extensions/VisualEditor
>    - /srv/deployment/integration/slave-scripts/bin/mw-core-get.sh master
> 
>    In that order.
> 
> 3) Even if this wouldn't happen, I think it's fine to use full wipe instead
> of git-clean since all these run on the local system and should be able to
> use hardlinks during the cloning (for the extension repo), and for mwcore we
> use archive from a central repo /srv/ssd/gerrit/mediawiki/core.git so its'
> fine to wipe the workspace of the extension repo)
> 
> 4) Confirmed by running mkdir ~/test; export WORKSPACE=~/test;
> /srv/ssd/gerrit/mediawiki/core.git master; that the central git repo is not
> dirty, the old files and skin directories are not in there.
> 
> 
> So then why do we have RELEASE-NOTES-1.22 and skins/Modern.php in
> mwext-VisualEditor-qunit workspace still?


Looks like the real underlying bug is that the 'non-git' wipe option we have (the one that fully removes the work tree and will require a new git-clone on each build, the option we actually *are* using for extensions jobs) that wipe option (not git-clean) still restricts itself to where the git-clone will be. It doesn't wipe the workspace, it wipes workspace/{basedir}.

Change 138648 had a related patch set uploaded by Krinkle:
defaults: Make values for 'wipe-workspace' and 'clean' explicit

https://gerrit.wikimedia.org/r/138648

Change 138657 had a related patch set uploaded by Krinkle:
mw-core-get: Clean workspace first

https://gerrit.wikimedia.org/r/138657

Change 138657 merged by jenkins-bot:
mw-core-get: Clean workspace first

https://gerrit.wikimedia.org/r/138657

Confirmed fix by seeing that old files in skins/* are gone in workspaces on gallium, and confirmed again by touching a file `krinkle.txt` and confirming it was gone the next build.

Thank you!

Change 138682 had a related patch set uploaded by Bartosz Dziewoński:
Revert "Revert "Officially deprecate skin autodiscovery""

https://gerrit.wikimedia.org/r/138682

Change 138648 merged by Krinkle:
defaults: Make values for 'wipe-workspace' and 'clean' explicit

https://gerrit.wikimedia.org/r/138648

Change 138682 merged by jenkins-bot:
Revert "Revert "Officially deprecate skin autodiscovery""

https://gerrit.wikimedia.org/r/138682