Last modified: 2014-06-08 02:29:34 UTC
CORE repo broken ? "git pull" fails: fatal: pack has 14 unresolved deltas fatal: index-pack failed
root@openid-wiki:/srv/mediawiki# git pull remote: Counting objects: 894, done remote: Finding sources: 100% (477/477) remote: Getting sizes: 100% (489/489) remote: Compressing objects: 100% (72/72) remote: Total 477 (delta 345), reused 405 (delta 331) Receiving objects: 100% (477/477), 279.19 KiB, done. Resolving deltas: 96% (379/393), completed with 222 local objects. fatal: pack has 14 unresolved deltas fatal: index-pack failed root@openid-wiki:/srv/mediawiki#
root@openid-wiki:/srv/mediawiki# git fsck Checking object directories: 100% (256/256), done. Checking objects: 100% (43356/43356), done.
WFM. This might be related to working on a shallow repo )side note: my local full clone (with some branches fetched apart from master) has ~400k objects instead of ~40k).
In my view, it is a blocker
even when having a shallow repo, git pull must not fail
This WFM as well on a shallow clone. Can you try on a fresh clone?
I can try. It means, that the puppet class which installed that wiki onm labsconsole is corrupted, apparently.
What does git fsck --full give you?
pls. wear your glasses : https://bugzilla.wikimedia.org/show_bug.cgi?id=44129#c2
I saw that, and you only seemed to have done `git fsck`. I asked for the results of `git fsck --full`
Uh, *sorry*, root@openid-wiki:/srv/mediawiki# git fsck --full Checking object directories: 100% (256/256), done. Checking objects: 100% (43356/43356), done. root@openid-wiki:/srv/mediawiki#
Could you clone a fresh copy of the repository?
(In reply to comment #12) > Could you clone a fresh copy of the repository? Yes. "Worksforme" But looks, as if the "puppet class" has a corrupted repo. But anyway, if you wish, you can close this bug.
[ Reset to UNCO and moving to Wikimedia Labs. ]
Wondering if further info would help to track down the offending commits/deltas, e.g. by using the env variable GIT_TRACE=1 for git pull?
(In reply to comment #15) > Wondering if further info would help to track down the offending > commits/deltas, e.g. by using the env variable > GIT_TRACE=1 > for git pull? I made a tgz of the whole installation (with that problem) before I made a fresh clone, perhaps I can find some time to track down the problem.
Decreasing priority as nobody can reproduce this, and severity as there is a workaround (clone a fresh copy).
closing this. nobody cares, and since I cloned a fresh copy, the problem does not exist anymore.
Reopening this bug... I can reproduce it. I used puppet to install mediawiki/Wikidata on two Labs instances a week ago. Today, I ran git pull for the first time since, and it didn't work. On a third instance, I first had puppet install it two days ago. Same error, but less "deltas". root@wikidata-testrepo /srv/mediawiki (master)$ git pull remote: Counting objects: 3626, done remote: Finding sources: 100% (1746/1746) remote: Getting sizes: 100% (2360/2360) remote: Compressing objects: 100% (480/480) remote: Total 1746 (delta 1247), reused 1310 (delta 1083) Receiving objects: 100% (1746/1746), 3.84 MiB | 4.24 MiB/s, done. Resolving deltas: 88% (1309/1479), completed with 652 local objects. fatal: pack has 170 unresolved deltas fatal: index-pack failed root@wikidata-testrepo /srv/mediawiki (master)$ git fsck --full Checking object directories: 100% (256/256), done. Checking objects: 100% (43644/43644), done. Note that I *only* encountered this problem when pulling core, everything worked fine for the extensions.
I'm still not convinced it's Gerrit's fault at all. From my (brief) time on Google, it looks like it could be a generic problem with shallow clones.
Looking at operations/puppet/manifests/generic-definitions.pp where git::clone is defined, it says $depth="full", which doesn't sound shallow.
On a newly create mediawiki cloned from git, git log only shows me the latest three commits, though.
(In reply to comment #21) > Looking at operations/puppet/manifests/generic-definitions.pp where > git::clone > is defined, it says > > $depth="full", > > which doesn't sound shallow. Looking at operations/puppet/manifests/mediawiki.pp where mediawiki/core is actually cloned, it says depth => 1, which makes a shallow clone.
Cannot reproduce it using the test repo over http: $ git clone --depth 1 http://l.saper.info:8888/test-project.git $ cd test-project $ git log commit 425953c4bb34df0e3304179261097766929828f4 Author: Marcin Cieślak <marcin.cieslak@gmail.com> Date: Fri Apr 13 09:23:43 2012 +0200 Force merge Change-Id: I419a1d16bb06600d8c9db2907bb8aef1bbf0593a commit 15e2af9cde0578f21056980d7f608f459ce75af9 Author: Marcin Cieślak <marcin.cieslak@gmail.com> Date: Fri Apr 13 09:09:50 2012 +0200 More Change-Id: Iefe0f48624a059474ff9f69da02bdfe59a0394db $ (...) after push of some change to the repository: $ git pull --depth 1 remote: Counting objects: 4, done remote: Finding sources: 100% (3/3) Unpacking objects: 100% (3/3), done. remote: Total 3 (delta 0), reused 3 (delta 0) From http://l.saper.info:8888/test-project 425953c..ea80f4a master -> origin/master Updating 425953c..ea80f4a Fast-forward DOIT | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) create mode 100644 DOIT $ git log commit ea80f4a2e524a3db676cb4401c620c8fb5f9d291 Author: saper <saper@saper.info> Date: Mon Feb 11 11:07:25 2013 +0100 Another change Change-Id: I0f92085acb6281b8539f164d36a35f12cfae983f commit 425953c4bb34df0e3304179261097766929828f4 Author: Marcin Cieślak <marcin.cieslak@gmail.com> Date: Fri Apr 13 09:23:43 2012 +0200 Force merge Change-Id: I419a1d16bb06600d8c9db2907bb8aef1bbf0593a But this was fast-forward update, using gerrit close to master (2.5.1-1020-gef6b83b) and git version 1.7.3.4
Ok got it reproduced with git talking to my test gerrit instance (above), over HTTP using mediawiki core repository. $ git remote -v origin http://l.saper.info:8888/mediawiki-core (fetch) origin http://l.saper.info:8888/mediawiki-core (push) $ git branch -vv * master cd382de [origin/master] Merge "Revert "Updated RELEASE NOTES for 1.20"" (this some local change for testing) $ git pull remote: Counting objects: 24778, done remote: Finding sources: 100% (22391/22391) remote: Total 22391 (delta 18522), reused 21888 (delta 18522) Receiving objects: 100% (22391/22391), 23.16 MiB | 1.04 MiB/s, done. Resolving deltas: 84% (15736/18522), completed with 1237 local objects. fatal: pack has 2786 unresolved deltas fatal: index-pack failed
As discussed on IRC with Marcin C.: - this is no puppet issue, he was not using puppet when he reproduced the error. - it is *probably* not related to http(s) vs. ssh either - the mediawiki core pack seems to lack information; normally git pull is possible on shallow clones.
Using direct access to the GIT repository (avoiding Gerrit and JGit) seems to avoid the problem: $ git remote add directssh ssh://l.saper.info/home/saper/sw/test-site-pg/git/mediawiki-core.git $ git fetch -k --depth 1 directssh refs/heads/master:refs/remotes/directssh/master seems to load objects and the tree can be updated with $ git merge directssh/master For now I suspect that JGit does not fully support sending object packs for shallow clients.
(In reply to comment #9) > pls. wear your glasses : > https://bugzilla.wikimedia.org/show_bug.cgi?id=44129#c2 can we have more respect, even in bugzilla? There are other ways to express the same and they are more friendly. Thanks... I flagged this as normal priority, since it affects only 1 user and is likely a problem of local copy... If puppet class is broken, fix it! But IMHO I don't see why a puppet class would clone a repository in a way that it's broken later.
(In reply to comment #28) > I flagged this as normal priority, since it affects only 1 user and is > likely a > problem of local copy... If puppet class is broken, fix it! But IMHO I don't > see why a puppet class would clone a repository in a way that it's broken > later. It's been replicated, and it's not puppet. Please see recent comments from Marcin.
ok that's true but is it really "critical"?
This issue may not be related to git at all. Were you using /data/project or /home to store your repository? If so, it's possible that glusterfs got corrupted (git files were damaged or aren't accessible), which happens quite a lot these days and may need a recovery. Usually a best way is to reboot and mount glusterfs again using latest version of gluster daemon. If it's not a problem of gluster I don't see how it is related to wikimedia labs.
(In reply to comment #30) > ok that's true but is it really "critical"? No, I don't think it's critical (I wasn't disagreeing with lowering the priority). I was just making sure we were all clear that it's a jgit bug most likely. I highly doubt it's a gluster issue, since replication of the bug was possible locally. In fact, moving it to Git/Gerrit where it belongs.
I cloned mediawiki to /srv/mediawiki, so no glusterfs partition.
(In reply to comment #27) > For now I suspect that JGit does not fully support sending object packs for > shallow clients. Is this still the case?
This may now be moot... the actual use case (in Labs) now does a full clone, thanks to performance improvements in gerrit.
I reported this bug. I think, I can now close it.
Related URL: https://gerrit.wikimedia.org/r/65873 (Gerrit Change If5cf784f37aea49ca6c96c9fb1f50104d073b5b2)
The snapshots tool for MediaWiki nightly branches started throwing this warning recently -- -- Thu, 27 Feb 2014 17:00:07 +0000 -- Starting update process for snapshots of mediawiki-core... -- Forced clean up and reset... $ rm -f .git/index.lock $ git clean -d -x --force -q $ git reset --hard HEAD -q $ git checkout 'master' -q Fetch updates from remotes... fatal: pack has 21 unresolved deltas fatal: index-pack failed error: Could not fetch origin As a result 1.23wmf15 and 1.23wmf16 never made it there, for example. Is there something I can do (short of rm -rf the clone, and re-init/re-clone every time) to defend against this so that I can have this cronjob run without any worries?
[17:24 UTC] local-snapshots at tools-login in ~/src/mwSnapshots/remotes/mediawiki-core (master) $ git fsck --full Checking object directories: 100% (256/256), done. Checking objects: 100% (63675/63675), done. dangling commit 6146a7d310181b23618b74fcba580f8919a8ba6d dangling commit 10ba3d7caa1e4bb4d521384bebbf42976cea4a22
Didn't solve it apparently. [17:24 UTC] local-snapshots at tools-login in ~/src/mwSnapshots/remotes/mediawiki-core (master) $ git remote update Fetching origin remote: Counting objects: 10144, done remote: Finding sources: 100% (1201/1201) remote: Getting sizes: 100% (286/286) remote: Compressing objects: 100% (2020458/2020458) remote: Total 1201 (delta 788), reused 1023 (delta 768) Receiving objects: 100% (1201/1201), 2.84 MiB | 2.58 MiB/s, done. Resolving deltas: 97% (823/844), completed with 331 local objects. fatal: pack has 21 unresolved deltas fatal: index-pack failed error: Could not fetch origin 1 $ git fsck --full Checking object directories: 100% (256/256), done. Checking objects: 100% (63675/63675), done. dangling commit 6146a7d310181b23618b74fcba580f8919a8ba6d dangling commit 10ba3d7caa1e4bb4d521384bebbf42976cea4a22 $ git remote update Fetching origin remote: Counting objects: 10144, done remote: Finding sources: 100% (1201/1201) remote: Getting sizes: 100% (286/286) remote: Compressing objects: 100% (2020458/2020458) remote: Total 1201 (delta 788), reused 1023 (delta 768) Receiving objects: 100% (1201/1201), 2.84 MiB | 2.61 MiB/s, done. Resolving deltas: 97% (823/844), completed with 331 local objects. fatal: pack has 21 unresolved deltas fatal: index-pack failed error: Could not fetch origin
I tried the trick saper mentioned about using another git remote (e.g the git.wikimedia.org mirror) to fix it up. $ git fetch origin .. fatal: pack has 26 unresolved deltas fatal: index-pack failed $ git remote add mirror https://git.wikimedia.org/git/mediawiki/core.git $ git remote -v mirror https://git.wikimedia.org/git/mediawiki/core.git (fetch) mirror https://git.wikimedia.org/git/mediawiki/core.git (push) origin https://gerrit.wikimedia.org/r/p/mediawiki/core.git (fetch) origin https://gerrit.wikimedia.org/r/p/mediawiki/core.git (push) $ git fetch mirror remote: Counting objects: 3886, done remote: Finding sources: 100% (1927/1927) remote: Getting sizes: 100% (3130/3130) remote: Compressing objects: 99% (59991/60007) remote: Total 1927 (delta 716), reused 1044 (delta 495) Receiving objects: 100% (1927/1927), 14.18 MiB | 2.99 MiB/s, done. Resolving deltas: 75% (903/1197), completed with 170 local objects. fatal: pack has 294 unresolved deltas fatal: index-pack failed # Still fails.. Trying again with limited depth $ git fetch -k --depth 1 mirror remote: Counting objects: 9253, done remote: Finding sources: 100% (9253/9253) remote: Getting sizes: 100% (8258/8258) remote: Compressing objects: 99% (165096/165109) remote: Total 9253 (delta 2901), reused 4979 (delta 978) Receiving objects: 100% (9253/9253), 55.62 MiB | 3.59 MiB/s, done. Resolving deltas: 100% (3895/3895), done. From https://git.wikimedia.org/git/mediawiki/core * [new branch] wmf/1.23wmf16 -> mirror/wmf/1.23wmf16 * [new tag] 1.22.3 -> 1.22.3 # That's better, now trying origin again $ git remote rm mirror $ git fetch origin remote: Counting objects: 14492, done remote: Finding sources: 100% (8/8) remote: Getting sizes: 100% (2/2) remote: Compressing objects: 100% (7196/7196) remote: Total 8 (delta 4), reused 5 (delta 4) Unpacking objects: 100% (8/8), done. fatal: unresolved deltas left after unpacking fatal: unpack-objects failed -- Still broken..
Worked around it for now by force fetching using `fetch -k --depth 1` and then switching to only having git.wikimedia.org as the remote (dropping gerrit.wikimedia.org entirely for now): $ git remote add mirror https://git.wikimedia.org/git/mediawiki/core.git $ git remote rm origin $ git remote rename mirror origin $ git remote -v origin https://git.wikimedia.org/git/mediawiki/core.git (fetch) origin https://git.wikimedia.org/git/mediawiki/core.git (push) $ git fetch -k --depth 1 origin remote: Counting objects: 33054, done remote: Finding sources: 100% (27705/27705) remote: Getting sizes: 100% (25252/25252) remote: Compressing objects: 99% (1019631/1019645) remote: Total 27705 (delta 15890), reused 14921 (delta 7700) Receiving objects: 100% (27705/27705), 141.58 MiB | 10.39 MiB/s, done. Resolving deltas: 100% (19416/19416), completed with 2419 local objects. From https://git.wikimedia.org/git/mediawiki/core .. * [new branch] REL1_21 -> origin/REL1_21 * [new branch] REL1_22 -> origin/REL1_22 .. * [new branch] wmf/1.23wmf15 -> origin/wmf/1.23wmf15 * [new branch] wmf/1.23wmf16 -> origin/wmf/1.23wmf16 $ git fetch --prune $ git checkout 'master' 0
Nevermind, that was a false positive. fatal: unresolved deltas left after unpacking fatal: unpack-objects failed $ git fetch --prune Command failed: $ git fetch --prune Return: 128 Created a new git clone from git.wikimedia.org in a temp directory and removed the old one and moved this one in its place. Whatever happens, if you get this bug, don't waste time trying to fix it. Just re-clone and hope you don't have any local commits or branches you care about.
Meh.
