Last modified: 2013-07-30 00:10:42 UTC

Wikimedia Bugzilla is closed!

Wikimedia migrated from Bugzilla to Phabricator. Bug reports are handled in Wikimedia Phabricator.
This static website is read-only and for historical purposes. It is not possible to log in and except for displaying bug reports and their history, links might be broken. See T35558, the corresponding Phabricator task for complete and up-to-date bug report information.
Bug 33558 - update.php stuck on update revision on large db
update.php stuck on update revision on large db
Status: NEW
Product: MediaWiki
Classification: Unclassified
Installer (Other open bugs)
1.20.x
All Linux
: Normal normal (vote)
: ---
Assigned To: Nobody - You can work on this!
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2012-01-06 12:54 UTC by Peter Bena
Modified: 2013-07-30 00:10 UTC (History)
4 users (show)

See Also:
Web browser: ---
Mobile Platform: ---
Assignee Huggle Beta Tester: ---


Attachments

Description Peter Bena 2012-01-06 12:54:07 UTC
Hi, I am working on deployment.wmflabs.org together with Mark Herhberger

I imported full db of simple wiki there to mediawiki 1.18wmf1 using mwimport, the wiki db was ok, however when I attempted to upgrade to latest head of svn, the update.php stuck on updating rev_sha1 on revision table

the process is now running, but seems to be idle,

desc revision;
+----------------+---------------------+------+-----+----------------+----------------+
| Field          | Type                | Null | Key | Default        | Extra          |
+----------------+---------------------+------+-----+----------------+----------------+
| rev_id         | int(10) unsigned    | NO   | PRI | NULL           | auto_increment |
| rev_page       | int(10) unsigned    | NO   | MUL | NULL           |                |
| rev_text_id    | int(10) unsigned    | NO   |     | NULL           |                |
| rev_comment    | tinyblob            | NO   |     | NULL           |                |
| rev_user       | int(10) unsigned    | NO   | MUL | 0              |                |
| rev_user_text  | varbinary(255)      | NO   | MUL |                |                |
| rev_timestamp  | binary(14)          | NO   | MUL |                |                |
| rev_minor_edit | tinyint(3) unsigned | NO   |     | 0              |                |
| rev_deleted    | tinyint(3) unsigned | NO   |     | 0              |                |
| rev_len        | int(10) unsigned    | YES  |     | NULL           |                |
| rev_parent_id  | int(10) unsigned    | YES  |     | NULL           |                |
+----------------+---------------------+------+-----+----------------+----------------+

it seems that update script didn't even start with generation of sha_1 keys for some reason, when I tried this on smaller db I've seen the output after few seconds, however on this db it stuck for some reason. Did anyone experienced anything similar? is it a bug of update.php or there is a better way to convert table
Comment 1 Peter Bena 2012-01-06 12:59:28 UTC
petrb@deployment-dbdump:/tmp$ tail nohup.out
...user_last_timestamp is already nullable.
...user_email key already set on user table.
...up_property in table user_properties already modified by patch patch-up_property.sql.
...uploadstash table already exists.
...user_former_groups table already exists.
...config table already exists.
...type_action key already set on logging table.
...batch conversion of user_options: nothing to migrate. done.
...user table does not contain user_options field.
Adding rev_sha1 field to table revision...

this is there in a log for several hours
Comment 2 Chad H. 2012-01-06 13:34:06 UTC
You could run populateRevisionSha1.php.
Comment 3 Mark A. Hershberger 2012-01-06 20:53:03 UTC
sounds like this should block deployment or at least tarball.
Comment 4 Chad H. 2012-01-06 21:17:11 UTC
(In reply to comment #3)
> sounds like this should block deployment or at least tarball.

Definitely doesn't block deployment, we don't use update.php.
Comment 5 Antoine "hashar" Musso (WMF) 2012-01-24 16:53:32 UTC
Does that mean that the Adding rev_sha1 field to table revision... does not output anything? I guess it should either:
- output something every X sha1 are computed
- skip that step and raise a nice warning explicitly asking user to run populateRevisionSha1.php
Comment 6 Sam Reed (reedy) 2012-02-08 18:53:01 UTC
(In reply to comment #5)
> Does that mean that the Adding rev_sha1 field to table revision... does not
> output anything? I guess it should either:
> - output something every X sha1 are computed
> - skip that step and raise a nice warning explicitly asking user to run
> populateRevisionSha1.php

Indeed, with it potentially being really slow, there's little point actually just automagically running it IMHO
Comment 7 Sam Reed (reedy) 2012-02-08 18:54:39 UTC
Oh wait. It's not an issue populating the field, it's the issue of it taking a long time (Asher suggests about 30 hours for enwiki size) to add the field to the revision table
Comment 8 Peter Bena 2012-02-08 23:14:07 UTC
no actually it didn't work at all, after 10 hours of run it didn't even update structure of db, no update of table was done either
Comment 9 Sam Reed (reedy) 2012-02-08 23:33:44 UTC
How big is the table?
Comment 10 Peter Bena 2012-02-09 07:12:59 UTC
I will check the details soon, the db itself (all tables) have around 38gb, it
was a complete dump of simple db
Comment 11 Peter Bena 2012-02-09 10:39:49 UTC
this query was run after the start of update.php

desc revision;
+----------------+---------------------+------+-----+----------------+----------------+
| Field          | Type                | Null | Key | Default        | Extra   
      |
+----------------+---------------------+------+-----+----------------+----------------+
| rev_id         | int(10) unsigned    | NO   | PRI | NULL           |
auto_increment |
| rev_page       | int(10) unsigned    | NO   | MUL | NULL           |         
      |
| rev_text_id    | int(10) unsigned    | NO   |     | NULL           |         
      |
| rev_comment    | tinyblob            | NO   |     | NULL           |         
      |
| rev_user       | int(10) unsigned    | NO   | MUL | 0              |         
      |
| rev_user_text  | varbinary(255)      | NO   | MUL |                |         
      |
| rev_timestamp  | binary(14)          | NO   | MUL |                |         
      |
| rev_minor_edit | tinyint(3) unsigned | NO   |     | 0              |         
      |
| rev_deleted    | tinyint(3) unsigned | NO   |     | 0              |         
      |
| rev_len        | int(10) unsigned    | YES  |     | NULL           |         
      |
| rev_parent_id  | int(10) unsigned    | YES  |     | NULL           |         
      |
+----------------+---------------------+------+-----+----------------+----------------+

it's clear that it didn't even update structure so that I doubt it would actually work. But it happened to me only on larger db's
Comment 12 Sam Reed (reedy) 2012-02-09 14:21:38 UTC
I've no idea how table alters work if you cancel them, rather than letting them execute through
Comment 13 Peter Bena 2012-02-09 14:35:44 UTC
I didn't cancel anything that is how table was looking during the update, I mean that it was running for hours and meanwhile there was no activity in the sql server
Comment 14 Max Semenik 2012-02-21 07:56:10 UTC
(In reply to comment #13)
> I didn't cancel anything that is how table was looking during the update, I
> mean that it was running for hours and meanwhile there was no activity in the
> sql server

What did SHOW PROCESSLIST output during that time of seemingly no activity?
Comment 15 Mark A. Hershberger 2012-03-02 21:38:27 UTC
removing from tarball blocker:

<hashar> I guess we want to try to reproduce 33558 with Peter using the labs
<hashar> then we will be able to investigate the issue
<hashar> not much we can do right now

Note You need to log in before you can comment on or make changes to this bug.


Navigation
Links