Last modified: 2014-11-20 19:08:58 UTC
Various bot commands end with the message: WARNING: API warning (categorymembers): Too many values supplied for parameter 'gcmnamespace': the limit is 50 Trying to find the reason in the code, grep does not find various pieces of that message. Thus I was not able to locate its origin in the code. :-(
I guess the method is site.categorymembers which launches the request. With Namespace Parameter the Maximum values to be returned is limited to 500 for bots and 50 for others. Are you running a bot or without bot flag when that error occurs. Could you give an example to reproduce that bug?
Seems this is a warning only due to $wgMiserMode Parameter and does not affect the results.
Change 136632 had a related patch set uploaded by Xqt: (bug 65981) reduce limit if $wgMiserMode is set and generator is affected https://gerrit.wikimedia.org/r/136632
(In reply to xqt from comment #1) > I guess the method is site.categorymembers which launches the request. With > Namespace Parameter the Maximum values to be returned is limited to 500 for > bots and 50 for others. Are you running a bot or without bot flag when that > error occurs. Could you give an example to reproduce that bug? Yes, running a test on categories with -simulate and without bot flag. The wiki is at http://wiki.piratenpartei.de/wiki/index.php and not available in git. It is fairly standard but has tons of local name spaces. In my test, I do not refer to name spaces explicitly, but of course, category is one of them.
(In reply to xqt from comment #2) > Seems this is a warning only due to $wgMiserMode Parameter and does not > affect the results. The bot terminates after the warning. While I guess it might generally be able to continue, it does nothing and at least not what it is expected to do.
(In reply to Gerrit Notification Bot from comment #3) > Change 136632 had a related patch set uploaded by Xqt: > (bug 65981) reduce limit if $wgMiserMode is set and generator is affected > > https://gerrit.wikimedia.org/r/136632 I tried the patch. It seems not to solve the problem in one instance, but creates more output: ... max WARNING: API warning (categorymembers): Too many values supplied for parameter 'gcmnamespace': the limit is 50 max max Dropped throttle(s). ... The "max" are new. In another test, the patch seems to avoid the error - maybe because another error is hit earlier: - - - cut - - - $ python pwb.py category -family:piratewiki -lang:de move -nodelete -inplace -from:ababa -to:abba -simulate -v === Pywikibot framework v2.0 -- Logging header === COMMAND: ['category', '-family:piratewiki', '-lang:de', 'move', '-nodelete', '-inplace', '-from:ababa', '-to:abba', '-simulate', '-v'] VERSION: ('[ssh] pywikibot-core.git (3615274, g3256, 2014/06/01, 13:13:55, OUTDATED)', 'b807b4f704dbfd042b2c1ae0b362a5b04093750e', (1, 22, u'.7')) SYSTEM: ('Linux', 'tools-login', '3.2.0-59-virtual', '#90-Ubuntu SMP Tue Jan 7 23:02:33 UTC 2014', 'x86_64') MODULES: pywikibot/comms/http.py affc571 2014-06-01 00:29:06.175927 pywikibot/data/api.py 1a1ae93 2014-06-01 12:39:25.389090 pywikibot/textlib.py 43dc65b 2014-06-01 00:29:06.535937 pywikibot/i18n.py a7dd59b 2014-06-01 00:29:06.439934 pywikibot/comms/threadedhttp.py 5dd0840 2014-06-01 00:29:06.187927 pywikibot/date.py a1b85ad 2014-02-21 14:38:22 pywikibot/exceptions.py d9e869a 2014-06-01 00:29:06.291930 pywikibot/site.py 3d3f2cf 2014-06-01 00:29:06.515936 pywikibot/bot.py 2e7e859 2014-06-01 00:29:06.159927 pywikibot/throttle.py 1d8da7a 2014-06-01 00:29:06.543937 pywikibot/page.py 01efdf2 2014-06-01 00:29:06.475935 pywikibot/family.py 75ed448 2014-06-01 00:29:06.411933 pywikibot/plural.py 2b02046 2014-02-21 14:38:22 pywikibot/version.py 6d0d30d 2014-06-01 00:29:06.615939 pywikibot/userinterfaces/terminal_interface.py 0e8db29 2014-02-21 14:38:23 pywikibot/userinterfaces/terminal_interface_base.py 745aa26 2014-06-01 00:29:06.579938 pywikibot/config2.py 18b9ca6 2014-06-01 00:29:06.231929 pywikibot/pagegenerators.py 87472d0 2014-06-01 00:29:06.491935 pywikibot/userinterfaces/terminal_interface_unix.py 30e64ec 2014-06-01 00:29:06.591938 pywikibot/tools.py 31ca0ef 2014-06-01 00:29:06.567937 pywikibot/login.py 8f167ac 2014-06-01 00:29:06.455935 pywikibot/userinterfaces/transliteration.py 288348d 2014-06-01 00:29:06.607939 MESSAGES: unknown (not logged in) === === === === === === === === === === === === === === Pywikibot ra774f3344f63c8a549779793c192b517e0866527 Python 2.7.3 (default, Feb 27 2014, 19:58:35) [GCC 4.6.3] Found 1 piratewiki:de processes running, including this one. Password for user Purbo_T on piratewiki:de (no characters will be shown): Logging in to piratewiki:de as Purbo_T Should be logged in now WARNING: deleteEmptySourceCat argument of __init__ is deprecated; use delete_oldcat instead. WARNING: titleRegex argument of __init__ is deprecated; use title_regex instead. WARNING: withHistory argument of __init__ is deprecated; use history instead. Moving text from Kategorie:Ababa to Kategorie:Abba. ERROR: Fatal error: Traceback (most recent call last): File "scripts/category.py", line 1089, in <module> main() File "scripts/category.py", line 1059, in main bot.run() File "scripts/category.py", line 473, in run self._movecat() File "scripts/category.py", line 517, in _movecat authors = ', '.join(self.oldcat.contributingUsers()) File "/home/purodha/pywikibot/pywikibot/page.py", line 1297, in contributingUsers edits = self.getVersionHistory(step=step, total=total) File "/home/purodha/pywikibot/pywikibot/__init__.py", line 418, in wrapper return method(*__args, **__kw) File "/home/purodha/pywikibot/pywikibot/__init__.py", line 418, in wrapper return method(*__args, **__kw) File "/home/purodha/pywikibot/pywikibot/__init__.py", line 418, in wrapper return method(*__args, **__kw) File "/home/purodha/pywikibot/pywikibot/page.py", line 1244, in getVersionHistory step=step, total=total) File "/home/purodha/pywikibot/pywikibot/site.py", line 1935, in loadrevisions raise NoPage(page) NoPageDropped throttle(s). Waiting for 1 network thread(s) to finish. Press ctrl-c to abort All threads finished. max $ - - - cut - - - But here: wiki.piratenpartei.de/Spezial:Kategorien/Ababa you see that the category has one page, so here is something wrong.
Could not reproduce it one wikimedia site. Unfortunately I haven't an account at piratewiki. Anyway: could you give me more results starting of the beginning of your command line. I'll commit a new patch set with hardcoded Limit of 50. Could you try it.
Just a quick look at the patch showed me that this patch won't solve the problem. I hope I'm wrong. I will try to install this on a local wiki and try to test it if it works
Purodha Blissenbach, do you still see the problem you mentioned on 2014-06-01 13:02:28 UTC . That will be masking this bug, and should be raised as a separate bug if it is still occurring. I believe the root of the problem is that category.py uses moves from the old category 1) pages using pagegenerators.CategorizedPageGenerator, and then 2) subcats using pagegenerators.SubCategoriesPageGenerator Each of those filters on namespace. It should do both using the same generator. I believe this is what Amir Ladsgroup was saying in the previous comment. The alternative is to not use cmnamespace, but use cmtype
Tried again with the most recent version of 2014-09-06 00:00 UTC and still not working: $ python pwb.py category -family:piratewiki -lang:de move -nodelete -inplace -from:ababa -to:abba -simulate -v === Pywikibot framework v2.0 -- Logging header === COMMAND: ['category', '-family:piratewiki', '-lang:de', 'move', '-nodelete', '-inplace', '-from:ababa', '-to:abba', '-simulate', '-v'] Found 1 piratewiki:de processes running, including this one. VERSION: ('[ssh] pywikibot-core.git (3615274, g3256, 2014/06/01, 13:13:55, OUTDATED)', 'd3414c6abffdf7dcd0cb3d2bed35430030110a40', (1, 22, u'.9')) SYSTEM: ('Linux', 'tools-login', '3.2.0-59-virtual', '#90-Ubuntu SMP Tue Jan 7 23:02:33 UTC 2014', 'x86_64') MODULES: pywikibot/comms/http.py affc571 2014-06-01 00:29:06.175927 pywikibot/data/api.py 1a1ae93 2014-06-01 12:39:25.389090 pywikibot/textlib.py 43dc65b 2014-06-01 00:29:06.535937 pywikibot/i18n.py a7dd59b 2014-06-01 00:29:06.439934 pywikibot/comms/threadedhttp.py 5dd0840 2014-06-01 00:29:06.187927 pywikibot/date.py a1b85ad 2014-02-21 14:38:22 pywikibot/exceptions.py d9e869a 2014-06-01 00:29:06.291930 pywikibot/site.py 3d3f2cf 2014-06-01 00:29:06.515936 pywikibot/bot.py 2e7e859 2014-06-01 00:29:06.159927 pywikibot/throttle.py 1d8da7a 2014-06-01 00:29:06.543937 pywikibot/page.py 01efdf2 2014-06-01 00:29:06.475935 pywikibot/family.py 75ed448 2014-06-01 00:29:06.411933 pywikibot/plural.py 2b02046 2014-02-21 14:38:22 pywikibot/version.py 6d0d30d 2014-06-01 00:29:06.615939 pywikibot/userinterfaces/terminal_interface.py 0e8db29 2014-02-21 14:38:23 pywikibot/userinterfaces/terminal_interface_base.py 745aa26 2014-06-01 00:29:06.579938 pywikibot/config2.py 18b9ca6 2014-06-01 00:29:06.231929 pywikibot/pagegenerators.py 87472d0 2014-06-01 00:29:06.491935 pywikibot/userinterfaces/terminal_interface_unix.py 30e64ec 2014-06-01 00:29:06.591938 pywikibot/tools.py 31ca0ef 2014-06-01 00:29:06.567937 pywikibot/login.py 8f167ac 2014-06-01 00:29:06.455935 pywikibot/userinterfaces/transliteration.py 288348d 2014-06-01 00:29:06.607939 MESSAGES: unknown (not logged in) === === === === === === === === === === === === === === Pywikibot ra774f3344f63c8a549779793c192b517e0866527 Python 2.7.3 (default, Feb 27 2014, 19:58:35) [GCC 4.6.3] Password for user Purbo_T on piratewiki:de (no characters will be shown): Logging in to piratewiki:de as Purbo_T Should be logged in now WARNING: deleteEmptySourceCat argument of __init__ is deprecated; use delete_oldcat instead. WARNING: titleRegex argument of __init__ is deprecated; use title_regex instead. WARNING: withHistory argument of __init__ is deprecated; use history instead. Moving text from Kategorie:Ababa to Kategorie:Abba. max ERROR: Fatal error: Traceback (most recent call last): File "scripts/category.py", line 1089, in <module> main() File "scripts/category.py", line 1059, in main bot.run() File "scripts/category.py", line 473, in run self._movecat() File "scripts/category.py", line 517, in _movecat authors = ', '.join(self.oldcat.contributingUsers()) File "/home/purodha/pywikibot/pywikibot/page.py", line 1297, in contributingUsers edits = self.getVersionHistory(step=step, total=total) File "/home/purodha/pywikibot/pywikibot/__init__.py", line 418, in wrapper return method(*__args, **__kw) File "/home/purodha/pywikibot/pywikibot/__init__.py", line 418, in wrapper return method(*__args, **__kw) File "/home/purodha/pywikibot/pywikibot/__init__.py", line 418, in wrapper return method(*__args, **__kw) File "/home/purodha/pywikibot/pywikibot/page.py", line 1244, in getVersionHistory step=step, total=total) File "/home/purodha/pywikibot/pywikibot/site.py", line 1935, in loadrevisions raise NoPage(page) NoPageDropped throttle(s). Waiting for 1 network thread(s) to finish. Press ctrl-c to abort All threads finished. $
Change 158861 had a related patch set uploaded by XZise: [FIX] Don't use wikibase and move pages only if valid https://gerrit.wikimedia.org/r/158861
Change 158861 merged by jenkins-bot: [FIX] Don't use wikibase and move pages only if valid https://gerrit.wikimedia.org/r/158861
https://gerrit.wikimedia.org/r/#/c/136632/ mentioned here has some -1's.
Is this bug still alive?
Still reproducible. $ python pwb.py category -family:piratewiki -lang:de move -nodelete -inplace -from:ababa -to:abba -simulate -v === Pywikibot framework v2.0 -- Logging header === COMMAND: ['category', '-family:piratewiki', '-lang:de', 'move', '-nodelete', '-inplace', '-from:ababa', '-to:abba', '-simulate', '-v'] DATE: 2014-11-20 16:13:09.409555 UTC VERSION: [ssh] pywikibot-core.git (60b788c, g4605, 2014/11/20, 22:48:28, OUTDATED) SYSTEM: ('Linux', '...', '3.17.2-200.fc20.x86_64', '#1 SMP Tue Nov 4 18:04:56 UTC 2014', 'x86_64') CONFIG FILE DIR: .. PACKAGES: _struct (/usr/lib64/python2.7/lib-dynload/_struct.so) = 0.2 backports (/usr/lib/python2.7/site-packages/backports/) = ?? cPickle (/usr/lib64/python2.7/lib-dynload/cPickle.so) = 1.71 ca_certs_locater (/usr/lib/python2.7/site-packages/ca_certs_locater/) = ?? distutils (/usr/lib64/python2.7/distutils/) = 2.7.5 email (/usr/lib64/python2.7/email/) = 4.0.3 google ([path unknown]) = ?? httplib2 (...) = 0.9 jaraco ([path unknown]) = ?? json (/usr/lib64/python2.7/json/) = 2.0.9 logging (/usr/lib64/python2.7/logging/) = 0.5.1.2 logilab (/usr/lib/python2.7/site-packages/logilab/) = ?? mpl_toolkits ([path unknown]) = ?? mwparserfromhell (/usr/lib/python2.7/site-packages/mwparserfromhell-0.3.3-py2.7-linux-x86_64.egg/mwparserfromhell/) = 0.3.3 parser (/usr/lib64/python2.7/lib-dynload/parsermodule.so) = 0.5 paste ([path unknown]) = ?? peak ([path unknown]) = ?? pickle (/usr/lib64/python2.7/pickle.pyc) = $Revision: 72223 $ pkg_resources (/usr/lib/python2.7/site-packages/pkg_resources.pyc) = ?? pywikibot ([path unknown]) = ?? re (/usr/lib64/python2.7/re.pyc) = 2.2.1 sphinxcontrib (/usr/lib/python2.7/site-packages/sphinxcontrib_napoleon-0.2.9-py2.7.egg/sphinxcontrib/) = ?? urllib (/usr/lib64/python2.7/urllib.pyc) = 1.17 urllib2 (/usr/lib64/python2.7/urllib2.pyc) = 2.7 zlib (/usr/lib64/python2.7/lib-dynload/zlibmodule.so) = 1.0 zope ([path unknown]) = ?? MODULES: pywikibot/comms/http.py f3e93c8 2014-11-20 05:02:35.475199 pywikibot/data/api.py 8aa97b4 2014-11-20 05:02:35.488199 pywikibot/textlib.py 33003b4 2014-11-17 12:49:31.747458 pywikibot/i18n.py 23f218e 2014-11-17 18:36:28.172387 pywikibot/comms/threadedhttp.py 32e55bd 2014-11-17 12:49:31.664459 pywikibot/exceptions.py e711f48 2014-11-20 07:29:58.869259 pywikibot/site.py 8f260e8 2014-11-20 22:48:03.183176 pywikibot/bot.py 9fb6254 2014-11-20 22:24:46.860068 pywikibot/throttle.py 5cdc30d 2014-11-17 12:49:31.756458 pywikibot/page.py 2d71d8a 2014-11-20 07:29:58.918259 pywikibot/family.py 9daf228 2014-11-20 05:02:35.501199 pywikibot/plural.py 02a50e4 2014-11-15 21:56:35.098131 pywikibot/version.py e82f0c5 2014-11-17 12:49:31.789458 pywikibot/userinterfaces/terminal_interface.py b0e2743 2014-11-15 21:56:35.114131 pywikibot/date.py 350bdf7 2014-11-17 12:49:31.701459 pywikibot/userinterfaces/terminal_interface_base.py 5f37e69 2014-11-17 12:49:31.770458 pywikibot/config2.py bdb3e50 2014-11-20 07:29:58.850259 pywikibot/pagegenerators.py f2e8360 2014-11-20 05:02:35.529198 pywikibot/userinterfaces/terminal_interface_unix.py bb6384d 2014-11-17 12:49:31.784458 pywikibot/tools.py d71ad8a 2014-11-20 07:29:58.948258 pywikibot/diff.py 7773b7e 2014-11-17 12:49:31.702459 pywikibot/login.py db7be8f 2014-11-20 07:29:58.891259 pywikibot/userinterfaces/transliteration.py 1d8e217 2014-11-15 21:56:35.120131 PYWIKI REPO VERSION: 7d1d1c761b697b568b91e55746e57dfb73e939d3 === === === === === === === === === === === === === === Pywikibot re20d7c4e05f696821260ba612d13e1b672759e0a Python 2.7.5 (default, Nov 3 2014, 14:26:24) [GCC 4.8.3 20140911 (Red Hat 4.8.3-7)] Found 1 piratewiki:de processes running, including this one. Logging in to piratewiki:de as Jayvdb Should be logged in now WARNING: Moving category page 'Kategorie:Ababa' requested, but the page doesn't exist. Moving category talk page 'Kategorie Diskussion:Ababa' requested, but the page doesn't exist. WARNING: API warning (categorymembers): Too many values supplied for parameter 'gcmnamespace': the limit is 50 Retrieving 1 pages from piratewiki:de. Cosmetic changes for piratewiki-de enabled. WARNING: API warning (tokens): Unrecognized values for parameter 'type': deleteglobalaccount, setglobalaccountstatus, centralauth SIMULATION: edit action blocked. Page [[Benutzer:Purodha]] saved without any changes. Page [[Benutzer:Purodha]] saved Dropped throttle(s). Waiting for 1 network thread(s) to finish. Press ctrl-c to abort All threads finished. The wiki has 106 namespaces. http://wiki.piratenpartei.de/wiki/api.php?action=query&meta=siteinfo&siprop=namespaces The provided patch is not related to this problem. page.articles/etc should not be doing the following: namespaces = [x for x in self.site.namespaces() if x >= 0 and x != 14] They should be using cmtype, but we need to check which version cmtype was introduced. https://www.mediawiki.org/wiki/API:Categorymembers