Last modified: 2012-11-07 08:10:35 UTC
If a page "Origin" contains an ask query 1 with format=template, template=Nested1, and Nested1 contains an ask query 2 with template=Nested2, and Nested2 contains an ask query 3 with template=Nested3 (which contains no further ask-query, but simple display), the query fails. We tested, that both the sequence: Origin-with-ask-query 1 + Nested1-with-ask-query 2 + Nested2-with-ask-query 3 (with format changed to table instead of template, making Nested3 unused) and Nested1-with-ask-query 2 + Nested2-with-ask-query 3 + Nested3-simple-rendering works fine. This seems to indicate that it is not the number of ask-queries (which is the same in both cases), but the depth of nested ask-template-formats is limiting. The described depth of nesting is necessary if a result shall be presented with two levels of subheadings (grouped by factor 1, then grouped by factor 2). I have not found this limitation documented and believe it is unexpected low. Some limitation of levels of subheadings may be reasonable, allowing only one level of subheadings in a report seems over-constrained and I wonder whether this is intentional or a bug.
This is an omission in the documentation. The template nesting depth can be controlled with the parameter SMWResultPrinter::$maxRecursionDepth (in LocalSetting.php; to be set only after enableSemantics() was called; else the class won't be known). The default of 2 is chosen since the total length of pages grows exponentially in this number. If the wikitext is too large, PHP will run out of memory, the page will appear blank and be hard to edit to correct the problem. This should be avoided by default. If you know what you are doing, feel free to increase the number to a higher value. It would be good to document this setting under http://semantic-mediawiki.org/wiki/Help:Configuration, and maybe also have pointers from the affected formats template and embedded.
Thanks! With this information I now found: http://semantic-mediawiki.org/wiki/Help_talk:Semantic_templates/archive but it is not documented in the main docu, as Markus confirms above.
As an afterthought: I wonder whether SMWResultPrinter::$maxRecursionDepth could default to a high value in the future, provided $smwgQMaxLimit = 10000 would be enforced across the sum of all nested template queries? I our case, grouping the results by subheading, the maxRecursionDepth does not add at all to the number of results or length of pages. While this is certainly different in other cases, it illustrates that maxRecursionDepth does not necessarily address the cause of the problem.
(In reply to comment #3) > As an afterthought: I wonder whether SMWResultPrinter::$maxRecursionDepth could > default to a high value in the future, provided $smwgQMaxLimit = 10000 would > be enforced across the sum of all nested template queries? There are several problems that prevent this from working, the main one being that we cannot count the total number of results. In fact, even a few results can be shown many times with recursive embedding. The number of query results is not added up or multiplied by higher depths, it is exponentiated. If a query has 10 results and recursively embeds it's own page for rendering, you get 100 results at level 2, 1000 at level three, and so on. And the query could easily have 100 results instead of 10 to start with. This cannot be limited at the storage level because the query is only answered once (with 10 results); the rest is rendering and, more specifically, recursive template transclusion in MW. The problem are not (just) the 10000 result rows, but the 1000 template inclusions that need to processed to get them.
(In reply to comment #4) > is not added up or multiplied by higher depths, it is exponentiated. If a query > has 10 results and recursively embeds it's own page for rendering, you get 100 > results at level 2, 1000 at level three, and so on. And the query could easily > have 100 results instead of 10 to start with. This cannot be limited at the > storage level because the query is only answered once (with 10 results); the > rest is rendering and, more specifically, recursive template transclusion in > MW. The problem are not (just) the 10000 result rows, but the 1000 template > inclusions that need to processed to get them. I agree but I believe these are separate issues. Recursive template transclusion is independent of ask queries and exists in non-SMW installations as well. Mediawiki carries a cost-based limitation mechanism that prevents this from creating a DoS situation. Is it perhaps possible to hook into this mechanism, feeding it with appropriate cost estimates? Is this perhaps the missing piece? With respect to the queries alone, I think we agree that it is possible to manage a cumulative result count in a recursion. I personally believe it is not necessary to have full prediction, it is acceptable to limit each query to a given result count and only stop once the template/ask recursion exceeds the cumulative maximum. That is with single = 500 and total = 1000, it would be possible to generation max 1499 result (if the penultimate ask was 999 when the last query was started.
(In reply to comment #5) ... > > With respect to the queries alone, I think we agree that it is possible to > manage a cumulative result count in a recursion. I personally believe it is not > necessary to have full prediction, it is acceptable to limit each query to a > given result count and only stop once the template/ask recursion exceeds the > cumulative maximum. That is with single = 500 and total = 1000, it would be > possible to generation max 1499 result (if the penultimate ask was 999 when the > last query was started. I am not sure how this could be done. The recursive subqueries are executed as part of a normal page parsing process; SMW does not see that it runs as a subquery there. It might even be that the embedded subpage was parsed eariler and the result cached; so when the parsing happened, there was no information about the current number of results and the remaining resources that could be used. Likewise, we have no mechanism to find out how many query results are displayed on a page in total, so we cannot easily access this data to decide whether to transclude a page or not. It might be doable, but it is not straightforward.
In respect to the reported problem (and not its inherent discussion), documentation is provided [1] therefore this bug is resolved. [1] http://www.semantic-mediawiki.org/wiki/Help:$maxRecursionDepth
It should be mentioned and linked on: http://www.semantic-mediawiki.org/wiki/Help:Template_format I tried, but user G.Hagedorn cannot edit there.
Thanks Gregor. Your user account should be able to edit the page now.
