Last modified: 2013-03-29 20:00:09 UTC
Basically continue the work discussed in https://gerrit.wikimedia.org/r/#/c/26711/r.
Gerrit messed up the url after the merge.. https://gerrit.wikimedia.org/r/#/c/26711/
Not sure if we'll still get to this in the JS implementation, but still something worth keeping in mind.
We are now handling noinclude etc in the tokenizer, and don't use the TokenAndAttributeCollector any more.