Last modified: 2014-01-03 16:02:38 UTC
This issue was converted from https://jira.toolserver.org/browse/DBQ-140. Summary: Selecting top contributors with 10 or more edits from a list of science articles in the English wikipedia and count all contributor edits for all articles they have edited across Wikipedia. Issue type: Task - A task that needs to be done. Priority: Major Status: Done Assignee: Hoo man <hoo@online.de> ------------------------------------------------------------------------------- From: Jim Hutchinson <jim.hutchinson@weldre4.k12.co.us> Date: Tue, 03 May 2011 20:36:08 ------------------------------------------------------------------------------- This analysis is part of a graduate research project. The data will be used to explore possible patterns in the connections between articles based on how often and in what ways multiple contributors overlap in various articles. The source articles for the analysis are those listed as "Part of a series on Science" listed on http://en.wikipedia.org/wiki/Science and consists of about 200 articles. I can provide a list of all article URLs if necessary or if helpful for clarity. Data from the SQL tables will be read into UCINET and used to weight edits and create an activity map based on articles most frequently edited by overlapping contributors. This will be done for each of the selected science articles and the resulting "maps" will contain the other articles the top contributors contributed to and showing the strength of the relationships.
------------------------------------------------------------------------------- From: Betacommand <phoenixoverride@gmail.com> Date: Tue, 03 May 2011 20:38:56 ------------------------------------------------------------------------------- Can you provide a clean list of all titles?
------------------------------------------------------------------------------- From: Platonides <Platonides@gmail.com> Date: Tue, 03 May 2011 21:53:24 ------------------------------------------------------------------------------- Please specify what you want. (I include my guesses below) For which articles? (NS main articles linked from http://en.wikipedia.org/wiki/Template:Science) What users do you want? (users with more than 10 edits to a single one of those articles) What data for each user? (list of all articles in main namespace they have edited, with number of edits to each one)
------------------------------------------------------------------------------- From: Jim Hutchinson <jim.hutchinson@weldre4.k12.co.us> Date: Tue, 03 May 2011 22:47:37 ------------------------------------------------------------------------------- Pardon my noob-ness as I adjust to this work flow. Thanks for the clarification questions. Hopefully this will help. 1. Start with these articles: Formal sciences <http://en.wikipedia.org/wiki/Formal_sciences> Mathematics <http://en.wikipedia.org/wiki/Mathematics> Mathematical logic <http://en.wikipedia.org/wiki/Mathematical_logic> Computer science <http://en.wikipedia.org/wiki/Theoretical_computer_science> Mathematical statistics <http://en.wikipedia.org/wiki/Mathematical_statistics> Physical sciences <http://en.wikipedia.org/wiki/Physical_science> Physics <http://en.wikipedia.org/wiki/Physics> Applied physics <http://en.wikipedia.org/wiki/Applied_physics> Atomic physics <http://en.wikipedia.org/wiki/Atomic_physics> Computational physics <http://en.wikipedia.org/wiki/Computational_physics> Condensed matter physics <http://en.wikipedia.org/wiki/Condensed_matter_physics> Experimental physics <http://en.wikipedia.org/wiki/Experimental_physics> Mechanics <http://en.wikipedia.org/wiki/Mechanics> Particle physics <http://en.wikipedia.org/wiki/Particle_physics> Plasma physics <http://en.wikipedia.org/wiki/Plasma_%28physics%29> Quantum mechanics <http://en.wikipedia.org/wiki/Quantum_mechanics> Solid mechanics <http://en.wikipedia.org/wiki/Solid_mechanics> Theoretical physics <http://en.wikipedia.org/wiki/Theoretical_physics> Thermodynamics <http://en.wikipedia.org/wiki/Thermodynamics> Entropy <http://en.wikipedia.org/wiki/Introduction_to_entropy> General relativity <http://en.wikipedia.org/wiki/Introduction_to_general_relativity> M-theory <http://en.wikipedia.org/wiki/Introduction_to_M-theory> Special relativity <http://en.wikipedia.org/wiki/Introduction_to_special_relativity> Chemistry <http://en.wikipedia.org/wiki/Chemistry> Acid-base reaction theories <http://en.wikipedia.org/wiki/Acid%E2%80%93base_reaction> Alchemy <http://en.wikipedia.org/wiki/Alchemy> Analytical chemistry <http://en.wikipedia.org/wiki/Analytical_chemistry> Astrochemistry <http://en.wikipedia.org/wiki/Astrochemistry> Biochemistry <http://en.wikipedia.org/wiki/Biochemistry> Crystallography <http://en.wikipedia.org/wiki/Crystallography> Environmental chemistry <http://en.wikipedia.org/wiki/Environmental_chemistry> Food science <http://en.wikipedia.org/wiki/Food_science> Geochemistry <http://en.wikipedia.org/wiki/Geochemistry> Green chemistry <http://en.wikipedia.org/wiki/Green_chemistry> Inorganic chemistry <http://en.wikipedia.org/wiki/Inorganic_chemistry> Materials science <http://en.wikipedia.org/wiki/Materials_science> Molecular physics <http://en.wikipedia.org/wiki/Molecular_physics> Nuclear chemistry <http://en.wikipedia.org/wiki/Nuclear_chemistry> Organic chemistry <http://en.wikipedia.org/wiki/Organic_chemistry> Photochemistry <http://en.wikipedia.org/wiki/Photochemistry> Physical chemistry <http://en.wikipedia.org/wiki/Physical_chemistry> Radiochemistry <http://en.wikipedia.org/wiki/Radiochemistry> Solid-state chemistry <http://en.wikipedia.org/wiki/Solid-state_chemistry> Stereochemistry <http://en.wikipedia.org/wiki/Stereochemistry> Supramolecular chemistry <http://en.wikipedia.org/wiki/Supramolecular_chemistry> Surface science <http://en.wikipedia.org/wiki/Surface_science> Theoretical chemistry <http://en.wikipedia.org/wiki/Theoretical_chemistry> Astronomy <http://en.wikipedia.org/wiki/Astronomy> Astrophysics <http://en.wikipedia.org/wiki/Astrophysics> Cosmology <http://en.wikipedia.org/wiki/Cosmology> Galactic astronomy <http://en.wikipedia.org/wiki/Galactic_astronomy> Planetary geology <http://en.wikipedia.org/wiki/Planetary_geology> Planetary science <http://en.wikipedia.org/wiki/Planetary_science> Stellar astronomy <http://en.wikipedia.org/wiki/Star> Earth sciences <http://en.wikipedia.org/wiki/Earth_science> Atmospheric sciences <http://en.wikipedia.org/wiki/Atmospheric_sciences> Ecology <http://en.wikipedia.org/wiki/Ecology> Environmental science <http://en.wikipedia.org/wiki/Environmental_science> Geodesy <http://en.wikipedia.org/wiki/Geodesy> Geology <http://en.wikipedia.org/wiki/Geology> Geomorphology <http://en.wikipedia.org/wiki/Geomorphology> Geophysics <http://en.wikipedia.org/wiki/Geophysics> Glaciology <http://en.wikipedia.org/wiki/Glaciology> Hydrology <http://en.wikipedia.org/wiki/Hydrology> Limnology <http://en.wikipedia.org/wiki/Limnology> Mineralogy <http://en.wikipedia.org/wiki/Mineralogy> Oceanography <http://en.wikipedia.org/wiki/Oceanography> Paleoclimatology <http://en.wikipedia.org/wiki/Paleoclimatology> Palynology <http://en.wikipedia.org/wiki/Palynology> Physical geography <http://en.wikipedia.org/wiki/Physical_geography> Soil science <http://en.wikipedia.org/wiki/Soil_science> Space science <http://en.wikipedia.org/wiki/Space_science> Life sciences <http://en.wikipedia.org/wiki/Life_sciences> Biology <http://en.wikipedia.org/wiki/Biology> Anatomy <http://en.wikipedia.org/wiki/Anatomy> Astrobiology <http://en.wikipedia.org/wiki/Astrobiology> Biochemistry <http://en.wikipedia.org/wiki/Biochemistry> Biogeography <http://en.wikipedia.org/wiki/Biogeography> Biological engineering <http://en.wikipedia.org/wiki/Biological_engineering> Biophysics <http://en.wikipedia.org/wiki/Biophysics> Behavioral neuroscience <http://en.wikipedia.org/wiki/Behavioral_neuroscience> Biotechnology <http://en.wikipedia.org/wiki/Biotechnology> Botany <http://en.wikipedia.org/wiki/Botany> Cell biology <http://en.wikipedia.org/wiki/Cell_biology> Conservation biology <http://en.wikipedia.org/wiki/Conservation_biology> Cryobiology <http://en.wikipedia.org/wiki/Cryobiology> Developmental biology <http://en.wikipedia.org/wiki/Developmental_biology> Ecology <http://en.wikipedia.org/wiki/Ecology> Ethnobiology <http://en.wikipedia.org/wiki/Ethnobiology> Evolutionary biology <http://en.wikipedia.org/wiki/Evolutionary_biology> Genetics <http://en.wikipedia.org/wiki/Genetics> Gerontology <http://en.wikipedia.org/wiki/Gerontology> Immunology <http://en.wikipedia.org/wiki/Immunology> Limnology <http://en.wikipedia.org/wiki/Limnology> Marine biology <http://en.wikipedia.org/wiki/Marine_biology> Microbiology <http://en.wikipedia.org/wiki/Microbiology> Molecular biology <http://en.wikipedia.org/wiki/Molecular_biology> Neuroscience <http://en.wikipedia.org/wiki/Neuroscience> Paleontology <http://en.wikipedia.org/wiki/Paleontology> Parasitology <http://en.wikipedia.org/wiki/Parasitology> Physiology <http://en.wikipedia.org/wiki/Physiology> Radiobiology <http://en.wikipedia.org/wiki/Radiobiology> Soil biology <http://en.wikipedia.org/wiki/Soil_biology> Systematics <http://en.wikipedia.org/wiki/Systematics> Theoretical biology <http://en.wikipedia.org/wiki/Theoretical_biology> Toxicology <http://en.wikipedia.org/wiki/Toxicology> Zoology <http://en.wikipedia.org/wiki/Zoology> Social <http://en.wikipedia.org/wiki/Social_sciences> Behavioral sciences <http://en.wikipedia.org/wiki/Behavioural_sciences> Anthropology <http://en.wikipedia.org/wiki/Anthropology> Archaeology <http://en.wikipedia.org/wiki/Archaeology> Criminology <http://en.wikipedia.org/wiki/Criminology> Demography <http://en.wikipedia.org/wiki/Demography> Economics <http://en.wikipedia.org/wiki/Economics> Geography <http://en.wikipedia.org/wiki/Geography> History <http://en.wikipedia.org/wiki/History> Linguistics <http://en.wikipedia.org/wiki/Linguistics> Political science <http://en.wikipedia.org/wiki/Political_science> Psychology <http://en.wikipedia.org/wiki/Psychology> Sociology <http://en.wikipedia.org/wiki/Sociology> Applied sciences <http://en.wikipedia.org/wiki/Applied_science> Engineering <http://en.wikipedia.org/wiki/Engineering> Agricultural <http://en.wikipedia.org/wiki/Agricultural_engineering> Aerospace <http://en.wikipedia.org/wiki/Aerospace_engineering> Biomedical <http://en.wikipedia.org/wiki/Biomedical_engineering> Chemical <http://en.wikipedia.org/wiki/Chemical_engineering> Civil <http://en.wikipedia.org/wiki/Civil_engineering> Computer <http://en.wikipedia.org/wiki/Computer_engineering> Electrical <http://en.wikipedia.org/wiki/Electrical_engineering> Fire protection <http://en.wikipedia.org/wiki/Fire_protection_engineering> Genetic <http://en.wikipedia.org/wiki/Genetic_engineering> Industrial <http://en.wikipedia.org/wiki/Industrial_engineering> Mechanical <http://en.wikipedia.org/wiki/Mechanical_engineering> Military <http://en.wikipedia.org/wiki/Military_engineer> Mining <http://en.wikipedia.org/wiki/Mining_engineering> Nuclear <http://en.wikipedia.org/wiki/Nuclear_engineering> Operations research <http://en.wikipedia.org/wiki/Operations_research> Software <http://en.wikipedia.org/wiki/Software_engineering> Health sciences <http://en.wikipedia.org/wiki/Health_science> Biological engineering <http://en.wikipedia.org/wiki/Biological_engineering> Dentistry <http://en.wikipedia.org/wiki/Dentistry> Epidemiology <http://en.wikipedia.org/wiki/Epidemiology> Health care <http://en.wikipedia.org/wiki/Health_care> Medicine <http://en.wikipedia.org/wiki/Medicine> Nursing <http://en.wikipedia.org/wiki/Nursing> Pharmacy <http://en.wikipedia.org/wiki/Pharmacy> Social work <http://en.wikipedia.org/wiki/Social_work> Veterinary medicine <http://en.wikipedia.org/wiki/Veterinary_medicine> Interdisciplinarity <http://en.wikipedia.org/wiki/Interdisciplinarity> Applied physics <http://en.wikipedia.org/wiki/Applied_physics> Artificial intelligence <http://en.wikipedia.org/wiki/Artificial_intelligence> Bioethics <http://en.wikipedia.org/wiki/Bioethics> Bioinformatics <http://en.wikipedia.org/wiki/Bioinformatics> Biogeography <http://en.wikipedia.org/wiki/Biogeography> Biomedical engineering <http://en.wikipedia.org/wiki/Biomedical_engineering> Biostatistics <http://en.wikipedia.org/wiki/Biostatistics> Cognitive science <http://en.wikipedia.org/wiki/Cognitive_science> Computational linguistics <http://en.wikipedia.org/wiki/Computational_linguistics> Cultural studies <http://en.wikipedia.org/wiki/Cultural_studies> Cybernetics <http://en.wikipedia.org/wiki/Cybernetics> Environmental studies <http://en.wikipedia.org/wiki/Environmental_studies> Ethnic studies <http://en.wikipedia.org/wiki/Ethnic_studies> Evolutionary psychology <http://en.wikipedia.org/wiki/Evolutionary_psychology> Forestry <http://en.wikipedia.org/wiki/Forestry> Health <http://en.wikipedia.org/wiki/Health> Library science <http://en.wikipedia.org/wiki/Library_science> Logic <http://en.wikipedia.org/wiki/Logic> Mathematical biology <http://en.wikipedia.org/wiki/Mathematical_biology> Mathematical physics <http://en.wikipedia.org/wiki/Mathematical_physics> Scientific modelling <http://en.wikipedia.org/wiki/Scientific_modelling> Neural engineering <http://en.wikipedia.org/wiki/Neural_engineering> Neuroscience <http://en.wikipedia.org/wiki/Neuroscience> Political economy <http://en.wikipedia.org/wiki/Political_economy> Science and technology studies <http://en.wikipedia.org/wiki/Science_and_technology_studies> Science studies <http://en.wikipedia.org/wiki/Science_studies> Semiotics <http://en.wikipedia.org/wiki/Semiotics> Sociobiology <http://en.wikipedia.org/wiki/Sociobiology> Systems theory <http://en.wikipedia.org/wiki/Systems_theory> Transdisciplinarity <http://en.wikipedia.org/wiki/Transdisciplinarity> Urban planning <http://en.wikipedia.org/wiki/Urban_planning> Scientific method <http://en.wikipedia.org/wiki/Scientific_method> History of science <http://en.wikipedia.org/wiki/History_of_science> Philosophy of science <http://en.wikipedia.org/wiki/Philosophy_of_science> Science policy <http://en.wikipedia.org/wiki/Science_policy> Humanities <http://en.wikipedia.org/wiki/Humanities> Fringe science <http://en.wikipedia.org/wiki/Fringe_science> Pseudoscience <http://en.wikipedia.org/wiki/Pseudoscience> 2. Identify the contributors of each article with 10 or more edits (contributors tool in "article history" on each page will show this. For example, http://toolserver.org/~daniel/WikiSense/Contributors.php?wikilang=en&wikifam=.wikipedia.org&grouped=on&page=Science 3. For each identified contributor in step 2, list all articles they have edited with an edit count for each. For example, http://en.wikipedia.org/wiki/Special:Contributions/Vsmith lists all their contributions. These articles would need to be identified and then a sum of their edit counts for each article. A very basic example for users A, B, C, and D is at http://goo.gl/VIWd6 4. Repeat for each article listed in step 1. Let me know if there is anything else. Thanks again.
------------------------------------------------------------------------------- From: Betacommand <phoenixoverride@gmail.com> Date: Tue, 03 May 2011 23:48:41 ------------------------------------------------------------------------------- Running
------------------------------------------------------------------------------- From: Betacommand <phoenixoverride@gmail.com> Date: Wed, 04 May 2011 03:24:13 ------------------------------------------------------------------------------- see http://toolserver.org/~betacommand/articleinfo.zip and http://toolserver.org/~betacommand/userinfo.zip
------------------------------------------------------------------------------- From: Jim Hutchinson <jim.hutchinson@weldre4.k12.co.us> Date: Wed, 04 May 2011 04:38:27 ------------------------------------------------------------------------------- Wow. That was fast. Thank you very much. It will take me a while to work through this, but a quick question based on a quick look through the data. The userinfo data seems to include all article edits for each user which is precisely what I was looking for. However, I do need to differentiate the activities of each contributor based on their activities in one of the original science articles (i.e. their appearance as a contributor in a particular science article). Is there a way to reconstruct just the contributors to each individual article, such as those contributing to quantum mechanics or astrophysics (obviously some will appear in multiple articles as well) in order to do a per article analysis? Perhaps that is the data included in the articleinfo files. Thanks again. This is greatly appreciate.
------------------------------------------------------------------------------- From: Jim Hutchinson <jim.hutchinson@weldre4.k12.co.us> Date: Wed, 04 May 2011 04:58:34 ------------------------------------------------------------------------------- One more question for clarification. Was the query run on the live version of Wikipedia or a mirror copy updated on some schedule? It looks like the query may have taken several hours. If this was run on the live Wikipedia then it's possible that some changes in edit counts could have occurred if a contributor made any edits during the time the query was running. Given the relatively short time it's unlikely there would have been any huge changes, but I will need to explain the details in my final writeup. Thanks.
------------------------------------------------------------------------------- From: Manish Goregaokar <manishsmail@gmail.com> Date: Wed, 04 May 2011 14:09:39 ------------------------------------------------------------------------------- Fix priority..
------------------------------------------------------------------------------- From: Betacommand <phoenixoverride@gmail.com> Date: Wed, 04 May 2011 14:22:49 ------------------------------------------------------------------------------- The queries ran for less than an hour on the toolserver replicated copy of the database. In the articleinfo.zip there is a file that lists all of the page ids/article name pairs. each of the text files with a number refers to the page id of the relevant article, and lists all users who have made 10 or more edits to that given article.
------------------------------------------------------------------------------- From: Jim Hutchinson <jim.hutchinson@weldre4.k12.co.us> Date: Wed, 04 May 2011 14:45:28 ------------------------------------------------------------------------------- Thanks again. I see that file now. Looks great. Sorry for all the questions, but one more. In the articleinfo files there are a dozen or so with no data. For example, one of them is 46771 which is the article "Agricultural" which actually maps to http://en.wikipedia.org/wiki/Agricultural_engineering and looks like it would have returned 2 users with more than 10 edits. I think I can probably just exclude these articles from the final analysis, but I'm just wondering what might explain it as I will probably have to explain why they are excluded. On a separate note, I would like to include a "special thanks" section in the final write up. Clearly I wouldn't be able to continue without the help of people here and Betacommand in particular. Please let me know if it would be okay to include your screen name and/or real name if you wish to share. There is probably a way to send a private message or email in response if anyone wishes to avoid posting personal information publicly. I will not include anything unless I receive explicit permission. Thanks again.
------------------------------------------------------------------------------- From: Betacommand <phoenixoverride@gmail.com> Date: Wed, 04 May 2011 15:15:03 ------------------------------------------------------------------------------- I just used what was listed before the <URL> so you listed Agricultural<URL> and I just stripped out the <URL> in order to get the page title. That is probably the cause for that issue. You can just use my screen name is fine.
------------------------------------------------------------------------------- From: Jim Hutchinson <jim.hutchinson@weldre4.k12.co.us> Date: Wed, 11 May 2011 04:14:00 ------------------------------------------------------------------------------- Thank you once again for the help on this. I've had a chance to work with the data a bit and it looks like the easiest way to run the stats is to actually use an SQL database and read that into the software. Basically, it would work best if I had the SQL dump of the query that you ran with the relationships of the primary and foreign keys intact as this would make the process a whole lot easier. Is there any chance you still have this? Thanks again.
This bug was imported as RESOLVED. The original assignee has therefore not been set, and the original reporters/responders have not been added as CC, to prevent bugspam. If you re-open this bug, please consider adding these people to the CC list: Original assignee: hoo@online.de CC list: manishsmail@gmail.com, hoo@online.de, Platonides@gmail.com, phoenixoverride@gmail.com
