Last modified: 2014-08-29 21:56:52 UTC
Too much Copyright violations via Mobile... See... https://commons.wikimedia.org/wiki/Category:MobileUpload-related_deletion_requests https://commons.wikimedia.org/wiki/Category:Mobile_uploads_lacking_EXIF_data https://commons.wikimedia.org/wiki/Category:Mobile_uploads_lacking_EXIF_data_and_with_multiple_Tineye_matches Mobile Upload related: "1,156 open deletion requests" "4268 closed (as deleted) deletion requests" "And a lot of speedy deletion requests... (See: https://commons.wikimedia.org/wiki/Commons:Mobile_app/deletion_request_tracking/archive (1, 2, ...)" Pleas set up a filter. **** See also wikimedia.mingle.thoughtworks.com/projects/mobile/cards/1314... (reported via IRC) * Add filter for new user uploads (0 uploads) on images with no exif data * Present these with either a nag or with a block and an educational statement.
Prioritization and scheduling of this bug is tracked on Mingle card https://wikimedia.mingle.thoughtworks.com/projects/mobile/cards/1768
Yes, please! If I say 90% of the mobile uploads is copyvio or otherwise out of scope, it would be very little.
@Andre Klapper: Copyright Violations in Wikipedia is LOW PRIO for the WMF? Explanation please. :)
(In reply to Steinsplitter from comment #3) > @Andre Klapper: Copyright Violations in Wikipedia is LOW PRIO for the WMF? > Explanation please. :) The mobile team does not use Bugzilla priority anyway. I'll poke the team about scheduling for this.
(In reply to Bingle from comment #1) > Prioritization and scheduling of this bug is tracked on Mingle card Bugzilla not suitable for tracking mobile-related requests? --------------------------------- Please prevent Mobile uploads that: * Have Tineye search results * Are not photos (check metadata) Thank you. --------------------------------- (In reply to Steinsplitter from comment #3) > Copyright Violations in Wikipedia is LOW PRIO for the WMF? Unfortunately this is true because the mobile team has not yet assigned someone to this bug. Our former bugmeister, Mark A. Hershberger, wrote something about that in his blog post: http://hexmode.com/2011/04/a-philosophy-of-bugs-on-an-open-project/(In reply to
http://hexmode.com/2011/04/a-philosophy-of-bugs-on-an-open-project
(In reply to Rainer Rillke @commons.wikimedia from comment #5) > (In reply to Bingle from comment #1) > > Prioritization and scheduling of this bug is tracked on Mingle card > > Bugzilla not suitable for tracking mobile-related requests? Nobody ever wrote that before? The line above mentioned "Prioritization and scheduling", see comment 1. For some dev teams, assignee field in Bugzilla doesn't necessarily reflect if or who's working on an item as some teams prefer using other planning tools. Or in short: There's not that one homogenous development workflow that fits all teams. Anyway, offtopic here...
Can this be fixed as soon as possible? (Though off-topic images should handled via DRs.)
Max has already sent a mail to the mobile mailing list. http://lists.wikimedia.org/pipermail/mobile-l/2014-March/006637.html
This is a serious problem in Commons. Can it be done as a "priority"?
Folks, every comment creates bugmail. Priority was already requested & Jon already answered, so I don't see much sense in asking the question again...
I am appalled: The PAYED mobile team staff is ignoring the communety... The Commons community is spending every day hours to delete the mobile uploads... Congratulations, Mobile Team!
What‽ Did you read the link in comment 9? The email says "Time to reevaluate". How is that "ignoring"? It's the opposite. Can everybody please calm down a bit? I don't understand the need for drama and overinterpretation of words here. Mobile team obviously does care (as the request wasn't ignored), but it's not "Community say and mobile team has to drop every other task they're working on" either.
(In reply to Andre Klapper from comment #13) > Community say and mobile team has to drop every other task they're working on Indeed. We can simply block mobile uploads or bot-delete them. Sorry, couldn't resist ;-)
Hi this is Kenan. I'm the Product Manager for Mobile. We will be working on this next iteration. Our plan is to provide some abuse filter like functionality along with recommendations for abuse filter settings. Does this sound reasonable? Would anyone here like to be involved in the requirement gathering for this? Thanks!
(In reply to kenan wang from comment #15) Hi Kenan, great to see you commenting here and for the small introduction about you. > Our plan is to provide some abuse filter like functionality along with > recommendations for abuse filter settings. If this would be able to scan the metadata of the file or otherwise is able to protect us from the copyright violations flood, why not? > Would anyone here like to be involved in the requirement gathering for this? Feel free to e-Mail me, if you like. I am also sometimes on IRC. Usually a good place to get feedback by admins quickly is joining #wikimedia-commons and saying "!admin@commons + your request" if no one is responding in time.
(In reply to kenan wang from comment #15) >Would anyone here like to be involved in the > requirement gathering for this? Thanks! Yes :)
Just to follow up from the mobile web team, we are going to be exploring the possibility of filtering photo uploads based on missing EXIF data. We'll be doing an assessment of which devices do not provide EXIF data and see if we can programmatically determine whether or not a phone can provide EXIF data, so we can start figuring out a safe approach to using this as a filter. (https://wikimedia.mingle.thoughtworks.com/projects/mobile/cards/1782) Kenan should also be following up with those of you who expressed interest in helping drive requirements for this stuff soon :)
90% of the phones are using exif. so i see no need. new users (-5 edits), upload data from phone with no exif --> upload forbidden. Autopatroller (etc.) excluded.
now the bug is + one week open. Status report pleas?
Comment 15 is the status. I don't see much use in spamming your favorite bug reports on a regular basis with "Hi Steinsplitter! No news here." comments. In case you refered to specific aspects (e.g. gathering requirements and how to move forward), you might want to explicitly mention those specific aspects in a question.
Andre, this is not my favourite bug, it is a prio bug with every day new upload... and you know: We can simply block mobile uploads or bot-delete them...
*every day new copyright violations uploads on commons. It think such things schould be fixed in weeks but not months... for this reason my questions :) and sorry for flooding you'r mailaccont (cc) :/
(In reply to Steinsplitter from comment #23) > *every day new copyright violations uploads on commons. It think such things > schould be fixed in weeks but not months... for this reason my questions :) > > and sorry for flooding you'r mailaccont (cc) :/ Thanks for prodding, Steinsplitter. Over the course of the next two weeks, we will be investigating a sensible and safe approach to implementing something to help curb the copy vios coming from mobile uploads (see https://wikimedia.mingle.thoughtworks.com/projects/mobile/cards/1782). We recognize the importance of the issue, yet we also have to balance all of our work against other contending priorities. If anyone is interested in pitching in on the engineering side, let us know and we may be able to tackle this even sooner.
Spent a few hours today researching EXIF filtering. First some background... We allow uploading from the following mobile browsers: * iOS Safari 6 and 7 * Android browser 3 and 4 * Blackberry 10 browser * IE Mobile 10 Of those, the vast majority of uploads come from iOS Safari and Android browser. iOS Safari 6 and 7 both strip out all EXIF data from photos except for Orientation, ColorSpace, PixelXDimension, and PixelYDimension. Android browser does not appear to strip any EXIF data. I did a random sampling of copyrighted images on the web. Most of them had no EXIF data whatsoever, but several had ColorSpace, PixelXDimension, and PixelYDimension. There are also a couple apps available for Android (like Image Privacy) that strip all the EXIF data from your photos. Given this information, we could probably block upload of all images without any EXIF data without blocking a large percentage of legitimate uploads. Such a block would not be foolproof, however, and would still allow a large number of copyvios to get uploaded.
One other option would be to have a more stringent EXIF filter for non-iPhone mobile uploads. In other words, require non-iPhone mobile uploads to have more EXIF data such as DateTimeOriginal.
One other option: If the user das - 10/-5 (?) edits and multiple Tineye_matches, block upload. I am not sure if this is possible, becous mobile web needs to query Tineye...
We're also doing analysis as to what call to actions produce more deleted images, that way we can also analyze UI level fixes to this issue. I'm thinking right now that we can first try an EXIF filter where we WARN users instead of just blocking them outright based on EXIF data. i.e. iPhone uploads we look for EXIF Data: Orientation, ColorSpace, PixelXDimension, and PixelYDimension And other photos have a more stringent requirement like Kaldari is proposing. In addition to this we can look at BLOCKING images that both lack EXIF data and have multiple tineye matches. But yes we will need to investigate how feasible using tineye is. I've emailed them about using their API and if we would be able to do that for free given this is a non-commercial enterprise.
https://gerrit.wikimedia.org/r/#/c/127179/
Just a heads up that the an experimental fix for this has been merged into the beta mode of mobile, but it will still take a while until it gets to stable.
still take a while... monts, weeks?
This should be live on Thursday. Let's see if this improves the situation somewhat...
https://commons.wikimedia.org/wiki/Commons:Forum#A_propos_.22mobile_upload.22 The same problems, pleas BLOCK the uploads, a warning is nonsense. If the WMF don't made the necessary changes the tool need to be disabled: The Community is actually planning a RFC to disable this mobile upload for all non-auto patrolled users completely. And i am sure the RFC will be successful.
ca. 84% provable copyvios, 14% unused & unusable selfies. If the devs are ignoring this bug i and my admincolleges will take the necessary steps to block mobile upload onwiki or to batchdelete new mobile uploads.
The numbers Steinsplitter mentioned above (85%, 14%) are my estimates after having patrolled the Mobile/Web uploads for the past weeks closely. The remaining 1% contains some usable images. I didn't keep track of exact counts, though. In any case the number of copyvios and unusable selfies/instagram/twitter images is overwhelming. If in a day's uploads through this channel we get a handful of useful pictures, we're lucky. The very large majority is just copied from the web somewhere. It takes substantial effort to track down these copyvios, tag them, delete them. This is not worth our volunteers' time. I don't think the uploaders act in bad faith. But I do think that they just do not get at all what this feature does. The uploaders have no clue about what they are doing (witness the countless selfies that then remain unused -- in fact, many other uploads through Mobile/Web also remain unused), they have no clue about what the Commons is, no clue about free licenses, no clue about the fact that Wikipedia is supposed to be "free", and no clue that such a thing as copyright even exists. This twitter/instagram/pinterest generation just sees another upload button through which it can send its pixel diarrhea. If the warning that was implemented in response to this bug report is shown at all (I didn't notice any improvement), it's clicked away. I do think this feature needs to be shut off.
Sorry for not replying here in a long time. There was a change of product manager in the mobile team which slowed things down and made us lose focus. Just to let you know, the warning message was deployed to stable mode (that reaches the vast majority of users) on May 15. Is the data that Steinsplitter quotes based on period after that date? I will let the team know that the problem persists.
(In reply to Juliusz Gonera from comment #36) > Is the data that > Steinsplitter quotes based on period after that date? Yes. If it was enabled on the 15th, I had been patrolling that image stream both before and after. I didn't notice any difference at all. We still get about the same number of uploads, and the success quota (useful, non-copyvio uploads) through that Mobile/Web upload is still very, very low. It's truly hideous. I invite you to just monitor that stream yourself for some time: https://commons.wikimedia.org/wiki/Special:RecentChangesLinked/Category:Uploaded_with_Mobile/Web Or see https://commons.wikimedia.org/wiki/Category:MobileUpload-related_deletion_requests or https://commons.wikimedia.org/wiki/Commons:Mobile_app/deletion_request_tracking with all its archives: https://commons.wikimedia.org/w/index.php?title=Special:PrefixIndex/Commons:Mobile_app/deletion_request_tracking/archive And that's only the ones that we catch... Or just look a little through the by now nearly 15'000 files in https://commons.wikimedia.org/wiki/Category:Uploaded_with_Mobile/Web You should have no problem finding copyvios galore, or unused & basically unusable selfies. Just shows how much slips through the first level screening. (The statistics for the category itself may be a little bit better since a lot already got deleted, but still: Augias' stable was a cleanroom compared to that.)
It was good to try the EXIF filter for combatting low quality uploads, but it seems pretty clear that this isn't the right strategy for dealing with the issue. Lupo, I think you're right that this isn't about bad faith users, but rather about new people who haven't really figured out what uploading entails on Wikimedia projects. We're going to try limiting mobile web uploading to autoconfirmed users, which should drastically lower the volume of incoming uploads (because currently only ~3% of new mobile web editors make it to 5 edits, which is in the ballpark of autoconfirmed thresholds across all the different WM projects) -- if our theory on clueless newbies holds true, this should also increase the ratio of quality uploads in the right direction :) As always, thanks for the feedback; we'll continue to monitor the feature and make permissions tweaks as needed.
(In reply to Maryana Pinchuk from comment #38) > We're going to try limiting mobile web uploading to autoconfirmed users, Limiting this to autoconfirmed users is certainly worth a try. I agree that it _should_ help improve the quality of that stream. Thanks a lot! Could you please post here again when that is active? (If it works, I guess we'll notice right way :-) It might also be worth analyzing/thinking about why the Android and iOS upload streams seem to have a much better success ratio.
See https://gerrit.wikimedia.org/r/137462
(In reply to Jon from comment #40) > See https://gerrit.wikimedia.org/r/137462 That is about not displaying the upload button in some other cases. The change about disabling this for non-autoconfirmed users appears to be https://gerrit.wikimedia.org/r/#/c/137461/ Glad to see that something is going on.
Hopefully when this change goes live you will see a big difference :)
And *when* does this go live? So far I have not seen any improvement.
(In reply to Lupo from comment #43) > And *when* does this go live? So far I have not seen any improvement. Could Mobile folks please provide an answer? Thanks.
If i'm right, it's still online :)
(In reply to Florian from comment #45) > If i'm right, it's still online :) What does _that_ mean? What is "it", and what does "being online" mean here? The two Gerrit commits mentioned in comment 40 and comment 41 were merged on June 7 and June 12. The fix for bug 66169 was merged on June 11 and definitely is active now at the Commons. 1. Is the "autoconfirmed users only" implemented as part of this bug now active or not? 2. Should we be seeing any improvements at the Commons already or not? 3. If not, *when* will this be deployed?
1. Is the "autoconfirmed users only" implemented as part of this bug now active or not? Yes. 2. Should we be seeing any improvements at the Commons already or not? Yes. I wanted a bit of time to go by before digging into the data, since deletion doesn't always happen immediately post upload -- I can run some stats on uploads & deletions from before and after the change went live and share with you. Keep in mind, however, that we recently (since June 17) began redirecting users on tablets to the mobile site (they'd previously been going to the desktop site), which appears to have increased the number of new mobile editors and uploaders, so while the ratio of good-to-bad uploads may have changed, the absolute numbers of both have probably gone up. I'll dig into the data on that, too.
Okay, There are still 80% nonsense/copyright violation uploads.
Mabye you like to IF globa_editcount = +900 allow mobile upload.?
Change 143751 had a related patch set uploaded by Florianschmidtwelzow: WIP: Add Uploadrestriction using edit count https://gerrit.wikimedia.org/r/143751
According to http://mobile-reportcard.wmflabs.org/#uploads_daily-graphs-tab, restricting the upload page-action button to autoconfirmed users had no effect on mobile uploads whatsoever, which is a bit strange.
I took a look at data and can confirm the filter is working on /English/ Wikipedia. There have been no uploads from users with an edit count of 0. On the other hand there were uploads from the following wikis by users with an edit count of 0. Maybe autoconfirmed status is different there? arwiki 95 eswiki 92 dewiki 76 ruwiki 59 trwiki 49 itwiki 48 jawiki 39 fawiki 34 kowiki 32 thwiki 30 ptwiki 28 commonswiki 27 idwiki 20 zhwiki 17 nowiki 17 nlwiki 16 frwiki 16 plwiki 14 cswiki 13 ukwiki 12 hewiki 11 svwiki 11 bgwiki 10 fiwiki 7
Kaldari pointed out the above data is not accurate as it uses the edit count on English Wikipedia not the native wiki. I took a look at tablet vs mobile through the lead photo upload icon. 5199 mobile uploads vs 236 tablet uploads so the tablet redirect shouldn't have effected this too much. 3740 of these users come from English Wikipedia across devices. 3585 of these users come from English Wikipedia on mobile and none of them had an edit count of 0. Very bizarre. Not sure what's going on here.
Change 143778 had a related patch set uploaded by Kaldari: Hide Uploads button unless user has adequate permissions https://gerrit.wikimedia.org/r/143778
Change 143778 merged by jenkins-bot: Hide Uploads button unless user has adequate permissions https://gerrit.wikimedia.org/r/143778
In the meantime, we've added a new change which also hides the Upload button in the left menubar unless the user is autoconfirmed. If that doesn't improve things, we may need to resort to Florian's idea in https://gerrit.wikimedia.org/r/#/c/143751/ (a hard edit count limit for all wikis that is separately configured).
I just took a look at the deletion stats -- while the autoconfirmed change did decrease the total number of uploads coming from the in-article upload button, as opposed to the Uploads menu item in the site navigation menu (in-article uploads dropped to 25% of all mobile web uploads, whereas they had previously accounted for 50%), the overall deletion rate doesn't look to have been affected: 66% of images uploaded from mobile web in the 3 days pre and post autoconfirmed change have since been deleted. Since it's been over 2 weeks since the change went live, and since these images tend to be fairly stringently scrutinized by the Commons community, it's pretty safe to say that's probably the majority of the egregious violators. (So, Steinsplitter, I don't think the 80% junk rate you quote above is entirely accurate.) Implementing the autoconfirmed gate on the Uploads menu item would be a good first step, and we can reevaluate from there.
(In reply to Maryana Pinchuk from comment #57) > (So, Steinsplitter, I don't think the 80% junk rate you quote above is entirely > accurate.) Just have a look at https://commons.wikimedia.org/wiki/User:Didym/Mobile_upload , unfortunately, it is accurate.
(In reply to Maryana Pinchuk from comment #57) > (So, Steinsplitter, > I don't think the 80% junk rate you quote above is entirely accurate.) Not all uploads will be deleted immediately, we don't have enough admins doing this jobs. Cleaning up Mobile Uploads is not a nice job, there are atm hunderts of files waiting for deletion.
FYI, the latest change (hiding the upload button in the left sidebar menu unless the user is autoconfirmed) should take affect on Thursday, July 10. This basically makes it impossible to upload from mobile unless you are autoconfirmed (as now both entry points will be hidden). Let's see if that has a more dramatic effect than the previous changes.
> makes it impossible to upload from mobile unless you are autoconfirmed What if some users have a bookmark to Special:Uploads or still navigate directly to upload page :) they still can upload images. I think we can add the check for mf-uploadbutton to SpecialUploads, too, there is already a right check (for upload enabled).
Change 143822 had a related patch set uploaded by Florianschmidtwelzow: Show Uploadbutton only when user has the permission https://gerrit.wikimedia.org/r/143822
Change 143822 merged by jenkins-bot: Show Uploadbutton only when user has the permission https://gerrit.wikimedia.org/r/143822
(In reply to Ryan Kaldari from comment #60) > should take affect on Thursday, July 10 If i'm right, the change is now in production for 9 days. I'm very interested in, if you see some changes, specially @Steinsplitter? :)
Yes, we do see some changes. The number of Mobile/Web uploads has been about halved, from about 200 to 100 per day. However, of these 100, the vast majority (in the area of 70-95%) is still crap that should not have been uploaded in the first place. You can find some statistics at https://commons.wikimedia.org/wiki/Commons:Forum#A_propos_.22mobile_upload.22 (for days since July 16; in German). While this change helped reducing the number of uploads through this channel, it did not improve the overall quality of uploads. I said so already above: this feature suffers from the systemic problem that the users of the feature have NO CLUE about copyrights, free licenses, or the Commons. The vast majority of uploads also isn't used anywhere. Technical solution attempts won't help. You will need to either * figure out a way to really educate users of this feature, or * restrict the use of the feature to users that can be assumed to know what this is about, * or switch this feature off altogether. For the second approach, someone suggested restricting the feature to users who have a global edit count > 900.
The most important statement from comment #65: > the users > of the feature have NO CLUE about copyrights, free licenses, or the Commons.
BTW, on July 19, 2014, I've seen three uploads through Mobile/Web without proper {{information}} template: https://commons.wikimedia.org/wiki/File:Keilana_portrait.jpeg https://commons.wikimedia.org/wiki/File:Noratramloroli.jpeg https://commons.wikimedia.org/wiki/File:Mareecheatham.jpeg The file pages did contain some text, but no proper template. I also noticed that something (bits.wikimedia.org?) was very slow and intermittently did not respond at all yesterday. What's up?
I suggest to allow uploads only by "autopatrolled" users and where global usercount is +400 edits.
(In reply to Lupo from comment #67) > BTW, on July 19, 2014, I've seen three uploads through Mobile/Web without > proper {{information}} template: ... Happened again. Opened bug 68321 for this.
(In reply to Steinsplitter from comment #68) > I suggest to allow uploads only by "autopatrolled" users and where global > usercount is +400 edits. ^^ can we do this? @Jon
http://sustainableman.org/wp-content/uploads/2013/07/DrIanMalcom.jpg Of course, anything is possible, but let's look at this differently. We have significantly more users from mobile uploading bad images than we do on desktop. This got personal for me as I have 294 edits on Wikipedia and I frequently enjoy uploading images from mobile to articles which need them. Upping the limit to 400 stops me from doing this which makes me really sad :-(. I think something like an edit count of 10 makes sense but I worry anything higher basically turns Commons into an elite members only club. We cut out rare images from developing worlds (see this inspiring success story that was enabled by mobile uploads for example - http://comments.gmane.org/gmane.org.wikimedia.foundation/67982 - by a user who only has an edit count of 77 [1]. To me this opens up the bigger and more important question: How can we get these users uploading good images? Open questions: * Are we seeing the same repeated offenders uploading images in good faith? Is there a way we can communicate to them better? * Are we sending messages to users? Are they responding? Do they repeat upload? [1] https://tools.wmflabs.org/supercount/index.php?user=Dileepunnikri&project=en.wikipedia
Sorry for the lengthy reply, but I felt your questions deserved detailed answers. (In reply to Jon from comment #71) > http://sustainableman.org/wp-content/uploads/2013/07/DrIanMalcom.jpg That remark applies perfectly to the development of this whole feature. The Commons community has pointed out these problems before this feature went live, but the powers that be chose not to listen. So don't give me witty quotes! > This got personal for me as I have 294 edits on Wikipedia... Please don't take this personally. That's an unhelpful frame of mind to try to solve this problem. Besides, you could upload your images through other channels, could you? (Normal desktop upload...) And perhaps it would also help you understand the problems better if you had more edits on WMF projects and thus more direct experience with the projects this feature was developed for. (No offense intended, but with that few edits you're really still a Wikipedia noob. There are noobs who know what they're doing, and I trust you're one of those. But don't generalize from yourself to other noobs. Most first have to learn the ropes, and this copyright thing is more complicated than writing your first (or 2nd, 3rd, or 4th) Wikipedia article.) > Open questions: > * Are we seeing the same repeated offenders uploading images in good faith? In general, no. Most try this once, upload a few or a batch of files (if a batch, it's frequently sports images taken from various websites and invariably rights managed by some agency such a Getty or USA TODAY Sports). Relatively few uploaders try again a few hours or days later and keep uploading images the next days. However, those who do keep making the same mistakes. Some uploaders even keep uploading the same image repeatedly, even though it was deleted as copyvio and they were notified about that. Either they did not understand why the file they uploaded is gone, or they don't care. We also sometimes see people uploading the same image again a few hours or days later even if the first upload has _not_ been deleted and is still there. Evidently, they don't get warnings about duplicates (or ignore them), and evidently they don't remember what they already did or don't know how to find their own uploads. I have no numbers on this; it isn't very frequent, but it does happen regularly. Personally, I give repeat offenders a personalized talk page message (not autotranslated, but very simple English text: "Stop uploading images taken from other websites. If you continue to upload copyright violations, you will be blocked."), especially if I catch them in near real-time. That works most of the time; most _do_ stop after that and I rarely have to block. (But that apparent success rate of that simple message may be a coincidence: typically I give that after the third bad upload, then allow for a fourth "grace" bad upload, and block on the fifth. Not very many uploaders do actually upload that many files.) If that doesn't help, I block for a short duration (for first-timers; if it happens on several days, I increase block lengths) and leave yet another message that tells them why they are blocked and that again instructs them to read up on scope, copyrights, and licensing. BTW: about messages: there is a reason I use this simple blunt message. I made the same experience already years ago at the English Wikipedia fighting vandalism: templating vandals with {{test}} templates of increasing severity (last time I looked they all had weasly text like "please use the sandbox for tests", "please don't vandalize", "you apparently vandalized, you might be blocked") has absolutely no effect and leads to a block. I had much better success by simply telling vandals after the first or second incident right away: "stop vandalizing or you will be blocked". Most stopped after that without me having to block. A concise, stern, and clear message giving a direct order ("stop doing this") and giving a clear promise of consequences ("you _will_ be blocked") is much better. It tells them "hey, somebody is watching my fooling around, and if I continue this crap, there'll be consequences". Of course, if they don't stop, you then have to block. Actually, I don't think "good faith" is part of this problem. I stated somewhere above already that I do *not* think these uploaders acted in bad faith: they just don't grok copyrights, free licenses, what the Commons is, or what a "free encyclopedia" is. I see very few clear vandalism attempts of "bad faith" uploads. Let's keep "good faith" out of this discussion: it is a given for all involved. You have to realize that nowadays especially the younger generation thinks anything on the web was up for grabs. No wonder with instagram, tumblr, twitter, facebook, pinterest, ... and all the "share this" buttons everywhere (or "re-tweet this" encouragements). *Of course* people think it was fine to re-post stuff taken from elsewhere. They do not realize that the Commons, being limited to freely licensed media, *is different*. > * Are we sending messages to users? Yes; uploaders are normally notified on their talk pages when one of their uploads is marked as problematic. The messages tell them that something is wrong with their uploads, that they should read the relevant pages about our scope, copyright, and licensing, and instructs them to react (and even tell them where and how). The messages used are even auto-translated in the user's language, if possible. > Is there a way we can communicate to them better? Define "better". Possibly the wording of the templates could be improved. But see below. > Are they responding? No. I did already wonder whether they actually do see the talk page messages we leave (especially if they're logged in on some Wikipedia, upload from there, and we reply on their Commons talk page). But even if they see them: we can't educate all these uploaders individually through personal interaction. We also can't contact them on their local Wikipedia talk pages: unless you automate this, it's too much hassle for already overloaded curators at the Commons. > Do they repeat upload? See above. Please don't muddy the waters with "success stories". Of course there are some. The problem here is the success-failure ratio of about 10%-90%. And those 90% bad uploads are creating too much work. I might add that any bot operator whose upload bot produces 10% of bad uploads gets into trouble quickly and will see his bot blocked. Here we have an upload channel that generates 90% of bad uploads. In summary: I think you're right that this is more a social engineering problem than a technical problem (and I said so before). Technical measures alone will not be sufficient to solve this. But either solve this social engineering problem quickly, or shut off this feature until you have a clear idea how to tackle it, or severely throttle the feature through technical means until you know how to solve the social part. (And "solving" the social problem might involve educating Internet users at large; see above about "share this" buttons and the like.) Right now, this upload channel with its failure rate of 90% is not acceptable.
I think, too, that the actual situation of mobile uploadsmust be better, specially for the community, which review the uploads. But, i think that an editcount of 400 is too big. Isn't a minimum editcount of 100 enough? For the implementation of this, i would prefer to add a new configuration variable (with default 0), so that third party users and Wikipedia-projects can set it self.
Just in case it helps, Commons' Picture of the Year contest defines a voter eligibility of 75 live edits on any single Wikimedia project. Could this be considered a reasonable requirement to upload pictures from mobile?
@Quim Gil: That sounds good for me, but i think (for the reasons wrote above), that a fixed level in code of MF isn't the best way. For Wikimedia-Projects (in generally and Commons as an example) the Editcount can be 75, that sounds good, too :)
@Florian, yes, I fully agree that this is Wikimedia specific problem that deserves a Wikimedia specific solution. I'm also not implying that requiring a minimum Editcount will solve the entire problem. I was just trying to find a reasonable number that the Commons community is already familiar with.
> Technical measures alone will not be sufficient to solve this. A well-thought video clip or slideshow before anybody can start uploading mobile with a simple multiple-choice question that must be answered correctly after the clip was played could have an extraordinary positive impact on this problem. Get away from the idea that people can "quickly" upload files. Most of the time people do it "quickly", they get it wrong.
> A well-thought video clip or slideshow before anybody can start uploading mobile I personally disagree for that :/ I think alot of mobile users visit Wikipedia and Commons with a limited data plan. That means, that our way should be to reduce the transferred content as much as possible and not blow it up with videos or slideshows, where the user don't know, that they come. That's my personally thought about that :)
Maybe you're also interested in my first-time mobile user experiences reported in bug 68375. Summary: * the upload tutorial doesn't work for me on my desktop browser, but that may be a quirk on my end. Please verify all the same. * maybe add a "respect third-party copyrights" line on the description page. Part of the problem really appears to be that the process is so streamlined and simplified, that it is OK *only* for self-taken pictures but completely neglects that people can and do upload other images through it. It is indeed a very quick process, and it is right only in one particular use case, and completely wrong in many other use cases. Which explains the poor success-failure rate. Use cases: * I want to upload an image I have just taken with my smartphone - OK, though the selfies are not really in scope and will be deleted unless used on a user page. * I want to illustrate this article with this image I've just taken myself - OK for uploading, but I didn't see any post-upload help to actually use the uploaded image anywhere, i.e., to include it in the article I wanted to illustrate. * I want to illustrate this article with this old image I've found at XYZ - process is wrong for that. * I want to illustrate this article with this image I saw on that other website - process is wrong for that, and user should be educated not to do this at all. * I saw this cool picture and want to share it with Wikipedia - process is wrong for that, and user should be educated not to do this at all. I did mention in bug 68375 one idea to enforce that the process is really only used for self-taken images: do a server side Google image search for each Mobile/Web upload and refuse the upload if more than one hit. Just a crazy idea, but it would catch most copied-from-somwhere files. Instagram uploads could probably be caught by looking at the width/height ratio; they're usually square. If the ratio is in the range [0.9, 1.1] or some such, refuse the upload.
(In reply to Florian from comment #78) > I think alot of mobile users visit Wikipedia and Commons with a limited data > plan They probably want to avoid uploading at all because images are usually bigger than people would expect. (In reply to Lupo from comment #79) > do a server side Google image search I'd also like to endorse the idea of a Google Image Search for each upload that is claimed to be own work. On Desktop, this could be maybe a warning but on mobile I agree that refusal is okay. On top of that it'd be great if we would have a database of pHashes or similar techniques to allow similarity detection of files already uploaded (to address the duplicate issue).
> to address the duplicate issue Duplicate images will be detected already, maybe there is a problem in detection of scaled images. But, like i said in german discussion on commons, (@mobile team, if i'm completly wrong, then please say anything :)): These suggestions are all good, and yeah, i would prefer to imrpvode the upload workflow and functionallity to make it more useful for the whole community and less work for admins and "cleaner" in the community. But this are all function suggestions, and in my opinion, mobile frontend is that: a forntend. This functionallity is good for desktop as well, so in my point of view it's better to make this changes in core, or with an extension, but not in mobile frontend (this isn't a mobilefrontend problem generally, it comes on mobile frontend only, because it's much easier to ignore the non-existing warnings in mobile as on desktop. And the upload in mobile is much easier as on desktop. Just my 2 cents :D
(In reply to Florian from comment #81) > Duplicate images will be detected already If they are exact duplicates (i.e. their data-hash matches), true. Consider an App that allows altering metadata and this whole file hash becomes useless. Perceptual hashing would also allow finding similar images, thus allowing suggesting categories based on file contents. > These suggestions are all good Which suggestions? Please clarify. > This functionallity is good for desktop as well What? Blocking users with less than X edits in any WMF project from uploading? Maybe ... but make sure administrators at Commons are able to exempt users from this restriction. > it's much easier to ignore the non-existing warnings in mobile as on desktop Huh, can you please clarify? Is it a good or bad thing? Should there be a timeout on mobile, for example before the warning can be dismissed? Please elaborate. > And the upload in mobile is much easier as on desktop ... comparing which aspects? Is this something you want to see changed or adopted to desktop, or ... ?
(In reply to Rainer Rillke @commons.wikimedia from comment #82) > (In reply to Florian from comment #81) > > These suggestions are all good > Which suggestions? Please clarify. E.g. a better detection of duplicate images (not only copied), or automatic search in image search engines, if imeages marked as "own work" are really not uploaded anywhere else. > > This functionallity is good for desktop as well > What? Blocking users with less than X edits in any WMF project from > uploading? Maybe ... but make sure administrators at Commons are able to > exempt users from this restriction. No, look above. Blocking users for less than X edits is in discussion for mobile only?! > > it's much easier to ignore the non-existing warnings in mobile as on desktop > Huh, can you please clarify? Is it a good or bad thing? It isn't a bad, nor a good thing. It depends on the devices the frontend is made for. On mobile it's terrible to scroll down a huge amount of text and warning messages on a (mostly) tiny screen. If i remember correct, that's the reason why there is no warning message above the upload formular. There was a similar discussion about anon warnings in edit forms (if anonymous editing is enabled), see https://bugzilla.wikimedia.org/show_bug.cgi?id=59937 > Should there be a > timeout on mobile, for example before the warning can be dismissed? Please > elaborate. Maybe that, maybe there is only one line which indicates, that there are information/warnings about uploads or something else. > > And the upload in mobile is much easier as on desktop > ... comparing which aspects? Is this something you want to see changed or > adopted to desktop, or ... ? On desktop you have much more text and information to read before you can upload (in general you read a the text before uploading). On mobile you simply choose the file, give a description and click upload -> finished. On desktop you have much more to do (e.g. with UploadWizard, the default uploader): - (can be hidden, but default you see this) Information, what you can upload and what not - Choose file - release rights - add description, category, date... - Finished That i mean.
Can somone pleas set ip a patch for reqired usergroup to upload: diff: - autoconfimred + autopatrolled AND mimimun editcount on commons: 75 And then let uns see if works. I am sure this reduces the crapuploads. Thanks.
sorry for posting again, i mean: diff (required usergroup to upload with -75 edits): - autoconfimred + autopatrolled __OR__ mimimun editcount on commons is "75" Thanks.
Note: since we deployed the last patch (to add the autoconfirmed threshold to the left nav Uploads feature as well as the in-article upload) on July 10th, unique mobile web uploaders dropped by half (from ~120-140/day to ~60/day),[1] and the number of deleted uploads this month is also extremely low compared to what we've seen in other months.[2] Let's focus on the actual data, please :) 1. http://mobile-reportcard.wmflabs.org/#uploads_daily-graphs-tab 2. http://mobile-reportcard.wmflabs.org/#monthly_reports-graphs-tab
(In reply to Steinsplitter from comment #85) > sorry for posting again, i mean: > diff (required usergroup to upload with -75 edits): > - autoconfimred > + autopatrolled > > __OR__ mimimun editcount on commons is "75" > > > Thanks. You mean only Editcount on commons or globally? @Maryana: Does we have a statistic of delted mobile uploads or is it what i see on german commons discussion?
Regarding the Google Image search idea, this is a frequent suggestion that we have looked into. The main problem is that the only sites that offer this as an API service either charge money for it or have strict usage limits (or both). If anyone knows of a service that would be usable for this, please let us know.
(In reply to Florian from comment #87) > You mean only Editcount on commons or globally? Yes. Commons editcount. > @Maryana: Does we have a statistic of delted mobile uploads or is it what i see on german commons discussion https://commons.wikimedia.org/wiki/Category:MobileUpload-related_deletion_requests And a lot of "non logged" deletion. (In reply to Ryan Kaldari from comment #88) > Regarding the Google Image search idea, this is a frequent suggestion that > we have looked into. The main problem is that the only sites that offer this > as an API service either charge money for it or have strict usage limits (or > both). If anyone knows of a service that would be usable for this, please > let us know. Additional requests cost $5 per 1000 queries, up to 10k queries per day. This is _not_ a lot of money for the wikimedia foundation.
@Florian - sorry, wrong link on #2 - here's live-updating deleted mobile (web and apps) uploads per month: http://mobile-reportcard.wmflabs.org/graphs/deleted-uploads
(In reply to Maryana Pinchuk from comment #86) > and the number of deleted uploads this month is also extremely > low compared to what we've seen in other months.[2] Let's focus on the > actual data, please :) We do. Again: A lot of files are in queue for deletion. We don't have engough admins to delete allteh crap.
FWIW, I think it's still too early to see what effect the latest change has on percentage of uploads deleted. There is one reason to be cautiously optimistic though. Typically, mobile web deletions outstrip mobile app deletions by at least a 10:1 margin. So far for July, mobile app deletions actually exceed mobile web deletions: http://mobile-reportcard.wmflabs.org/graphs/deleted-uploads Of course it's possible this is due to app uploads skyrocketing for some reason, but there haven't been any significant changes on the app side, so I doubt that's the case. We should have a clearer picture in the next couple weeks.
(In reply to Ryan Kaldari from comment #92) > FWIW, I think it's still too early to see what effect the latest change has > on percentage of uploads deleted. There is one reason to be cautiously > optimistic though. Typically, mobile web deletions outstrip mobile app > deletions by at least a 10:1 margin. So far for July, mobile app deletions > actually exceed mobile web deletions: > > http://mobile-reportcard.wmflabs.org/graphs/deleted-uploads > > Of course it's possible this is due to app uploads skyrocketing for some > reason, but there haven't been any significant changes on the app side, so I > doubt that's the case. We should have a clearer picture in the next couple > weeks. We need to wait weeks again? Somone with access to prod can run a query to get actualo data / data from the last weeks.
> This is _not_ a lot of money for the wikimedia foundation. Argh :/ What is with third-party wikis? Yeah, maybe they haven't this problem, but in my opinion it's not the best way to introduce a function in an free and open source software that maybe cost you later something :) Maybe i'm the only one who thinks so :P > The main problem is that the only sites that offer this as an API service Unhappily Google does not provide an API for that, has anyone a contact in Mountain View to request one? :P That's a really problem :( > Yes. Commons editcount. Basically this is much easier to implement as global editcount, mobile team? Give it a try? (look at https://gerrit.wikimedia.org/r/#/c/143751/ it's only a little change) :) (Just for saying this ;)) > @Florian - sorry, wrong link on #2 - here's live-updating deleted mobile That's much clearer now, thx Maryana :)
(In reply to Maryana Pinchuk from comment #90) > @Florian - sorry, wrong link on #2 - here's live-updating deleted mobile > (web and apps) uploads per month: > http://mobile-reportcard.wmflabs.org/graphs/deleted-uploads That graph is not up to date. (In reply to Maryana Pinchuk from comment #86) > Let's focus on the actual data, please :) Right, let's. Up-to-date manually collected statistics are at https://commons.wikimedia.org/wiki/Commons:Forum#A_propos_.22mobile_upload.22 for days since July 16, 2014. From July 16 to July 21 (including), there were at least 330 speedy deletions of Mobile/Web uploads. Plus another 110 in various deletion queues (no permission, no source, no license, deletion requests). Your limn statistics show only 149 right now, so they are clearly not up to date or plain incorrect. Thank you for your attention.
(In reply to Ryan Kaldari from comment #92) > FWIW, I think it's still too early to see what effect the latest change has > on percentage of uploads deleted. I disagree. It's not too early. Just see the link I gave in comment 95. Also see my comment 65. Here's a rough summary of the situation since July 10: We have roughly 100 Mobile/Web uploads a day. Roughly 60% are speedy deleted, another 20% are in 7-day-deletion queues (of which most *will* be deleted), and of the remaining 20% about half is arguably not useful (unused selfies, digitally altered retrica images, some instagram stuff that we can't pinpoint since Google doesn't index it and because it's usually also digitally altered, etc). We get at most 10% halfway acceptable images, and 90% crap. That's the same relations as before July 10, only the volume dropped by half. And having played with it myself, it's also clear why this is so: mobile/web uploads is a single-purpose process (upload self-taken photos) that is being used outside its specification boundaries (it's also and mostly being used to upload images copied from arbitrary websites).
>From July 16 to July 21 (including), there were at least 330 speedy deletions of >Mobile/Web uploads. Where do you get 330? I count 93. Does that list include both app and web uploads or just web?
(In reply to Ryan Kaldari from comment #97) > >From July 16 to July 21 (including), there were at least 330 speedy deletions of > >Mobile/Web uploads. > > Where do you get 330? I count 93. Does that list include both app and web > uploads or just web? Just mobile/web. They are tagged in the upload log as "mobile web edit". July 16: 87 uploads; next morning 33 remained: 44 speedy deletions July 17: 84 uploads; next day 44 remaining: 40 deletions July 18: 91 uploads; next day 44 remaining: 47 deletions July 19: 123 uploads; next day 42 remaining: 81 deletions July 20: 104 uploads, next day 40 remaining: 64 deletions July 21: 96 uploads; next day 42 remaining: 54 deletions 44 + 40 + 47 + 81 + 64 + 54 = 330 speedy deletions out of 585 total mobile/web uploads in these 6 days. Of the remaining 255, 110 are in deletion queues. Or rather, were when I collected the statistics and posted them at https://commons.wikimedia.org/wiki/Commons:Forum#A_propos_.22mobile_upload.22 The precise numbers may have changed a little in the meantime. For instance, some of the remaining files were already tagged for speedy deletion. And starting tomorrow or the day after, the ones from July 16 still in deletion queues should slowly go away, too.
Sorry for being dense, but where are you getting those numbers from? Here's what I see by counting what's in User:Didym/Mobile_upload/2014_July_17-21: July 17: 104 uploads; 40 deleted July 18: 71 uploads; 14 deleted July 19: 107 uploads; 10 deleted July 20: 87 uploads; 18 deleted July 21: 82 uploads; 6 deleted
(In reply to Ryan Kaldari from comment #99) > Sorry for being dense, but where are you getting those numbers from? Here's > what I see by counting what's in User:Didym/Mobile_upload/2014_July_17-21: > > July 17: 104 uploads; 40 deleted > July 18: 71 uploads; 14 deleted > July 19: 107 uploads; 10 deleted > July 20: 87 uploads; 18 deleted > July 21: 82 uploads; 6 deleted These pages are not entirely correct, only files still existing at the bot run are listed, but a lot of files are already deleted at this point.
Thanks for the explanation! So where do Lupo's numbers come from?
I get those numbers directly from the upload logs. For your convenience, here they are: July 16: https://commons.wikimedia.org/w/index.php?title=Special:Log&offset=201407170000&limit=87&type=upload&user=&page=&tagfilter=mobile+web+edit July 17: https://commons.wikimedia.org/w/index.php?title=Special:Log&offset=201407180000&limit=84&type=upload&user=&page=&tagfilter=mobile+web+edit July 18: https://commons.wikimedia.org/w/index.php?title=Special:Log&offset=201407190000&limit=91&type=upload&user=&page=&tagfilter=mobile+web+edit July 19: https://commons.wikimedia.org/w/index.php?title=Special:Log&offset=201407200000&limit=123&type=upload&user=&page=&tagfilter=mobile+web+edit July 20: https://commons.wikimedia.org/w/index.php?title=Special:Log&offset=201407210000&limit=104&type=upload&user=&page=&tagfilter=mobile+web+edit July 21: https://commons.wikimedia.org/w/index.php?title=Special:Log&offset=201407220000&limit=96&type=upload&user=&page=&tagfilter=mobile+web+edit Did you actually visit https://commons.wikimedia.org/wiki/Commons:Forum#A_propos_.22mobile_upload.22 ?? Those links are all there.
I think it is OKAY to implant the things suggested in comment 85. I hope we can resolve this in a few day (and not weeks) :-)
>I think it is OKAY to implant the things suggested in comment 85. >I hope we can resolve this in a few day (and not weeks) That change would eliminate virtually all mobile uploaders. Only a miniscule percentage of mobile users have a significant edit count on Commons. I would be very reluctant to make such a dramatic change without waiting for more data to back it up with. I'm not saying I doubt your data, but it is a very tiny data set, and as soon as we would implement such a change, someone would inevitably file a new bug saying it is too hard to upload on mobile and we would go through the entire discussion again. I would favor: 1. Waiting until the end of the month so that we have more of a basis for comparison and can solidly justify crippling the feature. 2. Implementing a less dramatic change. For example, requiring a minimum 75 edit count locally rather than on Commons. This is just my personal opinion though.
@lupo @rillke: I think we can do something for mobile.js - It is annoying to ask the devs and to wait every time months.
(In reply to Ryan Kaldari from comment #104) > That change would eliminate virtually all mobile uploaders. Why is that worse than wasting even more community members' time needed to delete the rather constant percentage of useless uploads? What's the exact reason to not make a more drastic change and afterwards experiment with changing the threshold again, instead of the other way round? Is it "only" about gathering more data wanted by developers, with the obvious trade-off that the involved community wastes more time deleting stuff though folks have patiently provided data here several times now to describe the underlying problem?
How about: ask uploaders where they took the image from? * With options like "I made it myself", "my grandma made it", "found it somewhere on the interwebz". * Until they've selected something, don't allow uploading. * If they selected something other than "I made it myself", show a very short "copyright for idiots" style tutorial and disallow the upload. This should filter out people who care but are about to make a mistake and most of drones, leaving only persistent drones and malicious uploaders. Reperesentatives of both of these categories can be blocked fairly liberally.
This dos not help. Our experience. (In reply to Max Semenik from comment #107) > How about: ask uploaders where they took the image from? This dos not help. It is our experience.
>What's the exact reason to not make a more drastic change and afterwards >experiment with changing the threshold again, instead of the other way round? There are two reasons: 1. To get enough data to actually understand what's going on. The latest change only went into effect 12 days ago. Even if there was only a small effect on upload quality, it would be good to know exactly what that effect was. This will help to inform future decisions about things like anonymous editing on mobile. 2. Because we are a wiki. Allowing people to share isn't just a motto we're supposed to give lip service to. We should exhaust as many plausible solutions as possible before we effectively disable a core feature like uploading. In many parts of the world, mobile phones are the predominant way people access Wikipedia. Also, it would be helpful to have an actual goal in mind. What percentage of bad mobile web uploads can be tolerated? 75%? 50%? 25%? Does volume affect that number?
(In reply to Ryan Kaldari from comment #104) > 2. Implementing a less dramatic change. For example, requiring a minimum 75 > edit count locally rather than on Commons. Can somebody summarize what the current restrictions are? It's just autoconfirmed on local wiki, right? Can somebody explain to me why this user https://en.wikipedia.org/w/index.php?title=Special:Log&user=Owndesd (account created 07:05 on 22 July 2014) could then 9 minutes later upload a file through mobile/web https://commons.wikimedia.org/wiki/Special:Log/Owndesd ?? Autoconfirmed at the English Wikipedia means 4 days AND 10 edits. (At the Commons, it is just 4 days AFAIK.) Yet this user had 1 edit when he made the upload, and his account was brand-new. https://en.wikipedia.org/wiki/Special:Contributions/Owndesd What's wrong?
Thanks, Ryan – totally agreed. So, I just pulled the data for all deleted uploads in July, not just mobile web: Total uploads to Commons contributed from 7/1-today (7/22) that have been deleted as of today: 2117 Total uploads to Commons contributed from 7/1-today *from mobile web* that have been deleted as of today: 103 So, mobile web is currently responsible for less than 5% of all deleted uploads on Commons. That really doesn't seem like an unfair burden on the community to me, and I don't see why we're treating this specific contribution funnel differently from any other that we have on Commons. If Commons admins truly feel overworked dealing with inappropriate uploads, why aren't they focused on improving the desktop upload workflow, which is contributing far more inappropriate/deleted images to the project?
(In reply to Ryan Kaldari from comment #109) > 2. Because we are a wiki. Allowing people to share isn't just a motto we're > supposed to give lip service to. We should exhaust as many plausible > solutions as possible before we effectively disable a core feature like > uploading. In many parts of the world, mobile phones are the predominant way > people access Wikipedia. While I would tend to agree with you in general, in this case it's not "switching off a core feature". It'd be switching off a "relatively recently added feature that has proven problematic". It's not "Smartphones are evil, and poor countries be damned; we shut you out", it'd be more like, "Sorry guys, we tried, but this isn't it yet. We're going back to the drawing boards and hope to come back again later". > Also, it would be helpful to have an actual goal in mind. What percentage of > bad mobile web uploads can be tolerated? 75%? 50%? 25%? Does volume affect > that number? The goal is actually quite simple: the ratio must be such that you don't get complaints from the Commons community. Only half joking. The percentage of speedy deletions and deletion requests for mobile/web must not exceed that of other uploads. Imagine if we had a 80-90% rejection rate on normal desktop uploads... if mobile/web uploads blend in with other uploads in that respect, then they're acceptable and you won't get complaints about mobile/web uploads in particular. Maybe it is then discovered that the crap rate is too high in general, but that would then be a different problem.
(In reply to Maryana Pinchuk from comment #111) > Total uploads to Commons contributed from 7/1-today (7/22) that have been > deleted as of today: 2117 > Total uploads to Commons contributed from 7/1-today *from mobile web* that > have been deleted as of today: 103 I don't know where you get your numbers from, but if you look at the upload logs (deletions are redlinked there), this is simply not true. Stop denying the problem by presenting fake or wrong numbers.
@Lupo, I'm getting the numbers directly from the Commons database. Here's my SQL if anyone with access to the database wants to double-check: select fa_name, log_timestamp, fa_deleted_timestamp from filearchive, logging where fa_name = log_title and log_action = "upload" and log_timestamp >= '20140701000000' If you really feel that I'm misrepresenting the numbers, I'm afraid we can't continue this discussion productively.
I agree with Lupo. Stop denying the problem by presenting fake or wrong numbers. Why it is so difficult to get a correct statistic? Why volunteer need to explain thousand times the same thing and why volunteers need to educate you how to make correct statistics? NONONO... You get payed for doing mobile related work but i am wasting my time here with emxplain you thinks again and again... I have deleted THOUSANDS of uploads. Engough is Engougs. Really. AND Please use commons sense. There are ~80% (or moor, see Lupos stats) crap uploads... AND the volonteers need to wast a lot of time to sort this out. Important note: Maybe i find time to emergency switch off this crap tomorrow using this mobile.js
>Can somebody summarize what the current restrictions are? It's just >autoconfirmed on local wiki, right? That is correct. >Can somebody explain to me why this user >https://en.wikipedia.org/w/index.php?title=Special:Log&user=Owndesd >(account created 07:05 on 22 July 2014) >could then 9 minutes later upload a file through mobile/web Looks like the restriction is not working on lazy-loaded pages (pages loaded immediately after editing). I'll file a separate bug for this. >I don't know where you get your numbers from, but if you look at the upload logs >(deletions are redlinked there), this is simply not true. Looks like there's definitely some discrepancy here. Let's all assume good faith and figure out how we can make sure everyone's looking at the same data. It may be that there's an upload pathway on mobile that is not hooked up for eventlogging.
mariadb> select count(log_timestamp) from filearchive, logging where fa_name = log_title and log_action = "upload" and log_timestamp >= '20140701000000'; 1 row(s) returned # count(log_timestamp) '14257' "Total uploads to Commons contributed from 7/1-today (7/22) that have been deleted as of today" should be that number, right?
http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-mobile/20140722.txt and http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-mobile/20140723.txt Mainly focusing on what I said (trimmed for relevance): [16:51:53] <legoktm> MariaDB [commonswiki_p]> select count(*) from filearchive join logging on log_title=fa_name where log_action = "upload" and log_timestamp >= '20140701000000' group by log_title ; [16:51:57] <legoktm> 11949 rows in set (7.40 sec) [16:57:10] <legoktm> MariaDB [commonswiki_p]> select count(*) from filearchive join logging on log_title=fa_name join change_tag on ct_log_id=log_id where log_action = "upload" and ct_tag="mobile edit" and log_timestamp >= '20140701000000' group by log_title ; [16:57:13] <legoktm> it's running [16:57:18] <legoktm> 1583 rows in set (9.82 sec) [16:57:47] JamesR (~JamesR@wikipedia/JamesR) left IRC. (Quit: Goodbye.) [16:57:52] <legoktm> [16:57:47] <EarwigBot> legoktm: 1583/11949 = 1583/11949 (approx. 0.13247970541467905) [16:57:57] <legoktm> 13% is a lot. ^ mobile deletions / total deletions [16:58:35] <Maryana> but my point still stands :) [16:58:47] <legoktm> how? [16:58:55] <Maryana> the vast majority of deleted uploads are coming from desktop [16:59:14] <legoktm> that's because the vast majority of uploads are from desktop [16:59:24] <Maryana> yes [16:59:34] <legoktm> what's the upload to deletion ratio of mobile versus desktop? [16:59:37] <Steinsplitter> mobile uploads are only a fre usable [16:59:44] <legoktm> that's a more interesting stat IMO [17:01:00] <legoktm> there were 3060 mobile web uploads in July. [17:01:04] <legoktm> over half were deleted. [17:02:24] <legoktm> desktop had 360236 uploads [17:02:39] <legoktm> [17:02:35] <EarwigBot> legoktm: 11949/360236 = 11949/360236 (approx. 0.033169921940061515) [17:02:50] <legoktm> so 3% deletion rate on desktop, compared to over 50% on mobile. Someone should also sanity check my SQL...
[17:05:11] <Maryana> legoktm: can you run those numbers for after july 10? [17:05:36] <legoktm> sure. [17:06:49] <legoktm> 219237 uploads overall [17:07:04] <legoktm> 1308 in mobile [17:07:26] <legoktm> 672 mobile deletions, still over half [17:08:06] <legoktm> 6319 deletions overall 2.8% deletion rate overall 51% deletion rate on mobile 2.6% deletion rate on desktop (calculated by overall-mobile)
Patch for lazy-loading bug (bug 68414) checked in: https://gerrit.wikimedia.org/r/#/c/148560/ There's a decent chance this loophole is responsible for the discrepancy in uploading statistics as well.
(In reply to Maryana Pinchuk from comment #114) > @Lupo, I'm getting the numbers directly from the Commons database. Here's my > SQL if anyone with access to the database wants to double-check: > > select fa_name, log_timestamp, fa_deleted_timestamp from filearchive, > logging where fa_name = log_title and log_action = "upload" and > log_timestamp >= '20140701000000' > > > If you really feel that I'm misrepresenting the numbers, I'm afraid we can't > continue this discussion productively. @Maryana: I wrote in comment 113 that you were presenting fake or wrong numbers. I did *not* say "you faked" the numbers. Small but important grammatical difference. I just pointed out that the numbers you were using have nothing to do with reality. I didn't say you were misrepresenting your numbers; I said your numbers were bad to begin with. Besides, from the fact that you kept using and defending these numbers it's evident that you didn't read comment 95 and comment 97 and following, where I pointed out this discrepancy already. It is not helpful to ignore bug reporters' comments.
BTW, since we're all so focused on "statistics" right now, a cautionary note: If the statistics say X% were deleted, that does *not* mean that the remaining (100-X)% were OK. I just went over the mobile/web uploads from July 5 and July 6, where already a lot had been deleted, and I had no trouble at all to find still numerous copyright violations that had slipped through the first screening. Including very obvious ones such as [[:commons:File:Photo of Hill 2014-07-06 19-59.jpg]] (which is a low-res jpg copy of the "fair use" TV screengrab https://it.wikipedia.org/wiki/File:Augustus_Hill.png ), or [[:commons:File:Scott kazmir 2014-07-06 09-52.jpg]], which is an AP photo: http://www.apimages.com/metadata/Index/Mariners-Athletics-Baseball/e89985f7b99645b48188914a5a60ee7b/47/0 , or [[:commons:File:Perišić with VFL Wolfsburg 2014 2014-07-06 11-54.jpg]], which is a crop of a Getty photo: http://gty.im/478110559 . Take this just as anecdotal evidence that the curators at the Commons _are_ indeed overworked and stretched beyond their limits.
I just want to emphasize that hosting copyright violation is a crime in certain countries. There were a few bills (DMCA safe harbour) that allows WMF to operate, though, but this liberal legislation should not be exploited or overstressed. Not to mention how it harms Commons hosting copyright violations as it poses threats to re-users. Every single blatant copyright violation is one too much.
(In reply to Gerrit Notification Bot from comment #63) > Change 143822 merged by jenkins-bot: > Show Uploadbutton only when user has the permission > > https://gerrit.wikimedia.org/r/143822 This patch (don't show uploadbutton to users without autoconfirmed status on special:uploads, e.g. if they navigate directly to the page) was merged at July, 15. That means that this patch isn't deployed at July 10 with wmf13. It was deployed yesterday with wmf14 to commons, so that can be a problem, too, when we discuss the next steps :) @Mobile-Team: Maybe i overlook something, but in the changelog of MW1.24wmf14 for MobileFrontend isn't this change listed?! Is that only an documentation error? https://www.mediawiki.org/wiki/MediaWiki_1.24/wmf14#MobileFrontend
Florian: Actually, that change doesn't really take effect until tomorrow. It was deployed to Commons yesterday, but very few mobile users go to Special:Uploads directly on Commons. Most use it from the Wikipedias, which get the change tomorrow. (Just to clarify, uploads from Special:Uploads on the Wikipedias actually get uploaded to Commons, not the Wikipedias.) I have no idea why that isn't listed in the changelog. I've confirmed, however, that the change is in effect in wmf14, however.
Bug 66762 has been fixed, so this bug is no longer needed (by me at least). Reclosing.
oops, wrong bug :P
Had a quick discussion with the PM, design, and the other developers. The outcome of the discussion was to go ahead and implement some minimum local edit count threshold across all projects (in addition to requiring autoconfirmed status). The threshold will be configurable. We're also considering having a sprint devoted to improving the mobile upload workflow in general (ala Lupo's bug 68375). Thanks for your undieing patience. And yes, Rainer, we know that hosting copyvio images is a serious problem. I'm a Commons administrator myself, so I understand your pain (believe it or not).
Thank you, Ryan, and all others involved. Max already said so at bug 68375. If you indeed do a sprint on that rather general "improvement" bug, don't forget the (quickly drawn up and surely incomplete) list of use cases in comment 79.
Change 143751 merged by jenkins-bot: Add Uploadrestriction using edit count https://gerrit.wikimedia.org/r/143751
@Ryan (and others): Do you think it's a good idea to backport change https://gerrit.wikimedia.org/r/143751 to wmf15? So it will deployed at July 29?
@Florian: I think it's fine to let it get deployed normally. FWIW, the uploads graph shows that unique uploaders have been halved again since the lazy-loading loophole was closed last week: http://mobile-reportcard.wmflabs.org/#uploads_daily-graphs-tab Also I just noticed that that graph shows "uploaders" not "uploads", which may have been the source of earlier confusion.
@Ryan: OK :)
Change 150075 had a related patch set uploaded by Kaldari: Add Uploadrestriction using edit count https://gerrit.wikimedia.org/r/150075
Change 150075 merged by jenkins-bot: Add Uploadrestriction using edit count https://gerrit.wikimedia.org/r/150075
@Florian: Actually, Maryana asked me to go ahead and deploy it to wmf15 so it should be live on the Wikipedias on July 31. We'll also have to deploy a config update sometime between now and then for it to actually have an effect though.
@ryan: Thanks for info! > We'll also have to deploy a config update sometime between now and then for it to actually have an effect though. Working on it...
Change 150145 had a related patch set uploaded by Florianschmidtwelzow: WIP: Add Uploadrestriction for Commons in MF https://gerrit.wikimedia.org/r/150145
Since the LIMN graphs are broken, I ran some numbers manually: 2014-01: 70% deleted (2598 of 3697) 2014-02: 74% deleted (2732 of 3705) 2014-03: 72% deleted (2698 of 3767) 2014-04: 56% deleted (2321 of 4174) 2014-05: 67% deleted (4200 of 6229) 2014-06: 63% deleted (3794 of 5976) 2014-07: 61% deleted (2160 of 3526) (will increase further) Personally, I think we should shoot for less than 50% deletion rate as a short-term goal.
SQL queries used... Images deleted: SELECT EXTRACT(YEAR_MONTH FROM log_timestamp) AS month, count(*) AS count FROM logging INNER JOIN tag_summary ON log_id = ts_log_id WHERE tag_summary.ts_tags LIKE 'mobile edit%' AND log_action = 'upload' AND NOT EXISTS ( SELECT img_name FROM image WHERE log_title=img_name ) GROUP BY month; Images uploaded: SELECT EXTRACT(YEAR_MONTH FROM log_timestamp) AS month, count(*) AS count FROM logging INNER JOIN tag_summary ON log_id = ts_log_id WHERE tag_summary.ts_tags LIKE 'mobile edit%' AND log_action = 'upload' GROUP BY month;
Change 150145 merged by jenkins-bot: Add Uploadrestriction for Commons in MF https://gerrit.wikimedia.org/r/150145
I don't know if I can resolve this fixed or not... but I did just deploy the configuration change.
> I don't know if I can resolve this fixed or not I think we can keep it on patch to review since we have feedback from Commons users and/or the new statistic after this change. Thanks @Matt Walker for merge.
*On Commons, mobile upload should be only possible for users with an editcount higher as 75* should fix this definitive. Closing this for now as *resoled*. Thanks to all for commenting and contributing patches.
Perhaps I missed something, but is this live? For example, recent entry: (Upload log); 17:49:57 . . R3k813 (talk | contribs) uploaded "File:R3K Official Grafitti Logo 2014-08-03 17-49.jpg" (Contributed image from Special:Uploads) (Tags: Mobile edit, Mobile web edit) That is the user's first edit to commons. My understanding is that the change presented by this bug should have prevented uploads like that (Or was the change only for the callout from article viewing, and not special:Uploads?).
> Or was the change only for the callout from article viewing, and not special:Uploads? No, with this[1] patch the upload restriction should be on Special:Uploads, too. I have tested it with my account (20 edits on commons) and i don't have the link to Special:Uploads (and no button to contribute an image, if i open Special:Uploads directly) and no button to add an image to an article. In this special case i think this image isn't uploaded from commons to commons. I think, he uploaded the image from Special:Uplaods of his home wikipedia (en.wiki). Like Ryan said in Comment 125: (In reply to Ryan Kaldari from comment #125) > (Just to clarify, uploads from Special:Uploads on > the Wikipedias actually get uploaded to Commons, not the Wikipedias.) For wikis except Commons we set a minimum edit count (local, so from en.wiki) to 10, and the user has 12 Edits, so he can use Special:Uploads and the upload for an article on en.wiki. [1] https://gerrit.wikimedia.org/r/#/c/143822/
(In reply to Florian from comment #146) > > Or was the change only for the callout from article viewing, and not special:Uploads? > > No, with this[1] patch the upload restriction should be on Special:Uploads, > too. I have tested it with my account (20 edits on commons) and i don't have > the link to Special:Uploads (and no button to contribute an image, if i open > Special:Uploads directly) and no button to add an image to an article. > > In this special case i think this image isn't uploaded from commons to > commons. I think, he uploaded the image from Special:Uplaods of his home > wikipedia (en.wiki). Like Ryan said in Comment 125: > (In reply to Ryan Kaldari from comment #125) > > (Just to clarify, uploads from Special:Uploads on > > the Wikipedias actually get uploaded to Commons, not the Wikipedias.) > > For wikis except Commons we set a minimum edit count (local, so from > en.wiki) to 10, and the user has 12 Edits, so he can use Special:Uploads and > the upload for an article on en.wiki. > > [1] https://gerrit.wikimedia.org/r/#/c/143822/ Ah. I misunderstood the change. Thanks for clarifying.
(In reply to Bawolff (Brian Wolff) from comment #147) > Ah. I misunderstood the change. Thanks for clarifying. Np :) Maybe @Lupo can give us (after several days) a new statistic about the uploaded/deleted statistic. Maybe we have to adjust the 10 edits for non-commons-wikis, if needed :)
There is the question whether the restriction will only affect the web interface or also the Apps: [[:c:Commons:Village pump#Mobile upload restriction]]
If I understand it right, https://gerrit.wikimedia.org/r/#/c/150145/ should be active now? So people who are either not autoconfirmed locally or who have less than 10 edits at the local wiki (whether autoconfirmed or not, and 75 if the local wiki is Commons) should not be able to upload through the Mobile/Web interface. Right so far? Then why can [[:commons:Special:Contributions/Pmirosee]] upload [[:commons:File:Kristen stewart 2014.jpeg]] (a clear copyvio from Getty) in his first edit, when he has exactly one single edit at the Turkish wikipedia ([[:tr:Special:Contributions/Pmirosee]]), and the account was created the same day?? Appears to me that this feature doesn't work yet. Or I don't understand it. I also do not see any effect on the uploads at all. See the continued statistics at https://commons.wikimedia.org/wiki/Commons:Forum#A_propos_.22mobile_upload.22
(In reply to Lupo from comment #150) > Then why can [[:commons:Special:Contributions/Pmirosee]] upload > [[:commons:File:Kristen stewart 2014.jpeg]] (a clear copyvio from Getty) in > his first edit, when he has exactly one single edit at the Turkish wikipedia > ([[:tr:Special:Contributions/Pmirosee]]), and the account was created the > same day?? Hm, that upload does not have the characteristic edit comment, so maybe it came in via Special:Upload (without "s"). However, [[:commons:File:Jessiann Gravel Beland 2014-08-05 11-21.png]] was uploaded by a user whose home account at tr-Wikipedia was created yesterday and who has no edits anywhere except the Commons, where he had 1 edit (before the file got deleted). So I still think this feature isn't quite working as it should.
No, it couldn't have gone live since July 31.
I reopen this, because of: This[1] user, without 10 edit's on no wikimedia wiki uploaded yesterday (after the registration of the account yesterdaay) this [2] image. I have tested Special:Uploads (directly, the link in the sidebar isn't there, that's ok) on en.wiki. There i'm not autoconfirmed and have no 10 edits. But i can contribute an image, if i want. After relook the code here [3] it's because of $wgMFPhotoUploadEndpoint, which is set to commons by default [4]. For now, the user can still upload (using Special:Uploads directly), if he is in all wikis except Commons ($wgMFPhotoUploadEndpoint not empty). This needs to be rewritten. [1] https://commons.wikimedia.org/wiki/Special:CentralAuth/Chetwilliams [2] https://commons.wikimedia.org/wiki/File:Alan_Wall_Photography_2014-08-06_22-14.jpg [3] https://gerrit.wikimedia.org/r/#/c/143751/9/includes/specials/SpecialUploads.php [4] https://github.com/wikimedia/operations-mediawiki-config/blob/master/wmf-config/InitialiseSettings.php#L11764
Change 152748 had a related patch set uploaded by Florianschmidtwelzow: Check userCanUpload when wgMFPhotoUploadEndpoint is set https://gerrit.wikimedia.org/r/152748
Review needed at https://gerrit.wikimedia.org/r/152748 . Could somebody please go take a look? This should be merged as soon as possible. The success ratio for mobile/web uploads is still very bad. I'd like to have that in 1.24wmf18. See continued statistics at https://commons.wikimedia.org/wiki/Commons:Forum#A_propos_.22mobile_upload.22 Or -- since the limn statistics have been fixed -- see those: http://mobile-reportcard.wmflabs.org/graphs/month-uploads 711 uploads in August http://mobile-reportcard.wmflabs.org/graphs/deleted-uploads 962 deletions at the time of this writing. (Looks like those statistcs are based on the deletion date, not the upload date. At the beginning of August a lot of uploads from July were still in deletion queues. Otherwise, how could there be more deletions than uploads?)
(In reply to Lupo from comment #155) > Otherwise, how could there > be more deletions than uploads?) I think yes, the statistic shows, how many images are uploaded and how many images are deleted (no matter when uploaded) per month.
Change 152748 merged by jenkins-bot: Check userCanUpload when wgMFPhotoUploadEndpoint is set https://gerrit.wikimedia.org/r/152748
Change 154364 had a related patch set uploaded by Florianschmidtwelzow: Check userCanUpload when wgMFPhotoUploadEndpoint is set https://gerrit.wikimedia.org/r/154364
Change 154364 merged by jenkins-bot: Check userCanUpload when wgMFPhotoUploadEndpoint is set https://gerrit.wikimedia.org/r/154364
The last patch is now backported to MW1.24wmf17 and works for me on commons and wikivoyage. Closing again as resolved. @Lupo (or/and others) can you reopen this, if there is still no change? :)
Info: Patch successfully deployed on all WMF Wikis (including Wikipedias)
It seems, that there are still very useless images. The last days statistic: 24. Aug: 90% useless 25. Aug: 82% useless 26. Aug: 100% useless The rule implementation itself works good, but it seems, that the rules itself doesn't help to improve the quality of uploads :/
(In reply to Florian from comment #162) > It seems, that there are still very useless images. The last days statistic: > > 24. Aug: 90% useless > 25. Aug: 82% useless > 26. Aug: 100% useless > > The rule implementation itself works good, but it seems, that the rules > itself doesn't help to improve the quality of uploads :/ That is absolutely correct. All the measures implemented so far have helped reduce the volume of uploads through mobile/web, but the have not improved the success ratio at all. The hope had been, if I understood this correctly, that by introducing minimum limits for this functionality we would get less clueless people using it, and of those remaining more would have a clue what this is about. However, this did not happen. Most people using this feature still have no clue about copyrights, free licenses, or the Commons, and still upload the same kind of crap as before July 10. Just that we now get instead of 200 problematic uploads daily around 25-60 per day. I see nearly no experienced uploaders using the feature. I don't know why that is so, but I might think that more experienced people typically upload more than just one or two pictures at a time and prefer to use the desktop upload wizard or one of the external batch upload tools for that. So mobile/web upload is completely uninteresting for experienced users. (That's my personal speculation, though.) Most uploaders using the feature are still clueless; many are drive-by uploaders. As a result, we still have an upload channel through which nearly no useful but many problematic images come in, and that still has to be monitored closely. Increasing the minimum limits for using the feature will only reduce the volume even more. It will _not_ attract experienced people who know about copyrights and free licenses and the Commons to use mobile/web uploads, and it will also _not_ solve the systemic problem that the workflow is utterly wrong for anything but really self-taken photos. (The workflow caters well to that one use case, but it does nothing at all to discourage the undesirable use cases that are, unfortunately, the norm.) I therefore would like to take up Max Semenik's promise in bug 68375 comment 7: Please switch off mobile/web uploads altogether until a better approach has been developed.
Change 156523 had a related patch set uploaded by MaxSem: Disable mobile uploads https://gerrit.wikimedia.org/r/156523
I wouldn't disable it completely, if you have autopatrolled rights (or higher) on Commons you should be able to use mobile uploads. Could be extended to similar rights on home (or at least one?) wiki.
(In reply to Denniss from comment #165) > I wouldn't disable it completely, if you have autopatrolled rights (or > higher) on Commons you should be able to use mobile uploads. Could be > extended to similar rights on home (or at least one?) wiki. Please provide statistics about the number of autopatrolled users who actually used this feature in the past two moths, and about the number of images they uploaded, and the number of images retained from those. And what are the rules about becoming autopatrolled anyway? At the Commons? At the English Wikipedia? At the Arabic Wikipedia? It makes no sense to throw still more technical hacks and patches and warts at this very fundamentally broken feature. (It's not broken technically -- but its process in very flawed and it's broken socially.)
Change 156523 merged by jenkins-bot: Disable mobile uploads https://gerrit.wikimedia.org/r/156523
As Max said, we've decided to remove all uploading features from the mobile site until the team can devote more focused time to improving the workflow. Please see the thread on the mobile mailing list for more details on the rationale and next steps: https://lists.wikimedia.org/pipermail/mobile-l/2014-August/007927.html
(In reply to Max Semenik from comment #107) > How about: ask uploaders where they took the image from? > > * With options like "I made it myself", "my grandma made it", "found it > somewhere on the interwebz". > * Until they've selected something, don't allow uploading. > * If they selected something other than "I made it myself", show a very > short "copyright for idiots" style tutorial and disallow the upload. > > This should filter out people who care but are about to make a mistake and > most of drones, leaving only persistent drones and malicious uploaders. > Reperesentatives of both of these categories can be blocked fairly liberally. At minimum, it would be good to look to the desktop UploadWizard for guidance on how to communicate what is wanted, and what is required. It's worthwhile to imagine yourself in a user's shoes (as I hope software designers already do!) Imagine this: * You have never heard of Wikimedia or Wikimedia Commons, but you like Wikipedia, and downloaded the Commons app for Android because you saw it was made by the same people. * The app offers you, essentially, one feature: upload media. * Where in the app's interface, even if you desperately WANTED to, could you gain any insight into WHERE it is uploading to, WHY, or what is desired by that mysterious entity? I don't think there's any opportunity to learn these things. So are we surprised that the uploads are mostly junk? Are we concerned about the amount of volunteer effort it takes to process these uploads? Will the Wikimedia Commons app be taken out of the Android marketplace along with the other Mobile Upload capability?
