Last modified: 2012-02-21 23:26:26 UTC
opensource Artifactory software should be installed (possibly on the Jenkins Machine) to allow us to manage versioned libreries for use with Maven built projects.
We don't distribute any of our other software via random 3rd party repo systems -- I'm not sure what the benefit is here for such an underused piece of software. What problem are we solving?
Please, please no more repo software. We can't properly maintain the billion we already use.
I was led to believe that these bugs are a formality and that this would be done by the end of the day. I've placed two abandoned projects into continuous Integration (CI) with the express intention to facilitate renewal of their maintenance by my self and volunteers (referred above as 3rd parties). This took up significant allocation time (4 days) of my free time in January and February. Since our Jerkins is geared to PHP and not Java but I've believe that this issue is important because: -- 1. CI empowers 3rd parties by allowing them to figure out which versions they should work on based on reports of the version which last compiled and which tests it passed. 2. It ensures that the software won't only build work on a single machine. The same goes to testing. 3. It makes all the dependencies explicit. The production version of Search code and its tests depends on files and scripts only available to users with root access to production servers 4. It has end product - downloadable versions which can be consumed by downstream projects and by end users - ops who require these extensions without getting bogged down by setting up build environments. B. In response to comment 1: I asked to use Artifactory as explained in my update "MWDumper - CI + Artifactory" last monday on 'Wikitext-l'. But to recap - the alternatives is to ask an external repository to distribute our visioned binaries. If hosting such an artifact repository is to difficult for WMF we could come to an arrangement with Codehaus, Ibilio, or Nexes. Let me know and I'll update POMs and Jenkins and I'll close this bug. However as you pointed out it is not the way we work. I still think that we should host our own Repository. I've evaluated Artifactory on the same Tomacat hosting my local Jenkins. I don't care which product is used as long as it integrates with Jenkins. C. In reponse to comment 2: Pretty please - just one more repo software.... soon you can remove one of the old ones like subversion.
Can this be done inside of Labs, and community managed? I don't think ops is willing to take on the responsibility of this, but if you'd like to, that would be fine (as long as everyone understands that it is community managed - meaning ops won't handle outages).
I'm fairly new to open source development but I'm pretty sure that if this project was hosted at Java.Net the infrastructure such as SCM, Issue managment CI etc could be taken for granted. The logic being to reduce the barriers to code contibution since it enables a long tail of contributors (AKA a developer community) who's single code contributions in aggregate account for t almost all of code over time. (like wikipedia ...) In theory I would not mind agreeing to your request. In practice however your request is not practical -- coordination costs far outweigh the benefits provided by the next best alternative: 1. At this point Ryan Lane is the goto guy with respect to labs. I make this bold statement since i the last 3 months your collegues on have had to deffer to you on every issue that I've raised. 2. Even if I recruit some new volanteers and if they eventualy get commit + labs access, they would still have to chase you down to get anything done. 3. AFAIK Labs is not a stable environment but still very much under development -- which boils down to your request being an unreasonable demand from the community at this point in the lab project life time. Therefore I entreat of you once again -- please check with other ops regarding supporting a modern and robust build workflow is not in WMF best intreset -- it does seem to be the gist of WMF 5 year startegic plan. If there are no longer sufficent resources for supporting long running projects in WMF and we are asked to roll our own we should migrate/outsorce the projects to more dynamic community such as Github or Java.net. That to will have a strategic cost, consider that Despite the growth of WMF dev staff in recent years there is a decline of general interest in MediaWiki software c.f. http://www.google.com/trends/?q=mediawiki
(In reply to comment #5) > I'm fairly new to open source development but I'm pretty sure that if this > project was hosted at Java.Net the infrastructure such as SCM, Issue managment > CI etc could be taken for granted. > It is. This is why we provide SVN (now Git) for SCM, issue management with Bugzilla, and CI with Jenkins.
I just skimmed the Artifactory site, so all it does is build the java project every X, push it to the unit testing and offer it for download? Could this be used for other language projects (to bundle code up, not necessarily to build projects)? or is this Java building only? (In reply to comment #3) > I was led to believe that these bugs are a formality and that this would be > done by the end of the day. Just wondering, Who said this? So we can improve on future communications if it is needed. > 1. CI empowers 3rd parties by allowing them to figure out which versions they > should work on based on reports of the version which last compiled and which > tests it passed. Will there actually be that much work on it, that it needs frequent testing and compiling? > B. In response to comment 1: > > I asked to use Artifactory as explained in my update "MWDumper - CI + > Artifactory" last monday on 'Wikitext-l'. But to recap - the alternatives is to That email looks it should of probably gone to Wikitech-l, the more general tech/dev list for mediawiki and wikimedia stuff. Wikitext-l is a seperate list for discussing the syntax used by our parser and such. (Which might explain why you didn't get many responses from it) > ask an external repository to distribute our visioned binaries. If hosting such > an artifact repository is to difficult for WMF we could come to an arrangement > with Codehaus, Ibilio, or Nexes. Let me know and I'll update POMs and Jenkins > and I'll close this bug. Will there be all that many binaries to offer? Couldn't you just put them in git directly (eww I know) or somewhere else?
So, let me just ask directly. Is this required for us to properly CI java applications that we use?
(In reply to comment #5) > 1. At this point Ryan Lane is the goto guy with respect to labs. I make this > bold statement since i the last 3 months your collegues on have had to deffer > to you on every issue that I've raised. The community members in the #wikimedia-labs channel can often answer many questions that I'd be able to answer. Yes, though, I'm the project lead, and the project launched like three months ago, so not everyone will be able to answer all of your questions. > 2. Even if I recruit some new volanteers and if they eventualy get commit + > labs access, they would still have to chase you down to get anything done. There are over 100 people with Labs access. I have a hard time believing that I'm a blocker for all of their work. > 3. AFAIK Labs is not a stable environment but still very much under development In which way is it not stable? It occasionally has some bugs with new instance creation, but we've only had one minor outage that basically no one even noticed. > -- which boils down to your request being an unreasonable demand from the > community at this point in the lab project life time. > Well, before Labs existed, the answer would have simply been: go buy a server and run this yourself. Now we're able to say "ops can't support this, but you can still run it on our resources". > Therefore I entreat of you once again -- please check with other ops regarding > supporting a modern and robust build workflow is not in WMF best intreset -- it > does seem to be the gist of WMF 5 year startegic plan. > > If there are no longer sufficent resources for supporting long running > projects in WMF and we are asked to roll our own we should migrate/outsorce > the projects to more dynamic community such as Github or Java.net. > > That to will have a strategic cost, consider that Despite the growth of WMF dev > staff in recent years there is a decline of general interest in MediaWiki > software c.f. http://www.google.com/trends/?q=mediawiki You are assuming that supporting a software repo for Java will grow our developer community. I don't think this is really the case. If this is something required to properly test software and there's not already a way to handle it with the tools available, we'll consider it; otherwise, you can either build it in labs, or can get help from other communities for this.
This is not a testing issue - it is a build issue as outlined in bug 34392. I'm pretty sure that a projects that only builds in our CI and nowhere is the kind of show stopper that stops community input and looking at any apache project one cannot help to see that the whole community becomes centered about the CI and thier micro sites. Regarding other questions - I'd rather not rehash the same message ad infinitum and get back to coding. I've check with the central repository. They have some pretty problematic restrictions: 1. We sign all our files with pgp. Can Jenkins be secured to protect such a certificate from tempering? 2. The other issues are unlikly to be understood here.
We don't wish to support another repository tool. If you'd like to maintain this in Labs, then we'd be more than happy to give you a public IP and DNS entry for this.
