Is A Plugin Validation Team A Pipe Dream?
Now that the WordPress plugin competition for 2009 is complete, Ozh has published his reviews of each plugin that was entered into the competition. Some of his reviews mention particular security issues that were discovered along with some shoddy coding. Once I read his reviews, it occurred to me that if the plugins in the competition had these issues, just imagine the state of plugins being hosted in the repository. If every plugin in the repository was reviewed as closely as the competitions were by someone like Ozh, I’m afraid of what the results would be.
This leads me back to the idea that the WordPress plugin repository needs a plugin validation team. This team would be a group of individuals of Ozh’s caliber that would review each line of code for any plugin that was initially submitted to the repository. They would make sure that the coding guidelines were met, bad code was corrected, and would not approve of a plugin until all checks were passed. Unfortunately, it’s too late for the repository to have such a team/process. In order for it to be truly effective, the plugin repository would need to be wiped out to start fresh with these new guidelines/practices in place. Something I don’t see happening anytime soon.
Currently, WordPress plugins are encouraged to be built around these WordPress Coding Standards. There is also documentation on how to write a plugin for WordPress. When a plugin is submitted to the repository for inclusion, the plugin is manually checked by someone to make sure that the submission came from a human rather than an automated process. I’m not sure if each line of code is checked or if it’s just a once over but once the plugin is approved, the author then has SVN access to that plugin which is an automated process. There in lies another problem. These revisions are not checked because there are hundreds to thousands of them each day. So the other risk here is a popular plugin authors account gets hacked, that person publishes malicious code into the plugin to trigger an upgrade notice and now potentially hundreds of blogs are compromised. Thankfully, this hasn’t happened yet but I wonder if it’s a matter of when, not if.
So even if there was a plugin validation team, there is no second line of defense unless you consider that to be the countless number of eyeballs reviewing the code as they install plugins onto their blogs. However, this raises another point.
I believe that the expectation for end users to dive into the code for plugins and themes they install onto their site is unrealistic. Essentially, everyone who uses WordPress would need to have a good understanding of PHP including the security aspects of it in order to make good decisions on whether a theme or plugin would be safe to install on their site. Matt Mullenweg and the WordPress project can not control end users blogs to make sure they are upgraded, they can’t control plugins or themes to ensure they are upgraded, they can’t control end user decisions but what they do control are the repositories. I think these repositories are a huge responsibility for the WordPress project and that in the beginning, more should have been done to ensure quality code would only be accepted into the repository. This is the way it’s worked for phpBB3 and so far, seems to be working well for them. Sure, if something like this were put into place, there wouldn’t be as many plugins or themes on the repositories but in this case, the end justifies the means. Good code and less insecure plugins and themes more than trump the amount of choices that would be available.
Now that a validation team simply wouldn’t work because the repository is in full swing, I think the potential is high that it is a ticking time bomb filled with insecure or improperly coded material. But, I don’t know PHP from ABC so I can’t confirm or deny my suspicion. The worst case scenario is that a new line of thinking takes place regarding the repository. Great place to pick from a large variety of plugins that fill a need but if you don’t know PHP to check how they were created, you may be installing the demise of your site. Of course, it would take some popular plugins to be compromised or, some popular sites to be compromised due to a plugin installed from the repository for this line of thought to gain any steam but considering how easy it is now to install plugins from the repository, it could certainly happen.
I see this entire issue as one of the big problems facing WordPress. Thankfully, nothing drastic has taken place so far but as more plugins are added to the repository, old ones are not pruned off (yes I know that some WordPress 1.5 plugins work on 2.8), and the bar to entry to use WordPress is lowered even more, the ingredients are slowly coming together to increase the risk for all who use the software.
As far as solutions to the problem, I can’t think of any off the top of my head that would not require the plugin repository to just start over. I suppose all end users of WordPress could read PHP For Dummies to be better educated on what is going on between plugins and their WordPress powered site but that would be expensive and some folks just want to publish content, not develop. Any solution that goes into place now which would make inclusion into the repository a more stringent process wouldn’t work because of the thousands of plugins submitted before them.
If the team is not going to make it harder to get into the repository then there should be clear warnings letting users know that while they host plugins, they can not be held responsible for any damage that may occur to their site from a plugin they installed from the repository. If I installed a plugin from the repository and it caused my site to be compromised, I would blame not only the plugin author, but much of the blame would be pushed onto the repository for hosting the plugin which made it easily available to millions of people.
Your Answers:
I find this subject to be a huge problem. You may not. Let me know in the comments what you think. Also, if you have any ideas on how this problem can be solved, particularly without requiring the plugin repository to start over, tell me about them.
16 Comments
It would make sense, if a new repository was started under new policies and validation. Then, once a significant amount of plugin authors had migrated, the old one be shut down.
To go through the current repo and all the plugins would take a considerable amount of time. Maybe plugins that have been checked could just get a “Verified” logo in the listings, or something of that nature?
Tons of ideas floating around, hopefully we can get some answers in today’s dev meeting.
Report
Other than the time factor for volunteers, probably one of the reasons that something more formal hasn’t been implemented is that some plugin developers would be more inclined to host the plugin themselves.
We host some of our own. For example, I have a login widget which I’ve maintained since WPMU 1.2. When I wrote it there wasn’t a repository. Now there are 3 or 4 other slightly different login widgets in the repository. So, I haven’t rushed to add it.
Report
For now, many security consultants offer code-reviews and fixes for problems like these. If the WordPress community does make it too restrictive, it will have the same image as the Linux kernel development community – a place where you can try and try harder but can’t get your plugin out there unless you’re lucky. That’s what has been happening with the Apple iPhone AppStore.
Report
there also needs to be a way to search through plugins that are compatible with the current version of WP. I’ve been looking for different things, and you always have to look under the junk that piles on top. By junk, I mean plugins that don’t even work with 2.8.4
Report
I’d say this would be on the Important but not Urgent list. They’d definitely have to pay someone to keep up with it though. Possibly going back to the older, less popular plugins and do them first. Maybe even cut off access temporarily. That way they can clean up plugins not in active use to disturb the least amount of people at a time.
Ron & I were just discussing about maybe even better idea is to have approved plugin frameworks for plugin devs to learn from properly the first time around. many of us got our feet wet by example code. If you start from a bad or unsecure example, you’ll just write more.
Report
@Ron – I didn’t mention it but that is probably the largest deterrent for this idea. However, why does phpBB3 have a group of people dedicated to performing these roles for free on their time. I think WordPress could do the same. Don’t know who would want to do it though.
@Nitin Reddy Katkam – I wouldn’t want that to happen and if handled correctly, it wouldn’t happen. But I see where you’re coming from.
@Andrea_R – The only reason it is not urgent right now is because none of the worst case scenario has happened yet. I think for it to work and to get a team together, money would have to be involved but I’m not sure Automattic and its investors would like to put money down that hole.
Approved plugin frameworks is something I haven’t heard of. But I think that is a large part of the problem. Crappy coded plugins in the repository which work but don’t work in the way that they should and then others coming along using those as examples to build on. This is a bad domino effect where at some point, those dominoes will fall.
Report
@Scott I agree with the secondary repository that would have enhanced features like searching on additional fields like version, author, update date range, screenshots available and better categorization than the current tags. WordPress.org could link to this additional site with enhanced features/security/testing which would encourage developers to submit their plugins (for more targeted exposure). They might even have a section for non-GPL and search on that field as well.
@trent I thought of creating a page on my site just with Google custom searches for the repository because I’ve had to build these searches to ensure I was getting the best plugin for the task. I saw one site that had done this with their own collection of plugins, but not linked to the repository (so the data was old). This seems like a simple interim fix that I’ll get around to at some point. I’m just getting started. Maybe Jeff wants to do it.
Report
I seem to recall a discussion a little while back abut optional standards which authors could build to which would given them the right to state they develop according to that standard.
It is self-certification, but, as with web standards, these sorts of standards would be a good first step to letting users know that the developer cared to meet those standards.
Report
I think leaving the repository open with some behind the scenes code validation makes the most sense. I haven’t seen a deliberately malicious plugin yet, and I have a feeling if it existed it would get pulled down pretty quickly.
I think the fundamental reason why WordPress has gained the popularity that it has, is because even a novice can join in and learn with the pros. Everyone is at a different stage of experience and knowledge, and there are bound to be flaws and bugs in just about every plugin out there when someone higher up the ladder looks down at it.
I agree that the “just because it works doesn’t mean it’s right” philosophy is one that as a developer you have to adopt fairly quickly, but not everyone has the resources to have their plugin combed through.
What I would hate to see is for the repo to follow the iPhone Store way of life and start rejecting or removing plugins. WordPress has always respected its legacy and the code it left behind, so if a stagnant plugin is sitting there and maybe needs some TLC, in our GPL open source world someone is free to pick up that lost code and give it a new home. If plugins were removed or rejected, that option disappears.
Imagine the kid that’s submitting their first plugin to the repo, waiting anxiously to show their friends what they did, only they get an email two days later saying it was rejected. With that kind of setup, it breaks their spirits and maybe makes them stop trying. With the current setup, they submit their code to the repo and if people download it they can give feedback to the developer via the wp.org forums, so that kid can learn from their peers and grow to be a better developer.
If there’s going to be a community of developers combing through plugin code looking for vulnerabilities or exploits, I hope it’s a positive experience for everyone and it’s treated like the customer service/public relations interaction that it is and that it happens pretty transparently.
Report
Interesting thought Jeff. The time is naturally going to make this hard to implement, but from an end user standpoint it makes a lot of sense.
@Ron – think of extensions for Firefox. Some host extensions on their own sites instead of through the addons at Mozilla. As an end user I feel a little more secure installing them from Mozilla, but I’ll still install directly from developer sites.
The majority will probably get plugins directly from WordPress, certainly those who would most likely benefit from having the plugin code go through some kind of validation.
Perhaps WordPress could give more guidance as to a set of standards for developing plugins and only those plugins adhering to the standards could be accepted into the repository. It doesn’t solve the problem, but it could be a step in the right direction.
Report
Why not learn from how projects are handled on github. That way everyone can check the code while it is in the repository and can easily fork etc and everyone is in the open.
Report
The main issue with reviewing is, obviously, time. I don’t know if I’m slow or if there are tools to automate parts of what I did but it took me 30 to 60 minutes to review a plugin. That’s more than 1 year, doing this 9 hours a day, if you want to review the 6500+ plugins for the repository. Not counting the countless plugin upgrades of course.
(Also, don’t assume that if trivial coding errors were spotted in the competition, things would be worse in the repository. It would be the same proportions I think, some contestant were obviously novice coders, some were skilled guys, just like the people who submit to the repo.)
I’m often a guy who sees the “that’s not possible” part of thing, but I don’t believe a validation team is possible, I mean volunteers who would check for vulnerabilities and other potential nightmares. This idea has been thrown a couple of times in the wp-hackers mailing list over the last few years, without gaining any real traction. There’s just too much time needed for such a task.
Now, there might be a team of, say, accredited consultants who could be hired to review plugins and give an official badge stating that the code looks good, but who would pay for this? Who to chose for such a reviewing task? Who to sue if you install a validated plugin that eventually contains a security hole and get your blog hacked?
This said, I’ve had a couple request for code review since the plugin comp. I’ll let the idea bounce into my mind for one or two days and maybe post something about it :)
Report
This is something I have been musing for a while and don’t think it is something that should necessarily be a centralised thing which is done by a team.
I think we need to build into the work-flow more social/community features so that:
It is really easy to report an issue to the person that can fix it – i.e. the plugin author
It is really easy to keep track of the developments on a plugin – there are feeds etc which can be leveraged (http://blog.ftwr.co.uk/archives/2009/09/10/feeding-on-feedback-and-progress/) but it would be nice for this to be even easier.
I would like to have the time to review more plugin code as I think the best way the community of developers can learn and the code quality can improve is constructive feedback and best practise documentation.
In my wildest dreams I would like to dedicate as much of my time as possible on making plugins better – but some things, like $dayjob and earning money, get in the way!
Report
@Ozh – good point about the potential liability held by a reviewer.
Report
[…] Is A Plugin Validation Team A Pipe Dream? […]
Report
This was not discussed in depth during the development meeting however, there is a central place where we should be having the discussion located here.
http://wpdevel.wordpress.com/2009/10/01/use-this-thread-to-brainstorm-ways-to-ma/
The more I think about it, the more I’m in line with the idea that a dedicated team is probably not the way to go but a hybrid of code sniffers and guidelines. Build in some crowd sourcing tools and such.
Report
Comments are closed.