A question I hear pretty frequently is, “Why a redesign of the admin panel so soon after 2.5?” Those who have attended WordCamps in the past few months have already heard the answer, but for the people who haven’t had that opportunity, this post is for you.
When the community response to the 2.5 admin redesign was mixed, it seemed like a good idea to do usability testing to find out which issues were based on actual interface problems vs. which complaints were just a result of not liking change. To prevent bias, a third party was contracted to conduct usability testing, Ball State University’s Center for Media Design, Insight and Research division. Try saying that three times fast with a mouth full of peanut butter. Or fitting it on a business card. To save time, we’ll just call that third party CMD, since that’s what they call themselves.
The plan that was developed involved multiple rounds of testing, as well as the creation of two prototypes, hardcore! The first phase involved a usability review of 2.5 by CMD, the results of which were discussed with lead developers. A quick prototype was created that addressed some of the lightweight issues, so that the test participants could use both 2.5 on their own blogs and the prototype on a test site. Results would be analyzed and compared, leading to a second round of suggestions. A second prototype would be developed over a week or two, which would then be tested with the same participants, and a final report delivered. But you know what they say… the best laid plans of designers and developers often go awry.
After the first round of testing, it was clear that a prototype delivering the kind of fixes that could be coded in a week or two wouldn’t make much of a difference overall. We all decided a more ambitious prototype was in order, one that would experiment with a new approach to screen real estate and attempt to address as many of the issues from 2.5 as was possible with a few extra weeks of time. A rapid design process was followed by an even more rapid development cycle. The second prototype is what you know as Crazyhorse.
The second round of testing blew everyone away. The research team had never seen such consistent results. Tasks were completed faster, participant opinions rated it higher, understanding of how interface elements worked was greater, and it wasn’t even a fully functional application. Of the test participants, every single one said they would choose the prototype over their current administrative interface, and it wasn’t even pretty (those of you who remember the original Crazyhorse will vouch for this).
A presentation on the process from start to finish was part of the schedule at WordCamp 2008 in San Francisco, and the slides are available online, but as always the slides only tell you so much without the videos, live demo and verbal narration that went with it. (Use Google and you can see audience videos of the presentation.)
Here, then, is a PDF of reasonable size that you can download and peruse at your leisure that outlines the usability testing project in some detail. I wanted to include some eye tracking videos, but the file was so huge it would have been ridiculous for anyone to download it, so I stuck with eye tracking outputs called gaze trails to illustrate the findings. I also tried to pare down the text to the more salient points, since more than 50 hours of test video really does reveal an insane amount of data. I also cut out the section about designing Crazyhorse in the interest of staying under 25 pages. Hopefully you’ll think it’s a good balance. I’ll try to put together a separate document on the design process of 2.7 in a couple of weeks that will include the early Crazyhorse material.
So, if you want to know what we learned from the usability testing this summer that caused us to create what is now 2.7, go ahead and read the report.
[…] to complain about the new administrative system. In the process, they decided to fix a thing or two until it naturally developed in a real design landslide: The second round of testing blew everyone away. The research team had never seen such consistent […]
Pingback from Screen Estate Management | AlphaDesigner Blogue on October 29, 2008
[…] Jane Wells at Automattic posted a report on their usability testing for the WordPress admin panel. Not only are they addressing most of my WordPress UI gripes, they’re also showing great […]
Pingback from Antipode - Article - WordPress admin usability study on October 29, 2008
[…] the company behind WordPress, released a report today which summarises their recent project to test the usability of WordPress. It’s an interesting […]
Pingback from More like a web application and less like a website | ../learninglab/joss/ on October 29, 2008
[…] Wells at WordPress opens the kimono further on the decision to redesign the administrative user interface for 2.7. I griped a while back that it seemed like the design changes were being made in a non-systematic […]
Pingback from A report on the usability testing for WordPress 2.7 (Jarrett House North) on October 29, 2008
[…] die Macher in einem längeren Bericht im offiziellen WordPress-Blog. Nach den sehr gemischten Rückmeldungen zum Redesign für WordPress 2.5 hatten sie Usability-Tests […]
Pingback from Ξ Warum WordPress schon wieder einen neuen Adminbereich bekommt Ξ UPLOAD - Magazin für digitales Publizieren Ξ on October 29, 2008
[…] Aw写的官方发布的WordPress用户体验调查报告提供了PDF原版报告下载，可是国内用户要穿墙才能下载，在这里可以看官方的详细报告，本站提供了个新下载地址。 […]
Pingback from wordpress官方发布体验报告 | 放羊博客 on October 30, 2008
[…] Check it out. It is a very interesting read: https://wordpress.org/development/2008/10/usability-testing-report-25-and-crazyhorse/ […]
Pingback from WordPress Usability test « Mcloide’s resources library on October 30, 2008
[…] SEO, WordPress, wordpress 2.5, wordpress.2.7 Imponerande värre när man ser vad som kommer i nya WP. Det som är mindre bra är att det åter kastar om adminfunktionerna för att bygga vidare på en […]
Pingback from WordPress ligger i frontlinjen « Lågprisflygbolag - och om livet bortom horisonten on October 30, 2008
[…] our finger tips. To read up more on the subject, check out the official WordPress post about is at Usability Testing Report: 2.5 and Crazyhorse . Share and […]
Pingback from WordPress Shows Us All What it Means To Be Hardcore With Crazyhorse Admin Dashboard | Top Blogger on October 31, 2008
[…] 在这里可以看到一份非常详细的调查报告。（PDF下载地址，请自行穿墙） […]
Pingback from 官方发布的WordPress用户体验调查报告（PDF） | 拽拽资讯 on October 31, 2008
[…] WordPress overhauls its user interface Posted by Sam Glover on October 31, 2008, to SoloSmallTech Think open source is unsophisticated? Think again. Take a look at the lengths the WordPress team went to in order to redesign the user interface. […]
Pingback from WordPress overhauls its user interface | Lawyerist on October 31, 2008
[…] releases a usability study conducted by Ball State. [Their 1st class report is available in both Slides and […]
Pingback from A Digital Outrigger / Library Usability Links 10/31/08 on November 1, 2008
[…] Puedes leer la explicación aquí. […]
Pingback from El porque de los cambios en WordPress 2.7 - Carrero Bitácora de los Hermanos Carrero, David Carrero Fernández-Baillo y Jaime Carrero Fernández-Baillo. on November 1, 2008
[…] Beta bringt vor allem ein völlig überarbeitetes Admin-Interface. Dieser Variante sind umfangreiche Usability-Tests vorausgegangen und nach ein paar Testrunden hat man sich mit der neuen Oberfläche sehr schnell […]
Pingback from WordPress 2.7 Beta 1 – Eindrücke ‹ bensite.net on November 2, 2008
[…] ziet niet vaak dat bedrijven hun usability rapporten op straat gooien, maar WordPress heeft dit wel gedaan. Een demo van WordPress 2.7 Crazy Horses is hier te bekijken. Dowload hier het usability rapport […]
Pingback from WordPress 2.7 krijgt opnieuw een volledig nieuwe adminsectie - MediaKip on November 3, 2008
[…] Na het één en ander getest te hebben, heeft WordPress 2.7 een volledige nieuwe design gekregen en meedere nieuwe functies. Zo kan je bijvoorbeeld ook reacties toevoegen vanuit je admin pagina, je kan sticky posts toevoegen, je kan meerdere artikelen tegelijkertijd wijzigen en nog veel meer. […]
Pingback from WordPress 2.7 Beta 1 nu te downloaden on November 4, 2008
[…] 1. Usability Testing Report: 2.5 and Crazyhorse […]
Pingback from 3 must read documents for usability experts — Jure Cuhalev on November 5, 2008
[…] Usability Testing Report: 2.5 and Crazyhorse – WordPress Blog This is the kind of testing I love to see. Although we don’t like to be tied to any one tool or Content Management System, we are suckers for WordPress. Usability is not something that is synonymous with programmers. It’s nice to see they are keeping up and doing a good job. Now if they will stop releasing updates every other week. Sheeesh! […]
Pingback from Blog Roundup for October | VIA Studio on November 6, 2008
[…] open-source CMS invites users to propose and vote on new features or kvetch; conducts usability testing; and offers the most customizable publishing interface […]
Pingback from WordPress 2.7 and why it will rock | Jessibird on November 9, 2008
[…] I can’t begin to tell you how excited I am about the upcoming version of WordPress (officially due tomorrow!). The development team has done a remarkable job of revamping the entire user dashboard based on usability test results. […]
Pingback from WordPress 2.7 « Anything Goes on November 9, 2008
[…] I’ve upgraded to WordPress 2.7 beta 2 and the new admin interface is really brilliant. I wish the corporate systems I use at work were as well laid out as this is. The developers have talked a bit about usability testing on their development blog. […]
Pingback from WordPress 2.7 first thoughts by Caspar Aremi on November 11, 2008
[…] prossima versione 2.7 di WordPress, attesa per fine novembre, avrà una serie di miglioramenti di usabilità. Questo è stato possibile grazie a un bel lavoro di redesign condotto dagli sviluppatori assieme […]
Pingback from WordPress e l’usabilità - Alberto Mucignat on November 24, 2008
[…] Il test, somministrato quest’estate fra maggio e agosto, è stato svolto inizialmente su una versione “potenziata” (corretta) di wordpress 2.5. Questa prima fase è terminata con delle raccomandazioni così sostanziali da obbligare gli sviluppatori, per la seconda fase, a creare un nuovo prototipo di test completamente diverso dal primo e quindi dalla versione classica di wordpress, il Crazyhorse Prototype. […]
Pingback from Sketchin - There is life inside experience design » WordPress 2.7: la rivoluzione on December 4, 2008
[…] Usability Testing Report: 2.5 and CrazyHorse […]
Pingback from WordPress › Blog » WordPress 2.7 “Coltrane” on December 11, 2008
[…] Usability Testing Report: 2.5 and CrazyHorse […]
Pingback from WordPress 2.7 “Coltrane” | The Blog Specialist on December 11, 2008
[…] Usability Testing Report: 2.5 and CrazyHorse […]
Pingback from WordPress 2.7 “Coltrane” release « From 1000 goto 2000 on December 11, 2008
[…] Usability Testing Report: 2.5 and CrazyHorse […]
Pingback from WordPress 2.7 “Coltrane” >> Fu4ny | Fun for You on December 11, 2008
[…] Usability Testing Report: 2.5 and CrazyHorse […]
Pingback from WordPress Taiwan 正體中文 › WordPress 2.7 “Coltrane” on December 11, 2008
[…] Usability Testing Report: 2.5 and CrazyHorse […]
Pingback from พีเอชพีวินด์! บล็อก » WordPress 2.7 “Coltrane” on December 11, 2008
[…] Usability Testing Report: 2.5 and CrazyHorse […]
Pingback from WordPress | Brasil » Tradução do post de lançamento do 2.7 on December 11, 2008
[…] Usability Testing Report: 2.5 and CrazyHorse […]
Pingback from WordPress-BR » Arquivo » Tradução do post de lançamento do 2.7 on December 11, 2008
[…] Usability Testing Report: 2.5 and CrazyHorse […]
Pingback from missing out on the latest wordpress update | KuroData.net on December 11, 2008
[…] lavoro dietro alla nuova interfaccia di backend, lavoro che ha coinvolto decine di sviluppatori, consulenti di usabilità e designer. E che ha portato come risultato finale una dashboard più semplice e veloce da usare […]
Pingback from » WordPress 2.7, qualche considerazione per i non sviluppatori » Full(o)bloG on December 11, 2008
[…] Usability Testing Report: 2.5 and CrazyHorse […]
Pingback from Running on WordPress 2.7 | Planet Lowyat on December 11, 2008
[…] Usability Testing Report: 2.5 and CrazyHorse […]
Pingback from WordPress 2.7 “Coltrane” @ 锐伊·拙笔 | 庸人自扰 ： 蔡伊琳 （Elaine Chua - rayvolvez） on December 11, 2008
[…] Usability Testing Report: 2.5 and CrazyHorse […]
Pingback from WbLogIn » WordPress 2.7 “Coltrane” on December 11, 2008
[…] Usability Testing Report: 2.5 and CrazyHorse […]
Pingback from WordPress 2.7 “Coltrane” - |VαмρWireleX| Weblog on December 11, 2008
[…] Usability Testing Report: 2.5 and CrazyHorse […]
Pingback from WordPress 2.7 ya está aquí! | Weblog Stuff on December 11, 2008
[…] Usability Testing Report: 2.5 and CrazyHorse […]
Pingback from DaboBlog - Cibercultura | Seguridad | GNU/Linux | Redes | Mac OS X | CMS| Opinión | Por David Hernández (Dabo) on December 12, 2008
[…] Usability Testing Report: 2.5 and CrazyHorse […]
Pingback from WordPress 2.7 更新[译文] « 宁的小屋 on December 12, 2008
[…] Usability Testing Report: 2.5 and CrazyHorse […]
Pingback from teknomagix » Blog Archive » WordPress 2.7 “Coltrane” on December 12, 2008
[…] Usability Testing Report: 2.5 and CrazyHorse […]
Pingback from WordPress 2.7 “Coltrane” | Personal Blogger Blogging Tutorials And Tips on December 14, 2008
[…] Usability Testing Report: 2.5 and CrazyHorse […]
Pingback from OneHamdi.Com » WordPress 2.7 “Coltrane” on December 14, 2008
[…] The WikiMedia Foundation skulle kunna lära sig en hel del av teamet bakom WordPress vad gäller användarvänlighet. […]
Pingback from Vagnretur » Kan man få pengar om man satsar på fel häst? on December 15, 2008
[…] Usability Testing Report: 2.5 and CrazyHorse […]
Pingback from 魅力燃烧 » Blog Archive » WordPress 2.7 “Coltrane” on December 15, 2008
[…] Usability Testing Report: 2.5 and CrazyHorse […]
Pingback from WordPress 2.7 “Coltrane” | James McWhorter on December 18, 2008
[…] Think open source is unsophisticated? Think again. Take a look at the lengths the WordPress team went to in order to redesign the user interface. […]
Pingback from WordPress Overhauls User Interface « Insight and Research’s News Archive on December 18, 2008
[…] to complain about the new administrative system. In the process, they decided to fix a thing or two until it naturally developed in a real design landslide: The second round of testing blew everyone away. The research team had never seen such consistent […]
Pingback from Screen Estate Management « Insight and Research’s News Archive on December 18, 2008
[…] Usability Testing Report: 2.5 and Crazyhorse In WordPress on 10/29/2008 at 3:25 PM WordPress.org https://wordpress.org/development/2008/10/usability-testing-report-25-and-crazyhorse/ […]
Pingback from Usability Testing Report: 2.5 and Crazyhorse « Insight and Research’s News Archive on December 22, 2008
[…] Usability Testing Report: 2.5 and CrazyHorse […]
Pingback from 起点 » WordPress 2.7 “Coltrane” on January 8, 2009
[…] Usability Testing Report: 2.5 and CrazyHorse […]
Pingback from WordPress 2.7 “Coltrane” : my sandbox on February 15, 2009
[…] Usability Testing Report: 2.5 and CrazyHorse […]
Pingback from WordPress 2.7 “Coltrane” on March 11, 2009
[…] Usability Testing Report: 2.5 and CrazyHorse […]
Pingback from WordPress 2.7 “Coltrane” | Yansen's Blog on March 15, 2009
[…] Usability Testing Report: 2.5 and CrazyHorse […]
Pingback from Tradução do post de lançamento do 2.7 » Comunidade WordPress-BR on March 19, 2009
[…] Usability Testing Report: 2.5 and CrazyHorse […]
Pingback from WordPress 2.7 “Coltrane” | myadsenseworld.com on March 20, 2009
[…] 用Drupal的时候，看到了一写关于Drupal Usability方面的东西，可能因为Drupal功能过于强大，搞的问题很多，整个community对于可用性非常重视，目前University of Baltimore的一个team正在帮忙测试和重设计一部分界面，并宣称Usability一日不解决一日不推出版本7。哈哈。这里是Link。google了一下，wp 版本2.5也有可用性测试，据说其中很多建议被2.7采用付出实践。 […]
Pingback from Devils Haircut » 抱歉误删了一些留言 on March 25, 2009
[…] is the amount of usability testing that took place during the development cycle. Starting with testing 2.5 and the Crazyhorse prototype and following with the 2.7 beta, the testing program looked at almost every feature and function in […]
Pingback from WordPress › Blog » Contributing to WordPress, Part III: Usability Testing on May 4, 2009
[…] is the amount of usability testing that took place during the development cycle. Starting with testing 2.5 and the Crazyhorse prototype and following with the 2.7 beta, the testing program looked at almost every feature and function in […]
Pingback from Contributing to WordPress, Part III: Usability Testing « Meraktan on May 4, 2009
[…] is the amount of usability testing that took place during the development cycle. Starting with testing 2.5 and the Crazyhorse prototype and following with the 2.7 beta, the testing program looked at almost every feature and function in […]
Pingback from Contributing to WordPress, Part III: Usability Testing | KGRAND ONLINE NEWS on May 4, 2009
[…] is the amount of usability testing that took place during the development cycle. Starting with testing 2.5 and the Crazyhorse prototype and following with the 2.7 beta, the testing program looked at almost every feature and function in […]
Pingback from Contributing to WordPress, Part III: Usability Testing | Coded Ink,Inc on May 5, 2009
[…] is the amount of usability testing that took place during the development cycle. Starting with testing 2.5 and the Crazyhorse prototype and following with the 2.7 beta, the testing program looked at almost every feature and function in […]
Pingback from Contributing to WordPress, Part III: Usability Testing - Word Press Magazine on May 7, 2009
[…] is the amount of usability testing that took place during the development cycle. Starting with testing 2.5 and the Crazyhorse prototype and following with the 2.7 beta, the testing program looked at almost every feature and function in […]
Pingback from essai numero 1 « saintgeorgeskg.com on May 7, 2009
[…] is the amount of usability testing that took place during the development cycle. Starting with testing 2.5 and the Crazyhorse prototype and following with the 2.7 beta, the testing program looked at almost every feature and function in […]
Pingback from Contributing to WordPress, Part III: Usability Testing on May 29, 2009
[…] is the amount of usability testing that took place during the development cycle. Starting with testing 2.5 and the Crazyhorse prototype and following with the 2.7 beta, the testing program looked at almost every feature and function in […]
Pingback from Contributing to WordPress, Part III: Usability Testing | XBox 360 Guide on July 11, 2009