Francesca Marano 7:12 am on March 30, 2021
Tags: test ( 14 )

Discussion: add a testing template to Trac

During the release cycle of WordPress 5.6, the focus lead, @monikarao, together with other people in the cohort, brainstormed ideas to encourage more testing.

One of the ideas is to add a template to TracTrac An open source project by Edgewall Software that serves as a bug tracker and project management tool for WordPress. so that when a new ticketticket Created for both bug reports and feature development on the bug tracker. is created the reporter is prompted to add information about the testing steps, not only the bugbug A bug is an error or unexpected result. Performance improvements, code optimization, and are considered enhancements, not defects. After feature freeze, only bugs are dealt with, with regressions (adverse changes from the previous version) being the highest priority. or the enhancementenhancement Enhancements are simple improvements to WordPress, such as the addition of a hook, a new feature, or an improvement to an existing feature. proposed.

Right now tickets are opened without instruction on how and what to test. During triagetriage The act of evaluating and sorting bug reports, in order to decide priority, severity, and other factors. sessions, scrubbers sometimes add the request manually, but the process could be automated.

Add a template

When reporters create new tickets they will be prompted to add these information:

List the exact steps necessary to reproduce the problem.
Specify the environment details where you reproduced the issue, e.g. WordPress version, PHPPHP The web scripting language in which WordPress is primarily architected. WordPress requires PHP 5.6.20 or higher version, browser & version, operating system & version, plugins installed, and their versions.
Paste any relevant error messages.
Are there any workarounds for this issue?
How can this issue be tested to ensure it’s fixed? Apply the patchpatch A special text file that describes changes to code, by identifying the files and lines which are added, removed, and altered. It may also be referred to as a diff. A patch can be applied to a codebase for testing.? Check the console? Check the front-end before and after? etc…
Are there any testing dependencies, such as a pluginPlugin A plugin is a piece of software containing a group of functions that can be added to a WordPress website. They can extend functionality or add new features to your WordPress websites. WordPress plugins are written in the PHP programming language and integrate seamlessly with WordPress. These can be free in the WordPress.org Plugin Directory https://wordpress.org/plugins/ or can be cost-based plugin from a third-party or script?
What is the expected behavior after applying the patch?

How-to guide

Write a short page that explains every one of those prompts, with examples and explanations on why it is important to test. Add the link to new tickets.

Notify coreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress.-test

If the tickets is high-priority, notify the #core-test channel in SlackSlack Slack is a Collaborative Group Chat Platform https://slack.com/. The WordPress community has its own Slack Channel at https://make.wordpress.org/chat/..

What do you think?

I opened a ticket in Meta for this, and @dd32 encouraged me to discuss it here first, so here I am 🙂

Do you think having a template is useful? For every new ticket? What about stale tickets? Do you think notifying the core-test channel is necessary? What do you think should be the flow?

Thanks!

#test

Paal Joachim Romdahl 5:35 pm on March 30, 2021

That is a very good idea, Francesca!
Check out the various templates we use for GutenbergGutenberg The Gutenberg project is the new Editor Interface for WordPress. The editor improves the process and experience of creating new content, making writing rich content much simpler. It uses ‘blocks’ to add richness rather than shortcodes, custom HTML etc. https://wordpress.org/gutenberg/.
https://github.com/WordPress/gutenberg/issues/new/choose
Tonya Mork 6:21 pm on March 30, 2021
I’m happy to learn that a template is possible in TracTrac An open source project by Edgewall Software that serves as a bug tracker and project management tool for WordPress..

Do you think having a template is useful?

Yes, I do think a template is useful and a big leap forward. How so?

This information needs to be gathered for a ticketticket Created for both bug reports and feature development on the bug tracker. to be investigated and then worked on. When provided upfront, the ticket is created in an actionable state where investigation/triagetriage The act of evaluating and sorting bug reports, in order to decide priority, severity, and other factors. can start. It can reduce the time and effort to move the ticket to the next step on its path to resolution.

Having a template provides structured guidance to the reporter on what information is needed. It can save the back-and-forth of gathering it.

For every new ticket?

No. I think only defect/bugbug A bug is an error or unexpected result. Performance improvements, code optimization, and are considered enhancements, not defects. After feature freeze, only bugs are dealt with, with regressions (adverse changes from the previous version) being the highest priority. tickets.

What about stale tickets?

Stale tickets will need this information too if it doesn’t already exist.

Often times the info is there but spread across different threads within the ticket.
What could helpful is to collect it and then summarize using the template in a new comment.

Do you think notifying the coreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress.-test channel is necessary?

Depends on what workflow(s) is the trigger.

A ticket has different test triage touch points:
- Ticket doesn’t have testing information => need to ask for it
- Ticket is in a testable state for reproducing the reported problem
- Ticket is marked as needs-testing but the patchpatch A special text file that describes changes to code, by identifying the files and lines which are added, removed, and altered. It may also be referred to as a diff. A patch can be applied to a codebase for testing. is not yet in a testable state
- Ticket is marked for testing and is in a testable state
Each of these states need action. Pinging the channel for each of these would be very noisy. Pinging that a ticket is in a test-ready state could be beneficial to alert the Testing Team that a ticket is ready.
Helen Hou-Sandi 8:52 pm on March 31, 2021

Broadly I think that a more guided experience in creating a ticketticket Created for both bug reports and feature development on the bug tracker. can definitely be helpful, especially with TracTrac An open source project by Edgewall Software that serves as a bug tracker and project management tool for WordPress. fields being less familiar to many. My experience with freeform templates (e.g. the ones found in various GitGit Git is a free and open source distributed version control system designed to handle everything from small to very large projects with speed and efficiency. Git is easy to learn and has a tiny footprint with lightning fast performance. Most modern plugin and theme development is being done with this version control system. https://git-scm.com/.-based trackers) from a contributor perspective is pretty negative, I find them overwhelming and stressful because it’s not clear to me if everything is required, it’s easy to miss sections, and it’s a pretty significant hassle to select and delete or edit parts that are pre-filled. Some of these tools are moving toward a form-based interface, which I find a lot easier to fill out and parse, so my inclination would be to target that as the UXUX User experience, rather than a blob of Markdown-ish to be altered.

Specific to Trac and its fields, but more broadly than just testing, I think this would also be a great opportunity to provide plain language guidance for reporters to be better equipped to set some of the fields initially without follow up from bugbug A bug is an error or unexpected result. Performance improvements, code optimization, and are considered enhancements, not defects. After feature freeze, only bugs are dealt with, with regressions (adverse changes from the previous version) being the highest priority. gardeners. So for example, thinking of this as more of a flow perhaps via an external form that then submits to Trac leaving the existing new ticket interface as-is for the familiar, perhaps it goes:

1. What would you like to report? (Something is broken (bug), something could be better (enhancementenhancement Enhancements are simple improvements to WordPress, such as the addition of a hook, a new feature, or an improvement to an existing feature.), I’d like a new feature, I noticed a security problem) – note that I’m leaving out “task” on purpose here. Now let’s say they select “bug”.
2. “What did you see and what did you expect to see?”
3. “If possible, please list steps to reproduce the bug.” (If empty, maybe set something like a `needs-reproduction` keyword)
4. “Please list details of the site you noticed this on (WordPress version, browser, operating system, etc.)” – this could also perhaps prompt for Site Health info and tuck it into something that collapses/hides in the UIUI User interface.
5. “In what version of WordPress did you first notice this bug?” (this sets version field)
6. “How badly does this bug impact you?” (this sets severityseverity The seriousness of the ticket in the eyes of the reporter. Generally, severity is a judgment of how bad a bug is, while priority is its relationship to other bugs. field, which is one of the ones that is notably confusing because it’s severity for the user, vs. priority for the project).
7. “Which of these components do you feel this bug belongs to (leave as “General” if you’re not sure)?”
8. “Do you have any screenshots, screencasts, or other files to attach?” (sets `has-screenshots` with image/video files)

This is just me thinking through how it might go, and each of the types would need its own flow of things. Certainly we could start with bugs/testing instructions and expand from there. I think it’s something that would take a fair bit of thought and then implementation work but could very well produce better Trac tickets and if done as an external form would likely be portable to any other tracker were we to move in the future.
Jonathan Desrosiers 5:03 pm on April 2, 2021

I am all in favor of finding ways to encourage more testing. But I have some concerns with this.

First, people don’t read when opening tickets. We see this all the time when people do not read the security disclosure at the top (and even check the “I am not reporting a security vulnerability” checkbox) and publicly submit security issues on TracTrac An open source project by Edgewall Software that serves as a bug tracker and project management tool for WordPress.. This doesn’t mean adding more information is bad, but we should be cautious about how, where, and how much we add.

Second, Trac is a system that very few people have access to update, and an even fewer number of people that understand how it works/how to make changes to it. If we add something to Trac, we should be sure it’s as “evergreen” as possible, as something that needs to be update frequently would be a maintenance burden.

Third, I am concerned about raising the barrier of entry to creating a bugbug A bug is an error or unexpected result. Performance improvements, code optimization, and are considered enhancements, not defects. After feature freeze, only bugs are dealt with, with regressions (adverse changes from the previous version) being the highest priority. report ticketticket Created for both bug reports and feature development on the bug tracker.. Personally, I think it should be as easy as possible to open a new bug report. We want all of the bug reports, even the ones with little information, or no steps to reproduce. Often times contributors are able to figure out what the issue is with poor context. We should be careful to introduce information that does not add a huge amount of effort.

I think it’s also important to note that there are multiple kinds of testing, and there will likely be unique requirements for each. For example, there is testing patches, testing bug reports, testing features, etc.. Each happens at different points of the ticket lifespan, requires different information, and different steps.

WordPress.org

Welcome!

Communication

Discussion: add a testing template to Trac

Add a template

How-to guide

Notify coreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress.-test

What do you think?

WordPress.org

Welcome!

Communication

Add a template

How-to guide

Notify coreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress.-test

What do you think?

Share this: