The Test Team helps manage testing and triage across the WordPress ecosystem. They focus on user testing of the editing experience and WordPress dashboard, replicating and documenting bug reports, and supporting a culture of review and triage across the project.
If you’d like to help test Full Site Editing, please join the FSE Outreach Program. You can find current calls for testing for this program here and you can join the fun in #fse-outreach-experiment.
In the summer of 2021, the Test team started meeting again for chats, triage sessions, and scrubs. One of the things that keeps coming up is the need to have clear instructions for testing.
They are scattered across many Make websites, they are not all kept up-to-date with changes in the different environments they mention, they not always link to existing documentation and, in some cases, they link to pages that no longer exist.
This makes it more difficult for new contributors to join the sessions and actively participate in the team initiatives.
During one of the meetings, the attendees agreed that having good documentation is a priority to welcome new contributors.
At this stage, I am focusing on the first goal: review the testing instructions, simplify if possible, and make sure they are all up-to-date.
The process I have in mind is:
wordpress-developHere is the spreadsheet: https://docs.google.com/spreadsheets/d/1D4Q2_P_FriSxj19P2HIso81s2SZf5RcUVFaTUe12cuk/edit?usp=sharing
Then we can move on to goal number 2 (or another group of people can work on that simultaneously).
Wanna help? Comment in this blog post with your Slack username and we can start working 🙂
Thank you!
Props to @hellofromtonya and @mai21 for peer review.
The meeting started on SlackSlack Slack is a Collaborative Group Chat Platform https://slack.com/. The WordPress community has its own Slack Channel at https://make.wordpress.org/chat/., here.
@hellofromtonya described that this is our time together to talk about things for our team: blockers, needs, roadmap, learning, docs, etc.
@boniu91 reminded, that there’s open poll to decide when we’ll be meeting in the future, twice a week:
@hellofromtonya asked to post the answers as comments inside the mentioned post.
@hellofromtonya described what is the goal:
And also what’s the current status:
CoreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. tests were blocked for over 2 years from running PHPUnit 8 and 9. They are not blocked anymore!!
@pbearne asked if we have any list of backports that needs to be finished
@hellofromtonya write a list of PRs:
There are 2 PRs for backports:
Plus, there’s still work to do in #53904
Also, the backports are waiting for PR 1587 to be fully reviewed and committed, we need to be sure that all paths are covered.
@lucatume offered, that he can take a look
@boniu91 asked if there’s any calendar with future WP Core releases
@francina answered that not yet
@pbearne asked what’s the plan for the @covers
@hellofromtonya answered that the plan for @covers is to address those during the reorganization and namespacing effort.
Why?
That’s when we’ll be thoroughly reviewing each test. At that point, we’ll know what each test is actually covering.
Why wait?
To avoid duplicating effort. We’re focusing first on PHP 8.1 and modernizing for latest PHPUnit. Once that’s done, then we plan to switch to the reorganization, namespacing, covers effort.
@hellofromtonya mentioned that there are many ways of contribution:
For those who don’t know how to build automated tests, you can contribute too:
@boniu91 commented that it would be good to have some kind of document that would describe the possible ways of contribution for new Test Team members.
Team agreed, @francina agreed to lead the effort, @hellofromtonya and @boniu91 offered help.
On July 30th, I shared a survey to gather insights around how theme authors are exploring theme.jsonJSON JSON, or JavaScript Object Notation, is a minimal, readable format for structuring data. It is used primarily to transmit data between a server and web application, as an alternative to XML. today in order to shape what’s possible in the future with theme design. Thank you to the 31 people who took the time to share their feedback! By having concentrated feedback like this, better decisions can be made around what makes sense to include in CoreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress.. What follows is a high level summary of the results.
As always, GitHub issues are welcomed for anything not covered here or that comes up in the future. Please keep sharing your feedback there and know it’s appreciated as the blockBlock Block is the abstract term used to describe units of markup that, composed together, form the content or layout of a webpage using the WordPress editor. The idea combines concepts of what in the past may have achieved with shortcodes, custom HTML, and embed discovery into a single consistent API and user experience. theme design system continues to evolve.
Most folks who responded to the survey have experimented with building block themes either by starting from scratch or by forking an existing theme. Of note, TT1 Blocks didn’t stand out as a base that folks relied upon with many choosing to fork their own theme or working to move a classic theme they made to a block. While most folks used color presets, there was a variety of approaches included a few mixed naming strategies showing that this is still an open ended area of exploration. Along with colors and typography, the ability to customize layout and spacing options proved to be favorites. There was a near 50/50 split in terms of those who had explored per block settings or styles, with the Button block being the most commonly referenced by those who had. From the wide variety of feedback gathered, a few suggestions emerged for how best to improve the experience going forward including expanding theme.json documentation, adding commenting and improving the structure of theme.json, and more.
If you want to read the full reports, I have included an option below. Keep in mind that I intentionally removed any personal identifying questions that listed the contact information for folks (name, email address, country, etc).
31 people participated from 17 countries with the longest time spent responding clocking in around 60 minutes and the shortest time just under 2 minutes (removed a 2d response assuming it was left open in their browser).
52% of participants said that they have built and launched block themes, which was exciting to see! For those who answered with Other Option, the responses were, for the most part, combining different responses into one:
I tried using theme.json in a classic theme and also experimented with TT1-Blocks theme and FSE. But haven’t dig in too deep to create a fully custom FSE solution.
I’ve built 5 experimental block themes and explored theme.json with classic themes.
I’m currently building a hybrid theme for a fairly large SaaS client that relies on theme.json and Block Patterns.
I have explored using theme.json with a classic theme and I have experimented with building block themes.
I have experimented with block themes and fse. I am currently building a custom block theme for a client site.
This question was intended to help get insight into what resources folks are relying on and what the process to get started looks like. The results showed that more could be done to improve this part of block theme development, whether through amplifying available resources or improving those resources themselves. For the Other Option selection, there were some insightful responses:
I converted an existing theme to a block theme.
All of the above.
Using my Genesis base child themeChild theme A Child Theme is a customized theme based upon a Parent Theme. It’s considered best practice to create a child theme if you want to modify the CSS of your theme. https://developer.wordpress.org/themes/advanced-topics/child-themes/. and implementing/testing what I have come across over several resources online.
The comment section was particularly lively for this question as it asked those who forked an existing theme to share which theme/what approach they took. All the responses aren’t included here, but here is a representative sampling:
I have used all three. I no longer use empty theme because it is too basic for me. When I fork, I copy one of my earlier themes.
I extended an existing custom theme from our agency.
We started from a classic theme, created a blocks pluginPlugin A plugin is a piece of software containing a group of functions that can be added to a WordPress website. They can extend functionality or add new features to your WordPress websites. WordPress plugins are written in the PHP programming language and integrate seamlessly with WordPress. These can be free in the WordPress.org Plugin Directory https://wordpress.org/plugins/ or can be cost-based plugin from a third-party (Nova Blocks) around it, and step-by-step evolve towards a block theme (Rosa2). It’s not a fully block-based theme, but the goal is to get there soon.
I have done most of my experimenting with tt1. I have been referencing and digging into the code many block themes, trying to come up with the best methodology for my custom theme which will need to be production ready at end of month. Blockbase, seedlet-blocks, astra-blocks, genesis block theme.
Across the board in the comments, these four patterns emerged: starting with an existing theme they know well, following tutorials like fullsiteediting.com, forking an existing block theme, or some combination of each option. It feels important to note that TT1 Blocks, the block version of the Twenty Twenty-One default theme, was only mentioned three times.
On the whole, most folks mentioned the following:
To get a broader sense of responses, here is a sampling:
This is an ecommerce theme using WooCommerce blocks and our custom block library. Templates: Front Page Single Post Archive Search 404 Page Several Post templates and Page templates with sidebars
Template parts: Header Footer Shop Archive-product (categories, tags, attributes) Loop and a few other templates (e.g., loop with sidebar) depending on the setup
We primarily use template parts for Header and Footer but evolving steadily to all other theme parts. Here is a list of all the templates: https://github.com/pixelgrade/rosa2/tree/main/template-parts
In particular, in that last comment, you can see a wide range of possible template parts mentioned from Pixelgrade, including site branding and meta social (shared with consent).
For the other options, the following themes emerged as a combination of feedback and approaches:
This question resulted in some lively additional comments with a sampling shared below:
Surely, this is a tricky question as there is no standard cross-theme compatible way of naming colors.
Nowadays I kind of use a mixed naming strategy:
– primary, secondary and tertiary (,…) for base theme color scheme,
– an black, white, gray, dark gray, light gray for standard grayscale colors.
All of these colors populate editor palette too. I do not set other colors if not required by the project as I feel a theme should adhere to some limited color palette (while still providing some options for basic colors of grayscale for possible tinting).
I have been adding the color names. But am keen to the idea of the primary, secondary, etc. The problem for me in doing so has always been that they are too limiting. The designs I work off of in client theming are usually pretty complex and my colors do not always neatly fall into those categories.
I use semantic names with a Tailwind-like shading system. For example, my “primary” color gets split between different shades from 100 (lightest) to 900 (darkest). So, you get primary-100, primary-200, … primary-900. These can be reduced or added to on a per-project basis. The system is based on the popular Tailwind CSSCSS CSS is an acronym for cascading style sheets. This is what controls the design or look and feel of a site. framework, but it also closely models common systems of theme authors I have surveyed. They tend to have neutral, primary, and/or secondary colors across the average project.
The !important that is added to all theme colors is DRIVING ME CRAZY.
A few folks mentioned block-specific settings likely because this question came before the one about block settings. I removed those responses from this section.
For Layout, I counted the following mentions altogether: Layout, contentSize, wideSize.
For Spacing, I counted the following mentions altogether: Spacing, Margin, Padding.
Mainly custom definitions: filling gaps that aren’t currently supported as presets, and duplicating layout definitions for contentSize and wideSize to expose these values.
Not much to date, but waiting on more to become available so I can use it more.
I think I’ve touched every piece of `{ settings: {} }`, configuring things to my liking. Some common `settings.custom` properties that I set are: – Spacing values, with the most-used being a `spacing.global` value. – Content and wide-layout values because these are not currently exposed as CSS properties via presets. – Google Fonts system. It’s a bit too early to say what I will use the most going forward though. It is still early. The biggest thing that is missing is a global spacing value as a preset. I have a feeling that will be a common use case for most theme authors.
This question was nearly split, with 16 people responding “Yes” (52%) and 15 responding “No” (48%). Those who said they do include per-block settings or styles were then given an additional question covered in the next section.
Which blocks do you tend to customize with per block settings or styles in theme.json and why?
Note that this question was only shown if someone indicated that they have included per-block settings or styles in their theme.json files. The responses were fairly split with folks either saying they tend to customize all of them or tending to mention specific blocks. The Button block was heavily mentioned with 11 out of 16 people noting it. The following specific blocks were mentioned:
Because this was an open-ended question, here is a sampling of longer form responses to help get a sense of what folks shared:
For per-block settings, I have not done much outside of enabling an option or two if it was disabled by default. However, for per-block (and per-element) styles, I am not sold on the system. Writing CSS in JSON, which is essentially what we are talking about, feels wrong on so many levels. There is the obvious issue that it is limited to merely a few styles that are actually configurable, so anything beyond that requires diving into an actual CSS file anyway. And, that is problematic. Why would I want half my CSS code in a JSON file and the other half elsewhere? From a development standpoint, it makes the codebase harder to maintain. Initially, I started down the path of configuring per-block and element styles from `theme.json`. However, I have since moved my styling back to CSS files. It feels more natural, and I have the added benefit of all the tooling I am accustomed to. Right now, I cannot imagine a scenario where I would move back.
I do customize pretty much all of them, as the default block styles rarely fit. I also remove every and all default block variations, and a lot of the patterns. Overall I’m not a fan of Core having “opinionated” styles, because that should be the theme’s job.
Individually styling a lot of blocks leads to a lot of customization when done explicitly in raw css – and potentially unnecessary bloat. Most of my customizations are set on the most used core blocks for layouting / synchronizing design aspects.
I’ve mostly enabled border support for the Button block in the past (not sure if that is enabled by default now). Adding default spacing to various blocks is another thing I’ve done. But, the biggest use case has been settings some default styles for the root/body (colors, typography, and spacing) and for other elements rather than specifically individual blocks. I’m more interested in configuring per-block settings in the long-term than per-block styles though.
This open-ended question resulted in an awesome abundance of feedback, ideas, and more help request style questions! If you’re keen to read more, I’d recommend downloading the full results. For now, here are the major themes that relate to more specific ideas than more general feedback/help requests:
Once more, since this is an open ended question, here are some of the responses:
We plan to use theme.json to *disable* all low-level controls like color pickers, padding controls, and even custom font sizes. We want to introduce some system-level controls similar to the Global Styles but couldn’t wait until that part of the project is finished.
I have to think more – but definitely have thoughts on this. MultisiteMultisite Multisite is a WordPress feature which allows users to create a network of sites on a single WordPress installation. Available since WordPress version 3.0, Multisite is a continuation of WPMU or WordPress Multiuser project. WordPress MultiUser project was discontinued and its features were included into WordPress core.https://codex.wordpress.org/Create_A_Network. functionality with theme.json in particular is a quite powerful use case. An “import” function, similar to wordpress importer to pass settings from one theme to another or one install to another or main theme to child theme – and consequently an exporter, would be super fantastic.
The theme.json file can get super big and a bit hard to peruse. My current project’s theme.json is 350+ lines long. I’ve seen two solutions already surface to try and address this a bit: Justin Tadlock shared a webpack tool that merges multiple json files in to one, which is probably the most suitable for my needs. I’m just mentioning this as I think it’ll quickly become a common gripe for engineers that use theme.json heavily and especially in a team/agency setting.
I love, love, love using the json file! It made customizing so easy. I’m looking forward to working more with FSE and building more themes.
We need as many tools as possible in theme.json but an ability to use as few of them as we need.
This is the ninth call for testing as part of the Full Site Editing Outreach Program! For more information about this outreach program, please review this FAQ for helpful details. To properly join the fun, please head to #fse-outreach-experiment in Make Slack for future testing announcements, helpful posts, and more.
In comparison with previous calls for testing, this one is even more community driven with the suggestion to do a Higher Education themed call for testing coming from @blake. If you’d like to suggest an idea for a call for testing, know it’s very welcomed and all ideas will be weighed against current project priorities to figure out what makes the most sense to pursue. You can share ideas directly in the slackSlack Slack is a Collaborative Group Chat Platform https://slack.com/. The WordPress community has its own Slack Channel at https://make.wordpress.org/chat/. channel or via DM to me (@annezazu).
To ground this test in a real-world example, we’re going to go back to school as an administrator and recreate a customized headerHeader The header of your site is typically the first thing people will experience. The masthead or header art located across the top of your page is part of the look and feel of your website. It can influence a visitor’s opinion about your content and you/ your organization’s brand. It may also look different on different screen sizes. to welcome students, parents, and teachers alike to our hypothetical university. For inspiration, check out the following sample of university sites or just look up some near you! Since this test is focused on building out the header portion, focus in on that aspect and take note of what is done on each site:
https://www.ni.ac.rs/en/student-info
As you can imagine, this test is going to enable us to go deep into the Navigation BlockBlock Block is the abstract term used to describe units of markup that, composed together, form the content or layout of a webpage using the WordPress editor. The idea combines concepts of what in the past may have achieved with shortcodes, custom HTML, and embed discovery into a single consistent API and user experience.. As a refresher, it’s a powerful, new block that unlocks the ability to edit a site’s navigation menuNavigation Menu A theme feature introduced with Version 3.0. WordPress includes an easy to use mechanism for giving various control options to get users to click from one place to another on a site., both in terms of structure and design. To help prepare it for inclusion in a future WordPress release, this test is meant to explore the edges of what this block can do.
Similar to prior tests, if you choose to get super creative, please share a screenshot in your comment so we can celebrate what you’ve made. For inspiration, here’s my example below with the multiple layers of sub-menu items displayed:
While there’s more information below to ensure you get everything set up properly, here are the key aspects to have in place with your testing environment:
Generally speaking, please use the latest versions of each part of the setup and keep in mind that versions might have changed since this post was shared.
Known issues
While creating this call for testing, a few issues popped up that you, too, might experience as you go through this. Rest assured they have been reported. Here’s a nonexhaustive list of the most important items:
This section is for those who want to follow specific steps to create a header and might not have a lot of time to take the test further.
While this call for testing is focused on testing a specific feature, you’ll likely find other bugs in the process of testing with such a betaBeta A pre-release of software that is given out to a large group of users to trial under real conditions. Beta versions have gone through alpha testing in-house and are generally fairly close in look, feel and function to the final product; however, design changes often occur as part of the process. feature! Please know any bugs you find are welcome in your report for testing, even if they aren’t directly applicable to the tested feature.
Create structure (template part, columns, etc)
Build out site branding
Create a simple menu for high level items
Make a Donation to Donate to make it shorter by simply editing the text of that Page Link Block. Create a more complex menu for specifics
Save your work & customize further
This section is for those who have the time to take the test further and who are comfortable venturing into the site editor without much guidance.
The steps for this section are simple: find a university site’s header and try to recreate all or part of it. You’re welcome to continue to use TT1 Blocks or to use the block theme of your choosing (please note if you use a different theme). You can use the universities listed above or you can find your own. When leaving a comment, please share a screenshot of what you were attempting and a screenshot of what you were able to do. It’s very helpful to see what folks would like to be able to do so don’t hesitate to share different designs you see.
Remember to share a screenshot of what you created if you’re up for it!
Please leave feedback in the comments of this post. If you’d prefer, you’re always welcome to create issues in this GitHub repo directly for Gutenberg and in this GitHub repo for TT1 Blocks. If you leave feedback in GitHubGitHub GitHub is a website that offers online implementation of git repositories that can can easily be shared, copied and modified by other developers. Public repositories are free to host, private repositories require a paid subscription. GitHub introduced the concept of the ‘pull request’ where code changes done in branches by contributors can be reviewed and discussed before being merged be the repository owner. https://github.com/, please do still comment below with the link. If you see that someone else has already reported a problem, please still note your experience with it below, as it’ll help give those working on this experience more well-rounded insight into what to improve.
Test Team is meeting twice a week:
This schedule was discussed while ago, when the team wasn’t crystalized as it’s right now. This is why it’s a good time to ask members of this Team whether the time of meetings is okay, or shall we reschedule it.
Please post in the comments your opinion about that, we’ll be collecting votes until 30th of August and then make a decision.
Thank you for reading!
The meeting started on SlackSlack Slack is a Collaborative Group Chat Platform https://slack.com/. The WordPress community has its own Slack Channel at https://make.wordpress.org/chat/., here.
Announcement starts here.
@hellofromtonya mentioned that two testers were nominated for this position to work together, both of them accepted the nomination, @hellofromtonya @boniu91
Team had no objections to those nominations
@hellofromtonya published and shared previously prepared introductions of newly nominated Test Team Reps
The discussion starts here.
@francina shared her list, people present on the meeting agreed:
@lucatume added, that’s not clear for him:
@hellofromtonya replied to the second point, saying that:
@lucatume expanded the previous (first) point and for the most effective contribution, the following things should be clear:
@hellofromtonya answered, that tickets with needs-unit-tests keyword are the ones for the PHPUnit side of things. We don’t have anything for the e2e tests though.
@lucatume suggested adding needs-e2e-tests keyword and the Team agreed that’d be useful.
@francina mentioned, that the current test setup is Jest + Puppeteer. We need developers to set up a tool for the Team that will make creating e2e tests easy.
Team agreed that it’s not possible to have 100% of automated tests, with no human review.
@francina shared what the Team needs right now to kick off the e2e testing:
@hellofromtonya said, that most likely, the bottleneck will be the group of automation test engineers to do the review, tuning, and maintenance work.
@lucatume explained how the “autogenerating” of e2e tests looks like (here)
@hellofromtonya confirmed the workflow:
Mai mentioned, that generated code can be worse in terms of quality than the one written by a human. Team agreed that it’ll need to be refined and maintained. @hellofromtonya shared improved workflow:
Team agreed to start a pilot initiative:
@lucatume will start creating the prototype
@francina offered reviewing the test handbooks
Two candidates were nominated during the open nomination period, both of whom accepted their nominations. The new Test Team Reps for 2021 (and beyond) are Piotr Boniu (@boniu91) and Tonya Mork (@hellofromtonya)!
Piotr is fascinated by WordPress since 8 years. Firstly, it was just a fun for him, later it became a way to live. He attends WordCamps whenever possible since WCEU 2017 in Paris. In 2019 in Brighton he was a speaker.
Within the project, he served as the 5.9 Test Lead.
In his career, he’s transitioning into technical product management for a WordPress performance product company, after having previously served as a QA Engineer and Technical Support Engineer.
On the personal side, Piotr is a resident of Canary Islands, previously lived in Madrid and Warsaw. He resides in a small town called Puerto de la Cruz with a girlfriend and his two cats. He’s obsessed with technology and gadgets that are making life easier, his free time is filled with hiking, sports and swimming in the ocean.
You can read more about him on his profile page.
Tonya is a leader, architect, engineer (software and electrical), educator, and learner. She became hooked on contributing during the 5.6 release cycle. She saw the impact the Test Team can and could have on the project. She brings her empowerment and transformational approach to help the team rebuild itself, help fuel continuous improvement, and help contributors get started, gain and grow their skills, and grow their impact.
Within the project, she served as the 5.6 and 5.7 Triage Lead, coached the 5.8 triage and test release squad, is a Build/Test Tooling co-maintainer, and is a full-time contributor (as of February).
Outside of the project, she’s likely best known as a developer educator. She’s been leading and advising multi-disciplined engineering teams for over 3 decades in the areas of architecture, quality, testing, performance, and operations.
On the personal side, Tonya lives in a small fishing village on the NW coast of Lake Michigan in the US with her spouse (celebrated their 35th anniversary this year), dog (who thinks he’s a cat), and cat (who thinks he’s a dog and ruler of the house).
You can read more about her journey on HeroPress and her profile page.
Here is the agenda for the upcoming Test Team Chat scheduled for 2021-08-03 13:00.
This meeting is held in the #core-test channel in the Making WordPress slackSlack Slack is a Collaborative Group Chat Platform https://slack.com/. The WordPress community has its own Slack Channel at https://make.wordpress.org/chat/..
A list of last week’s posts and live streams.
Posts:
Live streams:
custom_logo and option site_logoDo you have something to propose for the agenda? Please leave a comment below.
Can’t make the meeting? Share anything relevant for the discussion in a comment below.
Props: @boniu91 for peer review.
You must be logged in to post a comment.