I believe we have agreed on the parts and can consider that step complete.

We must not forget about documentation, and choosing a format for the documentation.

For step 2, Associate the correct tools to the correct parts

Here are some tools or references:

• Trac / theme upload classes in the theme directory
• Theme review action (including unit tets, e2e tests, AXE accessibility tests with Jest and Puppeteer)
• Theme check (used by theme review action)
• PHPCS / WPCS / WPThemeReview rules
• Plugin Readme validator
• Accessibility auditing tools for WordPress themes.

Also:
https://wordpress.org/plugins/child-theme-check/

I have made the following changes:

Updated the text in Structure 4

Removed Structure 20, (child themes)
Removed safe 7-9, demo content, and moved the following items up the list.
Removed duplicate text for files required for block themes.

Added quality 19-21 (custom roles, mime types, user contact methods)

system3 -A user can only submit a complete theme
This is covered by checking that the required files are included,

• Names cannot be “reserved” for future use or to protect brands.
  Authors used to upload a blank theme and then close the Trac ticket themselves to make sure that no one else could use the theme name. I have not seen this happen for a long time, partly because most authors no longer have the Trac privilege to close tickets. This is more about author behavior and I don't think it can be checked automatically.

System2 -A user can submit unlimited updates for their existing themes that are in the theme directory.
This is already true, it is how Trac and uploads work right now. It is not a requirement but more of a way to answer a frequently asked question.

I updated the text in distribute 11 to clarify that this requirement is for copyright displayed on the front.
distribute11 | All Themes | must | only display the user’s copyright on the front end | not the theme author’s copyright.

For distribute9, the themes team goes further than what public domain requires, because, we need to be able to confirm that the image is in the public domain.
License information and source still needs to be included, but the copyright information can not be included because public domain works are not protected by copyright.

I updated the text to:
distribute9 | All Themes | must | declare license, copyright information, and source for all resources included such as fonts or images. | that is provided in a list of all resources in one file. | unless the assets are public domain, where copyright is not included.

I updated structure2 to say:
Structure2 | All Themes | must | include the main stylesheet style.css
Previous:
Structure2 | All Themes | Main stylesheet

I updated quality 8 from should to must.
quality8 | Block Themes | must | have completed block templates.

distribute 5 changed to:
distribute5 | All Themes | must | disclose all affiliates

system3 -A user can only submit a complete theme
This is covered by checking that the required files are included,

• Names cannot be “reserved” for future use or to protect brands.
  Authors used to upload a blank theme and then close the Trac ticket themselves to make sure that no one else could use the theme name. I have not seen this happen for a long time, partly because most authors no longer have the Trac privilege to close tickets. This is more about author behavior and I don't think it can be checked automatically.

Let's remove any items that aren't a test (or tests) itself. The structure checks will cover it.

[Edit]
Same with system4, the distribution checks should cover it.

@carolinan Apart from my last comment, is this the final list of checks? Have they all been validated and agreed upon?

I've moved Structure11, Structure12 to the distribution list. I left the ids as is.

[Edit] Moved Structure1, Structure10 to "safe".

We'll need to move to numberless & category-less ids in the next iteration to allow us flexibility as we storm this out.

Here are my thoughts on: Associate the correct tools to the correct parts.

System Checks

Tool: class-wporg-themes-upload.php

This should continue to run its checks, which I documented in #13. Its checks will have some overlap with other tools but it does so with respect to its own purposes which are specific to creating the entry on WordPress.org and filling in the trac ticket. We should not rely on any of these outcomes for the theme review.

Instead of returning each error 1 at a time, we should combine all the errors and output them so users can work on multiple problems at once.

If the theme passes these checks, the following checks should be triggered:

• Structure Checks
• Basic Distribution Checks

Any errors found by the tests above should trigger the same error messages and prevent the theme from being uploaded.

Structure Checks

Tool: Static Code Analysis( Theme-Check/Theme Sniffer/Other )

I'm not sure which tools make the most sense here, but I think the key is that we want to run a subset, priority-1 tests, that look specifically for missing files (and core content). We can circle back and validate more of the details of files with more specialized tools and tests in the Static Code and User Experience Review.

Basic Distribution Checks

Tool: Static Code Analysis( Theme-Check/Theme Sniffer/Other )

Here too, we should focus on priority-1 issues that look specifically for missing licensing directly related to the theme. We can cycle back and validate other licensing and imagery in the Static Code and User Experience Review.

Static Code and User Experience Review rules

Tool:

• Static Code Analysis ( Theme-Check/Theme Sniffer/Other )
• User Experience (Theme review action via jest)

Provided we made it through the priority-1 tests above, the rest of the suite can be divided between the User Experience and the Static Code toolset and triggered asynchronously.

Notes

I've include Theme-Check/Theme Sniffer/Other because I'm not certain which would be the best static code tool and I'm probably not the best resource to lead that discussion.

My gut tells me to keep things simple and use the theme-check for the priority-1 structure and distribution tests and then get into a bigger debate on what to use for the more complicated Static Code review.

What are others thinking? Any other ideas?

This match what I had in mind, thank you for putting words to it.

@carolinan Apart from my last comment, is this the final list of checks? Have they all been validated and agreed upon?

There is a request for feedback out this week, where we are requesting comments until July 26.

I am still hoping that we can find more ways to reduce the requirements.
For example, the screenshot size and ratio requirements could be a fix for the theme directory itself:
https://meta.trac.wordpress.org/ticket/5834

I am still hoping that we can find more ways to reduce the requirements.
For example, the screenshot size and ratio requirements could be a fix for the theme directory itself:
https://meta.trac.wordpress.org/ticket/5834

What about global14... can use any text, must be the same.

I don't know the history but... to me, it applies to a small subset of themes and I would think that other factors like the quality of designs and quality of the copy have a bigger impact on a theme's quality.

I am still hoping that we can find more ways to reduce the requirements.
For example, the screenshot size and ratio requirements could be a fix for the theme directory itself:
https://meta.trac.wordpress.org/ticket/5834

What about global14... can use any text, must be the same.

I don't know the history but... to me, it applies to a small subset of themes and I would think that other factors like the quality of designs and quality of the copy have a bigger impact on a theme's quality.

Like WordPress, all themes must be translatable. This is a pretty common problem, and it prevents translation.
If a theme is in both Japanese and English, and I only know English, then I can't translate it.

Structure tests, Priority 1

? Valid theme name (does not contain WordPress, Theme, Twenty Twenty-series) (system6)

Include style.css, index.php, readme.txt, screenshot (Structure2, Structure7)
Block themes also include theme.json, block-templates/index.html (structure18, structure19)

Basic Distribution Checks Priority 1

Confirm that the style.css and readme.txt headers contains a valid license and license URL. (distribute6)
Confirm that the theme includes copyright information for the theme itself. Unless the theme is in the public domain. (distribute8)

Removed:

|Structure6|All Themes|must|have tags in style.css that match what the theme actually does in respect to functionality and design|and doesn't use more than 3 subject tags (See: Theme Tag List).||