Skip to content

/ gutenberg

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix e2e test failure screenshots not capturing at the right time #28449

Draft
wants to merge 2 commits into
base: master
from
Draft

Fix e2e test failure screenshots not capturing at the right time #28449

wants to merge 2 commits into from
+334 −55

Conversation

@kevin940726
Copy link
Member

@kevin940726 kevin940726 commented Jan 24, 2021

Description

Continued from #26664. Fix the screenshots not being captured at the right time.

Specifically, if the tests have afterEach or afterAll defined, the screenshots will be captured after these hooks are run. This PR uses jest-circus as the test runner and extends jest-environment-puppeteer to run the capturing scripts in the test_fn_failure event.

How has this been tested?

Added some intentionally failing tests to demo the result. See the failing tests for the stored artifacts. The screenshots captured should correctly point to when it failed.

npm run test-e2e -- packages/e2e-tests/specs/widgets/adding-widgets.test.js

Types of changes

Bug fix

Checklist:

  • My code is tested.
  • My code follows the WordPress code style.
  • My code follows the accessibility standards.
  • My code has proper inline documentation.
  • I've included developer documentation if appropriate.
  • I've updated all React Native files affected by any refactorings/renamings in this PR.
kevin940726 added 2 commits Jan 24, 2021
@kevin940726
Fix e2e test failure screenshots not capturing at the right time
bc78e3a
@kevin940726
Add intentonally failing tests to demo
Loading status checks…
cedeac5
@github-actions
Copy link

@github-actions github-actions bot commented Jan 24, 2021

Size Change: 0 B

Total Size: 1.39 MB

ℹ️ View Unchanged
Filename Size Change
build/a11y/index.js 1.14 kB 0 B
build/annotations/index.js 3.77 kB 0 B
build/api-fetch/index.js 3.4 kB 0 B
build/autop/index.js 2.84 kB 0 B
build/blob/index.js 665 B 0 B
build/block-directory/index.js 9.08 kB 0 B
build/block-directory/style-rtl.css 1.01 kB 0 B
build/block-directory/style.css 1.01 kB 0 B
build/block-editor/index.js 123 kB 0 B
build/block-editor/style-rtl.css 11.9 kB 0 B
build/block-editor/style.css 11.9 kB 0 B
build/block-library/blocks/archives/editor-rtl.css 196 B 0 B
build/block-library/blocks/archives/editor.css 196 B 0 B
build/block-library/blocks/audio/editor-rtl.css 194 B 0 B
build/block-library/blocks/audio/editor.css 194 B 0 B
build/block-library/blocks/audio/style-rtl.css 225 B 0 B
build/block-library/blocks/audio/style.css 225 B 0 B
build/block-library/blocks/block/editor-rtl.css 283 B 0 B
build/block-library/blocks/block/editor.css 283 B 0 B
build/block-library/blocks/button/editor-rtl.css 576 B 0 B
build/block-library/blocks/button/editor.css 577 B 0 B
build/block-library/blocks/button/style-rtl.css 552 B 0 B
build/block-library/blocks/button/style.css 552 B 0 B
build/block-library/blocks/buttons/editor-rtl.css 345 B 0 B
build/block-library/blocks/buttons/editor.css 346 B 0 B
build/block-library/blocks/buttons/style-rtl.css 419 B 0 B
build/block-library/blocks/buttons/style.css 419 B 0 B
build/block-library/blocks/calendar/style-rtl.css 319 B 0 B
build/block-library/blocks/calendar/style.css 319 B 0 B
build/block-library/blocks/categories/editor-rtl.css 210 B 0 B
build/block-library/blocks/categories/editor.css 209 B 0 B
build/block-library/blocks/categories/style-rtl.css 208 B 0 B
build/block-library/blocks/categories/style.css 208 B 0 B
build/block-library/blocks/code/style-rtl.css 216 B 0 B
build/block-library/blocks/code/style.css 216 B 0 B
build/block-library/blocks/columns/editor-rtl.css 300 B 0 B
build/block-library/blocks/columns/editor.css 299 B 0 B
build/block-library/blocks/columns/style-rtl.css 529 B 0 B
build/block-library/blocks/columns/style.css 528 B 0 B
build/block-library/blocks/cover/editor-rtl.css 505 B 0 B
build/block-library/blocks/cover/editor.css 503 B 0 B
build/block-library/blocks/cover/style-rtl.css 1.35 kB 0 B
build/block-library/blocks/cover/style.css 1.35 kB 0 B
build/block-library/blocks/embed/editor-rtl.css 594 B 0 B
build/block-library/blocks/embed/editor.css 595 B 0 B
build/block-library/blocks/embed/style-rtl.css 489 B 0 B
build/block-library/blocks/embed/style.css 489 B 0 B
build/block-library/blocks/file/editor-rtl.css 314 B 0 B
build/block-library/blocks/file/editor.css 313 B 0 B
build/block-library/blocks/file/style-rtl.css 352 B 0 B
build/block-library/blocks/file/style.css 352 B 0 B
build/block-library/blocks/freeform/editor-rtl.css 2.55 kB 0 B
build/block-library/blocks/freeform/editor.css 2.55 kB 0 B
build/block-library/blocks/gallery/editor-rtl.css 783 B 0 B
build/block-library/blocks/gallery/editor.css 783 B 0 B
build/block-library/blocks/gallery/style-rtl.css 1.17 kB 0 B
build/block-library/blocks/gallery/style.css 1.17 kB 0 B
build/block-library/blocks/group/editor-rtl.css 433 B 0 B
build/block-library/blocks/group/editor.css 432 B 0 B
build/block-library/blocks/group/style-rtl.css 190 B 0 B
build/block-library/blocks/group/style.css 190 B 0 B
build/block-library/blocks/heading/editor-rtl.css 248 B 0 B
build/block-library/blocks/heading/editor.css 248 B 0 B
build/block-library/blocks/heading/style-rtl.css 212 B 0 B
build/block-library/blocks/heading/style.css 212 B 0 B
build/block-library/blocks/html/editor-rtl.css 384 B 0 B
build/block-library/blocks/html/editor.css 385 B 0 B
build/block-library/blocks/image/editor-rtl.css 801 B 0 B
build/block-library/blocks/image/editor.css 800 B 0 B
build/block-library/blocks/image/style-rtl.css 569 B 0 B
build/block-library/blocks/image/style.css 570 B 0 B
build/block-library/blocks/latest-comments/editor-rtl.css 277 B 0 B
build/block-library/blocks/latest-comments/editor.css 275 B 0 B
build/block-library/blocks/latest-comments/style-rtl.css 382 B 0 B
build/block-library/blocks/latest-comments/style.css 382 B 0 B
build/block-library/blocks/latest-posts/editor-rtl.css 254 B 0 B
build/block-library/blocks/latest-posts/editor.css 254 B 0 B
build/block-library/blocks/latest-posts/style-rtl.css 634 B 0 B
build/block-library/blocks/latest-posts/style.css 634 B 0 B
build/block-library/blocks/list/editor-rtl.css 203 B 0 B
build/block-library/blocks/list/editor.css 203 B 0 B
build/block-library/blocks/list/style-rtl.css 201 B 0 B
build/block-library/blocks/list/style.css 201 B 0 B
build/block-library/blocks/media-text/editor-rtl.css 311 B 0 B
build/block-library/blocks/media-text/editor.css 311 B 0 B
build/block-library/blocks/media-text/style-rtl.css 642 B 0 B
build/block-library/blocks/media-text/style.css 640 B 0 B
build/block-library/blocks/more/editor-rtl.css 545 B 0 B
build/block-library/blocks/more/editor.css 545 B 0 B
build/block-library/blocks/navigation-link/editor-rtl.css 503 B 0 B
build/block-library/blocks/navigation-link/editor.css 504 B 0 B
build/block-library/blocks/navigation-link/style-rtl.css 805 B 0 B
build/block-library/blocks/navigation-link/style.css 803 B 0 B
build/block-library/blocks/navigation/editor-rtl.css 1.49 kB 0 B
build/block-library/blocks/navigation/editor.css 1.48 kB 0 B
build/block-library/blocks/navigation/style-rtl.css 289 B 0 B
build/block-library/blocks/navigation/style.css 289 B 0 B
build/block-library/blocks/nextpage/editor-rtl.css 507 B 0 B
build/block-library/blocks/nextpage/editor.css 507 B 0 B
build/block-library/blocks/paragraph/editor-rtl.css 236 B 0 B
build/block-library/blocks/paragraph/editor.css 236 B 0 B
build/block-library/blocks/paragraph/style-rtl.css 392 B 0 B
build/block-library/blocks/paragraph/style.css 392 B 0 B
build/block-library/blocks/post-author/editor-rtl.css 329 B 0 B
build/block-library/blocks/post-author/editor.css 329 B 0 B
build/block-library/blocks/post-author/style-rtl.css 303 B 0 B
build/block-library/blocks/post-author/style.css 303 B 0 B
build/block-library/blocks/post-comments-form/style-rtl.css 358 B 0 B
build/block-library/blocks/post-comments-form/style.css 358 B 0 B
build/block-library/blocks/post-content/editor-rtl.css 262 B 0 B
build/block-library/blocks/post-content/editor.css 262 B 0 B
build/block-library/blocks/post-excerpt/editor-rtl.css 206 B 0 B
build/block-library/blocks/post-excerpt/editor.css 206 B 0 B
build/block-library/blocks/post-featured-image/editor-rtl.css 453 B 0 B
build/block-library/blocks/post-featured-image/editor.css 453 B 0 B
build/block-library/blocks/post-featured-image/style-rtl.css 223 B 0 B
build/block-library/blocks/post-featured-image/style.css 223 B 0 B
build/block-library/blocks/preformatted/style-rtl.css 193 B 0 B
build/block-library/blocks/preformatted/style.css 193 B 0 B
build/block-library/blocks/pullquote/editor-rtl.css 304 B 0 B
build/block-library/blocks/pullquote/editor.css 304 B 0 B
build/block-library/blocks/pullquote/style-rtl.css 428 B 0 B
build/block-library/blocks/pullquote/style.css 428 B 0 B
build/block-library/blocks/query-loop/editor-rtl.css 217 B 0 B
build/block-library/blocks/query-loop/editor.css 216 B 0 B
build/block-library/blocks/query-loop/style-rtl.css 427 B 0 B
build/block-library/blocks/query-loop/style.css 429 B 0 B
build/block-library/blocks/query-pagination-numbers/editor-rtl.css 243 B 0 B
build/block-library/blocks/query-pagination-numbers/editor.css 240 B 0 B
build/block-library/blocks/query-pagination/editor-rtl.css 390 B 0 B
build/block-library/blocks/query-pagination/editor.css 379 B 0 B
build/block-library/blocks/query-pagination/style-rtl.css 288 B 0 B
build/block-library/blocks/query-pagination/style.css 288 B 0 B
build/block-library/blocks/query/editor-rtl.css 279 B 0 B
build/block-library/blocks/query/editor.css 279 B 0 B
build/block-library/blocks/quote/editor-rtl.css 195 B 0 B
build/block-library/blocks/quote/editor.css 195 B 0 B
build/block-library/blocks/quote/style-rtl.css 284 B 0 B
build/block-library/blocks/quote/style.css 285 B 0 B
build/block-library/blocks/rss/editor-rtl.css 307 B 0 B
build/block-library/blocks/rss/editor.css 309 B 0 B
build/block-library/blocks/rss/style-rtl.css 394 B 0 B
build/block-library/blocks/rss/style.css 393 B 0 B
build/block-library/blocks/search/editor-rtl.css 285 B 0 B
build/block-library/blocks/search/editor.css 285 B 0 B
build/block-library/blocks/search/style-rtl.css 454 B 0 B
build/block-library/blocks/search/style.css 456 B 0 B
build/block-library/blocks/separator/editor-rtl.css 229 B 0 B
build/block-library/blocks/separator/editor.css 229 B 0 B
build/block-library/blocks/separator/style-rtl.css 352 B 0 B
build/block-library/blocks/separator/style.css 352 B 0 B
build/block-library/blocks/shortcode/editor-rtl.css 603 B 0 B
build/block-library/blocks/shortcode/editor.css 603 B 0 B
build/block-library/blocks/site-logo/editor-rtl.css 321 B 0 B
build/block-library/blocks/site-logo/editor.css 321 B 0 B
build/block-library/blocks/site-logo/style-rtl.css 238 B 0 B
build/block-library/blocks/site-logo/style.css 238 B 0 B
build/block-library/blocks/social-link/editor-rtl.css 283 B 0 B
build/block-library/blocks/social-link/editor.css 283 B 0 B
build/block-library/blocks/social-links/editor-rtl.css 811 B 0 B
build/block-library/blocks/social-links/editor.css 810 B 0 B
build/block-library/blocks/social-links/style-rtl.css 1.48 kB 0 B
build/block-library/blocks/social-links/style.css 1.48 kB 0 B
build/block-library/blocks/spacer/editor-rtl.css 416 B 0 B
build/block-library/blocks/spacer/editor.css 416 B 0 B
build/block-library/blocks/spacer/style-rtl.css 184 B 0 B
build/block-library/blocks/spacer/style.css 184 B 0 B
build/block-library/blocks/subhead/editor-rtl.css 223 B 0 B
build/block-library/blocks/subhead/editor.css 223 B 0 B
build/block-library/blocks/subhead/style-rtl.css 210 B 0 B
build/block-library/blocks/subhead/style.css 210 B 0 B
build/block-library/blocks/table/editor-rtl.css 593 B 0 B
build/block-library/blocks/table/editor.css 593 B 0 B
build/block-library/blocks/table/style-rtl.css 501 B 0 B
build/block-library/blocks/table/style.css 501 B 0 B
build/block-library/blocks/tag-cloud/editor-rtl.css 237 B 0 B
build/block-library/blocks/tag-cloud/editor.css 235 B 0 B
build/block-library/blocks/tag-cloud/style-rtl.css 221 B 0 B
build/block-library/blocks/tag-cloud/style.css 221 B 0 B
build/block-library/blocks/template-part/editor-rtl.css 794 B 0 B
build/block-library/blocks/template-part/editor.css 794 B 0 B
build/block-library/blocks/text-columns/editor-rtl.css 220 B 0 B
build/block-library/blocks/text-columns/editor.css 220 B 0 B
build/block-library/blocks/text-columns/style-rtl.css 283 B 0 B
build/block-library/blocks/text-columns/style.css 283 B 0 B
build/block-library/blocks/verse/editor-rtl.css 194 B 0 B
build/block-library/blocks/verse/editor.css 194 B 0 B
build/block-library/blocks/verse/style-rtl.css 215 B 0 B
build/block-library/blocks/verse/style.css 215 B 0 B
build/block-library/blocks/video/editor-rtl.css 617 B 0 B
build/block-library/blocks/video/editor.css 617 B 0 B
build/block-library/blocks/video/style-rtl.css 303 B 0 B
build/block-library/blocks/video/style.css 304 B 0 B
build/block-library/common-rtl.css 1.01 kB 0 B
build/block-library/common.css 1.01 kB 0 B
build/block-library/editor-rtl.css 9.19 kB 0 B
build/block-library/editor.css 9.18 kB 0 B
build/block-library/index.js 143 kB 0 B
build/block-library/style-rtl.css 8.69 kB 0 B
build/block-library/style.css 8.7 kB 0 B
build/block-library/theme-rtl.css 860 B 0 B
build/block-library/theme.css 860 B 0 B
build/block-serialization-default-parser/index.js 1.88 kB 0 B
build/block-serialization-spec-parser/index.js 3.06 kB 0 B
build/blocks/index.js 48.3 kB 0 B
build/components/index.js 274 kB 0 B
build/components/style-rtl.css 15.5 kB 0 B
build/components/style.css 15.5 kB 0 B
build/compose/index.js 11.2 kB 0 B
build/core-data/index.js 15.2 kB 0 B
build/data-controls/index.js 830 B 0 B
build/data/index.js 8.97 kB 0 B
build/date/index.js 31.8 kB 0 B
build/deprecated/index.js 769 B 0 B
build/dom-ready/index.js 571 B 0 B
build/dom/index.js 4.93 kB 0 B
build/edit-navigation/index.js 11.1 kB 0 B
build/edit-navigation/style-rtl.css 938 B 0 B
build/edit-navigation/style.css 944 B 0 B
build/edit-post/index.js 306 kB 0 B
build/edit-post/style-rtl.css 6.51 kB 0 B
build/edit-post/style.css 6.5 kB 0 B
build/edit-site/index.js 24 kB 0 B
build/edit-site/style-rtl.css 4.01 kB 0 B
build/edit-site/style.css 4.01 kB 0 B
build/edit-widgets/index.js 23.6 kB 0 B
build/edit-widgets/style-rtl.css 3.17 kB 0 B
build/edit-widgets/style.css 3.18 kB 0 B
build/editor/editor-styles-rtl.css 543 B 0 B
build/editor/editor-styles.css 545 B 0 B
build/editor/index.js 41.8 kB 0 B
build/editor/style-rtl.css 3.89 kB 0 B
build/editor/style.css 3.89 kB 0 B
build/element/index.js 4.62 kB 0 B
build/escape-html/index.js 735 B 0 B
build/format-library/index.js 6.77 kB 0 B
build/format-library/style-rtl.css 637 B 0 B
build/format-library/style.css 639 B 0 B
build/hooks/index.js 2.27 kB 0 B
build/html-entities/index.js 623 B 0 B
build/i18n/index.js 3.56 kB 0 B
build/is-shallow-equal/index.js 699 B 0 B
build/keyboard-shortcuts/index.js 2.53 kB 0 B
build/keycodes/index.js 1.93 kB 0 B
build/list-reusable-blocks/index.js 3.14 kB 0 B
build/list-reusable-blocks/style-rtl.css 629 B 0 B
build/list-reusable-blocks/style.css 628 B 0 B
build/media-utils/index.js 5.32 kB 0 B
build/notices/index.js 1.85 kB 0 B
build/nux/index.js 3.41 kB 0 B
build/nux/style-rtl.css 731 B 0 B
build/nux/style.css 727 B 0 B
build/plugins/index.js 2.54 kB 0 B
build/primitives/index.js 1.42 kB 0 B
build/priority-queue/index.js 789 B 0 B
build/redux-routine/index.js 2.84 kB 0 B
build/reusable-blocks/index.js 2.92 kB 0 B
build/rich-text/index.js 13.4 kB 0 B
build/server-side-render/index.js 2.76 kB 0 B
build/shortcode/index.js 1.7 kB 0 B
build/token-list/index.js 1.27 kB 0 B
build/url/index.js 3.02 kB 0 B
build/viewport/index.js 1.86 kB 0 B
build/warning/index.js 1.14 kB 0 B
build/wordcount/index.js 1.22 kB 0 B

compressed-size-action
@kevin940726
Copy link
Member Author

@kevin940726 kevin940726 commented Jan 25, 2021

Somehow when using jest-circus, there will be an unexpected error when running multi-entity-editing.test.js.

Test suite failed to run

Expected an Error, but "" was thrown

I've narrowed it down to this following call:

await createNewPost({
  postType: "wp_template",
  title: templateName
});

Specifically the wp_template post type.

I have no idea why it's broken, and I don't have time to investigate it further now, might come back to it later.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
Automated Testing [Type] Bug
Projects
None yet
Milestone
No milestone
Linked issues

Successfully merging this pull request may close these issues.

None yet

1 participant
@kevin940726