New this week: Launching sites on a multisite network, Platform usage metrics, and an audit log UI

Something for everyone in our three new VIP Dashboard features: a straightforward multisite site launch tool, an audit log of all platform activity, and filters for your usage metrics.

Easily launch new sites in a WordPress multisite

Our tool allows you to easily map a production-ready domain and perform search/replace throughout your site content, so you can launch a new site on your multisite network without leaving the VIP Dashboard. Read our documentation for more instructions to try this out on your next launch.

Map your production domain and ready your content for launch

Track usage over time

We want you to have all the information about your usage on our platform. You can now see the total requests for all applications, or for one application in particular. Alternatively, you can choose a particular month, or view the last 30 days on a rolling basis. Hovering over the chart gives you a snapshot of the request breakdown for API vs Application requests for that day. Our documentation covers how requests are calculated and more details about what you can see. View your organization usage in the VIP Dashboard by clicking your organization name and then choosing “Usage.”

Breakdown your platform usage by date range and by application

Track activity using our Audit Log UI

With our Platform Audit Log you can follow and track all application management activity in your VIP Cloud organization, helping to meet your compliance standards. Read our Audit Log documentation for full details. You’ll find your Audit Log by clicking the name of your organization in the VIP Dashboard, then choosing “Audit Log” from the left hand menu.

The Audit Log: all your organization activity in one place

New Release: Jetpack 10.2

Jetpack 10.2 has been deployed to all VIP sites.

What is being added or changed?

ENHANCEMENTS

  • Contact Form: add filter to allow customizing email headers.
  • Contact Form: add two filters to improve anti-spam functionality.
  • Related Posts: enable lazy loading for images.
  • Search: improve filter label formatting for longer text entries.
  • Search: title customization improvements.
  • SEO Tools: add “Archive Title” option for custom archive page titles.
  • SmartFrame Embeds: add support for SmartFrame embed using URLs, embed code, and shortcodes.
  • VideoPress: add new VideoPress product and enables VideoPress to be used for free for 1 video upload.
  • Widgets: add visibility settings to widgets using the block editor, under the advanced section of the sidebar.

IMPROVED COMPATIBILITY

  • External Media: update verbiage around Google Photos to improve compatibility with their UX guidelines.
  • Instagram Widget: remove from Legacy Widget block.
  • Publicize: replace native text-area with custom/core-based component.
  • Search: as IE11 support was removed in Jetpack 10.1, remove an IE11 polyfill.
  • VideoPress: file url field now displays the HLS playlist if one was generated. The fragmented mp4 file is not meant to be played on its own. MP4 file will only be displayed as the video url if it was created prior to the HLS update.

BUG FIXES

  • Contact Form: make sure the set height of the Spacer block nested inside a Contact Form block is respected.
  • Custom Content Type: improve the performance of the Testimonial CPT.
  • Dashboard: improve the performance of the query used to display Anti-spam stats.
  • Embeds: add allow-popups permission to Google Calendar embed so that links internal to the iFrame will open.
  • Publicize: ensure the custom excerpt is used when sharing to Facebook.
  • Publicize: improve refreshing connections list.
  • Related Posts: avoid errors if main div is not in the DOM.
  • Search: ensure Safari input clear button is hidden.
  • Search: ensure translations are available for lazy-loaded payloads.
  • Search: fix dashboard buttons too big on Safari.
  • Sharing: fix the icon position in icon-only button style.
  • Shortcodes: remove Hulu from available embeds.
  • Slideshow: fix fade transition background.
  • Theme Tools: remove unnecessary extra Social Menu added in the Twenty Twenty theme.
  • VideoPress: fix a bug related to deleting VideoPress videos.
  • Widgets: fix a conflict between various Jetpack blocks and some WordPress core legacy widgets.

For more details about this release (including specific changes), please see the release notes.

Questions?

If you have any questions, related to this release, please open a support ticket and we will be happy to assist.

Incident Report: September 30 Service Disruption to BUR Data Center

Executive Summary

On Sept. 30, 2021, between 21:17 and 21:45 UTC, several of VIP’s data centers intermittently stopped forwarding traffic to the origin data center in Los Angeles.  During this time, sites hosted in the Los Angeles data center experienced elevated error rates for requests originating from Africa, Europe, Eastern and Central USA.  Requests from Western USA and Asia-Pacific were not affected.  By 21:45 UTC the issue was resolved and error rates returned to normal levels.

Chronology of Events

Time (UTC)DateUpdate
21:1730 Sep. 2021 High error volumes detected by VIP monitoring systems. 
21:2230 Sep. 2021 Investigation begins into the cause of the errors.
21:3030 Sep. 2021 Problem identified as a request forwarding issue to Los Angeles.
21:3730 Sep. 2021 Root cause identified and configuration change rollback initiated.
21:4530 Sep. 2021 Configuration rollback complete and all sites recovered.

Business Impact

WordPress VIP applications with a Los Angeles origin experienced elevated error rates between 21:17 and 21:45 UTC on September 30, 2021 for uncached requests originating from Africa, Europe, Western and Central USA.  


Root Cause Analysis 

On September 20th, 2021 we deployed a configuration change that improved routing performance and reduced error rates.  This involved always routing traffic from the edge to the closest origin data center and then allowing that origin data center to forward the traffic to the correct origin data center for a given site.  We also configured multiple paths to each origin data center from each edge location. Here is an example for a request originating in New York to a site hosted in our Los Angeles data center:

Before:  

Client → New York edge → Los Angeles 

After:  

Client → New York edge → Ashburn origin → Los Angeles (option #1) 

Client → New York edge → Dallas origin → Los Angeles (option #2)

Client → New York edge →  Los Angeles (option #3)

Even though the “before” option seems more direct, it’s less desirable because it spends more time on 3rd party provider networks.  In the “before” case if there was a network problem between New York and Los Angeles, an error would likely be returned to the client. In the “After” case the time on the provider network is minimized and multiple backup paths are provided, allowing requests to be retried in the case of network problems.  

Unfortunately, ten days later, a traffic anomaly triggered a software bug introduced by this optimization which caused multiple origin locations to incorrectly think that the Los Angeles origin data center was unreachable and stop forwarding traffic.  When this happened, the edge health checks failed and returned 503 responses for some requests.

Remediation

The routing improvements were rolled back which restored traffic routing to the pre-September 20th state and prevented the bug from impacting live requests.


Preventative Actions

We have audited our configurations to ensure a similar problem does not exist elsewhere.  Additionally, we are investigating ways to reduce the time between when the errors started and when the investigation began.  In this case it was five minutes, but we would like to reduce it further. 

Incident Report: September 29 Service Disruption to DCA Data Center

On Sept. 29, 2021 at 17:05 UTC (1:05 PM EDT) a VIP origin data center in Ashburn, Virginia (DCA) experienced a rapid increase in ambient temperature. The temperature increase triggered thermal throttling on a subset of VIP’s server infrastructure which caused some sites to experience errors and intermittent slowness.

As temperatures returned to normal levels, VIP discovered a resource limit triggered by the rapid reallocation of resources away from the impacted infrastructure.  This issue prevented immediate recovery of the affected sites, but by 18:00 UTC (2:00 PM EDT) the issue was resolved and all affected sites were responding as expected.

Chronology of Events

Times are UTC.

  • 17:17 VIP is notified by internal teams that alerts indicate that temperatures are higher than normal at DCA.
  • 17:19 VIP notifies customers via the VIP Lobby and begins providing regular status updates.
  • 17:20 Servers begin thermal throttling of CPU.
  • 17:24 VIP customers with sites at DCA begin reporting site availability issues.
  • 17:32 Cooling restored and data center temperatures begin to decrease.
  • 17:34 Thermal throttling ends.
  • 17:41 VIP monitors recovery and discovers a etcd resource limit issue preventing full recovery
  • 17:59 VIP increases the limit, which resolves the issue.
  • 18:00 All sites have recovered.

Business Impact

WordPress VIP customer sites with DCA as their origin experienced intermittent slowness and reduced availability for approximately 40 minutes between 17:20 and 18:00 UTC (1:20 – 2:00 PM EDT) on September 29, 2021.

Root Cause Analysis

Why did this happen?

VIP leases data center space in Ashburn from a well-known vendor.  The vendor is responsible for providing the space, power, and cooling. Their investigation into the root cause of the temperature issue is ongoing, so we don’t have a RCA at this time. 

The problem that prevented immediate recovery once the thermal event had ended was caused by etcd exceeding the maximum configured database size.  Etcd is a distributed key value store used to track the state of VIP hosted sites.  During normal operation, the configured size was sufficient, but during the thermal event, thousands of sites were scheduled to move to unaffected servers simultaneously.  This spike in activity caused etcd to exceed its configured limit. 

Remediation

Immediate Fix

  • The data center temperature issue was addressed, ensuring that temperatures returned to normal.
  • The etcd quota was increased, which immediately allowed sites to resume starting.

Preventative Actions

Data Center

VIP is awaiting the official RCA from the data center, and looks forward to reviewing and discussing their plan to mitigate future risk.

Etcd Quota

The immediate action taken by VIP to increase the etcd quota should prevent a similar issue from occurring again. In addition, VIP is adding additional monitoring to ensure the configured limits can absorb large spikes in activity.

Call for Testing: Jetpack 10.2 Beta

The beta for Jetpack 10.2 is available now, and the download link is at the top of the beta release notice.

Jetpack 10.2 will be deployed to VIP Go on Thursday, October 7, 2021. The upgrade is expected to be performed at 15:00 UTC (11:00am EDT). This deployment date and time is subject to change if issues are discovered during testing of the Jetpack release.

What is being added or changed?


Widget Visibility

Widget visibility controls have been added for blocks when using the block-based widget editor. To test:

  • In Jetpack writing settings /wp-admin/admin.php?page=jetpack#/writing, make sure the Enable widget visibility controls to display widgets only on particular posts or pages setting under “Widgets” is enabled.
  • Then access the block-based widget editor, /wp-admin/widgets.php.
  • Add any block widget, a Paragraph block will work for example.
  • With that block selected, navigate to the block settings panel and open the Advanced options.
  • Under Visibility, try adding new visibility rules.
  • Visit the frontend of the site to verify your rules are working.

VideoPress

VideoPress is being added as a standalone Jetpack product. This is best tested on a Jetpack connected site without a paid Jetpack plan:

  • The VideoPress module can now be activated from the Modules page (even for sites without a paid plan): /wp-admin/admin.php?page=jetpack_modules
  • Once VideoPress is activated, on the Jetpack dashboard /wp-admin/admin.php?page=jetpack#/dashboard you will see a VideoPress card that will mention the status such as 1 free video available. Upgrade now to unlock more videos and 1TB of storage. You can toggle the VideoPress module on/off from this card.
  • If VideoPress is enabled on a site without a paid Jetpack plan, you are able to upload one free video to VideoPress.
  • For uploading videos to VideoPress there are two methods:
    1. From WordPress.com, go to My Site(s) → Media. Drag the video file from your computer into the media library, or click Add New and select the video file from your computer.
    2. From the WP Admin dashboard, go to Media → Library and drag the video file from your computer into the media library. Note: Clicking Media → Add New and selecting the video file in WP Admin will not upload the video to Jetpack. In order to upload video, make sure the grid view (Grid View) in media library is selected, and then you can drag the file into the WP Admin media library, or you can click Add New. Clicking Add New in the list view (List View) in media library will not upload the video to Jetpack.

Contact Form Custom Email Headers

There is a new filter available for customizing the email headers for Jetpack contact forms. For testing, try:

  • Add a new Jetpack contact form to a test page.
  • Add the following snippet to your site using a functionality plugin:
add_filter(
	'jetpack_contact_form_email_headers',
	function ( $headers, $comment_author, $reply_to_addr, $to ) {
		$headers .= 'Bcc: ' . $reply_to_addr . "\r\n";
		return $headers;
	},
	10,
	4
);
  • Make a test submission to the form you created.
  • An email should be sent to the email address specified in the added snippet.
  • Warning: DO NOT add headers or header data from the form submission without proper escaping and validation, or you’re liable to allow abusers to use your site to send spam. Especially DO NOT take email addresses from the form data to add as CC or BCC headers without strictly validating each address against a list of allowed addresses.

SEO Tools Archive Title

For custom Archive page titles a new Archive Title option replaces the Date option. To test:

  • At /wp-admin/admin.php?page=jetpack#/traffic make sure Customize your SEO settings is enabled.
  • Click on Expand to customize the page title structures of your site.
  • For Archives use the buttons to insert each token presented (Site Title, Tagline, and Archive Title).
  • The live preview should show “Example Archive Title/Date” for the inserted Archive Title
  • Save the SEO settings.
  • Check a custom post type archive page. For example you can enable Jetpack’s custom Portfolio type, then view that archive at example.com/portfolio/. In the <title> you should see “Projects” replacing the Archive Title" inserted via button.
  • Check a date archive (e.g. example.com/2019/09). You should see the appropriate date replacing the Archive Title inserted via button.

A full list of changes is available in the release notes and the commit log.

What do I need to do?

We recommend:

  1. Installing the beta release on your non-production sites using these instructions.
  2. Running through the testing flows outlined in the Jetpack Testing Guide.

As you’re testing a few things to keep in mind:

Questions?

If you have testing feedback or questions related to this release (or Jetpack in general), please open a support ticket and we will be happy to assist.

Resolved: Service Interruption in BUR Datacenter

22:30 UTC: From 21:17 to 21:45 sites hosted in our BUR datacenter may have seen an increase in server responses 500 or greater. Sites have now fully recovered. We will continue to monitor the situation and provide an update should anything change.


21:53 UTC: We are investigating a service disruption in our BUR datacenter. Our teams are currently working to identify the cause and will provide further updates as we are able. Our team will provide an update on the cause both here and on our Twitter account (@wpvipstatus) as soon as possible.

Parse.ly WordPress Plugin 2.6.0 Release

We are pleased to introduce the 2.6.0 release of the wp-parsely plugin. This release doesn’t introduce many user-facing changes, but it’s paving the way for next iterations of the plugin by cleaning up the internal code structure and a strong investment on automated testing.

Parse.ly gives creators, marketers and developers the tools to understand content performance, prove content value, and deliver tailored content experiences that drive meaningful results.

We encourage everyone using the plugin to update to the latest version.

If you have any questions about how WPVIP + Parse.ly can help you deliver meaningful customer experiences, please get in touch.

What is being added or changed?

Please, check the GitHub changelog.

Questions?

If you have any questions, related to this release, please open a support ticket and we will be happy to assist.

Resolved: Service Interruption in DCA Datacenter

17:19 UTC: We are investigating a service disruption in our DCA datacenter. Our teams are currently working to identify the cause and will provide further updates as we are able. Our team will provide an update on the cause both here and on our Twitter account (@wpvipstatus) as soon as possible.


18:01 UTC: Sites are operating normally and we continue to monitor. Our DCA datacenter experienced cooling issues that impacted requests, in some cases leading to timeouts or 503 errors. This is now resolved.


18:34 UTC: We continue to monitor the cooling situation at our DCA datacenter. Normal service has been restored for approximately 30 minutes. If you have any questions related to this incident, please open a support ticket and we will be happy to assist.

Parse.ly WordPress Plugin 2.5.2 Release

After the big 2.5 and the subsequent 2.5.1 releases, we are pleased to introduce a new minor release on the Parse.ly WordPress plugin, 2.5.2.

Parse.ly gives creators, marketers and developers the tools to understand content performance, prove content value, and deliver tailored content experiences that drive meaningful results.

We encourage everyone using the plugin to update to the latest version.

If you have any questions about how WPVIP + Parse.ly can help you deliver meaningful customer experiences, please get in touch.

What is being added or changed?

Changed

  • Specify that browserslist should use the defaults setting when building bundles.
  • Wrapping post list links in an opt-in filter.

Fixed

  • Fix notices that would appear if the plugin was set up to print repeating metas but those wouldn’t exist.
  • Fix cookie parsing. In some edge cases, a cookie that contained special characters would not be parsed correctly.

Questions?

If you have any questions, related to this release, please open a support ticket and we will be happy to assist.

New Release: Jetpack 10.1

Jetpack 10.1 has been deployed to all VIP sites.

What is being added or changed?

Enhancements

  • Carousel: photo info and comment sections stay open once toggled on.
  • Carousel: show image info icon even if Exif metadata option is off.
  • Carousel: use photo description as caption if caption and title fields are blank.
  • Form Block: scroll to subscription message after submitting the form.
  • Secure Sign On: remove additional redirect during sign in process.
  • Shortcodes: allow wrapping quizzes together with the Quiz shortcode.
  • Spotify Shortcode: add lazy loading support.
  • Stats: add the ability to collapse the nudges UI.
  • Tiled Gallery Block: ensure it can be used when in Offline Mode.
  • Widget Visibility: rename taxonomy label “All Pages” in dropdown.

Improved compatibility

  • Browser Compatibility: allow use of new JavaScript features not supported by Internet Explorer 11.
  • Mailchimp Block: updates terminology to use “Audience” instead of “Lists”.
  • Related Posts: allow shortcodes in Related Posts titles.

Bug fixes

  • Blocks: fix HTML markup of buttons added to blocks.
  • Carousel: fix a bug that changes the cursor to a pointer over all nested blocks.
  • Carousel: fix bug preventing user from swiping in gallery when images are configured to link to Attachment Page.
  • Carousel: hide UI controls in the carousel for single images.
  • Contact Form: disable the Block Editor for contact form submissions.
  • Custom CSS: ensure that short CSS is added inline instead of via an external file.
  • Dashboard: add a new message to the backup card if the site is too new to have an accurate status.
  • Facebook Page Widget: fix a bug that prevented widget from being added in the wp-admin widget editor.
  • Form Block: fix form button alignment in editor.
  • Slideshow Block: fix intermittent issue where next/prev buttons sometimes don’t respond to clicks.
  • Social Icons: fix being unable to remove icons from widget.
  • Social Icons Widget: fix issue in wp-admin widget editor where icons don’t save on refresh.
  • Stats: allow better translations of page view counts in the Stats widget.
  • Video Block: fix resizing so that video block doesn’t display black bars on the sides.

For more details about this release (including specific changes), please see the release notes.

Questions?

If you have any questions, related to this release, please open a support ticket and we will be happy to assist.