Write custom WP-CLI commands at scale
On the VIP Platform, WP-CLI commands run in a container that is separate from a site’s web application, but it shares the Memcache and database containers. It is very important to set protections in place to prevent custom WP-CLI commands from inadvertently affecting your site’s performance.
Avoid memory exhaustion
The WPCOM_VIP_CLI_Command class, included in VIP Go MU-plugins, provides helper functions for custom WP-CLI commands that need to run over large datasets. The helper functions can be used by extending this class.
If a large amount of data is being processed on a launched site, make sure to prepare the custom WP-CLI command for processing without exhausting memory and overloading the database by using these helper functions.
WPCOM_VIP_CLI_Command::stop_the_insanity()resets the in-memory local WordPress object cache, from the global$wp_object_cache, without affecting memcache, and resets the in-memory database query log.- Consider using this to clear memory after having processed 50-100 posts to avoid interruptions, especially when using
get_posts()orWP_Query.
- Consider using this to clear memory after having processed 50-100 posts to avoid interruptions, especially when using
WPCOM_VIP_CLI_Command::start_bulk_operation()defers term counting, so that as individual updates are made to the dataset, expensive term counting is not being triggered each update.- This is important when the command is issuing many writes or changes to the database.
WPCOM_VIP_CLI_Command::end_bulk_operation()restores and triggers term counting. This should be used as a companion tostart_bulk_operation()where the pair of functions bookends the command.
- Use sleep() in between updating records or batches of records to help with loads associated with cache re-validation and data replication.
Paginate results and process in batches
- Using a no-LIMIT query can lead to timeout and failure, especially if it takes longer than 30 seconds. Instead, we recommend using smaller queries and paging through the results:
class Test_CLI_Command extends WPCOM_VIP_CLI_Command {
/**
* Publishes all pending posts once they have had their metakeys updated.
*
* Takes a metakey (required) and post category (optional).
*
* @subcommand update-metakey
* @synopsis --meta-key=<meta-key> [--category=<category>] [--dry-run]
*/
public function update_metakey( $args, $assoc_args ) {
// Disable term counting, Elasticsearch indexing, and PushPress.
$this->start_bulk_operation();
$posts_per_page = 100;
$paged = 1;
$count = 0;
// Meta key is required, otherwise an error will be returned.
if ( isset( $assoc_args['meta-key'] ) ) {
$meta_key = $assoc_args['meta-key'];
} else {
// Caution: calling WP_CLI::error stops the execution of the command. Use it only in case you want to stop the execution. Otherwise, use WP_CLI::warning or WP_CLI::line for non-blocking errors.
WP_CLI::error( 'Must have --meta-key attached.' );
}
// Category value is optional.
if ( isset( $assoc_args['category'] ) ) {
$cat = $assoc_args['category'];
} else {
$cat = '';
}
// If --dry-run is not set, then it will default to true. Must set --dry-run explicitly to false to run this command.
if ( isset( $assoc_args['dry-run'] ) ) {
// Passing `--dry-run=false` to the command leads to the `false` value being set to string `'false'`, but casting `'false'` to bool produces `true`. Thus the special handling.
if ( 'false' === $assoc_args['dry-run'] ) {
$dry_run = false;
} else {
$dry_run = (bool) $assoc_args['dry-run'];
}
} else {
$dry_run = true;
}
if ( $dry_run ) {
WP_CLI::line( 'Running in dry-run mode.' );
} else {
WP_CLI::line( 'We\'re doing it live!' );
}
do {
$posts = get_posts(
array(
'posts_per_page' => $posts_per_page,
'paged' => $paged,
'category' => $cat,
'post_status' => 'pending',
'suppress_filters' => 'false',
)
);
foreach ( $posts as $post ) {
if ( ! $dry_run ) {
update_post_meta( $post->ID, $meta_key, 'true' );
wp_update_post( array( 'post_status' => 'publish' ) );
}
$count++;
}
// Pause.
WP_CLI::line( 'Pausing for a breath...' );
sleep( 3 );
// Free up memory.
$this->stop_the_insanity();
/* At this point, we have to decide whether to increase the value of $paged. In case a value which is being used for querying the posts (like post_status in our example) is being changed via the command, we should keep the WP_Query starting from the beginning in every iteration.
* If the any value used for querying the posts is not being changed, then we need to update the value in order to walk through all the posts. */
// $paged++;
} while ( count( $posts ) );
if ( false === $dry_run ) {
WP_CLI::success( sprintf( '%d posts have successfully been published and had their metakeys updated.', $count ) );
} else {
WP_CLI::success( sprintf( '%d posts will be published and have their metakeys updated.', $count ) );
}
// Trigger a term count as well as trigger bulk indexing of Elasticsearch site.
$this->end_bulk_operation();
}
/**
* Updates terms in that taxonomy by removing the "test-" prefix.
*
* Takes a taxonomy (required).
*
* @subcommand update-terms
* @synopsis --taxonomy=<taxonomy> [--dry_run]
*/
public function update_terms( $args, $assoc_args ) {
$count = 0;
// Disable term counting, Elasticsearch indexing, and PushPress.
$this->start_bulk_operation();
// Taxonomy value is required, otherwise an error will be returned.
if ( isset( $assoc_args['taxonomy'] ) ) {
$taxonomy = $assoc_args['taxonomy'];
} else {
WP_CLI::error( 'Must have a --taxonomy attached.' );
}
if ( isset( $assoc_args['dry-run'] ) ) {
if ( 'false' === $assoc_args['dry-run'] ) {
$dry_run = false;
} else {
$dry_run = (bool) $assoc_args['dry-run'];
}
} else {
$dry_run = true;
}
if ( $dry_run ) {
WP_CLI::line( 'Running in dry-run mode.' );
} else {
WP_CLI::line( 'We\'re doing it live!' );
}
$terms = get_terms( array( 'taxonomy' => $taxonomy ) );
foreach ( $terms as $term ) {
if ( ! $dry_run ) {
$args = array(
'name' => str_replace( 'test ', '', $term->name ),
'slug' => str_replace( 'test-', '', $term->slug ),
);
wp_update_term( $term->term_id, $term->taxonomy, $args );
}
$count++;
}
// Trigger a term count as well as trigger bulk indexing of Elasticsearch site.
$this->end_bulk_operation();
if ( false === $dry_run ) {
WP_CLI::success( sprintf( '%d terms were updated.', $count ) );
} else {
WP_CLI::success( sprintf( '%d terms will be updated.', $count ) );
}
}
}
WP_CLI::add_command( 'test-command', 'Test_CLI_Command' );Design for restarts
There are many reasons a command may stop mid-execution. Commands dealing with a lot of posts or other long-running actions should be prepared for restart. You might either design them to be idempotent (meaning they can safely be run multiple times) or provide an option to start from a certain point by using offset and limit arguments.
Save command output to a file
When running long-running custom WP-CLI commands, or commands that will affect many records, it is recommended to save the output to a file for review. Whether it is a natively supported WP-CLI command or a custom command, once the output for a command is more than 100 lines of information, and potentially impacting dozens or more records in your database, having a log to audit what changes occurred, and what succeed or failed, can become a critical step in debugging a root cause for any unexpected side effects.
Using tee to view and save output
A common option is to pipe (|) output to tee so that you can log responses while also being able to view them in the terminal shell to monitor progress. By design, this will only write the STDOUT, but the output from WP_CLI::warning(), WP_CLI::debug(), WP_CLI::error(), and WP_CLI::error_multi_line() all write to STDERR, which will display on the screen but will not be recorded into the log file.
To ensure that complete output from the command is logged, both STDERR and STDOUT should be redirected to the same place using 2>&1 . This will ensure that all of the output seen in the terminal shell will be recorded to the log.
Write STDOUT to a file, and show both STDOUT and STDERR in the terminal window
# Post ID 64791 does not exist, which will trigger a WP_CLI::warning(). $ wp demo 64783 64791 65384 | tee output.log 3 args provided. 1/3 Post found "New Product Hero Image" (ID 65384). Warning: Post not found (ID 64791). 3/3 Post found "Lab Testing Samples" (ID 64783). Success: Completed command. $ cat output.log 3 args provided. 1/3 Post found "New Product Hero Image" (ID 65384). 3/3 Post found "Lab Testing Samples" (ID 64783). Success: Completed command.
Write STDOUT and STDERR to a file, and show both STDOUT and STDERR in the terminal window
# Post ID 64791 does not exist, which will trigger a WP_CLI::warning(). $ wp demo 64783 64791 65384 2>&1 | tee all-output.log 3 args provided. 1/3 Post found "New Product Hero Image" (ID 65384). Warning: Post not found (ID 64791). 3/3 Post found "Lab Testing Samples" (ID 64783). Success: Completed command. $ cat all-output.log 3 args provided. 1/3 Post found "New Product Hero Image" (ID 65384). Warning: Post not found (ID 64791). 3/3 Post found "Lab Testing Samples" (ID 64783). Success: Completed command.
Using > to only write to a log file
If an output display in the terminal is not required while the command is running, the redirection operator (>) can be used. By default, the redirection operator will write all STDOUT to the file and output the STDERR to the terminal shell.
Alternatively the STDERR and STDOUT can be combined to the same file using 2>&1 or redirect each output to its own file.
Write STDOUT to file, and show STDERR in the terminal window
# Post ID 64791 does not exist, which will trigger a WP_CLI::warning(). $ wp demo 64783 64791 65384 > output.log Warning: Post not found (ID 64791). $ cat output.log 3 args provided. 1/3 Post found "New Product Hero Image" (ID 65384). 3/3 Post found "Lab Testing Samples" (ID 64783). Success: Completed command.
Write STDOUT to one file, STDERR to another file, with nothing showing in the terminal window
# Post ID 64791 does not exist, which will trigger a WP_CLI::warning(). $ wp demo 64783 64791 65384 > output.log 2>error.log $ cat output.log 3 args provided. 1/3 Post found "New Product Hero Image" (ID 65384). 3/3 Post found "Lab Testing Samples" (ID 64783). Success: Completed command. $ cat error.log Warning: Post not found (ID 64791).
Write both STDOUT and STDERR to a file, with nothing showing in the terminal window
# Post ID 64791 does not exist, which will trigger a WP_CLI::warning(). $ wp demo 64783 64791 65384 > output.log 2>&1 $ cat output.log 3 args provided. 1/3 Post found "New Product Hero Image" (ID 65384). Warning: Post not found (ID 64791). 3/3 Post found "Lab Testing Samples" (ID 64783). Success: Completed command.