Robots API

[kraevit]

Hello,
My name is Krasimir Kraev, I have frustrating issue with my wordpress installation version 5.7 using yoast seo plugin. My google search console reported that i have 1600 invalid url's due to having "noindex" in the <meta robots tag> of my pages/posts. Everything in the admin panel is set correctly(eg settings reading). I found that in this particular file of wordpress core files - /wp-includes/robots-template.php in the lines from 153-165 there is a function:

function wp_robots_no_robots( array $robots ) {

    // $robots['noindex'] = true;  THIS DEFAULT LINE COUSING ISSUES INDEXING MY PAGES ON GOOGLE SEARCH CONSOLE so i changed to false
    $robots['noindex'] = false;

	if ( get_option( 'blog_public' ) ) {
		$robots['follow'] = true;
	} else {
		$robots['nofollow'] = true;
	}

	return $robots;
}

Is this a bug or it is a normal wordpress 5.7 behavior?

Please respond.

website: ​https://webnavigator.bg
template: Listify 2.13.7
yoast seo 16.1.1

Regards!

[kraevit]

this tag has been automatically added to my header <meta name="robots" content="index, noindex, follow> which is what causing the issue.

[dd32]

Hi @kraevit,

The function in question should only be called when the blog visibility option is enabled, or another plugin is causing it to be called.

The functionality is controlled through the following setting under Settings -> Reading:

Search engine visibility	Discourage search engines from indexing this site

If that option is disabled, you should check where the function is being called from (It's new in 5.7), as I suspect it'll be being called from a plugin. The only place that calls it in WordPress that would affect most urls is wp_robots_noindex().

Without knowing which URLs are affected or why it's being called, it's not possible to determine if this is a WordPress issue or not, but it doesn't appear that it's likely to be a core bug due to the newness of the functionality and that more people aren't affected by it.

Unfortunately I'm closing this as invalid as there's no information which can be used to replicate it or point to the cause being WordPress itself.

[kraevit]

The entire website is affected all the links from pages to posts etc. Discourage is not enabled also. I commented the line as i said before and the site is back to normal. So it is definetely a bug.

[kraevit]

The entire website is affected all the links from pages to posts etc. Discourage is not enabled also. I commented the line as i said before and the site is back to normal. So it is definetely a bug.

[kraevit]

I checked the following: .htaccess file is ok, robots.txt is ok, all plugins are disabled. Caches of the website and browser are cleared, Discourage in settings ITS NOT enabled. The problem persist, and i have noindex in meta tag in the header on the entire website. When i comment the line in the wordpress core file all seems to be ok and the noindex disappears. So....

[adamsilverstein]

@kraevit it is possible your theme is causing the issue, or perhaps some other code running in your environment (maybe an mu plugin)?

Can you try switching temporarily to a core default theme, disabling all plugins and see if the issue persists? If possible can you enable logging and see if there are any errors in your logs?

As Dion points out above, this is very unlikely to be a widespread core bug or we would have had many reports of it already - it is somehow triggered by your setup.

[kraevit]

Replying to adamsilverstein:

@kraevit it is possible your theme is causing the issue, or perhaps some other code running in your environment (maybe an mu plugin)?

Can you try switching temporarily to a core default theme, disabling all plugins and see if the issue persists? If possible can you enable logging and see if there are any errors in your logs?

As Dion points out above, this is very unlikely to be a widespread core bug or we would have had many reports of it already - it is somehow triggered by your setup.

[kraevit]

I am really sorry for disturbing you. The problem was actually caused by my paid wp theme - listify. I didn't expect that latest theme update will cause this issue. Also I was frustrated since my client was angry because the bug affected his entire business. Now it's solved and all works fine. Sorry again and thanks for the provided information. My bad!