Tags Blocked when SEMRush crawls site
By Stephen Crewson on Tuesday, 15 December 2015
Posted in Technical Issues
Likes 0
Views 733
Votes 0
I'm doing some SEO work on two websites. I've noticed on the two sites that I have crawled using SEMRush that pages corresponding to the Easyblog tags are being blocked.

Please see the screenshot for more info.

The two sites are:
http://www.fairviewhifi.com
http://www.bicycleworks.ca

Do you know if I am missing something in the settings that might be causing the issue?

Thank you,

Steve
Hi Stephen,

I am sorry for the misunderstanding here. I just checked our code for tag's page and it seems like we have added the following code as default:
// Add noindex for tags listing by default

$this->doc->setMetadata('robots', 'noindex,follow');


When this code is added, any robot crawler will not be able to index the page hence returning the result that you have above. The reason why we added this is because many users are complaining that the tags view always yield a duplicate result by Google.

You can manually remove the code by going to your /components/com_easyblog/views/tags/view.html.php and remove the code above at line 92~93.

Hope these help.
Ezrul Fazwan
·
Thursday, 17 December 2015 13:23
·
0 Likes
·
0 Votes
·
0 Comments
·
Hi Stephen,

I am really sorry for the delay of this reply.

I notice both of your blog url is not exactly uniformed across your page. May I know did you sh404sef components on your site? Can you provide us with your Joomla backend and FTP access for both of your site so we can check on your current menu structure and SEO settings and see if its configured correctly?
Ezrul Fazwan
·
Tuesday, 15 December 2015 13:22
·
0 Likes
·
0 Votes
·
0 Comments
·
Hi Ezrul,

I've updated the ticket and have given you login info for only one of the two sites. If it is a quick fix I'd love to know the process of correcting the issue as I'm probably making the same mistake elsewhere.

Thanks again,

Steve
Stephen Crewson
·
Wednesday, 16 December 2015 10:39
·
0 Likes
·
0 Votes
·
0 Comments
·
Hi Stephen,

I'm just curious, can you try to test your SEO using SEMRush again but with sh404sef turn off and see if the result is the same?
Ezrul Fazwan
·
Wednesday, 16 December 2015 16:22
·
0 Likes
·
0 Votes
·
0 Comments
·
Hi Ezrul,

Thanks for looking into things for me.

I disabled sh404SEF cleared the cache and ran the report with the same result
I then enable sh404SEF, changed the router settings to joomla in the sh404SEF configuration and ran the report with the same result.

For some reason the tags are continuing to be blocked.

Steve
Stephen Crewson
·
Wednesday, 16 December 2015 22:21
·
0 Likes
·
0 Votes
·
0 Comments
·
Hi Ezrul,

Thank you for your thorough response. I'm going to experiment with the code change, add some additional content to the page and see how Google responds. Once again, thank you for the time you took to look into the matter.

Enjoy your day,

Steve
Stephen Crewson
·
Thursday, 17 December 2015 22:12
·
0 Likes
·
0 Votes
·
0 Comments
·
Hey Stephen,

We actually disable indexing for "tags" by default because a lot of our users are telling us Google is rendering tags views as "duplicates" and this is why we set a noindex by default
Mark
·
Thursday, 17 December 2015 22:38
·
0 Likes
·
0 Votes
·
0 Comments
·
Thanks Mark. Point taken. I'm not going to even experiment with it.

Thanks for the great work. Thanks for the great explanations. Thanks for the great product.
Stephen Crewson
·
Thursday, 17 December 2015 23:20
·
0 Likes
·
0 Votes
·
0 Comments
·
You are most welcome Stephen and thank you for the kind compliments
Mark
·
Thursday, 17 December 2015 23:27
·
0 Likes
·
0 Votes
·
0 Comments
·
View Full Post