Did you know that not everything that is visible to you, as a simple user, is also displayed to the Google indexing bots?
Elements such as forms, cookies and images without an alternative description are ignored, instead only links coded with certain anchor or href tags are read. Here are some elements that search engine bots block, to facilitate the indexing process!
Location based pages
There are certain websites that detect the user’s location and display their content based on their IP. But this technique is not the best! It is possible that a user with an IP from Bucharest receives information from this location, but he actually and legally lives in Timişoara. Therefore, he will not be interested in the content displayed. This mentality is also addressed by search engine bots. They allow the data to be displayed depending on the location, but only on the first entry to the website. Subsequent content should be based on the links accessed.
Content based on information from cookies
Since the GDPR entered into force, almost all users are familiar with the notion of cookies. Through them, users can have a personalized experience when browsing the website. However, be careful, the content that users access thanks to cookies will not be accessible to search engines.
Let’s use an example to better understand how they work. Let’s say you access a website and choose English as your main language. The cookies will remember this option, so they will display all your pages in English. Although the links will remain in Romanian, the content is in English. As long as the URLs don’t change as the content changes, search engines won’t index the alternatives.
JavaScript links
If so far we have talked about more general notions, in the following paragraphs we will address more technical terms. To a search engine a link isn’t really a link without an anchor tag and an href to a URL. It is preferable to opt for a descriptive text of the anchor, because it gives the search engine a suggestion about the content of the link.
Most e-commerce websites opt for links that contain the “onclick” attribute instead of anchor tags. It is not wrong for users who will browse the website, but Google bots will not read these links. What this means? Indexing problems!
URLs containing hashtags
If you’ve ever used AJAX, you probably know that it’s a form of JavaScript that refreshes the page’s content without reloading it. Refreshed content generates a hashtag in the page’s URL. If the search engine indexes URLs containing the hashtag, the content may not be what users are looking for.
The robots.txt file
This is a text document that forms the basis of a website. It tells the crawlers what content can be indexed and what pages can be ignored. Denial orders do not mean that those pages will be completely ignored, but they can prevent them from being ranked in search engines based on relevance. Pay attention to the pages you include in this file. It is possible to include website-relevant pages in opt-out orders. If you are experiencing a sudden drop in organic search traffic, check your robots.txt file and analyze the pages included in the disavow orders!
The noindex attribute
With the noindex attribute, you will tell the bots not to index a certain page. It is most often used within a single page, rather than in a single file that governs the entire site, such as opt-out commands.
However, noindex attributes have a bigger impact than the deny commands we were talking about above. They stop indexing the page entirely. Noindex attributes can be set accidentally and can sometimes be very hard to find on a website.
Canonical labels
These types of tags identify which page is being indexed, out of several similar versions. They are important tools to prevent content duplication. All non-canonical pages assign their link authority and will not be indexed.
Canonical tags are included in the source code and sometimes errors can be difficult to detect. If the desired pages on your site are not indexed, incorrectly set canonical tags may be the main cause.
Conclusion
Many of the problems that prevent successful optimization are not particularly difficult to solve, nor do they require a bachelor’s degree in automation and computers to identify. Follow the SEO Starter Guide, pay attention to the above elements, create authentic content, track your progress over time, and the results will not be long in coming!
