How To Help Webcrawlers Understand Scrolling Sites

Scott Lambert


As we recently reported on this blog, changes at Google have been causing problems for long scrolling sites. Other search engines also have difficulty crawling them properly, and businesses have been advised to move away from them.

In practice, though, the opposite has been happening, and they have been growing more and more fashionable. As their popularity extends to newspapers and other big players that cannot be easily dismissed, Google has now shifted its focus and published advice on how they might be reworked so that they get along with search engines better.

The basic problem is a simple one: webcrawlers work fast They do not wait around for more and more of a page to load; therefore, the SEO value of content that is far below the line can be lost if the search engine bot has moved on. For sites that have a lot of content but keep it all on one page, this can result in a significant, unintentional rankings penalty.

It is not in the search engines’ interests to rank pages poorly if they might still have strong appeal to human readers. The solution is to persuade people working in web design to adapt approaches that make it easier for webcrawlers to do their job.

The simplest solution Google proposed is to make sure that "infinite" scrolling pages are in fact split into sections, which may be invisible to the reader, so that they can be perceived as component pages and crawled individually. When doing this, it is important to make sure that there is no duplicate content across the pages that could lead to the site being ranked poorly.

Using component pages also makes it easier for internal searches to effectively identify different sections of content. This means that human users who find their way to the site after using obscure search terms that have not been used there for months will immediately be able to find the right section instead of having to wade through pages and pages of irrelevant content in search of them.

Some content management systems automatically provide pagination; however, if they are using JavaScript, they may not work with all search engines. More robust design is needed to ensure that the fundamental structure of the site is compatible.

Google has pointed out that, in general, what is good for webcrawlers also makes things easier for human users.

Topics: SEO