How To Help Webcrawlers Understand Scrolling Sites

Scott Lambert


As we recently reported on this blog, changes at Google have been causing problems for long scrolling sites. Other search engines also have difficulty crawling them properly, and businesses have been advised to move away from them.

In practice, though, the opposite has been happening, and they have been growing more and more fashionable. As their popularity extends to newspapers and other big players that cannot be easily dismissed, Google has now shifted its focus and published advice on how they might be reworked so that they get along with search engines better.

The basic problem is a simple one: webcrawlers work fast They do not wait around for more and more of a page to load; therefore, the SEO value of content that is far below the line can be lost if the search engine bot has moved on. For sites that have a lot of content but keep it all on one page, this can result in a significant, unintentional rankings penalty.

It is not in the search engines’ interests to rank pages poorly if they might still have strong appeal to human readers. The solution is to persuade people working in web design to adapt approaches that make it easier for webcrawlers to do their job.

The simplest solution Google proposed is to make sure that "infinite" scrolling pages are in fact split into sections, which may be invisible to the reader, so that they can be perceived as component pages and crawled individually. When doing this, it is important to make sure that there is no duplicate content across the pages that could lead to the site being ranked poorly.

Using component pages also makes it easier for internal searches to effectively identify different sections of content. This means that human users who find their way to the site after using obscure search terms that have not been used there for months will immediately be able to find the right section instead of having to wade through pages and pages of irrelevant content in search of them.

Some content management systems automatically provide pagination; however, if they are using JavaScript, they may not work with all search engines. More robust design is needed to ensure that the fundamental structure of the site is compatible.

Google has pointed out that, in general, what is good for webcrawlers also makes things easier for human users.

Scott Lambert

Written by Scott Lambert

For 35 years, Scott has been at the forefront of digital transformation and how it is effectively applied in businesses and organizations to market, sell, and support customers and members more successfully and cost effectively. Scott is President and Co-Founder of Xcellimark, an award-winning digital marketing agency serving clients throughout the U.S., Canada, Central America, and Germany. Xcellimark helps businesses succeed through the implementation of its unique digital marketing and sales approach specifically designed to meet their specific goals and objectives. Scott’s marketing, sales, and operations experience have included key management positions at industry leaders including AT&T, iXL (now SapientRazorfish), eSchool Solutions (now TalentED), and BellSouth, where he led the product development, marketing efforts, and market launch of BellSouth.net, the second public ISP developed in the U.S. Scott is the author of articles highlighted on leading online publishers such as LinkedIn, Business 2 Community, and he was recently featured by LinkedIn in the Entrepreneur magazine article on “No Time for Marketing? Hire a Freelancer.” His engaging personality and wealth of real-world and business experience make Scott a frequently requested speaker. Some of his engagements include being the featured speaker in a series of Orlando Business Journal seminars on Digital Marketing, speaking at the WSI London conference, and at international conferences in Belize. Scott has a Bachelor’s degree in Accounting from the University of Mississippi and a Master’s of Science in the Management of Technology from the Georgia Institute of Technology (Georgia Tech). Scott has been active in global mission efforts and is a board member of Global Teams, a Christian outreach organization. He lives in Winter Springs, FL with his wife Nancy and dog Sadie.

Topics: SEO

Reply or Leave Comments: