GoogleBot To Soon Crawl Over HTTP/2


GoogleBot To Soon Crawl Over HTTP/2

On the 17th of September, Google announced that it would soon start crawling some sites over HTTP/2 beginning November 2020.  

For those who don’t know about HTTP/2 – it is the next generation of HTTPS, an internet protocol primarily used for transferring data. 

HTTPS/2 requires less open connections and thus can be more effective on your server when crawling your web pages. 

However, crawling your site over HTTP/2 has no benefits on Google rankings. 

What Is HTTP/2? 

HTTP/2 is an upgraded version of the HTTP network protocol used by the World Wide Web. HTTP/2 originated from the earlier experimental SPDY developed by Google. It was developed by the HTTP Working Group of the Internet Engineering Task Force. 

According to Ilya Grigorik from Google – 

 “HTTP/2 will make our applications faster, simpler, and more robust — a rare combination — by allowing us to undo many of the HTTP/1.1 workarounds previously done within our applications and address these concerns within the transport layer itself. Even better, it also opens up a number of entirely new opportunities to optimize our applications and improve performance.”

HTTP/2, also known as h2, is more effective and efficient compared to other versions, and thus Google is taking these steps. 

Google said – 

“We expect this change to make crawling more efficient in terms of server resource usage. With h2, Googlebot is able to open a single TCP connection to the server and efficiently transfer multiple files over it in parallel, instead of requiring multiple connections. The fewer connections open, the fewer resources the server and Googlebot have to spend on crawling.”

Going further, Google said starting from November 2020; it will slowly and gradually begin with a small number of sites and increasing support for more sites with the time. 

In an initial phase, Google will only take up those sites that may benefit from the supported features like request multiplexing – said Google. 

What Will Happen To Those Sites Who Don’t Support HTTP/2? 

According to Google, it is fine if your server still only talks HTTP/1.1. It will have no explicit drawback for crawling over HTTP/1.1 protocol; crawling will remain the same, quality and quantity-wise.

What Are The Benefits Of HTTP/2 Crawling? 

According to Google, there are three primary benefits of HTTP/2 crawling – 

Multiplexing and Concurrency: 

There will be fewer TCP connections open, and thus it will decrease the number of resources spent. 

Header compression: 

The size of an HTTP header will be reduced significantly, resulting in the saving of the resources. 

Server push: 

Though this feature is not yet implemented as it is still in the development phase; it will be beneficial for rendering. 

How To Make Sure Whether A Site Is Supporting HTTP/2 – 

In order to verify the support of HTTP/2 for the site, one can ask a host or a developer to examine it. 

➤  How Can I Request Googlebot To Talk h2 With My Site? 

If the site is eligible for being crawled over h2 and it would be beneficial to both the site and Googlebot, it will automatically do – One can not request for it. 

If the bot finds that crawling over h2 would not result in significant resource savings, then it will continue to crawl the site over HTTP/1.1. 

➤  How To Opt-Out – 

Though there are no issues or negative impact on indexing while opting for HTTP/2 crawl, Google understands that one may still want to opt-out for various reasons. 

In order to opt-out, one needs to instruct the server to respond with a 421 HTTP status code when Googlebot attempts to crawl a website over h2. If that is not feasible, one can send a message to the Googlebot team. 

➤  How One Can Know If A Site Is Crawled Over HTTP/2 – 

When a site gets crawled by HTTP/2, a message pops-up in Google Search Console. 

According to Google, the message may read something like “Some of the crawling traffic may be over h2 going forward. 

Moreover, Google adds that one can also check serve log to ensure the HTTP/2 crawling.  

➤  Does One Need To Upgrade The Server ASAP? 

Google says it totally depends on the site owner. If Google finds that crawling over HTTP/2 will benefit the site and Googlebot, then only it will automatically switch to h2. 

And if that will not be the case, it will continue to crawl over HTTP/1.1. 

➤  How Can One Upgrade A Site To h2? 

By talking to the server administrator or hosting provider. 

➤  Does Googlebot Support Plaintext HTTP/2? 

No, a website must use HTTPS and support HTTP/2 in order to be eligible for crawling over HTTP/2. 

➤  Is Googlebot Going To Use The ALPN Extension To Decide Which Protocol Version To Use For Crawling? 

ALPN (Application-Layer Protocol Negotiation) will only be used for sites that are opted into crawling over h2, and the only accepted protocol for responses will be h2. So, if the server responds during the TLS handshake with a protocol version other than h2, Googlebot will back off and come back later on HTTP/1.1.


Keval BhuvaSr. SEO Executive

Keval is a young and enthusiastic person who enjoys and loves cricket the most. Besides that, he has extensive knowledge of Digital Marketing and, particularly, SEO. Watching web series and reading about politics is his favorite time pass. He is a great example of “Living a big life with an open heart,”

Leave a Reply

Your email address will not be published. Required fields are marked *

Related Blogs

Content Marketing(Read Time: min.)July 30, 2021Top 4 Reasons to Use Content Marketing For Your eCommerce Business| Here’s All You Need to Know About

If you are running an eCommerce store, you already know how important it is to bring more traffic and generate more conversions.      Getting higher traffic doesn’t guara

Read More

Development(Read Time: min.)July 15, 20215 Things You Should Know Before Designing Your Website | A Complete Guide to a Newbee

Have you ever seen a website and thought, “Wow! This is the best website one could have of both worlds”, and then you see another website that looks terrific and you think who

Read More

PPC(Read Time: 5 min.)June 30, 2021How To Set Up Time Of Day Bid Adjustments

What is Ad Scheduling in Google Ads and How to Set Up Time of Day Bid Adjustments -  As we humans evolve every day, our technologies are also enhancing every day. The way of ma

Read More

YOU’VE GOT A PROJECT IN MIND Let’s Build Something Together

Drop us a message with a brief description of your dream project.

Our industry domain expert will review it and get back to you within 24 hours with free consultation and best reliable solutions.