If Google keeps crawling your filter pages instead of your important product pages, there could be several reasons. Your filters might be creating too many URLs, or your internal links might be sending Google to pages it doesn’t need to visit. It can also be your website creating several URLs for the same products.
In this Motifo guide, we’ll explain where those URLs come from and how to check whether they’re wasting Google’s crawl activity. We’ll also show you how to use robots.txt, noindex and canonical tags to control which pages Google crawls and indexes.
Faceted Navigation Creates Too Many URLs for Google to Crawl
Every time a shopper applies a filter, your website may create a new URL. For example, someone browsing boardshorts might filter the products by colour, size and price.
One filter combination could produce this URL:
/boardshorts?colour=black&size=10
Another could produce:
/boardshorts?colour=blue&size=12
Each URL shows a different selection of products. When shoppers apply multiple filters, the number of possible URLs can grow quickly. This rapid growth is often called a URL explosion.
The problem gets even bigger when your website creates different URLs for the same selection.
For instance, these two addresses might show exactly the same products:
/boardshorts?colour=black&size=10
/boardshorts?size=10&colour=black
The URL parameters appear in a different order, but the product selection hasn’t changed. Google may still need to request both URLs before recognising that they contain duplicate content.
Google explains this problem in its guide to managing faceted navigation URLs. It warns that filter combinations can create an almost endless number of URLs, which wastes crawling and slows the discovery of useful pages.
Your Internal Links Keep Exposing Filter URLs
Google discovers many pages by following links from one URL to another. Your internal linking structure can therefore influence which filter pages Google finds.
Many ecommerce sites render each filter option as a standard link, which Google follows like any other. If those links stay crawlable, the crawler will keep finding new combinations as it moves through your categories.
For example, a category page might link to black boardshorts, blue boardshorts, size 10 boardshorts and several price ranges. Each filtered page could then expose another set of filter links.
Every new set of links gives Google more filter URLs to crawl, even when those pages add little to search results. To avoid this issue, keep standard links only for filter pages you want in search. For the rest, use URL fragments or form controls that don’t create crawlable links.
Sorting Options Create Duplicate URLs for the Same Content
Filters aren’t the only feature that can multiply URLs. Sorting options can do the same thing.
A shopper might sort boardshorts by price, popularity or newest arrivals. Your website could create a separate URL for each option:
/boardshorts?sort=price-low-high
/boardshorts?sort=popular
/boardshorts?sort=newest
These URLs can show the same products in different orders, which gives Google more variations to crawl without exposing additional products. Unless a particular sorted view serves a distinct search purpose, Google has little reason to crawl every variation.
Sitemaps and External Links Can Expose More Filter URLs
Sitemaps and links from other websites give Google two more ways to find filter URLs. If your XML sitemap lists hundreds of filtered pages, Google treats each one as worth discovering. A sitemap should focus on the pages you want search engines to find, not every possible version of a category.
External websites can create another discovery path. Someone might link to a filtered product listing, or a page might be shared with its filter settings in the URL. Google can find those addresses even if you don’t link to them from your main navigation.
Neither a sitemap entry nor an external link guarantees that Google will crawl a URL. However, both can help Google discover addresses that your site would otherwise expose less often.
How to Check Which Filter Pages Appear in Search Results
Now that you know why Google keeps finding filter URLs, it’s time to check whether those URLs are actually causing a problem on your website. Start by identifying which filter combinations Googlebot requests, then check which ones show up in search results.
Check Your Server Logs for Filter URL Requests
Your server logs record requests made to your website, including requests from Googlebot. They can show you exactly which URLs the crawler accesses.
Look for recurring URL patterns to see whether Google is crawling parameter URLs for filters, sorting options and other variations.
For example, if your filter URLs use colour=, size= and price=, group the requests by those terms. This helps you see which types of filters Googlebot accesses most often.
Three patterns point to a filter crawling problem:
- Large Numbers of Filter URLs: Googlebot requests many combinations of colour, size, price and other filters.
- Repeated Variations of the Same Page: Googlebot requests different URLs that display the same or very similar product listings.
- Unnecessary Sorting URLs: Googlebot repeatedly accesses pages that only change the order of products.
Before drawing conclusions, confirm the requests come from Google. Other crawlers can copy Googlebot’s user-agent name, so run a reverse DNS lookup on the IP addresses first.
Review Google’s Crawl Stats Report
You can find the report under Settings → Crawl Stats in Google Search Console.
The report covers total crawl requests, average response time, response codes, and the file types Googlebot fetches. You can also review examples of URLs requested by Google. Compare this information with your server logs to see whether filter URLs account for a substantial share of crawling activity.
However, a high number of filter requests doesn’t always mean Google is ignoring your product pages. Check whether new or updated products still get crawled soon after they go live.
Crawl data shows what Google visits, not what shoppers see. To see which filter URLs appear in search results, open the Performance report and filter the Pages tab by a parameter like colour=. Any filter URL with impressions is already showing up for shoppers.
How to Stop Google From Crawling or Indexing Unnecessary Filter Pages
Once you know which filter URLs Google requests, decide which ones deserve a place in search. Three options are available: robots.txt, noindex and canonical tags. Let’s see how they work.
Use Robots.txt to Stop Unnecessary Crawling
A robots.txt file tells search engine crawlers which parts of your website they can access.
If you don’t want Google to crawl certain filter URLs, you can use a Disallow rule to block matching URL patterns.
For example:
User-agent: Googlebot
Disallow: /*?*colour=
Disallow: /*?*size=
Disallow: /*?*sort=
These rules tell Googlebot not to crawl URLs containing those parameters. Before adding these rules, check whether any landing pages you want in search share the same parameters, because a broad rule will block those too.
Even then, robots.txt doesn’t guarantee that a blocked URL will stay out of Google Search. Google may still index a URL if it finds links pointing to it, even though it can’t crawl the page content.
Use Noindex When Google Can Crawl a Page but Shouldn’t Index It
A noindex directive tells Google not to include a page in its search results.
You can add it to a page’s HTML:
<meta name=”robots” content=”noindex”>
This can be useful when a filtered page helps shoppers browse your products but doesn’t need to appear in Google Search.
However, Google must be able to crawl the page to read the noindex directive. If robots.txt blocks the URL, Google can’t access the page to see that instruction.
So don’t block a page in robots.txt if you expect Google to discover and follow its noindex directive. Choose one approach based on whether you want Google to access the page.
Use Canonical Tags for Duplicate Filter URLs
Sometimes, different filter URLs show substantially the same products or content. In those cases, a canonical tag can tell Google which version you prefer to appear in search results.
Take the two boardshorts URLs from earlier, where only the parameter order changes.
/boardshorts?colour=black&size=10
/boardshorts?size=10&colour=black
If the first URL is your preferred version, you could add this canonical tag to the second:
<link rel=”canonical” href=”https://example.com/boardshorts?colour=black&size=10″>
However, a canonical tag is a signal, not a command. Don’t combine a canonical tag with a robots.txt block, because Google has to crawl the page to read the tag. Google can choose a different canonical URL if it considers another version more appropriate.
And canonical tags don’t stop Google from crawling a URL either. If your main goal is to stop Google from requesting unnecessary filter pages, robots.txt is the more direct option.
Check Whether Your Changes Are Working
After your changes go live, review your server logs again and compare them with your earlier data. You’re looking for fewer Googlebot visits to the filter URLs you restricted.
Then, use the URL Inspection tool in Search Console to confirm your main category and product pages are still crawlable and indexed. If a useful landing page drops out of search results, look for a robots.txt rule or noindex tag that caught it by mistake.
Results won’t show straight away, because Google has to recrawl each page and process the new signals first. If Googlebot keeps requesting unnecessary filter URLs after that, recheck your URL patterns, internal links and sitemap entries.
Need Help Fixing Excessive Filter Crawling?
You don’t need to panic if Google is still crawling some filter URLs. Review your findings, identify where unnecessary URLs are coming from and put the right restrictions in place.
If you’re struggling with faceted navigation SEO or need help managing your ecommerce site’s crawling and indexing, Motifo can help. You can also explore our guide to technical SEO, crawling and indexing for more ways to improve how search engines access your website.
Any questions? Feel free to reach out to our team.