What are multiple URL patterns?
Most websites have a URL pattern for every page type. For instance, your blog post URL template or pattern might be different from category or product page. For a website with multiple sections and page types, the number of URL patterns keeps increasing.
URL Pattern 1 – Buying Guide Page: https://www.example.com/guides/guide-name
URL Pattern 2 – Category Page: https://www.example.com/catalog/catalog-name
URL Pattern 3 – Product Page: https://www.example.com/product-name
URL pattern 4 – Contact or About Page: https://www.example.com/contact
Having multiple URL patterns in your site is not the problem. The problem occurs when multiple URL patterns serve the same content. When multiple URL patterns serve same or similar content your site has a risk of having many pages with duplicate content.
The issue with duplicate content
If two or more web pages on your site have same or nearly similar content then your site has duplicate content. There are multiple ways duplicate content on your site can affect the way search engines crawl your website.
- Firstly search engines find it difficult to decide which version of your duplicate page to include or exclude from their index.
- When there are multiple pages serving the same content then link metrics like trust, authority, anchor text, link equity get divided between multiple versions.
- When you have too many duplicate pages on your site Google would be spending most of its time crawling duplicate redundant content instead of discovering other valuable content on your site. This will lead to other important content on your site undiscovered.
All these lead to inefficient crawling, poor ranking and loss of search engine traffic.
Three strategies to fix duplicate content issues
To fix duplicate content caused due to multiple URL patterns serving the same content you must know the 3 most effective techniques.
301 Redirect – Using a 301 permanent redirect we redirect the duplicate URL to its original URL. This means of all the duplicate URLs will be merged into a single URL. This solution is ideal when you want to serve the same destination URL to both search engines and users.
Canonical Tag – Canonical tag is one way pointing Google to the original page. This suggests Google transfer all SEO and link equity to the page which is being pointed to using the canonical tag. Google consolidates all link and SEO equity from duplicate pages on to its original page and gradually drops the duplicate pages out from its index.
The canonical tag is best suitable when you want to show users and Google different set of pages.
Meta Noindex – Noindex tag is one way of communicating to search engines that the webmaster of the page in concern doesn’t want it to be indexed by search engines. By tagging all duplicate content pages with “noindex” you can fight duplicate content on your site.
A meta noindex tag is useful for duplicate pages which not have acquired much link equity as in the case of ‘filters’ and ‘sort’ pages.
How to check if multiple URL patterns have duplicate content?
Step 1: Run a complete crawl of your website using DeepCrawl. Once finished open the crawl dashboard.
Step 2: Visit “Duplicate Content Body” under Content > Body Content in the left sidebar.
If you have multiple URL patterns showing the same content you will see a list of URL sets along with other metrics.
How to fix duplicate content across multiple URL patterns?
Once you arrive at the list of “Duplicate Content Sets” in DeepCrawl you will need to bucket them into the following categories and apply the ‘three duplicate content strategies’ appropriately.
Same product/page under multiple categories
When your CMS uses a category name in the single product or post URL things can get a bit complex. In this situation when you create a post or a product under multiple categories then you might create 2 or more URLs with the same content.
For instance, if your online store sells a particular desk which is listed under both ‘office’ and ‘home’ category then you might create 3 pages with the following URLs.
The problem gets even more complex when multiple subcategories come into play.
Since you want to personalize the experience of the user visiting the URL via each category you should use a canonical tag pointing from each category based URL to the main product URL.
Which in this case means you use a canonical tag on
www.examplefurniture.com/desks/home/awesome-desk-product-name and www.examplefurniture.com/desks/office/awesome-desk-product-name
which points to
Filter pages which have the same content as category pages
In the URL example below you will see that these URLs are creating a duplicate of each other because of filters and parameters which are very close in context to the main category page.
For instance, www.tshirts.com/blue-tshirts?color=blue will be a duplicate of www.tshirts.com/blue-tshirts while www.tshirts.com/blue-tshirts?color=blue&cat=9 will a duplicate of www.tshirts.com/blue-tshirts?cat=9&color=blue
Since in this case you want to show the user, the filtered page, and the category page both but want to fight duplicate content you should use a canonical tag pointing from the filter/parameter based URL to the category URL.
The other way of fixing this is by using a “noindex” tag on these URLs so that Google doesn’t even index this content. To perfect the solution further you should disallow crawler access to these filter pages using a Disallow field in robots.txt. This will save crawl bandwidth on your site leading to more efficient crawling by search engine spiders.