What is a Directory/HTML Sitemap?
In SEO terms a site directory is called an HTML Sitemap. A site directory is an HTML page which links to all possible URLs on your website. Depending on the size and structure of your website there might be multiple pages under an HTML sitemap.
Directories can be used by a user who wants to understand what all content is present on a website. For instance, if a user lands on your page looking for a product he or she might want to check all your product categories, subcategories, and styles at one place. A directory enables the user pick the most relevant content from across your site.
For search engines, it is a great way to discover deep level links which cannot be possibly reached via regular website navigation. HTML sitemaps pass link equity to the pages it links to. This can help deep level content rank better in search engines.
Does my site need a directory of Links?
Not all websites need a site directory. Only if your website fits in any of the three buckets below you should consider building an HTML sitemap.
- Most modern search engines are capable of crawling most links on a certain domain as long as the site has just a few hundred pages. But as the number of pages on the site increases and goes beyond few thousands the likelihood of the deep level pages receiving the necessary link equity reduces. Resulting in loss of indexation and lowered ranking power of these pages.
- The second reason you might want to have an HTML sitemap would be that your site has a large archive of content pages which are not linked from the main site. If you have web pages which are standalone or landing pages and don’t link to each other then an HTML sitemap can solve this problem.
- If your site is new and has zero or very few backlinks then HTML site can help Googlebot discover your pages. With HTML sitemap you can link even to the deepest of site pages. Hence it might not need backlinks from other sites to get discovered.
Site Directory SEO and User Best Practices
Not all site directories are user-friendly or optimized. Larger your website, larger will be your site directory. And the only way to make your site directory effective in such cases is to follow these set of best practices.
Structure – A directory should have a flat architecture. In other words – A search engine or a user should be able to reach the destination URL within 3 clicks. Flatter the architecture more link equity search engines will be able to pass to your deeper level pages.
In the example below, you can reach any destination page within 3 clicks from the website homepage
Limit Links – Always limit the number of links on any page to under 200. When you use too many links on a single page the user might not feel like navigating through your sitemap. Also when you use too many links on a single page not all pages you link to gets crawled by search engines.
Automated – See that your HTML sitemap is automated. As an active business, you might be adding pages to your site frequently. You want to link to these new pages from the sitemap so that search engines pick this page quickly and start driving traffic to your new pages. Also having an automated sitemap gets rid of the manual task of editing everytime you publish a new piece of content.
Anchor Text – Use the right anchor text while linking to the destination site. This will help the user understand what he or she is clicking on. From a search engine’s perspective, the anchor text also passes link and content value to the linked page.
Static Links – Keep your site directory links in plain HTML. Make it easy for all search engines to crawl through them.
How to check if my site has a Directory?
If you have a page with links to all other pages on your site then you might have an HTML sitemap. Check with your developer if such a page was developed previously. A typical site directory page looks like the image below.
How to create an HTML Directory?
If you are using a custom CMS then follow the steps below –
Step 1: Make an audit of all links on your website
This will help you in 2 ways. Firstly it helps you give a bird’s eye view of your site which helps you chalk out a logical structure for your sitemap. Secondly, you can test your HTML sitemap(after you build it) for missing links or folders.
Step 2: Create a logical structure for your sitemap
The structure of your sitemap should make logical sense to your user. It should follow basic principles of information architecture. Although there is no such rule for maximum links on a page limiting the number to 200 will ensure both search engines and users navigate the directory better.
If you feel a certain directory page will have more than 200 links see if you can further split the parent directory into subcategories. You can even use pagination to limit 200 links per page.
Zomato uses Pagination in its directory as a way to deal with too many links on a page.
Step 3: Make sure you don’t add these links to your HTML directory
When a search engine crawls your directory you must present the links which are worthy of being crawled and indexed. Do NOT Link to the following pages from your directory – Noindex Pages, Redirect Pages, Canonical Pages, 4XX/5XX expired content, Folders and URLs blocked by Robots.txt
Step 4: Create a mockup of your Site Directory
Step 5: Link to the directory from every other page on your site
A search engine spider is forever on a hunt for finding fresh content. By adding a link from every other page on your site you are enabling search engines more possibilities of finding new content. Most sites link to their site directory from within their footer.
Step 6: Fetch as Google
Once you have created a sitemap, submit the new directory for indexation and crawling via Google search console. This will help Google discover a new page on your site which has a lot of links for its delight.
To do so, first visit Fetch As Google section in Search Console. Enter the site Directory URL and hit “Fetch and Render”.
Once the render is finished click on “Request Indexing”.
On clicking it you will see a popup as below. Pick “Crawl this URL and its direct links” as an option and hit submit.
Once done Google will crawl your sitemap and all associated links on the page.
If you have various subcategories under your sitemap then you have to repeat the exercise for each one of them.
Plugins and Tutorials to create HTML Sitemap For Popular Content Management Systems –
Great HTML sitemap Structure Examples