Question: How to find all pages on a website?
Answer: It’s easy and there are many different ways to do that!
And in this article, I am going to show you 9 ways how to find all pages on a website using free and paid methods.
Let’s get started!
Why Do You Need to Find Every Single Page on a Website?
Knowing all pages on a website can help you in several ways to prevent any ranking issues. It also helps you to see all the pages that search engines are seeing on a website in order to determine if any pages should not be ranked.
Based on this you can easily remove any unneeded pages to optimize your crawl budget and even improve your overall ranking by removing pages buried deep in SERP that are not supposed to be ranking.
However, it also helps you to do any SEO competitor analysis to see what pages your competitors have on their website, what SEO keywords they are trying to target, what pages are the most important for them and so much more.
So, let’s have a look at these 6 reasons why you should know how to find all pages on a website whether it is your website or not:
Reason #1: Find Orphan Pages
The first reason is to find any orphan pages on a website.
Orphan pages are pages without any internal links pointing to them from any other web page or section of a website, which makes the page inaccessible for website users by simply clicking through your website and even for crawlers as well.
Usually, these pages are rarely to be indexed by search engines because they can’t be found by search engine crawlers as the prior way to discover web pages on a website is by crawling links.
However, even if they are indexed within the SERP, their ranking will often be very poor, and they can be lowering your average ranking position and even damage your overall ranking.
Thus, you want to know about these pages in order to make a decision whether to create internal links to the page, delete or remove them from the index.
Of course, not every page is meant to be indexed and often purposely made orphan pages such as landing pages for Google adverting or Facebook advertising are meant to be inaccessible from within the website.
Reason #2: Find Weird Ranking Pages
Another reason why you want to find all pages on a website is to find those weird, low-quality, thin-content, and spammy types of content that should not be indexed or better not exist on the website unless there is a good reason for them.
And there can be a lot of types of pages that should not be indexed and perhaps the best way to see and learn how to find all those weird pages is by looking at Glen from Detailed site audit videos like this one:
But for example, when you look at Backlinko, you can see one of their admin pages for a blog post was indexed by Google.
And when I visit this page, it does not exist.
Of course, these kinds of web pages you don’t want to be indexed on Google, thus, should be removed as pages like this can waste server resources, can cause a significant delay in discovering important pages on a site, and plummet your ranking.
Reason #3: Removing Duplicated/Redundant Pages
Another reason why you want to know how to find all pages on a website is to remove any duplicated/redundant pages.
Duplicated content can cause a lot of problems for search engines and webmasters as well and if you want to ensure that you have applied the best SEO practices, you deal with them.
Now, when it comes to duplicated content, there can be two versions of that:
- Within your website: This is when you have two identical pages on your website. This often confuses search engines on which one to rank, plus you might be wasting your crawl budget.
- Outside your website: This is when different websites copy content from your website and publish it on their own website. This can be a huge problem as they can over-rank your original content and steal your traffic. Even worst you might get a manual penalty for duplicating content.
However, in this case, we are talking about duplicated pages within your website that you want to discover and remove, redirect or provide a canonical tag, so search engines know which one is the main version.
Duplicated content can be a real issue for your website and by knowing how to find all pages on a website you can fix issues like this.
Reason #4: Changing Website Architecture
The next reason why you would need to know how to find all the pages on a website is due to changing website architecture.
Website architecture for SEO is extremely important as it allows you to plan out all the different contents located on the website in the most efficient manner so that users can easily find the information they are looking for and search engines can properly discover, crawl, and index all your pages.
However, one of the common issues that webmasters are facing later in the stage when their website has grown is poor website architecture and their SEO URLs structure that needs to be updated.
As most website owners do not really plan what keywords they will target, what SEO content they will have on the website, and other important aspects that come with designing your website architecture.
Thus, if you are in need of redesigning your website architecture, then you will need to know all the pages on your website in order to move your content around without recreating content that you have already on your website and so avoid keyword cannibalization and wasting your time.
Reason #5: Optimizing for Internal Linking Strategy
Another important reason why you would want to know how to find all pages on a website is when you want to optimize for an internal linking strategy.
SEO cross linking arguably one of the best SEO techniques to improve your impressions, ranking, and traffic to your website that you have absolute control over.
As mentioned, links are the prior way used by search engines to discover and crawl the website.
And internal links also pass link juice. Thus, with internal links, you can redistribute the link juice, AKA PageRank to other pages such as your product pages to boost their rankings.
So, with a properly optimized website with an internal linking strategy, you can easily increase your overall website ranking.
You can also help search engines to better understand your content when it is indexing it. Search engines are reading the internal links or the link text, which is called anchor text to get better context about the page.
Therefore, knowing all your pages and what keywords they are targeting you can essentially use that to interconnect the pages together when mentioning the keyword or topic within other pages like in this example.
Honestly, sometimes I play around with the text to mention the keyword on other pages. However, always try to keep it natural and within the context.
Reason #6: Doing Competitor Research
And the last reason why you should know how to find all pages on a website is when you want to do any competitor research such as SEO competitor analysis or backlink competitor analysis.
Your competitors are arguably one of the best sources of ideas and what works and what doesn’t for you to improve your SEO strategy.
And knowing how to find all pages on a competitor’s website will help you to find what pages are important for your competitors, what keywords bring them the most traffic and so much more.
Thus, knowing how to find all pages on a website will help you when you are doing competitor research as well.
So, now that we have discussed why you need to know how to find all pages on a website, let’s have a look at the different ways how you can do that:
Way #1: Using Competitive Research Tool
The first way to find all your and your competitor’s pages is by using a competitive research tool.
For this and other ways, I will be using SE Ranking and if you do not own any SEO tool or even if you have one, then I can definitely recommend you try SE Ranking as they offer over 30 SEO tools with the best value for money.
So let me show you how you can find all pages on a website using SE Ranking Competitive Research tool.
Step #1: Go into the Competitive Research tool, where you enter your or your competitor’s domain, select your target location if applicable and hit “analyze”.
Step #2: Here you will find a high-level overview of the analyzed website such as the domain trust, organic traffic, ranking keywords, the backlink profile, and more.
So now, you just go to the “Top pages in organic search” report using one of these methods.
And voila! Here you have the website pages of the analyzed website. Not all, but most of them. At least the most important for organic search.
You see, super easy. So, let’s move to the second method.
Way #2: Using Website Crawler
The second way to you can use to find all pages on a website is by using website crawlers and website audit tools.
Website crawlers usually work behind the scenes for SEO tools to discover website pages and backlinks and save them in their databases.
And website audit tools such as SE Ranking Website Audit tool are perfect for you to find all pages on a website as they work as a website crawler, plus show you all the SEO issues for every web page. And they are super easy to use as well.
Step #1: To start using SE Ranking Website Audit tool, you just need to create a new project. This can be your website or any other website.
Step #2: Then you can “restart audit” most likely if you have created a new project then it will start automatically.
Step #3: Once the report is completed, then you go to the “Crawled Pages” report where you can find all pages on a website.
Plus, you can easily customize the table with additional columns to get more information about the pages.
This is a super helpful report for both fixing your web pages, learning more about the website’s overall architecture, internal linking strategy, and a great way to find all pages on a website.
Way #3: Using Sitemap & Sitemap Generator
The next way how to find all pages on a website is by looking at a sitemap or using a sitemap generator to create a sitemap for a website.
Most websites are having sitemap/s to help search engines discover new and newly updated pages on a website. Thus, improving the crawl efficiency.
So, to find a sitemap on a website you can use a tool like Sitemap Test that helps you to analyze the website and show you any sitemaps located there.
Once you have the sitemap, then you can go through it and you will find all the indexable pages on a website.
However, if there is no sitemap on the website, you can use a sitemap generator. For example, the SE Ranking Website Audit tool provides such a feature.
Or you can use free sitemap generators.
And if you are using CMS like WordPress you can easily install SEO plugins such as Rank Math that help you to create a sitemap for all your pages as well as create an image sitemap to improve the ranking of your images.
Way #4: Using Webmaster Tools
The fourth way how to find all pages on a website is by using webmaster tools like Google Search Console or Bing Webmaster Tools.
Webmaster tools are made for website owners to see their website performance in the search results as well as any issues discovered by them or show notification of any manual action taken against your website.
Of course, if the webmaster console is owned by Google, then you will see only the performance for the Google search and likewise for Bing and other webmaster tools.
Now, because most of the webmaster tools are very similar to each other and because most of your traffic will probably come from Google as they own approximately 90% of the market, I will show you how to find all your website pages using Google Search Console.
So, when you open up Google Search Console, you can see the “Overview” report of your website. From here you will want to go to the “Coverage” report.
In the coverage report, you will want to select the “Valid” pages and click on one of the options in the “Details” tab. Both reports together should give you all your website pages.
If not, you can also select the “Excluded” tab to get more options to go through.
Sometimes freshly new web pages can take a while before they will be indexed, or they can be excluded for reasons such as violating guidelines, adding a noindex tag, etc.
To learn more, about how Google finds all your pages check out this.
Way #5: Using Google Analytics
The next way to find all pages on a website is by using Google Analytics.
Google Analytics is great for finding orphan pages that do not have any links coming to them or are not included within the sitemap, and thus cannot be reachable by the crawlers.
Because, Google Analytics can access orphan pages if they receive at least one pageview, this can also include pageview from bots that are pretending to be the user.
With that, because there are two versions of Google Analytics, I’ll show you how you can find all pages on a website using each of them.
To find all pages on your website using Google Universal Analytics go to Behavior → Site Content → All Pages.
Now you want to change the date of the report from before you have launched your website and start using Google Analytics to today.
Now, unfortunately, this is when you will also get a lot of trash that you will have to manually sort out from what’s the actual content page and what’s a URL of a site builder, search results, and other non-relevant URLs like in my case.
Now let’s have a look at how you can find all your pages on a website using Google Analytics 4.
To find all pages using G4, go to the Reports → Engagement → Pages and screens.
Then you will want to change the settings to see page paths or the URL of your web pages if you don’t the page title of the same page can change based on the translation of the visitor.
You will want to change the setting from “Page title and screen class” to “Page path + query string and screen class”.
And lastly, you will want to change the date of the report from before you launch your website to today just like in the previous example.
And again, you will need to manually sort through the report to find actual content on the website and not search result pages, page builders, and other not relevant URLs.
Way #6: Using Site Operator
The next way to find pages on a website is by using the google site operator.
Simply, just go to the Google search and write site:yourdomain.com for example in my case it would be site:b2bdigitalmarketers.com
As you can see, all the search results are only from the specified domain.
And if I would remove the site operator you can see Google gives the results that can include other websites and not just mine.
Now I want to give you a tip on how to find quickly internal linking opportunities whenever you publish new content on your website.
For example, recently I published a new article on Blogger Outreach, so to find quickly any keywords mentioned within my other articles I can use this operator and add the keyword behind it.
And just like this, I can find all the pages on a website that mention any specific keyword and quickly create relevant internal links for every of my new web page.
Way #7: Using Robots.txt
Another way to find all pages on a website is by using the robots.txt file.
Robots.txt file is used by website owners to manage access for crawlers which sites, directories, subdomains, and categories can and cannot be accessed.
Therefore, by using the robots.txt file you can find any hidden pages on a website from search engines and crawlers.
So, to find the robot.txt file, type into the search bar: example.com/robots.txt for example in my case it would be b2bdigitalmarketers.com/robots.txt.
As you can see, the only disallow directory on my website is the WordPress admin section.
Way #8: Using CMS
The next way to find all website pages is by using your own CMS (content management system) that you run on your website.
Now, when it comes to CMS, there are several different versions you might be using such as WordPress, which is the most popular, and then others like Joomla, Drupal, or HubSpot CMS.
Each of these are slightly different but should give you the same information about your website pages. However, for this tutorial, I’ll show you only how to find all pages on a website using WordPress CMS.
Step #1: Login into your WP dashboard.
Step #2: In your sidebar, you will find a couple of options where you can find different website pages:
- Posts: Here you can find all your blog posts pages.
- Pages: Here you can find all website pages that are not blog posts.
- Portfolio: Here you can find your web pages for your different portfolios like pictures and videos, and some are even used for customers or to showcase any of your projects.
- Others: Then depending on your website purpose there can be other options like forums pages, eCommerce product pages, customer ticket portals, etc.
All these options should have all your pages. You can also look at the “At a Glance” box that should have all your pages and options on where to find all your pages.
It’s also important to look at the different sections of each of these options. For example, the Posts option also has:
- All Posts: This is where you find all your blog posts.
- Categories: This is where you create categories to group your blog posts. Each category has its page.
- Tags: Each blog post can have assigned different tags that work like topics that could be associated with relevant content. Something like #hashtags. Again, each of these tags are having a separate web page.
Quick Tip: Usually, I put noindex for tags as they have very little value for users, thus they would just hurt my overall ranking.
You can also use a WordPress plugin such as Show Pages URL List that allows you to view all the pages on your WordPress site and view the URLs as well.
Way #9: Using Other Tools
And the last way how to find all pages on a website is by using other tools, crawlers, scrapers, etc.
There are perhaps thousands of different tools that can scan your website to find all your pages and I obviously won’t mention every single one of them.
However, there are two more tools that I still want to mention, and that are:
- Screaming frog: This is probably the most famous and best spider website crawler on the market for finding any issues on your website as they provide an extensive amount of information about the crawled website. The tool is especially useful once your website crosses over 1,000 pages. (Also, checkout Screaming Frog alternatives)
- Netpeak Spider: This is another famous and excellent website crawler that helps you side audit your website and of course discover all pages on a website. Their feature-packed tool not only helps you to find all your pages but also optimizes them for SEO.
And then there are other ways and more technical that I did not mention and perhaps even don’t know of.
However, I believe these nine ways should be enough to find all pages on a website.
Now, most likely you will need to use several of these methods to find all pages on a website, and depending on the size of your website you might need to go for the more technical ways.
However, I am confident you should know how to find all website pages on any website.
Did I forget something, did you learn something new, or do you have some comments? Please leave them on my YouTube Channel, CAU!
Also, check out our SEO hub page to find all our SEO Resources.
This article was created by Eduard Dziak and may contain affiliate links. The following were used to optimize the article for the best user and search engine experience include:
- SE Ranking for keyword research and on-page SEO optimization
- Surfer SEO for SEO-friendly content creation for users and search engines.
- Jasper AI for grammar correction and information enhancement.
The article is based on the author’s own experience and knowledge, drawn from both their own work and that of their clients, to provide the latest, proven methods.