Firecrawl Map

What Is This?

Overview

Firecrawl Map is a command-line skill that discovers and lists all accessible URLs on a given website. By crawling a domain and returning a structured list of pages, it gives developers and researchers a clear picture of a site's structure without requiring manual browsing or custom scraping scripts. The tool integrates directly into development workflows through the Firecrawl CLI, making URL discovery a repeatable, scriptable operation.

The skill supports optional search filtering, which means you can narrow results to pages matching a specific keyword or pattern. Instead of retrieving thousands of URLs and sorting through them manually, you can query for pages related to a particular topic, product, or section. This targeted approach saves significant time when working with large or complex websites.

Firecrawl Map is part of the broader Firecrawl toolset, which is designed to support web data extraction and site analysis tasks. It works through a simple command interface and returns results that can be piped into other tools, stored in files, or used directly in automation pipelines.

Who Should Use This

• Web developers who need to audit a site's URL structure before a migration, redesign, or SEO review.
• QA engineers who want to enumerate all pages on a staging or production site for automated testing coverage.
• Data engineers who need a complete list of URLs as input for a scraping or indexing pipeline.

Why Use It?

Problems It Solves

• Unknown page locations: When you know a site contains the information you need but cannot find the exact URL, Firecrawl Map returns the full list so you can locate it quickly.
• Manual sitemap limitations: Official sitemaps are often incomplete or outdated. Firecrawl Map crawls the live site and reflects its actual current structure.
• Slow manual audits: Clicking through a large site to document its pages is time-consuming. This skill automates the process in seconds.

Core Highlights

• Crawls an entire domain and returns all discovered URLs
• Supports keyword-based filtering to narrow results
• Runs from the command line with a single command
• Integrates with Bash scripts and automation pipelines
• Works with both firecrawl CLI and npx firecrawl for environments without a global install
• Returns structured output suitable for further processing
• Fast execution compared to manual browsing or custom crawlers

How to Use It?

Basic Usage

To map all URLs on a website, run the following command:

firecrawl map https://example.com

To filter results by a search term, add the --search flag:

firecrawl map https://example.com --search "pricing"

If you do not have Firecrawl installed globally, use npx:

npx firecrawl map https://docs.example.com --search "authentication"

Specific Scenarios

Scenario 1: Locating a specific page on a large documentation site

You know a framework's documentation covers rate limiting, but you cannot find the page. Run the map command with a relevant search term to get the exact URL immediately.

firecrawl map https://docs.someframework.io --search "rate limiting"

Scenario 2: Generating a URL list for automated testing

Before running a test suite against a staging environment, generate a complete list of pages and save it to a file for the test runner to consume.

firecrawl map https://staging.example.com > urls.txt

Real-World Examples

• A developer preparing for a site migration runs Firecrawl Map on the old domain to produce a complete URL inventory, then compares it against the new site after launch.
• A data engineer uses the output of Firecrawl Map as the seed list for a content extraction job, ensuring no pages are missed.
• A QA engineer maps a client's e-commerce site before a regression test cycle to confirm all product and category pages are accessible.

When to Use It?

Use Cases

• Auditing a website's full URL structure before a redesign or migration
• Finding the exact URL of a page when you only know the domain
• Generating input URL lists for web scraping or indexing workflows
• Verifying that all expected pages are live after a deployment
• Mapping a competitor's or third-party site for research purposes
• Identifying orphaned or unexpected pages on a production domain
• Supporting security assessments by enumerating publicly accessible endpoints

Important Notes

Requirements

• Node.js must be installed to use the npx firecrawl variant.
• A valid Firecrawl API key is required for authenticated requests.
• The target website must be publicly accessible or reachable from your network.