This entire tool is thirty-five lines of backend

No crawler framework, no queue, no scraping service. One endpoint reads the sitemap as XML, loops the first fifty URLs, and for each one pulls the title, H1 and description straight out of the parsed document tree before converting the body to Markdown:

.arguments
   url:string
strings.concat
   get-value:x:@.arguments/*/url
   .:/sitemap.xml
http.get:x:-
xml2lambda:x:-/*/content
.result
for-each:x:@xml2lambda/*/urlset/*/url/*/loc/*/#text/[0,50]
   http.get:x:@.dp/#
   html2lambda:x:-/*/content
   .h1
   strings.join:x:@html2lambda/*/html/*/body/**/h1/[0,1]/**/#text
      .:" "
   set-value:x:@.h1
      strings.trim:x:@strings.join
   .title
   set-value:x:@.title
      get-value:x:@html2lambda/*/html/*/head/*/title/*/#text
   .description
   set-value:x:@.description
      get-value:x:"@html2lambda/*/html/*/head/*/meta/*/\\@name/=description/./*/\\@content"
   lambda2html:x:@html2lambda/*/html/*/body
   html2markdown:x:@lambda2html
      url:x:@.arguments/*/url
   unwrap:x:+/*/*/*
   add:x:@.result
      .
         .
            url:x:@.dp/#
            h1:x:@.h1
            title:x:@.title
            description:x:@.description
            markdown:x:@html2markdown
return-nodes:x:@.result/*

That is the whole thing. Describe an endpoint in a sentence, and the Hyperlambda Generator writes and deploys it — then it is a tool your own AI agents can call over MCP. Need just one page instead of a whole site? URL to Markdown does that, with token counts. Run the platform yourself:

curl -fsSL https://hyperlambda.dev/docker-compose.yaml | docker compose -f - up

More free tools

Crawling a whole site into JSON is one of ten free tools here, each with the handful of backend lines that powers it printed on the page. See all of them — no signup on any of it.

Questions

Why does it need a sitemap?

It discovers pages from /sitemap.xml rather than by following links, which keeps the crawl predictable and polite — it only visits pages the site itself publishes as worth indexing. A site without a sitemap cannot be crawled by this tool.

Why only 50 pages?

Because each page is fetched and converted individually, and fifty is roughly where a single request still returns in a sensible amount of time. It takes the first fifty URLs in the sitemap, in the order the sitemap lists them.

What is this actually for?

Feeding a whole site to a model in one go — competitor research, documentation ingestion, migration audits, or building a knowledge base. Markdown costs a fraction of the tokens the original HTML would, and the title, H1 and description give you something to index on without parsing anything yourself.

Does it run JavaScript on the pages?

No. It fetches server-rendered HTML and converts that. Pages that build themselves entirely in the browser will come back close to empty.