Search Engine Spider Simulator

What does a search engine spider see on your page?

A search engine spider first sees the raw HTML your server returns, before any JavaScript runs. This simulator fetches that HTML while identifying itself with Googlebot's user-agent string, then shows you the HTTP status, the <title>, and the first 300 characters of text inside <body>. If your main content isn't in that text, a crawler that doesn't execute scripts won't find it.

It's a fast way to answer "is my content server-rendered?" without opening DevTools and disabling JavaScript.

Two contrasting results

A server-rendered product page:

Status: 200
Title: Trail Running Shoes | Example Outfitters
Visible text preview (first 300 characters, as a crawler that does not run JavaScript would see it):
Trail running shoes Grippy outsoles, rock plates and a drop between 4 and 8 mm. Road shoes Old model Reviews Email Partner sale

A client-rendered dashboard whose body is an empty <div id="app"> plus a script tag:

Status: 200
Title: Dashboard
Visible text preview (first 300 characters, as a crawler that does not run JavaScript would see it):
(no visible text found — this page likely renders its content with JavaScript)

The first page's heading, paragraph and link text all exist in the HTML. The second page's content will only appear after a browser downloads and runs the script.

What the request looks like

The URL you enter goes to our server, which sends a GET request with this user-agent header:

Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

It follows up to ten redirects, waits up to eight seconds for each response, and reads at most 2,000,000 bytes. It refuses file:, ftp: and other non-web schemes, and any hostname that resolves to a private or reserved IP. No cookies are sent and no JavaScript, CSS or images are loaded.

To produce the preview, the text inside <body> is collected with <script>, <style> and <template> content left out, a space is inserted wherever a block element such as a heading, paragraph, list item or table cell starts or ends, runs of whitespace become one space, and the first 300 characters are kept.

How this differs from the real Googlebot

Be clear about three gaps before drawing conclusions:

Google's JavaScript SEO basics describes the crawl, render and index phases, and says rendering can wait in a queue. The same page recommends server-side or pre-rendering because not all bots can run JavaScript. So an empty preview here doesn't prove Google sees an empty page, but it does mean you're depending on rendering, and any crawler that skips JavaScript gets nothing.

The user-agent point cuts the other way too. Google warns that the user-agent header is often spoofed and tells site owners to verify Googlebot with a reverse DNS lookup on the requesting IP. A site or CDN that does that will see a "Googlebot" request from a non-Google address and may block it with a 403. That's correct behaviour on their part, not a problem with your page. You can try the same check yourself with the IP to hostname tool.

Reading the text preview

• Code is left out. Inline <script> and <style> blocks, such as a framework's embedded state (window.__STATE__={...}), never appear in the preview. To see how much of the page's weight they make up, use the code-to-text ratio checker.
• Blocks are separated, inline text isn't. <h1>Latest posts</h1><p>New reads as "Latest posts New", while Hel<b>lo</b> stays "Hello", as a browser would show it.
• Hidden elements still count. Text hidden with CSS or the hidden attribute is still in the HTML, so it appears here.
• <noscript> text counts. A "Please enable JavaScript" message in a <noscript> block is real HTML text and appears in the preview.
• Only the opening. 300 characters is usually the header and navigation. If your menu is long, the preview may never reach the main content even when it's there; view source to check further down.

Using it to debug rendering problems

1. Run the page here. If the preview is empty or only navigation, your content is probably client-rendered.
2. Check the status line. A 403 or 503 can mean bot protection is challenging the Googlebot user agent.
3. Confirm the title. If it's generic ("React App") here but correct in your browser, the title is being set by JavaScript.
4. Compare with the on-page SEO checker, which reads the same raw HTML for the meta description, robots and canonical tags.
5. Make sure the page isn't blocked from crawling in the first place. The robots.txt generator helps you write rules that leave your content pages open.