# Web Scraper

Fetch any URL and extract structured data including title, meta description, keywords, favicon, social links, all links on the page, emails, phone numbers, images, body text, and detected languages. Meta keywords are always a JSON array of strings: content="php,,laravel" returns data.keywords=["php","laravel"]. Empty entries are omitted, remaining order and duplicates are preserved, and missing or all-empty metadata returns []. Set javascript=true for client-rendered pages. In JSON responses, url is the requested URL (with a protocol added if omitted), and final_url is the destination after redirects. Relative links, image URLs, and favicons are resolved against final_url, preserving non-default ports. Query-only references keep the page path: ?page=2 at https://example.com/catalog/page?old=1 resolves to https://example.com/catalog/page?page=2. Relative dot segments are normalized: /catalog/../item becomes /item and /catalog/./item becomes /catalog/item. Reference queries and fragments are preserved. Default HTTP(S) ports may be omitted from resolved relative URLs. Only HTTP(S) destinations are returned; absolute HTTP(S) references are left unchanged. If the destination URL is unavailable, the URL used for the fetch is returned. Returns the site's HTTP status code separately so you can distinguish site errors (404, 500) from infrastructure errors. Use response_type=markdown to get only the page content as clean markdown.

- **Documentation:** [https://staging.scrappa.co/docs/web-scraper/web_scraper](https://staging.scrappa.co/docs/web-scraper/web_scraper)
- **API group:** Web Scraper API
- **Endpoint:** `GET https://scrappa.co/api/web-scraper`

## Authentication

Send your Scrappa API key in the `X-API-KEY` request header. Paid endpoints also support accountless x402 payments when called without an API key.

## Parameters

| Parameter | Type | Required | Description |
| --- | --- | --- | --- |
| `url` | string | Yes | The URL to scrape (must include protocol, e.g., https://example.com) |
| `include_html` | boolean | No | Set to true to include the raw HTML in the response. Default: false. Ignored when response_type=markdown |
| `javascript` | boolean | No | Set to true to execute page JavaScript before extracting content. Rendering uses Obscura first and can fall back to Cloudflare Browser Rendering if navigation fails. Default: false |
| `response_type` | string | No | Set to "markdown" to return only the page content as clean markdown text. When set to markdown, only the markdown content is returned (not JSON). Default: json |

## Example request

```bash
#!/bin/bash

curl -X GET \
    -H "x-api-key: YOUR_API_KEY_HERE" \
    "https://scrappa.co/api/web-scraper?url=https%3A%2F%2Fexample.com&include_html=0&javascript=0"
```

## Example response

```json
{
    "success": true,
    "site_status_code": 200,
    "url": "https://example.com",
    "final_url": "https://example.com",
    "data": {
        "title": "Example Domain",
        "description": "This domain is for use in illustrative examples.",
        "keywords": [],
        "favicon": "https://example.com/favicon.ico",
        "social_links": {
            "linkedin": null,
            "twitter": null,
            "facebook": null,
            "instagram": null,
            "youtube": null,
            "tiktok": null
        },
        "extracted_keywords": [
            "domain",
            "example",
            "illustrative"
        ],
        "links": [
            "https://www.iana.org/domains/example"
        ],
        "emails": [],
        "phone_numbers": [],
        "images": [],
        "body_text": "Example Domain This domain is for use in illustrative examples in documents.",
        "languages_detected": [
            "en"
        ],
        "html": null
    }
}
```

## Errors

| Status | Error | Description |
| --- | --- | --- |
| 400 | Invalid URL | The url parameter is not a valid absolute URL or domain. |
| 422 | Validation Error | Required parameters failed request validation. |
| 502 | The upstream service is temporarily unavailable. Please retry shortly. | The upstream service is temporarily unavailable. Please retry shortly. |
| 504 | Connection Timeout | The upstream service is temporarily unavailable. Please retry shortly. |
| 503 | JavaScript Renderer Unavailable | JavaScript rendering is temporarily unavailable or its bounded queue is full. Retry later; failed requests are not billed. |

## More Scrappa resources

- [API documentation](https://staging.scrappa.co/docs)
- [Full LLM-readable API reference](https://staging.scrappa.co/llms-full.txt)
- [OpenAPI specification](https://staging.scrappa.co/docs/api.json)
- [Pricing](https://staging.scrappa.co/pricing)