# Web Search for AI

Extend [Web Search](/docs/web-search) with content enrichment: semantically scored passages (`extra_snippets`) and/or full page body (`full_content`). Results are reranked by relevance — built for RAG pipelines, LLM agents, and context-augmented generation.

```
GET https://api.staan.ai/v2/search/web
POST https://api.staan.ai/v2/search/web
```

:::note

This is the same endpoint as Web Search — enrichment is activated by adding parameters. All base parameters (`q`, `market`, `offset`) apply here too, including [domain filtering](/docs/web-search#domain-filtering): `site:` / `-site:` operators in `q` on both methods, or `include_domains` / `exclude_domains` arrays on `POST`.

:::

## Quick start

```bash title="Semantic chunks (RAG)"
curl "https://api.staan.ai/v2/search/web?q=vector+database+comparison&extra_snippets=true&max_snippets=5&min_score=0.2" \
  -H "Authorization: Bearer YOUR_API_KEY"
```

```bash title="Full page content"
curl "https://api.staan.ai/v2/search/web?q=vector+database+comparison&full_content=markdown" \
  -H "Authorization: Bearer YOUR_API_KEY"
```

You can restrict enriched results to trusted sources with [domain filtering](/docs/web-search#domain-filtering) — either `site:` operators inside the query, or the `include_domains` array on `POST`:

```bash title="Trusted sources — site: operator (GET or POST)"
curl "https://api.staan.ai/v2/search/web?q=vector+database+comparison+site:qdrant.tech+OR+site:weaviate.io&extra_snippets=true" \
  -H "Authorization: Bearer YOUR_API_KEY"
```

```bash title="Trusted sources — include_domains (POST)"
curl -X POST "https://api.staan.ai/v2/search/web" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "q": "vector database comparison",
    "market": "en-us",
    "extra_snippets": true,
    "max_snippets": 5,
    "include_domains": ["pinecone.io", "qdrant.tech", "weaviate.io"]
  }'
```

:::tip{title="Which one for AI workloads?"}

LLM agents naturally write Google-style operators — a generated query like `pricing site:qdrant.tech` works as-is, no parsing needed on your side. Use `include_domains` when **your application** decides the source list: a curated allowlist in your config acts as a guardrail, enforced regardless of the query the model generates.

:::

```python title="Python — RAG pipeline"
import requests

response = requests.get(
    "https://api.staan.ai/v2/search/web",
    params={
        "q": "vector database comparison",
        "extra_snippets": "true",
        "max_snippets": 5,
        "min_score": 0.2,
        "full_content": "markdown",
    },
    headers={"Authorization": "Bearer YOUR_API_KEY"},
)

for result in response.json()["web"]["results"]:
    print(result["url"])
    for chunk in result.get("extra_snippets", []):
        print(f"  [{chunk['score']:.2f}] {chunk['chunk'][:120]}...")
```

<OpenPlaygroundButton
  server="https://api.staan.ai/v2"
  url="/search/web"
  method="GET"
  headers={[
    {
      name: 'Authorization',
      defaultValue: 'Bearer YOUR_API_KEY',
      defaultActive: true,
    },
  ]}
  queryParams={[
    {
      name: 'q',
      defaultValue: 'vector database comparison',
      defaultActive: true,
    },
    { name: 'market', defaultValue: 'en-us', defaultActive: true },
    { name: 'extra_snippets', defaultValue: 'true', defaultActive: true },
    { name: 'max_snippets', defaultValue: '5', defaultActive: true },
    { name: 'min_score', defaultValue: '0.2', defaultActive: true },
  ]}
>
  Try extra snippets
</OpenPlaygroundButton>

<OpenPlaygroundButton
  server="https://api.staan.ai/v2"
  url="/search/web"
  method="GET"
  headers={[
    {
      name: 'Authorization',
      defaultValue: 'Bearer YOUR_API_KEY',
      defaultActive: true,
    },
  ]}
  queryParams={[
    {
      name: 'q',
      defaultValue: 'vector database comparison',
      defaultActive: true,
    },
    { name: 'market', defaultValue: 'en-us', defaultActive: true },
    { name: 'full_content', defaultValue: 'markdown', defaultActive: true },
  ]}
>
  Try full content
</OpenPlaygroundButton>

## Enrichment parameters

| Parameter        | Type                     | Default | Description                                                                                                           |
| ---------------- | ------------------------ | ------- | --------------------------------------------------------------------------------------------------------------------- |
| `extra_snippets` | `boolean`                | `false` | Fetch result pages and return semantically scored chunks. Triggers reranking.                                         |
| `full_content`   | `"markdown"` \| `"html"` | —       | Return the full page body. `markdown` is recommended for LLM prompts; `html` for raw markup. Also triggers reranking. |
| `max_snippets`   | `number`                 | `3`     | Max scored chunks per URL (1–10). Only with `extra_snippets=true`.                                                    |
| `min_score`      | `number`                 | `0.1`   | Minimum relevance score (0–1) for a chunk to be included. Only with `extra_snippets=true`.                            |

:::tip

Start with `min_score=0.2` and `max_snippets=5` as a baseline. Raise `min_score` to reduce noise; increase `max_snippets` if you need more coverage per page.

:::

## How enrichment works

When `extra_snippets=true` or `full_content` is set, the API runs a post-search pipeline:

1. **SERP** — initial ranked results from the search engine
2. **Fetch** — parallel page downloads (3.5 s per URL, 4 s global timeout)
3. **Chunking** — content split on Markdown structure (H1–H6); long sections split at ~1,200 characters with overlap, heading context preserved
4. **Reranking** — all chunks scored against the query; URLs with the best chunks are promoted
5. **Filtering** — `min_score` and `max_snippets` applied per URL; URLs with no qualifying chunks are moved to the end

:::info

Result order reflects reranking when enrichment is on — not raw SERP order. The `snippet` field always reflects the original search provider preview.

:::

## Enrichment fields

These fields appear on each result when enrichment is active:

| Field            | When present                     | Description                                                         |
| ---------------- | -------------------------------- | ------------------------------------------------------------------- |
| `extra_snippets` | `extra_snippets=true`            | Array of `{ chunk, score }`, ordered by descending relevance score. |
| `full_content`   | `full_content` set               | `{ text, format, length }` — full page body.                        |
| `published_date` | Fetch occurred and date resolved | ISO 8601 publication date.                                          |

### `extra_snippets`

```json
"extra_snippets": [
  { "chunk": "Most relevant content segment from this page...", "score": 0.87 },
  { "chunk": "Next relevant segment...", "score": 0.72 },
  { "chunk": "A less relevant passage.", "score": 0.31 }
]
```

- `chunk`: text segment (~300–1,800 characters) with heading context preserved
- `score`: relevance in `[0, 1]`, descending

`extra_snippets: []` is returned when the page could not be fetched (anti-bot, timeout) or no chunk scored above `min_score`. Such URLs remain in the response, placed after reranked results.

### `full_content`

```json
"full_content": {
  "text": "# Page title\n\nPage body content...",
  "format": "markdown",
  "length": 12345
}
```

Always check `length > 0` before using `text`. On fetch failure, the field is present with `"text": ""` and `"length": 0`.

### `published_date`

```json
"published_date": "2026-04-24T00:00:00.000Z"
```

Only present when a fetch occurred and the HTML extractor resolved a date. Check for presence before use.

## Full response example

```json title="GET ?q=...&extra_snippets=true&full_content=markdown"
{
  "search_id": "01906c9e-7e3f-7000-8000-abc123def456",
  "query": {
    "q": "vector database comparison",
    "market": "en-us",
    "count": 10,
    "offset": 0
  },
  "web": {
    "results": [
      {
        "title": "Comparing vector databases in 2024",
        "url": "https://www.example.com/vector-dbs",
        "snippet": "A deep dive into Pinecone, Weaviate, Qdrant...",
        "display_url": "www.example.com > ai > vector-dbs",
        "hostname": "www.example.com",
        "published_date": "2024-09-10T00:00:00.000Z",
        "extra_snippets": [
          {
            "chunk": "Pinecone is a fully managed vector database...",
            "score": 0.91
          },
          {
            "chunk": "Qdrant offers a self-hosted alternative with...",
            "score": 0.78
          }
        ],
        "full_content": {
          "text": "# Comparing vector databases\n\nThis guide covers...",
          "format": "markdown",
          "length": 22100
        }
      }
    ]
  }
}
```

## Limits and latency

| Limit                      | Value                             |
| -------------------------- | --------------------------------- |
| Rate limit                 | 20 req/s                          |
| Recommended client timeout | 8–10 s when enrichment is enabled |

Plain search (no enrichment) is significantly faster. Budget for extra latency when `extra_snippets` or `full_content` are set.

## Best practices

- Use `extra_snippets=true` for RAG: targeted passages reduce prompt noise and improve answer quality.
- Use [domain filtering](/docs/web-search#domain-filtering) (`include_domains` on `POST`, or `site:` in `q`) to ground answers in a curated set of trusted sources.
- Prefer `full_content=markdown` over `html` for LLM context unless you need raw markup.
- Cite `url` (and chunk text) in generated answers for traceability.
- For end-to-end Q&A without managing retrieval yourself, consider [Answer](/docs/answer-v2).