# Jina AI Reader API Documentation

## Overview

The **Jina AI Reader API** converts any URL into LLM-friendly text by extracting clean, readable content from web pages. It handles complex HTML, JavaScript-rendered pages, PDFs, and images with automatic captioning.

**Official Documentation**: https://jina.ai/reader/  
**GitHub Repository**: https://github.com/jina-ai/reader

---

## Base Endpoints

| Endpoint | Purpose | Description |
|----------|---------|-------------|
| `https://r.jina.ai/` | URL Reader | Convert any URL to clean markdown text |
| `https://s.jina.ai/` | Web Search | Search the web and return top 5 results as LLM-friendly text |

---

## Authentication

### API Key (Optional but Recommended)

- **Without API Key**: 20 requests per minute (RPM), rate-limited by IP
- **With Free API Key**: 500 RPM, tracked by API key
- **With Premium API Key**: 5000 RPM

**How to use API Key**:
```bash
curl -H "Authorization: Bearer YOUR_API_KEY" https://r.jina.ai/https://example.com
```

Or via header:
```bash
curl -H "X-API-Key: YOUR_API_KEY" https://r.jina.ai/https://example.com
```

Get your free API key at: https://jina.ai/api-dashboard/

---

## Basic Usage

### Simple GET Request

The simplest way to use the Reader API is to prepend `r.jina.ai/` to any URL:

```bash
curl https://r.jina.ai/https://www.example.com
```

### POST Request (for URLs with hash routing)

For Single Page Applications (SPAs) with hash-based routing (e.g., `#/route`), use POST:

```bash
curl -X POST https://r.jina.ai/ \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/#/route"}'
```

---

## Key Features

### 1. **Clean Content Extraction**
- Removes ads, navigation, footers, and other clutter
- Converts HTML to clean markdown
- Optimized for LLM input (average 7.9s latency)

### 2. **PDF Support**
```bash
curl https://r.jina.ai/https://arxiv.org/pdf/2301.00001.pdf
```

### 3. **Image Captioning**
Automatically captions images using vision-language models:
```bash
curl -H "X-With-Generated-Alt: true" https://r.jina.ai/https://example.com
```

### 4. **Web Search (SERP API)**
Search the web and get top 5 results:
```bash
curl "https://s.jina.ai/?q=latest+AI+news"
```

Returns 5 results with `title`, `url`, and `content` for each.

### 5. **Streaming Mode**
For pages with delayed content loading:
```bash
curl -H "Accept: text/event-stream" https://r.jina.ai/https://example.com
```

The last chunk contains the most complete result.

### 6. **JSON Response**
```bash
curl -H "Accept: application/json" https://r.jina.ai/https://example.com
```

Returns:
```json
{
  "url": "https://example.com",
  "title": "Page Title",
  "content": "Clean markdown content..."
}
```

---

## Advanced Headers

### Content Control

| Header | Values | Description |
|--------|--------|-------------|
| `X-Respond-With` | `markdown`, `html`, `text`, `screenshot` | Control output format |
| `X-With-Generated-Alt` | `true` / `false` | Enable image captioning |
| `X-Remove-Images` | `true` / `false` | Remove all images from output |

### Caching

| Header | Values | Description |
|--------|--------|-------------|
| `X-No-Cache` | `true` / `false` | Bypass cached content (default cache: 3600s) |
| `X-Cache-Tolerance` | Integer (seconds) | Custom cache tolerance |
| `X-No-Track` | `true` / `false` | Don't cache or track this request |

### Page Rendering

| Header | Values | Description |
|--------|--------|-------------|
| `X-Timeout` | Integer (seconds) | Max page load wait time |
| `X-Wait-For-Selector` | CSS selector | Wait for specific element before returning |
| `X-Target-Selector` | CSS selector | Extract only content from specific element |
| `X-Remove-Selector` | CSS selector | Remove specific elements (headers, footers, etc.) |

### Authentication & Proxy

| Header | Values | Description |
|--------|--------|-------------|
| `X-Set-Cookie` | Cookie string | Forward cookies to target URL (disables caching) |
| `X-Proxy-URL` | Proxy URL | Use custom proxy server |
| `X-Locale` | Locale code | Set browser locale (e.g., `en-US`, `fr-FR`) |

### Advanced Features

| Header | Values | Description |
|--------|--------|-------------|
| `X-With-Links-Summary` | `true` / `false` | Add "Buttons & Links" section at the end |
| `X-With-Images-Summary` | `true` / `false` | Add "Images" section at the end |
| `X-With-Iframe` | `true` / `false` | Extract content from iframes |
| `X-With-Shadow-Dom` | `true` / `false` | Extract content from Shadow DOM |

---

## Code Examples

### PHP (Laravel)

```php
use Illuminate\Support\Facades\Http;

class JinaContentExtractor
{
    private $apiKey;
    private $baseUrl = 'https://r.jina.ai/';
    
    public function __construct()
    {
        $this->apiKey = config('services.jina.api_key');
    }
    
    public function fetchContent(string $url, array $options = [])
    {
        $headers = [
            'Authorization' => 'Bearer ' . $this->apiKey,
            'Accept' => 'application/json',
        ];
        
        // Add custom headers
        if (isset($options['timeout'])) {
            $headers['X-Timeout'] = $options['timeout'];
        }
        
        if (isset($options['with_images']) && $options['with_images']) {
            $headers['X-With-Generated-Alt'] = 'true';
        }
        
        if (isset($options['target_selector'])) {
            $headers['X-Target-Selector'] = $options['target_selector'];
        }
        
        $response = Http::withHeaders($headers)
            ->timeout(60)
            ->get($this->baseUrl . $url);
        
        if ($response->successful()) {
            return $response->json();
        }
        
        throw new \Exception('Jina API request failed: ' . $response->body());
    }
    
    public function search(string $query)
    {
        $headers = [
            'Authorization' => 'Bearer ' . $this->apiKey,
            'Accept' => 'application/json',
        ];
        
        $response = Http::withHeaders($headers)
            ->get('https://s.jina.ai/', ['q' => $query]);
        
        return $response->json();
    }
}
```

### Usage with Caching (Laravel)

```php
use Illuminate\Support\Facades\Cache;

class JinaService
{
    protected $extractor;
    
    public function __construct(JinaContentExtractor $extractor)
    {
        $this->extractor = $extractor;
    }
    
    public function getContentWithCache(string $url, int $cacheDays = 30)
    {
        $cacheKey = 'jina_content_' . md5($url);
        
        return Cache::remember($cacheKey, now()->addDays($cacheDays), function () use ($url) {
            return $this->extractor->fetchContent($url, [
                'timeout' => 30,
                'with_images' => false,
            ]);
        });
    }
}
```

### JavaScript/Node.js

```javascript
const axios = require('axios');

class JinaReader {
  constructor(apiKey) {
    this.apiKey = apiKey;
    this.baseUrl = 'https://r.jina.ai/';
  }

  async fetchContent(url, options = {}) {
    const headers = {
      'Authorization': `Bearer ${this.apiKey}`,
      'Accept': 'application/json',
    };

    if (options.timeout) {
      headers['X-Timeout'] = options.timeout;
    }

    if (options.withImages) {
      headers['X-With-Generated-Alt'] = 'true';
    }

    const response = await axios.get(this.baseUrl + url, { headers });
    return response.data;
  }

  async search(query) {
    const headers = {
      'Authorization': `Bearer ${this.apiKey}`,
      'Accept': 'application/json',
    };

    const response = await axios.get('https://s.jina.ai/', {
      params: { q: query },
      headers,
    });

    return response.data;
  }
}

// Usage
const reader = new JinaReader(process.env.JINA_API_KEY);
const content = await reader.fetchContent('https://example.com');
console.log(content);
```

### Python

```python
import requests
import hashlib
from datetime import datetime, timedelta

class JinaReader:
    def __init__(self, api_key):
        self.api_key = api_key
        self.base_url = 'https://r.jina.ai/'
        self.cache = {}
    
    def fetch_content(self, url, options=None):
        headers = {
            'Authorization': f'Bearer {self.api_key}',
            'Accept': 'application/json',
        }
        
        if options:
            if 'timeout' in options:
                headers['X-Timeout'] = str(options['timeout'])
            if options.get('with_images'):
                headers['X-With-Generated-Alt'] = 'true'
            if 'target_selector' in options:
                headers['X-Target-Selector'] = options['target_selector']
        
        response = requests.get(self.base_url + url, headers=headers, timeout=60)
        response.raise_for_status()
        return response.json()
    
    def search(self, query):
        headers = {
            'Authorization': f'Bearer {self.api_key}',
            'Accept': 'application/json',
        }
        
        response = requests.get('https://s.jina.ai/', params={'q': query}, headers=headers)
        response.raise_for_status()
        return response.json()

# Usage
reader = JinaReader(api_key='your_api_key')
content = reader.fetch_content('https://example.com')
print(content['content'])
```

---

## Rate Limits & Pricing

### Rate Limits

| Tier | RPM | TPM | Cost |
|------|-----|-----|------|
| No API Key | 20 | N/A | Free |
| Free API Key | 500 | Counted | Pay per token |
| Premium API Key | 5000 | Counted | Pay per token |

### Token Counting

- **Reader API (`r.jina.ai`)**: Counts tokens in the **output** response
- **Search API (`s.jina.ai`)**: Fixed cost starting from 10,000 tokens per request
- Tokens are counted using standard tokenization methods

### Pricing

- Free tier available with rate limits
- Premium pricing based on token usage
- Check current pricing at: https://jina.ai/reader/ (Pricing tab)

---

## Best Practices for SEO Tool Implementation

### 1. **Implement Aggressive Caching**

```php
// Cache Jina responses for 30 days
$cacheKey = 'jina_' . md5($url);
$cacheDuration = now()->addDays(30);

$content = Cache::remember($cacheKey, $cacheDuration, function () use ($url) {
    return $jinaClient->fetchContent($url);
});
```

### 2. **Rate Limiting**

```php
// Add 100ms delay between requests
usleep(100000); // 100ms in microseconds
```

### 3. **Error Handling**

```php
try {
    $content = $jinaClient->fetchContent($url);
} catch (\Exception $e) {
    Log::error('Jina API error', [
        'url' => $url,
        'error' => $e->getMessage(),
    ]);
    
    // Mark page as failed in database
    $page->update(['status' => 'failed', 'error' => $e->getMessage()]);
    
    // Continue with next page
    return null;
}
```

### 4. **Content Extraction for E-commerce**

For product pages, focus on descriptions:

```php
$content = $jinaClient->fetchContent($url, [
    'target_selector' => '.product-description, .product-features',
    'timeout' => 20,
]);

// Limit to 1000 characters
$cleanContent = substr($content['content'], 0, 1000);
```

### 5. **Batch Processing with Queue**

```php
// Laravel Job
class ProcessPageAnalysis implements ShouldQueue
{
    public function handle(JinaService $jinaService)
    {
        $pages = Page::where('status', 'pending')
            ->limit(10)
            ->get();
        
        foreach ($pages as $page) {
            try {
                $content = $jinaService->getContentWithCache($page->url);
                $page->update([
                    'content' => $content['content'],
                    'status' => 'completed',
                ]);
                
                usleep(100000); // 100ms delay
            } catch (\Exception $e) {
                $page->update(['status' => 'failed']);
            }
        }
    }
}
```

---

## Common Use Cases

### 1. **Extract Product Descriptions**
```bash
curl -H "X-Target-Selector: .product-description" \
     https://r.jina.ai/https://shop.example.com/product/123
```

### 2. **Get Blog Content Only**
```bash
curl -H "X-Target-Selector: article" \
     -H "X-Remove-Selector: nav,footer,.sidebar" \
     https://r.jina.ai/https://blog.example.com/post
```

### 3. **Wait for Dynamic Content**
```bash
curl -H "X-Wait-For-Selector: .main-content" \
     -H "X-Timeout: 30" \
     https://r.jina.ai/https://spa-website.com
```

### 4. **Search and Extract**
```bash
# Search for "best SEO tools 2024"
curl "https://s.jina.ai/?q=best+SEO+tools+2024"
```

Returns top 5 results with clean content from each page.

---

## Troubleshooting

### Issue: Incomplete Content

**Solution**: Use streaming mode or increase timeout
```bash
curl -H "Accept: text/event-stream" \
     -H "X-Timeout: 30" \
     https://r.jina.ai/https://example.com
```

### Issue: 404 or Blocked

**Solution**: Check if the URL is accessible, or use a proxy
```bash
curl -H "X-Proxy-URL: http://your-proxy.com:8080" \
     https://r.jina.ai/https://blocked-site.com
```

### Issue: Rate Limit Exceeded

**Solution**: 
1. Add API key to increase limit from 20 to 500 RPM
2. Implement caching to reduce API calls
3. Add delays between requests

### Issue: Content Behind Login

**Solution**: Forward cookies
```bash
curl -H "X-Set-Cookie: session=abc123; auth=xyz789" \
     https://r.jina.ai/https://members-only.com
```

Note: Requests with cookies are not cached.

---

## Additional Resources

- **Official Documentation**: https://jina.ai/reader/
- **API Dashboard**: https://jina.ai/api-dashboard/
- **GitHub Repository**: https://github.com/jina-ai/reader
- **Community Support**: https://discord.jina.ai/

---

## Summary for SEO Tool

For your SEO keyword optimizer project:

1. **Use `r.jina.ai/` endpoint** to extract clean content from URLs in Screaming Frog CSV
2. **Implement 30-day caching** using MD5 hash of URL as cache key
3. **Add 100ms delay** between requests to respect rate limits
4. **Use `X-Target-Selector`** for product/collection pages to focus on relevant content
5. **Limit content to 1000 characters** for AI processing efficiency
6. **Handle errors gracefully** and continue processing other pages
7. **Get a free API key** to increase rate limit to 500 RPM

**Cost Estimate**: For 100-page analysis with caching, expect minimal Jina API costs (most requests will be cached on re-analysis).