Rate Limiting

Complete reference for handling API rate limits and quotas.

Limits Matrix

Method / Endpoint	Limit	Description
POST /v1/scans	10/min	Create new URL threat detection scans
GET /v1/scans/:id	60/min	Check scan status and retrieve results
POST /v1/domain-risks	20/min	Analyze domains (synchronous)
GET /v1/domain-risks/:id	20/min	Retrieve domain risk analysis results
POST /v2/domain-classification	10/min	Classify or retrieve domain classification results
GET /v2/domain-classification/:domain	60/min	Retrieve domain classification results

Rate Limit Headers

Header	Description	Example
`X-RateLimit-Limit`	The maximum number of requests allowed per minute for this endpoint.	60
`X-RateLimit-Remaining`	The number of requests you can make in the current sliding window before hitting the limit.	59
`X-RateLimit-Reset`	Unix timestamp (seconds since epoch) indicating when your oldest request will expire from the sliding window.	1678892345

Handling 429 Errors

429

Too Many Requests

This error occurs when you exceed the rate limit for your organization. The response will include a Retry-After header indicating how many seconds to wait.

Resolution

Read the Retry-After header and pause your requests for that duration. Implementing exponential backoff is recommended to ensure your system recovers gracefully.

HTTP/1.1 429 Too Many Requests
X-RateLimit-Limit: 10
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1705315800
Retry-After: 42

{
  "error": "rate_limit_exceeded",
  "message": "Rate limit exceeded. Try again in 42 seconds.",
  "limit": 10,
  "retry_after": 42
}

Best Practices

Monitor Headers

Check the X-RateLimit-Remaining header in every response to track your usage and avoid hitting limits unexpectedly.

// Check remaining rate limit before making requests
const response = await fetch('https://api.urlert.com/v1/scans/some-id', {
  headers: { 'Authorization': 'Bearer u_sk_your_token' }
});

const remaining = parseInt(response.headers.get('X-RateLimit-Remaining'));
const limit = parseInt(response.headers.get('X-RateLimit-Limit'));

if (remaining < limit * 0.1) {
  console.warn(`Low rate limit: ${remaining}/${limit} remaining`);
  // Implement your throttling logic
}

Exponential Backoff

When you encounter a 429 error, don't retry immediately. Wait for the referenced time, and if errors persist, increase the wait time exponentially.

import requests
import time

def make_api_request_with_backoff(url, headers, data):
    max_retries = 3
    retry_count = 0
    
    while retry_count < max_retries:
        response = requests.post(url, headers=headers, json=data)
        
        if response.status_code == 429:
            # Rate limit exceeded
            retry_after = int(response.headers.get('Retry-After', 60))
            print(f"Rate limited. Waiting {retry_after} seconds...")
            time.sleep(retry_after)
            retry_count += 1
            continue
        
        # Success or other error
        return response
    
    raise Exception("Max retries exceeded")

# Usage
response = make_api_request_with_backoff(
    'https://api.urlert.com/v1/scans',
    headers={'Authorization': 'Bearer u_sk_your_token'},
    data={'url': 'https://example.com'}
)

async function makeRequestWithBackoff(url, options) {
  const maxRetries = 3;
  let retryCount = 0;
  
  while (retryCount < maxRetries) {
    const response = await fetch(url, options);
    
    if (response.status === 429) {
      // Rate limit exceeded
      const retryAfter = parseInt(response.headers.get('Retry-After') || '60');
      console.log(`Rate limited. Waiting ${retryAfter} seconds...`);
      await new Promise(resolve => setTimeout(resolve, retryAfter * 1000));
      retryCount++;
      continue;
    }
    
    // Success or other error
    return response;
  }
  
  throw new Error('Max retries exceeded');
}

// Usage
const response = await makeRequestWithBackoff(
  'https://api.urlert.com/v1/scans',
  {
    method: 'POST',
    headers: {
      'Authorization': 'Bearer u_sk_your_token',
      'Content-Type': 'application/json'
    },
    body: JSON.stringify({ url: 'https://example.com' })
  }
);

# Simple retry with exponential backoff
for i in {1..3}; do
  response=$(curl -X POST https://api.urlert.com/v1/scans \
    -H "Authorization: Bearer u_sk_your_token" \
    -H "Content-Type: application/json" \
    -d '{"url": "https://example.com"}' \
    -w "\n%{http_code}" -s)
  
  http_code=$(echo "$response" | tail -n1)
  
  if [ "$http_code" = "429" ]; then
    retry_after=$(echo "$response" | grep -i "retry-after" | cut -d' ' -f2)
    echo "Rate limited. Waiting $retry_after seconds..."
    sleep "$retry_after"
  else
    echo "$response"
    break
  fi
done