Rate Limiting Guide

Understanding rate limits, headers, and best practices for the URLert API.

Overview

All API endpoints are rate limited on a per-organization, per-minute basis. Rate limits are enforced using a sliding window algorithm to ensure fair usage and maintain service quality for all users.

Rate limits reset on a sliding window basis, not at fixed intervals. This means each request's impact on your limit expires exactly 60 seconds after it was made.

Rate Limits by API

Operation	Limit	Description
POST /v1/scans	10/minute	Create new URL threat detection scans
GET /v1/scans/:id	60/minute	Check scan status and retrieve results
POST /v1/domain-risks	20/minute	Analyze domains (synchronous)
GET /v1/domain-risks/:id	20/minute	Retrieve domain risk analysis results

Rate Limit Headers

Every API response includes headers that help you track your rate limit status and implement proper throttling:

X-RateLimit-Limit: 60
X-RateLimit-Remaining: 45
X-RateLimit-Reset: 1705315800

X-RateLimit-Limit

Maximum Requests

The maximum number of requests allowed per minute for this endpoint.

X-RateLimit-Remaining

Remaining Requests

The number of requests you can make in the current sliding window before hitting the limit.

X-RateLimit-Reset

Reset Timestamp

Unix timestamp (seconds since epoch) indicating when your oldest request will expire from the sliding window, freeing up capacity.

Handling 429 Errors

When you exceed your rate limit, the API returns a 429 Too Many Requests response with details about when you can retry:

HTTP/1.1 429 Too Many Requests
X-RateLimit-Limit: 10
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1705315800
Retry-After: 42

{
  "error": "rate_limit_exceeded",
  "message": "Rate limit exceeded. Try again in 42 seconds.",
  "limit": 10,
  "retry_after": 42
}

Important: Always respect the Retry-After header value.

Best Practices

1 Monitor Rate Limit Headers

Check X-RateLimit-Remaining in every response to track your usage and avoid hitting limits.

// Check remaining rate limit before making requests
const response = await fetch('https://api.urlert.com/v1/scans/some-id', {
  headers: { 'Authorization': 'Bearer u_sk_your_token' }
});

const remaining = parseInt(response.headers.get('X-RateLimit-Remaining'));
const limit = parseInt(response.headers.get('X-RateLimit-Limit'));

if (remaining < limit * 0.1) {
  console.warn(`Low rate limit: ${remaining}/${limit} remaining`);
  // Implement your throttling logic
}

2 Implement Exponential Backoff

When you receive a 429 error, wait for the duration specified in the Retry-After header before retrying.

import requests
import time

def make_api_request_with_backoff(url, headers, data):
    max_retries = 3
    retry_count = 0
    
    while retry_count < max_retries:
        response = requests.post(url, headers=headers, json=data)
        
        if response.status_code == 429:
            # Rate limit exceeded
            retry_after = int(response.headers.get('Retry-After', 60))
            print(f"Rate limited. Waiting {retry_after} seconds...")
            time.sleep(retry_after)
            retry_count += 1
            continue
        
        # Success or other error
        return response
    
    raise Exception("Max retries exceeded")

# Usage
response = make_api_request_with_backoff(
    'https://api.urlert.com/v1/scans',
    headers={'Authorization': 'Bearer u_sk_your_token'},
    data={'url': 'https://example.com'}
)

async function makeRequestWithBackoff(url, options) {
  const maxRetries = 3;
  let retryCount = 0;
  
  while (retryCount < maxRetries) {
    const response = await fetch(url, options);
    
    if (response.status === 429) {
      // Rate limit exceeded
      const retryAfter = parseInt(response.headers.get('Retry-After') || '60');
      console.log(`Rate limited. Waiting ${retryAfter} seconds...`);
      await new Promise(resolve => setTimeout(resolve, retryAfter * 1000));
      retryCount++;
      continue;
    }
    
    // Success or other error
    return response;
  }
  
  throw new Error('Max retries exceeded');
}

// Usage
const response = await makeRequestWithBackoff(
  'https://api.urlert.com/v1/scans',
  {
    method: 'POST',
    headers: {
      'Authorization': 'Bearer u_sk_your_token',
      'Content-Type': 'application/json'
    },
    body: JSON.stringify({ url: 'https://example.com' })
  }
);

# Simple retry with exponential backoff
for i in {1..3}; do
  response=$(curl -X POST https://api.urlert.com/v1/scans \
    -H "Authorization: Bearer u_sk_your_token" \
    -H "Content-Type: application/json" \
    -d '{"url": "https://example.com"}' \
    -w "\n%{http_code}" -s)
  
  http_code=$(echo "$response" | tail -n1)
  
  if [ "$http_code" = "429" ]; then
    retry_after=$(echo "$response" | grep -i "retry-after" | cut -d' ' -f2)
    echo "Rate limited. Waiting $retry_after seconds..."
    sleep "$retry_after"
  else
    echo "$response"
    break
  fi
done

3 Cache Results When Possible

• For read operations (GET requests), implement caching to reduce API calls
• Store scan results locally if you need to reference them multiple times within a short period
• Cache time-to-live (TTL) depends on your use case, but consider 5-15 minutes for URL scan results

Frequently Asked Questions

What happens if I exceed the rate limit?

You'll receive a 429 error with a Retry-After header. Wait for the specified time before retrying.

Are rate limits per API key or per organization?

Rate limits are enforced per organization. All API keys for the same organization share the same rate limit pool.

Can I request higher rate limits?

Yes. If you have specific high-volume requirements, please contact us at support@urlert.com to discuss your use case and custom rate limits.

Do rate limit headers show my exact remaining requests?

Yes. The X-RateLimit-Remaining header shows your exact remaining capacity in the current sliding window. This updates with every request.