firecrawl-reliability-patterns
Implement FireCrawl reliability patterns including circuit breakers, idempotency, and graceful degradation. Use when building fault-tolerant FireCrawl integrations, implementing retry strategies, or adding resilience to production FireCrawl services. Trigger with phrases like "firecrawl reliability", "firecrawl circuit breaker", "firecrawl idempotent", "firecrawl resilience", "firecrawl fallback", "firecrawl bulkhead".
- License 许可
- MIT
- 最近更新
- 2026-11-15
- Stars
- ★ 2.0k
- Forks
- 233
01 前言 Front Matter
- name
- firecrawl-reliability-patterns
- author
- Jeremy Longshore <jeremy@intentsolutions.io>
- license
- MIT
- version
- 1.0.0
- description
- Implement FireCrawl reliability patterns including circuit breakers, idempotency, and graceful degradation. Use when building fault-tolerant FireCrawl integrations, implementing retry strategies, or adding resilience to production FireCrawl services. Trigger with phrases like "firecrawl reliability", "firecrawl circuit breaker", "firecrawl idempotent", "firecrawl resilience", "firecrawl fallback", "firecrawl bulkhead".
- allowed-tools
- Read, Write, Edit
03 SKILL.md 正文
FireCrawl Reliability Patterns
Overview
Production-grade reliability patterns for FireCrawl integrations.
Prerequisites
- Understanding of circuit breaker pattern
- opossum or similar library installed
- Queue infrastructure for DLQ
- Caching layer for fallbacks
Circuit Breaker
import CircuitBreaker from 'opossum';
const firecrawlBreaker = new CircuitBreaker(
async (operation: () => Promise<any>) => operation(),
{
timeout: 30000,
errorThresholdPercentage: 50,
resetTimeout: 30000,
volumeThreshold: 10,
}
);
// Events
firecrawlBreaker.on('open', () => {
console.warn('FireCrawl circuit OPEN - requests failing fast');
alertOps('FireCrawl circuit breaker opened');
});
firecrawlBreaker.on('halfOpen', () => {
console.info('FireCrawl circuit HALF-OPEN - testing recovery');
});
firecrawlBreaker.on('close', () => {
console.info('FireCrawl circuit CLOSED - normal operation');
});
// Usage
async function safeFireCrawlCall<T>(fn: () => Promise<T>): Promise<T> {
return firecrawlBreaker.fire(fn);
}
Idempotency Keys
import { v4 as uuidv4 } from 'uuid';
import crypto from 'crypto';
// Generate deterministic idempotency key from input
function generateIdempotencyKey(
operation: string,
params: Record<string, any>
): string {
const data = JSON.stringify({ operation, params });
return crypto.createHash('sha256').update(data).digest('hex');
}
// Or use random key with storage
class IdempotencyManager {
private store: Map<string, { key: string; expiresAt: Date }> = new Map();
getOrCreate(operationId: string): string {
const existing = this.store.get(operationId);
if (existing && existing.expiresAt > new Date()) {
return existing.key;
}
const key = uuidv4();
this.store.set(operationId, {
key,
expiresAt: new Date(Date.now() + 24 * 60 * 60 * 1000),
});
return key;
}
}
Bulkhead Pattern
import PQueue from 'p-queue';
// Separate queues for different operations
const firecrawlQueues = {
critical: new PQueue({ concurrency: 10 }),
normal: new PQueue({ concurrency: 5 }),
bulk: new PQueue({ concurrency: 2 }),
};
async function prioritizedFireCrawlCall<T>(
priority: 'critical' | 'normal' | 'bulk',
fn: () => Promise<T>
): Promise<T> {
return firecrawlQueues[priority].add(fn);
}
// Usage
await prioritizedFireCrawlCall('critical', () =>
firecrawlClient.processPayment(order)
);
await prioritizedFireCrawlCall('bulk', () =>
firecrawlClient.syncCatalog(products)
);
Timeout Hierarchy
const TIMEOUT_CONFIG = {
connect: 5000, // Initial connection
request: 30000, // Standard requests
upload: 120000, // File uploads
longPoll: 300000, // Webhook long-polling
};
async function timedoutFireCrawlCall<T>(
operation: 'connect' | 'request' | 'upload' | 'longPoll',
fn: () => Promise<T>
): Promise<T> {
const timeout = TIMEOUT_CONFIG[operation];
return Promise.race([
fn(),
new Promise<never>((_, reject) =>
setTimeout(() => reject(new Error(`FireCrawl ${operation} timeout`)), timeout)
),
]);
}
Graceful Degradation
interface FireCrawlFallback {
enabled: boolean;
data: any;
staleness: 'fresh' | 'stale' | 'very_stale';
}
async function withFireCrawlFallback<T>(
fn: () => Promise<T>,
fallbackFn: () => Promise<T>
): Promise<{ data: T; fallback: boolean }> {
try {
const data = await fn();
// Update cache for future fallback
await updateFallbackCache(data);
return { data, fallback: false };
} catch (error) {
console.warn('FireCrawl failed, using fallback:', error.message);
const data = await fallbackFn();
return { data, fallback: true };
}
}
Dead Letter Queue
interface DeadLetterEntry {
id: string;
operation: string;
payload: any;
error: string;
attempts: number;
lastAttempt: Date;
}
class FireCrawlDeadLetterQueue {
private queue: DeadLetterEntry[] = [];
add(entry: Omit<DeadLetterEntry, 'id' | 'lastAttempt'>): void {
this.queue.push({
...entry,
id: uuidv4(),
lastAttempt: new Date(),
});
}
async processOne(): Promise<boolean> {
const entry = this.queue.shift();
if (!entry) return false;
try {
await firecrawlClient[entry.operation](entry.payload);
console.log(`DLQ: Successfully reprocessed ${entry.id}`);
return true;
} catch (error) {
entry.attempts++;
entry.lastAttempt = new Date();
if (entry.attempts < 5) {
this.queue.push(entry);
} else {
console.error(`DLQ: Giving up on ${entry.id} after 5 attempts`);
await alertOnPermanentFailure(entry);
}
return false;
}
}
}
Health Check with Degraded State
type HealthStatus = 'healthy' | 'degraded' | 'unhealthy';
async function firecrawlHealthCheck(): Promise<{
status: HealthStatus;
details: Record<string, any>;
}> {
const checks = {
api: await checkApiConnectivity(),
circuitBreaker: firecrawlBreaker.stats(),
dlqSize: deadLetterQueue.size(),
};
const status: HealthStatus =
!checks.api.connected ? 'unhealthy' :
checks.circuitBreaker.state === 'open' ? 'degraded' :
checks.dlqSize > 100 ? 'degraded' :
'healthy';
return { status, details: checks };
}
Instructions
Step 1: Implement Circuit Breaker
Wrap FireCrawl calls with circuit breaker.
Step 2: Add Idempotency Keys
Generate deterministic keys for operations.
Step 3: Configure Bulkheads
Separate queues for different priorities.
Step 4: Set Up Dead Letter Queue
Handle permanent failures gracefully.
Output
- Circuit breaker protecting FireCrawl calls
- Idempotency preventing duplicates
- Bulkhead isolation implemented
- DLQ for failed operations
Error Handling
| Issue | Cause | Solution |
|---|---|---|
| Circuit stays open | Threshold too low | Adjust error percentage |
| Duplicate operations | Missing idempotency | Add idempotency key |
| Queue full | Rate too high | Increase concurrency |
| DLQ growing | Persistent failures | Investigate root cause |
Examples
Quick Circuit Check
const state = firecrawlBreaker.stats().state;
console.log('FireCrawl circuit:', state);
Resources
Next Steps
For policy enforcement, see firecrawl-policy-guardrails.
04 同仓库更多 Skills
theme-factory
Toolkit for styling artifacts with a theme. These artifacts can be slides, docs, reportings, HTML landing pages, etc. There are 10 pre-set themes with colors/fonts that you can apply to any artifact that has been creating, or can generate a new theme on-the-fly.
web-artifacts-builder
Suite of tools for creating elaborate, multi-component claude.ai HTML artifacts using modern frontend web technologies (React, Tailwind CSS, shadcn/ui). Use for complex artifacts requiring state management, routing, or shadcn/ui components - not for simple single-file HTML/JSX artifacts.
webapp-testing
Toolkit for interacting with and testing local web applications using Playwright. Supports verifying frontend functionality, debugging UI behavior, capturing browser screenshots, and viewing browser logs.
neuropixels-analysis
Neuropixels neural recording analysis. Load SpikeGLX/OpenEphys data, preprocess, motion correction, Kilosort4 spike sorting, quality metrics, Allen/IBL curation, AI-assisted visual analysis, for Neuropixels 1.0/2.0 extracellular electrophysiology. Use when working with neural recordings, spike sorting, extracellular electrophysiology, or when the user mentions Neuropixels, SpikeGLX, Open Ephys, Kilosort, quality metrics, or unit curation.
skill-creator
Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Claude's capabilities with specialized knowledge, workflows, or tool integrations.
notebooklm
Use this skill to query your Google NotebookLM notebooks directly from Claude Code for source-grounded, citation-backed answers from Gemini. Browser automation, library management, persistent auth. Drastically reduced hallucinations through document-only responses.