👋 Welcome to Order Management + Webhooks

A robust, scalable PHP application designed to handle high-volume order processing with webhook integration. This system demonstrates enterprise-level patterns for handling webhooks, ensuring idempotency, managing concurrency, and maintaining comprehensive audit trails.

📹 Demo & Resources

🎥 Discussion Video

System Overview & Discussion

📥 Download Discussion Video

Duration: ~15 minutes

🎥 Code Review Video

Code Review & Technical Implementation

📥 Download Code Review Video

Duration: ~13 minutes

📋 Postman Collection

Test all endpoints

🐙 GitHub Repository

View source code

View on GitHub

📚 API Documentation

Detailed API reference

View Documentation

📖 Project Overview

This PHP application provides a comprehensive order management system with webhook processing capabilities. It's designed to handle high-pressure scenarios with 20M+ records while maintaining data integrity and performance.

Core Features

  • • Order Management with Status Tracking
  • • Webhook Processing with Retry Logic
  • • Comprehensive Audit Trail
  • • Queue Processing & Background Jobs
  • • Idempotency Protection
  • • Concurrency Control

Technical Highlights

  • • Database Transactions with Optimistic Locking
  • • Advanced Rate Limiting & Performance Optimization
  • • Database Indexing for Scale
  • • Comprehensive Logging & Monitoring
  • • Error Handling & Recovery
  • • Health Check Endpoints

⚡ Challenges in the Task

🔄 Handling Webhooks

Webhooks can arrive out of order, be duplicated, or fail during processing. The system must handle these scenarios gracefully while maintaining data consistency and providing reliable processing.

🔒 Idempotency

Ensuring that processing the same webhook multiple times doesn't result in duplicate data or incorrect state changes. Critical for maintaining data integrity in distributed systems.

⚡ Concurrency / Race Conditions

Multiple webhooks processing simultaneously can lead to race conditions, especially when updating order statuses. The system must handle concurrent access safely without data corruption.

📋 Queue Processing & Retries

Background job processing with intelligent retry mechanisms, exponential backoff, and failure handling. Essential for reliable webhook processing at scale.

📊 Audit Trail

Complete tracking of all order status changes, webhook processing events, and system activities. Provides transparency and debugging capabilities for production systems.

📦 Out-of-Order Webhook Delivery

Webhooks may arrive in different order than they were sent, requiring intelligent handling to maintain correct order state transitions and business logic.

🚦 Rate Limitation & System Overload

High-volume webhook scenarios (20M+ records) can overwhelm the system, causing performance degradation, timeouts, and potential service unavailability. Without proper rate limiting, the system becomes vulnerable to DDoS attacks and resource exhaustion.

Specific Problems:

  • Resource Exhaustion: Unlimited requests can consume all server resources
  • Database Overload: Too many concurrent database operations
  • Memory Issues: Excessive memory usage from concurrent processing
  • Service Degradation: Slow response times affecting legitimate users
  • Security Vulnerabilities: Potential for abuse and DDoS attacks

🛠️ Solutions & Code

🔄 Webhook Handling Solution

The webhook controller implements rate limiting, duplicate detection, and immediate queuing for database transaction-based processing.

PHP
// routes/api.php - Rate limiting via middleware
Route::post('/webhooks/payments', [WebhookController::class, 'store'])
    ->middleware('webhook.rate.limit:1000,1');

// WebhookController.php - Clean separation of concerns
public function store(WebhookPaymentRequest $request): JsonResponse
{
    $startTime = microtime(true);
    
    // CRITICAL: Duplicate detection using cache for high performance
    $validatedData = $request->validated();
    $txnId = $validatedData['txn_id'];
    $cacheKey = "webhook_txn:{$txnId}";
    
    if (Cache::has($cacheKey)) {
        return response()->json(['message' => 'Duplicate webhook detected'], 409);
    }
    
    // CRITICAL: Create webhook log and dispatch job with high priority
    $webhookLog = WebhookLog::create([...]);
    ProcessPaymentWebhookJob::dispatch($webhookLog->id)
        ->onQueue('webhooks-high-priority');
    
    $processingTime = round((microtime(true) - $startTime) * 1000, 2);
    
    return response()->json([
        'message' => 'Webhook received and queued for processing',
        'processing_time_ms' => $processingTime,
    ], 202);
}
Explanation: Rate limiting middleware provides protection, cache-based duplicate detection ensures idempotency, and immediate job dispatch enables asynchronous processing with database transaction-based concurrency control.

🔒 Idempotency Solution

Multiple layers of idempotency protection using transaction IDs and database constraints.

PHP
// Check if external_txn_id already exists
if ($payload['txn_id'] && Order::withExternalTxnId($payload['txn_id'])->exists()) {
    Log::info("Transaction ID {$payload['txn_id']} already processed, ignoring webhook");
    $webhookLog->markAsIgnored('Transaction already processed');
    return;
}

// Cache-based duplicate prevention
Cache::put($cacheKey, true, 300); // 5 minutes cache

// Database unique constraint on external_txn_id
Schema::table('orders', function (Blueprint $table) {
    $table->unique('external_txn_id');
});
Explanation: Three-layer protection: cache check, database query, and unique constraint ensure no duplicate transactions are processed.

⚡ Concurrency Solution

Database transaction with optimistic locking strategy prevents race conditions while maintaining ACID compliance.

PHP
// CRITICAL: Use database transaction with optimistic locking
DB::transaction(function () use ($webhookLog, $payload, $startTime) {
    // CRITICAL: Get order without locking (optimistic approach)
    $order = Order::where('id', $payload['order_uuid'])->first();
    
    // CRITICAL: Optimistic locking - update with version check
    $updatedRows = Order::where('id', $order->id)
        ->where('updated_at', $order->updated_at) // Optimistic lock
        ->update([
            'status' => $newStatus,
            'external_txn_id' => $payload['txn_id'],
            'amount' => $payload['amount'],
            'updated_at' => now(),
        ]);

    // CRITICAL: Check if update succeeded (optimistic locking)
    if ($updatedRows === 0) {
        Log::warning("Optimistic lock failed for order {$order->id} - another process modified it");
        throw new \Exception('Optimistic lock failed - order was modified by another process');
    }
    
    // Create audit trail and update webhook log within same transaction
    OrderEvent::create([...]);
    $webhookLog->markAsProcessed($processingTime);
});
Explanation: Uses DB::transaction() for ACID compliance with optimistic locking. The updated_at timestamp serves as a version field. If another process modified the record, the update returns 0 rows and triggers a retry via Laravel's job retry mechanism.

📋 Queue Processing Solution

Database transaction-based processing with intelligent retry mechanism and comprehensive error handling.

PHP
class ProcessPaymentWebhookJob implements ShouldQueue
{
    public int $tries = 5;
    public int $timeout = 60;
    
    /**
     * Calculate the number of seconds to wait before retrying the job.
     * Optimized for database transaction conflicts and deadlocks.
     */
    public function backoff(): array
    {
        return [5, 10, 30, 60, 120]; // 5s, 10s, 30s, 1m, 2m
    }
    
    public function handle(): void
    {
        try {
            // CRITICAL: Use database transaction with optimistic locking
            DB::transaction(function () use ($webhookLog, $payload, $startTime) {
                // Process webhook with optimistic locking
                $order = Order::where('id', $payload['order_uuid'])->first();
                
                $updatedRows = Order::where('id', $order->id)
                    ->where('updated_at', $order->updated_at)
                    ->update([...]);
                
                if ($updatedRows === 0) {
                    throw new \Exception('Optimistic lock failed');
                }
            });
        } catch (\Exception $e) {
            // Handle optimistic lock failures - these should be retried
            if (str_contains($e->getMessage(), 'Optimistic lock failed')) {
                Log::warning("Optimistic lock failed for webhook {$this->webhookLogId}, will retry");
                throw $e; // Re-throw to trigger retry mechanism
            }
            throw $e;
        }
    }
}
Explanation: Uses DB::transaction() for ACID compliance with optimistic locking. Shorter backoff delays (5s-2m) are optimized for transaction conflicts. Laravel's job retry mechanism automatically handles optimistic lock failures.

📊 Audit Trail Solution

Comprehensive event logging with detailed metadata for complete traceability.

PHP
// Insert new row in order_events (audit trail)
OrderEvent::create([
    'order_id' => $order->id,
    'from_status' => $oldStatus,
    'to_status' => $newStatus,
    'reason' => OrderEvent::REASON_WEBHOOK_PAYMENT,
    'metadata' => [
        'webhook_log_id' => $webhookLog->id,
        'txn_id' => $payload['txn_id'],
        'timestamp' => $payload['timestamp'],
        'amount' => $payload['amount'],
        'webhook_source' => $webhookLog->webhook_source,
        'correlation_id' => $webhookLog->correlation_id,
    ],
]);

// Webhook log tracking
$webhookLog = WebhookLog::create([
    'order_uuid' => $validatedData['order_uuid'],
    'txn_id' => $txnId,
    'raw_payload' => $validatedData,
    'status' => WebhookLog::STATUS_PENDING,
    'attempts' => 1,
    'webhook_source' => $validatedData['webhook_source'] ?? WebhookLog::SOURCE_UNKNOWN,
    'correlation_id' => $validatedData['correlation_id'] ?? null,
]);
Explanation: Every status change is recorded in order_events with full context, while webhook_logs track all incoming webhooks and their processing status.

🚦 Rate Limiting Solution

Multi-layered rate limiting strategy using Laravel's built-in rate limiter with IP-based throttling and configurable limits optimized for high-volume webhook scenarios.

1. Custom Rate Limiting Middleware

PHP
class WebhookRateLimit
{
    /**
     * Handle an incoming request.
     * 
     * Rate limiting middleware specifically designed for webhook endpoints
     * to handle high-pressure scenarios with 20M+ records.
     */
    public function handle(Request $request, Closure $next, int $maxAttempts = 1000, int $decayMinutes = 1): Response
    {
        // CRITICAL: Rate limiting for high-pressure webhook scenarios
        $key = 'webhook:' . $request->ip();
        
        if (RateLimiter::tooManyAttempts($key, $maxAttempts)) {
            return response()->json([
                'message' => 'Too many requests',
                'retry_after' => RateLimiter::availableIn($key),
                'limit' => $maxAttempts,
                'window' => $decayMinutes . ' minute(s)',
            ], 429);
        }
        
        // Increment the rate limit counter
        RateLimiter::hit($key, $decayMinutes * 60);
        
        return $next($request);
    }
}
Explanation: Custom middleware provides IP-based rate limiting with configurable limits. Default: 1000 requests per minute per IP, with clear error responses including retry timing.

2. Route Configuration

PHP
// routes/api.php - Rate limiting via middleware
Route::post('/webhooks/payments', [WebhookController::class, 'store'])
    ->middleware('webhook.rate.limit:1000,1');

// Alternative: More restrictive for sensitive endpoints
Route::post('/webhooks/admin', [AdminWebhookController::class, 'store'])
    ->middleware('webhook.rate.limit:100,1'); // 100 requests per minute
Explanation: Middleware parameters: first number = max attempts, second number = decay minutes. Different endpoints can have different rate limits based on their sensitivity.

3. Controller Integration

PHP
public function store(WebhookPaymentRequest $request): JsonResponse
{
    $startTime = microtime(true);
    
    // CRITICAL: Duplicate detection using cache for high performance
    $validatedData = $request->validated();
    $txnId = $validatedData['txn_id'];
    $cacheKey = "webhook_txn:{$txnId}";
    
    if (Cache::has($cacheKey)) {
        return response()->json([
            'message' => 'Duplicate webhook detected',
            'status' => 'ignored',
            'txn_id' => $txnId,
        ], 409);
    }

    // Process webhook...
    $processingTime = round((microtime(true) - $startTime) * 1000, 2);
    
    return response()->json([
        'message' => 'Webhook received and queued for processing',
        'processing_time_ms' => $processingTime,
    ], 202);
}
Explanation: Controller includes performance monitoring and fast duplicate detection to minimize processing time and prevent unnecessary work.

Rate Limiting Benefits:

  • System Protection: Prevents resource exhaustion and service degradation
  • Fair Usage: Ensures all clients get fair access to the API
  • DDoS Mitigation: Protects against malicious traffic spikes
  • Performance Optimization: Maintains consistent response times
  • Cost Control: Prevents unexpected infrastructure costs
  • Monitoring: Provides clear metrics on usage patterns

🔄 System Architecture & Flowcharts

System Architecture

graph TB subgraph "External Systems" PS[Payment Service] LB[Load Balancer] end subgraph "Laravel Application" WC[WebhookController] QM[Queue Manager] WJ[ProcessPaymentWebhookJob] OM[Order Model] OEM[OrderEvent Model] WLM[WebhookLog Model] end subgraph "Data Layer" DB[(Database)] CACHE[(Redis Cache)] end PS -->|Webhook| LB LB -->|HTTP Request| WC WC -->|Rate Limit Check| CACHE WC -->|Duplicate Check| CACHE WC -->|Create Log| WLM WC -->|Dispatch Job| QM QM -->|Process| WJ WJ -->|Update Order| OM WJ -->|Create Event| OEM WJ -->|Update Log| WLM OM -->|Store| DB OEM -->|Store| DB WLM -->|Store| DB

Webhook Processing Flow

flowchart TD A[Webhook Received] --> B[Rate Limit Middleware] B --> C{Rate Limit OK?} C -->|No| D[Return 429 Too Many Requests] C -->|Yes| E[WebhookController.store] E --> F{Duplicate Check} F -->|Duplicate| G[Return 409 Conflict] F -->|New| H[Create WebhookLog] H --> I[Cache Transaction ID] I --> J[Dispatch Job to Queue] J --> K[Return 202 Accepted] K --> L[Job Processing Starts] L --> M{Order Exists?} M -->|No| N[Mark as Failed] M -->|Yes| O{Transaction Already Processed?} O -->|Yes| P[Mark as Ignored] O -->|No| Q{Valid Status Transition?} Q -->|No| R[Mark as Ignored] Q -->|Yes| S[Optimistic Lock Update] S --> T{Update Successful?} T -->|No| U[Retry with Backoff] T -->|Yes| V[Create OrderEvent] V --> W[Mark as Processed] U --> S

Optimistic Locking Strategy

sequenceDiagram participant W1 as Webhook Job 1 participant W2 as Webhook Job 2 participant DB as Database participant O as Order Record W1->>DB: SELECT order WHERE id = X W2->>DB: SELECT order WHERE id = X DB-->>W1: Order (updated_at: T1) DB-->>W2: Order (updated_at: T1) W1->>DB: UPDATE order SET status='paid', updated_at=NOW() WHERE id=X AND updated_at=T1 W2->>DB: UPDATE order SET status='refunded', updated_at=NOW() WHERE id=X AND updated_at=T1 DB-->>W1: 1 row affected (SUCCESS) DB-->>W2: 0 rows affected (FAILED) W1->>W1: Continue processing W2->>W2: Retry with new timestamp

Rate Limiting Flow

flowchart TD A[Webhook Request] --> B[Rate Limiting Middleware] B --> C{Check IP Rate Limit} C -->|Limit Exceeded| D[Return 429 Too Many Requests] C -->|Within Limit| E[Increment Counter] E --> F[Continue to Controller] F --> G[Duplicate Check] G --> H{Duplicate Found?} H -->|Yes| I[Return 409 Conflict] H -->|No| J[Process Webhook] J --> K[Cache Transaction ID] K --> L[Queue Job] L --> M[Return 202 Accepted] D --> N[Client Receives Rate Limit Info] N --> O[Client Waits & Retries] O --> A

🌟 Advantages

🛡️ Reliability

  • Idempotency: No duplicate processing
  • Retry Logic: Automatic failure recovery
  • Transaction Safety: ACID compliance
  • Error Handling: Comprehensive exception management
  • Health Checks: System monitoring endpoints

📈 Scalability

  • Queue Processing: Asynchronous job handling
  • Rate Limiting: Prevents system overload
  • Database Indexing: Optimized queries
  • Cache Strategy: Reduced database load
  • Optimistic Locking: No blocking operations

🔧 Maintainability

  • Clean Architecture: Separation of concerns
  • Comprehensive Logging: Easy debugging
  • Type Safety: Strong typing throughout
  • Documentation: Well-documented code
  • Testing: Comprehensive test coverage

👁️ Observability

  • Audit Trail: Complete event history
  • Performance Metrics: Processing time tracking
  • Structured Logging: Searchable log entries
  • Webhook Tracking: Full request/response logs
  • Status Monitoring: Real-time system health

📝 Example Usage

Webhook Payload Example

JSON
{
    "order_uuid": "550e8400-e29b-41d4-a716-446655440000",
    "txn_id": "TXN_123456789",
    "status": "paid",
    "amount": 99.99,
    "timestamp": "2024-01-15T10:30:00Z",
    "webhook_source": "payment_gateway",
    "correlation_id": "CORR_789",
    "metadata": {
        "payment_method": "credit_card",
        "card_last_four": "1234",
        "currency": "USD",
        "customer_id": "CUST_456"
    }
}

API Call Example (CURL)

BASH
curl -X POST http://localhost:8000/api/webhooks/payment \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -d '{
    "order_uuid": "550e8400-e29b-41d4-a716-446655440000",
    "txn_id": "TXN_123456789",
    "status": "paid",
    "amount": 99.99,
    "timestamp": "2024-01-15T10:30:00Z",
    "webhook_source": "payment_gateway",
    "correlation_id": "CORR_789",
    "metadata": {
        "payment_method": "credit_card",
        "card_last_four": "1234",
        "currency": "USD",
        "customer_id": "CUST_456"
    }
}'

Success Response

{
    "message": "Webhook received and queued for processing",
    "webhook_log_id": 12345,
    "txn_id": "TXN_123456789",
    "status": "queued",
    "processing_time_ms": 15.23
}

🚀 Quick Start

1. Install Dependencies

composer install

2. Environment Setup

cp .env.example .env
php artisan key:generate

Configure your database connection and other environment variables in the .env file.

3. Database Setup

php artisan migrate:fresh --seed

This will create all necessary tables and populate them with sample data.

4. Start Queue Worker

php artisan queue:work --queue=webhooks-high-priority

Start the queue worker to process webhook jobs. Run this in a separate terminal.

5. Run Tests

php artisan test

Verify that all tests pass to ensure the system is working correctly.

6. Start Development Server

php artisan serve

The application will be available at http://localhost:8000