Scaling Batch Processing: Redis and Queues for Large Data Imports
Prevent PHP gateway timeouts by offloading heavy Excel/CSV parsing to Redis-backed background queue workers.
Scaling Batch Processing: Redis and Queues for Large Data Imports
High-Scale Real-Time & Streaming Systems
Scaling Batch Processing with Redis and Queues
Handling large Excel or CSV imports of 10,000+ entries in a web request is a common mistake that leads to HTTP 504 Gateway Timeouts. Let's explore how to design a resilient, non-blocking upload pipeline using Redis and Laravel Queues.
The Non-blocking Pipeline Architecture
1. Direct Upload: User uploads file.
2. Job Dispatch: The file is stored, and an import job is pushed to the Redis queue. The server responds with 200 OK instantly.
3. Worker Processing: A background queue worker retrieves the job, parses the rows in chunks, and updates progress.
4. Real-Time Broadcast: The user gets live progress updates via WebSockets.
1. Storing File and Dispatching Job
Instead of parsing files synchronously, we dispatch an asynchronous Job to the queue.
class=class="text-emerald-500 dark:text-emerald-400 font-medium">"text-slate-500 italic">// app/Http/Controllers/ImportController.php
namespace AppHttpControllers;
use AppJobsProcessExcelImport;
use IlluminateHttpRequest;
class ImportController extends Controller
{
public function upload(Request $request)
{
$request->validate(['file' => 'required|mimes:xlsx,csv']);
class=class="text-emerald-500 dark:text-emerald-400 font-medium">"text-slate-500 italic">// Save file to storage
$path = $request->file('file')->store('imports');
class=class="text-emerald-500 dark:text-emerald-400 font-medium">"text-slate-500 italic">// Dispatch background job to Redis
ProcessExcelImport::dispatch($path, auth()->user()->id);
return response()->json([
'message' => 'Upload successful. Processing in the background.',
]);
}
}2. Implementing Chunks in the Background Job
Inside the Job, we read the data in chunks of 500 rows to optimize database query memory and avoid hitting execution limits.
class=class="text-emerald-500 dark:text-emerald-400 font-medium">"text-slate-500 italic">// app/Jobs/ProcessExcelImport.php
namespace AppJobs;
use IlluminateContractsQueueShouldQueue;
use IlluminateFoundationBusDispatchable;
use IlluminateQueueInteractsWithQueue;
use IlluminateQueueSerializesModels;
use MaatwebsiteExcelFacadesExcel;
class ProcessExcelImport implements ShouldQueue
{
use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
protected $filePath;
protected $userId;
public function __construct($filePath, $userId)
{
$this->filePath = $filePath;
$this->userId = $userId;
}
public function handle()
{
class=class="text-emerald-500 dark:text-emerald-400 font-medium">"text-slate-500 italic">// Parse excel in chunks to save memory
Excel::import(new InventoryImport($this->userId), $this->filePath);
}
}Using Redis as a broker and background workers lets your application remain highly responsive and scalable under high user traffic load.