{"id":305,"date":"2025-08-27T19:31:20","date_gmt":"2025-08-27T19:31:20","guid":{"rendered":"https:\/\/1v0.net\/blog\/?p=305"},"modified":"2025-08-27T19:31:22","modified_gmt":"2025-08-27T19:31:22","slug":"handling-large-data-sets-in-laravel-with-chunking-cursors","status":"publish","type":"post","link":"https:\/\/1v0.net\/blog\/handling-large-data-sets-in-laravel-with-chunking-cursors\/","title":{"rendered":"Handling Large Data Sets in Laravel with Chunking & Cursors"},"content":{"rendered":"\n

Handling Large Data Sets in Laravel with Chunking & Cursors<\/strong><\/h2>\n\n\n\n

Loading tens or hundreds of thousands of rows into memory will crash your app or time out your requests. Laravel provides chunk()<\/code>, chunkById()<\/code>, cursor()<\/code> \/ lazy()<\/code>, and their *ById()<\/code> variants to process large data sets in small, memory-friendly pieces. In this guide you\u2019ll learn when to use each, how to avoid classic pitfalls (like missing\/duplicated rows when ordering), and how to wire them into real features like CSV exports and background jobs.<\/p>\n\n\n\n

<\/div>\n\n\n\n\n

1 – When to Use chunk(), chunkById(), cursor(), lazy()<\/strong><\/h2>\n\n\n\n

Rule of thumb: chunk()<\/code> pages using LIMIT\/OFFSET (simple but can skip\/duplicate if rows are inserted\/deleted mid-scan). chunkById()<\/code> uses the primary key for stable windows (best for large mutable tables). cursor()<\/code>\/lazy()<\/code> stream one row at a time using a generator (lowest memory, slowest per-row). Use the *ById()<\/code> variants when ordering by the primary key for safety.<\/p>\n\n\n\n\n

<\/div>\n\n\n\n\n

2 – Using chunk() for Medium Data (Static Windows)<\/strong><\/h2>\n\n\n\n

Process records in fixed pages. Works well for reporting tables that don\u2019t mutate during the scan.<\/p>\n\n\n

use<\/span> App<\/span>\\Models<\/span>\\Order<\/span>;\n\n\/\/ Process 2,000 records at a time<\/span>\nOrder::where('status'<\/span>, 'paid'<\/span>)\n    ->orderBy('id'<\/span>) \/\/ always order deterministically<\/span>\n    ->chunk(2000<\/span>, function<\/span> ($orders)<\/span> <\/span>{\n        foreach<\/span> ($orders as<\/span> $order) {\n            \/\/ do work (aggregate, export, etc.)<\/span>\n        }\n    });<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
chunk(2000, ...)<\/code> runs your callback for each page. Always add a deterministic orderBy<\/code> (typically the PK). Avoid using this on hot tables that change rapidly; OFFSET can drift if rows are inserted\/deleted between pages.<\/p>\n\n\n\n\n
<\/div>\n\n\n\n\n
3 – Using chunkById() for Large, Mutable Tables<\/strong><\/h2>\n\n\n\n
Prefer chunkById()<\/code> on big tables receiving writes while you scan. It uses the last seen ID instead of OFFSET, avoiding gaps\/duplication.<\/p>\n\n\n
use<\/span> App<\/span>\\Models<\/span>\\User<\/span>;\n\n\/\/ Process active users safely even if the table changes mid-scan<\/span>\nUser::where('is_active'<\/span>, true<\/span>)\n    ->chunkById(3000<\/span>, function<\/span> ($users)<\/span> <\/span>{\n        foreach<\/span> ($users as<\/span> $user) {\n            \/\/ email sync, billing, etc.<\/span>\n        }\n    }, $column = 'id'<\/span>);<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
chunkById(3000,...)<\/code> remembers the highest id<\/code> from the last page and continues from there, so inserts\/deletes won\u2019t shift previously scanned windows. If your PK is not monotonic (e.g., UUIDs), use a sortable indexed column like an auto-increment surrogate.<\/p>\n\n\n\n\n
<\/div>\n\n\n\n\n
4 – Using cursor() \/ lazy() for Streaming (Ultra Low Memory)<\/strong><\/h2>\n\n\n\n
cursor()<\/code> (alias: lazy()<\/code>) yields models one-by-one using generators\u2014minimal memory, at the cost of longer total runtime and sustained DB connection.<\/p>\n\n\n
use<\/span> App<\/span>\\Models<\/span>\\LogEntry<\/span>;\n\nforeach<\/span> (LogEntry::where('level'<\/span>,'error'<\/span>)->orderBy('id'<\/span>)->cursor() as<\/span> $row) {\n    \/\/ Stream process each $row; memory footprint stays tiny<\/span>\n}<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
cursor()<\/code> is ideal for exports or ETL tasks. Because it keeps a DB cursor open, run it in CLI\/queue workers, not behind slow web requests. Combine with set_time_limit<\/code> or queue timeouts appropriately.<\/p>\n\n\n\n\n
<\/div>\n\n\n\n\n
5 – Memory & N+1: Eager Load Carefully<\/strong><\/h2>\n\n\n\n
When chunking or streaming, eager load only what you need. Avoid loading heavy relations per item; consider precomputing or joining.<\/p>\n\n\n
\/\/ Keep the payload small when scanning<\/span>\nOrder::with(['user:id,name'<\/span>, 'items:id,order_id,price'<\/span>])\n    ->whereYear('created_at'<\/span>, now()->year)\n    ->chunkById(2000<\/span>, function<\/span> ($orders)<\/span> <\/span>{\n        foreach<\/span> ($orders as<\/span> $order) {\n            \/\/ $order->relation calls won't trigger extra queries<\/span>\n        }\n    });<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
Eager load only essential columns (select lists on relations) to control memory and query volume. Heavy relations can still cause memory spikes if the chunk size is too large\u2014tune chunk size down if needed (e.g., 500\u20131000).<\/p>\n\n\n\n\n
<\/div>\n\n\n\n\n
6 – UI Feature: CSV Export with Streaming<\/strong><\/h2>\n\n\n\n
Let\u2019s build a safe CSV export that streams rows instead of loading them all. We\u2019ll use cursor()<\/code> inside a streamed response so memory stays flat.<\/p>\n\n\n
\/\/ app\/Http\/Controllers\/ExportController.php<\/span>\nnamespace<\/span> App<\/span>\\Http<\/span>\\Controllers<\/span>;\n\nuse<\/span> App<\/span>\\Models<\/span>\\Order<\/span>;\nuse<\/span> Symfony<\/span>\\Component<\/span>\\HttpFoundation<\/span>\\StreamedResponse<\/span>;\n\nclass<\/span> ExportController<\/span> extends<\/span> Controller<\/span>\n<\/span>{\n    public<\/span> function<\/span> ordersCsv<\/span>()<\/span>: StreamedResponse<\/span>\n    <\/span>{\n        $response = new<\/span> StreamedResponse(function<\/span> ()<\/span> <\/span>{\n            $handle = fopen('php:\/\/output'<\/span>, 'w'<\/span>);\n            fputcsv($handle, ['ID'<\/span>,'User'<\/span>,'Total'<\/span>,'Created At'<\/span>]);\n\n            foreach<\/span> (\n                Order::with('user:id,name'<\/span>)\n                    ->orderBy('id'<\/span>)\n                    ->cursor() as<\/span> $o\n            ) {\n                fputcsv($handle, [\n                    $o->id,\n                    optional($o->user)->name,\n                    number_format($o->total \/ 100<\/span>, 2<\/span>),\n                    $o->created_at,\n                ]);\n            }\n\n            fclose($handle);\n        });\n\n        $response->headers->set('Content-Type'<\/span>, 'text\/csv'<\/span>);\n        $response->headers->set('Content-Disposition'<\/span>, 'attachment; filename=\"orders.csv\"'<\/span>);\n\n        return<\/span> $response;\n    }\n}<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
This streams CSV lines as they are read from the database with cursor()<\/code>. The browser starts downloading immediately and memory usage remains near-constant regardless of table size.<\/p>\n\n\n
\/\/ routes\/web.php (snippet)<\/span>\nuse<\/span> App<\/span>\\Http<\/span>\\Controllers<\/span>\\ExportController<\/span>;\n\nRoute::get('\/exports\/orders.csv'<\/span>, [ExportController::class, 'ordersCsv'<\/span>])\n    ->middleware(['auth'<\/span>]);<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
Protect exports behind auth (and permissions if needed). For very long streams, prefer running the query in a queue and emailing a signed URL to the file once ready.<\/p>\n\n\n\n\n
<\/div>\n\n\n\n\n
7 – Background Jobs: Chunk Work into Batches<\/strong><\/h2>\n\n\n\n
Break the work into manageable pieces and dispatch jobs per chunk to parallelize safely without memory bloat.<\/p>\n\n\n
\/\/ app\/Console\/Commands\/DispatchUserEmails.php<\/span>\nnamespace<\/span> App<\/span>\\Console<\/span>\\Commands<\/span>;\n\nuse<\/span> App<\/span>\\Jobs<\/span>\\SendMonthlyEmail<\/span>;\nuse<\/span> App<\/span>\\Models<\/span>\\User<\/span>;\nuse<\/span> Illuminate<\/span>\\Console<\/span>\\Command<\/span>;\n\nclass<\/span> DispatchUserEmails<\/span> extends<\/span> Command<\/span>\n<\/span>{\n    protected<\/span> $signature = 'users:email-monthly'<\/span>;\n    protected<\/span> $description = 'Dispatch monthly emails in chunks'<\/span>;\n\n    public<\/span> function<\/span> handle<\/span>()<\/span>: int<\/span>\n    <\/span>{\n        User::where('is_active'<\/span>, true<\/span>)\n            ->select('id'<\/span>,'email'<\/span>,'name'<\/span>)\n            ->chunkById(500<\/span>, function<\/span> ($users)<\/span> <\/span>{\n                dispatch(new<\/span> SendMonthlyEmail($users->pluck('id'<\/span>)->all()));\n            });\n\n        $this<\/span>->info('Dispatched email jobs.'<\/span>);\n        return<\/span> self<\/span>::SUCCESS;\n    }\n}<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
This command slices the user base into 500-row chunks and dispatches one job per slice. Each job can query those IDs again in isolation to send emails, avoiding giant payloads on the queue bus.<\/p>\n\n\n
\/\/ app\/Jobs\/SendMonthlyEmail.php<\/span>\nnamespace<\/span> App<\/span>\\Jobs<\/span>;\n\nuse<\/span> App<\/span>\\Models<\/span>\\User<\/span>;\nuse<\/span> Illuminate<\/span>\\Bus<\/span>\\Queueable<\/span>;\nuse<\/span> Illuminate<\/span>\\Contracts<\/span>\\Queue<\/span>\\ShouldQueue<\/span>;\n\nclass<\/span> SendMonthlyEmail<\/span> implements<\/span> ShouldQueue<\/span>\n<\/span>{\n    use<\/span> Queueable<\/span>;\n\n    public<\/span> function<\/span> __construct<\/span>(public array $userIds)<\/span> <\/span>{}\n\n    public<\/span> function<\/span> handle<\/span>()<\/span>: void<\/span>\n    <\/span>{\n        User::whereIn('id'<\/span>, $this<\/span>->userIds)\n            ->select('id'<\/span>,'email'<\/span>,'name'<\/span>)\n            ->chunkById(100<\/span>, function<\/span> ($users)<\/span> <\/span>{\n                foreach<\/span> ($users as<\/span> $u) {\n                    \/\/ Mail::to($u->email)->queue(new MonthlyDigest($u));<\/span>\n                }\n            });\n    }\n}<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
Jobs receive lightweight ID lists, then process sub-chunks (100) within the job to keep memory flat and respect queue time limits. Use retry\/backoff for transient failures.<\/p>\n\n\n\n\n
<\/div>\n\n\n\n\n
8 – Avoiding Pitfalls (Ordering, Mutations, Transactions)<\/strong><\/h2>\n\n\n\n
Large scans can behave badly if ordering is unstable or if you wrap everything in one giant transaction.<\/p>\n\n\n
\/\/ Bad: OFFSET can skip\/duplicate if rows change<\/span>\n\/\/ Good: use chunkById() with a stable increasing key<\/span>\nInvoice::where('status'<\/span>,'unpaid'<\/span>)\n    ->chunkById(2000<\/span>, function<\/span> ($invoices)<\/span> <\/span>{\n        foreach<\/span> ($invoices as<\/span> $inv) {\n            \/\/ safely process<\/span>\n        }\n    });\n\n\/\/ Avoid one huge transaction around a whole scan<\/span>\n\/\/ Instead, commit per item or per small group<\/span>\nDB::transaction(function<\/span> ()<\/span> use<\/span> ($orders)<\/span> <\/span>{\n    foreach<\/span> ($orders as<\/span> $order) {\n        \/\/ small, fast writes here<\/span>\n    }\n});<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
Use chunkById()<\/code> for mutable tables. Keep transactions small to prevent lock contention and long-running locks. If you must guarantee a consistent snapshot, consider DB-level snapshot isolation or doing the scan from a read replica.<\/p>\n\n\n\n\n
<\/div>\n\n\n\n\n
Wrapping Up<\/h2>\n\n\n\n
You learned how to pick the right tool for big reads: chunk()<\/code> for simple paging on static data, chunkById()<\/code> for safety on hot tables, and cursor()<\/code>\/lazy()<\/code> for true streaming. You also built a memory-safe CSV export and batched background jobs. With careful ordering, eager loading, and transaction strategy, you can process millions of rows reliably.<\/p>\n\n\n\n\n
<\/div>\n\n\n\n\n
What\u2019s Next<\/h2>\n\n\n\n