01 — Product Overview (Bulk Module)

• 1. What the module is
• 2. Scope in / scope out
• 3. Sync vs async — the two engines
• 4. Import pipeline (exact)
• 5. Duplicate handling
• 6. 1000-row behaviour (large files)
• 7. CSV column contract (import)
• 8. Platform & client scope notes
• 9. Goals (product)
• 10. Non-goals (per source)

Generic CSV bulk import/export for tenant entities. Derived from src/modules/bulk/** (controller, service, adapter, interface, module), src/modules/users/**, src/modules/students/**, src/modules/rbac/permissions.constants.ts, src/infrastructure/bullmq/{queue.constants.ts,event-queue-map.ts}, docs/IMPLEMENTATION_PLAN.md, and the shared ledger design-docs/00-shared/**. Nothing in this doc is invented; plan-only capability is marked (planned), client-only or roadmap-only capability (forward-looking), analytics (proposed).

1. What the module is

The Bulk module is the generic CSV pipeline for tenant data: one endpoint pattern imports any entity with a registered adapter, one exports it. Today it ships exactly one adapter — students (students-import.adapter.ts:15); the service itself notes the registry is deferred until a second entity joins (bulk-import.service.ts:15-16).

Key facts from source:

• Two routes, both Bearer-JWT protected (JwtAuthGuard, bulk.controller.ts:28-31):
  • POST /api/v1/bulk/import/:entity — multipart upload, field file (bulk.controller.ts:35-48).
  • GET /api/v1/bulk/export/:entity — CSV download (bulk.controller.ts:50-60).
• Import is synchronous and in-process: csv-parse/sync → per-row validate → per-row create → in-memory report returned in the same request (bulk-import.service.ts:22-65). There is no BullMQ queue, no worker, no result-polling endpoint (queue.constants.ts:1-17 lists 15 queues; none is a bulk queue).
• Report contract ImportReport {entity, totalRows, imported, failed, errors: [{rowNumber, errors[]}]} (import-adapter.interface.ts:14-25); every error carries the physical CSV row number (header = row 1, bulk-import.service.ts:46).
• The students adapter creates one user per row via UsersService.create (students-import.adapter.ts:66-71) — which emits UserCreated (users.service.ts:64-76) → in-app queue user-created-notification (event-queue-map.ts:10) — and one student via StudentService.create (students-import.adapter.ts:76-84) — which emits StudentCreated (student.service.ts:79-91) → in-app queue student-enrolled (event-queue-map.ts:28). Import rows therefore DO fan out notifications, unlike the users module's inline import which bypasses events (see 01_Product_Overview.md §4 in the users package).
• Permissions exist in permissions.constants.ts:10 (user.import) and :7 (user.create), :25-28 (student.read/create/update/delete) — but no RBAC guard is attached to the bulk routes; only JwtAuthGuard. Permission enforcement is (planned).
• Module wiring: bulk.module.ts:11-17 imports UsersModule, StudentsModule, AcademicsModule; registered in the root app (app.module.ts:126).

2. Scope in / scope out

3. Sync vs async — the two engines

There are two distinct CSV import implementations in source; both are synchronous:

Async import is planned but not built: IMPLEMENTATION_PLAN.md:172 ("Generic service, CSV/Excel, validation, BullMQ progress, rollback; integrate Students, Fees, Library, Exams") and a students-specific POST /api/v1/students/bulk-import (IMPLEMENTATION_PLAN.md:195). The shared ledger's glossary already says "Bulk import: CSV upload processed asynchronously by workers" (00-shared/01 §10) — that is ahead of the code; flag it as a documented discrepancy (see 14_QA_Checklist.md §9).

4. Import pipeline (exact)

1. Request → multipart file required, else 400 VALIDATION_ERROR "CSV file is required (multipart field "file")." (bulk.controller.ts:43-46).
2. Buffer decoded as UTF-8 (bulk.controller.ts:47).
3. Parse with csv-parse/sync {columns:true, skip_empty_lines:true, trim:true} (bulk-import.service.ts:26-30); unparsable CSV → 400 VALIDATION_ERROR "Malformed CSV: could not parse file." (:31-33); header-only/empty → 400 "CSV must include a header row and data." (:34-35).
4. Unknown entity → 404 RESOURCE_NOT_FOUND No import adapter for entity "X". (bulk-import.service.ts:17-20).
5. Per row (rowNumber = index + 2, header is row 1 — :45-46): adapter validate(row); any errors → failed+1, errors.push({rowNumber, errors}), row skipped (:47-52); else adapter create(row) → imported+1, or on throw → failed+1 with the thrown message as the row error (:53-62). One bad row never aborts the batch (bulk-import.service.spec.ts:53-65).
6. Report returned in the 200 envelope.

5. Duplicate handling

The students adapter resolves duplicates at validate time (row rejected, never re-used):

• admissionNumber exists → Admission number "X" already exists. (students-import.adapter.ts:49-54, via StudentRepository.findByAdmissionNumber).
• email registered → Email "X" already registered. (students-import.adapter.ts:55-56, via UsersService.findByEmail).

Race fallback at create time: if UsersService.create throws (email ConflictException, users.service.ts:50-54), the adapter re-fetches the existing user and reuses its _id (students-import.adapter.ts:72-75) — so a duplicate that slips past validation still results in a student row, not a crash.

6. 1000-row behaviour (large files)

• Every row is processed sequentially with await (bulk-import.service.ts:45), and each row performs multiple DB round-trips: duplicate checks (2 reads) + ref resolution (3-4 reads: academicYear, grade, section, class — students-import.adapter.ts:100-144) + user create (1 write) + student create + enrollment (2 writes). A 1,000-row file is therefore roughly 1,000 × 7-9 DB operations inside a single HTTP request, with the full report held in memory (bulk-import.service.ts:37-43).
• Consequences for the client: no progress events, no partial-report resume, risk of gateway/proxy timeouts on very large files, and the API rate-limit budget (api tier 100/min, 00-shared/07 §4) is consumed by one import.
• Design rule: the client must treat the import as a long-running synchronous call, show an indeterminate progress state, and cap file size client-side (see 06, 10, 15). The production answer — queue + progress
  • polling — is (planned) (IMPLEMENTATION_PLAN.md:172).

7. CSV column contract (import)

From the adapter (students-import.adapter.ts:17-26): header names are exact camelCase (no normalization like the users inline import):

References must resolve to existing records — the adapter never creates grades/sections/years (error strings in 08_Form_Specifications.md §2).

8. Platform & client scope notes

• Native-app exclusion (PRD): the PRD puts native mobile apps out of Phase 1 scope ("Native mobile apps (web-first)", PRODUCT_REQUIREMENTS_DOCUMENT.md:144); the shared ledger resolved to spec a full-featured Flutter client now, against the complete API surface (00-shared/01 §9). All screens, routes, and Flutter guidance in this package are (forward-looking) by extension. Bulk import is a desktop/web-admin workflow (file picker, large tables) with phone support limited to result review.
• API is v1, Bearer JWT, tenant from token only (00-shared/07 §1,§6); wire contract and error codes per 00-shared/07 §2-§3 (success {success:true,message:"OK",data,meta?,timestamp,requestId}; codes 400/401/403/404/409/422/429/5xx).
• Push/QR tooling (forward-looking); analytics (proposed) — no SDK chosen (00-shared/12 A4).

9. Goals (product)

1. Bulk onboarding — an admin imports hundreds of students from a school's existing spreadsheet in one upload, with a per-row error report so rows can be fixed and re-uploaded without re-entering good rows.
2. Safe by default — nothing is imported until the file parses; duplicates are rejected per row; a bad row never blocks the batch; re-upload is idempotent (already-imported rows fail as duplicates, not as corruption).
3. Round-trip — export CSV as a template/reference for future imports (GET /bulk/export/students, students-import.adapter.ts:87-98).
4. Extensible — the adapter contract (import-adapter.interface.ts:6-12) lets fees/library/exams join with one new file each (planned).

10. Non-goals (per source)

• No async import job, no queue, no progress events, no rollback — all (planned) (IMPLEMENTATION_PLAN.md:172).
• No generic template endpoint, no per-entity pagination of errors, no download-error-CSV endpoint (client can synthesize).
• No bulk update/delete — import only creates.
• No permission enforcement on the routes yet (user.import exists but is unbound, permissions.constants.ts:10) — (planned) RBAC guard.
• No analytics instrumentation in this module (proposed).