A reporting pipeline that closed the month in a day, not a fortnight

Closing the month took ten to twelve working days. Analysts exported from three systems and reconciled by hand in Excel, so breaks were typically found in the second week — long after the people who could explain them had moved on. The audit trail was, in practice, a folder of emails. Growth made it worse: volume had roughly doubled in two years while the process had not changed at all.

1. 01

   Mapped the full data lineage first — where every figure originated, what transformed it, and who touched it. Several long-standing 'adjustments' turned out to be correcting an upstream bug nobody had reported.

2. 02

   Built an ingestion layer pulling from all three source systems on a schedule, with schema validation at the boundary so bad data fails loudly rather than silently corrupting a report.

3. 03

   Automated reconciliation with an exception queue: matches clear themselves, and only genuine breaks reach a human, each with the underlying records attached.

4. 04

   Made the audit trail immutable and automatic, so evidence is a by-product of the process rather than something assembled afterwards.

5. 05

   Kept the final reporting layer in the format the team already knew, which removed most of the change-management friction from rollout.

Close now completes in a single working day. Breaks appear on day one while context is fresh, and the exception queue directs analyst time to genuinely ambiguous cases instead of ticking off thousands of matches. The audit trail is produced automatically and has survived two external reviews with no manual preparation.

• Python
• dbt
• PostgreSQL
• Airflow
• React
• AWS