Audience
- Analysts moving from spreadsheets towards repeatable data pipelines.
- Developers preparing data for reporting or AI applications.
- Technical managers who need to discuss data architecture with delivery teams.
AI & technology
Big Data
Develop a clear understanding of large-scale data, processing and business applications.
Learning with a practical purpose.
Build a practical understanding of how growing data volumes change the way information is collected, stored and processed. This module follows operational event data from its source to an analytical result, showing where partitioning, incremental processing and distributed approaches become useful and where a simpler solution is sufficient.
You will develop a small but realistic data pipeline, including validation, repeatable transformations and an analytical output for a business stakeholder. Classroom datasets keep the work manageable while the architecture exercises address larger-scale trade-offs. Attention is given to correctness, operating cost and recovery, so that technical choices can be explained in terms of a dependable service.
Live onlineTeaching through Kent Business College, UK
2 hoursOne live session each teaching week
14 sessionsA semester for each module
28 live hoursPer module, plus independent study
The outline below is a 14-session online module, with one two-hour live session each teaching week. Reading weeks, breaks and additional activities follow the agreed cohort calendar. The cohort syllabus, full route, dates, fees, independent study and assessment requirements are confirmed before enrolment.
Who this module is designed for.
Starting knowledge and preparation
- Basic SQL queries and introductory Python are recommended.
- Comfort with tabular data and simple business metrics.
- No existing enterprise data platform or large cloud budget is required.
What you will work towards.
- Translate an analytical requirement into a documented data pipeline.
- Choose suitable storage and processing patterns for a defined workload.
- Implement validated ingestion and repeatable transformations on a teaching dataset.
- Explain partitioning, incremental loading and distributed-processing trade-offs.
- Monitor data quality, recover failed work and estimate the main cost drivers.
Explore the 14-session learning plan.
Explore the focus and practical output of each teaching session. Open individual sessions below, or expand the complete plan.
Session 01 Why scale changes data work
Connect volume, arrival rate and data variety with a specific analytical requirement and service expectation.
Practical output: A workload profile and success criteria.
Session 02 Sources, contracts and lineage
Identify source owners, field meanings and dependencies so downstream users can understand where figures originate.
Practical output: A source-to-report lineage map.
Session 03 Storage patterns and file formats
Compare structured tables and common file-based approaches against access, update and analytical needs.
Practical output: A reasoned storage-design decision.
Session 04 Ingestion and validation
Load raw event data, check required fields and separate rejected records for later investigation.
Practical output: A repeatable ingestion routine.
Session 05 Transformations and analytical models
Define stable business keys and prepare a model that supports a clear operational reporting question.
Practical output: A transformation specification and analytical schema.
Session 06 Batch processing in practice
Build a batch transformation and record the inputs, outputs and checks for each run.
Practical output: A working batch pipeline.
Session 07 Partitions and efficient access
Explore how partition choices affect the amount of data scanned for common analytical questions.
Practical output: A partitioning experiment and query comparison.
Session 08 Distributed processing concepts
Examine parallel tasks, data movement and skew through a guided processing exercise and architecture discussion.
Practical output: A distributed-processing trade-off note.
Session 09 Incremental loading and late data
Process new or changed records while accounting for delayed events and corrections to earlier data.
Practical output: An incremental-load design with test cases.
Session 10 Streaming and event-time thinking
Compare scheduled batches with event-driven processing and explore ordering, windows and late arrivals.
Practical output: A streaming-versus-batch recommendation.
Session 11 Data quality and reconciliation
Design checks that connect row counts and business totals across pipeline stages and expose discrepancies.
Practical output: A reconciliation and quality report.
Session 12 Recovery, observability and cost
Track failed runs, plan safe retries and estimate storage, processing and data-transfer cost drivers.
Practical output: An operational runbook and cost assumptions.
Session 13 Operational analytics project
Assemble ingestion, transformations and a stakeholder-facing output with documented ownership and refresh expectations.
Practical output: An integrated operational data pipeline.
Session 14 Architecture review and handover
Demonstrate the pipeline, explain scaling choices and show how another team would diagnose a failure.
Practical output: A final architecture review and handover.
Apply the learning
An operational events pipeline
Build a pipeline that turns order, service or learning-event records into a useful operational report. Include duplicate detection, missing-data checks and a way to process later arrivals. Explain which parts of the design would need to change as the workload grows, using measured behaviour and explicit assumptions.
Your project evidence can include
- A data contract, lineage map and analytical schema.
- Runnable ingestion and transformation code with checks.
- An operational report, runbook and scaling recommendation.
Use anonymised, public or suitable synthetic material where workplace information is confidential. The issued assessment brief confirms the required evidence.
Assessment, feedback and certification.
- Practical exercises examine validation, partition choices and incremental processing.
- The project is assessed for correct results, traceability and repeatable execution.
- A design review tests the rationale for architecture, recovery and cost decisions.
Teaching and the relevant course certificate are provided through Kent Business College in the UK. Completion requirements are set out in the module information; attendance alone should not be assumed to meet every assessment requirement.
Questions worth asking.
Will we process terabytes in class?
Large spending is unnecessary for the learning objectives. Smaller datasets demonstrate the patterns, while exercises examine how the design changes at scale.
Is this tied to one cloud provider?
The module teaches transferable architecture and pipeline concepts. The teaching environment and any account requirements are confirmed for the cohort.
How does it support AI work?
AI applications depend on reliable data. This module develops the ingestion, quality and traceability foundations that can support later modelling or retrieval projects.