This book is licensed for the personal use of the buyer. Please do not share or resell the file. Short quotations with attribution are welcome.
The information in this book is provided for education. It is not legal, financial or other professional advice. Product and company names are used only to describe them and belong to their owners.
This is a free sample: the introduction and first chapter.
IntroductionIntroduction
Ask experienced machine-learning practitioners what made their most successful project work, and few will name an algorithm. They will talk about data: the moment they realised the labels were inconsistent, the leak they found just before launch, the extra month spent collecting examples of the rare cases that mattered most.
Models learn only what their data shows them. A brilliant architecture trained on biased, noisy or unrepresentative data produces a confident, fast, wrong system. A simple model trained on clean, well-defined, representative data often performs remarkably well. Yet data work is rarely taught with the same care as algorithms.
This book is a practical guide to the full life cycle of data for AI:
Chapter 1 explains why data quality dominates outcomes.
Chapter 2 turns a business question into precise data requirements.
Chapter 3 covers collecting data responsibly: consent, licensing and privacy.
Chapter 4 sets out a systematic cleaning process.
Chapter 5 shows how to explore data before modelling, so that surprises are found early.
Chapter 6 is about labelling: guidelines, quality control and agreement.
Chapter 7 covers splitting data, preventing leakage and checking for bias.
Chapter 8 deals with imbalanced data and rare events.
Chapter 9 presents privacy-preserving techniques, from pseudonymisation to differential privacy.
Chapter 10 shows how to document datasets so others can trust them.
Chapter 11 covers data pipelines, automated quality checks and drift after launch.
Chapter 12 addresses the special needs of data for language-model projects.
Chapter 13 covers images, audio and video.
Chapter 14 sets out data governance sized for real teams: ownership, catalogues, access, retention and incidents.
Chapter 15 follows a complete case study, a customer-churn dataset, through every stage.
Each chapter ends with a checklist, and exercises with answers and a glossary close the book. No advanced mathematics is required.
Chapter 1Why Data Decides Everything
The model is a mirror
A trained model is, in effect, a compressed summary of the patterns in its training data. It cannot know about situations the data never showed, cannot correct systematic errors in the labels, and will happily learn accidental patterns that happen to predict the label.
Three classic failures illustrate this:
Missing situations. A defect-detection camera trained on images from a factory's day shift performs poorly at night, when lighting differs. Nothing was wrong with the model; the data never showed night conditions.
Inconsistent labels. Two teams labelled customer complaints using different ideas of "urgent". The model learned a blurry average of both, satisfying neither.
Shortcut patterns. A model meant to recognise a medical condition learned partly from incidental markers in images that correlated with where patients were scanned. It scored well in testing and would have failed elsewhere.
Data-centric thinking
When a model underperforms, the instinct is to try a bigger model or tune settings. Often the better first question is: what is wrong with the data? Common answers:
Labels are noisy or inconsistently defined.
Important cases are rare or missing.
Training data differs from the data the model will see in use.
Features contain information that will not be available at prediction time.
The cost curve
Fixing data problems is cheapest early. A clear label definition written on day one costs an hour. Discovering inconsistent labels after 50,000 examples have been annotated costs weeks. Discovering them after deployment costs trust.
Checklist
Do we understand where our data comes from and how it was produced?
Have we looked at a random sample of raw examples by eye?
Do we have a plan to check data quality before training?
You have reached the end of the free sample. The full book continues with 14 more chapters, plus everything else in its 80 pages.