GATE Machine Learning Study Guide
From zero to working understanding — intuitive concepts with worked numerical calculations, interactive diagrams, scikit-learn code, and exam-grade practice questions covering the complete official GATE Data Science & AI syllabus.
01. Introduction to ML
Supervised vs Unsupervised, Inductive Bias, and the No Free Lunch theorem.
Official Exam Question Bank
MCQ, MSQ, and NAT numerical questions with instant scoring, negative marking, and step-by-step solutions.
Formulas & Mathematics
Essential equations, loss functions, matrix derivations, and academic textbook references.
How to Use This Guide
This guide is designed for intuitive conceptual progression. You do not need to have mastered advanced calculus or linear algebra before reading: every mathematical concept — from partial derivatives and gradients to eigenvalues and loss functions — is built progressively from geometric intuition before being formalized. If you understand basic algebra, functions, and coordinate graphs, you are ready to start. For the full GATE DA examination, rigorous mathematical foundations and formulas are consolidated in Appendix A and Appendix B.
What each unit contains
Every unit follows the same rhythm, so you always know what is coming next. It opens with an intuition section that explains the idea in ordinary language and an analogy. Then comes the formal explanation — the definition, the maths, and how the algorithm actually works step by step. Next is at least one worked numerical example where numbers are computed by hand so you can see the arithmetic, followed by Python code using scikit-learn that does the same job on a real dataset. Each unit closes with practice questions — conceptual, numerical, and interview-style — with full answers.
How the callout boxes work
Green boxes mark the key idea of a section — the one thing to remember if you forget everything else. If you are revising the night before an exam, read only the green boxes first.
Yellow boxes appear only for genuinely difficult ideas — gradient descent, backpropagation, SVM kernels, eigenvectors. These are the concepts where formal language often hides the meaning, so they are re-explained in a conversational Hindi–English mix.
Purple boxes connect a technical idea to something from everyday life. Analogies are not perfectly accurate — they are scaffolding. Use them to build intuition, then discard them once the real concept clicks.
Red boxes flag errors that beginners make again and again — data leakage, evaluating on training data, misreading accuracy on imbalanced datasets. Most of the marks people lose, and most of the bugs that reach production, come from a short list of mistakes. These are them.
A note on the mathematics
Formulas appear throughout, but they are never left unexplained. Where a formula is given,
the symbol list underneath tells you what each term means, and the paragraph before it tells you
what the formula is doing in plain language. You will see notation like
y = mx + c rather than heavy vector notation wherever possible, because the idea
matters more than the formalism at this stage. Appendix A collects the small amount of
mathematical background you will need, in case a symbol is unfamiliar.
Suggested study paths
| If you want to… | Study in this order | Notes |
|---|---|---|
| Understand ML from scratch | Units 1 → 2 → 3 → 4 → 5 → 6 → 7 → 8 | The natural sequence. Read every worked example. |
| Prepare for interviews | Units 1, 5, 8 first, then 3, 4, 6, 7 | Interviewers weight evaluation, overfitting and bias–variance heavily. |
| Build something this week | Unit 2 → 3 → 4 → 5 → Unit 10 | Skim the theory, focus on the workflow and code. |
| Revise before an exam | All green Insight boxes, then Appendix B | The cheat sheet in Appendix B is dense and exam-ready. |
Machine learning is not a large subject — it is a small subject with a lot of vocabulary. There are perhaps fifteen core ideas, and everything else in this guide is a variation, combination or refinement of those fifteen. By Unit 5 you will start recognising the same three questions behind every algorithm: What am I predicting? How do I measure error? How do I avoid memorising?
Table of Contents
What ML is, how it differs from traditional programming, the three learning paradigms, real-world applications, and what ML cannot do.
The end-to-end pipeline, data collection, cleaning, missing values, encoding, feature scaling, feature engineering, and data leakage.
Linear regression, the cost function, gradient descent, multiple regression, polynomial features, Ridge, Lasso and Elastic Net.
Logistic regression, K-Nearest Neighbours, Naive Bayes, Decision Trees, and Support Vector Machines.
Confusion matrix, precision, recall, F1, ROC-AUC, regression metrics, bagging, Random Forest, AdaBoost, Gradient Boosting, XGBoost, stacking.
K-Means, hierarchical clustering, DBSCAN, the elbow method, silhouette score, PCA, t-SNE, and association rules.
Perceptron, activation functions, multilayer networks, backpropagation by hand, optimisers, CNNs, RNNs, transformers and transfer learning.
Bias–variance decomposition, underfitting vs overfitting, cross-validation, learning curves, grid and random search, Bayesian optimisation.
Text and NLP basics, recommender systems, time series, imbalanced data, and handling images and tabular data.
Saving models, APIs, containers, MLOps, data drift, fairness, explainability and privacy.
Vectors, matrices, derivatives, probability and statistics — only what ML actually needs.
Every important formula in the guide, on a few dense pages, for quick revision.
Plain-language definitions of around 120 key machine learning terms, alphabetically.
60 questions with model answers, grouped by topic and difficulty.
A realistic ninety-day study roadmap and the working habits that separate knowing from being able.
Introduction to Machine Learning
1.1 The problem machine learning solves
For most of computing history, programmers wrote rules. If you wanted a program that detected spam email, you wrote rules: if the message contains "free money", mark it spam. If it contains "click here now", mark it spam. If it contains more than five exclamation marks, mark it spam. This works, for a while. Then spammers write "fr33 m0ney" and your rules break. You add another rule. Then they write "free money" as an image. You add another rule. You are now maintaining four thousand rules, each one slightly contradicting the others, and the spammers are still winning because they only need to find one hole and you need to plug all of them.
Machine learning inverts the approach. Instead of writing the rules, you show the computer a large number of examples that already have the correct answer attached, and the computer works out the rules itself. You show it fifty thousand emails, each labelled spam or not spam. It finds the patterns. You never write a single rule about exclamation marks. And critically, when spammers change their tactics, you do not rewrite rules — you retrain on fresh examples.
Traditional programming: rules + data → answers. Machine learning: data + answers → rules. That inversion is the entire subject in one line. Everything else in this guide is detail about how the computer finds those rules, and how you check that the rules it found are actually good.
1.2 A formal definition
Tom Mitchell gave the standard definition in 1997, and it has survived because it is precise without being useless:
"A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T, as measured by P, improves with experience E."
Unpack that with the spam example. The task T is classifying email as spam or not spam. The performance measure P is the percentage of emails classified correctly. The experience E is the set of previously labelled emails. The program learns if, after seeing more labelled emails, its classification accuracy on new emails goes up. Note the word new — this is the single most important qualification in the whole definition, and Unit 8 is largely about it.
1.3 Machine learning versus traditional programming
| Aspect | Traditional programming | Machine learning |
|---|---|---|
| Logic source | Written by a human programmer | Learned from examples |
| Inputs | Rules and data | Data and correct answers |
| Output | Answers | A model (the learned rules) |
| Handling novel cases | Fails unless a rule covers it | Generalises from similar examples |
| Maintenance | Edit rules as requirements change | Retrain on new data |
| Explainability | High — you can read the rules | Often low, especially for deep learning |
| Best when | Rules are simple, exact, or safety-critical | Rules are complex, fuzzy, or unknown |
| Data needed | Little | Usually a lot |
Neither approach is superior in general. Nobody uses machine learning to compute income tax — the rules are known exactly and must be applied identically every time. But nobody writes hand-crafted rules to recognise a face in a photograph either, because the rules are impossible to articulate. The dividing line is whether you can write down the rules. If you can, write them. If you cannot but you have examples, use machine learning.
Simple baat: agar aapko rule pata hai, toh rule likh do — ML ki zaroorat nahi. Jaise income tax calculate karna, usme rule fixed hai. Lekin agar aap rule bata hi nahi sakte — "yeh photo mein billi hai ya kutta" — toh aap examples dikhao aur computer khud pattern nikaal lega. ML wahan kaam aata hai jahan rules likhna mushkil ho lekin examples available ho.
1.4 The three learning paradigms
Almost every machine learning method falls into one of three families, distinguished by what kind of data you have and what you want out of it. Getting this distinction clear early will make every later unit easier, because the moment you see a new algorithm you can ask "which family is this?" and immediately know what shape the solution has.
Supervised learning
You have input–output pairs and you want to learn the mapping between them. The "supervisor" is the labels: each training example comes with the correct answer, so the model can be told when it is wrong. This is by far the most commercially useful category and occupies most of this guide.
Supervised learning splits into two sub-problems depending on what kind of output you are predicting. If the output is a number on a continuous scale — house price, temperature, sales revenue — the task is regression, covered in Unit 3. If the output is a category — spam/not spam, cat/dog/bird, disease present/absent — the task is classification, covered in Unit 4.
Regression is filling in a blank with a number: "This flat in Pune will sell for ₹84 lakh." Classification is filing a card into a box: "This email goes in the Spam box." The question to ask is whether the answer is a quantity or a choice.
Unsupervised learning
You have inputs only, no labels. There is nothing to predict, so instead you look for structure: are there natural groupings in this data? Can it be described more compactly? This is what Unit 6 covers. Clustering groups similar items together — customer segmentation is the classic example, where nobody has defined the segments in advance and you want to discover whether they exist. Dimensionality reduction compresses many features into fewer while preserving as much information as possible, which helps both visualisation and model performance.
Supervised aur unsupervised ka fark ek line mein: supervised mein teacher hai, unsupervised mein nahi. Supervised mein data ke saath answer bhi diya hota hai, toh model check kar sakta hai ki sahi hai ya galat. Unsupervised mein sirf data hai — model khud dhoondhta hai ki isme koi pattern ya group bana raha hai ya nahi. Jaise ek dukaan ke customers ka data diya, aur poocha "kaun kaun se type ke customer hain?" — ye unsupervised hai, kyunki kisi ne pehle se category nahi banayi.
Reinforcement learning
There is no dataset at all. Instead there is an agent that takes actions in an environment and receives rewards. The agent's goal is to learn a policy — a rule for choosing actions — that maximises total reward over time. The hard part is that actions have delayed consequences: the reward for a good move in chess might only arrive thirty moves later, so the agent must solve the credit assignment problem, working out which of its earlier decisions deserve the credit. Reinforcement learning is behind game-playing systems, robot locomotion and much of modern AI alignment research. This guide mentions it for completeness but does not develop it, because it requires a different mathematical toolkit.
1.5 Supervised learning in more detail
Since supervised learning dominates both industry and coursework, it is worth stating its anatomy precisely now, because the same vocabulary recurs in every unit.
You begin with a dataset of n examples. Each example has features — the measurable properties — and, in training, a label, which is the correct answer. Features are conventionally written as x and the label as y. The training set is written as pairs: (x₁, y₁), (x₂, y₂), … (xₙ, yₙ).
Notation used throughout this guide
- n — number of examples (rows) in the dataset
- m or d — number of features (columns)
- x — one input example, a vector of m feature values
- y — the true label or target value
- ŷ (read "y-hat") — the model's predicted value
- w or θ — the model's parameters, the numbers it learns
- J or L — the loss or cost function, measuring how wrong the model is
The model is a function with parameters. You choose the form of the function — a straight line, a decision tree, a neural network — and the learning algorithm finds the parameter values that make it fit the training data well. A straight line ŷ = w₁x + w₀ has two parameters; a neural network with a million weights has a million parameters. In both cases, learning means searching for parameter values that minimise error.
Every supervised learning algorithm answers the same three questions, and they are the only three questions that matter:
1. What family of functions can this model represent? (Its hypothesis space
— a straight line, a tree, a deep network.)
2. How do we measure how wrong it is? (Its loss function — squared error,
cross-entropy, hinge loss.)
3. How do we search for the best parameters? (Its optimisation method — the
normal equation, gradient descent, greedy splitting.)
Once you can answer those three for an algorithm, you understand it. When you meet an unfamiliar algorithm later, ask these three questions and you will orient yourself quickly.
1.6 The kinds of problems machine learning solves
Beyond the label/no-label distinction, problems are often described by what the output looks like, and this vocabulary appears constantly in job descriptions and papers.
| Problem type | Family | Output | Example |
|---|---|---|---|
| Regression | Supervised | A continuous number | Predict tomorrow's temperature |
| Binary classification | Supervised | One of two classes | Will this customer churn, yes or no? |
| Multi-class classification | Supervised | One of many classes | Which of ten crop diseases is this leaf showing? |
| Multi-label classification | Supervised | Several labels at once | Which of these ten topics does this article cover? |
| Ranking | Supervised | An ordered list | Order these search results by relevance |
| Clustering | Unsupervised | Groups of similar items | Segment customers by behaviour |
| Dimensionality reduction | Unsupervised | A compressed representation | Compress 500 features into 10 |
| Anomaly detection | Either | Flag unusual items | Detect fraudulent transactions |
Note the difference between multi-class and multi-label, because they are frequently confused. In multi-class, exactly one label is correct — a fruit is an apple or a mango, never both. In multi-label, several can be correct simultaneously — a news article about cricket and economics carries both tags. The distinction matters because the modelling approach and the evaluation metrics differ.
1.7 Where machine learning is used
The following are not hypothetical. These are the applications you will encounter in interviews and in your first job.
Healthcare. Classifying medical images such as X-rays and retinal scans, predicting patient readmission risk, and accelerating drug discovery by predicting molecular properties. The stakes here make evaluation especially careful: a false negative in cancer screening is far worse than a false positive, which is why Unit 5 spends considerable time on metrics beyond accuracy.
Finance. Credit scoring, fraud detection in real time, algorithmic trading, and anti-money-laundering. These are dominated by tabular data and gradient-boosted trees, which is why Unit 5 covers those methods in depth.
E-commerce and retail. Product recommendations, demand forecasting, dynamic pricing, and churn prediction. Recommendation systems get their own section in Unit 9.
Manufacturing and agriculture. Predictive maintenance — spotting that a machine will fail before it does — plus yield prediction, quality inspection from images, and, increasingly relevant in India, crop disease detection from phone photographs.
Natural language. Spam filtering, sentiment analysis, machine translation, chatbots, and summarisation. Unit 9 introduces the fundamentals and Unit 7 touches on the transformer architecture behind modern language systems.
Transport and logistics. Route optimisation, arrival-time prediction, self-driving perception systems, and delivery demand prediction.
Think of machine learning as a very literal apprentice. Show the apprentice a thousand invoices and which ones were fraudulent, and it will start flagging suspicious ones — but it will also faithfully copy any bias in your thousand examples. If every fraudulent invoice in your history happened to come from one region, the apprentice will learn that region equals fraud, which may be both wrong and illegal. An apprentice has no judgement; it has only your examples. Choosing good examples is therefore a moral act, not just a technical one — a theme that returns in Unit 10.
1.8 What machine learning is not
Clearing up misconceptions early prevents disappointment later. Machine learning does not understand anything. It has found statistical regularities in data, and it will exploit any regularity that improves its score on the training data, including regularities that are accidental. A model trained to detect wolves in photographs famously learned to detect snow instead, because every wolf photograph in the dataset happened to be taken in snow. The model was not wrong about the data; it was wrong about the world.
Machine learning also needs data appropriate to the problem. A million examples of one thing and fifty of another will produce a model that is excellent at the first and useless at the second, no matter how sophisticated the algorithm. And machine learning cannot tell you whether your question is the right question. It optimises whatever you measure — so if you measure the wrong thing, you will get a model that is very good at the wrong thing.
Beginners believe that a more advanced algorithm will solve a weak dataset. It will not. In practice, data quality, feature quality and careful evaluation matter far more than algorithm choice. A logistic regression on well-engineered features routinely beats a neural network on poorly-prepared data. Spend your effort on Units 2 and 8 before you spend it chasing exotic models.
1.9 A first end-to-end example
Before the theory deepens, here is the complete shape of a machine learning project on a real dataset. Read it now to see where all the pieces fit; every later unit fills in one of these steps in detail. The dataset is the Iris dataset — 150 flowers, each described by four measurements, each belonging to one of three species. It is the "hello world" of machine learning and ships with scikit-learn, so this code runs as written.
# A complete, runnable machine learning project — 15 lines from sklearn.datasets import load_iris from sklearn.model_selection import train_test_split from sklearn.preprocessing import StandardScaler from sklearn.linear_model import LogisticRegression from sklearn.metrics import accuracy_score, classification_report # 1. Load data — X is features, y is labels X, y = load_iris(return_X_y=True) # 2. Split: train on 80%, keep 20% unseen for honest testing X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, random_state=42, stratify=y) # 3. Scale features (fit ONLY on training data — see Unit 2) scaler = StandardScaler().fit(X_train) X_train = scaler.transform(X_train) X_test = scaler.transform(X_test) # 4. Train the model model = LogisticRegression(max_iter=1000).fit(X_train, y_train) # 5. Evaluate on data the model has never seen preds = model.predict(X_test) print(f"Accuracy: {accuracy_score(y_test, preds):.3f}") print(classification_report(y_test, preds, target_names=load_iris().target_names))
Those five steps — load, split, prepare, train, evaluate — are the skeleton of every supervised learning task you will ever do, whether it is predicting crop yield or detecting fraud. The complexity lives entirely inside steps 3 and 4. Unit 2 is about step 3, Units 3 through 7 are about step 4, and Unit 8 is about doing step 5 honestly.
1.10 Practice questions
The Machine Learning Workflow & Data Preparation
2.1 Why this unit matters more than the algorithms
Ask any working data scientist how they spend their time and the answer is consistent: the overwhelming majority goes to finding, cleaning, understanding and preparing data. Training a model is often twenty lines of code. Making the data fit to train on is weeks.
This is not an accident of tooling. A model can only learn patterns that exist in the data you give it, expressed in a form it can use. If a date is stored as a string, the model sees meaningless text. If a salary column ranges into crores while age ranges to 60, most algorithms will treat salary as overwhelmingly important purely because its numbers are larger. If you scale your data before splitting it, you leak information from the test set and your reported accuracy becomes fiction.
The quality of your data preparation sets a ceiling on how good any model can be. A brilliant model on badly prepared data loses to an ordinary model on well-prepared data, nearly every time. This unit is not the boring part before the interesting part — it is the work.
2.2 The end-to-end machine learning workflow
Every serious project — whether a weekend Kaggle attempt or a system serving a million requests a day — follows the same broad sequence. Figure 2.1 shows it. Notice that it is a loop, not a line: after evaluating, you almost always go back to improve features, try another model, or collect more data.
2.3 Step 1 — Problem framing
This step is skipped constantly and it is the most common reason projects fail. Before touching data, answer four questions in writing.
What exactly am I predicting? "Predict customer behaviour" is not a problem. "Predict whether this customer will not log in for 30 days, using data available at the moment of prediction" is a problem. The second version tells you what the label is, what the features may contain, and when the prediction is made.
What does success look like in business terms, and what metric represents it? A model with 92% accuracy may be worthless if the business needs to catch 80% of fraud cases and it catches 40%. The metric must come from the decision the model supports, not from convenience.
Is this even a machine learning problem? Often the honest answer is that a lookup table, a business rule, or a simpler statistical summary would do the job with less risk and less maintenance.
What happens when the model is wrong? If the model is wrong 10% of the time, what is the cost? Does a human review the output? This determines how much precision is required and whether the model is acceptable at all.
Sabse pehla aur sabse important step: clear likho ki kya predict karna hai. "Customer behaviour predict karo" — ye problem statement nahi hai, ye ek wish hai. Sahi statement aisa hota hai: "aaj se agle 30 din mein customer login karega ya nahi, aur ye prediction aaj ke data se hi karni hai." Ye ek line aapko batati hai ki label kya hai, features kya ho sakte hain, aur prediction kis waqt karni hai. Aur ye last point critical hai — agar feature future ka data use kar liya, toh model bekaar ho jayega.
2.4 Step 2 — Data collection
Data comes from somewhere, and understanding where matters for knowing what can go wrong. Common sources are databases and data warehouses containing historical records, application logs and event streams, public datasets, third-party APIs, survey and sensor data, and increasingly, data generated by labelling services or by the users themselves.
Two properties of the collected data determine what models are possible. The first is volume: simple models work with hundreds of examples, deep models often need hundreds of thousands. The second is label quality. If your labels were produced by a human annotator working quickly, they contain errors, and your model will learn those errors as if they were truth. If your labels were derived from a proxy — say, "customer churned" inferred from a cancelled subscription when some cancellations were actually upgrades — the model learns the proxy, not the truth.
Never assume your labels are correct. Before training, inspect a random sample of a hundred labelled examples by hand. In real projects this catches label errors, ambiguous category definitions and inconsistent annotation rules — all of which cap model performance no matter what algorithm you use. If two annotators disagree on 30% of examples, no model can beat 70% on that data.
2.5 Step 3 — Exploratory data analysis and cleaning
Exploratory data analysis, usually abbreviated EDA, is the process of looking at your data before modelling it. Not running statistics — looking. Open the table, read some rows, plot some distributions, and ask whether anything is surprising.
The concrete questions to answer are these. How many rows and columns, and what does each column mean? What are the data types, and do they match what the column represents? How many missing values are there, in which columns, and is the missingness random or meaningful? What are the distributions of the numeric columns — are any heavily skewed, are there impossible values such as negative ages? What do the categorical columns contain — how many distinct values, and are any of them inconsistent spellings of the same thing? Is the target variable balanced, or is one class rare? And are there duplicate rows, which often indicate an error in data collection?
# The ten lines of EDA that should start every project import pandas as pd df = pd.read_csv("data.csv") print(df.shape) # rows × columns print(df.dtypes) # is a numeric column stored as text? print(df.isnull().sum()) # missing values per column print(df.describe()) # count, mean, std, min, quartiles, max print(df.duplicated().sum()) # exact duplicate rows for col in df.select_dtypes("object"): print(col, df[col].nunique(), df[col].unique()[:10]) # category consistency print(df["target"].value_counts(normalize=True)) # class balance print(df.corr(numeric_only=True)["target"].sort_values()) # linear signal
Handling missing values
Missing data is guaranteed. The question is not whether values are missing but why, because the reason determines the correct response.
Data is missing completely at random if the probability of missingness is independent of everything — a sensor that occasionally drops a reading. It is missing at random if missingness depends on other observed variables — men are less likely to report their weight, but weight is missing independently of the weight value itself. It is missing not at random if missingness depends on the missing value itself — people with the highest incomes are least likely to report income. The third case is the dangerous one, because any imputation will systematically bias your data.
| Strategy | How it works | When to use it |
|---|---|---|
| Drop rows | Remove examples with any missing value | Very few rows affected (under ~5%), and missingness is random |
| Drop column | Remove the feature entirely | The column is mostly missing and not important |
| Mean / median fill | Replace with the column average | Numeric, missing at random. Median if the column is skewed |
| Mode fill | Replace with the most frequent value | Categorical columns |
| Forward / backward fill | Carry the previous or next value | Time series data where values change smoothly |
| Constant fill | Replace with a sentinel like "Unknown" or −1 | When missingness itself is informative — the absence carries a signal |
| Model-based imputation | Predict the missing value from other columns | Important features with substantial missingness |
Never fill missing values using statistics computed over the whole dataset before splitting.
The mean you computed includes test-set rows, which leaks information. Compute the mean on the
training set only, then apply that same value to validation and test data. In scikit-learn,
SimpleImputer inside a Pipeline handles this correctly for you.
Handling outliers
An outlier is a value far from the rest of the distribution. Two things are true about outliers and they are often confused. First, an outlier may be a data entry error — an age of 456, a temperature of 900°C — in which case it is wrong and should be fixed or removed. Second, an outlier may be a genuine extreme value — a single customer who spent ₹50 lakh when everyone else spent ₹2,000 — in which case it is real and removing it discards the most informative example in your dataset.
The practical way to tell them apart is to investigate. Find the row, check the source system, and ask whether the value is physically possible. Do not delete outliers merely because they are inconvenient for a linear model. If genuinely extreme values distort a linear model, that is a signal to use a model robust to them, or to transform the feature, rather than to hide the evidence.
Two detection rules are worth knowing. The IQR rule flags any value below Q1 − 1.5×IQR or above Q3 + 1.5×IQR, where Q1 and Q3 are the 25th and 75th percentiles. It makes no assumption about the distribution and is the standard default. The z-score rule flags values more than three standard deviations from the mean, which assumes a roughly normal distribution and is itself distorted by the outliers it is trying to find.
Handling duplicates and inconsistent categories
Exact duplicates are usually errors and can be dropped — but check first. In some datasets genuine repeated transactions look identical, and removing them destroys real information. For categorical columns, the most common problem is inconsistent labelling: "Mumbai", "mumbai", "Bombay" and "MUMBAI " are four values representing one city. Left alone, the model treats them as four unrelated categories. The fix is standardisation — lowercasing, trimming whitespace, and mapping known variants to a canonical form.
2.6 Step 4 — Feature engineering
Feature engineering is the craft of turning raw columns into columns a model can learn from effectively. It is the highest-leverage activity in applied machine learning, and no algorithm can compensate for features that hide the signal.
Encoding categorical variables
Models do arithmetic. They cannot multiply the word "Delhi" by 0.7. Categorical variables must therefore be converted to numbers, and how you do it matters.
One-hot encoding creates a new binary column for each category. A "City" column with values Delhi, Mumbai and Pune becomes three columns, each 1 or 0. This is the default choice for nominal categories — categories with no natural order. Its drawback is dimensionality: a column with 5,000 distinct values becomes 5,000 columns, most of them almost entirely zeros.
Label encoding assigns each category an integer: Delhi = 0, Mumbai = 1, Pune = 2. This is compact and is the correct choice for ordinal categories, where order is meaningful — "Low", "Medium", "High" genuinely has a ranking, and encoding it as 0, 1, 2 preserves that. It is a mistake for nominal categories, however, because it invents an ordering that does not exist. If Delhi = 0 and Mumbai = 1, a linear model will treat Mumbai as "numerically larger" than Delhi, which is meaningless.
Target encoding replaces each category with the average target value for that category. It is powerful for high-cardinality features but must be computed with care, inside cross-validation, or it leaks the target and causes severe overfitting. It is worth knowing about but should not be your first tool.
One-hot vs label encoding ka confusion bahut common hai. Simple rule: agar category ka koi natural order hai — jaise "Small, Medium, Large" — toh label encoding theek hai, kyunki Small < Medium < Large ka matlab rakhta hai. Lekin agar order nahi hai — jaise "Delhi, Mumbai, Pune" — toh label encoding galat hai, kyunki model samjhega ki Mumbai > Delhi, jo bilkul bewakoofi hai. Aise cases mein one-hot encoding use karo. Rule yaad rakho: order hai toh number do, order nahi hai toh alag alag columns banao.
Feature scaling
Many algorithms compare feature values arithmetically. Consider a dataset with two features: age, ranging 18 to 60, and salary, ranging 15,000 to 20,00,000. In a distance-based algorithm such as K-Nearest Neighbours, the distance between two people is dominated entirely by salary — a difference of ₹1,00,000 contributes a million times more to the squared distance than a difference of 1 year in age. Age effectively disappears from the model.
Scaling solves this by putting features on comparable ranges. Two methods dominate.
x′ = (x − μ) / σ Subtracts the mean and divides by the standard deviation. The result has mean 0 and standard deviation 1. No fixed range. Preferred when a feature is roughly normally distributed, and it is the safer default with outliers present.
x′ = (x − min) / (max − min) Rescales linearly into the range [0, 1]. Sensitive to outliers, because a single extreme value compresses everything else into a narrow band. Preferred when you need a bounded range, as with neural networks or image pixel values.
Which algorithms need scaling? The rule is simple: algorithms that depend on distances or on gradient magnitudes need it; algorithms that depend only on ordering do not.
Scaling required: K-Nearest Neighbours, Support Vector Machines, K-Means
clustering, Principal Component Analysis, linear and logistic regression trained by gradient
descent, and all neural networks.
Scaling unnecessary: decision trees, Random Forest, and gradient boosting,
because they split on thresholds and the split point moves with the scale, so the resulting
tree is identical either way.
Creating new features
The most valuable features are often not in the data at all — they are constructed. A raw timestamp column is nearly useless; decomposed into hour of day, day of week, month, and is-weekend, it becomes highly informative, because fraud, shopping and travel all follow strong temporal rhythms. From length and width, a "floor area" feature may carry signal neither dimension carries alone. A "days since last purchase" column is far more predictive of churn than the raw date of last purchase.
Transformations also help. A heavily right-skewed feature such as income can be made well-behaved with a logarithmic transform, which compresses large values and spreads small ones. Binning turns a continuous variable into categories — useful when the relationship is non-linear and step-shaped, for example "age under 25", "25 to 40", "over 40" — though it throws away information and should not be a first resort.
Feature engineering is domain knowledge made explicit. This is why a data scientist who understands agriculture will beat a better programmer who does not, on a crop-yield problem. The algorithm is generic and freely available to everyone; the insight that flowering date matters more than planting date is not. When you start a new domain, spend the first week talking to people who understand it.
2.7 Step 5 — Splitting the data
You cannot judge a model on the data it learned from. A student who has memorised the answers to last year's exam paper may score perfectly on that paper and fail the real exam. Models behave identically, and much more extremely. The solution is to hold data back.
The standard split is training, validation and test sets, typically 60/20/20, or 80/20 when cross-validation is used instead of a separate validation set. Each has a distinct job and they must not be confused.
| Set | Typical size | Purpose |
|---|---|---|
| Training | 60–80% | The model learns its parameters from this. The model may — and should — fit this data closely. |
| Validation | 10–20% | Used to compare models and tune hyperparameters. You may look at this repeatedly, and this is exactly why it must not be the test set. |
| Test | 10–20% | Touched exactly once, at the very end, to report final performance. Its entire value lies in having been untouched. |
The most damaging mistake in applied machine learning is tuning repeatedly against the test set. If you check test performance, adjust, check again, adjust again — you are fitting to the test set, and your reported number becomes optimistic. This is called test set leakage. The test set's value comes entirely from being unexamined. Use validation data or cross-validation for all decisions; open the test set once.
Two refinements matter for specific data shapes. When classes are imbalanced, use stratified splitting, which preserves the class proportions in each set — otherwise a rare class might vanish from the test set entirely. When data is a time series, split chronologically: train on the earliest period, test on the latest. Random splitting a time series lets the model train on the future and predict the past, which produces a beautiful accuracy score and a useless model.
2.8 Data leakage — the silent killer
Data leakage is the use of information during training that will not be available when the model is actually used. It produces models that look excellent in development and fail in production, and it is the single most common serious error in applied machine learning.
There are three recurring forms. Target leakage is including a feature that is a consequence of the target rather than a cause. Predicting whether a patient has a disease, using a "prescribed medication" column, gives near-perfect accuracy — because the medication was prescribed after diagnosis. The feature is not available at prediction time in the real world. Train–test contamination is fitting any preprocessing — scaling, imputation, feature selection, target encoding — on the combined dataset before splitting, so that test-set statistics influence training. Temporal leakage is using future information to predict the past.
# WRONG — scaling before splitting leaks test statistics into training X_scaled = StandardScaler().fit_transform(X) # sees ALL data X_train, X_test, y_train, y_test = train_test_split(X_scaled, y) # RIGHT — Pipeline fits the scaler on training folds only, every time from sklearn.pipeline import Pipeline from sklearn.preprocessing import StandardScaler from sklearn.linear_model import LogisticRegression pipe = Pipeline([ ("scaler", StandardScaler()), ("clf", LogisticRegression()), ]) pipe.fit(X_train, y_train) # scaler sees only X_train score = pipe.score(X_test, y_test) # honest score
Leakage samajhna asaan hai — model ko exam se pehle answer key mil gayi. Toh score 100% aayega, lekin real exam mein fail. Do sabse common galtiyan: pehli, kisi aise column ko feature bana dena jo target ke baad milta hai — jaise "dawai di gayi thi kya" se predict karna ki patient ko bimari hai. Dawai toh bimari ke baad hi mili, toh ye cheating hai. Doosri, scaling ya imputation poore dataset par karna, split se pehle. Solution ek hi hai: Pipeline use karo, aur hamesha poochho — "kya ye information prediction ke waqt actually available hogi?"
2.9 A complete worked example
Suppose you have 10 customer records and want to predict churn. The raw data is below.
| Age | City | Salary | Plan | Tenure | Churned |
|---|---|---|---|---|---|
| 23 | Delhi | 25000 | Basic | 3 | Yes |
| 45 | mumbai | 85000 | Premium | 24 | No |
| 31 | Pune | ? | Basic | 8 | No |
| 52 | MUMBAI | 150000 | Premium | 36 | No |
| 29 | Delhi | 42000 | Basic | 5 | Yes |
| 38 | Pune | 67000 | Standard | 15 | No |
| 26 | delhi | 31000 | Basic | 4 | Yes |
| 44 | Mumbai | 92000 | Premium | 28 | No |
| 35 | Pune | 58000 | Standard | 12 | No |
| 21 | Delhi | ? | Basic | 2 | Yes |
Problem identified 1 — inconsistent categories. The City column contains Delhi, delhi, Mumbai, mumbai, MUMBAI, Pune. These are three cities, not six values. Standardise to Delhi, Mumbai, Pune.
Problem identified 2 — missing values. Salary is missing for rows 3 and 10. Check whether missingness is random. Here, assume it is. Compute the mean over the 8 known values: (25000 + 85000 + 150000 + 42000 + 67000 + 31000 + 92000 + 58000) / 8 = 550000 / 8 = 68,750. Fill both gaps with 68,750.
Problem identified 3 — categorical encoding. City is nominal, so one-hot encode into three columns. Plan has a natural order — Basic < Standard < Premium — so label encode it as Basic = 0, Standard = 1, Premium = 2. Target Churned is binary, encode Yes = 1, No = 0.
Problem identified 4 — scaling. Age spans 21–52 while Salary spans 25,000–150,000. Any distance-based model would ignore age entirely. Since Salary is right-skewed and contains a large outlier relative to the rest, standardise both columns.
First split, then compute scaling statistics on the training portion only. Suppose the training set is rows 1–8. Age mean = (23 + 45 + 31 + 52 + 29 + 38 + 26 + 44)/8 = 288/8 = 36, standard deviation ≈ 9.9. Salary mean over rows 1–8 (with 68,750 filled) = (25000 + 85000 + 68750 + 150000 + 42000 + 67000 + 31000 + 92000)/8 = 560750/8 = 70,094. Standard deviation ≈ 38,600.
Scaled age for row 1 = (23 − 36) / 9.9 = −1.31. Scaled salary for row 1 = (25000 − 70094) / 38600 = −1.17.
Both features now sit on a comparable scale. Row 9 (in the test set) is scaled using the same training mean and standard deviation — never statistics computed from the test set itself.
Final prepared feature row for customer 1: Age = −1.31, Salary = −1.17, Tenure = 3, City_Delhi = 1, City_Mumbai = 0, City_Pune = 0, Plan = 0, Target = 1.
Eight original columns became seven numeric features plus a target — and every transformation is now something a model can actually use.
2.10 Putting it together in code
# A production-quality preprocessing pipeline import pandas as pd from sklearn.model_selection import train_test_split from sklearn.compose import ColumnTransformer from sklearn.pipeline import Pipeline from sklearn.impute import SimpleImputer from sklearn.preprocessing import StandardScaler, OneHotEncoder from sklearn.ensemble import RandomForestClassifier from sklearn.metrics import roc_auc_score df = pd.read_csv("churn.csv") # Standardise messy category labels df["City"] = df["City"].str.strip().str.title() X = df.drop(columns=["Churned"]) y = (df["Churned"] == "Yes").astype(int) # Split FIRST, before any fitting of preprocessing steps X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, random_state=42, stratify=y) numeric_cols = ["Age", "Salary", "Tenure"] categorical_cols = ["City", "Plan"] numeric_pipe = Pipeline([ ("impute", SimpleImputer(strategy="median")), ("scale", StandardScaler()), ]) categorical_pipe = Pipeline([ ("impute", SimpleImputer(strategy="most_frequent")), ("encode", OneHotEncoder(handle_unknown="ignore")), ]) preprocessor = ColumnTransformer([ ("num", numeric_pipe, numeric_cols), ("cat", categorical_pipe, categorical_cols), ]) model = Pipeline([ ("prep", preprocessor), ("model", RandomForestClassifier(n_estimators=200, random_state=42)), ]) model.fit(X_train, y_train) probs = model.predict_proba(X_test)[:, 1] print(f"Test ROC-AUC: {roc_auc_score(y_test, probs):.3f}") # The whole pipeline — imputation, scaling, encoding, model — is one object. # Save it, ship it, and it behaves identically in production. See Unit 10.
2.11 Practice questions
Regression
3.1 What regression is
Regression predicts a number. Not a category, not a label — a quantity on a continuous scale. House prices, tomorrow's temperature, the number of units a shop will sell, crop yield per acre, a patient's blood pressure. The distinguishing feature is that the output can take any value in a range, and being slightly wrong is acceptable, unlike classification where being wrong means choosing the wrong category entirely.
The idea is older than machine learning. Francis Galton fitted lines to data on parents' and children's heights in the 1880s and observed that tall parents tended to have somewhat shorter children and vice versa — a phenomenon he called "regression to the mean". The name stuck, even though the technique has nothing to do with reverting.
Imagine plotting dots on graph paper: on the x-axis, the size of a house; on the y-axis, its price. The dots form a rough upward cloud — bigger houses cost more — but they are scattered, because size is not the only thing that determines price. Regression draws the single best straight line through that cloud, then uses the line to read off a predicted price for any house size, including sizes not in your data. The whole of Unit 3 is about what "best" means and how to find it.
3.2 Simple linear regression
Simple linear regression assumes the relationship between one input and the output is a straight line. With one feature x and target y:
x — the input feature
w₁ — the slope or weight: how much ŷ changes when x increases by one unit
w₀ — the intercept or bias: the predicted value when x is zero
The parameters w₁ and w₀ are what learning finds. The model form — a straight line — is chosen by you. This is the hypothesis space from Unit 1, and it is a deliberately restrictive one: a straight line cannot bend, so it can only capture relationships that happen to be linear.
Once the line is chosen, prediction is trivial: plug in x, get ŷ. The interesting question is how to choose the line.
3.3 The cost function: how do we measure "best"?
We need a single number that says how bad a particular line is, so that we can search for the line that makes it as small as possible. The standard choice is the mean squared error.
For each data point, the residual is the vertical distance between the actual value and the line's prediction: yᵢ − ŷᵢ. A good line has small residuals. Squaring them does two things: it makes all residuals positive so positive and negative errors cannot cancel, and it punishes large errors much more than small ones — an error of 10 contributes 100, while ten errors of 1 contribute only 10.
J(w₀, w₁) = (1/n) Σᵢ₌₁ⁿ (yᵢ − ŷᵢ)² = (1/n) Σᵢ₌₁ⁿ (yᵢ − (w₁xᵢ + w₀))² n — number of data points
yᵢ — the true value of the i-th point
ŷᵢ — the predicted value for the i-th point
J — the cost, a function of the parameters. Smaller is better.
Note carefully what J is a function of. It is not a function of x or y — those are fixed, given by the data. J is a function of w₀ and w₁, the parameters. For every pair of parameter values you can compute one number: the average squared error that line would produce. Learning means finding the pair that makes this number smallest.
This is the central move of all of machine learning, and it is worth pausing on. We have turned "find a good line" — a vague notion — into "minimise a function of the parameters" — a precise mathematical problem. Every algorithm in this guide does the same thing with a different cost function. Classification uses cross-entropy (Unit 4), SVMs use hinge loss (Unit 4), neural networks use whatever you choose for the output layer (Unit 7). Learn this pattern once and you will recognise it everywhere.
Cost function ko ek scorecard ki tarah samjho. Har possible line ke liye ek score milta hai — batata hai ki line kitni kharab hai. Better line = lower score. Ab learning ka matlab sirf itna hai: aisi line dhoondo jiska score sabse kam ho. Ye baat yaad rakho — poore ML ka basic structure yahi hai. Bas cost function badalta hai, method badalta hai, lekin idea wahi: error ko number banao, phir usko minimum karo.
Why squared error rather than absolute error? Three reasons. Squared error is differentiable everywhere, which makes the calculus clean. It corresponds to the maximum likelihood estimate under the assumption that errors are normally distributed, which is often reasonable. And its cost function is convex for linear regression — it has a single global minimum shaped like a bowl, so there is no risk of getting stuck in a local dip. The downside is sensitivity to outliers, which is why absolute error is sometimes preferred when outliers are known to be present.
3.4 Finding the best line, method 1: the normal equation
For linear regression there is a closed-form solution. Because J is a convex quadratic, we can use calculus to find where its gradient is zero and solve directly. Writing the model in matrix form with a column of ones appended to X to absorb the intercept:
y — the vector of true values
w — the vector of parameters that minimises MSE
This gives the exact optimum in one step, with no iteration and no learning rate to tune. So why does anyone use gradient descent? Because computing (XᵀX)⁻¹ costs roughly O(m³) in the number of features. With 50 features that is instant; with 1,00,000 features — as in text classification or genomics — it is impossible. Gradient descent scales to such problems, and more importantly, it generalises to models where no closed form exists at all, such as logistic regression and neural networks. It is the workhorse of the entire field.
3.5 Finding the best line, method 2: gradient descent
Gradient descent is an iterative search. Start with any parameter values — often all zeros. Compute the direction in which the cost increases most steeply. Take a small step in the opposite direction. Repeat until you stop improving.
You are standing on a foggy hillside and want to reach the bottom of the valley. You cannot see the valley, but you can feel the slope under your feet. So you take a step downhill, feel the slope again, take another step, and repeat. Eventually the ground feels flat — you are at the bottom. The fog is the reason you cannot simply look and walk straight there; you only have local information. The size of your step is the learning rate, and it turns out to matter enormously.
3.6 Gradient descent for linear regression, step by step
The gradient of the MSE cost with respect to each parameter is:
With those gradients, one update step is:
The minus sign because we move against the gradient, downhill
3.7 The learning rate — the most important number to get right
The learning rate α controls how far you move at each step. It is not learned by the algorithm; you choose it. It is a hyperparameter, as opposed to the weights, which are parameters. Getting α wrong is the most common reason gradient descent fails.
| Learning rate | What happens | Symptom you will observe |
|---|---|---|
| Too large | Steps overshoot the minimum; the cost oscillates or grows without bound | Cost increases, or jumps around, or becomes NaN |
| Too small | Steps are tiny; convergence takes enormous time | Cost decreases very slowly; training seems stuck |
| Just right | Cost decreases smoothly and flattens | Cost curve falls and levels off within a reasonable number of steps |
Diagnose gradient descent by plotting the cost against iteration number. A healthy run shows a curve that drops steeply then flattens. A rising or oscillating curve means α is too large — reduce it by a factor of three or ten. A nearly flat curve means α is too small. This single plot tells you more about training than any other diagnostic, and it is the first thing to check when a model will not learn.
Learning rate ko pahadi se utarne wale kadam ki tarah socho. Bahut bada kadam liya toh aap valley ke us paar ja pahunchoge, phir wapas idhar, phir udhar — kabhi neeche nahi pahunchoge, bas idhar udhar uchhalte rahoge. Bahut chhota kadam liya toh neeche toh pahunchoge, lekin itni dheere ki raat ho jayegi. Sahi kadam = jaldi bhi, aur seedha neeche. Practical tip: 0.01 se shuru karo. Agar cost badh raha hai toh 0.001 karo. Agar bahut slow hai toh 0.1 try karo. Aur hamesha cost vs iteration ka graph dekho — wahi batayega ki kya galat ho raha hai.
Batch, stochastic and mini-batch gradient descent
The method above computes the gradient over all n data points before taking a step. That is batch gradient descent: stable, but each step costs a full pass over the data, which is prohibitive when n is in the millions.
Stochastic gradient descent (SGD) takes a step after every single example. Steps are cheap, but wildly noisy — the path zigzags toward the minimum rather than descending smoothly. This noise is not purely bad: it can help escape shallow local minima.
Mini-batch gradient descent is the practical compromise, stepping after batches of 32, 64, 128 or 256 examples. It is what "SGD" almost always means in practice and what every deep learning framework uses by default. It balances the stability of batch descent against the speed of stochastic descent, and it maps efficiently onto GPU hardware.
3.8 Worked example: three steps of gradient descent by hand
Data. Four houses. Feature x = size in hundreds of square feet, target y = price in lakh rupees.
| x (size) | y (price) | ŷ = 0.5x + 0 (initial guess) | error = ŷ − y |
|---|---|---|---|
| 1 | 2 | 0.5 | −1.5 |
| 2 | 4 | 1.0 | −3.0 |
| 3 | 6 | 1.5 | −4.5 |
| 4 | 8 | 2.0 | −6.0 |
The true relationship here is obviously y = 2x, so the correct answer is w₁ = 2, w₀ = 0. Let us see whether gradient descent finds it. Start with w₁ = 0.5, w₀ = 0, learning rate α = 0.05.
Step 0 — initial cost. Predictions are 0.5, 1.0, 1.5, 2.0 against targets 2, 4, 6, 8. Errors are −1.5, −3.0, −4.5, −6.0. J = (2.25 + 9 + 20.25 + 36) / 4 = 67.5 / 4 = 16.875. That is a large cost — the line is well below the data.
Step 1 — compute gradients.
For the intercept: ∂J/∂w₀ = (2/4) × (−1.5 − 3.0 − 4.5 − 6.0) = 0.5 × (−15) = −7.5. Negative, so increasing w₀ lowers the cost — correct, since the line sits too low.
For the slope: ∂J/∂w₁ = (2/4) × [(−1.5×1) + (−3.0×2) + (−4.5×3) + (−6.0×4)] = 0.5 × (−1.5 − 6 − 13.5 − 24) = 0.5 × (−45) = −22.5. Also negative, so w₁ should increase — the line is too flat as well as too low.
Update the parameters.
w₀ = 0 − 0.05 × (−7.5) = 0 + 0.375 = 0.375
w₁ = 0.5 − 0.05 × (−22.5) = 0.5 + 1.125 = 1.625
Step 2 — recompute. With ŷ = 1.625x + 0.375, predictions are 2.0, 3.625, 5.25, 6.875. Errors: 0.0, −0.375, −0.75, −1.125. Cost J = (0 + 0.1406 + 0.5625 + 1.2656)/4 = 1.9688/4 = 0.492. The cost has fallen from 16.875 to 0.492 — a 97% reduction in one step.
Step 3 — gradients again. ∂J/∂w₀ = 0.5 × (−2.25) = −1.125, so w₀ = 0.375 + 0.0563 = 0.431. ∂J/∂w₁ = 0.5 × [(0×1) + (−0.375×2) + (−0.75×3) + (−1.125×4)] = 0.5 × (−7.5) = −3.75, so w₁ = 1.625 + 0.1875 = 1.813.
Where this is heading. Continuing, w₁ approaches 2 and w₀ approaches 0 — the exact correct line. Notice how the steps automatically shrink: the first update moved w₁ by 1.125, the second by only 0.1875. As the slope flattens near the minimum, the gradient itself becomes small, so the algorithm takes smaller steps without any intervention. This is the elegant self-regulating behaviour of gradient descent.
| Iteration | w₀ | w₁ | Cost J |
|---|---|---|---|
| 0 | 0.000 | 0.500 | 16.8750 |
| 1 | 0.375 | 1.625 | 0.4922 |
| 2 | 0.431 | 1.813 | 0.0432 |
| 3 | 0.443 | 1.943 | 0.0049 |
| … | … | … | … |
| ∞ | 0.000 | 2.000 | 0.0000 |
3.9 Multiple linear regression
Real problems have more than one feature. House price depends on size, bedrooms, location, age and more. The model extends naturally:
Each wⱼ measures the effect of feature j holding all others constant
That phrase in the last line is the whole point and it is easily missed. In multiple regression, w₃ does not mean "larger x₃ means larger y". It means "if two houses are identical in every other feature but differ by one unit in x₃, the predicted price differs by w₃". This is how the model controls for confounding. A naive analysis might find that neighbourhoods with more hospitals have sicker residents; multiple regression reveals that age explains both, and once age is included the hospital effect shrinks toward zero.
Interpreting a regression coefficient as a causal effect. Regression coefficients describe association in your data, given the features you included. If an important confounder is missing, the coefficient absorbs its effect and will mislead you. A regression of ice-cream sales on drowning deaths gives a large positive coefficient — not because ice cream causes drowning, but because both rise in summer, and season was not in the model. Predictive models do not need causal correctness; interpreted coefficients absolutely do.
Feature importance from coefficients, and why it can mislead
A convenient property of linear regression is that coefficients look like importance scores: larger absolute value means more influence. This is only valid if all features are on the same scale. If x₁ is age in years (range 18–60) and x₂ is income in rupees (range 10,000–10,00,000), the coefficient on income will be tiny purely because a one-unit change in income is negligible. Always standardise features before comparing coefficients. Even then, correlated features split their shared effect between them, so neither individual coefficient reflects the true importance of the underlying concept.
3.10 Polynomial regression and the bias–variance trade-off
If the relationship is curved, a straight line will underfit. The remedy is to add polynomial terms: x², x³ and so on. The model is still linear in its parameters — it is still a weighted sum, so the same fitting methods apply — but the resulting curve can bend.
Increasing the degree increases flexibility. Degree 1 is a straight line, degree 2 a parabola, degree 10 a wildly wiggling curve that can pass through almost any set of points. This is the essential trade-off of all of machine learning, and polynomial degree makes it visible in a single picture.
Overfitting ko aise samjho: student ne sirf previous year ke question papers rat liye. Us paper mein 100% marks, lekin naya paper aaya toh fail — kyunki usne pattern samjha hi nahi, sirf answers yaad kiye the. Model bhi wahi karta hai. Degree-10 polynomial har training point se guzar jata hai, training error almost zero, lekin naya data aaya toh bilkul galat prediction. Iska ulta underfitting hai — student ne itna kam padha ki basics bhi nahi aaye. Solution: na bahut simple model, na bahut complex. Aur isi ko balance karne ke liye regularisation use karte hain, jo abhi aata hai.
3.11 Regularisation: Ridge, Lasso and Elastic Net
Overfitting happens when the model's parameters grow large to fit noise. Regularisation adds a penalty for large parameters directly into the cost function, so the model must balance fitting the data against keeping the weights small.
The parameter λ (lambda) controls the strength of the penalty, and it is a hyperparameter you tune. When λ = 0 there is no regularisation and the model is ordinary regression. As λ grows, weights shrink, the model becomes simpler, and the risk shifts from overfitting toward underfitting. Note that the intercept w₀ is conventionally not penalised — shrinking the intercept would simply shift all predictions and make the model worse for no benefit.
| Method | Penalty term | Effect on weights | Use when |
|---|---|---|---|
| Ridge | Σ wⱼ² | Shrinks all toward zero, none exactly zero | Many features, all contributing a little; correlated features |
| Lasso | Σ |wⱼ| | Sets some weights exactly to zero | You suspect only a few features matter and want a sparse model |
| Elastic Net | Both | Shrinks and selects | Many correlated features, and you still want selection |
Why does Lasso produce exact zeros while Ridge does not? Geometrically, the L1 penalty constrains weights to a diamond shape and the L2 penalty to a circle. The cost function's contours are ellipses centred on the unregularised optimum. An ellipse expanding outward hits a diamond at a corner — and corners lie on the axes, where one weight is zero — far more readily than it hits a circle at a point with a coordinate of exactly zero. For algorithms where sparsity matters, this difference is the whole point.
Regularisation requires scaled features. The penalty Σwⱼ² sums weights across features, but if one feature is measured in rupees and another in years, their weights live on incomparable scales, and the penalty is applied unevenly — effectively penalising the large-scale feature more. Always standardise before regularising. This is one place where forgetting to scale silently degrades your model rather than throwing an error.
3.12 Evaluating a regression model
Four metrics cover essentially every regression problem.
Choose your metric from the cost of being wrong, not from habit. If being off by 10 units is twice as bad as being off by 5, use MAE. If a single large error is catastrophic — say, under-forecasting demand and running out of stock — use RMSE, which punishes exactly that. R² is useful for comparing models on the same data because it is scale-free, but it says nothing about whether the errors are acceptable in business terms.
3.13 Code: linear regression and regularisation in scikit-learn
import numpy as np from sklearn.datasets import fetch_california_housing from sklearn.model_selection import train_test_split from sklearn.preprocessing import StandardScaler, PolynomialFeatures from sklearn.linear_model import LinearRegression, RidgeCV, LassoCV, ElasticNet from sklearn.pipeline import Pipeline from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score X, y = fetch_california_housing(return_X_y=True) X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=42) def evaluate(name, model): model.fit(X_tr, y_tr) pred = model.predict(X_te) print(f"{name:28s} RMSE={mean_squared_error(y_te, pred, squared=False):.3f} " f"MAE={mean_absolute_error(y_te, pred):.3f} R2={r2_score(y_te, pred):.3f}") # 1. Plain linear regression, scaled evaluate("Linear", Pipeline([("s", StandardScaler()), ("m", LinearRegression())])) # 2. Ridge with built-in cross-validated alpha selection from sklearn.linear_model import RidgeCV evaluate("Ridge (auto alpha)", Pipeline([("s", StandardScaler()), ("m", RidgeCV(alphas=np.logspace(-3, 3, 30)))])) # 3. Lasso — inspect which coefficients it zeroed out pipe = Pipeline([("s", StandardScaler()), ("m", LassoCV(cv=5, random_state=0))]) evaluate("Lasso (auto alpha)", pipe) coefs = pipe.named_steps["m"].coef_ print(f"Lasso kept {(coefs != 0).sum()} of {len(coefs)} features") # 4. Polynomial features — watch it overfit if degree is too high for deg in [1, 2, 3]: evaluate(f"Poly deg={deg}", Pipeline([ ("p", PolynomialFeatures(degree=deg, include_bias=False)), ("s", StandardScaler()), ("m", RidgeCV(alphas=np.logspace(-3, 3, 30)))]))
CV variants tune the penalty automatically by cross-validation.3.14 Practice questions
∂J/∂w₀ = (2/3)(−2 − 3 − 4) = (2/3)(−9) = −6.
∂J/∂w₁ = (2/3)[(−2)(1) + (−3)(2) + (−4)(3)] = (2/3)(−2 − 6 − 12) = (2/3)(−20) = −13.33.
Updates: w₀ = 0 − 0.1(−6) = 0.6. w₁ = 1 − 0.1(−13.33) = 2.333.
The true relationship is y = 2x + 1, and after one step the parameters have moved strongly toward w₁ = 2, w₀ = 1 — from a starting point that was badly wrong.
Classification Algorithms
4.1 The classification problem
Classification predicts which category something belongs to. Will this customer churn or not? Is this tumour benign or malignant? Which of five species is this bird? The output is a choice, not a quantity, and that changes everything about how models are built and judged.
The crucial difference from regression is what "being wrong" means. In regression, being wrong by a little is nearly as good as being right. In classification, predicting class B when the truth is class A is simply incorrect — there is no partial credit. This is why classification needs probabilities and decision thresholds rather than a single predicted number, and it is why Unit 5 spends so long on metrics: whether a model is good depends entirely on which mistakes you care about.
Every classification algorithm produces a score, and a separate step turns that score into a label. Logistic regression produces a probability; a threshold of 0.5 turns it into a class. KNN produces a vote share. An SVM produces a distance from a boundary. Keeping those two stages separate in your mind is the key to understanding classification, because the decision threshold is something you control independently of the model — and adjusting it is often the single most effective improvement you can make.
4.2 Logistic regression
Despite the name, logistic regression is a classification algorithm. The name is historical: it uses a logistic function to model a probability, and probability is continuous, so the machinery resembles regression. If that naming decision seems unfortunate, it is — and it confuses everyone at first.
Why we cannot just use linear regression
A natural first idea: fit a straight line and treat outputs above 0.5 as class 1. This fails for a specific and instructive reason. A straight line is unbounded — it can predict 1.7, or −3.4, or 85. These are not probabilities, and there is no sensible way to interpret them. Worse, adding a single extreme data point swings the line's slope dramatically, changing predictions for every other point. A model whose decisions depend heavily on one outlier is not a model you want.
So we need a function that takes any real number — from −∞ to +∞ — and squashes it into the range 0 to 1. That function is the sigmoid.
As z → +∞, σ(z) → 1. As z → −∞, σ(z) → 0. At z = 0, σ(z) = 0.5.
p = σ(w₀ + w₁x₁ + … + wₘxₘ) = 1 / (1 + e^−(w₀ + w₁x₁ + … + wₘxₘ)) p — the probability that this example belongs to class 1
Prediction rule: class 1 if p ≥ 0.5, otherwise class 0
Sigmoid ka kaam samajhna asaan hai — ye ek squasher hai. Aapka linear model koi bhi number de sakta hai, chahe −50 ho ya +200. Lekin probability −50 nahi ho sakti, probability sirf 0 se 1 ke beech hoti hai. Toh sigmoid us number ko 0 aur 1 ke beech daba deta hai. Bada positive number aaya toh 1 ke kareeb, bada negative aaya toh 0 ke kareeb, aur exactly 0 aaya toh 0.5. Isliye hum kehte hain: pehle score banao, phir sigmoid se usko probability banao. Aur wahi 0.5 wala point decision boundary hai.
The cost function for logistic regression
Mean squared error does not work here, and it is worth understanding why. Plugging the sigmoid into MSE gives a cost function that is non-convex — a lumpy landscape full of local minima, where gradient descent can get stuck far from the true optimum. We need a cost function that is convex for this model.
The right choice is log loss, also called binary cross-entropy. Intuitively, it punishes confident mistakes savagely: if the model says "99% certain this is class 1" and the truth is class 0, the penalty is enormous; if it says "51% certain" and is wrong, the penalty is small.
J = −(1/n) Σᵢ₌₁ⁿ [ yᵢ log(pᵢ) + (1 − yᵢ) log(1 − pᵢ) ] yᵢ — the true label, 0 or 1
pᵢ — the predicted probability of class 1
When y = 1 only the first term survives; when y = 0 only the second. Both cases penalise being confidently wrong.
Notice what happens at the extremes. If the true label is 1 and the model predicts p = 1, the term is log(1) = 0 — no penalty. If the model predicts p = 0.01, the term is log(0.01) ≈ −4.6, and the negative sign in front makes it a penalty of +4.6. Because log approaches −∞ as p approaches 0, a confident mistake carries an essentially infinite penalty, which is exactly the behaviour we want.
Log loss and MSE are not arbitrary choices — they come from probability theory. Minimising log loss is equivalent to maximising the likelihood of the observed labels under the model. Minimising MSE is equivalent to maximum likelihood under a Gaussian noise assumption, which is natural for regression and wrong for binary labels. The deeper lesson: choose the loss that matches the probabilistic assumption of your target. This principle returns in Unit 7 with softmax cross-entropy for multi-class problems.
4.3 K-Nearest Neighbours
K-Nearest Neighbours is the simplest classification idea possible, and it is worth studying precisely because of that simplicity: it reveals what "learning" can mean when there are no parameters at all.
The method stores the entire training set — there is no training phase in the usual sense. To classify a new point, find the k training points closest to it in feature space, and take a majority vote of their labels. If k = 5 and three neighbours are class A and two are class B, predict A.
You move to a new neighbourhood and want to guess the political leaning of your street. You ask your five nearest neighbours and go with the majority. If you lived on a boundary between two areas, the answer would depend heavily on whether you asked 3 people or 15 — that is the effect of k. And if you asked someone's opinion about a topic where their answer varies wildly, you would be misled by their confidence — which is why distance-weighted voting helps when neighbours vary in how close they are.
Choosing k
The value of k controls the bias–variance trade-off directly, and both extremes fail in instructive ways.
With k = 1, each point is classified by its single nearest neighbour. The decision boundary becomes extremely jagged, wrapping around every individual training point. Training error is zero — every point is its own nearest neighbour — but the model has memorised noise. This is maximum variance, minimum bias.
With a very large k, the neighbourhood covers a large region, and the model predicts the majority class almost everywhere. Boundaries become smooth and eventually vanish entirely. This is minimum variance, maximum bias.
KNN mein k ka matlab: kitne padosiyon se poochhna hai. k = 1 rakha toh sirf sabse nazdeek wale ek bande se poochhoge — agar woh galti se outlier hai, aapka answer galat. Bahut bada k rakha (jaise 100) toh aap itne door ke logon se poochhoge ki unka koi lena-dena hi nahi — answer hamesha majority class hoga. Isliye k ko odd rakhte hain (5, 7, 11) taaki vote tie na ho, aur k ko cross-validation se choose karte hain. Aur ek practical baat: KNN mein scaling bahut zaroori hai, kyunki ye distance par chalta hai — bina scaling ke, sabse bada range wala feature hi sab decide karega.
Distance metrics and the curse of dimensionality
"Nearest" requires a distance measure. Euclidean distance — the straight-line distance given by the Pythagorean formula — is the default. Manhattan distance, which sums absolute differences along each axis, is sometimes better for high-dimensional data. For categorical features, Hamming distance, which counts positions where two vectors differ, is appropriate.
KNN has a fundamental weakness that deserves specific attention: the curse of dimensionality. As the number of features grows, the volume of the space grows exponentially, and all points become roughly equidistant from each other. With 100 features, the nearest neighbour and the farthest neighbour are nearly the same distance away, so "nearest" stops carrying information. KNN therefore works well with a handful of informative features and badly with hundreds, unless dimensionality is reduced first — which is exactly what Unit 6's PCA is for.
4.4 Naive Bayes
Naive Bayes is a probabilistic classifier built on Bayes' theorem. Despite its name — the "naive" part is a genuinely wrong assumption — it works remarkably well, especially for text, and it is fast enough to train on millions of documents in seconds.
P(A | B) = P(B | A) · P(A) / P(B) P(A) — the prior: how common is A before seeing any evidence?
P(B | A) — the likelihood: how likely is this evidence if A is true?
P(A | B) — the posterior: how likely is A now that we have seen the evidence?
In classification terms, we want the probability of each class given the observed features, and we choose the class with the highest probability. The "naive" assumption is that all features are conditionally independent given the class — that knowing the value of one feature tells you nothing about any other. This is almost always false. In an email, the words "bank" and "account" are strongly correlated; seeing one makes the other more likely. Naive Bayes assumes they are unrelated.
Naive Bayes is wrong about the world and still useful. Why? Because classification only needs to know which class has the highest probability, not the exact probability values. The independence assumption distorts the numbers badly — predicted probabilities from Naive Bayes are notoriously overconfident — but it often distorts all classes in the same direction, so the ranking survives and the predicted label is still right. This is a valuable lesson: a model can be misspecified and still make good decisions.
The three common variants
| Variant | Assumes features are… | Typical use |
|---|---|---|
| Gaussian NB | Normally distributed within each class | Continuous features such as measurements |
| Multinomial NB | Counts or frequencies | Text classification with word counts or TF-IDF |
| Bernoulli NB | Binary presence/absence | Short texts where word presence matters more than frequency |
One practical detail matters enormously in text classification: Laplace smoothing. If a word never appeared in the spam training examples, its probability is zero, and because Naive Bayes multiplies probabilities together, one zero makes the entire product zero — the model would confidently declare such an email impossible. Smoothing adds a small count to every feature so no probability is ever exactly zero.
A worked example: detecting spam
Training data. Eight emails, labelled by whether they contain two words: "free" and "money".
| Contains "free" | Contains "money" | Class | |
|---|---|---|---|
| 1 | Yes | Yes | Spam |
| 2 | Yes | No | Spam |
| 3 | No | Yes | Spam |
| 4 | Yes | Yes | Spam |
| 5 | No | No | Not spam |
| 6 | Yes | No | Not spam |
| 7 | No | No | Not spam |
| 8 | No | Yes | Not spam |
Step 1 — priors. 4 of 8 emails are spam, so P(Spam) = 0.5 and P(Not spam) = 0.5.
Step 2 — likelihoods. Among the 4 spam emails: "free" appears in emails
1, 2, 4 → P(free | Spam) = 3/4 = 0.75. "money" appears in emails 1, 3, 4 →
P(money | Spam) = 3/4 = 0.75.
Among the 4 non-spam emails: "free" appears in email 6 → P(free | Not spam) = 1/4 = 0.25.
"money" appears in email 8 → P(money | Not spam) = 1/4 = 0.25.
Step 3 — classify a new email containing both words. By the independence assumption:
Score(Spam) = P(Spam) × P(free|Spam) × P(money|Spam) = 0.5 × 0.75 × 0.75 =
0.28125
Score(Not spam) = 0.5 × 0.25 × 0.25 = 0.03125
Step 4 — decision. Spam scores roughly nine times higher, so the email is classified as spam. Normalising the scores into probabilities: 0.28125 / (0.28125 + 0.03125) = 0.90. The model is 90% confident it is spam.
Why we work in log space. With 10,000 words, multiplying 10,000 probabilities each below 1 produces a number so small that computers represent it as zero — a problem called underflow. Adding logs instead of multiplying probabilities avoids this entirely, and turns the product into a sum. Since log is monotonic, the class with the highest log-score is still the class with the highest score.
4.5 Decision trees
A decision tree classifies by asking a sequence of yes/no questions about the features. Is income above ₹5 lakh? If yes, is the account older than two years? If no, has the customer contacted support in the last month? Each answer leads down a branch until a leaf is reached, and the leaf holds the prediction.
Decision trees are the most interpretable models in common use. You can print the tree and show it to a domain expert, trace any prediction by hand, and explain a decision to a customer or a regulator. That interpretability is why they remain popular in regulated industries even where more accurate models exist.
A decision tree is the game "Twenty Questions". Each question splits the remaining possibilities, and a good player asks questions that divide the space as evenly as possible — "is it alive?" rather than "is it a giraffe?" A good tree does the same: it picks the feature and threshold that most cleanly separates the classes at each step, so that after as few questions as possible the answer is determined.
How a tree chooses its splits: entropy and information gain
At each node, the tree considers every possible feature and every possible threshold, and picks the split that best separates the classes. "Best" is measured by information gain — how much uncertainty about the label is removed by knowing the answer to the question.
Uncertainty is measured by entropy. A node containing a 50/50 mix of two classes has maximum uncertainty; a node containing only one class has zero uncertainty and needs no further splitting.
A pure node (one class only) gives H = 0. A 50/50 node gives H = 1 bit. This is the same entropy as in information theory, measured in bits.
|Sᵥ| / |S| — the fraction of examples going to child v
Setup. Ten customers. We want to predict churn from two binary features: "Contract = Monthly?" and "Support calls > 3?".
| Customer | Monthly contract | Support calls > 3 | Churned |
|---|---|---|---|
| 1 | Yes | Yes | Yes |
| 2 | Yes | Yes | Yes |
| 3 | Yes | No | Yes |
| 4 | Yes | No | No |
| 5 | Yes | No | No |
| 6 | No | Yes | No |
| 7 | No | No | No |
| 8 | No | No | No |
| 9 | No | No | No |
| 10 | No | No | No |
Step 1 — parent entropy. Overall, 3 of 10 churned and 7 did not. H(parent) = −(0.3 log₂ 0.3) − (0.7 log₂ 0.7) = −(0.3 × −1.737) − (0.7 × −0.515) = 0.521 + 0.360 = 0.881 bits.
Step 2 — split on "Monthly contract".
Left branch — Monthly = Yes (customers 1–5): 3 churned, 2 not. H = −(0.6 log₂ 0.6) − (0.4 log₂ 0.4) = 0.442 + 0.529 = 0.971.
Right branch — Monthly = No (customers 6–10): 0 churned, 5 not. This node is pure, so H = 0.
Weighted child entropy = (5/10)(0.971) + (5/10)(0) = 0.4855.
Information gain = 0.881 − 0.4855 = 0.396 bits.
Step 3 — split on "Support calls > 3".
Left branch — calls > 3 (customers 1, 2, 6): 2 churned, 1 not. H = −(0.667 log₂ 0.667) − (0.333 log₂ 0.333) = 0.390 + 0.528 = 0.918.
Right branch — calls ≤ 3 (customers 3, 4, 5, 7, 8, 9, 10): 1 churned, 6 not. H = −(0.143 log₂ 0.143) − (0.857 log₂ 0.857) = 0.401 + 0.190 = 0.591.
Weighted child entropy = (3/10)(0.918) + (7/10)(0.591) = 0.2754 + 0.4137 = 0.6891.
Information gain = 0.881 − 0.6891 = 0.192 bits.
Step 4 — choose. "Monthly contract" gives a gain of 0.396 against 0.192, so it is chosen as the root split. It perfectly separates one side — every customer on a monthly contract who is not in the data, and every customer on an annual contract did not churn. The tree then asks only about the monthly-contract customers, and would continue splitting on support calls within that branch.
What this tells you. Contract type is the dominant signal here. In real churn data this is common: commitment length is usually the strongest predictor of whether a customer leaves.
Gini impurity: the alternative criterion
Scikit-learn defaults to Gini impurity rather than entropy, because it is slightly faster to compute and gives nearly identical trees. It measures the probability of misclassifying a randomly chosen element if it were labelled according to the node's class distribution.
Overfitting in decision trees, and how to prevent it
Left unconstrained, a decision tree will keep splitting until every leaf is pure — meaning it has memorised every training example, including its noise. A tree that grows to depth 40 on 1,000 examples is not a model, it is a lookup table. Several mechanisms control this.
| Technique | What it does | Effect |
|---|---|---|
max_depth | Limits how many questions deep the tree may go | The single most effective control |
min_samples_split | Refuses to split a node with fewer than this many samples | Prevents splits based on tiny samples |
min_samples_leaf | Every leaf must contain at least this many samples | Ensures predictions rest on evidence |
max_features | Considers only a random subset of features per split | Decorrelates trees; the basis of Random Forest |
| Pruning | Grow fully, then cut back branches that do not help on validation data | Post-hoc complexity control |
Decision trees are unstable in a way that surprises people. Change a handful of training examples and an entirely different tree can emerge, because a different feature may win the first split and everything downstream changes. This sensitivity is not a bug — it is a fundamental property of greedy recursive partitioning. It is also exactly why ensembles of trees work so well: averaging many unstable trees cancels their individual errors. That is Unit 5.
4.6 Support Vector Machines
A support vector machine finds the boundary that separates the classes with the widest possible margin — the largest buffer zone between the boundary and the nearest points of either class.
The intuition is that many boundaries might separate the training data perfectly, but some leave almost no room for error. A boundary that passes safely down the middle of the gap is more likely to generalise, because a new point near the boundary is less likely to land on the wrong side.
Most training points are irrelevant to an SVM. The boundary is determined entirely by the support vectors — the handful of points closest to it. Delete any other point and the boundary does not move. This is why SVMs are described as sparse: the solution depends on a small subset of the data. It also explains their robustness, since a distant outlier cannot shift the boundary the way it would in logistic regression.
The mathematics, lightly
The boundary is a hyperplane defined by w·x + b = 0. The margin — the distance from the boundary to the nearest points — equals 2/‖w‖. Maximising the margin therefore means minimising ‖w‖, which gives the optimisation problem:
Real data is rarely perfectly separable, so a soft margin version allows some points to violate the constraint, with a penalty C for each violation. C is a hyperparameter and it is the most important one to tune: a large C punishes violations harshly, producing a narrow margin that fits training data closely and risks overfitting; a small C tolerates violations, giving a wider margin and a more regularised model.
The kernel trick
When classes cannot be separated by a straight line, the SVM can map the data into a higher-dimensional space where they can be. A dataset of points on a circle, where the inner points are one class and the outer ring another, cannot be split by a line in two dimensions — but lift it into three dimensions by adding x₁² + x₂² as a third coordinate, and suddenly a flat plane separates them perfectly.
Doing this explicitly is expensive and sometimes infinite-dimensional. The kernel trick avoids it entirely: it turns out the SVM optimisation only ever needs the dot products of data points, never the points themselves. A kernel function computes that dot product in the higher-dimensional space directly from the original coordinates, so the computation stays cheap even when the space is enormous.
Kernel trick ko samajhne ka best tarika: socho zameen par do rang ke dots hain jo aapas mein mix ho rakhe hain, aur aap ek seedhi lakir se unko alag nahi kar sakte. Ab agar aap zameen ko utha lein — jaise ek kapde ko beech se upar khinchein — toh woh mixed dots upar neeche alag ho jaate hain, aur ab ek flat plane se dono groups alag ho jaate hain. Yahi kernel karta hai: data ko higher dimension mein le jaata hai jahan separation aasan ho jaati hai. Aur kamaal ki baat — kernel trick actually us higher dimension mein jaata hi nahi. Woh sirf dot product calculate karta hai jaise woh wahan ho. Isliye infinite dimensions bhi handle ho jaate hain, bina kisi extra cost ke.
Common kernels are the linear kernel, appropriate when data is already separable and features are numerous, as in text; the RBF (radial basis function) kernel, which maps into infinite dimensions and is a strong general-purpose default; and the polynomial kernel, useful when feature interactions matter. Kernel choice is a hyperparameter, and linear is always worth trying first because it is fast and interpretable.
4.7 Choosing between classifiers
| Algorithm | Strengths | Weaknesses | Best for |
|---|---|---|---|
| Logistic regression | Fast, interpretable, gives probabilities, hard to overfit with regularisation | Only linear boundaries unless features are engineered | A baseline for anything; medical and credit scoring where explanation matters |
| KNN | No training, naturally handles complex boundaries, easy to explain | Slow at prediction, needs scaling, fails in high dimensions | Small datasets with few features and non-linear structure |
| Naive Bayes | Extremely fast, works with tiny data, handles many features | Independence assumption is false; overconfident probabilities | Text classification, spam filtering, sentiment analysis |
| Decision tree | Highly interpretable, no scaling needed, handles mixed types | Overfits badly, unstable, usually lower accuracy alone | When you must explain decisions; as a building block for ensembles |
| SVM | Effective in high dimensions, strong with clear margins, kernel-flexible | Slow on large data, hard to tune, no natural probabilities | Text, genomics, image features — medium-sized datasets |
In practice, the algorithm matters less than you would expect. On tabular data, a well-tuned gradient-boosted tree usually wins, a well-tuned logistic regression is usually within a few percent, and the difference between the two is often smaller than the improvement from one good engineered feature. Start with logistic regression because it is fast, easy to debug and hard to overfit. Move to ensembles only when you have clear evidence you need them.
4.8 Code: five classifiers compared
from sklearn.datasets import load_breast_cancer from sklearn.model_selection import train_test_split, cross_val_score from sklearn.preprocessing import StandardScaler from sklearn.pipeline import make_pipeline from sklearn.linear_model import LogisticRegression from sklearn.neighbors import KNeighborsClassifier from sklearn.naive_bayes import GaussianNB from sklearn.tree import DecisionTreeClassifier from sklearn.svm import SVC X, y = load_breast_cancer(return_X_y=True) X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y) models = { "Logistic Regression": make_pipeline(StandardScaler(), LogisticRegression(max_iter=5000)), "KNN (k=5)": make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=5)), "Gaussian Naive Bayes": GaussianNB(), # no scaling needed "Decision Tree": DecisionTreeClassifier(max_depth=5, random_state=42), "SVM (RBF)": make_pipeline(StandardScaler(), SVC(kernel="rbf", C=1.0, probability=True)), } for name, model in models.items(): # 5-fold cross-validation on training data — more reliable than one split cv = cross_val_score(model, X_tr, y_tr, cv=5, scoring="f1") model.fit(X_tr, y_tr) test_f1 = model.score(X_te, y_te) print(f"{name:24s} CV F1 = {cv.mean():.3f} ± {cv.std():.3f}") # Tuning KNN's k — the elbows tell you where it stops improving for k in [1, 3, 5, 9, 15, 25]: score = cross_val_score(make_pipeline(StandardScaler(), KNeighborsClassifier(k)), X_tr, y_tr, cv=5).mean() print(f"k={k:3d} accuracy={score:.3f}") # Inspecting a decision tree's rules from sklearn.tree import export_text tree = DecisionTreeClassifier(max_depth=3, random_state=42).fit(X_tr, y_tr) feat = load_breast_cancer().feature_names print(export_text(tree, feature_names=list(feat)))
4.9 Practice questions
max_depth to a small value such as 3–7; set min_samples_leaf to a
meaningful number such as 20 so no prediction rests on a handful of examples; and prune the
tree after growing it fully, cutting branches that do not improve validation performance.
The deeper fix, however, is to replace the single tree with a Random Forest, which averages
many trees and is far more accurate than any pruned tree.Model Evaluation & Ensemble Methods
5.1 Why accuracy is usually the wrong metric
Accuracy is the fraction of predictions that were correct. It is intuitive, it is the first metric anyone reaches for, and it is misleading often enough that experienced practitioners treat it with suspicion.
The problem appears when classes are imbalanced. Consider a test for a rare disease affecting 1 in 1,000 people. A model that predicts "healthy" for every single patient achieves 99.9% accuracy. It is also completely useless — it detects zero cases. Accuracy hid the failure entirely because the class we care about was 0.1% of the data.
The question is never "how often is this model right?" but "how often is it right in the way that matters?" Different mistakes have different costs. Failing to flag a fraudulent transaction costs money; mistakenly blocking a legitimate one annoys a customer. Missing a cancer diagnosis can be fatal; a false alarm leads to a follow-up test. Metrics are how you encode those costs. Choosing the metric is a business decision that happens to be expressed in mathematics.
5.2 The confusion matrix
Every binary classification result has exactly four possible outcomes, and laying them out in a 2×2 grid is the foundation of everything in this unit.
| Actual: Positive | Actual: Negative | |
|---|---|---|
| Predicted: Positive | True Positive (TP) — correctly flagged. The fraud we caught. | False Positive (FP) — false alarm. A legitimate customer blocked. |
| Predicted: Negative | False Negative (FN) — missed case. The fraud we let through. | True Negative (TN) — correctly ignored. A normal transaction. |
The convention worth remembering is that "positive" means the thing you are trying to detect, which is not always the desirable thing — in disease screening, positive means diseased. Every metric below is built from these four counts.
5.3 Precision, recall, and the trade-off between them
Precision aur recall ka fark yaad rakhne ka easy tarika: Precision = "jo maine pakda, woh kitna sahi tha?" — matlab aapki bharose ki baat hai. Agar aapne 100 logon ko chor bola aur unme se sirf 60 hi actually chor the, toh precision 60%. Recall = "jo chor the, unme se kitne pakde?" — matlab aapki poori coverage ki baat hai. Agar shahar mein 200 chor the aur aapne 60 pakde, toh recall 30%. Dono ko saath dekhna padta hai, kyunki ek badha ke doosra ghata sakte ho — agar aap sabko chor bol dein toh recall 100% ho jayega lekin precision girega. Isliye F1 use karte hain.
These two metrics pull against each other, and the reason is the decision threshold. A classifier outputs a probability; you choose where to cut. Lower the threshold and you flag more things as positive — you catch more real cases (recall up) but also generate more false alarms (precision down). Raise it and the reverse happens. There is no threshold that optimises both, so the choice depends on which error costs more.
5.4 F1 score and its variants
F1 combines precision and recall into a single number using their harmonic mean, which unlike the arithmetic mean punishes imbalance — a model with precision 1.0 and recall 0.0 has an arithmetic mean of 0.5 but an F1 of 0.
The more general F-beta score lets you weight one metric more heavily. F₂ weights recall twice as much as precision, appropriate when missing a positive case is worse than a false alarm — cancer screening. F₀.₅ weights precision twice as much, appropriate when false alarms are worse — spam filtering, where marking a genuine email as spam is a serious failure.
5.5 ROC curves and AUC
Rather than picking one threshold, the ROC curve shows performance across all thresholds at once. It plots the True Positive Rate (recall) against the False Positive Rate (FP / (FP + TN)) as the threshold sweeps from 1 down to 0.
AUC — area under this curve — compresses the whole curve into one number. It has a beautifully intuitive interpretation: AUC is the probability that the model ranks a randomly chosen positive example above a randomly chosen negative example. An AUC of 0.85 means that 85% of the time, a genuine positive gets a higher score than a genuine negative.
AUC is threshold-independent, which is both its strength and its trap. It tells you how well the model ranks, but not whether the specific threshold you deploy is any good. A model with AUC 0.95 can still perform terribly at the 0.5 threshold if the classes are imbalanced and the optimal threshold is 0.92. Always report AUC alongside the precision and recall at your chosen operating threshold.
Precision–Recall curves and when to prefer them
On heavily imbalanced data, ROC curves are deceptively optimistic, because the False Positive Rate denominator (FP + TN) is dominated by the huge negative class — a large number of false positives barely moves the FPR. The precision–recall curve does not have this problem, because precision has FP in its denominator directly. For rare-event problems such as fraud or disease detection where positives are under 5% of the data, report the PR curve and Average Precision instead.
5.6 A complete worked evaluation
Setup. A model classifies 1,000 transactions. In reality, 50 are fraudulent and 950 are legitimate. The confusion matrix is:
| Actual fraud | Actual legitimate | Total | |
|---|---|---|---|
| Predicted fraud | 40 (TP) | 60 (FP) | 100 |
| Predicted legitimate | 10 (FN) | 890 (TN) | 900 |
| Total | 50 | 950 | 1,000 |
Accuracy = (40 + 890) / 1,000 = 930 / 1,000 = 93%. This sounds impressive.
But consider the baseline. A model that predicts "legitimate" for everything: accuracy = 950 / 1,000 = 95%. The baseline is more accurate than our model while catching zero fraud. This is why accuracy is the wrong metric here.
Precision = 40 / (40 + 60) = 40 / 100 = 40%. Of the transactions we flagged, only 40% were actually fraudulent. 60 innocent customers had their cards blocked.
Recall = 40 / (40 + 10) = 40 / 50 = 80%. We caught 80% of the fraud. Ten fraudulent transactions went through.
Specificity = 890 / (890 + 60) = 890 / 950 = 93.7%. We correctly left 93.7% of legitimate transactions alone.
F1 = 2 × (0.40 × 0.80) / (0.40 + 0.80) = 0.64 / 1.20 = 0.533.
Interpretation. Whether this model is acceptable depends entirely on the business. If blocking a legitimate card costs ₹200 in support time and a missed fraud costs ₹10,000, the expected cost is: 60 × ₹200 + 10 × ₹10,000 = ₹12,000 + ₹1,00,000 = ₹1,12,000. A stricter threshold catching 45 fraud cases with 120 false alarms would cost 120 × ₹200 + 5 × ₹10,000 = ₹24,000 + ₹50,000 = ₹74,000 — better, despite lower precision. The metric did not decide this; the cost structure did.
5.7 Regression metrics, briefly
Regression was covered in Unit 3, but for completeness the metrics are MAE, MSE, RMSE and R². The choice follows the same logic as classification: RMSE if large errors are disproportionately bad, MAE if errors scale linearly, and R² for comparing models on the same dataset. There is no precision–recall analogue in regression because there are no discrete classes.
5.8 Ensemble methods: the general idea
An ensemble combines several models to produce a better prediction than any individual model. The principle is the wisdom of crowds — many imperfect opinions, aggregated sensibly, can beat one expert.
For this to work, two conditions must hold. The individual models must be better than random, or the crowd is simply pooling ignorance. And they must make different errors — if every model fails on the same examples, averaging changes nothing. Diversity is the entire game in ensembling, and every technique below is a different way of manufacturing diversity.
Ask one person to guess the number of sweets in a jar and they will be off by a lot. Ask a hundred people and average their guesses, and the answer is usually remarkably close to correct. Nobody became smarter; the errors cancelled. But this only works if people guess independently. If everyone copies the first person's answer, the average is exactly as wrong as that one guess — which is why ensembling methods deliberately inject randomness to force their models apart.
Ensembles fall into two families. Bagging trains many models in parallel on different subsets of data and averages them, primarily to reduce variance — this is Random Forest. Boosting trains models sequentially, each one focusing on the mistakes of the previous, primarily to reduce bias — this is AdaBoost, Gradient Boosting and XGBoost.
5.9 Bagging and Random Forest
Bootstrap aggregating, or bagging, trains many copies of the same model on different random samples of the training data, drawn with replacement, and combines their predictions by majority vote for classification or averaging for regression.
Bagging works because a single decision tree is unstable — recall from Unit 4 that small data changes produce completely different trees. That instability, which is a weakness in a single tree, becomes a strength in an ensemble: the trees genuinely make different errors, so averaging cancels them.
Random Forest adds one crucial refinement. If one feature is very strong, every tree will split on it at the root and the trees will be highly correlated — limiting the benefit of averaging. So at each split, a Random Forest considers only a random subset of features (by default √m for classification). This forces trees to use different features and decorrelates them, which is what makes the forest substantially better than plain bagged trees.
Random Forest is the most reliable off-the-shelf classifier for tabular data. It requires almost no tuning, is robust to outliers and unscaled features, handles missing values and mixed types, rarely overfits badly, and provides feature importances for free. If you need a strong baseline quickly and have no reason to prefer anything else, use it. Its main drawbacks are size — hundreds of deep trees consume real memory — and reduced interpretability compared with a single tree.
Random Forest also gives an out-of-bag error estimate. Because each tree is trained on a bootstrap sample, roughly 37% of rows are excluded from each tree. Those rows can be used to evaluate that tree, and averaging across trees yields a validation estimate without setting aside a separate validation set — a genuinely elegant property of the method.
5.10 Boosting
Boosting takes the opposite approach to bagging. Instead of independent models combined by voting, boosting builds models sequentially, where each new model is trained to correct the errors of the ensemble so far.
AdaBoost
AdaBoost maintains a weight for every training example, initially equal. It trains a weak learner — usually a shallow tree called a stump — and then increases the weights of the examples that were misclassified. The next learner therefore focuses on the hard cases. Each learner gets a say in the final vote proportional to its accuracy.
Boosting ko coaching class ki tarah samjho. Pehle teacher ek simple test leta hai. Jo students galat karte hain, teacher unpe zyada dhyan deta hai — agla lesson unhi ke weak topics pe focus karta hai. Phir dobara test, phir wahi process. Aakhir mein class ka result individual students se bahut better hota hai, kyunki har round mein galtiyon ko theek kiya gaya. Isliye boosting sequentially chalta hai — ek ke baad ek, aur har naya model pichle ki galti sudhaarta hai. Bagging isse ulta hai: sab models saath saath, independently, aur vote se decision.
Gradient Boosting
Gradient Boosting generalises the idea into a clean framework: each new model is fitted to the residual errors of the current ensemble — the difference between the true values and current predictions. This is literally gradient descent in function space, with each tree being one step.
For regression, the procedure is: fit a tree to the data; compute residuals; fit the next tree to those residuals; add it to the ensemble with a small learning rate; repeat. Because each tree corrects what is left over, the ensemble's training error decreases steadily with every iteration.
Two hyperparameters dominate. The learning rate (often called shrinkage, typically 0.01–0.1) controls how much each tree contributes — smaller values need more trees but generalise better. n_estimators is the number of trees; too many causes overfitting, since the model eventually begins fitting noise in the residuals. These two trade off against each other and must be tuned together.
XGBoost, LightGBM and CatBoost
These are highly optimised implementations of gradient boosting that dominate structured-data competitions and much of industry. XGBoost adds regularisation directly into the objective to prevent overfitting, uses second-order (Newton) approximations for faster convergence, and handles sparse data efficiently. LightGBM grows trees leaf-wise rather than level-wise, making it dramatically faster on large datasets. CatBoost handles categorical features natively without manual encoding. All three routinely outperform Random Forest on tabular data, at the cost of more careful tuning.
| Method | Training | Primarily reduces | Key idea |
|---|---|---|---|
| Bagging | Parallel | Variance | Average many models on bootstrap samples |
| Random Forest | Parallel | Variance | Bagging + random feature subsets to decorrelate |
| AdaBoost | Sequential | Bias | Reweight misclassified examples |
| Gradient Boosting | Sequential | Bias | Fit each new model to the residuals |
| XGBoost / LightGBM | Sequential | Bias | Gradient boosting with regularisation and engineering optimisations |
| Stacking | Both | Both | Learn how to combine model outputs |
5.11 Stacking
Stacking, or stacked generalisation, combines models of different kinds — a logistic regression, a Random Forest, an SVM — by training a meta-model on their predictions. Instead of a fixed rule like majority vote, the meta-model learns how much to trust each base model and in what circumstances.
The mechanics require care. Base models are trained on the training data and their predictions on the validation folds become the meta-model's input features. If you train the meta-model on predictions the base models made on data they were trained on, those predictions are optimistically biased and the meta-model learns the wrong weights. Cross-validated predictions are therefore essential — this is the one place where leakage is easy to introduce even for experienced practitioners.
Ensembles are not free accuracy. They cost training time, memory, inference latency and — most importantly — interpretability. Before reaching for a stacked ensemble of five models, ask whether the 1–2% accuracy gain justifies losing the ability to explain a decision. In regulated domains such as credit and healthcare, the answer is frequently no. The correct approach is to establish a simple, well-understood baseline first, then add complexity only when you can measure a real benefit.
5.12 Code: evaluation metrics and ensembles
from sklearn.datasets import make_classification from sklearn.model_selection import train_test_split, cross_val_predict from sklearn.ensemble import (RandomForestClassifier, GradientBoostingClassifier, AdaBoostClassifier) from sklearn.tree import DecisionTreeClassifier from sklearn.linear_model import LogisticRegression from sklearn.metrics import (confusion_matrix, classification_report, roc_auc_score, precision_recall_curve, average_precision_score) import numpy as np # Imbalanced dataset: 5% positives — the realistic hard case X, y = make_classification(n_samples=4000, n_features=20, weights=[0.95, 0.05], random_state=42) X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, stratify=y, random_state=42) rf = RandomForestClassifier(n_estimators=300, class_weight="balanced", random_state=42).fit(X_tr, y_tr) probs = rf.predict_proba(X_te)[:, 1] preds = (probs >= 0.5).astype(int) print(confusion_matrix(y_te, preds)) # the four counts print(classification_report(y_te, preds, digits=3)) # precision/recall/F1 print(f"AUC: {roc_auc_score(y_te, probs):.3f}") print(f"Average Precision: {average_precision_score(y_te, probs):.3f}") # Find the threshold that maximises F1 on validation-style predictions prec, rec, thresholds = precision_recall_curve(y_te, probs) f1 = 2 * prec * rec / (prec + rec + 1e-9) best = np.argmax(f1[:-1]) print(f"Best threshold = {thresholds[best]:.3f}, F1 = {f1[best]:.3f}") # Often far from 0.5 — always tune the threshold on imbalanced data # Compare single tree vs ensembles for name, m in { "Decision Tree": DecisionTreeClassifier(max_depth=5, random_state=42), "Random Forest": RandomForestClassifier(n_estimators=300, random_state=42), "AdaBoost": AdaBoostClassifier(n_estimators=200, random_state=42), "Gradient Boosting": GradientBoostingClassifier(n_estimators=200, learning_rate=0.05, random_state=42), }.items(): m.fit(X_tr, y_tr) print(f"{name:20s} AUC = {roc_auc_score(y_te, m.predict_proba(X_te)[:, 1]):.3f}") # Feature importance from the forest for i in np.argsort(rf.feature_importances_)[::-1][:5]: print(f"Feature {i}: {rf.feature_importances_[i]:.4f}")
5.13 Practice questions
Unsupervised Learning
6.1 Learning without answers
In supervised learning you always know the right answer for the training data, which is what makes the problem tractable — you can measure error and improve. Remove the labels and that entire apparatus disappears. There is no error to minimise, because there is no correct output to compare against.
This makes unsupervised learning fundamentally different in character. The algorithms do not find "the answer" — they find structure, and whether that structure is meaningful is a judgement only you can make. A clustering algorithm will always return clusters, even on pure random noise. It is a mirror: whatever groupings exist, it will show you, including the ones that are artefacts of your feature choices.
Supervised learning has a measurable objective, so success is objective. Unsupervised learning does not, so success is interpretive. This is why clustering is used for exploration — generating hypotheses, segmenting customers, finding anomalies — rather than for automated decisions. Treat the output as a question well asked, not an answer delivered.
Unsupervised learning has two broad families. Clustering groups similar examples together. Dimensionality reduction compresses many features into fewer while retaining as much structure as possible.
6.2 K-Means clustering
K-Means partitions data into k groups, where each point belongs to the group whose centre is nearest. The number of groups k is chosen by you in advance. The algorithm proceeds by alternating two steps until the centres stop moving.
Initialisation. Place k centroids — one per cluster — at random positions in the feature space, or by choosing k random data points as the initial centres.
Assignment step. Assign each data point to the nearest centroid, using Euclidean distance.
Update step. Recompute each centroid as the mean of all points assigned to it.
Repeat the assignment and update steps until assignments stop changing, or a maximum number of iterations is reached. The algorithm is guaranteed to converge — each step reduces the total within-cluster distance — though it may converge to a local optimum rather than the global best, which is why scikit-learn runs it several times with different initialisations and keeps the best.
The objective K-Means optimises
μⱼ — the centroid (mean) of cluster j
This is the within-cluster sum of squares, often called inertia. It measures how tight the clusters are.
6.3 Worked example: two iterations of K-Means by hand
Data. Six points in two dimensions, in a clear two-group arrangement: A(1,1), B(2,1), C(1,2), D(8,8), E(9,8), F(8,9). We choose k = 2.
Initialisation. Take the two initial centroids as the first two points: μ₁ = A = (1,1) and μ₂ = D = (8,8).
Iteration 1 — assignment. Compute squared Euclidean distance from each point to both centroids.
| Point | Dist² to μ₁ (1,1) | Dist² to μ₂ (8,8) | Assigned to |
|---|---|---|---|
| A (1,1) | 0 | 98 | Cluster 1 |
| B (2,1) | 1 | 85 | Cluster 1 |
| C (1,2) | 1 | 85 | Cluster 1 |
| D (8,8) | 98 | 0 | Cluster 2 |
| E (9,8) | 113 | 1 | Cluster 2 |
| F (8,9) | 113 | 1 | Cluster 2 |
Iteration 1 — update. Recompute the centroids as the means of their assigned points.
μ₁ = ((1+2+1)/3, (1+1+2)/3) = (4/3, 4/3) = (1.333, 1.333)
μ₂ = ((8+9+8)/3, (8+8+9)/3) = (25/3, 25/3) = (8.333, 8.333)
Inertia after iteration 1: distances from each point to its assigned centroid are (A: 0.222), (B: 0.556), (C: 0.556), (D: 0.222), (E: 0.556), (F: 0.556). Sum = 2.667.
Iteration 2 — reassignment. Check whether any point now prefers the other centroid.
Point B (2,1): distance² to μ₁ = (2−1.333)² + (1−1.333)² = 0.444 + 0.111 = 0.555.
Distance² to μ₂ = (2−8.333)² + (1−8.333)² = 40.11 + 53.78 = 93.89. Still cluster 1.
Point C (1,2): distance² to μ₁ = 0.111 + 0.444 = 0.555. To μ₂ = 53.78 + 40.11 = 93.89.
Still cluster 1.
Checking all six points, none changes its assignment. The algorithm has converged after two iterations.
Final result. Cluster 1 = {A, B, C} with centroid (1.33, 1.33). Cluster 2 = {D, E, F} with centroid (8.33, 8.33). This matches the obvious visual grouping — the algorithm found two tight, well-separated groups.
What to notice. The final centroids are not data points; they are averages, and they sit in the middle of their groups. Also notice that the initialisation mattered — had we started with two centroids both near the left group, the algorithm might have converged to a much worse solution, splitting the left group in two and lumping the right group with half of it. This is why K-Means is run multiple times with different initialisations and the run with the lowest inertia is retained.
6.4 Choosing k: the elbow method and silhouette score
K-Means requires you to specify k, and it will happily produce that many clusters whether or not they exist. Two techniques help.
The elbow method runs K-Means for a range of k values and plots the inertia against k. Inertia always decreases as k increases — with k equal to the number of points, inertia is zero — so the goal is not to minimise it but to find the point where the improvement flattens sharply. That bend, shaped like an elbow, suggests the natural number of clusters.
The silhouette score is more rigorous. For each point it compares the average distance to its own cluster (cohesion, a) with the average distance to the nearest other cluster (separation, b):
Do not assume K-Means will find the clusters you expect. It has three strong assumptions built in: clusters are roughly spherical, of comparable size, and separated by distance. Feed it two concentric rings and it cannot separate them, because the inner ring's centre is the same as the outer ring's centre. Feed it clusters of very different sizes and it will split the large one to balance. When your data does not meet these assumptions, use DBSCAN or hierarchical clustering instead.
6.5 Hierarchical clustering
Rather than producing a single flat partition, hierarchical clustering builds a complete tree of nested groupings, called a dendrogram. You can then cut the tree at any level to obtain any number of clusters, and the tree itself shows how clusters relate to one another.
The agglomerative approach is the common one and works bottom-up. Begin with every point as its own cluster. Repeatedly merge the two closest clusters until only one remains. The order of merges is recorded, and that record is the dendrogram. The divisive approach works top-down, splitting the whole dataset repeatedly.
The crucial design decision is how to measure the distance between two clusters, as opposed to two points. This is called the linkage criterion, and it changes the result substantially.
| Linkage | Distance between clusters | Character |
|---|---|---|
| Single | Closest pair of points across the two clusters | Can follow long chains; produces elongated clusters; prone to chaining |
| Complete | Farthest pair of points | Produces compact, evenly-sized clusters; sensitive to outliers |
| Average | Average distance over all pairs | Balanced compromise; commonly used |
| Ward | Merge that minimises the increase in within-cluster variance | Produces the most balanced clusters; often the best default |
The main advantage of hierarchical clustering is that the dendrogram shows structure at every granularity simultaneously. In customer segmentation this is genuinely valuable: rather than committing to five segments, you can see that the data supports a two-way split into "high-value" and "low-value", each of which subdivides further — and you can present that hierarchy to the business, which can then choose the level of detail it wants. The cost is computational: naive implementations are O(n³), so it is impractical beyond a few thousand points.
6.6 DBSCAN — density-based clustering
DBSCAN defines clusters as dense regions separated by sparse regions. It requires no specification of k, and it can find clusters of arbitrary shape.
Two parameters define the method. eps (ε) is the radius of a neighbourhood around each point. min_samples is how many neighbours a point must have within that radius to be considered a core point. Three kinds of points emerge: core points, which have enough neighbours; border points, which are within eps of a core point but not core themselves; and noise points, which are neither and are labelled as outliers.
DBSCAN ka idea simple hai: bheed dekho, deewar nahi. Jahan points paas paas hain woh ek cluster, jahan khaali space hai woh cluster ki boundary. Iska sabse bada fayda — isko bataana nahi padta ki kitne clusters chahiye, ye khud decide karta hai, aur moons ya rings jaise ajeeb shapes bhi pakad leta hai jinme K-Means fail ho jaata hai. Doosra bada fayda — jo points kisi bheed mein nahi hain, unko ye outlier bol deta hai, useful hai anomaly detection ke liye. Nuksan: eps set karna mushkil hai, aur agar aapke clusters ki density alag alag ho — ek bahut tight, doosra thoda faila hua — toh ye dono ko theek se handle nahi kar paata.
| Algorithm | Must specify k? | Cluster shapes | Handles outliers? |
|---|---|---|---|
| K-Means | Yes | Spherical, similar size | No — every point is assigned |
| Hierarchical | No (cut later) | Depends on linkage | No |
| DBSCAN | No | Arbitrary | Yes — explicit noise label |
6.7 Principal Component Analysis
Principal Component Analysis is the most widely used dimensionality reduction technique. Its purpose is to express a dataset with many features using fewer new features, while losing as little information as possible.
The central idea is that features are usually correlated, and correlated features carry redundant information. In a dataset containing height in centimetres and height in inches, one feature is entirely redundant. In a dataset of 50 survey questions, the responses may be driven by three underlying attitudes. PCA finds those underlying directions.
Concretely, PCA finds new axes, called principal components, with three properties. They are ordered by how much variance of the data they explain, so PC1 captures the most. Each is perpendicular to all the others, so the components are uncorrelated by construction. And they are linear combinations of the original features.
Computationally, PCA computes the covariance matrix of the standardised data and finds its eigenvectors and eigenvalues. The eigenvectors are the principal component directions; the eigenvalues say how much variance each direction accounts for. Sorting eigenvalues from largest to smallest orders the components by importance.
PCA ko socho jaise tum ek moti roti (chapati) ki taraf dekh rahe ho jo tedhi rakhi hai. Ab tum spoon se usko flatten karna chahte ho. Sabse aasan tarika kya hai? Usko uske lambai ki direction mein dekho — kyunki us direction mein woh sabse zyada faili hui hai. Agar tum usko uske chaudai ki taraf se dekho, toh sab kuch ek chhoti si line mein dabas jayega aur bahut information chali jayegi. PCA wahi karta hai: data ko sabse zyada faili hui direction mein dekhta hai (PC1), phir uske perpendicular (PC2), aur aage. Isliye PCA ko "maximum variance preserve karna" kehte hain. Aur important baat — PCA se pehle scaling karna zaroori hai, warna jo feature sabse bada range wala hai wahi PCA ko chala dega.
Explained variance and choosing how many components
Each component has an associated proportion of total variance it explains. The standard practice is to keep enough components to explain 90–95% of the variance. If the first three of fifty components explain 92%, you have compressed fifty features to three with minimal loss.
Applications fall into three groups. Visualisation: projecting onto the first two components lets you plot high-dimensional data on a screen. Noise removal: discarding the low-variance components often discards mostly noise, improving downstream models. Speed: fewer features means faster training, which matters with hundreds of thousands of features.
PCA components are linear combinations of all original features, so they are usually not interpretable. PC1 is not "size" or "income" — it is 0.3 × age + 0.7 × income − 0.2 × tenure. If you need to explain to a stakeholder which factors drive a prediction, PCA will not help, and you should use feature selection instead. This is a frequent source of frustration for beginners who expect the components to come out with meaningful names.
PCA is a linear method. It finds straight-line directions of maximum variance. Real data often lies on a curved manifold — think of a rolled-up sheet of paper, which is two-dimensional but curved in three-dimensional space. PCA cannot unroll it. This is exactly what autoencoders (Unit 7) and manifold learning methods such as t-SNE and UMAP are for.
6.8 t-SNE and UMAP
t-SNE (t-distributed Stochastic Neighbour Embedding) is designed specifically for visualisation. It maps high-dimensional data to two or three dimensions such that points that were close together remain close, and points that were far apart are pushed further apart.
Its outputs are visually striking — it reveals tight clusters and clear separations that other methods miss. But it must be used with care. The absolute distances and cluster sizes in a t-SNE plot are not meaningful; only the relative neighbourhood relationships are. Two clusters far apart in the plot are not necessarily far apart in the original space. And the perplexity parameter, roughly the number of effective neighbours, changes the picture substantially, so results must be checked across several values.
UMAP is a more recent alternative that is faster, better at preserving global structure, and more stable across runs. It has largely become the default for high-dimensional visualisation, though t-SNE remains widely used and understood.
Never use t-SNE or UMAP output as features for a downstream model, and never interpret the distance between clusters as meaningful. These methods are for looking at data, not for modelling it. If you want reduced features, use PCA or an autoencoder. A published biology paper once claimed a rare cell type formed a distinct cluster when the apparent separation was purely an artefact of the perplexity setting — a cautionary example of reading too much into a pretty picture.
6.9 Code: clustering and dimensionality reduction
from sklearn.datasets import load_iris, make_moons from sklearn.preprocessing import StandardScaler from sklearn.cluster import KMeans, AgglomerativeClustering, DBSCAN from sklearn.decomposition import PCA from sklearn.metrics import silhouette_score from sklearn.manifold import TSNE import numpy as np X, y_true = load_iris(return_X_y=True) X = StandardScaler().fit_transform(X) # ALWAYS scale before K-Means or PCA # --- Elbow method: inertia for a range of k --- for k in range(1, 9): km = KMeans(n_clusters=k, n_init=10, random_state=42).fit(X) print(f"k={k} inertia={km.inertia_:8.2f}") # Look for the bend — here it appears at k=3 # --- Silhouette score to confirm the choice --- for k in range(2, 9): labels = KMeans(n_clusters=k, n_init=10, random_state=42).fit_predict(X) print(f"k={k} silhouette={silhouette_score(X, labels):.3f}") # --- Compare algorithms on non-spherical data --- Xm, _ = make_moons(n_samples=600, noise=0.06, random_state=42) Xm = StandardScaler().fit_transform(Xm) km_labels = KMeans(n_clusters=2, n_init=10, random_state=42).fit_predict(Xm) db_labels = DBSCAN(eps=0.25, min_samples=8).fit_predict(Xm) print(f"K-Means on moons: silhouette {silhouette_score(Xm, km_labels):.3f}") # DBSCAN labels noise as -1; exclude those when scoring mask = db_labels != -1 print(f"DBSCAN on moons: silhouette {silhouette_score(Xm[mask], db_labels[mask]):.3f}") print(f"DBSCAN found {(db_labels == -1).sum()} outliers") # --- Hierarchical clustering --- hier = AgglomerativeClustering(n_clusters=3, linkage="ward") print(f"Hierarchical silhouette: {silhouette_score(X, hier.fit_predict(X)):.3f}") # --- PCA: how many components to keep? --- pca = PCA().fit(X) cumulative = np.cumsum(pca.explained_variance_ratio_) for i, v in enumerate(cumulative, 1): print(f"PC{i}: {v:.3f} cumulative variance") # Keep the smallest number of components reaching 0.95 n_keep = np.argmax(cumulative >= 0.95) + 1 X_reduced = PCA(n_components=n_keep).fit_transform(X) print(f"Reduced 4 features to {n_keep}, keeping 95% of variance") # --- t-SNE for visualisation only, never as model features --- embedding = TSNE(n_components=2, perplexity=30, random_state=42, init="pca").fit_transform(X)
6.10 Practice questions
Neural Networks & Deep Learning
Everything so far has been a human choosing the shape of the pattern. Neural networks let the model discover the shape itself. This unit walks from a single artificial neuron all the way to transformers, and does backpropagation by hand on a network small enough to fit on one page.
7.1 Why we needed something new
Look back at what we have built. Linear regression assumes the relationship is a straight line. Logistic regression assumes the boundary is a straight line. Polynomial regression works, but only if you guess the right degree. SVM with an RBF kernel is flexible, but you still hand-pick the kernel. Decision trees split on one feature at a time, so a diagonal boundary comes out as an ugly staircase.
In every case a human decided the form of the function, and the algorithm only filled in the numbers. That is fine for tabular data with 20 clean columns. It falls apart for a photograph. What is the "right feature" for recognising a cat? Nobody can write it down. Whisker length? Ear angle? A cat curled up asleep has no visible ears.
A neural network is a stack of very simple functions. Each layer transforms its input a little, and the next layer works on that transformed version. Because the layers are learned, the network invents its own features. Early layers of an image network learn edges. Middle layers learn textures and shapes. Later layers learn "wheel", "ear", "eye". No human wrote any of that down — it emerged from the training data.
7.2 The artificial neuron
One neuron does exactly three things. It multiplies each input by a weight, adds a bias, and pushes the result through a non-linear function.
Notice that z alone is just linear regression. The entire power of a neural network comes from φ, the activation function. Remove it and something surprising happens.
Stack 50 layers with no activation function and you get… a single linear layer. A linear function of a linear function is still linear. Mathematically, W3(W2(W1x)) = (W3W2W1)x = Wx. All that depth collapses into one matrix. The non-linearity is what makes depth mean something.
Think of a hospital triage desk. Each nurse (neuron) looks at a few symptoms, weighs them by importance (weights), has a personal threshold for concern (bias), and then makes a yes/no-ish judgement (activation). One nurse is not very smart. A hospital of nurses passing notes to each other in layers, where later nurses only see earlier nurses' conclusions, can diagnose things no single nurse could.
7.3 Activation functions: the practical shortlist
| Name | Formula | Output range | Use it when | Watch out for |
|---|---|---|---|---|
| Sigmoid | 1 / (1 + e−z) | (0, 1) | Final layer of binary classification | Saturates — gradients vanish for large |z|. Avoid in hidden layers. |
| Tanh | (ez − e−z) / (ez + e−z) | (−1, 1) | Hidden layers in RNNs; when zero-centred output helps | Still saturates, just symmetrically. |
| ReLU | max(0, z) | [0, ∞) | Default choice for hidden layers | Dying ReLU — a neuron stuck at negative z outputs 0 forever and never recovers. |
| Leaky ReLU | max(0.01z, z) | (−∞, ∞) | When you suspect dying ReLUs | One more hyperparameter (the slope). |
| Softmax | ezi / Σj ezj | (0, 1), sums to 1 | Final layer of multi-class classification | Only for mutually exclusive classes. Use sigmoid per-label for multi-label. |
Hidden layers: ReLU. Output layer: depends on the task. Regression → no activation (raw number). Binary classification → sigmoid (one probability). Multi-class → softmax (probabilities that sum to 1). Multi-label → sigmoid on every output independently.
Why is ReLU the default when it looks so crude? Three reasons. It is trivially cheap to compute (a comparison, not an exponential). Its gradient is exactly 1 for positive inputs, so gradients pass through many layers without shrinking. And it produces sparse activations — roughly half the neurons output zero for any given input, which acts as a mild regulariser.
7.4 From one neuron to a network
Arrange neurons in columns. Every neuron in a column receives the outputs of every neuron in the previous column. That is a fully connected or dense network, also called a multilayer perceptron (MLP).
Some vocabulary you will meet constantly. The width of a layer is its number of neurons. The depth of the network is its number of layers. "Deep learning" simply means a network with many hidden layers — there is no official threshold, but people usually mean more than two or three.
For a network with layers 1…L, written with matrices instead of individual neurons:
a(0) = x then for each layer z(l) = W(l)a(l−1) + b(l), a(l) = φ(z(l)) and finally ŷ = a(L)
That is it. A neural network is a matrix multiply, an add, and a squash — repeated. If W(l) has shape (neurons in layer l × neurons in layer l−1), the shapes always line up.
7.5 Universal approximation — and why it is less useful than it sounds
There is a famous theorem: a network with a single hidden layer, given enough neurons, can approximate any continuous function to arbitrary accuracy. Students often read this and conclude that depth is pointless.
The theorem says such a network exists. It does not say you can find it, or that "enough neurons" is a reasonable number. In practice a shallow network may need exponentially more neurons than a deep one to represent the same function. Depth buys you reuse: layer 3 can build on the edge-detectors layer 1 discovered, instead of rediscovering them separately for every output. Deep networks are not more powerful in theory, they are vastly more efficient in practice.
7.6 Loss functions for networks
Nothing new here — the same losses from Units 3 and 4 carry straight over.
| Task | Output layer | Loss |
|---|---|---|
| Regression | 1 neuron, no activation | Mean squared error (or MAE / Huber if outliers) |
| Binary classification | 1 neuron, sigmoid | Binary cross-entropy |
| Multi-class (one label) | k neurons, softmax | Categorical cross-entropy |
| Multi-label (many labels) | k neurons, sigmoid each | Binary cross-entropy, summed over labels |
7.7 Backpropagation: the heart of the whole thing
We have 35 weights in Figure 7.1 (and modern networks have billions). Gradient descent needs ∂L/∂w for every single one of them. Computing each derivative separately from scratch would be hopelessly slow. Backpropagation is the trick that gets all of them in one backward sweep, at roughly the same cost as one forward pass.
Socho ek company mein galti ho gayi. CEO ko pata chala ki profit target se 10 crore kam hai. Ab CEO neeche poochta hai: "Sales team, tumhara kitna contribution tha is galti mein? Marketing, tumhara kitna?" Sales head phir apni team se wahi sawaal poochta hai, aur aage. Har level apni local zimmedari ka hisaab lagata hai, aur blame ko neeche pass karta hai — proportion ke hisaab se, jitna kisne contribute kiya.
Backpropagation exactly yahi hai. Output pe error nikalti hai. Us error ko peeche bhejte hain, layer by layer. Har weight ko pata chal jata hai: "meri wajah se error kitni badhi?" Aur phir har weight utna hi adjust hota hai. Chain rule is blame-passing ka mathematical naam hai — bas aur kuch nahi.
The chain rule from calculus says that if L depends on a, and a depends on z, and z depends on w, then:
Two local derivatives you should simply memorise, because they appear everywhere:
Sigmoid: if a = σ(z) then ∂a/∂z = a(1 − a). Beautifully, it is written in terms of the output you already computed — no need to touch z again.
ReLU: if a = max(0, z) then ∂a/∂z = 1 when z > 0, and 0 otherwise. A gradient of exactly 1 is why ReLU networks train so well at depth.
Network. The smallest thing that still counts as deep: one input, one hidden neuron, one output neuron, sigmoid everywhere.
x → [ w₁, b₁ ] → a₁ → [ w₂, b₂ ] → a₂ = ŷ
Starting values. x = 1.0, target y = 0, w₁ = 0.5, b₁ = 0.0, w₂ = 0.8, b₂ = 0.0, learning rate η = 1.0 (deliberately large so the change is visible). Loss is L = ½(a₂ − y)².
Step 1 — forward pass.
| z₁ = w₁x + b₁ | = 0.5(1.0) + 0 = 0.5000 |
| a₁ = σ(0.5) | = 1/(1 + e−0.5) = 1/1.6065 = 0.6225 |
| z₂ = w₂a₁ + b₂ | = 0.8(0.6225) + 0 = 0.4980 |
| a₂ = σ(0.4980) | = 1/1.6078 = 0.6220 |
| L = ½(0.6220 − 0)² | = 0.1934 |
The network says 0.62 but the truth is 0. It is badly wrong. Now we fix it.
Step 2 — backward pass, output layer. We want how much the loss changes per unit change in z₂. Call it δ₂.
| ∂L/∂a₂ = a₂ − y | = 0.6220 − 0 = 0.6220 |
| ∂a₂/∂z₂ = a₂(1 − a₂) | = 0.6220 × 0.3780 = 0.2351 |
| δ₂ = 0.6220 × 0.2351 | = 0.1463 |
| ∂L/∂w₂ = δ₂ × a₁ | = 0.1463 × 0.6225 = 0.0911 |
| ∂L/∂b₂ = δ₂ × 1 | = 0.1463 |
Step 3 — backward pass, hidden layer. The hidden neuron's blame is δ₂ passed back through w₂, then multiplied by its own local sigmoid slope.
| ∂L/∂a₁ = δ₂ × w₂ | = 0.1463 × 0.8 = 0.1170 |
| ∂a₁/∂z₁ = a₁(1 − a₁) | = 0.6225 × 0.3775 = 0.2350 |
| δ₁ = 0.1170 × 0.2350 | = 0.0275 |
| ∂L/∂w₁ = δ₁ × x | = 0.0275 × 1.0 = 0.0275 |
| ∂L/∂b₁ = δ₁ | = 0.0275 |
Step 4 — update every weight. Same rule as Unit 3: w ← w − η(∂L/∂w).
| Parameter | Old | Gradient | New |
|---|---|---|---|
| w₂ | 0.8000 | 0.0911 | 0.7089 |
| b₂ | 0.0000 | 0.1463 | −0.1463 |
| w₁ | 0.5000 | 0.0275 | 0.4725 |
| b₁ | 0.0000 | 0.0275 | −0.0275 |
Step 5 — did it help? Run the forward pass again with the new weights.
| z₁ = 0.4725(1.0) − 0.0275 | = 0.4450 → a₁ = 0.6095 |
| z₂ = 0.7089(0.6095) − 0.1463 | = 0.2858 → a₂ = 0.5710 |
| L = ½(0.5710)² | = 0.1630 |
Loss fell from 0.1934 to 0.1630. One step, and the prediction moved from 0.622 towards the target of 0. Repeat this a few thousand times and the network converges. Every deep learning framework in existence is doing exactly this arithmetic, just with matrices instead of scalars and millions of parameters instead of four.
Notice one thing. The hidden-layer gradient (0.0275) is far smaller than the output-layer gradient (0.1463) — roughly 5× smaller after only one layer of travel. That shrinkage is the seed of a serious problem.
7.8 The vanishing gradient problem
Look at what happened in Worked Example 7.1. Each time the gradient crossed a sigmoid, it got multiplied by a(1 − a), which is at most 0.25 and usually much less. Cross ten sigmoid layers and the gradient is multiplied by something like 0.210 ≈ 0.0000001. The early layers receive essentially zero signal. They never learn. This is the vanishing gradient problem, and it is why neural networks barely worked before roughly 2010.
Four fixes, all still standard practice today. ReLU instead of sigmoid in hidden layers, because its gradient is exactly 1 rather than 0.25. Careful weight initialisation (He or Xavier), so signals neither explode nor die at the start. Batch normalisation, which re-centres each layer's inputs so activations stay in the healthy part of the curve. And skip connections (ResNets), which give the gradient a shortcut that bypasses layers entirely. Together these turned "3 layers maximum" into "150 layers, no problem".
The mirror image also exists. If weights are large, gradients get multiplied up instead of down and you get exploding gradients — loss suddenly becomes NaN. The standard fix is gradient clipping: if the gradient's magnitude exceeds a threshold, rescale it down before applying it.
7.9 Weight initialisation
You cannot start all weights at zero. If every neuron in a layer has identical weights, they compute identical outputs, receive identical gradients, and update identically — forever. The layer behaves like a single neuron no matter how wide it is. This is the symmetry problem, and random initialisation breaks it.
| Scheme | Draw weights from | Use with |
|---|---|---|
| Xavier / Glorot | variance = 1 / nin (or 2/(nin+nout)) | tanh, sigmoid |
| He / Kaiming | variance = 2 / nin | ReLU and its variants — the usual default |
The logic behind both: scale the random values so that the variance of a layer's output roughly matches the variance of its input. He initialisation uses 2 instead of 1 because ReLU throws away half the signal (everything negative becomes zero), so you compensate by starting with twice the variance. You will almost never set this by hand — every framework picks a sensible default — but knowing it exists explains a lot of "why won't my network train" situations.
7.10 Optimisers: smarter ways downhill
Plain gradient descent takes a step of fixed size in the steepest direction. It works, but it is naive. Three problems: it crawls through long flat valleys, it oscillates across narrow ravines, and it treats every parameter with the same learning rate even though some need big steps and some need tiny ones.
| Optimiser | The idea in one sentence | When to use |
|---|---|---|
| SGD | Step in the direction of the gradient of one mini-batch. | Baseline. Still wins on large vision models with a good schedule. |
| SGD + Momentum | Keep a running velocity, so consistent directions accelerate and oscillations cancel out. | Almost always better than plain SGD. Momentum 0.9 is the standard value. |
| RMSprop | Divide each parameter's step by the recent magnitude of its own gradients — a per-parameter learning rate. | RNNs; noisy or sparse gradients. |
| Adam | Momentum and per-parameter scaling, combined. | The default. Start here with lr = 0.001 and only change it if something is wrong. |
| AdamW | Adam with weight decay applied correctly (decoupled from the gradient). | Transformers and anything where regularisation matters. |
Plain gradient descent is walking: you stop completely, look around, take one careful step, stop again. Momentum is riding a bicycle down a hill: you build up speed in the direction you have consistently been going, so small bumps to the left and right do not throw you off course. That is why momentum both speeds up convergence and smooths out oscillation — one mechanism, two benefits.
Learning rate. If your loss becomes NaN or bounces wildly, it is too high. If loss decreases but agonisingly slowly, it is too low. Try powers of ten — 0.1, 0.01, 0.001, 0.0001 — before touching anything else. A learning-rate schedule that decays over training (step decay, cosine annealing) is nearly free performance: big steps early to explore, small steps late to settle.
7.11 Epochs, batches and iterations
Three words that get confused constantly, so let us pin them down with a number.
You have 10,000 training images and choose batch_size = 100.
Batch = the 100 images processed together in one weight update.
Iteration (step) = one weight update. 10,000 / 100 = 100 iterations per epoch.
Epoch = one complete pass over all 10,000 images.
Train for 30 epochs and the network performs 30 × 100 = 3,000 weight updates and sees each image
30 times.
Batch size trades off three things. Small batches (8–32) give noisy gradients, which is slow per epoch but the noise itself acts as a regulariser and often generalises better. Large batches (512+) use the GPU efficiently and give smooth gradients, but tend to find sharper minima that generalise slightly worse, and need a higher learning rate to compensate. 32 to 256 is the comfortable middle ground for most problems.
7.12 Regularisation for networks
A network with a million parameters and ten thousand training rows will memorise the training set perfectly and be useless on new data. Four defences, roughly in order of how often you will reach for them:
| Technique | What it does | Typical setting |
|---|---|---|
| Early stopping | Watch validation loss; stop when it stops improving and keep the best weights. | patience = 5–10 epochs. Use this always — it is free. |
| Dropout | During training, randomly switch off a fraction of neurons on every batch. At test time use all of them. | rate 0.2–0.5, after dense layers |
| Weight decay (L2) | Add a penalty on large weights to the loss — same Ridge idea as Unit 3. | 1e−4 to 1e−2 |
| Data augmentation | Create new training examples by transforming existing ones (flip, rotate, crop, colour-shift an image). | Enormously effective for images and audio |
Ek cricket team socho jahan har practice match mein coach randomly 3 players ko bench pe bhej deta hai. Result? Koi bhi player ek dusre pe poori tarah depend nahi kar sakta — "arre woh catch to Rohit hi lega" wali aadat khatam. Har player ko khud strong banna padta hai, aur team ke har combination mein kaam karna aata hai.
Dropout ka logic same hai. Neurons ek dusre pe over-depend karna band kar dete hain (isko co-adaptation kehte hain), aur har neuron independently useful feature seekhta hai. Ek important baat: dropout sirf training ke time hota hai. Testing ke time poori team khelti hai.
Not exactly regularisation, but it belongs in the same conversation. Batch norm standardises each layer's inputs to zero mean and unit variance across the mini-batch, then lets the network learn a scale and shift. Effect: training becomes far more stable, you can use higher learning rates, gradients vanish less, and you get a mild regularising side-effect from batch-to-batch noise. Standard placement is between the linear layer and the activation.
7.13 Convolutional neural networks (CNNs) — for images
Feed a 224×224 colour image to a dense layer and each neuron needs 224×224×3 = 150,528 weights. One layer of 1,000 such neurons is 150 million parameters — and it would still fail, because a dense layer treats every pixel as unrelated. Shift the cat two pixels to the right and every input has changed.
CNNs fix this with two ideas: local connectivity (a neuron looks at a small patch, not the whole image) and weight sharing (the same small filter slides across the entire image). A 3×3 filter on a colour image has 3×3×3 = 27 weights, and it detects its pattern anywhere in the picture.
The mechanics of one convolution: place the filter over a patch, multiply element-wise, sum, write the result to the output map, slide, repeat. Three settings control the geometry. Kernel size is the filter's dimensions (3×3 is the near-universal choice). Stride is how far it slides each time (1 keeps the size, 2 halves it). Padding adds a border of zeros so that output size matches input size — without it every conv layer shrinks the image and you run out of pixels.
Pooling downsamples by summarising each small region — max pooling keeps the largest value in each 2×2 block, halving both height and width. Its real value is not just speed: it makes the network slightly translation invariant. Whether the strongest edge was at pixel 5 or pixel 6, max pooling reports the same thing.
If you visualise what filters respond to, a clear progression appears. Layer 1: edges and colour blobs. Layer 2: corners, curves, simple textures. Layer 3: repeating patterns, mesh, fur. Layer 4: object parts — an eye, a wheel, a door handle. Layer 5: whole objects. Nobody designed this hierarchy. It is the structure that gradient descent found because it is the efficient way to describe natural images.
7.14 Recurrent networks (RNNs) — for sequences
A dense network has no memory. Feed it "the movie was not good" one word at a time and it has no idea that "not" came before "good". Sequences need state.
An RNN keeps a hidden state h that carries information forward through time. At each step it combines the new input with what it already remembers:
Plain RNNs have a fatal flaw, and it is the vanishing gradient problem in a new costume. Unrolling an RNN over 100 time steps is mathematically like a 100-layer deep network with tanh at every layer. The gradient from step 100 has effectively nothing left by the time it reaches step 1, so the network cannot learn long-range dependencies. It forgets the beginning of the sentence.
| Architecture | Key mechanism | Notes |
|---|---|---|
| LSTM | A separate cell state plus three learned gates — forget, input, output — that decide what to discard, what to store, and what to reveal. | Handles hundreds of steps. The reliable default when you need an RNN. |
| GRU | Same gating idea, simplified into two gates and no separate cell state. | Fewer parameters, trains faster, usually comparable accuracy. |
| Bidirectional | Run one RNN forwards and another backwards, concatenate their states. | Only valid when the full sequence is available — not for live prediction. |
Imagine reading a long novel while keeping a notebook. The forget gate is you crossing out a character who died in chapter 2. The input gate is you deciding this new plot twist is worth writing down (most sentences are not). The output gate is you choosing which notes are relevant to the question someone just asked you. The notebook is the cell state — it survives largely unchanged across many steps, which is precisely why gradients can flow back through it.
7.15 Attention and transformers
RNNs have a structural bottleneck: they must process a sequence one step at a time, which cannot be parallelised, and all information about the past has to squeeze through a single fixed-size hidden state. Attention removes both problems.
The idea: instead of relying on a summary carried forward, let every position look directly at every other position and decide for itself what is relevant. Processing the word "it" in "the trophy did not fit in the suitcase because it was too large", attention lets "it" look at all other words and assign a high weight to "trophy".
If components of q and k are independent random variables with mean 0 and variance 1, their dot product q · k has mean 0 and variance dk. For large dimensions (e.g. dk = 64), dot products grow large in magnitude, driving the softmax function into regions with vanishingly small gradients. Dividing by √dk pulls the variance back to 1, preserving healthy gradient flow during backpropagation.
headi = Attention(QWiQ, KWiK, VWiV)
Library mein ho, ek sawaal hai. Woh sawaal tumhara Query hai. Har kitaab ke cover pe title likha hai — woh Key hai. Kitaab ke andar ka content Value hai.
Tum apne sawaal ko har title se match karte ho, aur relevance score dete ho. Jo kitaabein zyada relevant hain, unse zyada padhte ho; jo kam relevant hain, unse thoda ya kuch nahi. Phir sab kuch mila ke ek answer banate ho.
Bas yahi attention hai — ek weighted average, jahan weights khud seekhe jaate hain. Aur multi-head attention ka matlab: ek hi sawaal ke liye 8 log parallel mein library scan kar rahe hain, har ek different angle se (ek grammar dekh raha hai, ek meaning, ek topic). Sabke answers combine ho jaate hain.
Because every position attends to every other position independently, the whole sequence can be processed in parallel — which is exactly what makes GPUs fast and what made training on internet-scale text feasible. That parallelism, not raw accuracy, is the reason transformers displaced RNNs.
The trade-off: attention cost grows as O(n²) with sequence length, since every position compares against every other. Doubling the context length quadruples the compute. And because there is no inherent notion of order in a set of comparisons, transformers must add positional encodings to the inputs to tell the model where each token sits.
7.16 Transfer learning — the most practical idea in this unit
Training a large network from scratch needs millions of labelled examples and a serious GPU budget. You almost never have either. Transfer learning sidesteps the problem: take a network someone else trained on a huge dataset, and reuse its learned features for your small problem.
The early layers of an image network learn edges, textures and shapes. Those features are not specific to the original task — edges are edges whether you are classifying dogs or detecting cracks in concrete. Only the last few layers are task-specific. So: keep the early layers, replace the final classifier, train on your 500 images. This routinely beats a from-scratch network trained on the same 500 images by a wide margin.
| Strategy | What you do | Use when |
|---|---|---|
| Feature extraction | Freeze all pretrained layers, train only a new head on top. | Very little data (a few hundred examples); your domain resembles the original. |
| Fine-tuning | Train the new head first, then unfreeze the top few layers and continue at a much lower learning rate. | Moderate data (thousands); your domain differs somewhat (e.g. X-rays vs photographs). |
| Full retraining | Use pretrained weights only as initialisation, train everything. | Lots of data and a domain very unlike the original. |
Unfreezing the pretrained layers at the same learning rate you used for the new head. The randomly initialised head produces large gradients at first, and those gradients will wreck carefully learned features in seconds — a phenomenon sometimes called catastrophic forgetting. Always train the head first with the base frozen, and when you do unfreeze, drop the learning rate by 10× or more.
7.17 When not to use a neural network
Deep learning is fashionable, which makes it easy to reach for by default. It is often the wrong choice.
Your data is tabular — rows and columns in a spreadsheet or database. On tabular data, XGBoost / LightGBM usually match or beat neural networks while training in seconds instead of hours, needing no scaling, handling missing values natively, and giving you feature importances for free. Also avoid deep learning when you have fewer than a few thousand rows, when explainability is a legal requirement, or when a logistic regression already meets the business target — a simpler model you can debug at 3am beats a better model you cannot.
Reach for neural networks when the input is unstructured — images, audio, video, free text — or when you can start from a pretrained model. That is where they are not merely better but in a different league.
import numpy as np # ---------- Part 1: backpropagation from scratch (no frameworks) ---------- # Learning XOR - deliberately chosen because it is NOT linearly separable, # so a single-layer model provably cannot solve it, but a hidden layer can. X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]]) y = np.array([[0], [1], [1], [0]]) # XOR targets def sigmoid(z): return 1 / (1 + np.exp(-z)) def d_sigmoid(a): return a * (1 - a) # note: in terms of the OUTPUT rng = np.random.default_rng(42) n_in, n_hidden, n_out = 2, 4, 1 # He-style initialisation: scale by sqrt(2 / fan_in) so signals neither die nor explode W1 = rng.normal(0, np.sqrt(2/n_in), (n_in, n_hidden)) b1 = np.zeros((1, n_hidden)) W2 = rng.normal(0, np.sqrt(2/n_hidden), (n_hidden, n_out)) b2 = np.zeros((1, n_out)) lr = 0.5 for epoch in range(6000): # --- forward pass --- z1 = X @ W1 + b1; a1 = sigmoid(z1) z2 = a1 @ W2 + b2; a2 = sigmoid(z2) loss = np.mean((a2 - y) ** 2) # --- backward pass: exactly the chain rule from Worked Example 7.1 --- d2 = (a2 - y) * d_sigmoid(a2) # error signal at output layer dW2 = a1.T @ d2 # gradient wrt W2 = input to layer x delta db2 = d2.sum(axis=0, keepdims=True) d1 = (d2 @ W2.T) * d_sigmoid(a1) # push blame back through W2, then local slope dW1 = X.T @ d1 db1 = d1.sum(axis=0, keepdims=True) # --- update every parameter --- W2 -= lr * dW2; b2 -= lr * db2 W1 -= lr * dW1; b1 -= lr * db1 if epoch % 1500 == 0: print(f"epoch {epoch:5d} loss {loss:.5f}") print("final predictions:", a2.ravel().round(3)) # -> approximately [0.02 0.98 0.98 0.02] : XOR solved by a 4-neuron hidden layer # ---------- Part 2: the same thing in scikit-learn ---------- from sklearn.neural_network import MLPClassifier from sklearn.preprocessing import StandardScaler from sklearn.pipeline import make_pipeline # SCALING IS MANDATORY for neural networks - unscaled features make the loss # surface a long narrow ravine that gradient descent struggles to descend. mlp = make_pipeline( StandardScaler(), MLPClassifier(hidden_layer_sizes=(64, 32), # two hidden layers activation="relu", solver="adam", alpha=1e-4, # L2 weight decay learning_rate_init=1e-3, batch_size=64, max_iter=500, early_stopping=True, # hold out 10% and watch it n_iter_no_change=10, # patience random_state=42) ) # mlp.fit(X_train, y_train); print(mlp.score(X_test, y_test)) # ---------- Part 3: a CNN sketch in Keras (for reference) ---------- # model = keras.Sequential([ # layers.Rescaling(1./255, input_shape=(32, 32, 3)), # layers.Conv2D(32, 3, padding="same", activation="relu"), # layers.BatchNormalization(), # layers.MaxPooling2D(), # 32x32 -> 16x16 # layers.Conv2D(64, 3, padding="same", activation="relu"), # layers.MaxPooling2D(), # 16x16 -> 8x8 # layers.Flatten(), # layers.Dropout(0.5), # regularise the dense head # layers.Dense(10, activation="softmax"), # ]) # model.compile(optimizer="adam", # loss="sparse_categorical_crossentropy", # metrics=["accuracy"]) # model.fit(X_train, y_train, epochs=30, validation_split=0.2, # callbacks=[keras.callbacks.EarlyStopping(patience=5, # restore_best_weights=True)])
When a network refuses to learn, work through this in order. 1. Did you scale the features? 2. Can it overfit a tiny subset — take 20 rows and check the loss goes to almost zero. If it cannot, there is a bug, not a capacity problem. 3. Is the learning rate sane? Try 10× up and 10× down. 4. Is the output activation right for the task, and does the loss match it? 5. Are labels aligned with inputs after shuffling? 6. Only once loss is falling should you worry about architecture.
7.18 Practice questions
Q1. Why can a neural network without activation functions never outperform linear regression, no matter how many layers it has?
A. Composing linear functions yields a linear function. W₃(W₂(W₁x)) = (W₃W₂W₁)x, and the product of matrices is a single matrix. So a 50-layer network with no activations is mathematically identical to one linear layer — it can only ever fit a hyperplane. The non-linearity is what allows depth to build increasingly complex functions.
Q2. Explain the vanishing gradient problem and state three ways to reduce it.
A. Backpropagation multiplies local derivatives along the chain. Sigmoid and tanh have derivatives well below 1 (sigmoid peaks at 0.25), so gradients shrink geometrically with depth — early layers receive almost no learning signal. Fixes: use ReLU in hidden layers (gradient exactly 1 for positive inputs); use He or Xavier initialisation to keep activations in a healthy range; add batch normalisation to re-centre each layer's inputs; and add skip/residual connections so gradients have a direct path backwards. Any three of those are a full answer.
Q3. Why must weights be initialised randomly rather than to zero?
A. With identical weights, every neuron in a layer computes the same output, receives the same gradient, and updates identically — forever. The layer collapses to the behaviour of a single neuron regardless of its width. Random initialisation breaks this symmetry so different neurons can specialise. Biases, however, may safely start at zero, since the weights already break symmetry.
Q4. You have 8,000 training samples and use a batch size of 200 for 25 epochs. How many weight updates occur?
A. Iterations per epoch = 8,000 / 200 = 40. Total updates = 40 × 25 = 1,000. Each sample is seen 25 times. Note that halving the batch size to 100 would double the updates to 2,000 without changing how many times each sample is seen — which is why smaller batches often converge in fewer epochs.
Q5. How does dropout prevent overfitting, and what changes between training and inference?
A. During training, each neuron is randomly zeroed with probability p on every batch. No neuron can rely on any specific other neuron being present, which prevents co-adaptation and forces individually useful features; it also approximates training an ensemble of many thinned networks. At inference time all neurons are active and no dropout is applied — frameworks compensate by scaling activations so the expected magnitude matches training. Forgetting to switch to eval mode is a classic bug that makes test predictions randomly noisy.
Q6. What two properties do convolutional layers have that dense layers lack, and why do they matter for images?
A. Local connectivity — a neuron sees only a small patch, which matches the fact that meaningful image structure is local. Weight sharing — the same filter slides across the whole image, so a pattern learned in one corner is detected everywhere. Together these cut parameter count by orders of magnitude (a 3×3×3 filter is 27 weights versus 150,528 for one dense neuron on a 224×224 image) and give approximate translation invariance, so a shifted object is still recognised.
Q7. Why did transformers replace LSTMs for most sequence tasks?
A. Two structural wins. Parallelism: an RNN must process step t before step t+1, so training cannot be parallelised across the sequence; attention compares all positions simultaneously, which suits GPUs and made internet-scale training practical. Direct access: an LSTM squeezes all history through one fixed-size state, while attention gives every position a direct connection to every other, so long-range dependencies do not decay. The cost is O(n²) compute in sequence length, and the need for explicit positional encodings.
Q8. You have 600 labelled X-ray images and need a classifier. Outline your approach and justify it.
A. Train from scratch is hopeless with 600 images — a CNN with millions of parameters will memorise them. Use transfer learning: take a network pretrained on ImageNet, freeze the convolutional base, replace the head with a small dense layer plus sigmoid, and train only the head. Then optionally unfreeze the top block and fine-tune at roughly 1/10th the learning rate, since X-rays differ from natural photographs. Add heavy data augmentation (flips, small rotations, zoom — but no augmentation that is medically implausible), use early stopping on a validation split, and evaluate with stratified cross-validation because 600 samples makes a single split unreliable. Report recall and PR-AUC rather than accuracy, since missing a positive diagnosis is far costlier than a false alarm.
Q9. Your validation loss decreases for 8 epochs, then steadily rises while training loss keeps falling. Diagnose and prescribe.
A. Textbook overfitting, beginning around epoch 8. The model has moved from learning general structure to memorising training noise. Prescription, in order of cheapness: enable early stopping with restore_best_weights so you keep the epoch-8 model; add dropout or increase it; add weight decay; apply data augmentation to effectively enlarge the dataset; and only then reduce model capacity (fewer layers or neurons). Getting more data is the most effective fix of all where feasible.
Q10. Your manager wants a deep learning model for a churn prediction problem with 40,000 rows and 25 tabular columns. Respond.
A. Push back, politely and with evidence. On tabular data of this size, gradient boosting (XGBoost / LightGBM) typically matches or beats a neural network while training in seconds, requiring no feature scaling, handling missing values natively, and producing feature importances the business can act on. A neural network here costs more engineering time, more tuning, and more explanation risk for no accuracy gain. The professional answer is to build both a logistic regression baseline and a gradient boosting model first, measure them on the metric that reflects business cost, and only escalate to a network if boosting demonstrably falls short.
Model Selection, Overfitting & Hyperparameter Tuning
Every model so far could be made arbitrarily complex. This unit answers the question that decides whether a project succeeds: how complex should it be? Bias and variance, cross-validation done properly, learning curves you can actually read, and a tuning strategy that does not waste a week of compute.
8.1 The only thing that matters: generalisation
A student who memorises last year's question paper scores 100% on last year's paper and fails this year's exam. A student who understands the subject scores well on both. Machine learning has exactly this failure mode, and it has a name: overfitting.
Training error is not the goal. It is not even a useful signal on its own — you can drive training error to zero on any dataset by memorising it. The goal is generalisation error: how wrong the model will be on data it has never seen. Everything in this unit is a technique for estimating that number honestly, and then reducing it.
You never get to see generalisation error. You can only estimate it, and every estimate is optimistic in proportion to how many times you peeked at the data used to compute it. Most real-world ML failures are not bad algorithms — they are honest people accidentally peeking.
8.2 Underfitting and overfitting, seen
| Symptom | Training error | Validation error | Diagnosis | Fix |
|---|---|---|---|---|
| Both bad | High | High (close to training) | Underfitting | More complex model, more/better features, train longer, reduce regularisation |
| Big gap | Very low | Much higher | Overfitting | More data, simpler model, stronger regularisation, feature selection, early stopping |
| Both good, small gap | Low | Low | Healthy | Ship it — but confirm on the untouched test set once |
| Validation better than training | Higher | Lower | Suspicious | Usually dropout/augmentation active only in training, or a leaky or too-easy validation split |
8.3 The bias–variance decomposition
The two failure modes in Figure 8.1 are not arbitrary categories. They fall out of a mathematical identity. For squared-error loss, the expected error of a model at a point x, averaged over all the training sets you could have drawn, splits exactly into three pieces:
Fire ten arrows at a bullseye. High bias, low variance: all ten arrows land tightly grouped, but in the top-left corner — consistent and consistently wrong. Your sight is misaligned. Low bias, high variance: arrows scattered all over the target, but their average position is the bullseye — your aim is right, your hand shakes. Low bias, low variance: tight group on the bullseye. High both: scattered and off-centre — the worst case, and more common than people admit.
The irreducible error σ² is wind. No archer, however good, can eliminate it.
Setup. The true value at x = 3 is 6.0. We train three different model types, each on three different random samples of the same population, and record what each predicts at x = 3. Assume the irreducible noise variance is σ² = 0.25.
| Model | Sample A | Sample B | Sample C | Mean prediction |
|---|---|---|---|---|
| M1 — always predicts a constant | 5.0 | 5.1 | 4.9 | 5.00 |
| M2 — degree-9 polynomial | 4.0 | 6.2 | 7.8 | 6.00 |
| M3 — degree-2 polynomial | 5.5 | 5.9 | 6.0 | 5.80 |
Step 1 — bias² (how far the average is from the truth of 6.0).
| M1: (5.00 − 6.0)² | = (−1.00)² = 1.0000 |
| M2: (6.00 − 6.0)² | = (0.00)² = 0.0000 |
| M3: (5.80 − 6.0)² | = (−0.20)² = 0.0400 |
Step 2 — variance (spread around the model's own mean).
| M1: [(5.0−5.0)² + (5.1−5.0)² + (4.9−5.0)²] / 3 | = [0 + 0.01 + 0.01] / 3 = 0.0067 |
| M2: [(4.0−6.0)² + (6.2−6.0)² + (7.8−6.0)²] / 3 | = [4.00 + 0.04 + 3.24] / 3 = 2.4267 |
| M3: [(5.5−5.8)² + (5.9−5.8)² + (6.0−5.8)²] / 3 | = [0.09 + 0.01 + 0.04] / 3 = 0.0467 |
Step 3 — total expected error = bias² + variance + σ².
| Model | Bias² | Variance | σ² | Total | Verdict |
|---|---|---|---|---|---|
| M1 | 1.0000 | 0.0067 | 0.25 | 1.2567 | Underfitting — stable but wrong |
| M2 | 0.0000 | 2.4267 | 0.25 | 2.6767 | Overfitting — right on average, useless on any single run |
| M3 | 0.0400 | 0.0467 | 0.25 | 0.3367 | ✔ Best total error |
The lesson. M2 is unbiased — on average it is exactly right — and it is still the worst model by a factor of eight. You do not deploy an average of three models; you deploy one, and M2's single prediction could be 4.0 or 7.8. M3 accepts a small deliberate bias (0.04) in exchange for a huge reduction in variance (2.43 → 0.05). That trade is what every regularisation technique in this book is buying. Ridge, Lasso, pruning a tree, dropout, early stopping, bagging — all of them add bias to remove more variance.
Also note the floor: no model can score below 0.25. If your business target requires error under 0.25 here, no amount of modelling will save you — you need better data or better features.
Bias ka matlab: model hamesha galat hai, ek hi tarike se. Jaise ek ghadi jo roz 10 minute peeche chalti hai — bharosemand hai, par galat.
Variance ka matlab: model har baar alag jawab deta hai. Jaise ek ghadi jo kabhi 5 minute aage, kabhi 8 minute peeche — average sahi nikal sakta hai, par kisi ek waqt pe uspe bharosa nahi kar sakte.
Simple model = zyada bias, kam variance. Complex model = kam bias, zyada variance. Dono ko ek saath zero nahi kar sakte — beech ka rasta dhoondhna hi asli kaam hai. Isi ko bias–variance tradeoff kehte hain.
The classic U-shaped curve is not the whole story. Very large, heavily over-parameterised models (huge neural networks) sometimes show double descent: test error rises as complexity grows, peaks around the point where the model can just barely fit the training data, and then falls again as you make the model even larger. This is why enormous networks do not overfit as catastrophically as the classical theory predicts. For everything in this guide except very large deep networks, the classical U-shape is the right mental model.
8.4 Train, validation and test — three sets, three jobs
Beginners use two sets. Professionals use three, and the third one is what separates a model that works in production from one that only looked good in a notebook.
| Set | Typical size | What it is for | How many times you may use it |
|---|---|---|---|
| Training | 60–70% | Fitting model parameters (weights, splits, coefficients) | Unlimited |
| Validation | 15–20% | Choosing hyperparameters, comparing models, deciding when to stop | Many times — that is its job |
| Test | 15–20% | One final honest estimate of generalisation error | Once. Genuinely once. |
You tune on the test set, get 94%, are not satisfied, tune again, get 95%, tune again, get 96%. You now report 96% and it is fiction. Each peek leaks a little information about the test set into your choices, and after twenty peeks you have effectively fitted your hyperparameters to it. The test set has silently become a second validation set, and you have no unbiased estimate left at all.
Discipline: split the test set off on day one, and do not touch it until the day you are done. If you must re-use it after a major change, be honest in your report that the estimate is now optimistic.
8.5 Cross-validation
A single validation split has a problem: with 1,000 rows, your 200-row validation set might, by luck, be unusually easy or unusually hard. Your estimate is noisy, and you cannot tell how noisy. Cross-validation fixes this by using every row for validation exactly once.
"Accuracy 82.6%" is an incomplete result. "82.6% ± 2.5%" is a result. If model A scores 0.85 ± 0.06 and model B scores 0.83 ± 0.01, model A is not reliably better — their ranges overlap heavily. Choosing A over B on the mean alone is exactly the kind of decision that evaporates in production.
Which cross-validation scheme?
| Scheme | How it splits | Use when |
|---|---|---|
| K-Fold | Random equal parts, usually k = 5 or 10 | Default for regression and balanced data |
| Stratified K-Fold | Same as K-Fold, but preserves the class ratio in every fold | Always, for classification. Non-negotiable when classes are imbalanced |
| Leave-One-Out (LOOCV) | k = n; each single row is a fold | Very small datasets (< 100 rows). Nearly unbiased but high variance and n model fits |
| Group K-Fold | Keeps all rows sharing a group ID in the same fold | Repeated measurements — multiple visits per patient, multiple photos per user |
| TimeSeriesSplit | Always trains on the past, validates on the future; never shuffles | Any time-ordered data. Ordinary K-Fold here is leakage |
| Repeated K-Fold | Run K-Fold several times with different random seeds | When you need a tighter estimate and can afford the compute |
1. Scaling before splitting. If you call StandardScaler().fit(X) on the whole dataset and then cross-validate, every fold's mean and standard deviation were computed using the validation rows. Information has leaked. The fix is structural, not a matter of care: put the scaler inside a Pipeline and pass the pipeline to cross_val_score, so it is refitted on the training part of each fold.
2. Shuffling time series. Random K-Fold on time-ordered data trains on Tuesday to predict Monday. Your CV score will look wonderful and the model will fail the moment it meets a genuinely unseen future. Use TimeSeriesSplit.
8.6 Learning curves — will more data help?
Before you spend three months and a large budget collecting more data, you can find out whether it will help. Plot training and validation score against training-set size.
| Shape you see | What it means | What to do |
|---|---|---|
| Both curves converge to a low score, and the gap is small | High bias. The model is too simple; it has already learned everything it is capable of learning. | More data will not help. Get a more expressive model or better features. |
| Training score stays high, validation far below, gap is still closing | High variance. The model can learn more, it just has not seen enough examples. | More data will help. Also try regularisation while you collect it. |
| Both high and converged, gap small | Healthy and near the ceiling for this model class. | Diminishing returns. Improve features or accept it. |
"Should we buy more labelled data?" is a question with a real price tag. A learning curve answers it with evidence instead of opinion, and takes twenty minutes to produce. It is one of the highest-value plots in applied machine learning and one of the least used.
8.7 Validation curves — how complex should this model be?
A validation curve fixes the dataset and varies one hyperparameter, plotting training and validation score against it. You get the U-shape from Figure 8.1 made literal: as complexity rises, training score climbs monotonically while validation score rises, peaks, then falls. The peak is your setting.
| Model | Parameter to sweep | Low value means | High value means |
|---|---|---|---|
| Decision tree | max_depth | Underfit | Overfit |
| Ridge / Lasso | alpha | Overfit | Underfit |
| SVM (RBF) | C, gamma | Underfit (smooth boundary) | Overfit (jagged boundary) |
| KNN | n_neighbors | Overfit (k=1 memorises) | Underfit (k=n predicts the mean) |
| Neural network | layers / neurons | Underfit | Overfit |
Note that the direction flips depending on the parameter. For alpha and n_neighbors, larger means simpler. Getting this backwards is a common source of confusion when reading someone else's tuning code.
8.8 Parameters versus hyperparameters
Parameters are learned from data by the training algorithm. You never set them by hand. Examples: the coefficients w and intercept b in linear regression, the split thresholds in a decision tree, every weight in a neural network.
Hyperparameters are set by you, before training begins, and control how learning happens. Examples: learning rate, max_depth, alpha, n_estimators, k in KNN, number of hidden layers.
Quick test: if gradient descent updates it, it is a parameter. If you typed it into the constructor, it is a hyperparameter.
8.9 Search strategies
Grid search
Define a list of candidate values for each hyperparameter and try every combination. It is exhaustive, reproducible, and easy to explain. It is also brutally expensive: the cost is the product of the list lengths, multiplied by the number of CV folds.
Tuning an SVM with C ∈ 5 values, gamma ∈ 5 values, kernel ∈ 2 values, using 5-fold CV:
5 × 5 × 2 = 50 combinations × 5 folds = 250 model fits
If one fit takes 40 seconds, that is 2 hours 46 minutes. Add one more hyperparameter with 4 options and it becomes 11 hours. This multiplicative blow-up is called the curse of dimensionality in hyperparameter space, and it is why grid search stops being practical past three or four parameters.
Random search — usually the better default
Instead of a grid, sample random combinations from distributions you specify, for a fixed budget of n_iter trials. This sounds worse than being exhaustive. It is reliably better, for a reason that is worth understanding rather than memorising.
In practice, only a few hyperparameters actually matter much; the rest barely move the score. Suppose you run a 5×5 grid over two parameters, where only the first one matters. Your 25 expensive fits explored just 5 distinct values of the parameter that matters — the other dimension was wasted effort. Run 25 random trials instead and you explore 25 distinct values of it. Five times the resolution where it counts, for the same compute.
Random search also lets you sample from continuous distributions, so you are not restricted to values you guessed in advance, and you can stop it at any point — whereas an interrupted grid search has covered only one corner of the space.
Bayesian optimisation
Grid and random search are memoryless: trial 40 knows nothing about trials 1–39. Bayesian optimisation builds a probabilistic model of "hyperparameters → score" as it goes, and uses it to choose the next point to try, balancing exploitation (search near the best result so far) against exploration (try regions it knows little about).
It typically finds a good configuration in far fewer trials, which matters enormously when a single fit takes hours. The cost is complexity, an extra library (Optuna, Hyperopt, scikit-optimize), and a sequential process that is harder to parallelise. Use it when fits are expensive; stick with random search when they are cheap.
| Strategy | Trials needed | Parallel? | Best for |
|---|---|---|---|
| Grid search | Many (exponential) | Fully | 1–3 hyperparameters, cheap fits, need reproducible exhaustiveness |
| Random search | Moderate (you choose) | Fully | The sensible default for 4+ hyperparameters |
| Bayesian / Optuna | Few | Partially | Expensive fits — large neural networks, big gradient boosting |
| Successive halving | Few (adaptive budget) | Mostly | Many candidates; kill bad ones early on small subsets |
8.10 Nested cross-validation — when you need an honest number
Here is a subtlety that catches even experienced practitioners. You run GridSearchCV with 5-fold CV, and the best configuration scores 0.91. Is 0.91 your generalisation estimate?
No. You chose that configuration because it scored highest on those folds. If you try 200 configurations, some will score well partly by luck, and you selected for exactly that. The reported best CV score is biased upwards — sometimes by several points.
Outer loop (say 5 folds): holds out a chunk purely for evaluation.
Inner loop (say 5 folds, on the outer training part only): does the entire hyperparameter search.
The winning configuration from the inner loop is then evaluated once on the outer held-out chunk, which the
search never saw. Average those five outer scores.
Cost: 5 × 5 × (number of configurations) fits. Expensive — but it is the only way to report a tuned model's performance without the selection bias baked in. Use it for papers, model comparisons, and any number a stakeholder will act on. For day-to-day iteration, a single held-out test set touched once is a reasonable approximation.
8.11 A practical tuning order
Do not tune everything at once. Most hyperparameters barely matter; a few dominate. Tune the dominant ones first, coarsely, then refine.
| Algorithm | Tune first (biggest effect) | Tune next | Usually leave alone |
|---|---|---|---|
| Random Forest | max_depth, max_features | min_samples_leaf, class_weight | n_estimators — just set it high (300–500); more trees never overfits |
| Gradient boosting / XGBoost | learning_rate, n_estimators (together), max_depth | subsample, colsample_bytree, min_child_weight | Regularisation terms, until the basics are settled |
| SVM (RBF) | C and gamma — they interact strongly, search jointly | class_weight | Kernel choice, unless RBF clearly fails |
| Neural network | learning_rate — by a wide margin the most important | Architecture size, batch_size, dropout rate | Optimiser choice — Adam is fine |
| Ridge / Lasso | alpha — that is essentially the whole model | — | Use RidgeCV/LassoCV, which tune it efficiently for you |
For learning_rate, C, gamma and alpha, the meaningful differences are multiplicative. Searching 0.1, 0.2, 0.3, 0.4 is almost pointless; searching 0.0001, 0.001, 0.01, 0.1, 1, 10 covers six orders of magnitude with the same number of trials. Use np.logspace, or loguniform for random search.
Hyperparameter tuning usually buys 1–3% over sensible defaults. Better features routinely buy 10–20%. More or cleaner data can buy more still. If you have spent two days tuning and gained 0.4%, you are optimising the wrong thing — go back to the data. Tuning is the last step of a modelling cycle, not the main one.
import numpy as np from scipy.stats import loguniform, randint from sklearn.model_selection import ( train_test_split, StratifiedKFold, TimeSeriesSplit, cross_val_score, GridSearchCV, RandomizedSearchCV, learning_curve, validation_curve) from sklearn.pipeline import Pipeline from sklearn.preprocessing import StandardScaler from sklearn.svm import SVC from sklearn.ensemble import RandomForestClassifier # ---------- 0. Split the test set off ONCE, on day one, and forget it ---------- X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, stratify=y, random_state=42) # From here on: X_test does not appear again until the very last line. cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42) # ---------- 1. Baseline with a leak-proof pipeline ---------- # The scaler lives INSIDE the pipeline, so it is refit on each fold's # training part only. Scaling outside this would leak the validation rows. pipe = Pipeline([("scale", StandardScaler()), ("clf", SVC())]) scores = cross_val_score(pipe, X_train, y_train, cv=cv, scoring="f1") print(f"baseline f1: {scores.mean():.3f} +/- {scores.std():.3f}") # ALWAYS report the std. A big std means the mean is not trustworthy. # ---------- 2. Learning curve: will more data help? ---------- sizes, train_sc, val_sc = learning_curve( pipe, X_train, y_train, cv=cv, scoring="f1", train_sizes=np.linspace(0.1, 1.0, 8), n_jobs=-1) gap = train_sc.mean(axis=1)[-1] - val_sc.mean(axis=1)[-1] if gap > 0.10: print("High variance -> collect more data, or regularise harder") elif val_sc.mean(axis=1)[-1] < 0.70: print("High bias -> more data will NOT help; need a better model/features") # ---------- 3. Validation curve: where does one parameter turn over? ---------- C_range = np.logspace(-3, 3, 7) # LOG scale, not linear tr, va = validation_curve(pipe, X_train, y_train, param_name="clf__C", # note the step__param syntax param_range=C_range, cv=cv, scoring="f1") best_C = C_range[va.mean(axis=1).argmax()] print(f"validation curve peaks at C={best_C:g}") # ---------- 4a. Grid search: exhaustive, for a small space ---------- grid = GridSearchCV( pipe, param_grid={"clf__C": np.logspace(-2, 3, 6), "clf__gamma": np.logspace(-4, 1, 6), "clf__kernel": ["rbf"]}, cv=cv, scoring="f1", n_jobs=-1, refit=True) grid.fit(X_train, y_train) print(grid.best_params_, round(grid.best_score_, 3)) # 6 x 6 x 1 = 36 combos x 5 folds = 180 fits. Count before you run. # ---------- 4b. Random search: better once you have 4+ parameters ---------- rf = RandomForestClassifier(n_estimators=400, random_state=42, n_jobs=-1) rand = RandomizedSearchCV( rf, param_distributions={ "max_depth": randint(3, 30), "max_features": loguniform(0.05, 1.0), "min_samples_leaf": randint(1, 20), "min_samples_split": randint(2, 30), }, n_iter=60, # fixed budget - you control the cost exactly cv=cv, scoring="f1", n_jobs=-1, random_state=42) rand.fit(X_train, y_train) # ---------- 5. Nested CV: an UNBIASED estimate of the tuned model ---------- # grid.best_score_ is optimistic: that config was CHOSEN for scoring high # on those folds. Nested CV puts the whole search inside an outer loop. outer = StratifiedKFold(n_splits=5, shuffle=True, random_state=0) nested = cross_val_score(grid, X_train, y_train, cv=outer, scoring="f1") print(f"nested (honest): {nested.mean():.3f} vs inner (optimistic): {grid.best_score_:.3f}") # ---------- 6. Time series data needs a different splitter entirely ---------- # Random K-Fold here would train on the future to predict the past = leakage. ts_cv = TimeSeriesSplit(n_splits=5, gap=0) # fold 1: train [0:100] validate [100:200] # fold 2: train [0:200] validate [200:300] ... training window only grows # ---------- 7. The test set. Touched once. ---------- final_model = grid.best_estimator_ # already refit on all of X_train print("FINAL held-out score:", final_model.score(X_test, y_test)) # Do not go back and tune after seeing this number. That is the whole point.
8.12 Practice questions
Q1. Your model scores 98% on training data and 71% on validation. Diagnose it and give four fixes in order of what you would try first.
A. A 27-point gap is clear overfitting — high variance. In order: (1) increase regularisation, since it is the cheapest lever — higher alpha, lower max_depth, more dropout; (2) get more training data, or augment what you have, which attacks variance directly; (3) reduce model complexity or apply feature selection, since irrelevant features give the model more noise to memorise; (4) use early stopping with the best-weights restored. Also verify there is no duplicate-row leakage inflating the training score.
Q2. Explain bias and variance to a non-technical manager in under 60 seconds.
A. "Bias is being consistently wrong in the same direction — like a scale that always reads two kilos light. Variance is being inconsistent — like a scale that gives a different answer every time you step on it. A very simple model has high bias: it is too crude to capture what is really going on. A very complex model has high variance: it chases every quirk in the data we happened to collect, so it does great on our historical data and poorly on next month's. Our job is to sit between those, and the way we check is by testing on data the model has never seen."
Q3. Why is it wrong to report GridSearchCV.best_score_ as your model's expected performance?
A. Because that configuration was selected for scoring highest on those exact folds. With enough candidates, some will score well partly through luck on that particular split, and the search picks precisely those — so the maximum of many noisy estimates is systematically optimistic. The honest options are nested cross-validation (the search runs inside an outer loop that never saw the evaluation fold) or a test set held out before the search began and scored exactly once.
Q4. You are tuning 5 hyperparameters with 6 values each, 5-fold CV. How many fits does grid search need? What would you do instead?
A. 65 = 7,776 combinations × 5 folds = 38,880 fits. At 10 seconds each that is over four days. Use random search with a budget you choose (say n_iter=100 → 500 fits), which in practice finds a comparable or better configuration because it explores far more distinct values of whichever parameters actually matter. If individual fits are very expensive, use Bayesian optimisation (Optuna) instead, which typically needs fewer trials still.
Q5. Why must classification use StratifiedKFold rather than plain KFold?
A. Plain KFold splits randomly, so fold class ratios drift from the overall ratio. With 2% positives, a random fold could easily contain zero positives — recall is then undefined or zero for that fold and the averaged score is meaningless. Stratification preserves the class proportion in every fold, which both removes that failure mode and reduces the variance of the estimate. It costs nothing, so there is no reason not to use it for classification.
Q6. A learning curve shows training and validation accuracy both flat at about 0.62, with almost no gap. Should you collect 50,000 more rows?
A. No. A small gap with both curves low and flat is the signature of high bias — the model has already extracted everything it is capable of extracting, and more of the same data will change nothing. Spend the budget on a more expressive model (gradient boosting instead of logistic regression), better feature engineering, or acquiring new kinds of information rather than more rows of the same kind. More data only helps when the gap is large and still narrowing.
Q7. You scale features with StandardScaler fitted on the full dataset, then run 10-fold CV and get 0.94. What is wrong?
A. Data leakage. The scaler's mean and standard deviation were computed using rows that later served as validation data, so every fold's "unseen" data influenced the preprocessing. The 0.94 is optimistic and will not reproduce in production, where genuinely new rows arrive after the scaler is frozen. Fix it structurally: wrap the scaler and the estimator in a Pipeline and pass that to cross_val_score, so the scaler is refit on each fold's training portion only. Relying on remembering to do it manually is how the bug recurs.
Q8. Model A scores 0.87 ± 0.09 in CV; model B scores 0.85 ± 0.01. Which do you deploy?
A. Most likely B. A's advantage of 0.02 is far smaller than its own fold-to-fold spread of 0.09, so the difference is not statistically meaningful — A's true performance could easily be worse than B's. B is also predictable, which matters operationally: a stakeholder can be told what to expect, and monitoring can flag genuine drift rather than normal noise. Before finalising, run a paired comparison on identical folds, and check whether A's variance comes from one pathological fold worth investigating.
Q9. Why does random search usually beat grid search at equal compute?
A. Because hyperparameter importance is highly uneven — typically one or two parameters dominate and the rest barely matter. A 5×5 grid spends 25 fits but tries only 5 distinct values of the important parameter; 25 random trials try 25 distinct values of it. Random search also samples continuous distributions rather than values you guessed in advance, gives you exact budget control via n_iter, and can be stopped at any time while still having covered the space broadly.
Q10. You have 80 labelled samples total. Design an evaluation strategy.
A. A single 20% test set here is 16 rows — far too small for a stable estimate. Use repeated stratified k-fold (for example 5-fold repeated 10 times with different seeds) and report the mean with its spread; with only 80 rows the compute cost is trivial. Leave-one-out is also defensible and nearly unbiased, though its estimates have high variance. Strongly prefer simple, heavily regularised models — regularised logistic regression or a shallow tree — because anything flexible will overfit 80 rows instantly. Use nested CV if you tune anything at all, and be explicit in your write-up that the confidence interval is wide. Honestly, the most valuable recommendation is usually to go and collect more data.
Applied Machine Learning
The textbook world hands you a clean table. The real world hands you raw text, a cold-start recommender, a sales series that trends and repeats, and a fraud dataset that is 99.8% "not fraud". This unit is about the four situations you will actually meet: text, recommendations, time series, and imbalanced data.
9.1 The gap between coursework and practice
Every algorithm in Units 3–8 assumed a tidy matrix: rows are examples, columns are numeric features, and a clean target sits in the last column. Almost no real problem arrives that way. An email is a paragraph of text. A customer has a viewing history, not a feature vector. Sales data has an order that matters. And in the problems where machine learning earns its keep — fraud, disease, defaults — the interesting class is rare.
The skill of applied ML is turning these messy realities into something the earlier algorithms can consume, without fooling yourself along the way. That framing — represent the data honestly, then apply what you already know — is the thread through this entire unit.
9.2 Text and natural language: from words to numbers
A model cannot multiply a weight by the word "excellent". Text has to become numbers first, and there is a ladder of ways to do it, from crude to powerful.
Step 1 — cleaning and tokenisation
Before any counting, text is normalised: lowercased, punctuation handled, split into tokens (usually words). Two classic operations you will see named constantly:
| Operation | What it does | Example | Caution |
|---|---|---|---|
| Stop-word removal | Drops ultra-common words (the, is, at, and) | "the cat sat on the mat" → "cat sat mat" | "not" is a stop word in some lists — removing it flips sentiment. Check the list. |
| Stemming | Chops words to a crude root | running, runs, ran → "run"; studies → "studi" | Fast but produces non-words. |
| Lemmatisation | Reduces to the real dictionary form using grammar | better → "good"; studies → "study" | Slower but linguistically correct. |
Step 2 — Bag of Words
The simplest representation: count how many times each vocabulary word appears in each document, ignoring order entirely. A document becomes a long vector of counts, mostly zeros.
Vocabulary = [good, movie, bad, great, boring]. Then:
"good movie great movie" → [1, 2, 0, 1, 0]
"bad boring movie" → [0, 1, 1, 0, 1]
"great good movie" and "good great movie" produce the identical vector — order is thrown away. That is the defining weakness of bag of words, and the reason "not good" and "good, not..." can look the same.
Step 3 — TF-IDF, the workhorse
Raw counts over-reward common words. The word "movie" appears in every film review and tells you nothing discriminating. TF-IDF fixes this by weighting each word by how distinctive it is.
Imagine describing a friend so others can pick them out of a crowd. Saying "has two eyes" is useless — everyone does (high document frequency, so IDF crushes it to nearly zero). Saying "has a small scar above the left eyebrow" is gold — rare and distinctive (low document frequency, high IDF). TF-IDF automatically down-weights the "two eyes" words and up-weights the "scar" words. It is a formula for "what makes this document different".
Jo word har document mein hai (jaise "movie" har review mein) — woh bekaar hai, uska weight kam. Jo word sirf kuch documents mein hai (jaise "refund" ya "terrible") — woh important hai, uska weight zyada. Matlab: "aam word = kam value, khaas word = zyada value." Bas yahi TF-IDF hai.
Step 4 — word embeddings
Bag of words and TF-IDF share a deep limitation: to them "excellent" and "superb" are as unrelated as "excellent" and "refrigerator" — just different columns. Embeddings fix this by mapping each word to a dense vector (say 300 numbers) learned so that words used in similar contexts land near each other in space.
Because meaning becomes geometry, you can do arithmetic on words. The classic result: vector("king") − vector("man") + vector("woman") ≈ vector("queen"). The direction that encodes "royalty" and the direction that encodes "gender" are separate axes in the space. This is not a party trick — it is evidence that the model captured genuine semantic structure, and it is why embeddings power modern NLP. Word2Vec and GloVe learn static embeddings; transformer models (Unit 7) learn contextual ones where "bank" gets a different vector in "river bank" than in "bank account".
| Representation | Captures meaning? | Captures order? | Use when |
|---|---|---|---|
| Bag of Words | No | No | Quick baseline, small data, interpretability |
| TF-IDF | No | No (unless n-grams) | Strong default for classification on modest data |
| Word embeddings | Yes | Partly | Larger data, synonyms matter |
| Transformer (BERT etc.) | Yes, in context | Yes | Best accuracy; needs a GPU or a pretrained model |
It is tempting to reach straight for a large transformer. Resist it. TF-IDF plus logistic regression trains in seconds, is fully explainable (you can read the top weighted words per class), and on many business text problems — spam, ticket routing, sentiment on short reviews — comes within a couple of points of a heavy model. Establish that baseline first. Only escalate when the numbers prove you need to.
A useful middle setting: n-grams. Instead of single words, count adjacent pairs or triples. "not good" becomes its own feature, so a bag-of-words model recovers some of the order information it normally throws away. In scikit-learn this is one argument: ngram_range=(1, 2).
9.3 Recommender systems
Netflix suggesting a film, Amazon suggesting a product, Spotify building a playlist — all recommender systems. There are two foundational strategies and a hybrid that most real systems actually use.
| Approach | Core idea | Needs | Strength | Weakness |
|---|---|---|---|---|
| Content-based | "Recommend items similar to ones you liked" — matches item features to your history | Item features (genre, director, tags) | Works for brand-new users on their first like; explainable | Never surprises you; stuck in a filter bubble |
| Collaborative filtering | "People similar to you liked this" — uses the crowd's behaviour, ignores item content | A big user–item interaction matrix | Discovers surprising cross-genre hits; needs no item features | Cold start: useless for a brand-new user or item |
| Hybrid | Combine both — content for cold start, collaborative once data accrues | Both | Best of each; what Netflix and Spotify really do | More engineering |
Content-based friend: "You loved that spy thriller, so here is another spy thriller by the same author." Safe, relevant, a little predictable. Collaborative friend: "Three people whose taste matches yours exactly all loved this cookbook — I have no idea why it works for you, but trust me." The collaborative friend surprises you; the content-based friend never will. A good recommender is both friends at once.
Put users as rows, items as columns, ratings as entries — a giant, mostly empty matrix R. Factorise it into two thin matrices: R ≈ U × I, where each user becomes a short vector of latent factors and each item becomes a vector in the same space. The predicted rating is their dot product. The latent factors are discovered, not designed — one might turn out to mean "seriousness", another "how action-heavy" — and multiplying the filled-in matrix back out gives a predicted rating for every user–item pair they have not touched yet. This is the algorithm that won the Netflix Prize.
A brand-new user has no history, so collaborative filtering has nothing to compare. A brand-new item has no ratings, so it can never be recommended by the crowd — a vicious circle. This is cold start, and it is the central practical headache of recommenders. Standard escapes: fall back to content-based for new users/items, recommend globally popular items initially, or ask a few onboarding questions ("pick three genres you like") to seed a profile.
9.4 Time series
Time series data — daily sales, hourly traffic, monthly revenue — breaks the single most important assumption underlying everything in Units 3–8: that examples are independent and identically distributed. Today's sales depend on yesterday's. Order is information, and shuffling destroys it.
Never shuffle. Never let the future leak into the past. Your train/test split must be a cut in time — train on January to September, test on October to December. Random splitting, random K-Fold, or fitting a scaler on the whole series all let the model peek at the future, producing a spectacular validation score and a worthless model. Use TimeSeriesSplit (Unit 8), which only ever trains on the past.
A time series is usually thought of as a sum of components:
| Component | Meaning | Example |
|---|---|---|
| Trend | Long-run direction, up or down | A company's revenue growing year on year |
| Seasonality | A fixed-period repeating pattern | Ice-cream sales peaking every summer; traffic peaking every Monday |
| Cyclic | Long, irregular swings with no fixed period | Economic boom-and-bust cycles |
| Residual / noise | What is left after removing the above | Random day-to-day fluctuation |
Two families of approach. Classical statistical models — ARIMA for trend, SARIMA when there is seasonality, exponential smoothing — are strong when the series is fairly regular and you have one series. Machine learning approaches reframe forecasting as supervised learning through feature engineering: turn the sequence into a table where the features are the recent past.
To let a normal regressor (even gradient boosting) forecast, build features from history:
Lag features: yesterday's value, the value 7 days ago, 365 days ago.
Rolling statistics: the mean and standard deviation over the last 7 or 30 days.
Calendar features: day of week, month, is-holiday, is-weekend.
Now each row is "given these facts about the recent past and the calendar, predict tomorrow" — ordinary supervised learning, and gradient boosting on these features is a genuinely strong forecaster. The one discipline: every feature must use only information available before the moment being predicted.
Time series mein sabse badi galti: data ko shuffle kar dena. Socho tum kal ka result predict kar rahe ho, lekin tumne galti se model ko parso ka data de diya jo abhi hua hi nahi. Exam mein aage ke questions ke answers pehle se dekh liye! Score to 99% aayega, par real life mein model faltu hai. Time series mein hamesha purane data pe train karo, naye pe test — kabhi ulta nahi.
9.5 Imbalanced data — when the interesting class is rare
Fraud is maybe 0.2% of transactions. A rare disease is 1% of patients. Loan defaults might be 3%. In every case the class you actually care about is drowned out — and a model can score brilliantly while being useless.
With 99.8% legitimate transactions, a model that predicts "not fraud" for everything scores 99.8% accuracy and catches zero fraud. It is the single most dangerous number in applied ML. Accuracy is meaningless on imbalanced data. Judge these models on precision, recall, F1, and PR-AUC (Unit 5), never on accuracy. Which of precision or recall matters more is a business decision: for fraud and disease, recall usually dominates — a missed case is far costlier than a false alarm.
Three broad families of fix, and they combine well.
| Family | Technique | How it works | Watch out for |
|---|---|---|---|
| Resampling | Oversampling / SMOTE | Add minority examples. SMOTE creates synthetic ones by interpolating between real neighbours. | Only ever resample the training fold — never the test set, or you measure fiction. |
| Undersampling | Throw away majority examples until classes balance | Discards potentially useful data; use only when the majority is huge. | |
| Algorithm-level | class_weight="balanced" | Tells the model each minority error costs more, so the loss stops ignoring the rare class. | Simplest first thing to try — one argument, no new data. |
| Threshold-level | Move the decision threshold | Predict positive above 0.2 instead of 0.5, trading precision for recall to suit the business cost. | Tune the threshold on validation, using the PR curve — not on the test set. |
SMOTE must be applied after the train/test split, and only to the training data — inside the cross-validation loop, on each fold's training part. If you SMOTE the whole dataset first and then split, synthetic points interpolated from a test-set example can leak into training, and your reported recall becomes fantasy. Use imblearn.pipeline.Pipeline, which applies SMOTE fold-correctly and skips it at prediction time automatically.
Maano 1000 transactions mein sirf 2 fraud hain. Agar model bole "sab safe hai" — toh 99.8% accuracy! Lekin ek bhi fraud nahi pakda. Bilkul bekaar. Isliye yahan accuracy dekhna band karo. Sawaal ye hai: "jitne actual fraud the, unme se kitne pakde?" — yeh recall hai, aur imbalanced problems mein yahi sabse zyada matter karta hai.
9.6 A note on images and audio
These are covered by CNNs in Unit 7, so only the applied reflex belongs here: you will almost never train an image model from scratch. Start from a network pretrained on ImageNet and use transfer learning (Unit 7). Rely heavily on data augmentation — flips, rotations, crops, colour shifts — to multiply a small labelled set. And be disciplined about the validation split: augment only the training data, and if the same physical object appears in several photos, keep them in the same fold (Group K-Fold) or the model will recognise the object rather than learn the task.
# ---------- Text classification: TF-IDF + logistic regression baseline ---------- from sklearn.feature_extraction.text import TfidfVectorizer from sklearn.linear_model import LogisticRegression from sklearn.pipeline import Pipeline text_clf = Pipeline([ ("tfidf", TfidfVectorizer( lowercase=True, stop_words="english", ngram_range=(1, 2), # unigrams + bigrams: recovers "not good" min_df=5, # ignore words in fewer than 5 docs (noise) max_df=0.9, # ignore words in >90% of docs (uninformative) sublinear_tf=True)), # dampen raw counts with log ("clf", LogisticRegression(max_iter=1000, C=1.0)), ]) text_clf.fit(train_texts, train_labels) # vectoriser is fit on TRAIN only - no leakage # Interpretability for free: read the most predictive words per class import numpy as np vocab = np.array(text_clf.named_steps["tfidf"].get_feature_names_out()) weights = text_clf.named_steps["clf"].coef_[0] print("most 'spam' words :", vocab[np.argsort(weights)[-10:]]) print("most 'ham' words :", vocab[np.argsort(weights)[:10]]) # ---------- Imbalanced classification: SMOTE done fold-correctly ---------- # NOTE: imblearn's Pipeline, not sklearn's - it applies SMOTE to the # training part of each fold only, and skips it at prediction time. from imblearn.pipeline import Pipeline as ImbPipeline from imblearn.over_sampling import SMOTE from sklearn.ensemble import RandomForestClassifier from sklearn.model_selection import cross_val_score, StratifiedKFold from sklearn.preprocessing import StandardScaler imb = ImbPipeline([ ("scale", StandardScaler()), ("smote", SMOTE(random_state=42)), # synthesises minority points ("clf", RandomForestClassifier( n_estimators=300, class_weight="balanced", # belt AND braces: also reweight random_state=42)), ]) cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42) # Score on the RIGHT metrics. Accuracy is meaningless here; watch PR-AUC + recall. for metric in ["average_precision", "recall", "f1"]: s = cross_val_score(imb, X, y, cv=cv, scoring=metric) print(f"{metric:20s}: {s.mean():.3f} +/- {s.std():.3f}") # ---------- Threshold tuning: choose the operating point deliberately ---------- from sklearn.metrics import precision_recall_curve imb.fit(X_train, y_train) proba = imb.predict_proba(X_val)[:, 1] prec, rec, thr = precision_recall_curve(y_val, proba) # Business rule example: cheapest threshold that still catches >= 90% of fraud ok = rec[:-1] >= 0.90 best_thr = thr[ok][np.argmax(prec[:-1][ok])] # among those, max precision print(f"deploy threshold = {best_thr:.3f} (NOT the default 0.5)") # ---------- Time series: feature engineering + a past-only split ---------- import pandas as pd df = df.sort_values("date") # order matters - sort first, never shuffle df["lag_1"] = df["sales"].shift(1) # yesterday df["lag_7"] = df["sales"].shift(7) # same weekday last week df["roll_7"] = df["sales"].shift(1).rolling(7).mean() # shift(1) FIRST = no leak df["dow"] = df["date"].dt.dayofweek df["month"] = df["date"].dt.month # Now an ordinary regressor forecasts tomorrow from these past-only features.
9.7 Practice questions
Q1. Why does TF-IDF weight a word higher when it appears in fewer documents?
A. A word appearing in almost every document (like "the" or "movie" in film reviews) cannot help tell documents apart, so it carries little information. A word confined to a few documents is distinctive — its presence strongly characterises those documents. The IDF factor log(N/df) is large when document frequency is small and near zero when the word is everywhere, so TF-IDF automatically emphasises discriminating words and suppresses ubiquitous ones.
Q2. What is the fundamental limitation shared by Bag of Words and TF-IDF, and what fixes it?
A. Both ignore word order and meaning. "The movie was not good" and "good, was the movie not?" yield similar vectors, and "excellent" and "superb" are treated as unrelated separate columns. Two fixes: n-grams partially recover local order by counting adjacent word pairs ("not good" becomes one feature), and word embeddings map words to dense vectors where synonyms sit close together, capturing meaning. Transformers capture both order and context fully.
Q3. Explain the cold-start problem and two ways to handle it.
A. Collaborative filtering recommends based on past interactions, so a brand-new user (no history) or brand-new item (no ratings) cannot be placed — there is nothing to compare. Handling: (1) fall back to content-based recommendations, which use item features rather than interaction history, so a single like or a stated preference is enough; (2) recommend globally popular items to new users as a safe default; (3) collect a little information at onboarding ("choose three genres") to seed a profile. Hybrid systems do this transition automatically.
Q4. Content-based versus collaborative filtering — give one strength of each that the other lacks.
A. Content-based works from day one for a new user (one like is enough) and can explain itself ("because you liked X, which is also sci-fi"), but it never surprises — it only ever recommends more of the same. Collaborative filtering discovers non-obvious cross-category hits by leveraging the behaviour of similar users and needs no item features at all, but suffers cold start and cannot justify its suggestions in human-readable terms. Hybrids combine the strengths.
Q5. Why is randomly splitting time series data into train and test a serious error?
A. It causes look-ahead leakage: random assignment puts future dates in the training set and past dates in the test set, so the model learns from information that would not exist at prediction time. Validation scores look excellent and collapse in production, where only the genuine past is available. Time series must be split by a cut in time — train on earlier data, test on later — using TimeSeriesSplit, and every engineered feature must use only pre-prediction information.
Q6. A fraud model has 99.7% accuracy. Your manager is delighted. What do you check first?
A. Whether it catches any fraud at all. With fraud around 0.3% of cases, predicting "not fraud" for everything scores 99.7% and detects nothing, so accuracy here is meaningless. Pull the confusion matrix and report recall (what fraction of real fraud was caught), precision (of flagged cases, how many were truly fraud), and PR-AUC. Then frame the recall/precision balance around business cost — usually catching fraud (recall) dominates — and tune the decision threshold accordingly rather than leaving it at 0.5.
Q7. What does SMOTE do, and what is the single most common mistake when using it?
A. SMOTE (Synthetic Minority Over-sampling Technique) creates new minority-class examples by interpolating between a minority point and its nearest minority neighbours, enriching the rare class without simply duplicating rows. The classic mistake is applying it to the whole dataset before splitting, so synthetic points derived from test examples leak into training and recall is wildly inflated. SMOTE must be applied only to the training portion, inside the CV loop — use imblearn's pipeline, which enforces this.
Q8. You must classify 2,000 short product reviews as positive or negative. Outline a sensible progression of approaches.
A. Start simple and escalate only if needed. (1) Baseline: TF-IDF with unigrams and bigrams into logistic regression — trains in seconds, fully interpretable via per-word weights, and often strong on short text. Evaluate with stratified CV. (2) If the baseline falls short, add light cleaning (lemmatisation), tune C and ngram_range, and try linear SVM. (3) Only if accuracy still matters and data grows, move to a pretrained transformer fine-tuned on the reviews. With just 2,000 examples the transformer's edge is often marginal and the baseline may well be the right production choice — simpler, faster, explainable.
Q9. For an imbalanced problem, compare class_weight="balanced" against SMOTE.
A. Both push the model to respect the rare class but by different means. class_weight="balanced" is algorithm-level: it multiplies the loss contribution of minority errors so they are not ignored — a single argument, no new data, no leakage risk, always worth trying first. SMOTE is data-level: it fabricates new minority samples to rebalance the training set, which can help models that have no class-weight option, but risks creating unrealistic synthetic points and must be applied fold-correctly. In practice they combine well, and threshold tuning on top of either is often what actually moves the business metric.
Q10. When forecasting sales, which features would you engineer from the raw date-and-value series, and what is the one rule they must all obey?
A. Lag features (value 1, 7, 365 days ago) to capture recent level and weekly/yearly repetition; rolling statistics (7- and 30-day moving mean and standard deviation) for local trend and volatility; and calendar features (day of week, month, holiday flag, weekend flag) for seasonality. The inviolable rule: every feature for predicting day t may use only information available strictly before t. Practically that means shifting before rolling (.shift(1).rolling(7)), never including the current day, and never fitting any transform on future data.
Deployment, Monitoring & Responsible AI
A model in a notebook has created exactly zero value. This unit covers the part most courses skip: getting a model into production, keeping it healthy as the world changes underneath it, explaining its decisions, and making sure it does not quietly harm the people it touches.
10.1 The uncomfortable truth about ML projects
Most machine learning models never reach production. Of those that do, many degrade silently within months and nobody notices until a business metric moves. The modelling code — the part every course teaches — is typically a small fraction of a working ML system. The rest is data pipelines, serving infrastructure, monitoring, retraining, and governance.
Traditional software is deterministic: the same input gives the same output forever, and the code only changes when a human edits it. A machine learning system has a third moving part — the world. Your code is unchanged, your model file is unchanged, and your accuracy still falls, because customer behaviour shifted. Software engineering assumes correctness is permanent. ML engineering assumes correctness decays, and builds for that.
10.2 The end-to-end lifecycle
10.3 Saving and packaging a model
Training produces an object in memory. To use it tomorrow, or on another machine, it has to be serialised to disk.
| Format | Best for | Notes |
|---|---|---|
| joblib | scikit-learn models and pipelines | Faster than pickle for large NumPy arrays. The usual choice. |
| pickle | General Python objects | Works, but slower for arrays and carries the same security caveat. |
| ONNX | Cross-language, cross-framework serving | Train in Python, serve in C++/Java/JavaScript. Also often faster at inference. |
| Framework-native | Deep learning (.keras, .pt) | Save architecture and weights; save the optimiser state too if you may resume training. |
1. Save the whole pipeline, not the estimator. If you save only the classifier and re-implement scaling and encoding in the serving code, the two will drift apart — a slightly different mean, a differently ordered one-hot, and predictions silently degrade. Serialise the entire Pipeline so preprocessing travels with the model.
2. Pin your versions. A model pickled under scikit-learn 1.3 may fail to load, or worse load and behave differently, under 1.5. Record library versions alongside the model file and install exactly those at serving time. Never unpickle a model file from an untrusted source — pickle can execute arbitrary code on load.
10.4 Ways to serve a model
| Pattern | How it works | Latency | Use when |
|---|---|---|---|
| Batch prediction | A scheduled job scores a large table overnight and writes results to a database | Hours | Churn scores, monthly risk ratings — anything not needed instantly. Simplest and cheapest. |
| REST API | A web service (FastAPI, Flask) loads the model once and answers requests | Milliseconds | Fraud checks at checkout, live recommendations. The most common pattern. |
| Streaming | The model consumes events from a queue (Kafka) and emits predictions continuously | Sub-second | High-throughput event pipelines, IoT, live anomaly detection. |
| Edge / on-device | The model runs on the phone, browser or sensor itself | Instant | Privacy-sensitive data, offline operation, no network round trip. |
Real-time serving brings latency budgets, autoscaling, uptime guarantees and on-call rotas. A great many business problems — "which customers are likely to churn this month" — do not need any of it. If a nightly job writing to a table solves the problem, build that. You can always promote it to an API later, and you will have learned what the model is actually worth before paying for infrastructure.
10.5 Deploying safely
Never flip 100% of traffic to a new model on day one. The staged approaches below cost little and have saved countless teams from a very bad week.
| Strategy | What happens | What it protects against |
|---|---|---|
| Shadow mode | The new model receives real traffic and logs predictions, but its output is not used | Everything — you see real-world behaviour with zero risk. Always do this first. |
| Canary release | Route 1–5% of traffic to the new model, watch, then widen gradually | Limits the blast radius of a bad model to a small slice of users |
| A/B test | Split traffic between old and new, measure the business metric, not just accuracy | The very common case where offline accuracy improves but revenue does not |
| Blue–green | Two identical environments; switch traffic over, keep the old one warm | Gives you instant rollback if something breaks |
Naya model seedha sabke liye mat chaalu karo. Pehle shadow mode — model chal raha hai, sab predictions log ho rahi hain, par koi faisla uspe nahi ho raha. Phir canary — sirf 2–3% users ko naya model. Sab theek lage tabhi poora rollout. Aur sabse zaroori: rollback ka rasta hamesha khula rakho. Offline accuracy achhi hona aur real users pe achha kaam karna — dono alag cheezein hain.
10.6 Drift: why good models go bad
This is the concept that separates people who have shipped ML from people who have only trained it. Your model was correct when you built it. The world moved. The model did not.
| Type | What changes | Example | Detect by |
|---|---|---|---|
| Data drift (covariate shift) |
The distribution of the inputs P(X) changes; the underlying relationship still holds | Your user base expands from one city to five; average income and age shift | Compare feature distributions now versus training (PSI, KS test) |
| Concept drift | The relationship P(y|X) changes — the same input now implies a different answer | Fraudsters change tactics; a transaction pattern that was safe is now the attack | Falling accuracy once labels arrive; this is the dangerous one |
| Label drift | The class balance P(y) changes | Default rate rises from 3% to 9% in a recession | Monitor the predicted-positive rate over time |
| Upstream data change | Nothing about the world changed — an engineer changed a column | Currency switched from rupees to lakhs; a field became nullable | Schema validation and range checks on every batch |
You learned to drive in Delhi. Data drift is being moved to Mumbai: the roads look different, the traffic is denser, everything you see has changed — but the rules of driving still work. You will be a bit worse until you adapt. Concept drift is being moved to London: same-looking roads, but now you must drive on the left. What you learned is no longer merely unfamiliar, it is actively wrong. Data drift degrades performance; concept drift inverts it.
A drifting model does not crash. It does not log an error. It returns confident, well-formatted predictions that are increasingly wrong, and unless someone is measuring, the first signal is a business metric moving three months later. This is why monitoring is not optional garnish — it is the only thing standing between you and a model that has been quietly wrong since March.
10.7 What to monitor
Four layers, from easiest to hardest to obtain.
| Layer | Examples | Available |
|---|---|---|
| System health | Latency (p50, p95, p99), error rate, throughput, memory, uptime | Immediately |
| Data quality | Missing-value rate per column, schema conformance, value ranges, unexpected new categories | Immediately — catches most real incidents |
| Drift | Feature distribution shift (PSI / KS), prediction distribution shift, predicted-positive rate | Immediately — no labels needed, which is what makes it valuable |
| Model performance | Accuracy, recall, precision, calibration — the ground truth | Only when labels arrive, which may take weeks or months |
For a loan model you might not know whether a decision was right for two years. You cannot wait that long to discover a problem. Drift metrics are leading indicators — they need no labels at all, only the inputs and outputs you already have. When the input distribution moves sharply, you get a warning long before the accuracy number could possibly tell you anything.
Retraining strategy follows from this. Scheduled retraining (monthly, quarterly) is simple and predictable. Triggered retraining fires when a drift metric or performance metric crosses a threshold — more efficient, but needs the monitoring to be trustworthy. Either way, the newly trained model goes through the same staged rollout as any other: shadow, canary, full.
10.8 Explainability
"The model said no" is not an acceptable answer to a customer denied a loan, a doctor questioning a diagnosis, or a regulator conducting an audit. In several jurisdictions it is not a legal one either.
| Method | Scope | How it works | Notes |
|---|---|---|---|
| Coefficients | Global | Read the weights of a linear or logistic model directly | Only meaningful on standardised features; only for linear models |
| Feature importance | Global | Tree models report how much each feature reduced impurity | Biased towards high-cardinality features; prefer permutation importance |
| Permutation importance | Global | Shuffle one column and measure how much the score drops | Model-agnostic, intuitive, honest. Misleading when features are correlated. |
| SHAP | Global and local | Game-theoretic: fairly divides the prediction among features | The current standard. Slower, but has strong theoretical guarantees and additive per-prediction values. |
| LIME | Local | Fits a simple interpretable model in the neighbourhood of one prediction | Fast and intuitive; can be unstable — two runs may differ |
| Partial dependence | Global | Plots the average effect of one feature across its range | Great for showing shape (is the effect linear? does it plateau?) |
A team wins a project worth ₹10 lakh in bonus. How much did each member contribute? The fair way: consider every possible order in which people could have joined the team, measure how much the expected outcome improved when each person joined, and average over all orderings. That is the Shapley value from cooperative game theory, and SHAP applies exactly this to features. It answers, for one specific prediction: "this applicant was scored 0.73; the low credit score pushed it down 0.12, the long employment history pushed it up 0.08…" and those contributions add up to the final prediction. That additivity is why it is trusted.
A feature having high importance means the model uses it, not that it causes the outcome. If "owns an umbrella" predicts "gets wet", the model will happily rank umbrella ownership as important — and confiscating umbrellas will not keep anyone dry. Explanations describe the model's behaviour, not the world's mechanics. Present them as "what drove this decision", never as "what would change the outcome" unless you have done genuine causal work.
10.9 Fairness and bias
A model trained on historical data learns historical patterns — including the unjust ones. This is not a hypothetical risk; it is the default behaviour, and avoiding it takes deliberate effort.
| Source of bias | What happens | Example |
|---|---|---|
| Historical bias | The training data faithfully records a biased past | Past hiring favoured one group, so a model trained on it recommends that group |
| Representation bias | Some groups are under-represented in the data | A medical model trained mostly on one demographic performs worse on others |
| Measurement bias | The proxy you measure is not the thing you care about | Using "arrests" as a proxy for "crime" imports policing disparities |
| Aggregation bias | One model is forced onto groups that genuinely behave differently | A single diagnostic threshold applied across populations with different baselines |
| Deployment bias | The model is used for something it was never validated for | A risk-triage tool repurposed as a sentencing input |
Deleting the "gender" column feels like a fix. It is not. Other features act as proxies — postcode correlates with ethnicity, certain job titles and career gaps correlate with gender, a school name can encode both. The model reconstructs the sensitive attribute from what remains and discriminates just as effectively, while you have lost the ability to measure whether it is doing so. Keeping the attribute for auditing (while not using it as a model input) is often the more responsible design.
Demographic parity: the positive prediction rate is equal across groups.
Equal opportunity: the true positive rate (recall) is equal across groups.
Equalised odds: both true positive and false positive rates are equal across groups.
Calibration: a predicted probability of 0.7 means 70% in every group.
It has been proven that when base rates genuinely differ between groups, you cannot satisfy calibration and equalised odds simultaneously. This is not an engineering shortcoming to be optimised away — it is a mathematical impossibility. Which definition applies is an ethical and often legal decision about the specific application, and it must be made explicitly, with the affected stakeholders, before modelling begins.
10.10 Privacy and governance
Three ideas worth knowing by name. Anonymisation is weaker than it sounds — combinations of "anonymous" fields such as postcode, birth date and gender can re-identify a large share of a population, so removing names is not sufficient. Differential privacy adds carefully calibrated mathematical noise so that no individual's presence in the dataset can be detected, providing a formal guarantee rather than a hope. Federated learning trains across many devices without the raw data ever leaving them — only model updates are shared, which is how phone keyboards improve without uploading your messages.
Governance is the unglamorous layer that makes all of this auditable: version your data as well as your code, log every model version with its training data, hyperparameters and metrics, write a model card documenting intended use, limitations and known failure modes, and keep a clear record of who approved deployment and on what evidence.
"Should we build this?" comes before "can we". A model that predicts something accurately can still be the wrong thing to deploy — because the decision it automates should involve human judgement, because the error costs fall on people who did not consent, or because it will be used in ways you cannot control. Asking this early is cheaper than asking it in a regulatory hearing.
# ---------- 1. Save the WHOLE pipeline, with metadata ---------- import joblib, json, sklearn, numpy as np from datetime import datetime joblib.dump(pipeline, "model_v3.joblib") # pipeline = preprocessing + estimator metadata = { "version": "3.0.0", "trained_at": datetime.utcnow().isoformat(), "sklearn": sklearn.__version__, # PIN THIS - loading under a "numpy": np.__version__, # different version can break "features": list(feature_names), # order matters at serve time "test_f1": 0.834, "train_rows": len(X_train), "threshold": 0.32, # the tuned one, NOT 0.5 } json.dump(metadata, open("model_v3.meta.json", "w"), indent=2) # ---------- 2. Serve it behind a REST API ---------- from fastapi import FastAPI, HTTPException from pydantic import BaseModel import pandas as pd, logging, time app = FastAPI() model = joblib.load("model_v3.joblib") # load ONCE at startup, not per request meta = json.load(open("model_v3.meta.json")) class Request(BaseModel): # pydantic validates types for free age: float income: float tenure_months: int @app.post("/predict") def predict(req: Request): t0 = time.time() X = pd.DataFrame([req.dict()])[meta["features"]] # enforce column ORDER p = float(model.predict_proba(X)[0, 1]) decision = int(p >= meta["threshold"]) # tuned threshold, not 0.5 # Log every prediction: this IS your monitoring dataset. logging.info(json.dumps({"input": req.dict(), "proba": p, "decision": decision, "model_version": meta["version"], "latency_ms": (time.time() - t0) * 1000})) return {"probability": round(p, 4), "decision": decision, "model_version": meta["version"]} # ---------- 3. Drift detection: no labels required ---------- from scipy.stats import ks_2samp def psi(expected, actual, buckets=10): """Population Stability Index. <0.1 stable | 0.1-0.25 watch | >0.25 act.""" edges = np.percentile(expected, np.linspace(0, 100, buckets + 1)) edges[0], edges[-1] = -np.inf, np.inf e = np.histogram(expected, edges)[0] / len(expected) + 1e-6 a = np.histogram(actual, edges)[0] / len(actual) + 1e-6 return float(np.sum((a - e) * np.log(a / e))) for col in feature_names: score = psi(X_train[col], X_live[col]) ks_p = ks_2samp(X_train[col], X_live[col]).pvalue if score > 0.25: print(f"ALERT drift in {col}: PSI={score:.3f}, KS p={ks_p:.4f} -> retrain") # Also watch the PREDICTION distribution - cheap and often the first signal. if abs(live_positive_rate - train_positive_rate) > 0.05: print("ALERT: predicted-positive rate has shifted materially") # ---------- 4. Explain individual predictions with SHAP ---------- import shap explainer = shap.TreeExplainer(model.named_steps["clf"]) shap_values = explainer.shap_values(X_transformed) # Global view: which features drive the model overall # shap.summary_plot(shap_values, X_transformed, feature_names=feature_names) # Local view: why THIS customer was declined (values sum to the prediction) i = 0 contrib = sorted(zip(feature_names, shap_values[i]), key=lambda kv: abs(kv[1]), reverse=True)[:5] for name, val in contrib: direction = "increased" if val > 0 else "decreased" print(f"{name:24s} {direction} risk by {abs(val):.4f}") # ---------- 5. Fairness audit: measure per group, always ---------- from sklearn.metrics import recall_score, precision_score for g in sensitive_attr.unique(): m = sensitive_attr == g print(f"group={g:12s} n={m.sum():5d} " f"recall={recall_score(y_test[m], y_pred[m]):.3f} " f"precision={precision_score(y_test[m], y_pred[m]):.3f} " f"positive_rate={y_pred[m].mean():.3f}") # Equal recall across groups -> equal opportunity # Equal positive_rate -> demographic parity # You generally CANNOT have every fairness criterion at once. Choose # deliberately, document the choice, and revisit it with stakeholders.
10.11 Practice questions
Q1. Distinguish data drift from concept drift, and say which is more dangerous.
A. Data drift is a change in the input distribution P(X) while the input–output relationship holds — your users get younger, but young users still behave as young users always did. Concept drift is a change in P(y|X): the same input now implies a different outcome, as when fraudsters adopt new tactics so a previously safe pattern becomes an attack. Concept drift is far more dangerous because the model's learned rules are now actively wrong rather than merely applied to unfamiliar inputs, and because it can only be confirmed once labels arrive — which may be months later.
Q2. Why should you serialise the entire pipeline instead of just the trained estimator?
A. Because the estimator alone is meaningless without the exact preprocessing it was trained under. If serving code re-implements scaling, imputation and encoding separately, the two implementations drift — a different mean, a differently ordered one-hot, a changed default for unseen categories — and predictions silently degrade with no error raised. Saving the whole Pipeline guarantees the fitted transformer parameters travel with the model and that training and serving apply identical transformations.
Q3. You cannot get labels for six months. How do you monitor the model in the meantime?
A. Monitor everything that needs no labels. Data quality: missing-value rates, schema conformance, out-of-range values, unexpected new categories. Input drift: PSI or KS tests on each feature against the training distribution. Output drift: the distribution of predicted probabilities and the predicted-positive rate — a sudden move here is an early warning. System health: latency and error rates. These are leading indicators; combine them with any partial or proxy feedback available (user overrides, complaint rates, manual review outcomes) and log every prediction so that a proper evaluation is possible the moment labels do arrive.
Q4. What is shadow deployment and why is it worth the extra effort?
A. The new model receives real production traffic and logs its predictions, but those predictions are not used for any decision — the old model still drives the system. You therefore observe real latency, real input distributions, real failure cases and real disagreement with the incumbent, at zero risk to users. It surfaces problems offline testing cannot: unexpected nulls, schema mismatches, feature pipeline bugs and latency blowups under real load. It is the cheapest insurance in ML deployment.
Q5. Your model is 4% more accurate offline but revenue drops in the A/B test. Explain how this happens.
A. Accuracy is not the business objective. Several mechanisms produce this: the model may have improved on the common, low-value cases while getting worse on the rare, high-value ones; the errors it now makes may be costlier even though there are fewer of them (more false negatives on high-margin customers); the offline test set may not represent live traffic; or the model may have changed user behaviour in a feedback loop — more conservative recommendations reduce discovery and therefore purchases. The lesson is to define the business metric first and treat offline accuracy as a proxy that must be validated online.
Q6. Why does dropping the sensitive attribute not make a model fair?
A. Because other features act as proxies. Postcode correlates with ethnicity, job title and career gaps correlate with gender, school name encodes both. The model reconstructs the protected attribute from correlated features and discriminates just as effectively — while you have removed your own ability to measure whether it is doing so, since you no longer have the attribute to audit against. The better approach is to retain the attribute for measurement and auditing, exclude it from model inputs if appropriate, explicitly test performance parity across groups, and apply a fairness constraint chosen deliberately for the application.
Q7. Compare SHAP and LIME.
A. Both explain individual predictions and both are model-agnostic. LIME perturbs the input around one instance and fits a simple interpretable model locally — fast and intuitive, but the explanation depends on the random perturbations, so repeated runs can disagree. SHAP computes Shapley values from cooperative game theory, fairly attributing the prediction among features; it is consistent, theoretically grounded, additive (contributions sum to the prediction), and offers both local and global views. SHAP is slower, especially on non-tree models, but it is the current default when the explanation must be defensible.
Q8. Give three reasons a model that worked perfectly in testing might fail immediately in production.
A. (1) Training–serving skew: preprocessing in the serving path differs from training — different column order, different encoding of unseen categories, a scaler refitted or not applied. (2) Data leakage in the original evaluation: a feature used in training is unavailable or computed differently at prediction time, so the impressive test score was never real. (3) Distribution mismatch: the test set came from the same historical snapshot as training, while live traffic differs — different user mix, seasonality, or an upstream schema change. Add to these latency or memory limits that were never exercised offline.
Q9. Two fairness criteria your stakeholders want cannot both be satisfied. What do you do?
A. First explain that this is a proven mathematical impossibility when base rates differ between groups, not an engineering failure — calibration and equalised odds cannot both hold. Then move the decision to where it belongs: present the concrete trade-off with numbers (what each criterion costs each group in real terms), involve legal, domain and affected-community stakeholders, and have them choose the criterion that fits the application's ethical and regulatory context. Document the choice and its rationale in the model card, implement and monitor against it, and schedule a review. The engineer's job is to make the trade-off visible and quantified, not to pick silently.
Q10. Outline what you would put in place before a credit-scoring model goes live.
A. Technical: the full pipeline serialised with pinned library versions and a tuned decision threshold; a model card recording intended use, training data, metrics and known limitations; per-prediction logging; drift and data-quality monitoring with alert thresholds; a staged rollout (shadow, then canary) with a tested rollback path. Governance: per-group performance audit with an agreed fairness criterion; SHAP-based adverse-action explanations so any declined applicant can be told why; a documented human appeal route; data-retention and privacy review; and a named owner with a scheduled review cadence. In regulated lending, explainability and the appeal mechanism are frequently legal requirements, not optional extras.
Mathematics Refresher
Only what you actually need, explained in plain language. If a formula in the main text felt opaque, the piece you were missing is almost certainly on these pages.
A.1 Notation you will meet constantly
| Symbol | Read it as | Meaning |
|---|---|---|
| xi | "x sub i" | The i-th training example (one row) |
| xij | "x i j" | Feature j of example i (one cell) |
| yi | "y sub i" | The true label for example i |
| ŷ | "y hat" | The predicted value. The hat always means "estimated". |
| n | "n" | Number of examples (rows) |
| d | "d" | Number of features (columns) |
| Σ | "sum over" | Add up everything that follows |
| Π | "product over" | Multiply everything that follows |
| θ, w | "theta", "w" | The model's learnable parameters |
| η, α | "eta", "alpha" | Learning rate (or regularisation strength — context decides) |
| ∂L/∂w | "partial L by partial w" | How much L changes when w changes a little |
| ∈ | "is in" | Belongs to a set, e.g. y ∈ {0, 1} |
| ‖v‖ | "norm of v" | The length of vector v |
| argmin / argmax | "the argument that minimises/maximises" | Not the smallest value — the input that produces it |
Σni=1 xi is just a for loop written in Greek. It says: start at i = 1, go to i = n, add up xi each time. In Python that is sum(x[i] for i in range(n)). Every intimidating summation in this book unpacks into a loop that fits on one line.
A.2 Linear algebra — the four things that matter
A vector is a list of numbers: one row of your data, or one set of model weights. A matrix is a grid: your entire dataset, rows as examples and columns as features.
a · b = a₁b₁ + a₂b₂ + … + anbn — multiply element by element, then add it all up. One number comes out.
Why it is everywhere: every linear model is a dot product. A prediction w₁x₁ + w₂x₂ + … + b is exactly w · x + b. A neural network layer is a matrix of dot products. An SVM kernel is a dot product in a transformed space. Cosine similarity is a normalised dot product. Learn this one operation properly and half the notation in machine learning stops being mysterious.
Geometric meaning: the dot product is large and positive when two vectors point the same way, zero when they are perpendicular, and negative when they oppose. That is why it measures "agreement".
Matrix multiplication C = AB is a grid of dot products: entry Cij is row i of A dotted with column j of B. The shapes must line up — (m×k)(k×n) = (m×n), with the inner dimensions matching. Most shape errors in ML code are this rule being violated.
The transpose AT flips rows into columns. It appears mainly to make shapes align, as in the normal equation from Unit 3.
Norms measure the size of a vector, and the two you need are exactly the two regularisers: the L2 norm ‖w‖₂ = √(w₁² + w₂² + …) is ordinary straight-line distance and underlies Ridge; the L1 norm ‖w‖₁ = |w₁| + |w₂| + … is "city-block" distance and underlies Lasso. L1's corners are precisely why Lasso drives coefficients to exactly zero while Ridge only shrinks them.
Eigenvectors and eigenvalues appear once, in PCA. An eigenvector of a matrix is a direction that the matrix stretches without rotating; the eigenvalue is how much it stretches. For a covariance matrix, the eigenvector with the largest eigenvalue is the direction of greatest variance in the data — which is exactly the first principal component.
A.3 Calculus — derivatives, and why we care
A derivative is a slope: how fast the output changes when you nudge the input. That is all training needs. If the slope of the loss with respect to a weight is positive, increasing the weight increases the loss, so you decrease the weight. Gradient descent is that sentence, repeated.
| Function | Derivative | Where it shows up |
|---|---|---|
| c (a constant) | 0 | Bias terms vanish when differentiating w.r.t. weights |
| xn | n·xn−1 | The power rule — used on squared error |
| x² | 2x | Why MSE gradients have a factor of 2 |
| ex | ex | Sigmoid and softmax derivations |
| ln(x) | 1 / x | Cross-entropy loss |
| σ(x) | σ(x)(1 − σ(x)) | Backpropagation through sigmoid |
| max(0, x) | 1 if x > 0, else 0 | Backpropagation through ReLU |
If y depends on u, and u depends on x, then dy/dx = (dy/du) × (du/dx).
In words: to find how much a change in x affects y, multiply the effects along the path. A neural network is a long chain of such dependencies, and backpropagation is nothing more than this rule applied repeatedly, from the loss backwards to every weight. Worked Example 7.1 is this formula and nothing else.
A gradient is simply the vector of all partial derivatives — the slope in every direction at once. It points in the direction of steepest increase, which is why gradient descent steps in the negative gradient direction. A convex function is bowl-shaped with a single minimum, so gradient descent is guaranteed to find the global optimum; linear and logistic regression are convex, while neural networks are not, which is why initialisation and multiple runs matter for them.
A.4 Probability and statistics
| Concept | Formula / meaning | Plain-language reading |
|---|---|---|
| Mean | μ = (1/n) Σ xi | The average — the centre of the data |
| Variance | σ² = (1/n) Σ (xi − μ)² | Average squared distance from the mean — how spread out |
| Std deviation | σ = √variance | Spread, back in the original units |
| Median | Middle value when sorted | Robust to outliers, unlike the mean |
| Covariance | Cov(X,Y) = E[(X−μX)(Y−μY)] | Do they move together? Scale-dependent. |
| Correlation | ρ = Cov(X,Y) / (σXσY) | Covariance normalised to −1…+1. Scale-free. |
P(A|B) = P(B|A) × P(A) / P(B)
P(A|B) is the posterior: what we believe about A after seeing evidence B.
P(B|A) is the likelihood: how probable the evidence is if A were true.
P(A) is the prior: what we believed before any evidence.
P(B) is the normalising constant that makes the probabilities sum to 1.
Read as a sentence: new belief = old belief, adjusted by how well the evidence fits. This is the whole engine of Naive Bayes (Unit 4), and the reason spam filters work.
A disease affects 1 in 1,000 people. A test is 99% accurate both ways. You test positive. What is the chance you have it?
Out of 100,000 people: 100 have the disease, and 99 of them test positive. 99,900 are healthy, and 1% of them — 999 people — test positive anyway. So of 1,098 positive tests, only 99 are real: about 9%. Most people guess 99%.
This is precisely why accuracy is a terrible metric on imbalanced data (Units 5 and 9). When the base rate is low, false positives from the huge majority class swamp the true positives from the tiny minority class — no matter how good the test sounds.
Two distributions to recognise. The normal (Gaussian) distribution is the bell curve, where roughly 68% of values fall within one standard deviation of the mean and 95% within two; it justifies z-score outlier rules and standardisation. The Bernoulli distribution describes a single yes/no trial with probability p, and it is the assumed target distribution behind logistic regression.
Finally, log probabilities. Multiplying many small probabilities underflows to zero on a computer, so we take logarithms and add instead — log(ab) = log(a) + log(b). Since log is monotonic, whatever maximises the probability also maximises the log probability, so nothing is lost. This is why nearly every loss function in machine learning has a log in it.
Formula Cheat Sheet
Every formula from the guide, on facing pages, with each symbol named. Built for the night before an exam.
B.1 Regression
| Name | Formula | Notes |
|---|---|---|
| Linear model | ŷ = w1x1 + … + wdxd + b | Same as w · x + b |
| MSE | (1/n) Σ (yi − ŷi)² | Penalises large errors heavily; units are squared |
| RMSE | √MSE | Back in original units — report this to stakeholders |
| MAE | (1/n) Σ |yi − ŷi| | Robust to outliers |
| R² | 1 − (SSres / SStot) | Fraction of variance explained; can go negative |
| Normal equation | w = (XTX)−1XTy | Exact solution; O(d³) so impractical for large d |
| Gradient descent | w ← w − η ∂J/∂w | η = learning rate |
| MSE gradient | ∂J/∂wj = (2/n) Σ (ŷi − yi)xij | Error × input, averaged |
| Ridge (L2) | J = MSE + αΣwj² | Shrinks all weights; keeps all features |
| Lasso (L1) | J = MSE + αΣ|wj| | Drives weights to exactly 0 — feature selection |
| Elastic Net | J = MSE + α1Σ|wj| + α2Σwj² | Both; good with correlated features |
B.2 Classification
| Name | Formula | Notes |
|---|---|---|
| Sigmoid | σ(z) = 1 / (1 + e−z) | Maps any real number to (0, 1) |
| Logistic regression | P(y=1|x) = σ(w · x + b) | Linear model passed through a sigmoid |
| Binary cross-entropy | −(1/n)Σ[y log ŷ + (1−y) log(1−ŷ)] | Punishes confident wrong answers severely |
| Softmax | ezi / Σj ezj | Multi-class; outputs sum to 1 |
| Odds / logit | log(p / (1−p)) = w · x + b | What logistic regression is linear in |
| Euclidean distance | √Σ(xi − yi)² | KNN, K-Means — requires scaling |
| Manhattan distance | Σ|xi − yi| | Better in high dimensions |
| Naive Bayes | P(y|x) ∝ P(y) Πj P(xj|y) | "Naive" = assumes features independent given y |
| Entropy | H = −Σ pi log₂ pi | 0 = pure node, 1 bit = 50/50 split |
| Information gain | IG = H(parent) − Σ(nk/n)H(childk) | Pick the split that maximises this |
| Gini impurity | 1 − Σ pi² | Like entropy, cheaper to compute; sklearn default |
| SVM margin | 2 / ‖w‖ | Maximise it → minimise ‖w‖ |
| RBF kernel | K(x, x′) = exp(−γ‖x − x′‖²) | High γ = tight, wiggly boundary |
B.3 Evaluation
| Name | Formula | Answers the question |
|---|---|---|
| Accuracy | (TP + TN) / (TP + TN + FP + FN) | Overall correctness — useless when imbalanced |
| Precision | TP / (TP + FP) | Of those I flagged, how many were right? |
| Recall (sensitivity, TPR) | TP / (TP + FN) | Of all real positives, how many did I catch? |
| Specificity (TNR) | TN / (TN + FP) | Of all real negatives, how many did I clear? |
| F1 score | 2PR / (P + R) | Harmonic mean — punishes imbalance between P and R |
| F-beta | (1+β²)PR / (β²P + R) | β > 1 favours recall; β < 1 favours precision |
| FPR | FP / (FP + TN) | The x-axis of the ROC curve |
| ROC-AUC | Area under TPR vs FPR | P(random positive ranked above random negative). 0.5 = chance |
| PR-AUC | Area under precision vs recall | Use this instead of ROC when imbalanced |
| Bias–variance | Error = Bias² + Variance + σ² | σ² is the irreducible floor |
| Silhouette | s = (b − a) / max(a, b) | Clustering quality; +1 good, 0 borderline, −1 wrong cluster |
| PSI (drift) | Σ(ai − ei) ln(ai/ei) | <0.1 stable, 0.1–0.25 watch, >0.25 retrain |
B.4 Unsupervised and neural networks
| Name | Formula | Notes |
|---|---|---|
| K-Means objective | min Σj=1..k Σx in Cj ‖x − μj‖² | Called inertia or within-cluster sum of squares |
| Standardisation | z = (x − μ) / σ | Mean 0, std 1. The usual default. |
| Min-max scaling | (x − min) / (max − min) | Squashes to [0, 1]; sensitive to outliers |
| Neuron | a = φ(w · x + b) | Dot product, add bias, squash |
| Layer (matrix form) | a(l) = φ(W(l)a(l−1) + b(l)) | The entire forward pass |
| ReLU | max(0, z) | Default hidden activation; derivative is 1 or 0 |
| Chain rule | ∂L/∂w = (∂L/∂a)(∂a/∂z)(∂z/∂w) | All of backpropagation |
| Sigmoid derivative | σ′(z) = a(1 − a) | Max value 0.25 → vanishing gradients |
| He initialisation | Var(w) = 2 / nin | For ReLU networks |
| Xavier initialisation | Var(w) = 1 / nin | For tanh / sigmoid networks |
| Attention | softmax(QKT / √dk) V | Query, Key, Value — a learned weighted average |
| TF-IDF | tf(t,d) × log(N / df(t)) | Common word = low weight; rare word = high weight |
| RNN step | ht = tanh(Wxxt + Whht−1 + b) | Same weights reused at every time step |
B.5 The decision tables worth photographing
| If your situation is… | Reach for… |
|---|---|
| Tabular data, need a strong result fast | Gradient boosting (XGBoost / LightGBM) |
| Tabular data, need explainability | Logistic regression or a shallow decision tree |
| Images, audio, video | CNN with transfer learning |
| Text classification, modest data | TF-IDF + logistic regression (then transformers if needed) |
| Sequences, time order matters | Lag features + gradient boosting, or LSTM / transformer |
| Fewer than a few hundred rows | Regularised linear model; repeated stratified CV |
| No labels at all | K-Means, DBSCAN, PCA |
| Rare positive class | class_weight="balanced", threshold tuning, PR-AUC |
| Algorithm | Needs feature scaling? | Why |
|---|---|---|
| KNN, K-Means, SVM, PCA, neural networks | Yes — always | They use distances or gradients, so larger-scaled features dominate |
| Ridge, Lasso, Elastic Net | Yes | The penalty is applied per coefficient, so scale changes the penalty |
| Logistic / linear regression (unregularised) | Helps convergence | Not required for correctness, but gradient descent converges faster |
| Decision trees, Random Forest, gradient boosting | No | Splits depend on order, not magnitude — scaling changes nothing |
| Naive Bayes | No | Works on per-feature distributions independently |
Glossary
Roughly 130 terms, defined the way you would explain them to a friend — not the way a textbook would. Where a term is genuinely slippery, there is a Hinglish line to anchor it.
A
| Accuracy | Fraction of predictions that were correct. Honest on balanced data, dangerously misleading on imbalanced data. |
| Activation function | The non-linear squash applied after a neuron's dot product (ReLU, sigmoid, tanh). Without it, stacking layers achieves nothing — the whole network collapses to one linear model. |
| AdaBoost | Boosting method that re-weights misclassified examples upward so the next weak learner focuses on them. |
| Adam | The default optimiser for neural networks. Combines momentum with per-parameter adaptive learning rates. |
| Attention | A mechanism that lets a model decide which parts of the input matter for each output. softmax(QKT/√dk)V. The foundation of transformers. |
| AUC | Area Under the Curve. Usually ROC-AUC; a single number summarising ranking quality across all thresholds. |
| Autoencoder | A neural network trained to reproduce its own input through a narrow bottleneck, forcing it to learn a compressed representation. |
B
| Backpropagation | The algorithm that computes how much each weight contributed to the error, by applying the chain rule backwards from the loss. Not a learning rule — just the gradient calculator. |
| Bagging | Bootstrap Aggregating. Train many models on random resamples of the data and average them. Reduces variance. |
| Batch size | How many examples the model looks at before updating its weights once. |
| Batch normalisation | Normalising a layer's activations during training. Stabilises and speeds up deep network training. |
| Bayes' theorem | How to update a belief when new evidence arrives. See Appendix A.4. |
| Bias (parameter) | The constant b added in w · x + b. Lets the model shift its output up or down. |
| Bias (error) | Error from the model being too simple to capture the real pattern. High bias = underfitting. |
| Bias (fairness) | Systematic unfairness toward a group. A social problem, not a statistical one — three different meanings share one word, so always check which is meant. |
| Bias–variance tradeoff | Simple models miss the pattern; complex models memorise the noise. The art is landing between them. |
| Boosting | Train models sequentially, each one correcting the previous one's mistakes. Reduces bias. |
| Bootstrap | Sampling with replacement to create a new dataset of the same size. Some rows appear twice, some not at all. |
C
| Calibration | Whether predicted probabilities mean what they say — if the model says 70%, does it happen 70% of the time? |
| Categorical variable | A feature with discrete labels ("Delhi", "Mumbai") rather than numbers. |
| Centroid | The mean point of a cluster. K-Means moves these around until they stop moving. |
| Class imbalance | When one class vastly outnumbers another (fraud, disease, churn). Breaks accuracy; demands PR-AUC and threshold tuning. |
| Classification | Predicting a category. Contrast with regression. |
| Clustering | Grouping similar examples without any labels. Unsupervised. |
| Collaborative filtering | Recommending based on what similar users liked. "People like you also bought…" |
| Concept drift | The relationship between features and target changes over time. Your inputs look the same but the right answer moved. |
| Confusion matrix | The 2×2 grid of TP, FP, TN, FN. Every classification metric is computed from it. |
| Convergence | When training stops improving — the loss has flattened out. |
| Convex | Bowl-shaped, single minimum, no local traps. Linear and logistic regression are convex; neural networks are not. |
| Convolution | Sliding a small filter across an image to detect a local pattern (edge, texture) anywhere it appears. |
| Cost function | Same as loss function — the number training tries to make small. |
| Cross-entropy | The standard classification loss. Rewards confident correct answers, punishes confident wrong ones harshly. |
| Cross-validation | Split into k folds, train on k−1, test on the held-out one, rotate. Gives a reliability estimate instead of one lucky number. |
| Curse of dimensionality | In high dimensions everything is far from everything else, so distance-based methods stop meaning anything. |
D – E
| Data drift | The input distribution changes (new user demographics, new sensor). The relationship may still hold. |
| Data leakage | Information from the test set or the future sneaking into training. Produces beautiful validation scores and catastrophic production failures. The single most common serious mistake in applied ML. |
| DBSCAN | Density-based clustering. Finds arbitrarily-shaped clusters and labels sparse points as noise. No need to choose k. |
| Decision boundary | The surface in feature space where the model switches its predicted class. |
| Decision tree | A flowchart of yes/no questions learned from data. Highly interpretable, highly prone to overfitting alone. |
| Deep learning | Neural networks with many layers. Excels where features are hard to hand-engineer: images, audio, text. |
| Dimensionality reduction | Compressing many features into few while keeping the structure. PCA, t-SNE, UMAP. |
| Dropout | Randomly switching off neurons during training so the network cannot rely on any one path. A regulariser. |
| Early stopping | Halt training when validation loss stops improving. The cheapest regulariser there is. |
| Elbow method | Plot inertia against k and pick the bend. A rough heuristic for choosing cluster count. |
| Embedding | A dense vector that represents something (a word, a user, a product) so that similar things sit close together. |
| Ensemble | Combining several models so their errors cancel. Bagging, boosting, stacking, voting. |
| Epoch | One full pass over the entire training set. |
| Exploding gradient | Gradients grow enormous through deep layers and training blows up. Fix with gradient clipping. |
F – H
| F1 score | Harmonic mean of precision and recall. High only when both are high. |
| False negative | Missed a real positive. The expensive error in cancer screening and fraud detection. |
| False positive | Flagged something that was fine. The expensive error in spam filtering and criminal justice. |
| Feature | One input column. Also called an attribute, predictor, or independent variable. |
| Feature engineering | Creating better inputs from raw data. Usually worth more than switching algorithms. |
| Feature importance | How much each feature contributed. Beware: tree-based impurity importance is biased toward high-cardinality features — prefer permutation importance or SHAP. |
| Federated learning | Train across many devices without the raw data ever leaving them. Only model updates travel. |
| Fold | One of the k slices in cross-validation. |
| Generalisation | Performing well on data never seen during training. The entire point of the field. |
| Gradient | The vector of slopes — which way is uphill, and how steeply. |
| Gradient boosting | Sequentially fit new trees to the residual errors of the current ensemble. XGBoost, LightGBM, CatBoost. The reigning champion on tabular data. |
| Gradient descent | Repeatedly step downhill on the loss surface. The core training loop of most of ML. |
| Grid search | Try every combination of hyperparameters. Thorough, and exponentially expensive. |
| Ground truth | The actual correct answer, as recorded in your labels. |
| Hyperparameter | A setting you choose before training (k, learning rate, tree depth). Contrast with parameters, which the model learns. |
| Hinglish note | Parameter woh hai jo model khud seekhta hai; hyperparameter woh hai jo aap pehle set karte ho. Weights are learned, learning rate is chosen. |
I – L
| Imputation | Filling in missing values — with the median, the mode, or a model. Always fit the imputer on training data only. |
| Inertia | K-Means' objective: total squared distance from each point to its centroid. Always falls as k rises, which is why the elbow method exists. |
| Inference | Using a trained model to make predictions. The production phase. |
| Information gain | How much a split reduces entropy. The tree picks the split that maximises it. |
| Irreducible error | Noise you can never model away. The floor beneath every model's performance. |
| Kernel trick | Computing dot products in a high-dimensional space without ever constructing that space. Gives SVMs curved boundaries cheaply. |
| KNN | K-Nearest Neighbours. Predict by asking the k closest training points. No training phase; all the cost is at prediction time. |
| Label | The answer you are trying to predict. Also called the target or dependent variable. |
| Lasso | L1-regularised regression. Shrinks some coefficients to exactly zero, performing feature selection. |
| Latent variable | A hidden factor that was never measured but explains the observations — "taste for sci-fi" in a recommender. |
| Learning curve | Performance plotted against training set size. Tells you whether more data would help. |
| Learning rate | Step size in gradient descent. Too big and you overshoot; too small and you never arrive. |
| Linear separability | Whether a straight line (or flat plane) can cleanly divide the classes. XOR famously is not. |
| Logistic regression | Despite the name, a classifier. A linear model passed through a sigmoid to yield probabilities. |
| Loss function | The number that measures how wrong the model currently is. Training minimises it. |
| LSTM | Long Short-Term Memory. An RNN with gates that let it carry information across long sequences without the gradient vanishing. |
M – O
| MAE | Mean Absolute Error. Average size of the mistake, robust to outliers. |
| Matrix factorisation | Splitting a big sparse user×item matrix into two small dense ones. The classic recommender engine. |
| MLOps | The engineering discipline of getting models into production and keeping them healthy there. |
| Model card | A short document stating what a model does, on what data it was trained, how it performs per group, and where it should not be used. |
| Momentum | Letting the optimiser carry velocity from previous steps, so it rolls through small bumps and flat regions. |
| MSE | Mean Squared Error. Punishes large errors disproportionately; the default regression loss. |
| Multicollinearity | Two or more features carrying nearly the same information. Coefficients become unstable and uninterpretable; Ridge helps. |
| Naive Bayes | A probabilistic classifier assuming all features are independent given the class. The assumption is false; the classifier works anyway, especially on text. |
| Nested CV | Cross-validation inside cross-validation: an inner loop tunes hyperparameters, an outer loop measures performance honestly. |
| Neural network | Layers of neurons, each doing a dot product then a non-linearity. Universal function approximators given enough width. |
| Normalisation | Rescaling features to a common range. Min-max to [0, 1], or standardisation to mean 0, std 1. |
| One-hot encoding | Turning a category into a set of 0/1 columns, one per level. Prevents the model inventing a false ordering. |
| Ordinal encoding | Mapping categories to integers. Only valid when a real order exists (small < medium < large). |
| Outlier | A point far from the rest. Sometimes an error to remove, sometimes the exact thing you are trying to detect. |
| Overfitting | Memorising the training data, including its noise. Training score high, test score poor. |
| Hinglish note | Overfitting matlab ratta maar liya, samajh nahi aaya. The student who memorised last year's paper and fails when the questions change. |
P – R
| Parameter | A value the model learns from data — weights and biases. |
| PCA | Principal Component Analysis. Rotates the data so the first axis captures the most variance, then lets you drop the later axes. |
| Pipeline | Chaining preprocessing and model into one object, so every transform is fitted on training folds only. The main defence against leakage. |
| Precision | Of what I flagged, what fraction was actually positive. |
| PR-AUC | Area under the precision–recall curve. The right summary metric when the positive class is rare. |
| Pruning | Cutting back a grown decision tree to reduce overfitting. |
| PSI | Population Stability Index. A drift score comparing production feature distributions against training. Above 0.25, investigate. |
| Random Forest | Many decorrelated decision trees, each on a bootstrap sample with a random feature subset at each split, then averaged. |
| Random search | Sample hyperparameter combinations at random. Usually beats grid search per unit of compute. |
| Recall | Of all real positives, what fraction did I catch. Also called sensitivity or true positive rate. |
| Regression | Predicting a continuous number. |
| Regularisation | Any deliberate constraint that trades a little training accuracy for better generalisation. L1, L2, dropout, early stopping. |
| Reinforcement learning | Learning by trial, error, and reward rather than from labelled examples. |
| ReLU | max(0, x). The default hidden-layer activation: cheap, and its gradient does not vanish for positive inputs. |
| Residual | Actual minus predicted, for one example. Plotting residuals is the fastest way to spot a bad regression. |
| Ridge | L2-regularised regression. Shrinks coefficients toward zero without eliminating them. |
| RNN | Recurrent Neural Network. Processes sequences by carrying a hidden state forward through time. |
| ROC curve | True positive rate against false positive rate as the threshold sweeps. AUC summarises it. |
| R² | Fraction of the target's variance explained. 1.0 is perfect, 0 is no better than predicting the mean, negative is worse than the mean. |
S – Z
| Semi-supervised | Learning from a small labelled set plus a large unlabelled one. |
| SHAP | Shapley Additive exPlanations. Fairly divides a prediction's credit among the features, with solid game-theoretic grounding. |
| Sigmoid | S-shaped squash into (0, 1). Used for binary output probabilities; avoid in hidden layers because its gradient vanishes. |
| Silhouette score | Clustering quality from −1 to +1: how much closer a point is to its own cluster than to the nearest other one. |
| SMOTE | Synthetic Minority Over-sampling. Creates new minority examples by interpolating between real ones. Apply inside the CV fold, never before splitting. |
| Softmax | Turns a vector of scores into probabilities that sum to 1. The multi-class output layer. |
| Stacking | Train a meta-model on the predictions of several base models. |
| Standardisation | (x − μ) / σ. Mean 0, standard deviation 1. |
| Stratified split | Splitting so each fold preserves the original class proportions. Essential when imbalanced. |
| Supervised learning | Learning from labelled examples — you have the answer key. |
| SVM | Support Vector Machine. Finds the boundary with the widest margin; kernels let it curve. |
| Target encoding | Replacing a category with the mean target for that category. Powerful, and a leakage hazard unless done fold-wise. |
| Test set | Data touched exactly once, at the very end, to estimate real-world performance. Every extra peek inflates the estimate. |
| TF-IDF | Term Frequency × Inverse Document Frequency. Weights a word by how often it appears here and how rare it is overall. |
| Threshold | The probability cutoff for declaring a positive. Default 0.5 is a convention, not a law — tune it to your cost tradeoff. |
| Transfer learning | Starting from a model trained on a huge dataset and fine-tuning on your small one. The reason modern vision and NLP work with limited data. |
| Transformer | Architecture built on self-attention rather than recurrence. Processes sequences in parallel; underlies modern language models. |
| Underfitting | Model too simple to capture the pattern. Poor on training and test data. |
| Unsupervised learning | Finding structure with no labels at all. |
| Validation set | Held-out data used to tune hyperparameters and choose models. Distinct from the test set. |
| Vanishing gradient | Gradients shrink toward zero through deep layers, so early layers stop learning. Fixed by ReLU, residual connections, and careful initialisation. |
| Variance (error) | How much the model would change if trained on a different sample. High variance = overfitting. |
| Weight | A learned coefficient controlling how strongly an input influences the output. |
| Word embedding | Dense vector for a word, learned so that similar words land near each other. |
| XGBoost | A fast, regularised gradient boosting implementation. Still a default first choice for tabular problems. |
| Zero-shot | Performing a task the model was never explicitly trained on, using only a description of it. |
Interview Question Bank
Sixty questions that actually get asked, with answers written the way you should say them — short, confident, and with the "why" attached. Cover the answer, attempt it aloud, then compare.
Do not memorise these word for word — interviewers can hear a recited answer instantly. Instead, get the shape right: state the direct answer in one sentence, give the reason, then offer a concrete example. Almost every strong ML answer follows that three-beat pattern. If you can also name the tradeoff you are accepting, you sound like someone who has shipped a model rather than someone who has read about one.
D.1 Fundamentals (Q1–Q12)
Q1. What is machine learning, in one sentence?
Writing programs that improve at a task by learning patterns from data, instead of being given explicit rules. The classic test: if you cannot easily write down the rules yourself — "is this email spam?" — that is an ML problem.
Q2. Supervised vs unsupervised vs reinforcement learning?
Supervised has labelled answers and learns the mapping from input to output. Unsupervised has no labels and finds structure. Reinforcement learns from reward signals through trial and error. Spam filtering, customer segmentation, and game-playing agents respectively.
Q3. Parameter vs hyperparameter?
A parameter is learned by the model during training — the weights. A hyperparameter is set by me before training — learning rate, tree depth, k. Parameters come from data; hyperparameters come from search over validation performance.
Q4. Explain the bias–variance tradeoff.
Total error decomposes into bias squared, variance, and irreducible noise. Bias is error from the model being too simple; variance is how much it would change on a different training sample. Making the model more flexible lowers bias and raises variance. The goal is the sum's minimum, not either extreme.
Q5. How do you detect overfitting?
A large gap between training and validation performance — excellent on train, mediocre on validation. A learning curve confirms it: if the two curves stay far apart even as data grows, it is variance, and more data will help. If they converge but both sit at a poor level, it is bias, and more data will not.
Q6. Name five ways to reduce overfitting.
More training data; regularisation (L1/L2/dropout); reducing model complexity (shallower trees, fewer features); early stopping on a validation set; and ensembling, particularly bagging. Cross-validation does not itself reduce overfitting — it only reveals it.
Q7. What is data leakage and how do you prevent it?
Information from the test set or from the future entering training, so validation scores look great and production fails. Classic causes: scaling or imputing before splitting, target encoding computed on the full dataset, and features that could only be known after the event. Prevention is a scikit-learn Pipeline fitted inside each CV fold, plus asking of every feature, "would I actually have this at prediction time?"
Q8. Why split into train, validation, and test rather than just train and test?
Because every hyperparameter decision made against a set contaminates it as an unbiased estimate. Validation is for choosing; test is for reporting, opened once. If I tune on the test set, I am overfitting to it and my reported number is optimistic.
Q9. Which algorithms need feature scaling and which do not?
Anything based on distances or gradients needs it: KNN, K-Means, SVM, PCA, neural networks, and all regularised linear models. Tree-based methods do not, because splits depend on the ordering of values, not their magnitude — rescaling produces identical trees.
Q10. Standardisation or normalisation — how do you choose?
Standardisation, (x−μ)/σ, is the safer default and handles outliers better. Min-max normalisation to [0,1] is useful when a bounded range is required, such as image pixels, but a single extreme value squashes everything else. With heavy outliers I would use RobustScaler, which uses the median and IQR.
Q11. How do you handle missing values?
First ask why they are missing, because missingness is often informative — I usually add a binary "was missing" indicator. Then impute: median for numeric, most-frequent or an explicit "Unknown" category for categorical, and iterative or KNN imputation when relationships are strong. Critically, fit the imputer on training data only, inside the pipeline.
Q12. What is the curse of dimensionality?
As dimensions grow, data becomes sparse and all pairwise distances converge toward being equally large, so "nearest neighbour" stops being meaningful. It also means the data needed for reliable estimates grows exponentially. Remedies are feature selection, dimensionality reduction such as PCA, and choosing algorithms that are less distance-dependent.
D.2 Regression and linear models (Q13–Q22)
Q13. What assumptions does linear regression make?
Linearity in the parameters, independent errors, constant error variance (homoscedasticity), approximately normal residuals for inference, and no severe multicollinearity. Prediction still works when these bend; it is confidence intervals and p-values that break first.
Q14. Ridge vs Lasso — when would you use each?
Ridge (L2) shrinks all coefficients smoothly and keeps every feature; it is the right choice when features are correlated and I believe most of them matter a little. Lasso (L1) drives some coefficients to exactly zero, so it performs feature selection; it is right when I suspect only a handful of features matter. Elastic Net blends both and is the pragmatic default with many correlated features.
Q15. Why does L1 produce exact zeros while L2 does not?
Geometrically, the L1 constraint region is a diamond with sharp corners lying on the axes, and the loss contours are very likely to first touch it at a corner — where a coefficient is exactly zero. The L2 region is a circle with no corners, so contact almost always occurs at a point where every coefficient is small but non-zero.
Q16. R² is 0.95. Is the model good?
Not necessarily. R² can be high while residuals show clear structure, it rises mechanically as features are added (use adjusted R²), and a suspiciously high value on business data often signals leakage. I would check residual plots, the test-set R², and whether any feature encodes the target.
Q17. MSE or MAE?
MSE when large errors are disproportionately costly and I want the model to prioritise them; it is also differentiable everywhere, which suits gradient methods. MAE when outliers are present and I want a robust, evenly-weighted average error. MSE optimises toward the conditional mean, MAE toward the median — that difference matters on skewed targets.
Q18. Gradient descent or the normal equation?
The normal equation gives the exact answer in one step but requires inverting a d×d matrix at O(d³) cost, so it is fine below roughly ten thousand features and impractical above. Gradient descent is iterative and approximate but scales to enormous feature counts and huge datasets, and it is the only option once the model is non-linear.
Q19. Batch, stochastic, or mini-batch gradient descent?
Batch uses all data per update — stable but slow and memory-hungry. Stochastic uses one example — fast and noisy, and the noise can help escape shallow minima. Mini-batch, typically 32 to 256 examples, is the practical compromise and what essentially everyone uses, because it is both reasonably stable and GPU-efficient.
Q20. Your learning rate is too high. What do you observe?
Loss oscillating rather than descending, or diverging to NaN. Too low, and the loss falls but agonisingly slowly and may stall in a plateau. I would plot loss against iteration, try a log-spaced sweep, and use a scheduler or Adam to reduce the sensitivity.
Q21. What is multicollinearity and does it matter?
Two or more features carrying nearly the same information. It does not usually hurt predictive accuracy, but it makes individual coefficients unstable and their signs unreliable, so any interpretation is worthless. Detect with VIF or a correlation matrix; fix by dropping one, combining them, or using Ridge.
Q22. How would you model a non-linear relationship with a linear model?
Add engineered features: polynomial terms, interactions, splines, or a log transform of a skewed variable. The model stays linear in its parameters, which is all "linear model" actually requires, while the fitted curve becomes non-linear in the inputs. Polynomial degree then becomes a hyperparameter to cross-validate, since high degrees overfit fast.
D.3 Classification and evaluation (Q23–Q36)
Q23. Why is it called logistic regression if it classifies?
Because it regresses the log-odds. log(p/(1−p)) is a linear function of the inputs; the sigmoid simply maps that back into a probability. The regression is on the logit scale, and the classification is what we do with the resulting probability.
Q24. Why not use MSE as the loss for logistic regression?
Two reasons. MSE combined with a sigmoid gives a non-convex loss surface with local minima, so optimisation is unreliable. And MSE's gradient shrinks exactly when the model is confidently wrong, which is when you most want a large correction. Cross-entropy is convex here and produces a strong gradient for confident mistakes.
Q25. Explain precision and recall to a non-technical stakeholder.
Precision: "when we raise an alarm, how often is it real?" Recall: "of all the real cases out there, how many alarms did we raise?" Then anchor it to their cost: for fraud, a missed case costs money, so recall matters; for a spam filter, blocking a real customer email is worse, so precision matters.
Q26. 99% of transactions are legitimate. My model has 99% accuracy. Comment.
It may be predicting "legitimate" for everything and catching zero fraud. Accuracy is meaningless at this imbalance. I would look at the confusion matrix, recall on the fraud class, and PR-AUC, and I would set the threshold from the business cost of a missed fraud versus a false alarm rather than leaving it at 0.5.
Q27. ROC-AUC or PR-AUC?
ROC-AUC when classes are roughly balanced or when both errors matter symmetrically. PR-AUC when positives are rare, because ROC's false positive rate has a huge denominator of true negatives and therefore barely moves — a model can look excellent on ROC while its actual precision is terrible. On a 1%-positive problem, PR-AUC is the honest metric.
Q28. What does an ROC-AUC of 0.5 mean? Of 0.3?
0.5 is random ranking — no discriminative signal. 0.3 is worse than random, which is actually informative: it usually means the labels or the predicted probabilities are inverted somewhere in the code. A genuinely useless model sits at 0.5, not below it.
Q29. How do you choose the classification threshold?
From costs, not from convention. I estimate the cost of a false positive and a false negative, sweep the threshold over the validation set, and pick the point that minimises expected cost or hits a required recall. 0.5 is only optimal when the two errors cost the same and the model is calibrated.
Q30. What is model calibration and when does it matter?
Calibration means predicted probabilities match observed frequencies — among the cases scored 0.7, about 70% should be positive. It matters whenever the probability itself is used downstream: expected-value pricing, risk scores, medical decisions. Tree ensembles and SVMs are often poorly calibrated; CalibratedClassifierCV with Platt scaling or isotonic regression fixes it.
Q31. Why is Naive Bayes "naive", and why does it still work?
It assumes every feature is conditionally independent given the class, which is plainly false for text — "New" and "York" are not independent. It works because for classification we only need the correct class to score highest, not the probabilities to be accurate. The estimates are badly calibrated but the ranking survives.
Q32. How does a decision tree decide where to split?
It evaluates candidate splits on every feature and picks the one that most reduces impurity, measured by Gini or entropy. Information gain is parent impurity minus the weighted average of the children's. It repeats greedily until a stopping rule fires — which is why a single tree is greedy and locally optimal rather than globally optimal.
Q33. Gini or entropy?
In practice they produce nearly identical trees. Gini is slightly cheaper because it avoids logarithms, which is why it is scikit-learn's default. This is not a hyperparameter worth spending a tuning budget on; depth and minimum samples per leaf matter far more.
Q34. Bagging vs boosting?
Bagging trains models independently in parallel on bootstrap samples and averages them; it attacks variance and is hard to overfit by adding trees. Boosting trains sequentially, each model correcting the last one's errors; it attacks bias, usually reaches higher accuracy, and can overfit with too many rounds, so it needs early stopping.
Q35. What makes Random Forest better than a single deep tree?
Decorrelation. Each tree sees a bootstrap sample and, crucially, only a random subset of features at each split, so the trees make different mistakes. Averaging cancels those independent errors while preserving the shared signal. Without the feature subsampling the trees would be too similar and averaging would gain far less.
Q36. Explain the kernel trick.
SVMs only ever need dot products between examples. A kernel computes what the dot product would be in a much higher-dimensional space, without ever building the coordinates in that space. So we get a curved decision boundary at roughly the cost of a linear one — the RBF kernel corresponds to an infinite-dimensional space we never materialise.
D.4 Unsupervised learning (Q37–Q43)
Q37. How does K-Means work, and what are its weaknesses?
Place k centroids, assign each point to the nearest, move each centroid to its cluster's mean, repeat until stable. Weaknesses: you must pick k in advance; it assumes roughly spherical, similar-sized clusters; it is sensitive to initialisation (hence k-means++) and to feature scale; and it forces every point into a cluster, including outliers.
Q38. How do you choose k?
The elbow on the inertia curve, the silhouette score, and gap statistic are the technical tools — but the honest answer is that domain meaning usually decides. If the business can act on four customer segments and not seventeen, k is four. I would present the metrics and then reconcile them with what is operationally usable.
Q39. When would you prefer DBSCAN over K-Means?
When clusters are non-spherical or of very different densities, when I do not know k, and when I want outliers explicitly labelled as noise rather than forced into a group. The cost is two less-intuitive hyperparameters, eps and min_samples, and poor behaviour when cluster densities vary a lot.
Q40. What does PCA actually do?
It finds the orthogonal directions of greatest variance — the eigenvectors of the covariance matrix — and re-expresses the data along them. Keeping the first few components retains most of the variance in far fewer dimensions. It must be run on standardised data, otherwise the largest-scaled feature dominates the first component.
Q41. What is lost when you use PCA?
Interpretability, primarily — components are linear blends of all original features, so "PC1" has no business meaning. You also lose whatever low-variance information you discarded, and low variance does not guarantee low importance for the target. PCA is unsupervised, so it can throw away exactly the direction that predicts y.
Q42. PCA vs t-SNE?
PCA is linear, deterministic, fast, and preserves global structure, and it can transform new data. t-SNE is non-linear, stochastic, slow, and preserves local neighbourhoods at the expense of global distances, so cluster sizes and inter-cluster gaps in a t-SNE plot are not meaningful. PCA is for preprocessing; t-SNE and UMAP are for visualisation only.
Q43. How do you evaluate clustering when there are no labels?
Internal metrics such as silhouette, Davies–Bouldin, and Calinski–Harabasz measure cohesion and separation. Stability checks — do the clusters survive resampling? — are underrated. Ultimately the real test is external: does acting on these clusters produce a better business outcome than not doing so?
D.5 Neural networks and deep learning (Q44–Q52)
Q44. Why do neural networks need non-linear activation functions?
Because a composition of linear functions is still a linear function. Without non-linearity, a hundred-layer network is mathematically equivalent to a single linear layer and cannot represent anything a linear model could not. The activation is what makes depth meaningful.
Q45. Explain backpropagation in plain terms.
It is the chain rule applied backwards. After a forward pass computes the loss, we work from the output layer back, at each step asking how much this layer's output affected the loss and how much each weight affected that output, multiplying the two. Every weight ends up with a gradient, and the optimiser then updates them. It computes gradients; it does not itself learn.
Q46. What causes vanishing gradients and how do you fix them?
Sigmoid and tanh have derivatives at most 0.25 and 1, so multiplying many of them through deep layers drives the gradient toward zero and early layers stop learning. Fixes: ReLU-family activations, He initialisation, batch normalisation, and residual (skip) connections that give gradients a direct path backwards.
Q47. Why not initialise all weights to zero?
Symmetry. Every neuron in a layer would compute the same output and receive the same gradient, so they would remain identical forever — the layer would have the capacity of a single neuron. Random initialisation breaks the symmetry; He or Xavier scaling keeps the variance sensible as signals propagate.
Q48. Why is Adam usually the default optimiser?
It combines momentum, which smooths the trajectory, with per-parameter adaptive step sizes, which handle features and layers that need very different scales. It converges quickly with little tuning. Plain SGD with momentum sometimes generalises slightly better on vision tasks, but Adam is the safer starting point, and AdamW fixes its weight-decay behaviour.
Q49. What makes CNNs suited to images?
Three properties. Local connectivity, because nearby pixels are related; parameter sharing, because an edge detector is useful anywhere in the image, which cuts parameters dramatically; and translation invariance via pooling, so a cat is recognised wherever it appears. A fully connected net on the same image would have orders of magnitude more parameters and no built-in notion of spatial structure.
Q50. Why did transformers displace RNNs?
RNNs process tokens one at a time, so they cannot parallelise across the sequence and they struggle to carry information over long distances. Self-attention lets every position look at every other in a single step, giving constant path length between any two tokens and full parallelism on GPUs. That combination is what made training on internet-scale text feasible.
Q51. What is transfer learning and when is it worth it?
Taking a model trained on a large general dataset and adapting it to a smaller specific one, usually by freezing early layers and retraining the head. It is worth it almost any time the domains are related and my dataset is small — a few thousand images with a pretrained backbone will beat a from-scratch network trained on the same data by a wide margin.
Q52. Your deep network's training loss falls but validation loss rises. What now?
Classic overfitting. In order: add early stopping, increase dropout or weight decay, augment the data, reduce model size, and check for a train/validation distribution mismatch. If validation loss rises while validation accuracy also rises, that is often just growing overconfidence on already-correct predictions, which matters less.
D.6 Applied ML, deployment and judgement (Q53–Q60)
Q53. Walk me through how you would approach a new ML problem.
Define the business outcome and the metric that tracks it before touching data. Establish a trivial baseline — predict the majority class or the mean. Explore the data and check for leakage. Build the simplest reasonable model in a pipeline with proper cross-validation. Only then iterate on features, and only after that on model complexity. Test set opened once, at the end.
Q54. How do you handle a severely imbalanced dataset?
Start with the cheapest lever: class_weight="balanced" and threshold tuning, evaluated on PR-AUC with stratified folds. If that is insufficient, try resampling — SMOTE or undersampling — strictly inside the CV fold so the validation data stays untouched. And I would question whether more minority data can be collected, which usually beats all of it.
Q55. Why can't you use standard k-fold cross-validation on time series?
Because random folds let the model train on the future and test on the past, which leaks information and produces wildly optimistic scores. I use TimeSeriesSplit — expanding or rolling windows where every test fold is strictly after its training data — and I make sure lag and rolling features never reach across the split boundary.
Q56. How do you convert text into model input?
For a fast strong baseline, TF-IDF over word or character n-grams into logistic regression — on many classification tasks this is within a few points of far heavier models. If more is needed, pretrained embeddings, or fine-tuning a transformer. The choice depends on data volume, latency budget, and how much the extra accuracy is actually worth.
Q57. What is the cold-start problem in recommenders?
Collaborative filtering needs interaction history, so it fails for a brand-new user or item that has none. Mitigations: fall back to content-based features (genre, category, text), use popularity or demographic defaults initially, ask a few onboarding preference questions, and switch to collaborative signals as history accumulates — a hybrid system.
Q58. Your model performed well at launch and is degrading. Diagnose.
I would separate the causes. Data drift — inputs have shifted, check PSI or KS per feature. Concept drift — the input-output relationship itself changed, visible as falling accuracy with stable inputs. Then the unglamorous ones: an upstream pipeline change, a feature silently going null, a training/serving skew where production computes a feature differently. I would check the boring engineering causes first, because they are the most common.
Q59. How do you deploy a new model safely?
Shadow mode first, where the new model scores live traffic but its outputs are logged rather than used, so I can compare against the incumbent on real data. Then a canary to a small traffic percentage with monitoring and an automatic rollback trigger. Then a proper A/B test measuring the business metric, not just the offline one. Blue-green if instant full rollback matters more than gradual exposure.
Q60. Your model is accurate but performs worse for one demographic group. What do you do?
First measure it properly — per-group confusion matrices and metrics, with confidence intervals, since small groups produce noisy estimates. Then trace the cause: under-representation in training data, a proxy feature encoding group membership, or historically biased labels. Remedies range from collecting better data to reweighting to group-specific thresholds. I would also be explicit that demographic parity, equal opportunity, and calibration cannot generally all hold at once, so this is a documented decision with stakeholders, not a purely technical fix — and it belongs in a model card.
Asked what you would do first, say "establish a baseline" — it signals discipline. Asked about any metric, tie it to a business cost rather than reciting the formula. And when you genuinely do not know, say so and then say how you would find out; interviewers trust that far more than a confident guess. The strongest candidates are not the ones who know the most algorithms — they are the ones who reliably ask what problem is actually being solved.
Where To Go From Here
You have the map. The territory is walked by building things.
If you have read this far, you now know more machine learning than most people who claim it on a CV. What separates knowing from being able is roughly a dozen projects, each finished badly and then improved.
A realistic ninety-day plan
Weeks 1–3, foundations in the hands. Re-read Units 1–3 with a notebook open beside them. Take one small tabular dataset and take it all the way: load, explore, clean, split, build a pipeline, fit a linear model, read the residuals, write down what surprised you. Do it a second time with a different dataset without looking at the first notebook.
Weeks 4–6, classification and the metrics that matter. Work Units 4–6. Build a classifier on an imbalanced dataset and force yourself to justify the threshold in terms of cost. Compare logistic regression, a random forest, and gradient boosting on the same problem with identical folds — and notice how often the simple model is close enough.
Weeks 7–9, breadth. Units 7–9. Train a small neural network from scratch in NumPy once, because doing it by hand permanently demystifies backpropagation. Then fine-tune a pretrained image model, and build a text classifier. Compare TF-IDF plus logistic regression against a transformer and observe the accuracy-per-rupee.
Weeks 10–12, make something real. Unit 10. Take your best model, wrap it in a FastAPI endpoint, log its inputs and predictions, write a drift check, and produce a model card. A deployed mediocre model teaches more than an undeployed excellent one.
Always build the stupid baseline first — you cannot claim improvement without it. Always ask "could I know this at prediction time?" of every feature. Always look at the residuals or the misclassified examples by hand, at least twenty of them; that single habit finds more bugs than any metric. Write down what you expected before you run the experiment, because a prediction you got wrong teaches you something and a result you rationalise afterwards does not.
Machine learning ka sabse bada raaz yeh hai: zyada tar kaam algorithm ka nahi hota. Data samajhna, sahi sawaal poochhna, leakage pakadna, aur sahi metric chunna — asli kaam yahi hai. Fancy model toh last 10% hai. Jo log yeh baat jaldi samajh lete hain, woh doosron se bahut aage nikal jaate hain.
Toh ab kitaab band kijiye aur ek project shuru kijiye. Pehla model kharaab hoga. Yeh bilkul theek hai — wahi se seekhna shuru hota hai.
Good luck. Build something.
Standard References & Authoritative Textbooks
To deepen your theoretical intuition or explore rigorous proofs beyond the GATE DA exam syllabus, consult these primary textbooks and seminal papers:
| Reference | Authors & Edition | Key Topics Covered in This Guide |
|---|---|---|
| The Elements of Statistical Learning | Trevor Hastie, Robert Tibshirani, Jerome Friedman (Springer, 2nd ed.) | Supervised learning theory, bias-variance decomposition, L1 Lasso and L2 Ridge shrinkage, Random Forests, AdaBoost, and Gradient Tree Boosting. |
| Pattern Recognition and Machine Learning | Christopher M. Bishop (Springer, 2006) | Bayesian linear regression, Gaussian Mixture Models (GMM), Expectation-Maximization (EM) algorithm, and Support Vector Machines with Mercer kernels. |
| Deep Learning | Ian Goodfellow, Yoshua Bengio, Aaron Courville (MIT Press, 2016) | Computational graphs, backpropagation calculus, convolutional neural networks, and modern stochastic gradient descent optimizers (Adam, RMSprop). |
| Mathematics for Machine Learning | Marc Peter Deisenroth, A. Aldo Faisal, Cheng Soon Ong (Cambridge Univ. Press, 2020) | Vector calculus, spectral decomposition, eigenvalues/eigenvectors, Principal Component Analysis derivation, and Lagrange multipliers. |
| Attention Is All You Need | Ashish Vaswani, Noam Shazeer, Niki Parmar, et al. (NeurIPS, 2017) | Scaled Dot-Product Attention, Multi-Head Attention mechanisms, and Transformer encoder-decoder sequence modeling. |
Continue learning
Solidify your understanding through practical problem-solving: practise GATE Machine Learning questions, take a timed GATE DA full-length mock test, or explore topic-specific deep dives — probability, statistics, linear algebra, machine learning, and deep learning. The complete formula sheet is available in Appendix B.
Last updated: 26 October 2026 · Maintained by Gaurav Tiwari · See about and contact.