M O D U L E 3 Statistics & Exploratory Data Analysis Introduction to Data Science • B.E./B.Tech — 5th Semester Dr. APJ Abdul Kalam School of Engineering | Garden City University, Bengaluru W H AT T H I S M O D U L E C O V E R S Module 3 Roadmap Introduction to Statistics Why statistics underpins every data- science decision Descriptive Statistics Summarising a dataset with a few honest numbers Central Tendency Mean, median and mode — the typical value Dispersion Variance and standard deviation — the spread Basics of Probability Quantifying uncertainty and likelihood Correlation & Covariance How two variables move together Exploratory Data Analysis Interrogating data before modelling Patterns & Trends Reading structure, seasonality and drift 02 F O U N D AT I O N S What is Statistics? Statistics is the science of collecting, organising, summarising and interpreting data so that sound conclusions can be drawn in the presence of variability and uncertainty. In data science it is the layer between raw records and reliable insight: before any model is trained, statistics tells us what the data actually says. Descriptive Statistics Summarises and describes the data you already have. Reports the typical value, the spread and the shape of a dataset without claiming anything beyond it. Example: the average marks of one class. Inferential Statistics Uses a sample to draw conclusions about a larger population it cannot fully observe. Relies on probability to state how confident those conclusions are. Example: predicting all students from a survey of 200. Why it matters in Data Science • Detects errors, outliers and bias before modelling • Turns millions of rows into a few meaningful numbers • Chooses the right features and transformations • Quantifies how much to trust a result 03 F O U N D AT I O N S Population, Sample & Variables Population The complete set of all items under study — every student at GCU, every Swiggy order in a city. Often too large to measure fully. Sample A manageable subset drawn from the population. Good samples are representative, so conclusions generalise back to the whole. Parameter vs Statistic A parameter describes a population (true mean μ). A statistic estimates it from a sample (sample mean ). We compute statistics to infer x̄ parameters. Types of Variables Categorical (Qualitative) Labels or groups with no numeric meaning. e.g. Gender, city, product category Numerical — Discrete Countable whole numbers. e.g. No. of orders, students present Numerical — Continuous Any value in a range, measured. e.g. Height, temperature, delivery time 04 S E C T I O N 0 1 Descriptive Statistics Summarising a dataset with a few honest numbers. D E S C R I P T I V E S TAT I S T I C S Describing a Dataset A descriptive summary answers three questions about a column of numbers: where is the centre, how spread out are the values, and what is the overall shape. Together these let us characterise thousands of records with a handful of trustworthy figures. Central Tendency The single most typical value. Mean, Median, Mode. Dispersion How much values scatter. Range, Variance, Std. Dev. Shape Symmetry and peakedness. Skewness, Kurtosis. 06 C E N T R A L T E N D E N C Y Mean, Median & Mode Mean (Average) Sum of all values divided by the count. Uses every data point, but a few extreme values drag it sharply. = ( x ) / n μ Σ ᵢ Median (Middle) The middle value once data is sorted. Splits the set in half and is robust to outliers and skew. Middle of sorted data Mode (Most frequent) The value that occurs most often. The only measure that works for categorical data; a set may have none or several. Most repeated value 07 W O R K E D E X A M P L E Central Tendency in Action Scenario: Swiggy delivery times (minutes) recorded for 9 orders in Koramangala. 22 25 25 28 30 31 34 38 62 (sorted; 62 min is an outlier — a rainy-day delay) Mean (22+25+25+28+30+31+34+38+62) / 9 = 295 / 9 ≈ 32.8 min Median 5th value of 9 sorted = 30 min Mode value appearing twice = 25 min Insight: the single 62-minute order pulls the mean up to 32.8, but the median stays at 30. When outliers exist, the median is the more honest “typical” value. 08 C E N T R A L T E N D E N C Y Which Measure Should You Trust? Use the Mean when Data is roughly symmetric with no extreme outliers — e.g. exam scores in a well-set test. Use the Median when Data is skewed or has outliers — PG/hostel rents, incomes, delivery times. Use the Mode when Data is categorical or you need the most common category — favourite payment method. Right-skewed data: mean > median 10 20 30 40 50 60 70 80 0 5 10 15 20 25 Value 09 S E C T I O N 0 2 Measures of Dispersion How spread out the values are around the centre. M E A S U R E S O F D I S P E R S I O N Why the Centre Is Not Enough Two datasets can share the same mean yet behave completely differently. Dispersion measures how far values sit from the centre — it is the difference between a reliable process and an erratic one. Restaurant A — consistent Mean = 30 min in both cases 28 29 30 31 32 Values hug the mean → low spread → dependable. Restaurant B — erratic Mean = 30 min in both cases 12 22 30 38 48 Same mean, but values scatter widely → unpredictable service. 11 M E A S U R E S O F D I S P E R S I O N Range, Variance & Standard Deviation Range The gap between the largest and smallest value. Quick to compute but driven entirely by the two extremes. Range = max − min Variance (σ²) The average of the squared distances from the mean. Squaring stops positive and negative gaps cancelling, but the unit becomes squared. ² = (x − )² / n σ Σ ᵢ μ Standard Deviation (σ) The square root of variance, back in the original unit. The everyday measure of spread — “on average, how far from the mean.” = √(variance) σ Note: for a sample we divide by (n − 1) instead of n (Bessel’s correction) to avoid underestimating the population spread. 12 W O R K E D E X A M P L E Computing Variance & Std. Deviation Marks of 5 students (out of 50): 36, 40, 42, 44, 48 → Mean μ = 210 / 5 = 42 x ᵢ x − ᵢ μ (x − )² ᵢ μ 36 −6 36 40 −2 4 42 0 0 44 +2 4 48 +6 36 (x − )² = Σ ᵢ μ 80 Result Variance σ² = 80 / 5 = 16 Std. Deviation σ = √16 = 4 marks Interpretation: student marks lie, on average, about 4 marks away from the class mean of 42 — a fairly tight, consistent set of scores. 13 S E C T I O N 0 3 Basics of Probability The mathematical language of chance and uncertainty. B A S I C S O F P R O B A B I L I T Y Quantifying Uncertainty Probability measures how likely an event is, on a scale from 0 (impossible) to 1 (certain). It is the mathematical language of chance and the foundation on which inferential statistics and machine learning rest. P(event) = ( Number of favourable outcomes ) / ( Total number of possible outcomes ) Experiment & Outcome An action with uncertain results (rolling a die) and each possible result (1–6). Sample Space (S) The set of all possible outcomes. For one die, S = {1,2,3,4,5,6}. Event Any subset of outcomes we care about, e.g. “roll an even number” = {2,4,6}. Range of P Always 0 ≤ P ≤ 1. All outcome probabilities in S sum to exactly 1. 15 B A S I C S O F P R O B A B I L I T Y Key Rules & a Worked Example Complement P(not A) = 1 − P(A) Addition (mutually exclusive) P(A or B) = P(A) + P(B) Multiplication (independent) P(A and B) = P(A) × P(B) Conditional P(A | B) = P(A and B) / P(B) Example — a deck of cards Draw one card from 52. P(King) = 4/52 = 1/13 ≈ 0.077 P(Heart) = 13/52 = 1/4 = 0.25 P(King or Heart) = 4/52 + 13/52 − 1/52 = 16/52 ≈ 0.31 We subtract the King of Hearts once so it is not counted twice — the general addition rule. 16 S E C T I O N 0 4 Correlation & Covariance Measuring how two variables move together. C O R R E L AT I O N & C O VA R I A N C E How Two Variables Move Together Covariance Measures the direction of the joint variation of two variables. Positive means they rise together; negative means one rises as the other falls. Its size depends on the units, so it is hard to compare across datasets. Correlation (r) A standardised covariance, always between −1 and +1. It captures both direction and strength on a fixed scale, so different variable pairs can be compared directly. Pearson correlation coefficient r r = −1 Perfect negative r ≈ −0.5 Moderate negative r = 0 No linear relation r ≈ +0.5 Moderate positive r = +1 Perfect positive 18 C O R R E L AT I O N & C O VA R I A N C E Reading a Scatter Plot Scenario: does more study time relate to higher marks? Each dot is one student. 20 40 60 80 100 0 2 4 6 8 Study hours per day Marks (out of 100) What the plot tells us Points rise from lower-left to upper-right → positive correlation. The trend is tight and near-linear → strong, roughly r ≈ 0.98. Caution: correlation is not causation. Study habits, sleep and prior ability may all be at play. 19 W O R K E D E X A M P L E Computing the Correlation r Attendance (%) vs marks (out of 100) for 5 students. Do the two rise together, and how strongly? x (Att%) y (Marks) x− x̄ y−ȳ (x− )(y−ȳ) x̄ (x− )² x̄ (y−ȳ)² 60 50 −15 −16 240 225 256 70 60 −5 −6 30 25 36 75 68 0 2 0 0 4 85 74 10 8 80 100 64 85 78 10 12 120 100 144 Column sums Σ → 470 450 504 Means: = 375/5 = 75% attendance, ȳ = 330/5 = 66 marks (Σx=375, Σy=330) x̄ Pearson's formula r = (x− )(y− ) / √[ (x− )² × (y− )² ] Σ x̄ ȳ Σ x̄ Σ ȳ Substitute the column sums: r = 470 / √(450 × 504) r = 470 / √226800 r = 470 / 476.2 r ≈ +0.99 A strong positive correlation: students with higher attendance tend to score higher marks. 20