INTRODUCTION TO DATA SCIENCE • MODULE 2 Data Acquisition & Data Preparation Sources & types of data · collection · quality issues · missing & noisy data · cleaning · integration · transformation · preprocessing B.E / B.Tech • 5th Semester • 9 Hours Dr. APJ Abdul Kalam School of Engineering · Garden City University, Bengaluru BEFORE WE BEGIN Why Data Preparation Matters Imagine Swiggy wants to predict which orders will be delivered late. The model is only as good as the data fed into it. 70-80% of a data scientist's time is spent finding, cleaning & preparing data - not modelling. ⚠ Wrong decisions A duplicated customer inflates sales counts and misleads managers. ⚠ Broken models Missing values crash training or bias predictions badly. ⚠ Wasted money Acting on dirty data costs businesses real revenue. ⚠ Lost trust One bad dashboard and stakeholders stop believing the data team. WHAT THIS MODULE COVERS The Data Preparation Journey 01 Sources of Data Where data comes from 02 Types of Data Structured · Semi · Unstructured 03 Collection Methods How we gather it 04 Data Quality Issues Missing & noisy data 05 Data Cleaning Fix, fill, smooth 06 Integration Combine many sources 07 Transformation Reshape & rescale 08 Preprocessing Ready for modelling TOPIC 1 Sources of Data Before any analysis can begin, we have to decide where the raw data will come from. Data scientists group every possible origin into three broad families, each with its own cost, speed and reliability trade-offs. ◉ Primary Sources Data you gather yourself, specifically for the problem you are trying to solve. • Surveys & questionnaires — directly ask people the exact questions you need answered • Experiments & sensors — measure results under conditions you control • Interviews & observation — record behaviour or opinions first-hand ▣ Secondary Sources Data that already exists because someone else collected it earlier for another purpose. • Government & census records — large, authoritative, usually free to reuse • Kaggle & research datasets — curated and ready for experimentation • Company databases — sales, HR or ERP data your organisation already stores ☁ Web & Digital Data continuously generated online, often automatically and at enormous scale. • Social media & clickstreams — what users post, click and watch every second • APIs & web scraping — pull structured data from services or web pages • IoT device streams — sensors and smart devices reporting non-stop TOPIC 1 · GOING DEEPER Primary vs Secondary Sources Primary Sources You collect it yourself, for your exact question. + Fresh, specific, you control quality + Owns the data & rights - Expensive, slow to gather - Needs effort & expertise Secondary Sources Reusing data someone else already collected. + Cheap and fast to obtain + Large historical coverage - May not fit your question - Quality/bias unknown SCENARIO GCU wants to know why students skip 8 AM classes. Primary: run a fresh survey of students. Secondary: reuse existing attendance data already in the ERP. TOPIC 2 Types of Data Data is classified by how much internal structure it has. This single property decides how the data can be stored, how easily it can be searched, and how much processing is needed before we can analyse it. 1 Structured Organised into fixed rows and columns with a defined type for every field. › Relational tables in SQL databases › Spreadsheets and CSV files › ATM & banking transactions Easy to store, query and analyse 2 Semi-structured No rigid table, but tags or keys mark each value so the data describes itself. › JSON, XML and YAML files › Emails (header fields + free body) › NoSQL document databases Flexible: has order, not rigid 3 Unstructured No predefined model at all — just raw content a computer cannot read directly. › Text: reviews, PDFs, chats › Images, audio and video › WhatsApp & social media posts Rich in meaning, hard to analyse TOPIC 2A Structured Data Highly organised into a fixed schema of rows and columns . Each column has a defined data type. Lives in relational databases, queried with SQL. ✓ Predefined schema (columns fixed in advance) ✓ Every value has a clear data type ✓ Easy to search, sort, filter, aggregate ✓ Stored in SQL databases / spreadsheets Example: student marks table ID Name CGPA 101 Aditya 8.4 102 Meera 9.1 103 Rohan 7.6 -- one line gets what you want SELECT Name FROM students WHERE CGPA > 8.0; TOPIC 2B Semi-structured Data No rigid table, but tags and markers describe the data so it's self-organising. The schema can vary from record to record. ✓ Uses tags/keys (JSON, XML, YAML) ✓ Flexible - records can differ ✓ Self-describing & hierarchical ✓ Common in APIs and NoSQL databases Example: a Zomato API response (JSON) { "restaurant": "Truffles", "rating": 4.6, "cuisines": ["Burger", "Continental"], "delivery": { "time_min": 32, "free": true }, "veg_only": false } TOPIC 2C Unstructured Data No predefined model at all - the ~80% of real-world data that's richest in meaning but hardest to process directly. ✉ Text Reviews, emails, chats, PDFs ▦ Images Photos, X-rays, scanned forms ♪ Audio Call recordings, voice notes ▶ Video CCTV, YouTube, reels ⚇ Social Tweets, WhatsApp forwards ⌘ Sensor blobs Raw logs, satellite feeds WHY IT'S HARD To read 10,000 Swiggy reviews a computer can't just SELECT the angry ones - it needs NLP / machine learning to turn that text into structured sentiment scores first. TOPIC 2 · SIDE BY SIDE Comparing the Three Types Aspect Structured Semi-structured Unstructured Schema Fixed, predefined Flexible (tags) None Storage Relational DB / SQL NoSQL, JSON/XML Data lakes, blobs Example ATM transactions JSON API response CCTV video, tweets Query ease Very easy (SQL) Moderate Hard (needs AI/ML) Share ~20% Growing ~80% TOPIC 3 Data Collection Methods Once we know which source to use, we need a technique to actually capture the data. Each method below suits a different situation, and every one carries its own strengths and limitations. ✎ Surveys & Questionnaires A fixed set of questions is put to a sample of people. Cheap and easy to scale, but answers can be biased because people self-report. ⚙ Sensors & IoT Devices capture readings automatically and continuously, giving high-volume real-time data — but they need hardware and infrastructure. ⛓ Web Scraping Code extracts data straight from web pages when no other access exists. Powerful, but you must respect each site's legal terms. ⇄ APIs A service hands you clean, structured data on request — e.g. weather, maps or Twitter/X. The sanctioned, reliable route to web data. ▤ Existing Databases Internal systems already log every transaction and event, so the data is sitting there waiting — usually the fastest place to start. ◎ Observation & Logs Behaviour is recorded as it happens: server logs, website clickstreams, or a researcher watching users directly in the field. TOPIC 3 · APPLIED Which Method Fits the Problem? The method depends on the question, budget, and how fast the data changes. • Measure student satisfaction Survey / questionnaire One-time opinion, need direct input • Track campus power usage IoT sensors Continuous, automatic, real-time • Get live cricket scores Public API Structured, someone else maintains it • Study competitor pricing Web scraping Public web data, no API available • Analyse past admissions Existing database Data already sits in the ERP TOPIC 4 Data Quality Issues "Garbage in, garbage out." A model can only ever be as good as the data behind it. We judge that data against six quality dimensions — if any one fails, the results downstream are quietly wrong. ✓ Accuracy Every value truly matches the real world, with no errors or typos. ✓ Completeness No required values are blank, absent or lost during collection. ✓ Consistency The same fact never contradicts itself across different sources. ✓ Timeliness The data is recent enough to still reflect the current situation. ✓ Validity Values obey the expected rules, formats and ranges (e.g. age > 0). ✓ Uniqueness Each real-world entity appears once — no accidental duplicate rows. TOPIC 4 · SEE IT Spot the Problems: A Dirty Dataset A tiny customer table from two merged systems. How many quality issues can you find? ID Name City Age Phone 1 Ravi Kumar Bengaluru 28 98xxxxxx01 2 ravi kumar Bangalore 28 98xxxxxx01 3 Priya S Mysuru -5 (missing) 4 Amit Delhi 210 98xxxxxx88 5 Sara Ali Chennai (missing) 98xxxxxx90 ! Duplicate Rows 1 & 2 same person ! Inconsistent Bengaluru vs Bangalore ! Invalid Age = -5 and 210 ! Missing Blank phone & age TOPIC 4A Missing Values Why data goes missing ! Data entry errors Skipped fields, typos ! Non-response User declines to answer ! Sensor / system failure Device drops readings ! Merging mismatches No match across sources How to handle them Delete - Drop rows/columns - only if few & random Mean / Median / Mode - Impute with a central value Forward / Backward fill - Carry last known value (time series) Predict (regression / KNN) - Model the value from other columns Flag & keep - Mark as 'Unknown' category TOPIC 4A · WORKED EXAMPLE Filling a Missing Value Class marks: 72, 68, 90, ?, 85, 76 - one student's mark is missing. What do we put? Mean (72+68+90+85+76) / 5 result: = 78.2 Good when data is roughly symmetric Median sort -> 68,72,76,85,90 result: = 76 Best when outliers exist (robust) Mode most frequent value result: (none repeats) Used for categories, not marks here TOPIC 4B Noisy Data Noise = random error or variance in a measured variable - outliers, wrong entries and fluctuation that hides the true signal. 1 Binning Sort into bins, smooth by bin mean / median / boundary 2 Regression Fit a function; use the fitted value to smooth 3 Clustering Group similar points; values outside clusters = outliers 4 Outlier removal Detect with IQR / Z-score, then drop or cap 1 2 3 4 5 6 7 8 9 10 11 12 0 2 4 6 8 10 12 14 16 18 20 Smoothing removes the jitter Raw (noisy) Smoothed TOPIC 4B · WORKED EXAMPLE Smoothing by Binning Prices: 4, 8, 15, 21, 21, 24, 25, 28, 34 -> split into 3 equal-frequency bins of 3. Bin Values Mean Smooth by mean Smooth by boundary Bin 1 4, 8, 15 9 4, 4, 4 4, 4, 15 Bin 2 21, 21, 24 22 22, 22, 22 21, 21, 24 Bin 3 25, 28, 34 29 29, 29, 29 25, 25, 34 TAKEAWAY Smoothing by bin mean replaces every value with its bin's average; by boundary snaps each value to the nearer bin edge. Both flatten the random noise. TOPIC 4B · OUTLIERS Detecting Outliers with the IQR Rule An outlier is a value far from the rest. The Inter-Quartile Range (IQR) rule flags anything unusually low or high. Q1 25th percentile (lower quartile) Q3 75th percentile (upper quartile) IQR = Q3 - Q1 (the middle 50% spread) Bounds Low = Q1 - 1.5xIQR , High = Q3 + 1.5xIQR Flag Any value outside [Low, High] is an outlier Salaries (Rs k): 20,22,24,25,27,90 Q1 = 22 Q3 = 27 IQR = 27 - 22 = 5 High = 27 + 1.5x5 = 34.5 90 > 34.5 -> Rs 90k is an OUTLIER TOPIC 5 Data Cleaning Techniques A repeatable pipeline that turns raw, dirty data into an analysis-ready dataset. 1 Handle Missing Impute or drop 2 Remove Duplicates Deduplicate rows 3 Fix Errors Typos, wrong types 4 Smooth Noise Bin / regression 5 Handle Outliers IQR / Z-score 6 Standardise Formats Dates, units, case Golden rule: clean a COPY, log every change, keep it reproducible - never overwrite raw data.