Assignment 1: Perform the following operations using Python on any open-source dataset (e.g., data.csv) the web site). 1. Import all the required Python Libraries. 2. Locate an open-source data from the web (e.g. https://www.kaggle.com). Provide a clear description of the data and its source (i.e., URL of 3. Load the Dataset into pandas’ data frame. 4. Data Preprocessing: check for missing values in the data using pandas isnull (), describe() function to get some initial statistics. Provide variable descriptions. Types of variables etc. Check the dimensions of the data frame. 5. Data Formatting and Data Normalization: Summarize the types of variables by checking the data types (i.e., character, numeric, integer, factor, and logical) of the variables in the data set. If variables are not in the correct data type, apply proper type conversions. 6. Turn categorical variables into quantitative variables in Python. In addition to the codes and outputs, explain every operation that you do in the above steps and explain everything that you do to import/read/scrape the data set. # Import pandas library for data manipulation and working with DataFrames import pandas as pd # Read the Titanic CSV file and store it in a pandas DataFrame named df df = pd read_csv("Titanic-Dataset.csv") # Display the first 5 rows of the dataset to check if data is loaded correctly df head() PassengerId Survived Pclass Name Sex Age SibSp Parch Ticket Fare Cabin E 0 1 0 3 Braund, Mr. Owen Harris male 22.0 1 0 A/5 21171 7.2500 NaN 1 2 1 1 Cumings, Mrs. John Bradley (Florence Briggs Th... female 38.0 1 0 PC 17599 71.2833 C85 2 3 1 3 Heikkinen, Miss. Laina female 26.0 0 0 STON/O2. 3101282 7.9250 NaN 3 4 1 1 Futrelle, Mrs. Jacques Heath (Lily May Peel) female 35.0 1 0 113803 53.1000 C123 4 5 0 3 Allen, Mr. William Henry male 35.0 0 0 373450 8.0500 NaN In [1]: In [12]: Out[12]: Explore our developer-friendly HTML to PDF API Printed using PDFCrowd HTML to PDF df tail() PassengerId Survived Pclass Name Sex Age SibSp Parch Ticket Fare Cabin Emb 886 887 0 2 Montvila, Rev. Juozas male 27.0 0 0 211536 13.00 NaN 887 888 1 1 Graham, Miss. Margaret Edith female 19.0 0 0 112053 30.00 B42 888 889 0 3 Johnston, Miss. Catherine Helen "Carrie" female NaN 1 2 W./C. 6607 23.45 NaN 889 890 1 1 Behr, Mr. Karl Howell male 26.0 0 0 111369 30.00 C148 890 891 0 3 Dooley, Mr. Patrick male 32.0 0 0 370376 7.75 NaN # Returns the number of rows and columns in the dataset # Output format: (number_of_rows, number_of_columns) df shape (891, 12) # Displays all column names present in the dataset df columns Index(['PassengerId', 'Survived', 'Pclass', 'Name', 'Sex', 'Age', 'SibSp', 'Parch', 'Ticket', 'Fare', 'Cabin', 'Embarked'], dtype='object') # Provides statistical summary of numerical columns # Includes count, mean, standard deviation, minimum, # quartiles (25%, 50%, 75%), and maximum values df describe() PassengerId Survived Pclass Age SibSp Parch Fare count 891.000000 891.000000 891.000000 714.000000 891.000000 891.000000 891.000000 mean 446.000000 0.383838 2.308642 29.699118 0.523008 0.381594 32.204208 std 257.353842 0.486592 0.836071 14.526497 1.102743 0.806057 49.693429 min 1.000000 0.000000 1.000000 0.420000 0.000000 0.000000 0.000000 In [13]: Out[13]: In [15]: Out[15]: In [16]: Out[16]: In [18]: Out[18]: Explore our developer-friendly HTML to PDF API Printed using PDFCrowd HTML to PDF PassengerId Survived Pclass Age SibSp Parch Fare 25% 223.500000 0.000000 2.000000 20.125000 0.000000 0.000000 7.910400 50% 446.000000 0.000000 3.000000 28.000000 0.000000 0.000000 14.454200 75% 668.500000 1.000000 3.000000 38.000000 1.000000 0.000000 31.000000 max 891.000000 1.000000 3.000000 80.000000 8.000000 6.000000 512.329200 # Counts the total number of missing values in each column df isnull() sum() PassengerId 0 Survived 0 Pclass 0 Name 0 Sex 0 Age 177 SibSp 0 Parch 0 Ticket 0 Fare 0 Cabin 687 Embarked 2 dtype: int64 # Displays the data type of each column # int64 -> Integer values # float64 -> Decimal values # object -> String/text values # category -> Categorical variables df dtypes PassengerId int64 Survived int64 Pclass int64 Name object Sex object Age float64 SibSp int64 Parch int64 Ticket object Fare float64 Cabin object Embarked object dtype: object # Convert Survived column from integer to categorical type # 0 = Did not survive # 1 = Survived df['Survived'] = df['Survived'] astype('category') # Convert Pclass column from integer to categorical type # 1 = First class In [20]: Out[20]: In [24]: Out[24]: In [31]: Explore our developer-friendly HTML to PDF API Printed using PDFCrowd HTML to PDF # 2 = Second class # 3 = Third class df['Pclass'] = df['Pclass'] astype('category') df dtypes PassengerId int64 Survived category Pclass category Name object Sex object Age float64 SibSp int64 Parch int64 Ticket object Fare float64 Cabin object Embarked object dtype: object # Convert Sex column into categorical type # Categories: male and female df['Sex'] = df['Sex'] astype('category') # Convert Embarked column into categorical type # Categories: C, Q, S df['Embarked'] = df['Embarked'] astype('category') df dtypes PassengerId int64 Survived category Pclass category Name object Sex category Age float64 SibSp int64 Parch int64 Ticket object Fare float64 Cabin object Embarked category dtype: object # Calculate the average age and replace all missing Age values with the mean age # This prevents machine learning algorithms from failing due to missing data df['Age'] fillna(df['Age'] mean(), inplace =True ) # Find the most frequently occurring value in Embarked column # Replace missing Embarked values with the mode value df['Embarked'] fillna(df['Embarked'] mode()[0], inplace =True ) In [32]: Out[32]: In [33]: In [34]: Out[34]: In [41]: In [42]: In [46]: Explore our developer-friendly HTML to PDF API Printed using PDFCrowd HTML to PDF # Remove Cabin column because it contains too many missing values # inplace=True permanently removes the column from df df drop(columns = ['Cabin'], inplace =True , errors = 'ignore') df columns Index(['PassengerId', 'Survived', 'Pclass', 'Name', 'Sex', 'Age', 'SibSp', 'Parch', 'Ticket', 'Fare', 'Embarked'], dtype='object') # Convert Sex column into numerical values # male -> 0 # female -> 1 # Machine learning algorithms require numerical input df['Sex'] = df['Sex'] map({'male': 0, 'female': 1}) # Convert Embarked categories into separate binary columns # Creates columns such as Embarked_Q and Embarked_S # drop_first=True removes one category to avoid multicollinearity df = pd get_dummies(df, columns = ['Embarked'], drop_first =True ) # Display the first 5 rows of the processed dataset # Used to verify that preprocessing steps were applied correctly df head() PassengerId Survived Pclass Name Sex Age SibSp Parch Ticket Fare Embarked_Q 0 1 0 3 Braund, Mr. Owen Harris 0 22.0 1 0 A/5 21171 7.2500 Fals 1 2 1 1 Cumings, Mrs. John Bradley (Florence Briggs Th... 1 38.0 1 0 PC 17599 71.2833 Fals 2 3 1 3 Heikkinen, Miss. Laina 1 26.0 0 0 STON/O2. 3101282 7.9250 Fals 3 4 1 1 Futrelle, Mrs. Jacques Heath (Lily May Peel) 1 35.0 1 0 113803 53.1000 Fals 4 5 0 3 Allen, Mr. William Henry 0 35.0 0 0 373450 8.0500 Fals df shape In [45]: Out[45]: In [47]: In [48]: In [49]: Out[49]: In [50]: Explore our developer-friendly HTML to PDF API Printed using PDFCrowd HTML to PDF (891, 12) Out[50]: Explore our developer-friendly HTML to PDF API Printed using PDFCrowd HTML to PDF