Unit 4: Data Analysis with NumPy and Pandas - Subjective Questions

NumPy ndarray is a multidimensional, homogeneous array object provided by the NumPy library. It is designed for efficient numerical computation.

Main characteristics:

• All elements generally have the same data type.
• It can have one or more dimensions.
• It supports vectorized operations without explicit Python loops.
• It stores data more compactly than ordinary lists.
• It provides attributes such as ndim, shape, size, and dtype.

Difference from a Python list:

• A list can store values of different data types, whereas an ndarray normally stores values of a single data type.
• NumPy arrays support fast mathematical and matrix operations.
• Arrays usually consume less memory for numerical data.
• NumPy provides advanced indexing, slicing, broadcasting, and aggregation functions.

In NumPy, the structure of an array is described using dimensions and axes.

• Dimension: The number of levels in an array. A one-dimensional array is similar to a list, while a two-dimensional array is similar to a table.
• Axis: A direction along which operations are performed. In a two-dimensional array, axis 0 usually represents rows and axis 1 represents columns.
• Shape: A tuple indicating the length of the array along each dimension. For example, an array with 3 rows and 4 columns has shape (3, 4).
• Size: The total number of elements in the array. For shape (3, 4), the size is .
• ndim: The number of dimensions in the array.

For example, if a = np.zeros((2, 3, 4)), then a.shape is (2, 3, 4), a.ndim is 3, and a.size is 24.

NumPy provides several functions for creating arrays:

• From a Python list: np.array([1, 2, 3])
• Using a range: np.arange(0, 10, 2) creates values from 0 to 8 with a step of 2.
• Using evenly spaced values: np.linspace(0, 1, 5) creates five equally spaced values between 0 and 1.
• Zeros: np.zeros((2, 3)) creates a two-by-three array filled with zeros.
• Ones: np.ones((2, 2)) creates a two-by-two array filled with ones.
• Identity matrix: np.eye(3) creates a three-by-three identity matrix.
• Random values: np.random.rand(2, 3) creates a two-by-three array of random floating-point values between 0 and 1.

The choice of function depends on whether the program needs a sequence, initialized matrix, evenly spaced data, or random test data.

NumPy uses zero-based indexing, meaning that the first element has index 0.

One-dimensional array:

• a[0] accesses the first element.
• a[-1] accesses the last element.

Two-dimensional array:

• a[1, 2] accesses the element in row 1 and column 2.
• a[0] accesses the first row.
• a[:, 1] accesses the second column.

Three-dimensional array:

• a[0, 1, 2] accesses an element using three indices representing the first, second, and third axes.

NumPy also supports slicing. For example, a[1:4] selects elements from index 1 up to, but not including, index 4. In two dimensions, a[0:2, 1:3] selects the first two rows and columns 1 through 2.

Slicing selects a portion of an array using the form start:stop:step.

Examples:

• a[1:5] selects elements from index 1 to index 4.
• a[::2] selects every second element.
• a[::-1] reverses the array.
• a[0:2, 1:4] selects a rectangular section of a two-dimensional array.

Boolean indexing selects elements that satisfy a condition. For example, a[a > 10] returns all elements greater than 10. Multiple conditions can be combined using operators such as & and |, with each condition enclosed in parentheses.

Boolean indexing is useful for filtering data, removing invalid values, identifying values within a range, and performing conditional updates. For example, a[a < 0] = 0 replaces all negative values with zero.

NumPy provides functions for changing the structure of arrays without necessarily changing their data.

• Reshaping: a.reshape(2, 3) changes an array into two rows and three columns. The total number of elements must remain unchanged.
• Flattening: a.flatten() returns a one-dimensional copy of the array. a.ravel() also produces a one-dimensional form and may return a view when possible.
• Transposing: a.T exchanges rows and columns in a two-dimensional array.
• Concatenation: np.concatenate((a, b), axis=0) joins arrays along a specified axis.
• Vertical stacking: np.vstack((a, b)) combines arrays row-wise.
• Horizontal stacking: np.hstack((a, b)) combines arrays column-wise.
• Splitting: np.split(a, parts) divides an array into multiple subarrays.

These operations are useful when preparing data for analysis, visualization, or machine-learning algorithms.

Broadcasting is NumPy's mechanism for performing arithmetic operations on arrays with different but compatible shapes.

General rules:

• NumPy compares shapes from the last dimension toward the first.
• Two dimensions are compatible if they are equal or if one of them is 1.
• Missing dimensions are treated as having length 1.
• If the dimensions are incompatible, NumPy raises a broadcasting error.

For example, adding a scalar to an array, such as a + 5, adds 5 to every element. A one-dimensional array of shape (3,) can be added to a two-dimensional array of shape (2, 3) because the last dimensions match.

Broadcasting avoids explicit loops, improves readability, and usually provides efficient vectorized computation. However, it should be used carefully because broadcasting large arrays can require substantial memory.

NumPy supports element-wise mathematical operations and statistical aggregations.

Element-wise operations:

• np.add, np.subtract, np.multiply, and np.divide
• np.sqrt, np.exp, and np.log
• Comparisons such as a > 5

Aggregation functions:

• np.sum() calculates the total.
• np.mean() calculates the arithmetic average.
• np.median() finds the middle value.
• np.min() and np.max() find extreme values.
• np.std() calculates standard deviation.
• np.var() calculates variance.

The axis argument controls the direction of aggregation. For a two-dimensional array, np.sum(a, axis=0) calculates column-wise totals, while np.sum(a, axis=1) calculates row-wise totals. If no axis is supplied, the operation usually considers all elements.

An N-dimensional array, or ndarray, is an array with any number of dimensions. A one-dimensional array has one axis, a matrix has two axes, and an array representing multiple images or data batches may have three or more axes.

For an array with shape (2, 3, 4):

• There are three dimensions.
• The first axis has length 2.
• The second axis has length 3.
• The third axis has length 4.
• The total number of elements is .

Elements are accessed with one index per dimension, such as a[1, 2, 3]. Operations can be performed element by element or along a selected axis. N-dimensional arrays are useful for representing tables, images, videos, scientific measurements, and collections of multidimensional observations.

A Pandas Series is a one-dimensional labeled array. It contains values and an associated index. The values may be numeric, textual, or of another data type.

A Pandas DataFrame is a two-dimensional labeled data structure consisting of rows and columns. Each column may have a different data type, and every row and column can have labels.

Comparison:

• A Series represents a single labeled column or sequence.
• A DataFrame represents a complete table or dataset.
• A Series uses one index, while a DataFrame has row and column labels.
• A DataFrame can be created from dictionaries, lists, NumPy arrays, or files.

Series objects are useful for individual variables, while DataFrames are preferred for data analysis, filtering, grouping, and tabular reporting.

Data inspection is the first step in understanding a dataset.

• head() displays the first few rows.
• tail() displays the last few rows.
• info() shows the number of rows, columns, non-null values, and data types.
• describe() provides statistical summaries for numeric columns, including count, mean, standard deviation, minimum, quartiles, and maximum.
• shape returns a tuple containing the number of rows and columns.
• dtypes displays the data type of each column.
• columns returns the column names.
• index describes the row labels.

These tools help identify missing values, incorrect data types, unusual ranges, duplicate fields, and the general size and structure of the dataset before analysis begins.

Pandas provides convenient functions for importing and exporting tabular data.

CSV files:

• pd.read_csv("data.csv") loads a CSV file into a DataFrame.
• Important options include sep, header, index_col, usecols, and na_values.
• df.to_csv("output.csv", index=False) writes a DataFrame to a CSV file without saving the index.

HTML data:

• pd.read_html("page.html") or pd.read_html(url) extracts HTML tables and returns a list of DataFrames.
• The required table can be selected from that list.

During import, the analyst should verify column names, encoding, delimiters, missing-value markers, and data types. When exporting, options such as index=False, selected columns, and appropriate formatting can be used to produce a clean output file.

Pandas supports several selection techniques.

• df["Name"] selects one column as a Series.
• df[["Name", "Marks"]] selects multiple columns as a DataFrame.
• df.loc["row_label", "column_label"] selects data using labels.
• df.iloc[0, 1] selects data using integer positions.
• df.iloc[0:3, 1:4] selects a positional slice.
• df[df["Marks"] >= 50] selects rows satisfying a condition.
• df.loc[df["City"] == "Delhi", ["Name", "Marks"]] combines conditional filtering with column selection.

Boolean conditions can be combined using & for AND and | for OR. Parentheses should be placed around each condition. Label-based selection is preferred when index or column labels are meaningful, while positional selection is useful when working by numeric location.

Pandas allows DataFrames to be transformed using a variety of operations.

• Adding a column: df["Total"] = df["Test"] + df["Exam"]
• Renaming columns: df.rename(columns={"old": "new"})
• Changing values: values can be assigned using .loc or .iloc.
• Sorting: df.sort_values("Marks", ascending=False) sorts by a column.
• Sorting by index: df.sort_index() orders rows according to their index.
• Deleting a column: df.drop("Total", axis=1) removes a column.
• Deleting rows: df.drop(index=[0, 1]) removes selected rows.
• Changing the index: df.set_index("ID") uses a column as the index.

Many operations return a new DataFrame unless inplace=True is used. Explicit assignment is often clearer and helps avoid unintended modifications.

The groupby() operation divides data into groups according to one or more categorical columns and then applies an aggregation to each group.

For example, df.groupby("Department")["Salary"].mean() calculates the average salary for each department.

Common aggregation functions include:

• sum() for totals
• mean() for averages
• count() for the number of observations
• min() and max() for extreme values
• median() for middle values
• std() for standard deviation

Multiple aggregations can be applied with agg(), such as df.groupby("Department")["Salary"].agg(["mean", "max", "min"]). Grouping is useful for departmental reports, category-wise comparisons, regional analysis, and summarizing large datasets.

Pandas provides function-based operations for transforming data.

• map() is commonly used with a Series to transform each value. For example, df["Marks"].map(lambda x: x + 5) adds 5 to every mark.
• apply() applies a function along a Series or DataFrame axis. For example, df["Name"].apply(len) calculates the length of each name, while df.apply(np.mean, axis=0) calculates column-wise means.
• Element-wise DataFrame transformation: functions such as df.map() in newer Pandas versions can apply a function to every cell. In older versions, applymap() was commonly used.

Functions may be named functions, lambda expressions, or built-in functions. Vectorized operations are generally preferred when available because they are usually faster and clearer than applying Python functions element by element.

Data cleaning is the process of detecting and correcting inaccurate, incomplete, inconsistent, duplicate, or improperly formatted data.

Typical steps include:

• Inspecting the data structure and data types.
• Identifying missing and invalid values.
• Removing or correcting duplicate records.
• Standardizing text, capitalization, units, and category names.
• Converting columns to suitable data types.
• Detecting and handling outliers.
• Validating ranges and relationships between columns.

Cleaning is essential because poor-quality data can produce misleading statistics, incorrect visualizations, and unreliable conclusions. A clean dataset improves accuracy, consistency, reproducibility, and the validity of later analytical or machine-learning results.

Missing values can be detected using df.isna(), df.isnull(), and df.notna(). The number of missing values in each column can be found with df.isna().sum().

Deletion methods:

• dropna() removes rows or columns containing missing values.
• Deletion is simple but may discard useful observations and introduce bias if many values are missing.

Imputation methods:

• fillna(0) replaces missing values with zero when appropriate.
• Numeric values may be replaced with the mean or median.
• Categorical values may be replaced with the mode or a label such as "Unknown".
• Forward filling and backward filling are useful for ordered or time-series data.

The appropriate method depends on the amount, pattern, and meaning of the missing data. The chosen method should be documented and validated.

Data inconsistencies can be handled through systematic inspection and transformation.

• Duplicate records: Use df.duplicated() to identify duplicates and df.drop_duplicates() to remove them. A subset of columns can be supplied when only certain fields define a duplicate.
• Inconsistent text: Methods such as .str.strip(), .str.lower(), .str.upper(), .str.title(), and .str.replace() can remove extra spaces, standardize case, and correct patterns.
• Incorrect data types: df.dtypes helps identify problems. Numeric text can be converted with pd.to_numeric(..., errors="coerce"), dates with pd.to_datetime(), and categories with .astype("category").

After correction, the data should be rechecked for remaining duplicates, conversion failures, missing values, and invalid categories.

An outlier is an observation that differs unusually from the general pattern of a dataset. Outliers may represent errors, rare but valid events, or exceptional cases.

Detection methods:

• Visual inspection using box plots or scatter plots.
• The IQR method, where values below or above are potential outliers.
• The z-score method, where values with a large absolute standardized score may be flagged.
• Domain-specific minimum and maximum limits.

Handling methods:

• Verify and correct data-entry errors.
• Remove values only when justified.
• Cap or winsorize extreme values.
• Transform the data using logarithms.
• Use robust statistics such as the median.

The method must depend on the context; an outlier should not be removed automatically.