Unit 13: NumPy and Pandas - Subjective Questions

Python is a high-level, interpreted, general-purpose programming language known for its simple and readable syntax.

Python is widely used in data science because:

• Ease of learning: Its syntax is concise and easy to understand.
• Rich ecosystem: Libraries such as NumPy, Pandas, Matplotlib, and Scikit-learn support scientific computing and data analysis.
• Interactivity: Tools such as Jupyter Notebook allow code, output, and documentation to be combined.
• Community support: Python has extensive documentation and a large developer community.
• Integration: It can interact with databases, web applications, and languages such as C and C++.

Thus, Python enables analysts to perform the complete data science workflow, from data collection and cleaning to modeling and visualization.

The main built-in Python data structures are:

• List: An ordered, mutable collection that permits duplicate values. It is suitable for storing a sequence that may change, such as scores = [75, 82, 91].
• Tuple: An ordered, immutable collection. It is useful for fixed records, such as point = (4, 7).
• Dictionary: A mutable collection of key-value pairs. It is suitable for labeled data, such as student = {'name': 'Asha', 'marks': 90}.
• Set: An unordered collection of unique values. It is useful for removing duplicates or performing union and intersection operations.

Lists and tuples are generally accessed by position, dictionaries by key, and sets through membership tests and set operations.

NumPy, or Numerical Python, is a Python library for efficient numerical and scientific computing. Its central object is the multidimensional array called ndarray.

Important features of an ndarray include:

• It stores elements of a single homogeneous data type.
• It supports one-dimensional and multidimensional data.
• Its shape specifies the size along each axis.
• Its ndim gives the number of dimensions.
• Its size gives the total number of elements.
• Its dtype identifies the element data type.
• It supports vectorized calculations, broadcasting, slicing, aggregation, and linear algebra.

Because the data is stored compactly and operations are implemented in optimized compiled code, NumPy arrays are generally faster and more memory-efficient than Python lists for numerical work.

Python lists and NumPy arrays differ in several ways:

For example, multiplying a list by 2 repeats it, whereas multiplying a NumPy array by 2 doubles every element. Lists are appropriate for general-purpose heterogeneous collections, while arrays are preferable for large numerical datasets.

NumPy provides several array-creation methods:

• From a Python sequence: np.array([1, 2, 3])
• Array of zeros: np.zeros((2, 3))
• Array of ones: np.ones((3, 2))
• Array filled with a value: np.full((2, 2), 7)
• Evenly spaced integer-like values: np.arange(0, 10, 2)
• A fixed number of evenly spaced values: np.linspace(0, 1, 5)
• Identity matrix: np.eye(3)
• Random values: np.random.random((2, 2))

The required dtype can be stated explicitly, as in np.array([1, 2, 3], dtype='float64'). The method should be selected according to the desired shape, value pattern, and data type.

For a NumPy array:

• Dimension (ndim) is the number of axes.
• Shape (shape) is a tuple containing the length of each axis.
• Size (size) is the total number of elements.
• Axis identifies a direction along which an operation is performed.

For an array with shape (3, 4), ndim is 2 and size is . Axis 0 runs down the rows, while axis 1 runs across the columns.

reshape() changes the arrangement without changing the element count. For example, a.reshape(3, 4) can transform a 12-element vector into a matrix. The condition is:

Using -1 allows NumPy to infer one dimension, as in a.reshape(3, -1). Depending on memory layout, reshaping may return a view rather than an independent copy.

NumPy supports several forms of data selection:

• Basic indexing: For a = np.array([10, 20, 30, 40]), a[1] returns 20, and a[-1] returns 40.
• Slicing: a[1:3] returns the elements at indices 1 and 2. The general form is start:stop:step, where stop is excluded.
• Two-dimensional indexing: If m is a matrix, m[1, 2] selects the element in row 1, column 2; m[:, 0] selects the first column; and m[0:2, 1:3] selects a subarray.
• Boolean masking: a[a > 20] returns only values greater than 20.
• Fancy indexing: a[[0, 3]] selects elements at explicitly listed positions.

Basic slices commonly return views, so modifying a slice may alter the original array. Boolean and fancy indexing generally return copies.

Vectorization means applying an operation to an entire array without writing an explicit Python loop. For example, a * 2 multiplies every element by 2.

Broadcasting allows NumPy to perform operations on arrays with compatible but different shapes. Dimensions are compared from right to left. Two dimensions are compatible when:

• They are equal, or
• One of them is 1.

For example, an array with shape (3, 4) can be added to an array with shape (4,); the second array is treated as though it were repeated across the three rows. Similarly, shapes (3, 1) and (1, 4) broadcast to (3, 4).

Shapes (3, 4) and (2,) are incompatible because the trailing dimensions 4 and 2 are unequal and neither is 1.

Vectorization and broadcasting make code shorter and typically faster while avoiding unnecessary explicit data replication.

Python is dynamically typed, meaning a variable name can refer to objects of different types at different times. For example, x can first refer to an integer and later to a string. Built-in types include int, float, complex, bool, str, and NoneType.

Each Python object stores both its value and type-related metadata. A Python collection such as a list can therefore contain mixed objects, but this flexibility introduces memory and processing overhead.

A NumPy array normally uses a fixed data type, represented by dtype, such as int32, float64, or bool. Its elements have a common size and memory representation. This enables:

• Compact storage
• Predictable numerical behavior
• Fast vectorized operations
• Efficient interaction with compiled code

Thus, Python objects emphasize flexibility, while NumPy arrays emphasize efficient homogeneous numerical computation.

NumPy supports data types such as signed and unsigned integers, floating-point numbers, complex numbers, Boolean values, strings, and objects. Examples include int8, int32, uint16, float32, float64, complex128, and bool.

When np.array() is called without a dtype, NumPy infers a common type. For example, combining integers and floating-point values normally produces a floating-point array. A type can also be specified explicitly with dtype.

Conversion can be performed using astype(), such as a.astype('float64'). Important risks include:

• Converting floats to integers discards the fractional part.
• Storing a value outside an integer type's range can cause overflow or rejection, depending on the operation.
• Converting higher-precision values to lower precision can lose information.
• Mixed strings and numbers may force an unintended string or object type.

The dtype should therefore be selected according to range, precision, memory, and computational requirements.

Pandas is a Python library that provides labeled data structures and tools for data cleaning, transformation, aggregation, and analysis.

A Series is a one-dimensional labeled array. It contains:

• A sequence of values
• An associated index
• A single dtype
• An optional name

A DataFrame is a two-dimensional labeled table. It contains:

• Rows identified by an index
• Columns identified by labels
• Potentially different data types across columns
• Multiple Series aligned on a shared row index

A Series is suitable for one variable, while a DataFrame represents a complete tabular dataset. For example, one column of student marks can be a Series, whereas a table containing names, marks, and grades is a DataFrame.

A Pandas Series can be created in several ways:

• From a list: pd.Series([10, 20, 30]) creates a default integer index.
• From a list with labels: pd.Series([10, 20], index=['a', 'b']) creates a custom index.
• From a dictionary: pd.Series({'a': 10, 'b': 20}) uses dictionary keys as index labels.
• From a scalar: pd.Series(5, index=['a', 'b', 'c']) repeats the scalar for every supplied label.
• From a NumPy array: pd.Series(np.array([1.5, 2.5])) uses the array values.

Useful Series attributes include values, index, dtype, name, shape, and size. Explicit labels should be unique where possible because duplicate labels can make selection and alignment less intuitive.

Data in a Series can be selected as follows:

• Bracket notation: s['a'] selects the value labeled a.
• Label-based selection: s.loc['a'] explicitly selects by label.
• Position-based selection: s.iloc[0] selects the first value by integer position.
• Label slicing: s.loc['a':'c'] usually includes both endpoints.
• Position slicing: s.iloc[0:3] excludes the stop position, following normal Python slicing.
• Boolean masking: s[s > 50] returns values satisfying the condition.
• Multiple labels: s.loc[['a', 'c']] selects the listed labels.

Using loc and iloc avoids ambiguity when the Series has integer labels. For example, label 0 and position 0 may refer to different selection concepts even when they happen to identify the same item.

A DataFrame can be created from:

• A dictionary of lists: pd.DataFrame({'name': ['A', 'B'], 'score': [80, 90]})
• A dictionary of Series: Values are aligned according to their Series indices.
• A list of dictionaries: Each dictionary usually represents one row.
• A two-dimensional NumPy array: Row and column labels can be supplied separately.
• External data: Functions such as pd.read_csv() and pd.read_excel() load tabular files.

The row index identifies records, while column labels identify variables. Labels permit meaningful selection and automatic alignment. If custom labels are not provided, Pandas creates a RangeIndex. A suitable index can improve readability and selection, but it does not replace the need to maintain valid and meaningful data columns.

A DataFrame supports multiple selection methods:

• df['age'] returns one column as a Series.
• df[['name', 'age']] returns selected columns as a DataFrame.
• df.loc['r1'] selects a row by label.
• df.loc['r1':'r3', ['name', 'age']] selects labeled rows and columns; the label slice normally includes its endpoint.
• df.iloc[0] selects the first row by position.
• df.iloc[0:3, 1:4] selects rows and columns by positional slices, excluding stop positions.
• df[df['score'] >= 50] filters rows using a Boolean condition.
• (df['age'] > 18) & (df['city'] == 'Pune') combines conditions using element-wise logical operators.

Parentheses are required around individual compound conditions. loc should be used for labels and Boolean masks, while iloc should be used for integer positions.

The four indexers differ as follows:

Examples include df.loc['r1', 'score'], df.iloc[0, 2], df.at['r1', 'score'], and df.iat[0, 2].

loc label slices normally include the final label, while iloc slices exclude the stop position. at and iat are intended for a single value rather than selecting multiple rows or columns.

Missing data represents an unavailable, unknown, or inapplicable value. Pandas may represent it using:

• NaN for many numerical and object-based columns
• NaT for missing date or time values
• pd.NA for nullable extension data types
• None, which Pandas may convert or interpret as missing depending on the column type

Missing values can be detected using:

• df.isna() or df.isnull() to produce a Boolean mask
• df.notna() or df.notnull() to identify present values
• df.isna().sum() to count missing values in each column
• df.info() to inspect non-null counts and data types

Direct equality checks such as value == np.nan should not be used because NaN is not equal to itself. Missingness should be tested with isna() or notna().

dropna() removes rows or columns according to missing-value rules.

• axis=0 removes rows, while axis=1 removes columns.
• how='any' removes an item if at least one selected value is missing.
• how='all' removes an item only if all selected values are missing.
• thresh=n retains an item only when it contains at least n non-missing values.
• subset=['a', 'b'] limits the missing-value test to particular columns when rows are being removed.

For example, df.dropna(subset=['target']) removes rows whose target value is absent, while df.dropna(axis=1, thresh=80) retains columns having at least 80 non-missing values.

Dropping data is simple but may reduce sample size or introduce bias. The analyst should examine the amount and pattern of missingness before removing records or variables.

Missing values can be filled using several strategies:

• Constant replacement: df['city'].fillna('Unknown')
• Mean imputation: df['score'].fillna(df['score'].mean())
• Median imputation: Useful for skewed numerical data because it is less sensitive to outliers.
• Mode imputation: Suitable for categorical or discrete values.
• Forward filling: df.ffill() uses the last observed value.
• Backward filling: df.bfill() uses the next observed value.
• Grouped imputation: A value may be filled using a statistic computed within its category or group.
• Interpolation: df.interpolate() estimates values between observations, especially in ordered numerical or time-series data.

The strategy must match the meaning and structure of the data. Forward and backward filling are appropriate only when observation order matters. Imputation preserves rows but can distort distributions, variance, and relationships, so the chosen method should be documented.

A systematic Pandas workflow can include the following steps:

1. Load data: Use a function such as pd.read_csv().
2. Inspect structure: Examine df.head(), df.shape, df.columns, df.info(), and df.describe().
3. Check data types: Confirm that numerical, categorical, and date columns have suitable types; convert them where necessary.
4. Detect missingness: Use df.isna().sum() and calculate proportions with df.isna().mean().
5. Select relevant data: Use column lists, loc, iloc, and Boolean filters.
6. Handle missing values: Drop unusable rows or columns, or impute values using constants, group statistics, filling, or interpolation.
7. Validate cleaning: Recheck missing counts, ranges, duplicates, and data types.
8. Summarize: Use operations such as value_counts(), groupby(), agg(), and describe().

The workflow should avoid modifying the original data unexpectedly, justify every deletion or imputation, and verify that cleaning has not introduced invalid values or serious bias.