Unit 14: Futuristic World of Data Analytics

Data analytics is the practice of examining raw data to draw conclusions, spot patterns, and support decisions. This unit frames the field around big data, the way data is described and measured, how it is stored and retrieved efficiently, the statistical learning that turns data into predictions, and the toolset practitioners use.

I. Orientation: The Analytics Landscape

Data analytics converts observations into knowledge through a repeatable pipeline: collect, clean, store, model, interpret. Every later section depends on the vocabulary set out here.

• Data: raw, unprocessed facts (numbers, text, images) with no context on their own.
• Information: data processed and given meaning, e.g. "average temperature = 27 degC".
• Analytics stages: descriptive (what happened), diagnostic (why), predictive (what will happen), prescriptive (what to do).
• Pipeline convention: garbage-in, garbage-out — model quality is capped by data quality, so cleaning precedes modelling.
• Dataset shape: a rectangular table where rows are observations (records) and columns are attributes (fields).

II. Introduction to Big Data and Analysis Techniques

Big data is data whose volume, speed, or variety exceeds the capacity of traditional single-machine tools, forcing distributed storage and processing.

A. Introduction to Big Data

Big data is characterised by a set of "V" properties that explain why ordinary spreadsheets fail.

• Volume: scale from terabytes (10^12 bytes) to petabytes (10^15); e.g. clickstream logs of a global site.
• Velocity: rate of arrival, from batch to real-time streaming, e.g. sensor readings every millisecond.
• Variety: structured (tables), semi-structured (JSON, XML), unstructured (video, free text).
• Veracity: trustworthiness — noise, bias, and missing values that must be handled.
• Value: the usable insight extracted, the ultimate justification for storing the rest.
• Storage principle: big data spreads across clusters of commodity machines rather than one large server, enabling horizontal scaling.

B. Analysis Techniques

Techniques range from simple counting to automated learning, matched to the analytics stage.

• Descriptive statistics: summarise with mean, median, mode, standard deviation.
• Data mining: discover hidden patterns, e.g. association rules ("customers who buy bread also buy butter").
• Clustering: group similar records without labels, e.g. k-means grouping customers by spend.
• Classification and regression: predict a labelled category or a numeric value respectively.
• Text and sentiment analysis: parse unstructured language to score opinion polarity.
• Visualization: charts and dashboards that expose trends the eye can read quickly.

III. Elements, Variables, and Data Categorization

This section defines the building blocks of any dataset: the things observed, the properties recorded, and the kinds those properties fall into.

A. Elements

Elements are the individual entities on which data is collected.

• Definition: an element (also case, unit, or observation) is one row in the dataset.
• Example: in a student table, each student is an element; 200 students give 200 elements.
• Observation count: denoted n, the number of elements, drives statistical power.

B. Variables

A variable is a characteristic of an element that can take different values.

• Definition: each column is a variable, e.g. age, city, marks.
• Value: the specific entry for one element under one variable.
• Types by role: independent (predictor/feature) versus dependent (target/outcome).
• Types by nature:
  • Qualitative (categorical): labels or categories, e.g. gender, blood group.
  • Quantitative (numerical): measurable amounts, split into discrete (countable, e.g. number of children) and continuous (any value in a range, e.g. height 172.4 cm).

C. Data Categorization

Categorization organises variables so the right analysis and chart type can be chosen.

• Structured: fits a fixed schema in rows and columns, e.g. an SQL table.
• Semi-structured: carries tags but no rigid table, e.g. a JSON document.
• Unstructured: no predefined model, e.g. emails, audio, images.
• Why it matters: categorical variables need bar charts and mode; numerical variables allow means and histograms.

IV. Levels of Measurement

The level of measurement fixes which operations and statistics are valid for a variable. There are four, in increasing power.

A. The Four Levels

Each level adds a capability the previous one lacked.

1. Nominal: names or categories with no order.
   • Operations: equality only; count and mode.
   • Example: eye colour {blue, brown, green}.
2. Ordinal: ordered categories with unequal or unknown gaps.
   • Operations: ranking, median; differences are not meaningful.
   • Example: satisfaction {low, medium, high}.
3. Interval: ordered with equal spacing but no true zero.
   • Operations: addition and subtraction; ratios are meaningless.
   • Example: temperature in degC — 20 degC is not "twice as hot" as 10 degC.
4. Ratio: interval with a genuine zero, so ratios hold.
   • Operations: all arithmetic including division.
   • Example: weight 80 kg is truly twice 40 kg.

• Rule of thumb: the higher the level, the more statistical techniques become permissible.

V. Data Management and Indexing

Data management is the disciplined storage, organisation, and retrieval of data; indexing is the technique that makes retrieval fast.

A. Data Management

Management keeps data accurate, available, and secure across its lifecycle.

• Collection and ingestion: gathering from sources such as forms, sensors, APIs.
• Cleaning: removing duplicates, fixing formats, imputing missing values.
• Storage models: relational databases (SQL) for structured data; NoSQL stores (document, key-value, column) for flexible or large-scale data.
• Governance: rules for access, privacy, backup, and retention.
• Integrity: constraints (primary key, foreign key) that stop invalid records.

B. Indexing

An index is an auxiliary structure that lets a query find rows without scanning the whole table.

• Analogy: like a book index pointing straight to a page instead of reading every page.
• Cost model: a full scan is O(n); a balanced-tree index lookup is roughly O(log n).
• Common structure: B-tree indexes for range queries; hash indexes for exact matches.
• Trade-off: indexes speed reads but slow writes and consume extra storage, since each insert must update the index.

-- create an index to accelerate lookups by city
CREATE INDEX idx_city ON customers(city);
SELECT * FROM customers WHERE city = 'Pune';

VI. Introduction to Statistical Learning

Statistical learning is the set of methods for estimating a function that maps inputs to an output, so the model can predict or explain.

A. Core Framework

The goal is to learn f in Y = f(X) + e from data.

Y = f(X) + e
Y : response / target variable
X : predictors / features (X1, X2, ... Xp)
f : the unknown systematic relationship
e : random error, irreducible noise

• Prediction: use estimated f to guess Y for new X.
• Inference: understand how each predictor influences Y.

B. Supervised vs Unsupervised Learning

The presence or absence of a labelled target splits the field.

1. Supervised: training data includes known outputs.
   • Regression: predicts a continuous Y, e.g. house price.
   • Classification: predicts a category, e.g. spam or not spam.
2. Unsupervised: no labels; the aim is structure.
   • Clustering: group similar records, e.g. k-means.
   • Dimensionality reduction: compress features, e.g. PCA.

C. Model Fit and Generalization

A model must perform on unseen data, not just the training set.

• Training vs test split: hold back data to measure true performance.
• Overfitting: model memorises noise; low training error but high test error.
• Underfitting: model too simple to capture the pattern; high error everywhere.
• Bias-variance trade-off: flexible models cut bias but raise variance; the aim is the balance that minimises total test error.

VII. Overview of Tools Used for Data Analysis

Tools span programming languages, spreadsheets, statistical packages, big-data platforms, and visualization software, chosen by scale and skill level.

A. Programming Languages and Libraries

Code gives the most flexibility for custom analysis.

• Python: general-purpose; libraries pandas (tables), NumPy (arrays), scikit-learn (learning), Matplotlib (plots).
• R: built for statistics; strong in modelling and the ggplot2 visualization grammar.
• SQL: the standard language for querying relational databases.

B. Spreadsheet and BI Tools

Point-and-click tools serve analysts without heavy coding.

• Microsoft Excel / Google Sheets: formulas, pivot tables, quick charts for small datasets.
• Power BI and Tableau: interactive dashboards connecting to live data sources.

C. Statistical Packages

Dedicated software for rigorous statistical work.

• SPSS and SAS: menu-driven analysis widely used in social science and enterprise.
• Purpose: hypothesis testing, regression, survey analysis with validated procedures.

D. Big Data Platforms

Frameworks that distribute work across clusters when volume exceeds one machine.

• Apache Hadoop: stores data across nodes (HDFS) and processes with MapReduce.
• Apache Spark: in-memory engine, much faster than disk-based MapReduce for iterative jobs.
• Selection guide: small data suits Excel or a single Python script; petabyte-scale streaming demands Spark on a cluster.