Unit 1: Introduction to Data Science

I. Orientation — Foundations of Data Science

Data science is an interdisciplinary field that uses scientific methods, computing, statistics, and domain knowledge to extract useful insights from data. It turns raw observations into evidence that can support explanations, predictions, decisions, and operational improvements.

• Defining properties:
  • Data-driven: Conclusions are supported by recorded facts rather than intuition alone.
  • Interdisciplinary: It combines statistics, mathematics, programming, database systems, visualization, and subject expertise.
  • Goal-oriented: Every project begins with a practical question, such as predicting customer churn or detecting equipment failure.
  • Iterative: Analysts often revisit earlier stages after discovering poor-quality data or weak model performance.
  • Evidence-based: Results are evaluated with measurable criteria such as accuracy, error, cost, or business value.
• Core assumptions:
  • Available data meaningfully represents the phenomenon being studied.
  • Data quality and sampling methods affect the reliability of conclusions.
  • Models simplify reality and therefore always have assumptions and limitations.
  • Ethical requirements—including privacy, consent, fairness, and security—apply throughout the project.
• Common data forms:
  • Structured data: Organized into rows and columns, such as a sales table.
  • Semi-structured data: Uses flexible labels or keys, as in JSON and XML documents.
  • Unstructured data: Includes text, images, audio, and video without a fixed tabular schema.

II. Motivation — Value of Data Science

A. Why learn data science?

Learning data science develops the ability to convert growing volumes of data into reliable knowledge and practical action.

• Better decisions: Organizations can compare measurable evidence instead of relying only on experience; a retailer can use transaction records to identify products frequently purchased together.
• Career relevance: Data skills are used in finance, healthcare, manufacturing, education, logistics, marketing, government, and scientific research.
• Problem-solving ability: Data science provides a systematic path from a broad concern—such as declining sales—to testable questions about price, location, customer segment, or season.
• Automation: Models can perform repeatable tasks at scale, including classifying emails, recommending products, and forecasting demand.
• Scientific understanding: Statistical analysis distinguishes genuine patterns from random variation and helps assess whether evidence supports a claim.
• Innovation: Data enables new services, such as route optimization based on traffic observations or preventive maintenance based on sensor readings.
• Responsible interpretation: Training helps practitioners recognize biased samples, misleading visualizations, data leakage, and inappropriate causal claims.
• Transferable skills:
  • Technical: Programming, querying databases, modeling, and visualization.
  • Analytical: Framing questions, testing assumptions, and evaluating evidence.
  • Communicative: Explaining results to both technical and non-technical audiences.

III. Analytics Process — From Questions to Deployed Solutions

A. Life cycle of data analytics

The life cycle of data analytics is an iterative sequence that transforms a problem and its associated data into a tested, communicated, and operational solution.

• Typical sequence: Discovery → preparation → model planning → model building → communication → operationalization.
• Iteration: A failed validation test may send the team back to data preparation, while changing business requirements may require renewed discovery.
• Governance: Documentation, security, privacy, ethics, and reproducibility should be maintained across every stage.
• Success criteria: Technical performance must be connected to an operational measure, such as reducing delivery delays by 10%.
• Simplified workflow:

define_problem()
discover_data()
prepare_data()
plan_model()
build_and_validate_model()
communicate_results()
deploy_and_monitor()

B. Data discovery

Data discovery establishes the problem, available resources, relevant data, and criteria for project success.

• Problem definition: Convert a broad objective into an analytical question; “improve retention” may become “predict which subscribers will cancel within 30 days.”
• Stakeholders: Identify decision-makers, data owners, users, customers, and people affected by the resulting decisions.
• Data inventory: Locate internal databases, logs, surveys, spreadsheets, sensors, and legitimate external datasets.
• Initial exploration: Examine variable types, ranges, distributions, missing values, and relationships; a customer table might contain age, plan, usage, and churn status.
• Feasibility: Check whether sufficient data, computing resources, expertise, time, and permission are available.
• Hypotheses: Record expectations that can be tested, such as “frequent service interruptions are associated with churn.”
• Deliverables: A problem statement, data-source inventory, initial hypotheses, risks, constraints, and measurable success criteria.

C. Data preparation

Data preparation converts raw data into a consistent, accurate, and analysis-ready dataset.

• Selection: Retain records and variables relevant to the stated objective rather than collecting unnecessary personal information.
• Cleaning: Correct invalid values, remove inappropriate duplicates, standardize formats, and decide how to treat missing observations.
• Integration: Combine sources through valid keys; customer transactions may be joined to customer profiles using customer_id.
• Transformation: Normalize scales, encode categories, aggregate events, or convert dates into features such as month and weekday.
• Feature engineering: Create informative variables; total customer spending can be calculated as:

total_spending = sum(transaction_amount)

• Data partitioning: Separate training, validation, and test data before model tuning to prevent overly optimistic evaluation.
• Quality controls: Verify row counts, uniqueness constraints, valid ranges, lineage, and transformations through reproducible scripts.
• Ethical controls: Remove unjustified identifiers, restrict access, and inspect whether missingness or sampling disadvantages particular groups.

D. Model planning

Model planning determines which analytical techniques, variables, assumptions, and evaluation measures are appropriate.

• Method selection: Match the technique to the task:
  • Classification predicts categories, such as fraud or non-fraud.
  • Regression predicts numerical values, such as monthly demand.
  • Clustering discovers groups without predefined labels.
  • Time-series methods model observations ordered over time.
• Variable roles: Identify predictors, the target outcome, identifiers, and fields excluded because they leak future information.
• Evaluation metric: Select a measure aligned with consequences; recall may matter when failing to detect fraud is especially costly.
• Baseline: Define a simple reference model, such as predicting the majority class, which a useful model should outperform.
• Validation design: Use holdout validation, cross-validation, or time-based validation according to the data-generating process.
• Assumptions: Document requirements such as independence, linearity, stable distributions, or representative sampling.

E. Model building

Model building implements, trains, tunes, and validates the planned analytical approach.

• Training: Estimate model parameters from training data; linear regression estimates coefficients connecting predictors to a numerical target.
• Tuning: Select hyperparameters, such as tree depth, using validation data rather than the final test set.
• Evaluation: Compare predictions with known outcomes using the planned metric. For numerical predictions, mean absolute error is:

MAE = (1 / n) × Σ|yᵢ − ŷᵢ|

• n is the number of evaluated observations.
• yᵢ is the actual value for observation i.
• ŷᵢ is its predicted value.
  • Comparison: Evaluate several suitable models against the baseline while considering accuracy, speed, interpretability, and cost.
  • Error analysis: Inspect where errors occur, including performance across important subgroups.
  • Reproducibility: Preserve code, data versions, random seeds, model settings, dependencies, and evaluation results.

F. Communicate results

Communicating results turns technical findings into understandable evidence connected to stakeholder decisions.

• Audience alignment: Executives may need impact and risk, whereas engineers need data definitions, architecture, and implementation details.
• Narrative structure: State the problem, method, evidence, conclusion, limitations, and recommended action in logical order.
• Visualization: Choose charts suited to the comparison; line charts show time trends, while bar charts compare categories.
• Clarity: Report “the model correctly identified 82% of actual defaults” rather than presenting an unexplained metric.
• Uncertainty: Include error ranges, assumptions, sample limitations, and conditions under which conclusions may fail.
• Honesty: Distinguish association from causation; correlated advertising and sales do not alone prove that advertising caused the increase.
• Deliverables: Dashboards, reports, presentations, notebooks, model cards, and decision briefs should remain consistent with the evidence.

G. Operationalization

Operationalization integrates a validated solution into real workflows and maintains its usefulness over time.

• Deployment: Deliver the result through an application programming interface, dashboard, scheduled report, batch process, or embedded system.
• Workflow integration: Define who receives each output and what action follows; a high-risk alert may trigger manual review.
• Monitoring: Track service reliability, input-data quality, prediction distributions, model performance, fairness, and business outcomes.
• Drift detection: Identify changes in data or relationships; a fraud model may weaken when attackers adopt new behavior.
• Controls: Apply authentication, encryption, audit logs, access restrictions, fallback procedures, and human oversight.
• Maintenance: Establish schedules or triggers for retraining, testing, approval, versioning, rollback, and retirement.
• Value measurement: Compare post-deployment outcomes with the original success criterion rather than treating deployment itself as success.

IV. Analytical Purposes — Questions Data Can Answer

A. Type of data analysis

The type of data analysis is determined primarily by whether the goal is to summarize, explain, predict, or recommend.

• Descriptive question: What happened?
• Diagnostic question: Why did it happen?
• Predictive question: What is likely to happen?
• Prescriptive question: What action should be taken?
• Progression: The types often build on one another, but greater complexity does not automatically produce greater value.
• Selection principle: The decision need, available evidence, time horizon, and acceptable uncertainty determine the appropriate type.

B. Descriptive analysis

Descriptive analysis summarizes observed historical or current data without explaining causes or forecasting unseen outcomes.

• Measures: Counts, percentages, mean, median, minimum, maximum, variance, and frequency distributions describe a dataset.
• Techniques: Aggregation, filtering, tabulation, dashboards, and visualizations reveal patterns in recorded observations.
• Concrete example: If weekly sales are 100, 120, 80, 100, mean sales equal (100 + 120 + 80 + 100) / 4 = 100.
• Applications: Monthly revenue reports, website traffic summaries, attendance dashboards, and population profiles are descriptive.
• Limitation: A decline shown in a chart establishes what occurred, not why it occurred or whether it will continue.

C. Diagnostic analysis

Diagnostic analysis investigates factors associated with an observed outcome to explain why it may have occurred.

• Techniques: Drill-down, segmentation, correlation analysis, root-cause analysis, hypothesis testing, and comparison with benchmarks.
• Process: Detect an anomaly, generate possible explanations, examine relevant variables, and test whether evidence supports them.
• Concrete example: A sales decline concentrated in one region after stockouts suggests inventory availability as a plausible contributor.
• Causal caution: Correlation does not prove causation because confounding variables, reverse direction, or coincidence may explain an association.
• Value: Diagnosis directs attention toward controllable mechanisms rather than merely reporting symptoms.
• Limitation: Reliable causal conclusions may require experiments or stronger quasi-experimental designs, not observational comparison alone.

D. Predictive analysis

Predictive analysis uses patterns in existing data to estimate unknown or future outcomes.

• Methods: Regression, classification, decision trees, ensemble methods, survival models, and time-series forecasting support different targets.
• Output: A prediction may be a number, category, probability, ranking, or forecast interval.
• Concrete example: A model may assign a borrower a default probability of 0.18, meaning an estimated 18% risk under the model and its data.
• Evaluation: Predictions must be tested on unseen data with metrics such as MAE, precision, recall, or area under the ROC curve.
• Dependence on stability: Accuracy can deteriorate when future conditions differ substantially from training conditions.
• Limitation: A predictive feature need not be a cause; a model can forecast an event without explaining its mechanism.

E. Prescriptive analysis

Prescriptive analysis recommends actions by combining predictions, objectives, constraints, costs, and possible consequences.

• Methods: Mathematical optimization, simulation, decision analysis, business rules, and reinforcement learning can generate recommendations.
• Objective: A delivery system might minimize total travel time while satisfying vehicle-capacity and delivery-window constraints.
• Inputs: Prescriptions require predicted outcomes, feasible actions, resource limits, risk preferences, and a measurable utility or cost.
• Concrete formulation:

Choose action a* = argminₐ C(a)
subject to a ∈ F

• a is a possible action.
• a* is the selected action.
• C(a) is the cost associated with action a.
• F is the set of feasible actions.
  • Human oversight: Recommendations affecting employment, credit, health, or safety require review, accountability, and appeal mechanisms.
  • Limitation: A mathematically optimal answer can still be inappropriate if its objective omits fairness, safety, or practical constraints.

V. Analytics Classifications — How Analysis Is Delivered

A. Types of data analytics

Types of data analytics can also be classified by processing timing, implementation style, and level of automation.

• Batch analytics: Processes accumulated data at intervals; a nightly job may calculate daily sales totals.
• Real-time analytics: Processes events with low latency; payment systems may score transactions within milliseconds.
• Business intelligence analytics: Uses reports, dashboards, queries, and key performance indicators to support monitoring and routine decisions.
• Advanced analytics: Applies statistical modeling, machine learning, simulation, or optimization to complex predictive and prescriptive tasks.
• Exploratory analytics: Searches interactively for patterns and hypotheses without beginning with one fixed model.
• Confirmatory analytics: Tests predefined hypotheses using specified statistical procedures and decision thresholds.
• Relationship to analytical purpose: Batch and real-time describe when computation occurs, while descriptive through prescriptive describe the question being answered.
• Selection factors: Appropriate analytics depends on data volume, velocity, decision urgency, infrastructure, interpretability, cost, risk, and required human control.