Unit 6: Artificial Intelligence and Data Science

Artificial Intelligence (AI) and Data Science are complementary fields: AI builds systems that mimic intelligent behaviour, while Data Science extracts knowledge from data to feed and evaluate those systems. This unit establishes what AI is, traces its evolution, surveys its core techniques, and then covers the data pipeline that powers modern analytics.

I. Foundations of Artificial Intelligence

AI is the branch of computer science concerned with building machines that perform tasks normally requiring human intelligence.

• Working definition: AI is the simulation of human cognitive processes such as learning, reasoning, perception, and decision-making by computer systems.
• Narrow vs. general: Narrow (weak) AI solves one task, e.g. spam filtering; General (strong) AI would match human ability across any task and does not yet exist.
• Core goals: knowledge representation, reasoning, planning, learning, perception, and natural-language communication.
• Rationality test: the Turing Test (Alan Turing, 1950) judges a machine intelligent if a human evaluator cannot distinguish its text replies from a person's.

A. Definition and scope of AI

AI's scope spans several overlapping sub-disciplines applied across industry.

• Machine Learning: systems that improve from data without explicit rule coding.
• Computer vision: interpreting images and video, e.g. face detection.
• Robotics: physical agents that sense and act, e.g. warehouse robots.
• Expert systems: rule-based programs encoding human specialist knowledge, e.g. medical diagnosis aids.
• Application domains: healthcare (diagnostics), finance (fraud detection), transport (self-driving cars), retail (recommendation engines).

B. History and evolution of AI

AI has advanced in waves of optimism and stagnation ("AI winters").

• 1943–1956, birth: McCulloch and Pitts model an artificial neuron (1943); the term "Artificial Intelligence" is coined at the Dartmouth Conference (1956).
• 1956–1974, golden years: early reasoning programs and the Perceptron (Rosenblatt, 1958).
• 1974–1980, first AI winter: funding cut after limits of early models were exposed.
• 1980s, expert systems boom: commercial rule-based systems, then a second winter.
• 1997, milestone: IBM's Deep Blue defeats chess champion Garry Kasparov.
• 2012 onward, deep-learning era: the AlexNet neural network wins the ImageNet contest, triggering the current surge; large models like the Transformer (2017) enable today's generative systems.

II. Machine Learning — Learning From Data

Machine Learning (ML) is a subset of AI where algorithms build models from example data to make predictions or decisions.

A. Introduction to machine learning

ML replaces hand-written rules with patterns inferred from data.

• Supervised learning: trained on labelled data (input → known output); tasks are classification (spam/not-spam) and regression (predicting price).
• Unsupervised learning: finds structure in unlabelled data, e.g. clustering customers into segments.
• Reinforcement learning: an agent learns by trial and error using rewards, e.g. game-playing AI.
• Training vs. testing: data is split so the model learns on one portion and is evaluated on unseen data to measure generalisation.
• Overfitting: a model memorises training data but fails on new data; countered by more data or simpler models.
• Simple worked idea: linear regression fits a line y = mx + c by adjusting m and c to minimise the squared error between predicted and actual y.

error = Σ (actual_y - predicted_y)²   # minimise this over all data points

Here actual_y is the true value, predicted_y is the model output, and the sum runs across every training example.

III. Neural Networks — Layered Learning Models

A neural network is a computing model of interconnected nodes ("neurons") loosely inspired by the brain, forming the basis of deep learning.

A. Neural networks

Neurons process weighted inputs through layers to learn complex patterns.

• Neuron computation: each node computes a weighted sum of inputs plus a bias, then applies an activation function.

output = activation( (w1·x1 + w2·x2 + ... + wn·xn) + b )

• xi are inputs, wi are learned weights, b is the bias, and activation (e.g. ReLU or sigmoid) adds non-linearity.
  • Layer structure: an input layer, one or more hidden layers, and an output layer; "deep learning" means many hidden layers.
  • Training by backpropagation: the network compares its output to the correct answer, computes error, and adjusts weights backwards using gradient descent.
  • Strengths: excel at images, speech, and text where relationships are non-linear and high-dimensional.
  • Limitation: need large data and compute, and act as "black boxes" that are hard to interpret.

IV. Natural Language Processing — Machines and Human Language

Natural Language Processing (NLP) enables computers to understand, interpret, and generate human language.

A. Natural language processing

NLP bridges unstructured text and structured machine representation.

• Tokenisation: splitting text into words or sub-words, e.g. "data science" → ["data", "science"].
• Stop-word removal and stemming: dropping common words ("the", "is") and reducing words to roots ("running" → "run").
• Vectorisation: converting words into numeric vectors (embeddings) so meaning-similar words sit close together.
• Core tasks: sentiment analysis, machine translation, named-entity recognition, question answering.
• Everyday uses: chatbots, voice assistants (Alexa, Siri), autocomplete, and spam detection.

V. Generative AI — Creating New Content

Generative AI refers to models that produce new content, text, images, audio, or code, rather than only classifying existing data.

A. Generative AI

These models learn the statistical patterns of training data and sample from them to generate novel outputs.

• Large Language Models (LLMs): trained on vast text to predict the next token; power tools like ChatGPT.
• Transformer architecture: the 2017 design using an "attention" mechanism that weighs how much each word relates to others, enabling coherent long text.
• Generative techniques: GANs (two networks competing to produce realistic images) and diffusion models (refining random noise into images).
• Applications: drafting text, code generation, image creation, and summarisation.
• Risks: hallucination (confident but false output), bias inherited from training data, and misuse for deepfakes or misinformation.

VI. Data Science — Turning Data Into Knowledge

Data Science is the interdisciplinary field that uses statistics, programming, and domain knowledge to extract insight from data.

A. Data science: definition and importance

Data Science underpins evidence-based decisions across every modern sector.

• Definition: the systematic study of data to discover patterns, build models, and support decisions, combining maths, computing, and domain expertise.
• Lifecycle: problem definition → data collection → cleaning → analysis → modelling → communication of results.
• Importance: enables data-driven strategy, personalisation (recommendations), forecasting (demand, risk), and automation.
• Roles: data analyst, data scientist, data engineer, and ML engineer, differing in emphasis from reporting to modelling to infrastructure.

B. Big data

Big Data describes datasets too large or complex for traditional tools to handle.

• The five V's:
  • Volume: vast quantity, measured in terabytes and petabytes.
  • Velocity: high speed of generation, e.g. real-time sensor streams.
  • Variety: structured tables, plus unstructured text, images, and video.
  • Veracity: uncertainty and trustworthiness of the data.
  • Value: the usable insight extracted after processing.
• Enabling tools: distributed frameworks such as Hadoop (storage) and Spark (in-memory processing) split work across many machines.

VII. Data Collection and Sources

Reliable analysis depends first on gathering the right data from appropriate sources.

A. Data collection methods

Data collection is the process of gathering information to answer a defined question.

• Surveys and questionnaires: structured questions to samples of people.
• Interviews and observation: direct qualitative gathering, useful for context.
• Sensors and IoT devices: automatic capture of physical measurements.
• Web scraping and APIs: extracting data from websites or programmatic feeds.
• Transactional logs: records generated by systems, e.g. e-commerce purchases.

B. Primary vs. secondary data

Data is classified by whether the analyst gathered it first-hand.

1. Primary data: collected directly by the researcher for the specific problem.
   • Traits: original, current, and tailored, but costly and time-consuming.
   • Example: a company running its own customer satisfaction survey.
2. Secondary data: already collected by someone else for another purpose.
   • Traits: cheap and fast to obtain, but may be outdated or ill-fitted.
   • Example: using government census reports or published datasets.

VIII. Data Preparation and Analytics

Raw data is rarely usable directly; it must be cleaned, transformed, and then analysed.

A. Basics of data cleaning and transformation

Cleaning and transformation prepare raw data into a consistent, analysable form.

• Handling missing values: delete affected rows or impute with mean, median, or a placeholder.
• Removing duplicates and outliers: eliminate repeated records and flag extreme values that distort results.
• Standardising formats: unify dates, units, and text casing, e.g. "USA" and "United States" into one label.
• Transformation operations:
  • Normalisation/scaling: rescaling numeric features to a common range so no feature dominates.
  • Encoding: converting categories to numbers, e.g. one-hot encoding.
  • Aggregation: summarising rows, e.g. daily sales into monthly totals.

B. Introduction to data analytics

Data analytics is the examination of prepared data to draw conclusions and guide action.

• Descriptive analytics: what happened, e.g. last quarter's total sales.
• Diagnostic analytics: why it happened, e.g. cause of a sales drop.
• Predictive analytics: what is likely to happen, using ML models to forecast.
• Prescriptive analytics: what to do next, recommending optimal actions.
• Common tools: spreadsheets, SQL for querying, Python/R for statistics, and visualisation tools like Tableau or Power BI to communicate findings clearly.