Unit 5: Time Series Regression

1. Time Series Data Characteristics

Time series data is a sequence of data points collected or recorded at specific time intervals. Unlike standard supervised learning datasets where observations are assumed to be independent (i.i.d.), time series data possesses temporal dependence, meaning the current value is often correlated with past values.

Key Components of Time Series

A time series signal can be decomposed into four primary components:

1. Trend (): The long-term movement or direction of the data. Trends can be deterministic (linear, exponential) or stochastic.
   • Example: Increasing global temperatures over 50 years.
2. Seasonality (): Recurring patterns or cycles that occur at fixed intervals (e.g., daily, weekly, monthly, quarterly).
   • Example: Retail sales spiking every December.
3. Cyclicity (): Fluctuations occurring at irregular intervals, usually influenced by macroeconomic factors (business cycles). Unlike seasonality, the period is not fixed.
   • Example: Economic recessions and expansions.
4. Irregularity / Noise / Residuals (): The random variation left over after extracting trend, seasonality, and cyclicity. This should ideally resemble white noise.

Stationarity

A critical concept in time series analysis. A time series is stationary if its statistical properties do not change over time.

• Constant Mean: No trend.
• Constant Variance: No heteroscedasticity (volatility does not change).
• Constant Autocovariance: The relationship between and depends only on the lag , not on the actual time .

Why it matters: Most classical models (ARIMA) assume stationarity to make reliable future predictions.

2. Univariate vs. Multivariate Time Series

Univariate Time Series

Consists of a single variable observed over time. The objective is to predict future values of this variable based solely on its own history.

• Notation:
• Example: Predicting the price of Bitcoin using only past Bitcoin prices.

Multivariate Time Series

Consists of two or more variables observed over time. One variable is usually the target, while others are exogenous (predictor) variables that may influence the target.

• Notation: (target) and (features).
• Example: Predicting "Ice Cream Sales" () using "Past Sales" () and "Temperature" ().

3. Regular vs. Irregular Intervals

Regular Time Series

Data points are collected at equally spaced intervals (frequency is constant). This is the standard format required for most regression algorithms (ARIMA, RNNs).

• Examples: Hourly temperature readings, daily stock closing prices.

Irregular Time Series

Data points arrive sporadically or at arbitrary timestamps.

• Examples: Credit card transactions, server error logs, patient health records (visits occur only when sick).
• Handling Irregularity: Before applying standard regression, irregular data is often resampled (e.g., binning or interpolation) to convert it into a regular interval structure.

4. Preparing Time Series Data

Data preparation involves transforming raw temporal data into a format suitable for supervised learning algorithms.

A. Handling Missing Values

Standard mean imputation is often inappropriate because it destroys temporal continuity.

• Forward Fill (Last Observation Carried Forward): Propagates the last valid observation forward.
• Interpolation: Linear or spline interpolation based on time index.

B. Feature Engineering

Transforming a time series into a supervised learning problem ().

1. Lag Features: Using past values as input features.
   • Target:
   • Feature 1: (Lag 1)
   • Feature 2: (Lag 7 / Weekly seasonality)
2. Rolling Window Statistics: Calculating summary statistics over a moving window.
   • Rolling Mean (Moving Average).
   • Rolling Standard Deviation (Volatility).
3. Datetime Features: Extracting components from the timestamp.
   • Hour of day, Day of week, Is_Weekend, Month.

C. Transformations for Stationarity

• Differencing: Computing the difference between consecutive observations () to remove trend.
• Log Transformation: Applying to stabilize increasing variance (heteroscedasticity).
• Decomposition: Separating Trend and Seasonality mathematically.

5. Data Splitting Strategies

CRITICAL RULE: Never use random train_test_split or standard K-Fold Cross-Validation for time series. Random shuffling causes data leakage (using future information to predict the past).

A. Temporal Train-Test Split

Split the data at a specific cutoff point.

• Train: Data from to
• Test: Data from to
• Con: Uses only one snapshot of model performance.

B. Walk-Forward Validation (Time Series Cross-Validation)

Also known as "Rolling Origin" validation.

1. Train on initial window. Predict the next step.
2. Expand the training window to include the actual value of the predicted step.
3. Retrain and predict the subsequent step.
4. Repeat until the end of the dataset.

• Expanding Window: Training set grows larger with every step.
• Sliding Window: Training set size remains constant (oldest data drops off).

6. Autoregressive Models (AR)

An Autoregressive model forecasts the variable of interest using a linear combination of its past values. It assumes the current value depends linearly on its own previous values.

Formula (AR())

• : The order of the AR model (number of lags used).
• : Coefficients.
• : White noise error term.

Determining

Use the Partial Autocorrelation Function (PACF) plot.

• The PACF removes the indirect effect of intermediate lags.
• If the PACF cuts off (drops to zero) after lag , an AR() model is appropriate.

7. Moving Average Models (MA)

A Moving Average model forecasts the variable based on a linear combination of past forecast errors (shocks). It models the "noise" structure.

Formula (MA())

• : The order of the MA model.
• : Coefficients.
• : Prediction errors (residuals) from previous time steps.

Determining

Use the Autocorrelation Function (ACF) plot.

• If the ACF cuts off after lag , an MA() model is appropriate.

8. ARIMA (AutoRegressive Integrated Moving Average)

ARIMA combines AR and MA models while handling non-stationary data through "Integration" (Differencing).

Notation: ARIMA()

1. (AR order): Number of lag observations included in the model.
2. (Integrated order): Number of times the raw observations are differenced to make the data stationary.
3. (MA order): Size of the moving average window.

The Modeling Process (Box-Jenkins Method)

1. Identification:
   • Check for stationarity (Augmented Dickey-Fuller test).
   • If non-stationary, difference the data (increase ) until stationary.
   • Examine ACF and PACF plots to estimate and .
2. Estimation: Use Maximum Likelihood Estimation (MLE) or Least Squares to find coefficients ( and ).
3. Diagnostic Checking: Analyze residuals. They should be white noise (no correlation, constant variance).

9. SARIMA (Seasonal ARIMA)

Standard ARIMA struggles with strong seasonal patterns (e.g., sales always peaking in December). SARIMA extends ARIMA by adding seasonal hyperparameters.

Notation: SARIMA

Non-Seasonal Components:

• : Trend autoregression order.
• : Trend difference order.
• : Trend moving average order.

Seasonal Components:

• : The number of time steps for a single seasonal period (e.g., for monthly data, for daily data with weekly seasonality).
• : Seasonal autoregressive order (lags at multiples of ).
• : Seasonal difference order (subtracting from ).
• : Seasonal moving average order.

Application Note

SARIMA is computationally more expensive than ARIMA but essential for data with obvious cycles (electricity demand, temperature, retail sales).

Model Selection (Auto-ARIMA)

In practice, manually interpreting ACF/PACF plots can be subjective. Grid Search (via AIC/BIC scores) is often used to find the optimal parameters.

• AIC (Akaike Information Criterion): Estimator of prediction error. Lower AIC indicates a better model (balances goodness of fit vs. model complexity).