Unit 5: Role of Statistics in Data Science - Subjective Questions

Statistics is the discipline concerned with collecting, organizing, analyzing, interpreting, and presenting data.

Its role in the data science life cycle includes:

• Data collection: Helps design surveys, experiments, and sampling strategies.
• Data exploration: Uses descriptive measures and visualizations to identify patterns and anomalies.
• Inference: Draws conclusions about a population from sample data.
• Hypothesis testing: Evaluates claims and determines whether observed effects are statistically significant.
• Model building: Provides methods such as regression, classification, and analysis of variance.
• Uncertainty quantification: Uses confidence intervals, standard errors, and probability distributions.
• Decision-making: Converts analytical findings into evidence-based conclusions.

Thus, statistics provides the theoretical foundation needed to distinguish genuine patterns from random variation in data.

Hypothesis testing is a statistical procedure used to evaluate a claim about a population parameter using sample data.

It begins with two competing statements:

• Null hypothesis (): Represents no effect, no difference, or the existing assumption.
• Alternative hypothesis ( or ): Represents the effect or difference that the researcher wants to investigate.

A test statistic is calculated from the sample and compared with its distribution under . A p-value or critical-value rule is then used to decide whether to reject .

In data science, hypothesis testing is used to:

• Compare models or business strategies.
• Evaluate A/B experiments.
• Test relationships between variables.
• Validate data-driven claims.
• Determine whether observed patterns could reasonably result from random chance.

The conclusion is expressed as either reject or fail to reject ; the test does not prove a hypothesis with absolute certainty.

The two hypotheses make competing claims about a population.

• Null hypothesis (): Usually states that there is no difference, association, or effect. It is treated as true while the test statistic is evaluated.
• Alternative hypothesis (): States that a difference, association, or effect exists. It may be two-sided or directional.

Example: Suppose a company claims that the mean battery life of a device is 10 hours.

If the sample produces sufficiently strong evidence against , it is rejected in favor of . Otherwise, the analyst fails to reject .

Failing to reject does not prove that the mean is exactly 10 hours. It only means that the sample does not provide sufficient evidence of a difference at the selected significance level.

A statistical hypothesis test can be conducted through the following steps:

1. State the research question and identify the relevant population parameter.
2. Formulate hypotheses: Specify and before examining the test result.
3. Select a significance level: Common choices are or .
4. Choose an appropriate test: The choice depends on the variable types, study design, sample size, and statistical assumptions.
5. Check assumptions: Examples include independence, normality, and equality of variances.
6. Calculate the test statistic: A general standardized form is
   
7. Calculate the p-value or compare the statistic with a critical value.
8. Make a decision: Reject when ; otherwise, fail to reject .
9. Interpret the result in context: Report the estimated effect, uncertainty, practical importance, and limitations.

The final conclusion should not rely only on the p-value; effect sizes and confidence intervals should also be reported.

Statistical significance indicates that an observed result would be sufficiently unusual if the null hypothesis were true.

The significance level, denoted by , is the maximum probability of a Type I error that the researcher is willing to accept. It is selected before conducting the test.

The decision rule is:

• If , reject and call the result statistically significant.
• If , fail to reject .

For example, when , the testing procedure accepts a 5% long-run risk of rejecting a true null hypothesis under its assumptions.

Important points are:

• Statistical significance does not prove that is true.
• It does not indicate the size or practical importance of an effect.
• The result can be influenced by sample size.
• Choosing after seeing the data weakens the validity of the test.

The p-value is the probability, assuming and the statistical model are true, of obtaining a test statistic at least as extreme as the observed statistic.

Symbolically, for a right-tailed test with observed statistic :

Correct interpretation: A small p-value means that the observed data are relatively incompatible with . If , the null hypothesis is rejected at significance level .

Common misinterpretations:

• It is not the probability that is true.
• It is not the probability that the result occurred by chance.
• It does not measure the magnitude or importance of an effect.
• A large p-value does not prove .
• A p-value of is not meaningfully different from merely because one crosses a threshold.

The p-value should therefore be considered with effect sizes, confidence intervals, assumptions, and domain knowledge.

A hypothesis test can produce two kinds of decision errors:

• Type I error: Rejecting when is actually true. Its probability is .
• Type II error: Failing to reject when is actually false. Its probability is .

The power of a test is the probability of correctly rejecting a false null hypothesis:

Example: Fraud detection

Let state that a transaction is legitimate and state that it is fraudulent.

• A Type I error occurs when a legitimate transaction is incorrectly blocked.
• A Type II error occurs when a fraudulent transaction is incorrectly allowed.

The costs of these errors may differ. A data scientist should select , the sample size, and the decision procedure by considering these consequences rather than automatically using the same threshold in every application.

The probabilities of Type I and Type II errors are related to the decision threshold and the amount of information in the sample.

• Reducing makes the rejection rule stricter and generally increases when sample size and effect size remain fixed.
• Increasing generally decreases , but it raises the risk of a Type I error.
• Increasing the sample size reduces standard error and generally increases power without requiring a larger .
• A larger true effect size is easier to detect and therefore produces greater power.
• Greater data variability generally reduces power.

Power is

A power analysis commonly uses four quantities: significance level, desired power, effect size, and sample size. If three are specified, the fourth can often be estimated. For example, researchers may select , power , and a minimum meaningful effect to calculate the required sample size.

A one-tailed test has a directional alternative hypothesis. For example:

• Right-tailed:
• Left-tailed:

All of the rejection region is placed in one tail of the sampling distribution. A one-tailed test is appropriate only when effects in the opposite direction are irrelevant to the research question and the direction is specified before observing the data.

A two-tailed test uses a non-directional alternative:

Its rejection region is divided between both tails. It should be used when departures in either direction are possible or important.

For a two-tailed test at , each tail commonly contains of the rejection probability. Analysts must not choose a one-tailed test after seeing the direction of the observed effect, because doing so inflates the Type I error rate.

A confidence interval gives a range of parameter values compatible with the observed data under a specified model.

A two-sided confidence interval generally corresponds to a two-sided hypothesis test at significance level :

• If the null value lies outside the interval, reject .
• If the null value lies inside the interval, fail to reject .

For a mean, a confidence interval often has the form

where is an appropriate critical value.

For example, if a 95% confidence interval for a mean difference is , the null value of zero is excluded. Therefore, the difference is statistically significant in the corresponding two-sided test at .

Confidence intervals are more informative than a binary significance decision because they communicate the estimated effect's direction, magnitude, and uncertainty.

One-way analysis of variance (ANOVA) tests whether the population means of three or more independent groups are equal based on one categorical factor.

For groups, its hypotheses are:

ANOVA compares:

• Between-group variability: Variation of group means around the overall mean.
• Within-group variability: Variation of observations around their respective group means.

If between-group variability is large relative to within-group variability, the evidence against becomes stronger.

ANOVA is preferred to performing many independent t-tests because multiple t-tests inflate the family-wise Type I error rate. A significant ANOVA result establishes that not all means are equal, but it does not by itself identify which groups differ. Post-hoc comparisons are needed for that purpose.

Suppose there are groups, observations in group , and total observations. Let be group mean and be the grand mean.

The total sum of squares is

It is partitioned as

where the between-group sum of squares is

and the within-group sum of squares is

The corresponding degrees of freedom are and . Therefore,

The ANOVA statistic is

Under and the ANOVA assumptions, follows an distribution with degrees of freedom. A large value indicates that differences among group means are large relative to variation within groups.

The main assumptions of one-way ANOVA are:

• Independence: Observations must be independent within and across groups.
• Normality: The residuals within each group should be approximately normally distributed.
• Homogeneity of variance: Population variances should be approximately equal across groups.
• Appropriate measurement: The dependent variable should normally be continuous, while the factor defines categorical groups.

Assumptions can be examined through study design, residual plots, Q-Q plots, and tests such as Levene's test. ANOVA may be reasonably robust to moderate non-normality when samples are sufficiently large and balanced.

If ANOVA is significant, it only shows that at least one mean differs. The analyst should use planned contrasts or a multiple-comparison procedure, such as Tukey's HSD, to identify the differing groups while controlling the family-wise error rate. Effect sizes, such as

and confidence intervals should also be reported. For serious assumption violations, Welch's ANOVA or a suitable nonparametric method may be considered.

The chi-square goodness-of-fit test determines whether the observed frequencies of one categorical variable match a specified theoretical distribution.

If there are categories, the hypotheses are:

• : The population category probabilities equal the specified probabilities.
• : At least one population category probability differs from its specified value.

Expected frequency for category is

where is the sample size and is the probability expected under .

The test statistic is

where and are the observed and expected frequencies. If no parameters are estimated from the data, the degrees of freedom are

A large value indicates substantial disagreement between observed and expected frequencies. The test uses counts rather than percentages, and the observations should be independent with sufficiently large expected counts.

The chi-square test of independence evaluates whether two categorical variables are associated in a population.

Its hypotheses are:

• : The two categorical variables are independent.
• : The two categorical variables are associated.

For a contingency table, the expected frequency in row and column under independence is

The test statistic is

For a table with rows and columns, the degrees of freedom are

A large statistic and small p-value provide evidence of an association. However, the test does not establish causation or directly describe the association's strength. Measures such as Cramér's and examination of standardized residuals can help assess effect size and identify the cells contributing most strongly to the result.

The principal assumptions and requirements of chi-square tests are:

• Data are frequency counts in mutually exclusive categories.
• Each observation contributes to only one cell.
• Observations are independent.
• Categories are defined before examining the outcomes.
• Expected frequencies are sufficiently large for the chi-square approximation.

A common guideline is that no expected frequency should be below 1 and that no more than 20% of expected frequencies should be below 5, although exact recommendations depend on the setting.

Possible remedies for sparse expected counts include:

• Combining categories when it is scientifically meaningful.
• Collecting a larger sample.
• Using Fisher's exact test for a small table.
• Using an appropriate exact or simulation-based test for larger sparse tables.

Repeated measurements from the same subject violate independence and require a method designed for paired or clustered categorical data. Analysts should inspect expected counts rather than judging suitability from observed counts alone.

ANOVA and chi-square tests both evaluate evidence against a null hypothesis, but they address different questions.

ANOVA:

• Compares means of a continuous dependent variable across categorical groups.
• Uses .
• Uses the statistic .
• Common assumptions include independent observations, approximately normal residuals, and equal variances for standard one-way ANOVA.

Chi-square test:

• Analyzes frequency counts for categorical variables.
• Tests either agreement with a specified distribution or independence between categorical variables.
• Uses
  
• Requires independent observations and adequate expected frequencies.

Thus, ANOVA is suitable for questions such as whether average sales differ among three regions, while a chi-square test is suitable for whether purchase category is associated with region. Both tests indicate whether evidence of a difference or association exists, but additional analysis is needed to describe its size and practical importance.

Statistical significance concerns whether an observed effect is unlikely under at a selected significance level. Practical significance concerns whether the effect is large enough to matter in the real application.

For example, an online platform may test a new page design using one million visitors. Suppose conversion increases from 10.00% to 10.05% and the test gives .

• The increase is statistically significant because the large sample allows a very small effect to be estimated precisely.
• It may not be practically significant if implementation costs exceed the revenue generated by the 0.05 percentage-point increase.

A complete analysis should report:

• The p-value and chosen .
• The estimated absolute and relative effect.
• A confidence interval.
• A domain-specific minimum meaningful effect.
• Costs, benefits, risks, and model assumptions.

Statistical significance is therefore evidence about compatibility with , not a direct measure of scientific, operational, or economic importance.

The team can use a randomized experiment followed by one-way ANOVA.

1. Design: Randomly assign comparable users to algorithms A, B, and C, define engagement before the experiment, and ensure that each user contributes an independent observation.
2. Hypotheses:
   
   
3. Significance level: Choose , such as , before examining results.
4. Exploration: Report group sizes, means, standard deviations, distributions, and missing-data handling.
5. Assumptions: Examine independence, residual normality, and homogeneity of variance. Consider Welch's ANOVA if variances differ substantially.
6. Test: Calculate and its p-value.
7. Decision: Reject if ; otherwise, fail to reject it.
8. Follow-up: If significant, use prespecified contrasts or an adjusted post-hoc method such as Tukey's HSD.
9. Report: Include mean differences, confidence intervals, an effect size such as , adjusted p-values, and practical implications.

Randomization supports causal interpretation, whereas ANOVA applied to observational groups may only reveal association because confounding variables could explain the differences.

When many hypotheses are tested, the probability of obtaining at least one false-positive result increases. If independent true null hypotheses are each tested at level , the probability of at least one Type I error is

For example, with 20 independent tests at , this probability is approximately

Data-driven practices such as testing many variables, repeatedly changing model specifications, or selecting a one-tailed test after viewing results can produce misleadingly small p-values. This behavior is often called p-hacking when choices are made to obtain significance.

Appropriate safeguards include:

• Prespecifying hypotheses, outcomes, and analysis methods.
• Using a holdout or replication sample.
• Applying family-wise error controls such as the Bonferroni or Holm procedure.
• Controlling the false discovery rate when testing many exploratory hypotheses.
• Reporting all analyses rather than only significant ones.
• Reporting effect sizes and confidence intervals.

Confirmatory and exploratory analyses should be clearly distinguished so that users can interpret the strength of the evidence correctly.