An Explanation of P-Values and Statistical Significance (2024)

In statistics, p-values are commonly used in hypothesis testing for t-tests, chi-square tests, regression analysis, ANOVAs, and a variety of other statistical methods.

Despite being so common, people often interpret p-values incorrectly, which can lead to errors when interpreting the findings from an analysis or a study.

This post explains how to understand and interpret p-values in a clear, practical way.

Hypothesis Testing

To understand p-values, we first need to understand the concept of hypothesis testing.

Ahypothesis testis a formal statistical test we use to reject or fail to reject some hypothesis. For example, we may hypothesize that a new drug, method, or procedure provides some benefit over a current drug, method, or procedure.

To test this, we can conduct a hypothesis test where we use a null and alternative hypothesis:

Null hypothesis – There is no effect or difference between the new method and the old method.

Alternative hypothesis – There does exist some effect or difference between the new method and the old method.

A p-value indicates how believable the null hypothesis is, given the sample data. Specifically, assuming the null hypothesis is true, the p-value tells us the probability of obtaining an effect at least as large as the one we actually observed in the sample data.

If the p-value of a hypothesis test is sufficiently low, we can reject the null hypothesis. Specifically, when we conduct a hypothesis test, we must choose a significance level at the outset. Common choices for significance levels are 0.01, 0.05, and 0.10.

If the p-values isless thanour significance level, then we can reject the null hypothesis.

Otherwise, if the p-value isequal to or greater thanour significance level, then we fail to reject the null hypothesis.

How to Interpret a P-Value

The textbook definition of a p-value is:

A p-value is the probability of observing a sample statistic that is at least as extreme as your sample statistic, given that the null hypothesis is true.

For example, suppose a factory claims that they produce tires that have a mean weight of 200 pounds. An auditor hypothesizes that the true mean weight of tires produced at this factory is different from 200 pounds so he runs a hypothesis test and finds that the p-value of the test is 0.04. Here is how to interpret this p-value:

If the factory does indeed produce tires that have a mean weight of 200 pounds, then 4% of all audits will obtain the effect observed in the sample, or larger, because of random sample error. This tells us that obtaining the sample data that the auditor did would be pretty rare if indeed the factory produced tires that have a mean weight of 200 pounds.

Depending on the significance level used in this hypothesis test, the auditor would likely reject the null hypothesis that the true mean weight of tires produced at this factory is indeed 200 pounds. The sample data that he obtained from the audit is not very consistent with the null hypothesis.

How Notto Interpret a P-Value

The biggest misconception about p-values is that they are equivalent to the probability of making a mistake by rejecting a true null hypothesis (known as a Type I error).

There are two primary reasons that p-values can’t be the error rate:

1.P-values are calculated based on the assumption that the null hypothesis is true and that the difference between the sample data and the null hypothesis is simple caused by random chance. Thus, p-values can’t tell you the probability that the null is true or false since it is 100% true based on the perspective of the calculations.

2. Although alow p-value indicates that your sample data are unlikely assuming the null is true, a p-value still can’t tell you which of the following cases is more likely:

  • The null is false
  • The null is true but you obtained an odd sample

In regards to the previous example, here is a correct and incorrect way to interpret the p-value:

  • Correct Interpretation: Assuming the factory does produce tires with a mean weight of 200 pounds, you would obtain the observed difference that youdidobtain in your sample or a more extreme difference in 4% of audits due to random sampling error.
  • Incorrect Interpretation: If you reject the null hypothesis, there is a 4% chance that you are making a mistake.

Examples of Interpreting P-Values

The following examples illustrate correct ways to interpret p-values in the context of hypothesis testing.

Example 1

A phone company claims that 90% of its customers are satisfied with their service. To test this claim, an independent researcher gathered a simple random sample of 200 customers and asked them if they are satisfied with their service, to which 85% responded yes. The p-value associated with this sample data turned out to be 0.018.

Correct interpretation of p-value:Assuming that 90% of the customers actually are satisfied with their service, the researcher would obtain the observed difference that hedidobtain in his sample or a more extreme difference in 1.8% of audits due to random sampling error.

Example 2

A company invents a new battery for phones. The company claims that this new battery will work for at least 10 minutes longer than the old battery. To test this claim, a researcher takes a simple random sample of 80 new batteries and 80 old batteries. The new batteries run for an average of 120 minutes with a standard deviation of 12 minutes and the old batteries run for an average of 115 minutes with a standard deviation of 15 minutes. The p-value that results from the test for a difference in population means is 0.011.

Correct interpretation of p-value:Assuming that the new battery works for the same amount of time or less than the old battery, the researcher would obtain the observed difference or a more extreme differencein 1.1% of studies due to random sampling error.

An Explanation of P-Values and Statistical Significance (2024)

FAQs

An Explanation of P-Values and Statistical Significance? ›

A p-value measures the probability of obtaining the observed results, assuming that the null hypothesis is true. The lower the p-value, the greater the statistical significance of the observed difference. A p-value of 0.05 or lower is generally considered statistically significant.

What does statistical significance or the p-value represent? ›

The P value is defined as the probability under the assumption of no effect or no difference (null hypothesis), of obtaining a result equal to or more extreme than what was actually observed. The P stands for probability and measures how likely it is that any observed difference between groups is due to chance.

What is p-value and statistical significance for dummies? ›

The end result of a statistical significance test is a p value, which represents the probability that random fluctuations alone could have generated results that differed from the null hypothesis (H0), in the direction of the alternate hypothesis (HAlt), by at least as much as what you observed in your data.

What is the statement on p-values and statistical significance? ›

Principle 5: A P-value, or statistical significance, does not measure the size of an effect or the importance of a result. The threshold of statistical significance that is commonly used is a P-value of 0.05. This is conventional and arbitrary. It does not convey any meaningful evidence of the size of the effect.

What is the p-value equation for statistical significance? ›

The p-value is calculated using the sampling distribution of the test statistic under the null hypothesis, the sample data, and the type of test being done (lower-tailed test, upper-tailed test, or two-sided test). The p-value for: a lower-tailed test is specified by: p-value = P(TS ts | H 0 is true) = cdf(ts)

How do you interpret p-value and statistical significance? ›

A p-value measures the probability of obtaining the observed results, assuming that the null hypothesis is true. The lower the p-value, the greater the statistical significance of the observed difference. A p-value of 0.05 or lower is generally considered statistically significant.

How do you explain p-value to non-technicians? ›

Academically, the P-value is the probability of obtaining results as extreme as the observed data, assuming that the null hypothesis is correct1.

What is the difference between p-value and significant value? ›

The term significance level (alpha) is used to refer to a pre-chosen probability and the term "P value" is used to indicate a probability that you calculate after a given study.

What does a p-value of 0.9 mean? ›

If the null hypothesis holds, then the p-values of your statistic have a uniform distribution. A p-value of 0.2 just means that you'd get a statistic greater than that 20% of the time under the null hypothesis; a p-value of 0.9 means you'd see a greater value 90% of the time.

What happens if the p-value is 1? ›

A P-value measures how statistically significant the resulting data from the experiment is. A P-value of 1 says that the two sets of data are identical and that no change has been observed, supporting the null hypothesis that there have been no change. The value ranges from 0 to 1.

How to report significant p-value? ›

If P>. 01 then the P value should always be expressed to 2 digits whether or not it is significant. When rounding, 3 digits is acceptable if rounding would change the significance of a value (eg, you may write P=. 049 instead of .

What is the p-value for statistical significance and clinical significance? ›

What a P < 0.05 implies is that the possibility of the results in a study being due to chance is <5%. In clinical practice, the “clinical significance” of a result is dependent on its implications on existing practice-treatment effect size being one of the most important factors that drives treatment decisions.

How to determine statistical significance? ›

Start by looking at the left side of your degrees of freedom and find your variance. Then, go upward to see the p-values. Compare the p-value to the significance level or rather, the alpha. Remember that a p-value less than 0.05 is considered statistically significant.

What does a high p-value mean? ›

A high P-value, between 0.5 and 1.0, means that it is more likely that the results occurred by random chance, or that the difference is not statistically significant in the case of a hypothesis test.

What does the t-test tell you? ›

A t-test is an inferential statistic used to determine if there is a significant difference between the means of two groups and how they are related. T-tests are used when the data sets follow a normal distribution and have unknown variances, like the data set recorded from flipping a coin 100 times.

What is the p-value for a statistically significant confidence interval? ›

Thus, 'statistical significance' (corresponding to p<0.05) can be inferred from confidence intervals – but, in addition, these intervals show the largest and smallest effects that are likely, given the observed data.

When to use 0.01 and 0.05 level of significance? ›

How to Find the Level of Significance? If p > 0.05 and p ≤ 0.1, it means that there will be a low assumption for the null hypothesis. If p > 0.01 and p ≤ 0.05, then there must be a strong assumption about the null hypothesis. If p ≤ 0.01, then a very strong assumption about the null hypothesis is indicated.

What does p .05 mean in statistics? ›

A p-value of . 05 means that there is a 5% probability that a test statistic as extreme as or more extreme than the observed test statistic could occur, assuming that the null hypothesis is true.

What does the p-value tell us in a test of statistical inference? ›

P-value shows how likely it is that your set of observations could have occurred under the null hypothesis. P-Values are used in statistical hypothesis testing to determine whether to reject the null hypothesis. The smaller the p-value, the stronger the likelihood that you should reject the null hypothesis.

What does the AP value of 0.5 mean? ›

Mathematical probabilities like p-values range from 0 (no chance) to 1 (absolute certainty). So 0.5 means a 50 per cent chance and 0.05 means a 5 per cent chance. In most sciences, results yielding a p-value of . 05 are considered on the borderline of statistical significance.

Top Articles
Latest Posts
Recommended Articles
Article information

Author: Foster Heidenreich CPA

Last Updated:

Views: 5257

Rating: 4.6 / 5 (56 voted)

Reviews: 95% of readers found this page helpful

Author information

Name: Foster Heidenreich CPA

Birthday: 1995-01-14

Address: 55021 Usha Garden, North Larisa, DE 19209

Phone: +6812240846623

Job: Corporate Healthcare Strategist

Hobby: Singing, Listening to music, Rafting, LARPing, Gardening, Quilting, Rappelling

Introduction: My name is Foster Heidenreich CPA, I am a delightful, quaint, glorious, quaint, faithful, enchanting, fine person who loves writing and wants to share my knowledge and understanding with you.