DATA POINT - TESTS OF CLINICAL SIGNIFICANCE A HICS INITIATIVE
Dr Manoj Raju Prabandhakam
Aster Narayanadri Hospital, Tirupati
Appropriate interpretation of research results from the clinical significance point is crucial for clinical decision-making and Evidence-based practice. Clinical significance measures the practical, real-world impact of a treatment, distinct from statistical significance, which only indicates that findings are likely not due to chance (p< 0.05). From a clinical point of view, the statistically significant difference among groups is not of prime importance. (1)
Before diving into clinical significance, the six
points about p-value to consider are:(2)
1.
P values
can be indicative of the level of discrepancy between the data and a specific
statistical model.
2.
P values do not quantify
the probability of the tested hypothesis being true or the probability of the
data resulting solely from random chance.
3.
Making scientific conclusions or
business/policy decisions solely based on whether a P value
surpasses a specific threshold is not advisable.
4.
Proper inference necessitates complete
reporting and transparency of all relevant information.
5.
A P value,
or statistical significance, does not gauge the magnitude of an effect or the
significance of a finding.
6.
In isolation, a P value
does not serve as a reliable measure of evidence regarding a model or
hypothesis.
Let’s understand this with few examples.
For example, imagine a study evaluates the
effectiveness of a new medication (drug X) for treating depression. After
analyzing magnitude of improvement is minimal and may not translate into a
noticeable change in patients’ well-being or functioning which specifies
clinical significace.
Let us take another example for a study aims to
investigate the effectiveness of a new therapy for managing chronic pain in treatment
group and the control group. The researchers found that there is no
statistically significant difference (P > 0.05)) in pain
reduction but there is a noticeable and meaningful enhancement in pain levels
among the treatment group which is clinically significant
Which is more important?
Different research questions may prioritize different
aspects of significance.
Statistical significance takes precedence:
Purely exploratory study
designed to uncover potential associations or patterns. Researchers would focus
on determining whether observed differences or relationships hold statistical
significance, which would guide further investigation or hypothesis generation.
Clinical significance takes precedence:
Study aimed at informing
clinical decision-making or shaping public health policies, such as assessing
the efficacy of a new treatment or vaccine.
The primary focus here is to
ascertain whether that observed effect or relationship holds significance in
terms of its impact on patients’ health outcomes or the well-being.
How to determine clinical significance?
A) Effect size index:
Quantify the magnitude of a
relationship between variables or the difference between groups, independent of
sample size.
Provide a standardized
measure of practical significance (how big or meaningfulness of effect).
Common effect size indices are shown in table 1(3)
Table1:
|
Index |
Description |
Standard |
Comment |
|
|
Between
groups |
Cohen's
d or Glass's Δ |
d or Δ
= (Mean1 - Mean2) / SD* |
Small
0.2 |
For
continuous outcomes |
|
Odds
ratio (OR) |
OR = odds1 /
odds2 |
Small
1.5 |
Degree
of association between binary outcomes |
|
|
Relative
risk or risk ratio (RR) |
RR = p1 / p2 |
Small 2 |
For
binary outcomes, ratio of two proportions |
|
|
Measures
of association |
Pearson's r correlation |
Range -1 to 1 |
Small ±
0.2 |
Measures
the degree of linear relationship |
|
Pearson r correlation
coefficient |
Range 0 to 1 |
Small
0.04 |
Proportion
of variance |
|
1. Between groups
o 1) Cohen's d or Glass's
Δ: Defined by difference between two group means divided by standard deviation
for continuous outcomes. Cohen's d is calculated by dividing pooled standard
deviation under assumption of the equal variances while Glass's Δ is obtained
by dividing the standard deviation of control group.
o 2) Odds ratio: Defined by
ratio of odds of two compared groups for binary outcomes.
o 3) Relative ratio:
Defined by ratio of proportions of two compared groups for binary outcomes.
2. Measures of association
o 1) Pearson's r correlation:
Effect size representing association of two variables.
o 2) Pearson r correlation
coefficient: The amount of variation explained.
If Cohen's
d is calculated to be zero, it means that there is no mean difference between
two comparative groups and the position of the mean of experimental group is
exactly the same with the mean of control group. Therefore, 50% of observations
in control group locate below the mean of experimental group (Table 3). The
relative 'small' effect size '0.2' means the mean of experimental group is
located at 0.2 standard deviation above the mean of control group. The Z score
of 0.2 is at 58th percentile which have 58% of observations
below in control group (Figure 1).
Similarly, the Cohen's d values 0.5 and 0.8 locate at 69th and
79th percentile of the distribution of the control group,
respectively.
Pearson
correlation coefficient (r)
The image shows five scatter plots
illustrating how the Pearson correlation coefficient (r)
reflects the strength and direction of a linear relationship between two
variables. Here is the interpretation of each panel in practical terms:
1. Strong Positive
Correlation
·
The points lie very close to an
upward-sloping straight line.
·
As one variable increases, the other consistently
increases.
·
Pearson r is close to +1
(for example +0.8 to +1.0).
2. Medium Positive
Correlation
·
Points show an upward trend but with more
scatter around the line.
·
The relationship is positive but not
perfectly consistent.
·
r around +0.4 to +0.7.
3. Weak / No
Correlation
·
Points are widely scattered with almost a
flat line.
·
Change in one variable does not
predict change in the other.
·
r close to 0
(−0.1 to +0.1).
4. Medium Negative
Correlation
·
The trend slopes downward with moderate
scatter.
·
As one variable increases, the other tends
to decrease.
·
r around −0.4 to −0.7.
5. Strong Negative
Correlation
·
Points lie close to a downward straight line.
·
Increase in one variable is associated with a
consistent decrease in the other.
·
r close to −1
(−0.8 to −1.0).
Key
Points for Interpretation
·
Direction:
o
Positive r → variables move in same direction
o
Negative r → variables move in opposite
direction
·
Strength (absolute value of
r):
o
0–0.3 → weak
o
0.3–0.7 → moderate
o
0.7–1.0 → strong
·
Important cautions:
o
Correlation shows association, not
causation.
o
Pearson correlation measures only linear
relationships.
o
Outliers can markedly change r value.
Minimal Clinically Important
Difference
Operational definition of a minimal clinically important
difference was “…. The smallest difference in score in the domain of interest
which patients perceive as beneficial and which would
mandate, in the absence of troublesome side effects and excessive cost, a
change in the patient's management.” This definition involved two
constructs: 1) a minimal amount of patient reported change and 2) something
significant enough to change patient management.
Minimal Detectable Change (MDC)
MDC is the
smallest amount of change that exceeds the expected error in measurement. A
result that doesn’t meet the MDC threshold may not be reliable, even if it
looks improved.
CONFIDENCE INTERVALS
A confidence interval
(CI) is a range of values within which the true population parameter is
expected to lie with a certain level of confidence, usually 95 percent. Unlike
a single point estimate, the CI expresses the precision and reliability of
study results. A narrow interval indicates high precision, while a wide
interval shows uncertainty.
Interpretation in
Clinical Research
If the CI for a difference between two treatments does not cross the line of no
effect, the result is considered statistically significant. For example, a risk
ratio of 0.70 with 95 percent CI 0.60 to 0.82 shows clear benefit because the
entire interval is below 1.00. However, a risk ratio of 0.70 with CI 0.45 to
1.10 is not conclusive because the interval includes 1.00.
Clinical Importance of Width
In critical care trials, a mortality reduction of 10 percent with CI 2 to 18
percent is more convincing than the same reduction with CI minus 5 to 25
percent. The second interval is too wide and includes possible harm. Therefore,
clinicians must examine the CI rather than only the p value.Relationship with
Sample Size
Larger samples reduce standard error and produce narrower intervals. Small
pilot ICU studies often show wide CIs, explaining why early promising results
may later disappear in larger trials.
Practical Steps for Interpretation
- Look at whether CI crosses line of no effect.
- Assess whether the limits exceed the minimal clinically important
difference.
- Consider direction of possible harm or benefit.
- Combine CI with clinical judgment and patient values.
ODDS RATIO (OR)
1. Definition
Odds Ratio is a measure of the strength
of association between an exposure and an outcome.
It tells how many times higher (or lower) the odds of an outcome are in the
exposed group compared with the non-exposed group.
·
Used mainly in case–control studies
·
Can also be used in cohort and
cross-sectional studies
2. 2 × 2 Table
|
Disease Present |
Disease Absent |
|
|
Exposed |
a |
b |
|
Not Exposed |
c |
d |
3. Formula
Odds Ratio (OR)
OR=a/bc/d=a×db×cOR = \frac{a/b}{c/d} =
\frac{a \times d}{b \times c}OR=c/da/b=b×ca×d
Where
·
a = exposed with disease
·
b = exposed without disease
·
c = non-exposed with disease
·
d = non-exposed without disease
4. Interpretation
OR = 1
·
No association between exposure and outcome.
OR > 1
·
Exposure is a risk factor.
·
Odds of disease are OR times higher in
exposed group.
OR < 1
·
Exposure is protective.
References
1. Sharma H. Statistical
significance or clinical significance? A researcher's dilemma for appropriate
interpretation of research results. Saudi J Anaesth. 2021
Oct-Dec;15(4):431-434. doi: 10.4103/sja.sja_158_21. Epub 2021 Sep 2. PMID:
34658732; PMCID: PMC8477766.
2. AbdulRaheem, Yousif. Statistical Significance
versus Clinical Relevance: Key Considerations in Interpretation Medical
Research Data. Indian Journal of Community Medicine 49(6):p 791-795, Nov–Dec
2024. | DOI: 10.4103/ijcm.ijcm_601_23
3.
Statistical notes for
clinical researchers: effect size Hae-Young
Kim 2015;40(4) 331. DOI: https://doi.org/10.5395/rde.2015.40.4.328
Comments
Post a Comment