Scientific Critical Thinking

Systematic evaluation of research rigor through methodology assessment, bias detection, and evidence quality frameworks.

Triggers

• User asks to evaluate a study's quality
• User needs to assess evidence strength
• User is reviewing trial methodology
• User wants to identify limitations or biases
• User is critiquing research for an editorial

Core Capabilities

1. Methodology Critique

Validity Assessment:

Study Design Hierarchy:

1. Systematic reviews/meta-analyses of RCTs
2. Individual RCTs
3. Cohort studies
4. Case-control studies
5. Cross-sectional studies
6. Case series/reports
7. Expert opinion

2. Bias Detection

Cognitive Biases in Research:

• Confirmation bias: Interpreting data to support hypothesis
• HARKing: Hypothesizing after results known
• Publication bias: Positive results published more
• Spin: Overstating or misrepresenting findings

Selection Biases:

• Sampling bias (non-representative)
• Volunteer bias (healthier participants)
• Attrition bias (differential dropout)
• Survivorship bias (only studying survivors)

Measurement Biases:

• Observer/detection bias
• Recall bias
• Social desirability bias
• Hawthorne effect

Analysis Biases:

• P-hacking (multiple testing)
• Outcome switching
• Selective reporting
• Data dredging

3. Statistical Evaluation Checklist

• Sample size adequate? (power analysis done?)
• Statistical test appropriate for data type?
• Multiple comparison correction applied?
• Effect sizes reported (not just p-values)?
• Confidence intervals provided?
• Missing data handled appropriately?
• Assumptions of tests verified?

4. Evidence Quality Assessment (GRADE)

Quality Levels:

Downgrade Factors:

• Risk of bias
• Inconsistency across studies
• Indirectness (surrogate outcomes)
• Imprecision (wide CIs)
• Publication bias

Upgrade Factors:

• Large effect size
• Dose-response relationship
• Residual confounding would reduce effect

5. Logical Fallacy Detection

Causation Fallacies:

• Post hoc ergo propter hoc (after = because of)
• Correlation ≠ causation
• Reverse causation
• Confounding as causation

Generalization Errors:

• Hasty generalization (small sample)
• Ecological fallacy (group to individual)
• Exception fallacy (individual to group)

Statistical Fallacies:

• Texas sharpshooter (finding patterns in noise)
• Base rate neglect
• Regression to mean confusion
• Multiple endpoints fishing

6. Research Design Questions

When evaluating a study, ask:

1. Question: Is the research question clear and answerable?
2. Design: Is the study design appropriate for the question?
3. Population: Is the sample representative of target population?
4. Intervention: Was the intervention clearly defined and consistent?
5. Comparison: Was the control group appropriate?
6. Outcome: Were outcomes clinically meaningful and measured reliably?
7. Follow-up: Was follow-up long enough and complete enough?
8. Analysis: Was the analysis appropriate and pre-specified?

7. Claim Evaluation Framework

For any scientific claim:

1. Identify the assertion - What exactly is being claimed?
2. Evaluate supporting evidence - What studies support it?
3. Check logical connection - Does evidence actually support claim?
4. Assess proportionality - Is strength of claim proportional to evidence?
5. Detect overgeneralization - Are limits of findings respected?
6. Flag red flags - Conflicts of interest, spin, p-hacking?

Application to Cardiology Content

Evaluating Trial Results

1. Check randomization and blinding adequacy
2. Assess primary endpoint clinical relevance
3. Evaluate intention-to-treat vs per-protocol
4. Look for protocol changes mid-trial
5. Examine subgroup analyses critically
6. Consider funding source influence

For Editorials/Newsletters

• Acknowledge study limitations explicitly
• Don't overstate findings
• Note where evidence is weak
• Distinguish association from causation
• Highlight what questions remain

Critique Output Format

When critiquing research:

1. Summary: Brief overview of what study did
2. Strengths: What was done well
3. Critical concerns: Major methodological issues
4. Important limitations: Secondary concerns
5. Minor issues: Small points for completeness
6. Overall assessment: Balanced conclusion on reliability