Do People Appropriately Rely on AI-Advice? An Analytical Review of HCI Research on Human-AI Decision-Making
This is a solid CHI synthesis paper: its main contribution is not a new system but a structured review that argues current human-AI reliance studies often miss realism in decision scenarios. The paper’s claims are well matched to its evidence base, though the work remains bounded by review scope and coding choices.
Axes Lens
Rare contribution shape, typical evidence profile. The point here is not a score. It is to show what kind of claim the paper makes, and whether the evidence pattern is unusual or baseline in this 268 -review set.
Contribution shape
- Knowledge form
- descriptive knowledge typical · 92/268
- Novelty type
- synthesis typical · 16/268
- Abstraction level
- field typical · 41/268
- Generalization target
- field argument typical · 55/268
- Validation mode
- survey synthesis typical · 10/268
Evidence profile
- Evidence strength
- moderate typical · 105/268
- Claim alignment
- strong typical · 231/268
- Overclaim risk
- medium typical · 210/268
Review Summary
This paper reads as a field-level analytical synthesis rather than a conventional HCI artifact contribution, and that is exactly where its value lies. The central move is to argue that research on reliance on AI advice has too often been evaluated in settings that do not adequately replicate realistic decision-making scenarios, and then to organize the literature around “appropriate reliance” as the lens for interpretation. That is a meaningful reframing because it shifts the discussion from isolated interaction effects toward the validity of the decision context itself. The evidence base is also reasonably clear: the authors report a standardized and systematic review process and state that they analyzed 56 studies, which supports the scope of the synthesis claim. At the same time, the paper is appropriately cautious in its limitations, acknowledging that some studies may have been missed and that its categorization framework is only one possible way to structure the literature. That makes the contribution credible, but also keeps the overclaim risk from being low. In CHI terms, the paper’s novelty is best understood as a synthesis/framework contribution at the field level, with descriptive and normative implications for how future human-AI reliance studies should be designed and interpreted. It does not validate a deployed intervention or a new algorithm; instead, it offers a disciplined review that can guide future methodological choices and intervention design discussions.
What Changed
Canon before
Prior HCI work on reliance on AI advice is presented as fragmented across tasks, methods, and interaction factors, with insufficient attention to whether study scenarios mirror realistic decision-making contexts.
Departure from common sense
The paper argues that HCI research on human-AI reliance largely overlooks replicating realistic decision-making scenarios, implying that common evaluation setups may not reflect real-world reliance behavior.
Actual novelty
The paper’s novelty is a standardized analytical review that organizes recent human-AI reliance studies around appropriate reliance, synthesizing concepts, methods, and interaction factors while also assessing task validity in application-grounded contexts.
Evidence
The paper is an analytical review rather than an empirical system paper. It reports a standardized and systematic method, summarizes the landscape of human-AI reliance research, and states that it analyzed 56 studies. The review also explicitly acknowledges that it may have missed some studies and that its categorization is only one possible framing.
“ In this work, we use a standardized and systematic method to analyze studies by specifically focusing on research in human-AI appropriate reliance and covering recent work ”
actual novelty · Introduction / Our Contribution · confidence 0.69
“ While HCI research explores how people rely on AI advice, we argue that it largely overlooks an important aspect: replicating realistic decision-making scenarios”
departure from common sense · Abstract / Introduction · confidence 0.74
“ First, like most reviews, we may have missed some relevant studies, even with a diverse search strategy and concrete inclusion criteria”
limitation · Limitations section · confidence 0.86
“ This provided us with a total of 56 studies for final analysis”
validation scope · Methodology (Search/Selection) · confidence 0.82
Limits
Method limits
As a review, the paper may have missed relevant studies despite a diverse search strategy and concrete inclusion criteria, and its categorization framework is not unique. The evidence base is limited to the selected HCI literature and the review’s coding choices.
Deployment limits
The findings are most directly applicable to human-AI decision-making research and intervention design in HCI; they do not by themselves validate a deployed decision-support system or guarantee transfer across all domains, tasks, or AI advice settings.
Boundary conditions
The conclusions are bounded by the review corpus, the databases searched, the inclusion criteria, and the paper’s focus on application-grounded decision-making scenarios and objective reliance measures.
Position in field
This is a field-level synthesis paper that reframes human-AI reliance around appropriate reliance and realistic decision-making validity. Its contribution is primarily analytical and integrative, not a new interface or algorithm.