Neural computations underlying arbitration between model-based and model-free learning.

PubWeight™: 2.20‹?› | Rank: Top 2%

🔗 View Article (PMC 3968946)

Published in Neuron on February 05, 2014

Authors

Sang Wan Lee1, Shinsuke Shimojo2, John P O'Doherty3

Author Affiliations

1: Computation & Neural Systems, MC228-77, California Institute of Technology, Pasadena, CA 91125, USA; Behavioral & Social Neuroscience, MC228-77, California Institute of Technology, Pasadena, CA 91125, USA; Division of Humanities and Social Sciences, MC228-77, California Institute of Technology, Pasadena, CA 91125, USA. Electronic address: swlee@caltech.edu.
2: Computation & Neural Systems, MC228-77, California Institute of Technology, Pasadena, CA 91125, USA; Behavioral & Social Neuroscience, MC228-77, California Institute of Technology, Pasadena, CA 91125, USA; Division of Biology, MC228-77, California Institute of Technology, Pasadena, CA 91125, USA.
3: Computation & Neural Systems, MC228-77, California Institute of Technology, Pasadena, CA 91125, USA; Behavioral & Social Neuroscience, MC228-77, California Institute of Technology, Pasadena, CA 91125, USA; Division of Humanities and Social Sciences, MC228-77, California Institute of Technology, Pasadena, CA 91125, USA.

Articles citing this

Differentiating neural systems mediating the acquisition vs. expression of goal-directed and habitual behavioral control. Eur J Neurosci (2015) 1.43

Neural computations mediating one-shot learning in the human brain. PLoS Biol (2015) 1.39

The neural bases of emotion regulation. Nat Rev Neurosci (2015) 1.33

Model-based choices involve prospective neural activity. Nat Neurosci (2015) 1.15

Ventral striatal dopamine reflects behavioral and neural signatures of model-based control during sequential decision making. Proc Natl Acad Sci U S A (2015) 1.13

Goal-directed learning and obsessive-compulsive disorder. Philos Trans R Soc Lond B Biol Sci (2014) 1.07

Cognitive control predicts use of model-based reinforcement learning. J Cogn Neurosci (2015) 1.05

Active Inference, homeostatic regulation and adaptive behavioural control. Prog Neurobiol (2015) 1.01

Neural correlates of strategic reasoning during competitive games. Science (2014) 1.00

The neural network underlying incentive-based learning: implications for interpreting circuit disruptions in psychiatric disorders. Neuron (2014) 0.99

Model averaging, optimal inference, and habit formation. Front Hum Neurosci (2014) 0.96

Devaluation and sequential decisions: linking goal-directed and model-based behavior. Front Hum Neurosci (2014) 0.95

Habits as action sequences: hierarchical action control and changes in outcome value. Philos Trans R Soc Lond B Biol Sci (2014) 0.94

Cognitive effort: A neuroeconomic approach. Cogn Affect Behav Neurosci (2015) 0.92

Processing speed enhances model-based over model-free reinforcement learning in the presence of high working memory functioning. Front Psychol (2014) 0.90

Vicarious trial and error. Nat Rev Neurosci (2016) 0.86

Scaling prediction errors to reward variability benefits error-driven learning in humans. J Neurophysiol (2015) 0.84

Active inference and learning. Neurosci Biobehav Rev (2016) 0.83

Lateral prefrontal model-based signatures are reduced in healthy individuals with high trait impulsivity. Transl Psychiatry (2015) 0.83

Arbitration between Action Strategies in Obsessive-Compulsive Disorder. Neuroscientist (2015) 0.83

Neural Basis of Strategic Decision Making. Trends Neurosci (2015) 0.83

Multisensory causal inference in the brain. PLoS Biol (2015) 0.81

Active Inference, epistemic value, and vicarious trial and error. Learn Mem (2016) 0.80

The Computational Complexity of Valuation and Motivational Forces in Decision-Making Processes. Curr Top Behav Neurosci (2016) 0.79

Contextual modulation of value signals in reward and punishment learning. Nat Commun (2015) 0.79

Deep brain stimulation of the subthalamic nucleus modulates sensitivity to decision outcome value in Parkinson's disease. Sci Rep (2016) 0.79

Addiction History Associates with the Propensity to Form Habits. J Cogn Neurosci (2016) 0.78

Model-based action planning involves cortico-cerebellar and basal ganglia networks. Sci Rep (2016) 0.78

Characterizing the associative content of brain structures involved in habitual and goal-directed actions in humans: a multivariate FMRI study. J Neurosci (2015) 0.78

Translational Research on Habit and Alcohol. Curr Addict Rep (2016) 0.78

Goal-directed, habitual and Pavlovian prosocial behavior. Front Behav Neurosci (2015) 0.77

Confidence and psychosis: a neuro-computational account of contingency learning disruption by NMDA blockade. Mol Psychiatry (2015) 0.77

Slips of Action and Sequential Decisions: A Cross-Validation Study of Tasks Assessing Habitual and Goal-Directed Action Control. Front Behav Neurosci (2016) 0.77

Safety out of control: dopamine and defence. Behav Brain Funct (2016) 0.77

Reduced model-based decision-making in schizophrenia. J Abnorm Psychol (2016) 0.77

The role of habit in compulsivity. Eur Neuropsychopharmacol (2015) 0.76

The specificity of Pavlovian regulation is associated with recovery from depression. Psychol Med (2016) 0.76

Effects of fictive reward on rat's choice behavior. Sci Rep (2015) 0.76

Stress potentiates decision biases: A stress induced deliberation-to-intuition (SIDI) model. Neurobiol Stress (2016) 0.76

Dopamine, reward learning, and active inference. Front Comput Neurosci (2015) 0.76

Adaptive integration of habits into depth-limited planning defines a habitual-goal-directed spectrum. Proc Natl Acad Sci U S A (2016) 0.75

Intact Ventral Striatal Prediction Error Signaling in Medicated Schizophrenia Patients. Biol Psychiatry Cogn Neurosci Neuroimaging (2016) 0.75

Modeling choice and reaction time during arbitrary visuomotor learning through the coordination of adaptive working memory and reinforcement learning. Front Behav Neurosci (2015) 0.75

When Does Model-Based Control Pay Off? PLoS Comput Biol (2016) 0.75

Dissociable functions of reward inference in the lateral prefrontal cortex and the striatum. Front Psychol (2015) 0.75

Functional connectivity between prefrontal cortex and striatum estimated by phase locking value. Cogn Neurodyn (2016) 0.75

Intelligence moderates reinforcement learning: a mini-review of the neural evidence. J Neurophysiol (2014) 0.75

Challenges and promises for translating computational tools into clinical practice. Curr Opin Behav Sci (2016) 0.75

Gaze data reveal distinct choice processes underlying model-based and model-free reinforcement learning. Nat Commun (2016) 0.75

Divide and conquer: strategic decision areas. Nat Neurosci (2015) 0.75

The involvement of model-based but not model-free learning signals during observational reward learning in the absence of choice. J Neurophysiol (2016) 0.75

Decisions about decisions. Neuron (2014) 0.75

A Unified Theoretical Framework for Cognitive Sequencing. Front Psychol (2016) 0.75

Amygdala-prefrontal cortical functional connectivity during implicit emotion processing differentiates youth with bipolar spectrum from youth with externalizing disorders. J Affect Disord (2016) 0.75

Cognitive inflexibility in Obsessive-Compulsive Disorder. Neuroscience (2016) 0.75

'Proactive' use of cue-context congruence for building reinforcement learning's reward function. BMC Neurosci (2016) 0.75

Stress enhances model-free reinforcement learning only after negative outcome. PLoS One (2017) 0.75

Lever Insertion as a Salient Stimulus Promoting Insensitivity to Outcome Devaluation. Front Integr Neurosci (2017) 0.75

Articles cited by this

A neural substrate of prediction and reward. Science (1997) 31.30

Neural systems of reinforcement for drug addiction: from actions to habits to compulsion. Nat Neurosci (2005) 16.18

Psychophysiological and modulatory interactions in neuroimaging. Neuroimage (1997) 13.51

Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science (2004) 11.67

Inhibition and the right inferior frontal cortex. Trends Cogn Sci (2004) 9.78

A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J Neurosci (1996) 9.54

A model for Pavlovian learning: variations in the effectiveness of conditioned but not of unconditioned stimuli. Psychol Rev (1980) 9.23

Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat Neurosci (2005) 9.16

Temporal difference models and reward-related learning in the human brain. Neuron (2003) 8.50

Bayesian inference with probabilistic population codes. Nat Neurosci (2006) 8.39

Self-control in decision-making involves modulation of the vmPFC valuation system. Science (2009) 8.33

Habits, rituals, and the evaluative brain. Annu Rev Neurosci (2008) 7.14

Temporal prediction errors in a passive learning task activate human striatum. Neuron (2003) 6.44

Human and rodent homologies in action control: corticostriatal determinants of goal-directed and habitual action. Neuropsychopharmacology (2010) 6.00

Stop-signal inhibition disrupted by damage to right inferior frontal gyrus in humans. Nat Neurosci (2003) 5.94

Goal-directed instrumental action: contingency and incentive learning and their cortical substrates. Neuropharmacology (1999) 5.62

Bayesian model selection for group studies. Neuroimage (2009) 5.46

States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron (2010) 5.30

Human insula activation reflects risk prediction errors as well as risk. J Neurosci (2008) 4.68

Right hemispheric dominance of inhibitory control: an event-related functional MRI study. Proc Natl Acad Sci U S A (1999) 4.62

Model-based influences on humans' choices and striatal prediction errors. Neuron (2011) 4.58

The role of the ventromedial prefrontal cortex in abstract state-based inference during decision making in humans. J Neurosci (2006) 4.55

Probabilistic population codes for Bayesian decision making. Neuron (2008) 4.25

How green is the grass on the other side? Frontopolar cortex and the evidence in favor of alternative courses of action. Neuron (2009) 3.85

Frontal cortex and reward-guided learning and decision-making. Neuron (2011) 3.70

Defining the neural mechanisms of probabilistic reversal learning using event-related functional magnetic resonance imaging. J Neurosci (2002) 3.50

Anterior prefrontal function and the limits of human decision-making. Science (2007) 3.47

Determining the neural substrates of goal-directed learning in the human brain. J Neurosci (2007) 3.31

A specific role for posterior dorsolateral striatum in human habit learning. Eur J Neurosci (2009) 3.25

Evidence for a common representation of decision values for dissimilar goods in human ventromedial prefrontal cortex. J Neurosci (2009) 3.02

Multiple model-based reinforcement learning. Neural Comput (2002) 2.74

Disruption in the balance between goal-directed behavior and habit learning in obsessive-compulsive disorder. Am J Psychiatry (2011) 2.24

Optogenetic stimulation of lateral orbitofronto-striatal pathway suppresses compulsive behaviors. Science (2013) 2.21

Mapping value based planning and extensively trained choice in the human brain. Nat Neurosci (2012) 2.09

Role of the lateral prefrontal cortex in executive behavioral control. Physiol Rev (2008) 2.08

Calculating consequences: brain systems that encode the causal effects of actions. J Neurosci (2008) 2.05

Transformation of stimulus value signals into motor commands during simple choice. Proc Natl Acad Sci U S A (2011) 1.94

Inactivation of the infralimbic prefrontal cortex reinstates goal-directed responding in overtrained rats. Behav Brain Res (2003) 1.90

Expectancy-related changes in firing of dopamine neurons depend on orbitofrontal cortex. Nat Neurosci (2011) 1.81

Neural correlates of variations in event processing during learning in basolateral amygdala. J Neurosci (2010) 1.75

Differential roles of human striatum and amygdala in associative learning. Nat Neurosci (2011) 1.71

Avoiding non-independence in fMRI data analysis: leave one subject out. Neuroimage (2009) 1.68

Contributions of striatal subregions to place and response learning. Learn Mem (2004) 1.59

Differential engagement of the ventromedial prefrontal cortex by goal-directed and habitual behavior toward food pictures in humans. J Neurosci (2009) 1.53

Contributions of the ventromedial prefrontal cortex to goal-directed action selection. Ann N Y Acad Sci (2011) 1.49

Rostrolateral prefrontal cortex and individual differences in uncertainty-driven exploration. Neuron (2012) 1.46

Corticostriatal connectivity underlies individual differences in the balance between habitual and goal-directed action control. J Neurosci (2012) 1.42

Reversible online control of habitual behavior by optogenetic perturbation of medial prefrontal cortex. Proc Natl Acad Sci U S A (2012) 1.24

Neural substrates for reversing stimulus-outcome and stimulus-response associations. J Neurosci (2008) 1.00

Action selection: a race model for selected and non-selected actions distinguishes the contribution of premotor and prefrontal areas. Neuroimage (2010) 0.96

How initial confirmatory experience potentiates the detrimental influence of bad advice. Neuroimage (2013) 0.85

Separate encoding of model-based and model-free valuations in the human brain. Neuroimage (2011) 0.84