Browsing by Author "Cone, Ian"

Now showing 1 - 3 of 3
  • Thumbnail Image
    Item
    Biophysically Plausible Learning in the Brain via Eligibility Traces: Cortical Sequences, Hippocampal Place Cells, and Dopaminergic Reward Prediction Error
    (2021-08-25) Cone, Ian; Shouval, Harel; Pitkow, Xaq
    The brain’s ability to learn and associate temporally distal stimuli is one of its most fundamental (and puzzling) functions. The behaviorally relevant time scales (hundreds of milliseconds to seconds) at which the brain must link actions to reward are largely incompatible with Hebbian or STDP-like correlative learning rules. To solve this conundrum, we posit that two types of Hebbian activated, synapse specific “eligibility traces” – one associated with long term potentiation and the other long term depression – act as long lasting synaptic “tags” of previous activity . Upon presentation of a reinforcement signal, these two traces act in competition to determine long term changes in synaptic strength. In this work, we demonstrate the efficacy of this two-trace learning rule in three separate models. The first focuses on the learning and recall of uncompressed temporal sequences, based on recent experimental data from the visual cortex. The second model replicates so called “behavioral time scale plasticity” in hippocampal CA1, where the induction of a dendritic calcium spike triggers plasticity in place fields well in the past or future along the track traversal. Finally, this thesis showcases a model of dopaminergic cells demonstrating reward prediction error, including in the context of various “blocking” and “unblocking” paradigms. These models adhere to biophysical realism as much as possible; leaky-integrate-and-fire neurons with realistic noise are used when appropriate, and the models are either based on or replicate experimental results. Notably, and in contrast to many contemporary models which deal with the temporal credit assignment problem, eligibility traces allow for the principles of locality and causality to always be conserved. The success of these models presents a compelling case for the widespread utility of eligibility traces across a wide range of temporal tasks, and the models’ adherence to biophysical realism lend plausibility to the idea that eligibility traces are actually implemented in such a manner in the brain.
  • Thumbnail Image
    Item
    Learning precise spatiotemporal sequences via biophysically realistic neural circuits with modular structure
    (2020-05-27) Cone, Ian; Shouval, Harel; Pitkow, Xaq
    The ability to express and learn temporal sequences is an essential part of neural learning and memory. Learned temporal sequences are expressed in multiple brain regions and as such there may be common design in the circuits that mediate it. This thesis proposes a substrate for such representations, via a biophysically realistic network model that can robustly learn and recall discrete sequences of variable order and duration. The model consists of a network of spiking leaky-integrate-and-fire model neurons placed in a modular architecture designed to resemble cortical microcolumns. Learning is performed via a learning rule with “eligibility traces”, which hold a history of synaptic activity before being converted into changes in synaptic strength upon neuromodulator activation. Before training, the network responds to incoming stimuli, and contains no memory of any particular sequence. After training, presentation of only the first element in that sequence is sufficient for the network to recall an entire learned representation of the sequence. An extended version of the model also demonstrates the ability to successfully learn and recall non-Markovian sequences. This model provides a possible framework for biologically realistic sequence learning and memory, and is in agreement with recent experimental results, which have shown sequence dependent plasticity in sensory cortex.
  • Thumbnail Image
    Item
    Learning to express reward prediction error-like dopaminergic activity requires plastic representations of time
    (Springer Nature, 2024) Cone, Ian; Clopath, Claudia; Shouval, Harel Z.
    The dominant theoretical framework to account for reinforcement learning in the brain is temporal difference learning (TD) learning, whereby certain units signal reward prediction errors (RPE). The TD algorithm has been traditionally mapped onto the dopaminergic system, as firing properties of dopamine neurons can resemble RPEs. However, certain predictions of TD learning are inconsistent with experimental results, and previous implementations of the algorithm have made unscalable assumptions regarding stimulus-specific fixed temporal bases. We propose an alternate framework to describe dopamine signaling in the brain, FLEX (Flexibly Learned Errors in Expected Reward). In FLEX, dopamine release is similar, but not identical to RPE, leading to predictions that contrast to those of TD. While FLEX itself is a general theoretical framework, we describe a specific, biophysically plausible implementation, the results of which are consistent with a preponderance of both existing and reanalyzed experimental data.