Schultz, W., Dayan, P. & Montague, P. R. A neural substrate of prediction and reward. Science 275, 1593–1599 (1997).
Gadagkar, V. et al. Dopamine neurons encode performance error in singing birds. Science 354, 1278–1282 (2016).
Zhuo, Y. et al. Improved green and red GRAB sensors for monitoring dopaminergic activity in vivo. Nat. Methods 21, 680–691 (2024).
Chen, R. & Goldberg, J. H. Actor–critic reinforcement learning in the songbird. Curr. Opin. Neurobiol. 65, 1–9 (2020).
Joel, D., Niv, Y. & Ruppin, E. Actor–critic models of the basal ganglia: new anatomical and computational perspectives. Neural Netw. 15, 535–547 (2002).
Sutton, R. S. & Barto, A. G. Reinforcement Learning: An Introduction (MIT Press, 1998).
Botvinick, M. et al. Reinforcement learning, fast and slow. Trends Cogn. Sci. 23, 408–422 (2019).
Wickens, J. R., Reynolds, J. N. & Hyland, B. I. Neural mechanisms of reward-related motor learning. Curr. Opin. Neurobiol. 13, 685–690 (2003).
Costa, R. M. Plastic corticostriatal circuits for action learning: what’s dopamine got to do with it? Ann. N. Y. Acad. Sci. 1104, 172–191 (2007).
Jarvis, E. Vocal learning and spoken language. Science 366, 50–54 (2019).
Davenport, M. H. & Jarvis, E. D. Birdsong neuroscience and the evolutionary substrates of learned vocalization. Trends Neurosci. 46, 97–99 (2023).
Konopka, G. & Roberts, T. F. Insights into the neural and genetic basis of vocal communication. Cell 164, 1269–1276 (2016).
Prather, J., Okanoya, K. & Bolhuis, J. J. Brains for birds and babies: neural parallels between birdsong and speech acquisition. Neurosci. Biobehav. Rev. 81, 225–237 (2017).
Doupe, A. J. & Kuhl, P. K. Birdsong and human speech: common themes and mechanisms. Annu. Rev. Neurosci. 22, 567–631 (1999).
Brainard, M. S. & Doupe, A. J. Translating birdsong: songbirds as a model for basic and applied medical research. Annu. Rev. Neurosci. 36, 489–517 (2013).
Burke, J. E. & Schmidt, M. F. Neural control of birdsong. eLS 1, 345–355 (2020).
Person, A. L., Gale, S. D., Farries, M. A. & Perkel, D. J. Organization of the songbird basal ganglia, including area X. J. Comp. Neurol. 508, 840–866 (2008).
Lovell, P. V. et al. ZEBrA: Zebra finch Expression Brain Atlas—a resource for comparative molecular neuroanatomy and brain evolution studies. J. Comp. Neurol. 528, 2099–2131 (2020).
Tumer, E. C. & Brainard, M. S. Performance variability enables adaptive plasticity of ‘crystallized’ adult birdsong. Nature 450, 1240–1244 (2007).
Andalman, A. S. & Fee, M. S. A basal ganglia–forebrain circuit in the songbird biases motor output to avoid vocal errors. Proc. Natl Acad. Sci. USA 106, 12518–12523 (2009).
Duffy, A., Latimer, K. W., Goldberg, J. H., Fairhall, A. L. & Gadagkar, V. Dopamine neurons evaluate natural fluctuations in performance quality. Cell Rep. 38, 110574 (2022).
Roeser, A. et al. Dopaminergic error signals retune to social feedback during courtship. Nature 623, 375–380 (2023).
Hisey, E., Kearney, M. G. & Mooney, R. A common neural circuit mechanism for internally guided and externally reinforced forms of motor learning. Nat. Neurosci. 21, 589–597 (2018).
Xiao, L. et al. A basal ganglia circuit sufficient to guide birdsong learning. Neuron 98, 208–221 (2018).
Hoffmann, L. A., Saravanan, V., Wood, A. N., He, L. & Sober, S. J. Dopaminergic contributions to vocal learning. J. Neurosci. 36, 2176–2189 (2016).
Fee, M. S. & Goldberg, J. H. A hypothesis for basal ganglia-dependent reinforcement learning in the songbird. Neuroscience 198, 152–170 (2011).
Mackevicius, E. L. & Fee, M. S. Building a state space for song learning. Curr. Opin. Neurobiol. 49, 59–68 (2018).
Doya, K. & Sejnowski, T. A novel reinforcement model of birdsong vocalization learning. In Adv. Neural Information Processing Systems 7 (NIPS 7) (eds Tesauro, G. et al.) 101–108 (MIT Press, 1995).
Tchernichovski, O., Mitra, P. P., Lints, T. & Nottebohm, F. Dynamics of the vocal imitation process: how a zebra finch learns its song. Science 291, 2564–2569 (2001).
Kollmorgen, S., Hahnloser, R. H. R. & Mante, V. Nearest neighbours reveal fast and slow components of motor learning. Nature 577, 526–530 (2020).
Funabiki, Y. & Konishi, M. Long memory in song learning by zebra finches. J. Neurosci. 23, 6928–6935 (2003).
Steinfath, E., Palacios-Munoz, A., Rottschafer, J. R., Yuezak, D. & Clemens, J. Fast and accurate annotation of acoustic signals with deep neural networks. eLife 10, e68837 (2021).
Lerner, T. N., Holloway, A. L. & Seiler, J. L. Dopamine, updated: reward prediction error and beyond. Curr. Opin. Neurobiol. 67, 123–130 (2021).
Toutounji, H., Zai, A. T., Tchernichovski, O., Hahnloser, R. H. R. & Lipkind, D. Learning the sound inventory of a complex vocal skill via an intrinsic reward. Sci. Adv. 10, eadj3824 (2024).
Bayer, H. M. & Glimcher, P. W. Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron 47, 129–141 (2005).
Adam, I. et al. Daily vocal exercise is necessary for peak performance singing in a songbird. Nat. Commun. 14, 7787 (2023).
Fiete, I. R., Fee, M. S. & Seung, H. S. Model of birdsong learning based on gradient estimation by dynamic perturbation of neural conductances. J. Neurophysiol. 98, 2038–2057 (2007).
Ikeda, M. Z., Trusel, M. & Roberts, T. F. Memory circuits for vocal imitation. Curr. Opin. Neurobiol. 60, 37–46 (2019).
Louder, M. I. M. et al. Transient sensorimotor projections in the developmental song learning period. Cell Rep. 43, 114196 (2024).
Tian, J. et al. Distributed and mixed information in monosynaptic inputs to dopamine neurons. Neuron 91, 1374–1389 (2016).
Watabe-Uchida, M., Eshel, N. & Uchida, N. Neural circuitry of reward prediction error. Annu. Rev. Neurosci. 40, 373–394 (2017).
Chen, R. et al. Songbird ventral pallidum sends diverse performance error signals to dopaminergic midbrain. Neuron 103, 266–276 (2019).
Kearney, M. G., Warren, T. L., Hisey, E., Qi, J. & Mooney, R. Discrete evaluative and premotor circuits enable vocal learning in songbirds. Neuron 104, 559–575 (2019).
Bottjer, S. W., Brady, J. D. & Cribbs, B. Connections of a motor cortical region in zebra finches: relation to pathways for vocal learning. J. Comp. Neurol. 420, 244–260 (2000).
Mandelblat-Cerf, Y., Las, L., Denisenko, N. & Fee, M. S. A role for descending auditory cortical projections in songbird vocal learning. eLife 3, e02152 (2014).
Schrittwieser, J. et al. Mastering Atari, Go, chess and shogi by planning with a learned model. Nature 588, 604–609 (2020).
Fawzi, A. et al. Discovering faster matrix multiplication algorithms with reinforcement learning. Nature 610, 47–53 (2022).
Markowitz, J. E. et al. Spontaneous behaviour is structured by reinforcement without explicit reward. Nature 614, 108–117 (2023).
Colquitt, B. M., Merullo, D. P., Konopka, G., Roberts, T. F. & Brainard, M. S. Cellular transcriptomics reveals evolutionary identities of songbird vocal circuits. Science 371, eabd9704 (2021).
Pfenning, A. R. et al. Convergent transcriptional specializations in the brains of humans and song-learning birds. Science 346, 1256846 (2014).
Tchernichovski, O., Nottebohm, F., Ho, C. E., Pesaran, B. & Mitra, P. P. A procedure for an automated measurement of song similarity. Anim. Behav. 59, 1167–1176 (2000).
Immelman, K. in Bird Vocalizations (ed. Hinde, R. A.) 64–74 (Cambridge Univ. Press, 1969).
Krzanowski, W. J. Principles of Multivariate Analysis: A User’s Perspective (Oxford Univ. Press, 1988).
Seber, G. A. F. Multivariate Observations (Wiley, 1984).
Akaike, H. A new look at the statistical model identification. IEEE Trans. Autom. Control 19, 716–723 (1974).
Ljung, L. System Identification: Theory for the User (Prentice Hall, 1999).
Holm, S. A simple sequentially rejective multiple test procedure. Scand. J. Stat. 6, 65–70 (1979).
