Sensory Systems/NonPrimates BirdSong

Birds: Neural Mechanism for Song Learning in Zebra Finches

Introduction

Over the past four decades songbirds have become a widely used model organism for neuroscientists studying complex sequential behaviours and sensory-guided motor learning. Like human babies, young songbirds learn many of the sounds they use for communication by imitating adults. One songbird in particular, the zebra finch (Taeniopygia guttata), has been the focus of much research because of its proclivity to sing and breed in captivity and its rapid maturation. The song of an adult male zebra finch is a stereotyped series of acoustic signals with structure and modulation over a wide range of time scales, from milliseconds to several seconds. The adult zebra finch song comprises a repeated sequence of sounds, called a motif, which lasts about a second. The motif is composed of shorter bursts of sound called syllables, which often contain sequences of simpler acoustic elements called notes as shown in Fig.1. The songbirds learning system is a very good model to study the sensory-motor integration because the juvenile bird actively listens to the tutor and modulates its own song by correcting for errors in the pitch and offset. The neural mechanism and the architecture of the song bird brain which plays a crucial role in learning is similar to the language processing region in frontal cortex of humans. Detailed study of the hierarchical neural network involved in the learning process could provide significant insights into the neural mechanism of speech learning in humans.

Figure 1: Illustration of the typical song structure & learning phases involved in song bird. Upper panel: Phases involved in the song learning process. Middle panel: Structure of a crystallized song a,b,c,d,e denote the various syllable in the song. Lower panel: Evolution of the song dynamics during learning.

Illustration of the typical song structure & learning phases involved in song bird.

Song-learning proceeds through a series of stages, beginning with sensory phase where the juvenile bird just listens to its tutor (usually its father) vocalizing, often without producing any song-like vocalization itself. The bird uses this phase to memorize a certain structure of the tutor song, forming the neural template of the song. Then it enters the sensorimotor phase, where it starts babbling the song and correcting its errors using auditory feedback. The earliest attempt to recreate the template of the tutor song is highly noisy, unstructured and variable and it is called sub-song. An example is shown in the spectrogram in Fig.1. Through the subsequent days the bird enters a “plastic phase” where there is a significant amount of plasticity in the neural network responsible for generating highly structured syllables and the variability is reduced in the song. By the time they reach sexual maturity, the variability is substantially eliminated—a process called crystallization—and the young bird begins to produce a normal adult song, which can be a striking imitation of the tutor song (Fig.1). Thus, the gradual reduction of song variability from early sub-song to adult song, together with the gradual increase in imitation quality, is an integral aspect of vocal learning in the songbird. In the following sections we will explore several parts of the avian brain and the underlying neural mechanisms that are responsible for this remarkable vocal imitation observed in these birds.

Hierarchical Neural Network involved in the generation of song sequences

It is important to understand the neuroanatomy of the songbird in detail because it provides significant information about the learning mechanisms involved in various motor and sensory integration pathways. This could ultimately shed light on the language processing and vocal learning in humans. The exact neuroanatomical data about human speech processing system is still unknown and songbird anatomy and physiology will enable us to make plausible hypotheses. The comparison of the mammalian brain and a songbird (avian) brain is made in the final section of this chapter in (Fig. 6). The pathway observed in the avian brain can be broadly divided into motor control and anterior forebrain pathway as shown in (Fig.2). The auditory pathway provides the error feedback signals which leads to potentiation or depression of the synaptic connections involved in motor pathways, which plays a significant role in vocal learning. The motor control pathway includes Hyperstriatum Ventrale, pars Caudalis (HVC), Robust Nucleus of Acropallium (RA), Tracheosyringeal subdivision of the hypoglossal nucleus (nXIIts) and Syrinx. This pathway is necessary for generating the required motor control signals which produce highly structured songs and coordinating breathing with singing. The anterior forebrain pathway includes Lateral magnocellular nucleus of anterior nidopallium (LMAN), Area X (X) and the medial nucleus of dorsolateral thalamus (DLM). This pathway plays a crucial role in song learning in juveniles, song variability in adults and song representation. The auditory pathway includes substantia nigra (SNc) and the ventral tegmental area (VTA), which plays a crucial role in auditory inputs processing and analyzing the feedback error. The muscles of the syrinx are innervated by a subset of motor neurons from nXIIts. A primary projection to the nXIIts descends from neurons in the forebrain nucleus RA. Nucleus RA receives motor-related projections from another cortical analogue, nucleus HVC, which in turn receives direct input from several brain areas, including thalamic nucleus uvaeformis (Uva).

Figure 2. Architecture of the song bird brain & various pathways carrying motor and auditory feed- back signals.

Neural Mechanism for the generation of highly structured & temporally precise syllable pattern

Nuclei HVC and RA are involved in the motor control of song in a hierarchical manner (Yu and Margoliash 1996). Recordings in singing zebra finches have shown that HVC neurons that project to RA transmit an extremely sparse pattern of bursts: each RA-projecting HVC neuron generates a single highly stereotyped burst of approximately 6 ms duration at one specific time in the song (Hahnloser, Kozhevnikov et al. 2002). During singing, RA neurons generate a complex sequence of high-frequency bursts of spikes, the pattern of which is precisely reproduced each time the bird sings its song motif (Yu and Margoliash 1996). During a motif, each RA neuron produces a fairly unique pattern of roughly 12 bursts, each lasting ~10 ms (Leonardo and Fee 2005). Based on the observations that RA-projecting HVC neurons generate a single burst of spikes during the song motif and that different neurons appear to burst at many different times in the motif, it has been hypothesized that these neurons generate a continuous sequence of activity over time (Fee, Kozhevnikov et al. 2004, Kozhevnikov and Fee 2007). In other words, at each moment in the song, there is a small ensemble of HVC (RA) neurons active at that time and only at that time (Figure 3), and each ensemble transiently activates (for ~10 ms) a subset of RA neurons determined by the synaptic connections of HVC neurons in RA (Leonardo and Fee 2005). Further, in this model the vector of muscle activities, and thus the configuration of the vocal organ, is determined by the convergent input from RA neurons on a short time scale, of about 10 to 20 ms. The view that RA neurons may simply contribute transiently, with some effective weight, to the activity of vocal muscles is consistent with some models of cortical control of arm movement in primates (Todorov 2000). A number of studies suggest that the timing of the song is controlled on a millisecond-by-millisecond basis by a wave, or chain, of activity that propagates sparsely through HVC neurons. This hypothesis is supported by an analysis of timing variability during natural singing (Glaze and Troyer 2007) as well as experiments in which circuit dynamics in HVC were manipulated to observe the effect on song timing. Thus, in this model, song timing is controlled by propagation of activity through a chain in HVC; the generic sequential activation of this HVC chain is translated, by the HVC connections in RA, into a specific precise sequence of vocal configurations.

Figure 3. Mechanisms of sequence generation in the adult song motor pathway. Illustration of the hypothesis that RA-projecting HVC (HVC(RA)) neurons burst and activate each other sequentially in groups of 100 to 200 coactive neurons. Each group of HVC neurons drives a distinct ensemble of RA neurons to burst. The neurons converge with some effective weight at the level of the motor neurons to activate syringeal muscles.

Synaptic Plasticity in Posterior Forebrain Pathway is a potential substrate for vocal learning

A number of song-related avian brain areas have been discovered (Fig. 4A). Song production areas include HVC (Hyperstriatum Ventrale, pars Caudalis) and RA (robust nucleus of the arcopallium), which generate sequences of neural activity patterns and through motor neurons control the muscles of the vocal apparatus during song (Yu and Margoliash 1996, Hahnloser, Kozhevnikov et al. 2002, Suthers and Margoliash 2002). Lesion of HVC or RA causes immediate loss of song (Vicario and Nottebohm 1988). Other areas in the anterior forebrain pathway (AFP) appear to be important for song learning but not production, at least in adults. The AFP is regarded as an avian homologue of the mammalian basal ganglia thalamocortical loop (Farries 2004). In particular, lesion of area LMAN (lateral magnocellular nucleus of the nidopallium) has little immediate effect on song production in adults, but arrests song learning in juveniles (Doupe 1993, Brainard and Doupe 2000). These facts suggest that LMAN plays a role in driving song learning, but the locus of plasticity is in brain areas related to song production, such as HVC and RA. Doya and Senjowski in 1998 proposed a tripartite schema, in which learning is based on the interactions between actor and a critic (Fig.4B). The critic evaluates the performance of the actor at a desired task. The actor uses this evaluation to change in a way that improves its performance. To learn by trial and error, the actor performs the task differently each time. It generates both good and bad variations, and the critic’s evaluation is used to reinforce the good ones. Ordinarily it is assumed that the actor generates variations by itself. However, the source of variation is external to the actor. We will call this source the experimenter. The actor was identified with HVC, RA, and the motor neurons that control vocalization. The actor learns through plasticity at the synapses from HVC to RA (Fig. 4C). Based on evidence of structural changes like axonal growth and retraction that take place in the HVC to RA projection during song learning, this view is widely regarded as a plausible mechanism. For the experimenter & critic, Doya and Senjowski turned to the anterior forebrain pathway, hypothesizing that the critic is Area X and the experimenter is LMAN.

Figure 4. Plasticity in Specific pathways enabling learning. (A) Avian song pathways and the tripartite hypotheses. A: avian brain areas involved in song production and song learning. Premotor pathway (open) includes areas necessary for song production. Anterior forebrain pathway (filled) is required for song learning but not for song production. (B) Tripartite reinforcement learning schema: the actor produces behaviour; the experimenter sends fluctuating input to the actor, producing variability in behaviour that is used for trial-and-error learning; the critic evaluates the behaviour of the actor and sends a reinforcement signal to it. For birdsong, the actor includes premotor song production areas HVC and RA. (C) Plastic and empiric synapses. RA receives synaptic input from both HVC and LMAN. We will call the HVC synapses “plastic,” in keeping with the hypothesis that these synapses are the locus of plasticity for song learning.

Biophysically realistic synaptic plasticity rules underlying song learning mechanism

Biophysically realistic model The role of LMAN input to RA is to produce a fluctuation that is static over the duration of a song bout, directly in the synaptic strengths from premotor nucleus HVC to RA. From a functional perspective, the model of Doya and Sejnowski is akin to weight perturbation (Dembo and Kailath 1990, Seung 2003) and relatively easy to implement: a temporary but static HVC->RA weight change that lasts the duration of one song causes some change in song performance. If performance is good, the critic sends a reinforcement signal that makes the temporary static perturbation permanent. From a neurobiological perspective this model requires machinery whereby N-methyl-Daspartate (NMDA)-mediated synaptic transmission from LMAN to RA can drive synaptic weight changes that remain static over the 1 to 2 seconds. In short, LMAN appears to drive fast, transient song fluctuations on a subsyllable level, affected by ordinary excitatory transmission that drives dynamic postsynaptic membrane conductance fluctuations in the postsynaptic RA neurons. The goal of this model is to relate the highlevel concept of reinforcement learning by the tripartite schema to a biologically realistic lower level of description in terms of microscopic events at synapses and neurons in the birdsong system. It should demonstrate song learning in a network of realistic spiking neurons, and examine the plausibility of reinforcement algorithms in explaining biological fine motor skill learning with respect to learning time in the birdsong network. The present model is based on many of the same general assumptions that were made by Doya and Sejnowski. We assume a tripartite actor-critic-experimenter schema. The critic is weak, providing only a scalar evaluation signal. The HVC sequence is fixed, and only the map from HVC to the motor neurons is learned, through plasticity at the HVC->RA synapses. LMAN perturbs song through its inputs to the song premotor pathway. However, the structure and dynamics of LMAN inputs, and their influence on learning, are different, with distinct neurobiological implications. In keeping with our hypothesis that the function of LMAN drive to RA is to perform experiments for trial-and-error learning, the connections from LMAN to RA will be called empiric synapses (Fig. 4C). The conductance of the plastic synapse from neuron j in HVC to neuron i in RA is given by $W_{ij}S_{ij}HVC(t)$, where the synaptic activation $S_{ij}HVC(t)$ determines the time course of conductance changes, and the plastic parameter $W_{ij}$ determines their amplitude. Changes in $W_{ij}$ are governed by the plasticity rule is given by ${\frac {\partial {{W}_{ij}}}{\partial t}}=\eta \,R(t)\,{{e}_{ij}}(t)$ The positive parameter $\eta$, called the learning rate, controls the overall amplitude of synaptic changes. The eligibility trace $e_{ij}(t)$ is a hypothetical quantity present at every plastic synapse. It signifies whether the synapse is "eligible" for modification by reinforcement and is based on the recent activation of the plastic synapse and the empiric synapse onto the same RA neuron ${{e}_{ij}}(t)=\int \limits _{0}^{t}{d{t}'}\,G(t-{t}')\left[S_{i}^{LMAN}(t)-\left\langle S_{i}^{LMAN}\right\rangle \right]\,\,S_{ij}^{HVC}(t)$ Here $S_{i}^{LMAN}(t)$ is the conductance of the empiric (LMAN->RA) synapse onto the RA neuron. The temporal filter G(t) is assumed to be nonnegative, and its shape determines how far back in time the eligibility trace can "remember" the past. The instantaneous activation of the empirical synapses is dependent on the average activity $\left\langle \,S_{i}^{LMAN}(t)\right\rangle$. The learning principles follows two basic rules shown in (Fig.5). First rule: If coincident activation of a plastic (HVC->RA) synapse and empiric (LMAN->RA) synapse onto the same RA neuron is followed by positive reinforcement, then the plastic synapse is strengthened. Second rule: If activation of a plastic synapse without activation of the empiric synapse onto the same RA neuron is followed by positive reinforcement, then the plastic synapse is weakened. The rules based on dynamic conductance perturbations of the actor neurons perform stochastic gradient ascent on the expected value of the reinforcement signal. This means that song performance as evaluated by the critic is guaranteed to improve on average.

Comparison between Mammalian & Songbird brain architecture

The avian Area X is homologous to the mammalian basal ganglia (BG) and includes striatal and pallidal cell types. The BG forms part of a highly conserved anatomical loop-through several stations, from cortex to the BG (striatum and pallidum), then to thalamus and back to cortex. Similar loops are seen in the songbird: the cortical analogue nucleus LMAN projects to Area X, the striatal components of which project to the thalamic nucleus DLM, which projects back to LMAN. Striatal components accounts for reward basing learning and reinforcement learning. The neuron types and its functionality are exactly comparable in Area X of birds to basal ganglia in humans as shown (in Fig.6). The close anatomical similarity motivates us to learn the song bird brain in more detail because with this we can finally achieve some significant understanding of the speech learning in humans and treat many speech related disorders with higher precision.

Figure 6. Comparison of mammalian and avian basal ganglia–forebrain circuitry.

References

Brainard, M. S. and A. J. Doupe (2000). "Auditory feedback in learning and maintenance of vocal behaviour." Nat Rev Neurosci 1(1): 31-40.

Dembo, A. and T. Kailath (1990). "Model-free distributed learning." IEEE Trans Neural Netw 1(1): 58-70.

Doupe, A. J. (1993). "A neural circuit specialized for vocal learning." Curr Opin Neurobiol 3(1): 104-111.

Farries, M. A. (2004). "The avian song system in comparative perspective." Ann N Y Acad Sci 1016: 61-76.

Fee, M. S., A. A. Kozhevnikov and R. H. Hahnloser (2004). "Neural mechanisms of vocal sequence generation in the songbird." Ann N Y Acad Sci 1016: 153-170.Glaze, C. M. and T. W. Troyer (2007). "Behavioral measurements of a temporally precise motor code for birdsong." J Neurosci 27(29): 7631-7639.

Hahnloser, R. H., A. A. Kozhevnikov and M. S. Fee (2002). "An ultra-sparse code underlies the generation of neural sequences in a songbird." Nature 419(6902): 65-70.

Kozhevnikov, A. A. and M. S. Fee (2007). "Singing-related activity of identified HVC neurons in the zebra finch." J Neurophysiol 97(6): 4271-4283.

Leonardo, A. and M. S. Fee (2005). "Ensemble coding of vocal control in birdsong." J Neurosci 25(3): 652-661.

Seung, H. S. (2003). "Learning in spiking neural networks by reinforcement of stochastic synaptic transmission." Neuron 40(6): 1063-1073.

Suthers, R. A. and D. Margoliash (2002). "Motor control of birdsong." Curr Opin Neurobiol 12(6): 684-690.

Todorov, E. (2000). "Direct cortical control of muscle activation in voluntary arm movements: a model." Nat Neurosci 3(4): 391-398.

Vicario, D. S. and F. Nottebohm (1988). "Organization of the zebra finch song control system: I. Representation of syringeal muscles in the hypoglossal nucleus." J Comp Neurol 271(3): 346-354.

Yu, A. C. and D. Margoliash (1996). "Temporal hierarchical control of singing in birds." Science 273(5283): 1871-1875.