Peter Cariani
Eaton Peabody Laboratory of Auditory Physiology
Massachusetts Eye and Ear Infirmary
243 Charles St, Boston, MA 02114 USA
peter@epl.meei.harvard.edu
www.cariani.com
Communication and Cognition-Artificial Intelligence
(CC-AI),
March, 1995, Vol 12, nos. 1-2, pp. 157-219.
Special Issue on Self-Reference in Biological
and Cognitive Systems, L. Rocha (ed).
Communication and Cognition-Artificial Intelligence
The journal for the integrated study of artificial
intelligence,
cognitive science and applied epistemology.
Fernand Vandamme, Editor-in-chief.
CC-AI, Blandinjnberg 2, B-2000 Ghent, Belgium
Preprinted in: Origins: Brain and Self-Organization,
K Pribram, ed.
Lawrence Erlbaum Assoc., 1994; 208-252.
Pagination and formatting in this document differ from the original and some of the figures have been reformatted for portrait mode. I have generally left the text alone except for minor typographical errors. While my thinking has evolved substantially since this paper was written (1993), I still firmly believe that a general temporal theory of sensory coding is worth articulating, developing, and discussing. It is encouraging to see today a much more widespread awareness and understanding of temporal coding alternatives than was the case a decade ago. pac 1/2000. This HTML version was created June, 2000.
Contents
Abstract
1. The role of timing
in the brain
2. Coding by average
discharge rate
3. Possible
alternatives to average rate codes
3.1 Signs and codes
3.2
Scalar vs. multidimensional signalling
3.3
A space of possible neural codes
3.4
The many ways to send a message
3.5 Sensory
representations
3.6
Neural codes and perceptual qualities
4.
Evidence for temporal codes in sensory systems
4.1
Somatoception: vibration perception
4.2 Audition
4.3 Electroception
4.4 The chemical
senses
4.5 Pain
4.6 Vision
5.
Processing strategies for discriminating temporal patterns
6. Temporal
processing in a single axon tree
7. Temporal
autocorrelation architectures
8. Temporal
cross-correlation architectures
9. Higher order resonances
References
Potential strategies for temporal neural processing in the brain and their implications for the design of artificial neural networks are considered. Current connectionist thinking holds that neurons send signals to each other by changes in their average rate of discharge. This implies that there is one output signal per neuron at any given time (scalar coding), and that all neuronal specificity is achieved solely by patterns of synaptic connections. However, information can be carried by temporal codes, in temporal patterns of neural discharges and by relative times of arrival of individual spikes. Temporal coding permits multiplexing of information in the time domain, which potentially increases the flexibility of neural networks. A broadcast model of information transmission is contrasted with the current notion of highly specific connectivity. Evidence for temporal coding in somatoception, audition, electroception, gustation, olfaction and vision is reviewed, and possible neural architectures for temporal information processing are discussed.
1. The role of timing in the brain
The human brain is by far the most capable, the most versatile, and the most complex information-processing system known to science. For those concerned with problems of artificial intelligence there has long been the dream that once its functional principles are well understood, the design and construction of adaptive devices more powerful than any yet seen could follow in a straightforward manner. Despite great advances, the neurosciences are still far from understanding the nature of the "neural code" underlying the detailed workings of the brain. i.e. exactly which information-processing operations are involved.
If we choose to view the brain in informational terms, as an adaptive signalling system embedded within an external environment, then the issue of which aspects of neural activity constitute the "signals" in the system is absolutely critical to understanding its functioning. It is a question which must be answered before all others, because all functional assumptions, interpretations, and models depend upon the appropriate choice of what processes neurons use to convey information. The role of the time patterns of neural discharges in the transmission and processing of information in the nervous system has been debated since the pulsatile nature of nervous transmission was recognized less than a century ago. Because external stimuli can be physically well-characterized and controlled, the encoding of sensory information has always played a pivotal role in more general conceptions of neural coding.
2. Coding by average discharge rate
With the advent of single cell recording techniques in neurophysiology, it was generally assumed that neural information is encoded solely in the average neural discharge rates of neurons (Adrian 1928) . This notion of a average discharge rate code, sometimes called the Frequency Coding principle , has persisted and forms the basis for virtually all neural net design (Feldman 1990) and almost all neuroscientific investigations concerned with information processing (Barlow 1972) .
While there is much accumulated experimental evidence to support such a principle in many systems, it does not necessarily follow that only average rate codes are used in the nervous coding. From the advent of modern electrophysiology, there were always other conceptions of how sense information could be transmitted (Troland 1921; Troland 1929; Wever & Bray 1937; Boring 1942; Wever 1949) . Many other types of codes produce signals which co-vary with average rates, and these other coding schemes may actually contain much higher quality information than average discharge rates. In the auditory nerve, for example, stimulus periodicities below a few kHz are much more precisely represented by interspike interval statistics than by discharge rates (Goldstein & Srulovicz 1977) , but because both interval patterns and discharge rate patterns are observed together, it is difficult to determine directly which kinds of codes are functionally operant. However, since rate-coding has become the default assumption of practicing neuroscientists, the burden of proof generally falls on the alternatives.
The principle of rate coding has a number of wide-ranging ramifications in the way that neural networks, both wet and dry are conceptualized. A mean rate code entails some time window over which spikes are counted, and depending upon the system, this window is usually thought to be on the order of tens to hundreds of milliseconds or more. Long integration windows can present problems in sensory systems where coherent, detailed percepts can be generated with short stimulus durations (e.g. tachistoscopically presented images, tone bursts). The meaningful use of an average discharge rate is also stretched when only a handful of spikes are discharged within an integration window, as often occurs in cortical neurons.
Rate coding goes hand in hand with the doctrine of "specific nerve energies," as it was laid out by Müller and Helmholtz (see discussion in (Boring 1933; Boring 1942) ). The principle asserts that specific sensory modalities have specific types of sense receptors. Consequently it is by virtue of connection to a given type of receptor that a given neuron is interpreted to be sending a signal related to a particular quality (a visual signal as opposed to a smell). Helmholtz through his study of the cochlea elevated this principle to also include quality differences within a sense modality. Thus, in Helmholtz's view, because particular auditory nerve fibers are connected to receptors at specific places on the cochlear partition, and hence have different frequency sensitivities, they signal different pure tone pitches by virtue of their connectivity. Coding exclusively by average discharge rate necessitates this kind of "labelled line" or "place" coding because there is no other means internal to the spike train itself for conveying what kind of signal it is (e.g. a taste vs. a sound; the semantics of the message). While the doctrine of specific nerve energies does not mandate that average rate be the signal encoded in the spike train (e.g. see the discussion of Troland's resonance-frequency theory of hearing (Boring 1942) ), it has generally been taken on faith that sensory coding could be accomplished solely by rate-place codes. Unless temporal patterns are immediately obvious and impossible to ignore, looking elsewhere into coding alternatives has generally been regarded by neuroscientists as wasted effort.
In tandem with exclusive use of rate codes, it has often been assumed
that there is no usable temporal structure in spike trains, i.e. spike
trains can be functionally described as a Poisson process with one independent
parameter, the mean rate of arrivals. As a result, in many higher-level
models of neuronal networks, the temporal dynamics of spike generation
are ignored in favor of mean rates or discharge probabilities. One far
reaching consequence of these high level functional descriptions is that
the neural output signal in any given time period is conceived as a scalar
quantity. This effectively rules out the multiplexing of signals in the
time domain, which would require a finer grained representation of time
and a different (e.g. Fourier) interpretation of the signal. Since only
one output signal can be sent from each neural element, multiple input
signals converging on a given element must be converted into one output
signal. An analogy could be made to a telegraph network which recieves
messages from a hundred stations, but can only transmit one message to
all of its hundred connecting stations. Each additional signal must compete
with all others at each node. In contrast, a station which has several
frequency bands available can process meaningful information in one or
two bands and relay the other messages unchanged.
Even the assumption that all postsynaptic neurons receive the same
message can be called into question, since conduction blocks in different
branches of axon trees can filter the spike trains that arrive at the respective
synapses (Bittner 1968; Raymond & Lettvin 1978; Waxman 1978;
Raymond 1979; Wasserman 1992) . Instead of one informationally-passive
output line fanning out to send the same signal to all postsynaptic elements,
a branching structure is created which sequentially filters the signals.
Thus the shift from scalars to multidimensional signalling and the inclusion
of axonal operations can drastically the functional topology of the network,
and with it the flexibility of infomation processing.
Largely because of the ordering in cortical maps of retinotopic positions, cochleotopic positions, and somatotopic positions, it has long been assumed that the cortex is a spatial pattern processor. This view of cortical structures was crystallized in a set of far-reaching and of provocative papers by David Marr (Marr 1970; McNaughton & Nadel 1990; Marr 1991) . In these papers Marr proposed general information processing mechanisms for the major cortical structures in the brain: the cerebral cortex, the hippocampus ("archicortex") and the cerebellar cortex.
While it seems abundantly clear that spatially ordered maps are functionally very important, there is no inherent reason why the cortex must be only a spatial processor, why it cannot also be structured so as to effect time-space transformations (Pitts & McCulloch 1947) . Alternative time-place architectures, such as those first articulated by Licklider (Licklider 1951) and Braitenberg (Braitenberg 1961; Braitenberg 1967) take advantage of spatial orderings to perform computations in the time domain. After a long period of relative neglect, the recent discoveries of neuronal synchronies in the visual cortex have brought various time-place models back into more general consideration (e.g. (Reitboeck et al 1988; Pabst et al 1989; Baldi & Meir 1990; Singer 1990) ), but these models are still more the exception than the rule.
From the belief that the cortex is exclusively a spatial processor it follows that all information which is not place coded in sensory peripheries (e.g. time patterns in somatosensory and auditory systems) must eventually converted into the common language of the cortex, spatial excitation patterns. Thus whatever time patterns might exist in sensory peripheries, so this line of thinking runs, there must be a temporal feature detectors which will realize time-to-place transformations somewhere in the pathway. This transformation allows temporal features to be processed along with other place-coded forms of information by a common cortical architecture. An alternative, however, is to use a cortical spatio-temporal processing architecture capable of handling both types of information, taking advantage of temporal order in its spatially organized input channels when it is available. In this way all information might not need to be transformed into spatial excitation patterns and a mixture of spatial and temporal coding could then be utilized at all levels.
In summary, because of the twin postulates of specific connectivity and coding by discharge rate , almost all connectionist networks assume that time can be completely ignored, i.e. that all processing operates on spatial rather than spatio-temporal patterns of excitation (Barlow 1972; Arbib 1989; Feldman 1990; Churchland 1992) . These two assumptions together with variable synaptic weighting form the basis of virtually all neural network models now in currency (e.g. (Rosenblatt 1958; Selfridge 1958; Kabrisky 1966; Grossberg 1980; Hopfield 1982; Edelman 1987; Marr 1991) to list a few). The only exceptions have been adaptive time-delay networks ( (MacKay 1962; Tank & Hopfield 1987; Mozer 1993) ), whose temporal processing capabilities are only now beginning to be appreciated and developed more fully.
Although spatially-based neural nets have proven useful in an enormous array of applications, it does not immediately follow that the brain must code by average discharge rate scalars and process information by discharge rate integration. Since neural nets are finite state automata, given enough elements and processing steps, they can replicate any observed natural regularities or behavioral patterns. While this is extremely useful from a modelling perspective, the protean nature of the simulation also present pitfalls to the unwary. When we see that a complex and mysterious natural function can be realized with a computer program or neural net,. lacking viable alternative explanations, it is easy to be seduced into believing that this is the way that nature does it. Thus observations must always be used to test the validity of models, not the other way around.
Related to the difficulties of average rate coding mentioned above, there are some inconsistencies between the view of the cortex as a spatial pattern processor and the observed behavior of its elements. Some researchers have noted that the discharge statistics for cortical pyramidal cells are more consistent with coincidence detection than rate integration as the operational primitive for cortical information processing (Abeles 1982a; Abeles 1982b; Windhorst 1988; Abeles 1990; Softky & Koch 1992) ). In this light, it is conceivable that new kinds of neural net architectures based on temporal coincidence operations will be needed before we finally have artificial networks that function at all like those of our brains.
3. Possible
alternatives to average rate codes
Other kinds of neural coding schemes besides those based on average
discharge rates are possible, and diverse examples of neural timing patterns
have been found in nautre (Perkell & Bullock 1968) . In general
any property of a spike train which covaries with some property of a stimulus
can be used to transmit information about that stimulus. While a given
spike train over some time interval has but one average discharge rate,
there are a very large number of temporal patterns that are possible with
the same number of spikes. Some of these codes are potentially more efficient
at conveying information than average discharge rates (MacKay &
McCulloch 1952) . Still more codes are possible if the joint discharge
patterns of multiple neurons are considered.
The many different coding schemes in the time domain range from simple interspike interval codes to more complex temporal pattern codes, latency and history-dependent codes. Examples of many codes are shown in Figure 1. The spike trains on the left contain patterns that would be recognized as encoded "signals" while those on the right for the most part are examples of patterns that would either be interpreted as different signals or as the absence of signals. A synchrony code between two neurons is shown as a simple example of a population code, a coding scheme which requires the joint activity of multiple neurons to transmit information. In the late 1960's and early 1970's there were a number of significant efforts to systematically describe a wide array of possible neural codes (Bullock 1967; Morrell 1967; Mountcastle 1967; Perkell & Bullock 1968; Uttal 1973) . A large catalog of hypothetical and observed coding schemes can be found in the appendix of Perkel & Bullock, 1968.
To show that a given pattern of activity is a sign one must 1) show that the requisite information needed to effect a particular sensory discrimination or behavior is present in the characteristic patterns of neural activity that constitute the sign, and 2) that the particular sensory discrimination or behavior occurs when the sign is present and does not occur when it is not present. To show that a given pattern of activity is a code rather than a sign one must further show that the particular sign is sufficient by itself to cause the sensory discrimination or behavior to take place. This is a test of whether the system as a whole functionally relies on the particular form of the sign to make the perceptual or behavioral discrimination. In general it is much easier to ascertain whether the requisite information needed to encode a particular stimulus property is present in the discharge patterns of a given neural population than it is to determine whether the rest of the brain actually utilizes information encoded in that form to modify its behavior. One notable exception to this rule is the use of electrical stimulation: in several cases (e.g. (Young 1977; Eddington et al 1978; Covey 1980; Emmers 1981; Di Lorenzo & Hecht 1993; Mountcastle 1993) it has been shown that a particular time patterns of electrical stimulation evoke behavior similar to that of a natural stimulus. All of these cases are strongly suggestive of temporal coding mechanisms. For electrical stimulation both conditions (1) and (2) are likely to be met if it can be assured that the electrical stimulation is in fact inducing the particular pattern of neural activity that constitutes the sign and that the electrical activity is not evoking the behavior through some other set of nonspecific mechanisms or side effects that would not be present under natural stimulation.
3.2 Scalar vs. multidimensional signalling
It should be emphasized that these coding schemes are not mutually exclusive; they are complementary. Because of the non-exclusive nature of many of these coding schemes evidence in favor of one code is not necessarily evidence against another code. Thus the existence of neurophysiological data correlating a given stimulus property with average discharge rates is not necessarily inconsistent with the existence of a temporal code for that property. Indeed, in many cases it can be the case that the accumulated, coarser-grained observations of average discharge rates can be explained by complex, underlying time patterns of excitation and inhibition.
This complementarity of signalling modes also permits multiplexing. A given neuron conveying a spike train may synapse upon many other neurons so that a given spike train could be interpreted by one group of neurons in one way (e.g. by reading off the average rate of discharge), by a second group in a completely different way (e.g. by distinguishing particular temporal patterns), and by a third group in yet a third way (e.g. by examining the time of arrival of the first spike in a burst). In such a case the spike train would be conveying several signals at once; many kinds of information would be multiplexed in the single spike train. Thus depending upon the nature and diversity of the receiving neural assemblies, a spike train can convey several semi-independent signals at the same time. One of the advantages of temporal codes over mean rate codes is this capacity to convey higher dimensional signals rather than one, scalar signal (the mean rate).
Multidimensional signalling can also enhance the reliability of information transmission through the simulatanous use of different kinds of codes. Informational redundancy thus can be achieved not only by sending the same information over different lines using the same code, but also by sending the same information using different codes. Central processors could receive information about stimulus intensity not only by mean discharge rates in afferent channels, but also by the latencies of incoming spikes and the regularities of interspike interval patterns in those channels.
What determines how a given spike train is to be interpreted? Is a particular spike train signalling the presence of a pungent odor or a tap on one's back? A high-pitched tone or part of a visual scene? The type of information conveyed can be differentiated by 1) where it came from (which neurons produced it -- the place principle), 2) what characteristic form it takes (characteristic modality- or quality-specific temporal patterns), and 3) when it arrived relative to some reference time (characteristic latency). Conceptually, all neural pulse codes can be classified according to three sets of properties: those relating to the differential connectivity of neurons ("place" or spatial pattern codes), those relating to temporal patterns within spike trains, and those relating to precise time-of-arrival of spikes. Each general type of code can be based on the activity of single neurons or on patterns of activity of populations of neurons. These three different dimensions of pulse codes can be depicted in the form of a triangular space of possibilities. (Figure 2). Each vertex is an archetype for each coding type: connectivity or place coding, temporal pattern coding, and arrival time coding. Connectivity is described in terms of the pattern of synaptic connections in a given system of interest and the relative effectiveness of each synapse in eliciting discharges in each post-synaptic target (a set of "synaptic weights"). A temporal pattern can be described by a Fourier spectrum, a set of magnitude and phases for the various frequencies present in the spike train. Here a single interspike interval is a quantum of frequency information. An arrival time is described in terms of the time relative to some reference time, often called a latency. A pure place code need not convey any information in the temporal structure of spike trains or in precise times of arrival. Similarly, a pure temporal pattern code need not rely on specific neural connectivities or on the specific arrival times of the temporal patterns. And a code which relies on the latency of an event (e.g. a burst of spikes) need not rely on which neurons convey the event nor on the particular temporal patterning of the event (e.g. the temporal microstructure of the burst). Thus the three coding archetypes are orthogonal to each other; hence they are not mutually exclusive, but complementary, and can be combined in various ways. In general the term "temporal coding" includes both temporal pattern codes and time-of-arrival codes, i.e. any code which does not rely on particular neurons with specific connectivities to convey a message .
Associated with each kind of code are the processing elements best suited to produce and interpret it. A set of information processing elements should be capable of both generating the coded form (encoding) and interpreting signals sent in the coded form (decoding).
Generally speaking, rate codes can be generated and interpreted by populations of elements having long integrative time constants, place codes by populations of elements having very specific connectivities and a broad range of synaptic efficacies, temporal pattern codes by populations of elements with highly tuned intrinsic temporal resonances (e.g. recurrent conduction times or recovery kinetics), and time-of-arrival codes by populations of elements with sharply differentiated temporal windows (e.g. coincidence detectors, adaptive control of conduction times).
3.4 The many ways
to send a message
To make these distinctions more concrete, we could imagine ourselves
in an isolated room with a panel of 10 lights and 10 telegraph keys before
us. Another group of people are sitting in a similarly outfitted room far
removed from ours. Each telegraph key is connected in 1-to-1 fashion to
a particular light in the other room and depressing a key causes the corresponding
light in the other room to flash momentarily. The lights can be either
on or off at each instant, and the faster they flicker the brighter they
appear. With this setup there are many ways that a signalling system could
be set up so that messages could be passed from one room to another.
The two groups could decide that particular lights were reserved for
signalling special events and that any pattern or number of flashes would
signal that this event had occurred. This would be a binary, "labelled
line" scheme. It be could decided that the rate of flashes (or the perceived
brightness) of each light would signal the measured intensity of a different
sensory property. One light would signal temperature, another loudness,
another degree of bitterness, and so on. If the lights were arranged systematically
to signal different properties arranged in a continuum (e.g. light wavelength,
sound frequency) the pattern of lights could convey a spectrum. The rate
of flashing of each light would signal the relative intensity of a stimulus
parameter within some specified range. In all of these schemes the particular
connectivity between the telegraph keys in one room and the lights in another
would be critical, but the timing of the lights would be irrelevant. These
codes are all therefore purely place codes. Depending upon the complexity
of the light patterns, each group would require a device for reading the
spatial brightness patterns and deciding what the message was. The longer
the time that the device could read each light, the more flashes that could
be sent, and the finer would be the resolution of stimulus intensities.
Optimally this device would have a long time to count the number of flashes.
In a large network consisting of many such groups care would need to be
exercised to guarantee that all of the wiring connections remained stable
over time.
The groups could devise a code for sending messages in the temporal patterns of the light flashes. Particular rhythms of lights would signal different events. The simplest such scheme would assign particular intervals between flashes to particular stimulus parameters, and different stimulus properties could be represented by intervals spanning different time scales. Alternately, more elaborate patterns of flashes could be sent in which the kind of information would be encoded in one part of the pattern and its magnitude in another. These would all be purely temporal pattern codes, and these codes would be unaffected by rearranging the wiring between the two rooms or by changing the transmission time between the two rooms. Thus it would be irrelevant which lights carried the rhythms or exactly when the flashes arrived, so long as the appropriate pattern was conveyed. Rather than a spatial pattern analyzer, the groups would instead need a device that could recognize rhythms to correctly decode incoming messages. Here it would help to have processing elements that themselves had intrinsic temporal properties. Because the identity of the signal channel is contained in the signal itself (as in radio), this scheme is highly adapted for broadcasting messages in a large network of interacting groups.
The two groups could also send messages by the time of arrival of flashes relative to a reference event. One group would send an initial message, and the second group would send a return message at some prearranged time after the first message. The return time of the second message (its latency) would signal the nature of the event which had occurred. A flash returning 9 seconds after the initial message might mean rain, 10 seconds snow, 11 seconds sunny weather. The number of lights flashing at the appointed time could signal an intensity, so that the temperature could be conveyed by the number of flashes returning at 50 seconds, the humidity the number at 51 seconds, and so on. Such a code would be impervious to rearrangements of the wires, although a change in the transmission time between the two stations would completely alter the message which was received. Here each group would need some kind of resettable clock. This scheme would be useful in large networks where the transmission times between groups are stable and heterogeneous (they are different distances apart or conduction velocities differ), where the response return time (or reverberation time) could signal which stations are replying.
Combinations of these codes could also be arranged. Groups could decide that a temporal pattern of flashes in one light might mean something different from the same pattern in another light (interval-place code). Or that a pattern of flashes in one light must be accompanied by another pattern in another light, so that a complex spatiotemporal pattern is conveyed (spatiotemporal pattern code). If the transmission velocities of the wires are different, then a message could depend upon both the time of arrival of a flash and which light was activated (latency-place). Similarly, a temporal pattern arriving at one time after a reference event might connote something different from that same pattern arriving much later (gated-temporal pattern code).
A sensory map is formed by an ensemble of elements which represent information
using combinations of two or more codes whose parameters are systematically
ordered in some way. Usually this is cast in terms of a spatially-coded
parameter ("place") vs. some other parameter (e.g. average rate, latency,
preferred delay). Four sensory maps are schematized in Figure 3; obviously
other (e.g. spatially-distributed "mosaics") representations are possible.
Although the coding schemes are cast in terms of auditory representations,
all of these strategies are possible in any spatially ordered array of
sensory neurons where there is some time structure present in individual
channels. For many modalities, the one dimensional tonotopic axis would
become a two-dimensional map. Thus tonotopic position would be homologous
to somatotopic position in somatoception and retinotopic position in vision.
The rate-place scheme ( Fig. 3A) is the most familiar -- a central
representation is formed by spatially organized differences in firing rates.
The role of the sensory cortex in these schemes is to recognize complex
spatial patterns relay these recognitions to higher centers, and finally,
via motor cortex, to motor outputs. Three general difficulties for rate-place
coding as the sole representational vehicle are the dynamic range problem,
the pattern recognition problem, and the multiple object problem.
The "contrast degradation" problem arises because the discharge rates of primary sensory neurons tend to saturate at higher stimulus levels and the spatial excitaton patterns become broader. Spatial excitation patterns therefore should be less well delineated (lower "contrast") for moderate and high levels than at levels just above threshold, where discharge rates increase more rapidly with increasing intensity. However, for psychophysical discrimations (e.g. pitch), the opposite is usually the case -- moving from threshold to moderate levels, discrimination steadily improves and levels off. While peripheral compensatory mechanisms (e.g. cochlear efferents), ranges of thresholds (e.g. different spontaneous rate classes), and particular connectivities for threshold classes may theoretically allow the entire dynamic range to be encoded by mean rates, signal-to-noise ratios still change in the wrong direction with level unless very specific compensatory connectivity patterns are assumed.
The pattern recognition problem arises because extremely subtle and complex spatial patterns must be extracted to account for percepts such as periodicity pitch or visual texture discriminaton. A 200 Hz click train, for example, elicits the same pitch as a 200 Hz tone, even if the lower frequency components of the click train are masked with low frequency noise (so that one cannot hear them separately). although the click train consists of many harmonics spaced equally in frequency and covering the entire frequency map. Frequency separations by themselves are not what the putative central spatial processor uses because shifting all freuqencies by a constant amount results pitches not equal to the frequency spacings. However the central processor operates, it must perform analysis on frequency ratios. Thus to deduce the 200 Hz fundamental from a spatial frequency map, it is necessary to simultaneously extract and compare all of the frequency ratios of the peaks of excitation and to compute what the common fundamental frequency would be. This would demand an extremely powerful, subtle, and elaborate spatial processor.
While it appears that sufficiently large connectionist networks can handle any one pattern recognition task by brute force, it is not yet clear whether these networks could simultaneously handle all of the many simultanous pattern recognition tasks required of a single cortical field. And as the number of independent perceptual qualities to be discriminated increases, the problem of associating combinations of qualities increases combinatorically.
Yet a third difficulty for pure rate-place coding involves segregating multiple objects represented in the map, what von der Malsberg has called the "superposition catastrophe." This problem is especially apparent when there are multiple "transparent" auditory or visual objects in the auditory (Handel 1989) (Bregman 1990) or visual scene (Bruce & Green 1985) . Each element in the processing array responds to parts of one or the other object or even combinations of parts from both objects.In order to segregate and recognize the objects, it is necessary (possibly through an iterative process) to determine which elements go together to encode a given object. As the number of objects increases (visual surfaces, voices in a cocktail party), the problem becomes combinatorically more difficult. One can postulate an extensive library of stored spatial pattern templates, but this also involves very elaborate representation, storage, and retrieval mechanisms. If the channels have internal temporal structure, however, channels with similar temporal structures (temporal patterns, synchronicities, or common movements) can be grouped together, and objects can then be separated and recognized.
The localized time-place scheme (Fig. 3B) utilizes local correlations within a spatial map to sharpen the central spatial map. Since synchronization tends to improve as levels increase, the signal-to-noise ratio improves with level and the dynamic range problem is ameliorated somewhat. While the complexities of recognizing complex spatial patterns are not reduced by this scheme, the scheme does permit the possibility of segregating multiple objects in the time domain by grouping channels by common synchronies.
Latency-place representations (Fig. 3C) use the relative time-of-arrival in different spatial locations in order to encode intensity and other qualities. The contrast degradation problem is ameliorated because for virtually all stimuli, latencies shorten with increasing level. This makes latency distributions attractive candidates for the encoding of stimulus intensities over extremely wide ranges (Stevens 1971) . As absolute latencies decrease, so do the variances of latency distributions. Latency differences can be amplified more centrally by lateral inhibition, since earlier impulses can excite inhibitory units which can deliver inhibition to surrounding regions before those regions receive their (relatively delayed) excitatory inputs (manuscript ref). Latency variances can be detected by using temporal summation properties of cells with many convergent inputs, since excitatory inputs all arriving nearly simultaneously (small latency variances) produce more transient depolarization than those arriving at different times (large latency variances). The precision of latency estimation can be improved by increasing the number of convergent inputs. Latency-place mechanisms appear to be involved in a wide variety of sensory processes: electroception, vision (motion perception, Pulfrich illusion), and stimulus localization in auditory, somatosensory, olfactory and gustatory systems (see below).
Coding by temporal pattern distributions over a population of neurons is yet another possibility (Fig. 3D). Interspike intervals are perhaps the simplest temporal patterns, but distributions of more elaborate pattern types could also be employed to encode perceptual qualities. If the responses of sensory receptors follow the fine time structure of a stimulus, then periodicities in the stimulus waveform will also be found in the spike trains of primary sensory neurons. The resulting distribution of intervals across a neural population essentially forms an autocorrelation-like representation of the stimulus, which contains the same information as would be present in a power spectrum.
Temporal pattern codes are generally not faced with the contrast degradation problem because higher stimulus levels impress upon more primary sensory neurons the temporal form of the stimulus. As in latency coding, the variability of spike initiation times is reduced with increasing level, and since the absolute timing of intervals related to threshold crossings of the stimulus waveform is more precise, then interspike intervals will more precisely reflect intervals between peaks in the stimulus waveform. In addition, as intensity increases, stimulus time patterns are impressed upon more of the population and the relative proportion of temporally structured activity increases.
3.6 Neural codes and perceptual qualities
One of the central goals of neuroscience is to understand the relationship of neural activity to human perceptual and behavioral capacities (Boring 1933; Boring 1942; Teuber 1959; Uttal 1973; Uttal 1988) and ultimately to the structure and texture of human experience itself (Boring 1933) . What is the relationship between a particular coding scheme and the perceptual space of distinctions that it subserves?
Any perceptual distinction must be realized through the differential activity patterns of the nervous system. Thus each dimension of perceptual quality should be related to a dimension of neural activity. Since codes have been defined as sign systems which have functional roles, every code at the level of the entire organism should make a perceptual difference. Since codes themselves have structure, it should be possible, given an understanding of the nature of the codes employed, to correlate spaces of perceptual distinctions with spaces of neurally encoded distinctions.
Ratio codes and central spectra are relatively simple strategies for constructing a space of perceptual qualities through the activity of a population of neurons (Figure 4). One needs only three types of receptors to encode a continuous two dimensional space of qualities (e.g. a space of colors). This is accomplished by taking ratios between the respective degrees of excitations produced by the three receptors. Usually the degree of excitation is taken simply as the average discharge rates associated with the respective receptors and their primary sensory afferents, but characteristic temporal patterns can also serve the same role. If the receptors themselves have different time courses of activations or their associated primary afferent fibers have different conduction velocities, then different characteristic time patterns and latencies are produced by the receptor populations in proportion to their excitation. Once a set of temporal pattern or latency primitives is established, a space of perceptual qualities can be constructed by taking the ratios between them at some higher station. The number of distinguishable temporal pattern primitives thus determines the dimensionality of the quality space.
One of the intriguing properties of temporal codes is that the tuning of resonances in elements in a network can in effect introduce new temporal pattern primitives into that network, in effect increasing the dimensionality of the quality space being encoded. On an abstract level, this process of dimensional increase is related to the addition of new observables to a scientific model, hence to systems-theoretic definitions of emergence (Rosen 1985; Cariani 1989; Pattee 1989; Ferenandez et al 1991; Helighen 1991; Kampis 1991b; Kampis 1991a; Cariani 1992a; Cariani 1992b; Cariani 1993).
4. Evidence for temporal codes in sensory systems
In virtually every sensory system there is some evidence for the role of temporal discharge patterns for conveying complex stimulus qualities: olfaction (Gesteland et al 1968; Macrides & Chorover 1972) , gustation (Covey 1980; Di Lorenzo & Swartzbaum 1982) , nocioception (Emmers 1981) , somatoception (Keidel et al 1960; Mountcastle et al 1969; Morley et al 1990; Rowe 1990) , electroception (Bullock 1982; Carr et al 1986a; Carr et al 1986b; Hopkins 1988; Heiligenberg 1991; Carr 1993) , vision (Chung et al 1970; Festinger et al 1971; Kozak & Reitboeck 1974; Richmond et al 1987; Kozak et al 1989; Bialek et al 1991; McClurkin et al 1991a; McClurkin et al 1991b; Wasserman 1992) .
It is not necessarily surprising that this should be the case. In sensory systems with receptors capable of following the stimulating waveform (i.e. audition, vibration perception), action potentials are created at threshold crossings and hence their timings directly reflect stimulus periodicities. In sensory systems where there are no periodicities to be followed (e.g. the chemical senses) or those whose frequencies are too high to follow (e.g. color vision), different classes of receptors can have different time courses of activation and recovery. In these systems characteristic temporal patterns can arise through the interplay of receptor activation times and lateral inhibitory interactions. The lateral inhibitory interactions are driven by latency differences. Those channels most sensitive to the stimulus will have shorter latencies, so that excitation will precede lateral inhibition. Those channels which are least sensitive will have the longest latencies, and lateral inhibition will precede excitation. The time course of lateral inhibition can also interact with the activation kinetics of different receptor types to produce temporal patterns which contain information concerning the ratios of excitations of the different receptors. From this perspective, the noted structural similarity of the olfactory bulb and the retina (Shepherd 1970; Szentagothai & Arbib 1972) may be due to their common functions in the generation of characteristic stimulus-dependent temporal relationships between the various neural elements. Once temporal pattern primitives are established, a space of perceptual qualities within each modality can be established by taking the ratios between them (via temporal correlation).
4.1 Somatoception: vibration perception
The perception of vibration is one of several cutaneous qualities that also include temperature, pressure, and pain. Human beings can distinguish between different vibratory frequencies in the range of 5-1000 Hz (Morley et al 1990) . In the somatosensory system there is considerable accumulated evidence that temporal patterns of mechanical vibration of the skin are encoded in corresponding temporal patterns of primary somatosensory afferents (Mountcastle et al 1969; Morley et al 1990; Johnson & Hsiao 1992; Mountcastle 1993) . Tactile primary sensory nerve fibers consist of three classes, rapidly-adapting (RA) fibers, slowly-adapting (SA) fibers, and fibers associated with Pacinian corpuscle (PC) receptors. Of these three classes, two are responsive to vibratory stimuli. RA fibers are excited by vibration frequencies in the 5-100 Hz range, while PC fibers are excited in the 30-1000 Hz range, and together these two classes of fibers are thought to cover the human range of vibration perception.
Because all fibers of a given class (RA, SA, PC) have similar frequency tunings, a simple place code for vibration frequency in which each fiber conveys information about a different narrow frequency band appears unlikely. The only other obvious spatial mechanism for encoding the frequencies of vibratory patterns would be an across-neuron pattern code which estimated vibration frequencies by comparing discharge rates between RA and PC afferents. Low vibration frequencies should recruit relatively more RA fibers, evoking higher discharge rates in each fiber, while higher vibration frequencies should recruit relatively more PC fibers, thereby evoking higher discharge rates in that population. One of the consequences of ratio coding, however, is that the amplitude of a one frequency stimulus should change the perceived frequency of the stimulus as the recruitment of one group of fibers saturates and the other slowly increases. (Morley et al 1990) used 30 Hz and 150 Hz vibratory stimuli with different amplitues to test the ratio coding hypothesis psychophysically, and their findings do not appear to be consistent with a ratio code. Further evidence in favor of temporal codes as opposed to ratio pattern codes for flutter perception is that different frequencies of electrical microstimulation of the hand "elicit changes in the subjective sense of frequency" (Mountcastle 1993) .
The temporal patterns of the vibratory stimulus are evident in the temporal discharge patterns of units at all stations in the ascending somatosensory pathway: primary sensory fibers, dorsal column nuclei units, and somatosensory cortex neurons (Mountcastle 1993) . While patterns of first-order (successive) interspike intervals are gradually disrupted by jitter and intervening spikes as the pathway is ascended, all-order (nonsuccessive) interspike intervals related to stimulus periodicities persist into the somatosensory cortex. From Mountcastle's experiments in alert monkeys, it appears that differences in all-order interspike intervals (rather than differences in discharge rates) are used to discriminate between two vibration frequencies. From the neurophysiological evidence for temporal coding and psychophysical evidence against ratio codes, it therefore currently appears "that temporal patterning of impulse activity remains the major candidate code for pitch perception, at least over a substantial part of the vibrotactile frequency bandwidth" (Morley et al 1990) .
4.2 Audition
In the auditory system, neural time codes are thought to be involved
in both auditory localization and in the encoding of sound qualities such
as pitch, timbre, and phonemic identity. The literature on the timing of
neural discharges in the auditory system is extensive, so only a fraction
of the many phenomena involved can be discussed here.
In many vertebrates, interaural time differences are used effectively to estimate the azimuthal position of sound sources. A general mechanism for utilizing interaural time differences for localization uses two neural pathways, one from each ear (Jeffress 1948) . Each path originates in a specific frequency region of the cochlea and the spikes in each of these frequency channels have precise and reliable latencies (relatively low jitter) relative to the stimulus. The two pathways converge on an array of coincidence detectors in the brainstem, and a range of relative delays between the two pathways are supplied by varying the length of one of the pathways (and hence its conduction time). Those coincidence detectors with the relative delay between the two pathways will discharge the most, hence interaural time differences are converted by means of a latency code into a spatial pattern of excitations (place code). In addition to the spatial patterns which are generated, the time patterns of coincidences are preserved, so that the outputs of such coincidence arrays will also contain time patterns which are common to both ears. These time patterns may be responsible for the binaural "periodicity pitches" heard when continuous noise presented to one ear is delayed by a few milliseconds and presented to the other ear. Similarly, when two harmonically-related pure tones having a common fundamental are presented to separate ears, the low pitch of the fundamental can be heard.
Auditory localization by means of time differences is most highly developed in barn owls and bats, two kinds of flying animals which hunt prey in darkness. Acoustic echolocation ("sonar"), used by bats and some aquatic mammals, involves the measurement of time delays between an emitted sound and its returning echo, and can support very elaborate representations of distant three dimensional surfaces (Simmons 1990) . Comparing relative times of arrival of a stimulus to receptors positioned at different body points is a general strategy which can be utilized by many other diverse sense modalities such as touch, taste, and smell (von Bekesy 1963; von Bekesy 1964a; Bower 1974) .
The ability of temporal discharge patterns to convey information concerning pitch and other qualities, has long been appreciated by physiologists and theorists of the auditory system (Troland 1929; Wever & Bray 1937; Wever 1949; Kiang et al 1965; Rose et al 1967; Brugge et al 1969; Rose et al 1971; Goldstein & Srulovicz 1977; Evans 1978; Delgutte 1980) . It has been argued that the quality and robustness of interspike interval information for representing stimulus components below 4-5 kHz is superior to rate-place representations and is more consistent with human levels of performance (Srulovicz & Goldstein 1977; Srulovicz & Goldstein 1983; Javel et al 1988) . Although most auditory theorists, in the tradition of Helmholtz, have retained a spatial, spectral pattern approach to central (i.e. cortical) representations of complex stimuli, purely spectral pattern theories must invoke very sophisticated central processors to recognize periodicity pitches, pitch shifts, musical intervals (e.g. octave relations), and spectral shapes over a large range of stimulus intensities and background conditions. Interestingly, these phenomena are described very simply in terms of all-order interspike interval distributions of the auditory nerve, which are essentially the autocorrelation functions of response spike trains.Thus stimulus periodicities can be the form of temporal autocorrelations in addition to the spatially-encoded spectral representations.
The phenomenon of "periodicity pitch" has long been a testing ground for various psychophysical models and neural coding schemes. With very few exceptions, periodic waveforms evoke low pitches (50-500 Hz) associated with their fundamental frequencies (F0). These pitches have been variously called "periodicity pitch", "virtual pitch", "repetition pitch", "the pitch of the missing fundamental", or "musical pitch." (Small 1970; de Boer 1976; Evans 1978; Nordmark 1978) . Such pitches are reliably heard even if there is no energy present at F0 or if the frequency region of the fundamental is masked with noise (Licklider 1954) . Periodicity pitches can be produced by broadband stimuli, such as click trains and amplitude modulated noise, with frequency components too close together to be resolved by neural rate-place mechanism (and especially at higher levels or in noise). These stimuli with their unresolved spectral patterns, however, produce amplitude modulations in many frequency channels, and these modulations create clear interspike interval patterns in many parts of the auditory nerve array. Modern temporal theories for periodicity pitch have combined interspike interval distributions from many frequency regions of the auditory nerve to produce pooled interspike interval distributions from which the pitch is then extracted (Licklider 1951; Licklider 1956; Licklider 1959; Moore 1982; Van Noorden 1982; Lyon 1984; Lyon 1991; Ghitza 1992) ; (Lazzaro & Mead 1989; Meddis & Hewitt 1991b; Meddis & Hewitt 1991a) ).
These global temporal models for periodicity pitch were tested by the author and Bertrand Delgutte by recording the temporal discharge patterns of more than a thousand single auditory nerve fibers in Dial-anesthetized cats (Cariani & Delgutte 1992a; Delgutte & Cariani 1992; Cariani & Delgutte 1993) . The all-order interspike interval distributions of individual fibers of many characteristic frequencies were summed together to construct an estimate of the population interval distribution for the entire auditory nerve array. A diverse set of stimuli with variable fundamental frequencies were constructed to investigate many complex pitch phenomena.
In order to convey the pervasive nature of the temporal patterning in the auditory nerve array, a "neurogram" of several auditory nerve fibers is shown in Figure 5. To the left is the stimulus and the cochlea, which through its mechanical properties implements band-pass filtering in which a given "place" in the spiral structure is preferentially tuned to a given frequency. As a consequence the auditory nerve fibers innervating hair cells at a given place are similarly tuned (i.e. lower sound pressures are needed at the "characteristic frequency" to generate additional discharges than at other frequencies). The tuning curves of six fibers are shown. The stimulus is a periodic waveform with a fundamental period of 12.5 ms (F0=80 Hz) and one resonance period at 1.6 msec (F1=640 Hz, a "single formant vowel"). Inspecting the corresponding peristimulus time histograms of each fiber at right, one can see that the time patterns in the different frequency channels are different. Several discharge periodicities can be readily seen in the individual channels, related to the stimulus fundamental period 1/F0 and the formant period 1/F1, as indicated above. Thus many intervals corresponding to stimulus periodicities 1/F0 and 1/F1 are present in the time structure of the discharges in the array of fibers, and these periodicities can be seen in the population interval distribution (Figure 5, bottom center). For this stimulus the most common interspike interval (1/F0=12.5 msec) corresponds with the perceived pitch period of the stimulus (12.5 msec).
For the vast majority of periodic complex stimuli presented (click trains, vowels, AM and QFM tones, AM noise, equi-amplitude harmonic tone complexes, Shepard-Risset "continuously ascending pitch staircases) it was found that 1) the pitch heard by human listeners corresponded to the most common interspike interval in the auditory nerve (the aggregate interval distribution), and 2) that the salience of the pitch heard corresponded to peak-to-background ratio in the population interval distribution. Thus it was found physiologically what had been predicted in global temporal models (Meddis & Hewitt 1991b; Meddis & Hewitt 1991a) , that the aggregate interspike interval distribution of the auditory nerve can account for a great deal of the psychophysics of periodicity pitch: the pitch of harmonic tone complexes with “missing fundamentals”, the pitches of click trains, amplitude modulated noise, pitch shift and pitch ambiguity as found with inharmonic AM tones, and pitch dominance of lower harmonics over higher ones.
It has also been known that interspike intervals of single auditory nerve fibers can convey spectral information suitable for recognizing speech (Delgutte 1980; Voigt et al 1982) (Srulovicz & Goldstein 1977; Srulovicz & Goldstein 1983; Secker-Walker & Searle 1990) . but it has only been more recently that global temporal models have been developed for this purpose (Lyon 1984; Ghitza 1988; Ghitza 1992) . Aggregate interspike interval distributions in a simulated auditory nerve array have been shown to yield high quality, noise-resistant spectral representations sufficient for speech recognition (Ghitza 1992) . Physiological experiments have similarly shown that the population interval distribution of the auditory nerve is sufficient to discriminate vowels and in many cases to identify the two single vowels present in a concurrently presented vowel pair (Cariani & Delgutte 1993) . The interval distributions characteristic of several 5-formant synthetic vowels are shown in Figure 6. Here only intervals shorter than 5 msec are needed to identify a vowel, and the response population interspike interval distributions resemble the autocorrelation functions of their respective vowel stimuli. Thus it appears that a common temporal code for pitch and vowel identity exists in the auditory periphery.
Several groups of fish have evolved the capacity to sense changes in electric fields in their immediate vicinity. Some fish passively sense changing nearby electric fields, some generate their own weak, electric fields and sense the pattern of field potentials over their body surface, and some use these generating and sensing capacities for social communication (Bullock 1982; Carr et al 1986a; Carr et al 1986b; Hopkins 1988; Heiligenberg 1991) . The mechanisms by which electroceptive fishes construct a spatial representation from very small differences in field potentials involve comparisons of spike latencies from different parts of the fish's body (Carr et al 1986a; Carr et al 1986b) . The pathways by which spikes are conveyed from the electroceptors to more central structures which do the time comparisons utilize electrical synapses (gap junctions) which produce less jitter than chemical ones.
"In gymnotoform fish, T-type receptor afferents mark the timing of the zero crossing, or phase, of a sinusoidal signal by firing a single spike at a fixed latency within each cycle of the signal. This information is coded for many points on the body surface, and it is relayed in the same form and in somatotopic order, by the spherical cells of the electrosensory lateral line lobe, to lamina 6 of the torus semicircularis in the midbrain. A network within lamina 6 compares the arrival times of spike from pairs of points on the body surface, and "small cells" at a location in lamina 6 representing a given point A on the body surface modulate their rate of firing in accordance with the difference between the timing of the signal in A and some other area, B. The firing of these small cells is irregular and longer locked to individual cycles of the sinusoidal signal." (Heiligenberg 1991)
Thus, in electroception it is believed that a latency-place code is converted to a place code in which various combinations of body points are represented, although the functional organization of this place-based electroceptive map for external space is still not well understood.
In the chemical senses of taste and smell, hypotheses for neural coding have generally assumed labelled line or across-pattern theories of sensory quality. Simple place coding hypotheses have encountered several difficulties. Each receptor responds to many different types of stimuli, each receptor apparently responds to a different stimulus set, and these sets do not appear to be ordered in any obvious way in order to code some perceptual dimension. This makes labelled line codes extremely unlikely -- one would need a neuron for each combination of molecular species present, and stimulus generalization would be quite problematic. "Across-pattern" codes might be hypothesized: combinations of ideosyncratic receptors are adaptively wired together so that specific combinations signal particular stimulus qualities. However, in both gustatory and olfactory systems the receptors turn over, and this poses problems for place- or connectivity-based coding of taste and odor qualities.
"The cells making up the taste bud have limited life spans. Radioactive labelling shows that they die and are replaced by new cells...As a cell goes through its life cycle, it appears to move from the edge to the center of the taste bud. Since the nerve fibers do not move, receptor cells are presumably innervated by different nerve fibers as they change location. This poses a problem for stable quality perception. The population of receptor cells that synapse with a single fiber at various times should have the same sensitivities in order to ensure that a stimulus always evokes the same neural signal." (Bartoshuk 1988)
"Perhaps the most intriguing aspect of the constant turnover of olfactory receptor neurons is the ability of olfactory receptor neurons to achieve perceptual constancy. There is good evidence for a number of species of an ability to make consistent responses to a given odorant over periods of time that rival (and even surpass) the lifespan of the receptor neuron. If this is the case then the olfactory system must be able to produce a consistent response to a given odorant even though most, if not all, receptor neurons have been replaced and the newly formed cells have made synaptic contacts within the olfactory bulb." (Mair 1986)
One solution for this problem of perceptual constancy might be through temporal pattern coding of tastes and smells. Both receptor systems have a number of structural features that could give rise to different time course of receptor activation and both systems have lateral inhibitory connections (Bartoshuk 1988; Cain 1988) that could generate patterned phase relations between different types of neurons. Here a relatively small number of temporal pattern primitives could encode independent perceptual dimensions, and their relative ratios would form the continuum of perceptual qualities that are experienced (see the discussion of color coding below).
Despite nearly complete omission in the standard textbooks and reviews (e.g. (Bartoshuk 1988) ) there is considerable evidence in favor of a temporal pattern code for taste (Di Lorenzo & Hecht 1993) :
"In the study of the neural code for gustation in the central nervous system, the temporal patterns of responses to taste are most often ignored. Typical measures of taste responses account for the overall amount of neural activity evoked by a tastant but do not reflect the temporal arrangement of spikes during the response. These measures would be adequate descriptors if the total number of spikes associated with a given response were equally distributed within the response interval; however, that is almost never the case. Instead, most taste responses are characterized by variations in the rate of firing. The time course and magnitude of these variations defines the temporal pattern of a response. Given numerous reports that different taste stimuli appear to evoke distinctive temporal patterns of response in a number of taste-related neural structures and that similar-tasting stimuli evoke similar temporal patterns of response (Fishman 1957; Scott & Erickson 1971; Ogawa et al 1973; Ogawa et al 1974; Perotto & Scott 1976; Funakoshi & Ninomiya 1977; Covey 1980; Scott & Perotto 1980; Nagai & Ueda 1981; Di Lorenzo & Swartzbaum 1982; Pritchard & Scott 1982b; Pritchard & Scott 1982a; Bradley et al 1983; Yamamoto et al 1984; Travers & Norgren 1989) , it is not surprising that several investigators have suggested that this feature of the neural response may contain important, if not essential, information about taste stimuli."
Perhaps the most direct evidence that temporal neural patterns have functional significance and underlie the perception of tastes come from electrical stimulation experiments (Covey 1980; Di Lorenzo & Hecht 1993) . In the mammalian gustatory system, three primary sensory nerves convey information from the taste buds to the nucleus of the solitary tract (NTS), which is the first nucleus in the ascending gustatory pathway. The three primary sensory nerves are the chorda tympani/greater superficial petrosal, glossopharygneal, and vagus, which are, respectively, branches of cranial nerves VII, IX, and X. In these experiments the responses of neurons in the NTS of a rat, are recorded when particular tastants are applied to the tongue. The temporal response patterns of the NTS neurons are then digitized and fed into a stimulating electrode situated in either the chorda tympani or the NTS of another rat and the behavior of the rat is observed. The control is application of a pulse train of equally spaced pulses with the same average pulse rate. Since rats have highly stereotyped orofacial behavioral responses to different tastants ("acceptance" or "rejection" licking behaviors or a neutral "jaw snap"), the behavioral responses serve as a reliable indication of how the rat perceived the electrical stimulus. When a given temporal pattern normally associated with a perception of sweetness and evoking an "acceptance" licking behavior was used to electrically stimulate the chorda tympani (Covey 1980) or the NTS (Di Lorenzo & Hecht 1993) , rats exhibited all of the behavioral signs associated with a sweet tastant. Analogously, rejection behaviors were elicited by bitter tastants and their corresponding temporal patterns of electrical stimulation. When the control temporal patterns, uniform pulse trains, were used for electrical stimulation, no such behaviors were observed. (It has been found that electrical stimulation of individual taste buds can evoke particular tastes (von Bekesy 1964b) and that different buds have different electrical frequency response curves, but this is prior to the lateral inhibitory interactions that may be in part generating the temporal patterns observed by (Covey 1980) , so that the two sets of results are not necessarily contradictory). Since electrical stimulation indiscriminately stimulates all neurons in a region without regard to their connectivities, thereby removing spatial cues, these experiments are strong evidence that temporal discharge patterns by themselves are capable of conveying gustatory quality.
Despite relatively little attention given to analysis of temporal response
patterns in the olfactory system, some early experiments gave indications
that the responses could be temporally complex and highly dependent on
the history of stimulation (Gesteland et al 1968) . Although electrical
stimulation experiments with recorded spike trains have not yet been attempted
in the olfactory system, there is a fair amount of evidence for temporal
discharge patterns characteristic of particular odor types (Macrides
& Chorover 1972; Macrides 1977; Meredith & Moulton 1978; Meredith
1981).
One difficulty with a temporal pattern theory of odor quality has been
that the observed temporal patterns can change with changing stimulus concentrations
and may be somewhat dependent upon cycles of air inhalation (sniffing).
The discussion is complicated by the difficulty of the experiments themselves
and weaknesses in many of the methods commonly used to search for temporal
patterns. The relative underdevelopment of temporal coding hypotheses and
the analytical methods needed for their validation/falsification is a serious
and pervasive problem in research in all sensory systems, including audition.
As a consequence, one must be extremely careful not to rule out whole coding
schemes on the basis of incomplete analyses in the existing literature.
In general, temporal pattern hypotheses in olfaction have not conceptualized in terms of time patterns in individual spike trains, but as changes in firing rates over time. A typical way that unit responses are analyzed (in olfaction and elsewhere) is to sum together response spike trains of many stimulus presentations to form a peristimulus time (PST) histogram. PST histograms are then analyzed for time patterns. If the temporal patterns in question are not rigidly locked to the stimulus, then much of the timing information that might be present is destroyed. This method also presents problems if the time patterns in individual spike trains are interleaved (as (Emmers 1981) reportedly found for nocioception) or not synchronized to inhalation cycles. It may be, for example, that the crudeness of the analytical methods only allows temporal patterns to be seen when neural populations have been synchronized. When stimuli are presented asynchronously with respect to inhalations, the temporal patterns may be interleaved or jittered with respect to each other (perhaps depending upon the immediate history of the unit (Gesteland et al 1968) ) and therefore would not be visible in the PST histograms.
In a series of papers and a monograph (Emmers 1969; Emmers 1970; Emmers 1976; Emmers 1981) , Raimond Emmers has reported a complex spike interval code for pain and several other sense modalities: touch, temperature, nocioception, and taste. The code, which he observed at the level of the thalamus, consists of an initial burst followed by a modality specific interval, then by several other intervals of a different, characteristic duration (as in Figure 1D). Under natural stimulation, he found these patterns interleaved with each other, so the analytical methods typically used by investigators would almost certainly miss them. Applying electrical stimulation, he was able to evoke the behavioral signs for pain when the correct temporal patterns were induced. When external pain stimuli were applied, he was able to achieve analgesic effects with electrical stimulation that disrupted these characteristic patterns. Unfortunately, his work has been largely ignored, he has retired, and there have been no reported replications or followups to this interesting work.
Historically vision has been regarded as the archetypal sensory modality, and one where time plays little or no role. Outside of the Gestaltists and the Gibsonians, vision has usually been conceptualized in static terms, where layers of successive "feature detectors" operate on localized spatial retinal patterns, and the retinal image is progressively reconstructed at higher and higher levels of abstraction (e.g. (Marr 1982) ; see discussion in (Uttal 1988)). Motion, however, appears to be essential for vision (Ditchburn & Ginsborg 1952) , and moving images would be expected to set up coherent time patterns in ON and OFF units of the retina. While temporal structure in the optic nerve is much less well understood than that in the auditory nerve, it is known that retinal ganglion cells, unlike the auditory nerve, exhibit temporally correlated discharges (Mastronarde 1989) , so that cross-neuron time patterns are a possible coding mechanism (see especially the work of (Bialek et al 1991) on the use of temporal correlations in insect vision). There is also some psychophysical evidence that several aspects of vision (color, texture, form) may utilize temporal codes, although no comprehensive temporal theory of vision based on these principles has been yet proposed.
One of the advantages of temporal coding is that it permits the multiplexing of visual information. In the late 1960's, Jerry Lettvin and co-workers (Chung et al 1970) found that information concerning conditions of illumination could be transmitted the interspike interval statistics in the dimming fibers of the frog's optic nerve (see also (Wasserman 1992) ). They observed different sets of interspike intervals which corresponded to different levels, types and time courses of illumination, and combinations of various intervals occurred together depending upon light level and other factors. Thus their experiments serve as an example of the multiplexing of visual information in interspike interval distributions of the frog optic nerve.
More recent investigations in primate visual systems have found evidence for multiplexing of information concerning visual form and color (Richmond et al 1987; Richmond et al 1989; McClurkin et al 1991a; McClurkin et al 1991b) . In these experiments images consisting of a set of black and white squares aranged in different configurations (i.e. 2-D spatial isoluminant, Walsh patterns) are presented to an awake monkey and the temporal discharge patterns of visual neurons are recorded. Temporal patterns in the response spike trains are then extracted using using principal component analysis and the informational content of the temporal patterns is assessed (i.e. how well can the stimulus that was presented be predicted from temporal or average rate patterns of discharge. In all regions of the visual system studied, lateral geniculate, primary visual cortex, and inferior temporal cortex, the response time patterns contained considerably more information than the average discharge rates. More recent work suggests the same is true for color (McClurkin et al 1993) . Other investigators analyzing spike trains in single units of the visual cortex have found joint interval patterns ("precisely replicating" spike triplets) present in numbers significantly greater than random process models would predict. (Strehler & Lestienne 1986; Lestienne & Strehler 1987) .
There is a substantial literature on the psychophysics and physiology
of "subjective color" -- colors induced by achromatic temporal patterns
(the Prevost-Fechner-Benham effect or Benham's Top (Benham 1894;
Benham 1895) ). Several patterns which evoke these colors are shown in
Figure 7A. In these patterns, weakly saturated colors are seen around the
edges of the black areas. For the colors to be seen there must be a particular
phase relationship between the three different sectors of the disk (black,
white, line pattern). Any color can be induced by the appropriate temporal
pattern of luminance changes, as shown in Figure 7B (Festinger et
al 1971) , and television shuttering devices have been built to evoke colors
with with appropriately flickered black and white images (the Butterfield
Color Encoder) (Sheppard 1968) . Perhaps even more strikingly, when
the characteristic temporal patterns are induced via electrical stimulation
of the retina, humans see phosphenes of the corresponding color (Young,
1975). Since electrical stimulation presumably excites all retinal cells
to fire in the same temporal pattern, this is strong evidence in favor
of a temporal code for color. Physiological studies of the optic nerve
(Kozak & Reitboeck 1974; Kozak et al 1989) , lateral geniculate
(Young & De Valois 1977) and visual cortex (Richmond et
al 1989) show characteristic temporal patterns when colored stimuli
are presented. Underlying the traditionally cited response patterns of
color opponency (as manifest in discharge rates) may be relative
timings of various excitatory and inhibitory events (van Esch et
al 1988) . From this perspective, the Benham top induces the appropriate
temporal responses characteristic of each color in the retinal ganglion
cells, by inducing the temporal patterns that would normally be produced
by lateral interactions between different types of (inhibitory and
excitatory) retinal cells (von Campenhausen 1969; Festinger et al
1971; von Campenhausen 1973; Jarvis 1976; Adamczak 1981; Zrenner 1983;
Tritsch 1992; von Campenhausen et al 1992) . It is thus conceivable
that the distribution of interspike intervals or some higher order time
pattern in a particular patch of visual cortex determines the color perceived
in the corresponding visual region. Since each patch is connected to other
patches by horizontal connections, time patterns in one patch can interact
with those of other patches, and the color perceived in one stimulus region
can be influenced by those in surrounding regions.
That particular temporal patterns can mimic the effects of particular colors and spatial patterns is very suggestive of the presence of generalized time-place cortical transformations that could also potentially underlie many other illusions such as periodicity pitch. As the Gestaltists held, the cortex may be the site of complex interactions between dynamic, two-dimensional spatiotemporal patterns mediated by horizontal connections (Siegel & Read 1993) , and flicker stimuli may serve to mimic some of the two-dimensional "standing wave" patterns that would be evoked by regular, textured stimuli.
5. Processing strategies for discriminating temporal patterns
One of the barriers to more serious consideration of temporal coding alternatives has been the relative underdevelopment of information processing strategies which can handle time patterns. The remainder of this paper will outline some of the possibilities for the general kinds of computations that could be performed using temporal information and the kinds of neural architectures that could conceivably realize these computations. While this is an initial, highly speculative excursion, eventually, one would want to specify the models more precisely and test them against known anatomical constraints and physiological behaviors.
What kinds of representations could be computed? One of the great advantages
of temporal codes is that correlations between patterns can be implemented
by the convergence of axons carrying the patterns onto coincidence detectors
(Figure 9) (Longuet-Higgins 1989) . If there are a range of relative delays
available in a neural subpopulation, either through conduction delays or
intrinsic oscillatory periods, then all lags in the cross-correlation function
can be computed by that subpopulation by temporal coincidence. This is
markedly simpler than a similar computation using spatially-coded patterns.
If all of the the delays are present within a given sensory channel
(e.g. an auditory frequency channel or a retinal position channel), then
the time patterns in the channel can be correlated with themselves to form
an autocorrelation function. As discussed above, global temporal autocorrelations
in the auditory nerve are effective for representing pitch (Licklider
1951; Licklider 1955; Licklider 1959; Lyon 1984; Lazzaro & Mead 1989;
Meddis & Hewitt 1991b; Meddis & Hewitt 1991a; Slaney & Lyon
1991; Cariani & Delgutte 1992a; Cariani & Delgutte 1992b; Delgutte
& Cariani 1992; Slaney & Lyon 1993) and speech (Palmer
1988; Meddis & Hewitt 1990; Ghitza 1992; Palmer 1992; Cariani &
Delgutte 1993) .
In the auditory system if a global interspike interval distribution can be computed over an entire auditory map and analyzed, then the percepts of periodicity pitch, musical intervals, and spectral shapes can all be subsumed into one general processing scheme. Because intervals intrinsically carry harmonic structure (e.g. octave relations), many form invariances (musical intervals, fusion of chords) naturally fall out of such a representation. Interestingly, in vision, autocorrelation functions which operate on spatial intervals have proven effective in modelling texture perception (Uttal 1975; Uttal 1988) . Since auto- and cross-correlations on place-based excitation patterns have been proposed for universal information processing operations (Reichardt 1961; Kabrisky 1967; von der Malsburg & Schneider 1986) , such universality would also be a property of similar functions implemented in the time domain.
Where might such functions be computed? Since the computation of all perceptual Gestalts involve assembling information across large portions of a sensory map, a high degree of connectivity across tonotopic regions ("lateral" or "horizontal" connections) must be present in the anatomical substrate. In the auditory case, the computation suberving a percept such as periodicity pitch or vowel identity requires the merging of information from all parts of the frequency map. The primary auditory cortical fields are the first stations in the ascending auditory pathway which generally meet this requirement, since the cerebral cortex contains many horizontal fiber systems coursing through the apical dendrites of its principal (pyramidal) cells (Ramon y Cajal 1894/1990; Lorente de No & Fulton 1933/1949; Gilbert 1983; Imig & Morel 1983; Braitenberg 1986; Ts'o et al 1986; Braitenberg & Schüz 1991; Gilbert & Wiesel 1992) . Lesion and ablation studies in animals and humans also suggest that the auditory cortex may be necessary for discrimination of complex acoustic patterns, such as speech and periodicity pitch (Dewson III 1964; Symmes 1966; Whitfield 1980) , though not for discrimination of simple tones (Elliot & Trahoitis 1972) .
How much temporal information reaches the cortex from sensory peripheries? Unfortunately, in no sensory system has the form and quality of temporal information available to the primary sensory cortices yet been properly characterized. Even in the auditory system, where the temporal patterns present in the auditory nerve have been well described, the time patterns present in primary auditory cortices are poorly understood. Spike latencies for stimulus onsets can be quite precise, with jitters comparable to those found in the auditory nerve (Phillips et al 1991) . There is evidence that stimulus periodicities up to several hundred Hz (Goldstein et al 1959; Kiang & Goldstein 1959; Steinschneider et al 1980; Schreiner et al 1983; Phillips 1989; Mäkelä et al 1990; Langner 1992) and perhaps up to 1 kHz (de Ribaupierre et al 1972) .are present in primary auditory cortex, so that periodicity pitch could potentially be temporally coded at the cortical level. These periodicity limits should be taken as lower ones, because the recording and analytical methods used (gross potentials, synchronization indices, first order intervals, post-stimulus time histograms) are not powerful enough to detect jittered, desynchronized, or complex temporal patterns. Until more powerful analytical methods are utilized (e.g. autocorrelation, power spectra, joint interval statistics), more elusive temporal patternings (such as those found elsewhere: (Covey 1980; Emmers 1981; Lestienne & Strehler 1987; Richmond et al 1989; Bialek et al 1991; Mountcastle 1993) ) cannot be ruled out.
How might global correlation functions be computed in the cerebral cortex? Temporal information could be stored in the resonance patterns of networks with recurrent connections (Greene 1962; McCulloch 1969) . Alternately, information could be stored in temporal correlations using adaptively tuned delay lines, coincidence detectors, and/or sets of filters (MacKay 1962; Longuet-Higgins 1969; Longuet-Higgins 1987; Longuet-Higgins 1989) . Adaptively tuned conduction blocks in axon trees could potentially parse out particular periodicities from spike trains, thereby implementing temporal pattern recognition in a single neuron (Chung et al 1970; Raymond & Lettvin 1978; Pratt 1990) . Learning could be built into various timing nets by adaptively altering conduction times (MacKay 1962) or by strengthening synapses corresponding to particular sets of existing delays, or by tuning membrane properties of pacemaker neurons (Torras i Genis 1985) . Currently time delays are being incorporated into discrete neural networks to recognize warped symbol sequences and time-varying patterns (Tank & Hopfield 1987; Mozer 1993) . Of these general alternatives, the three possible temporal processing architectures depicted in Figure 8 will be discussed in greater depth.
6. Temporal
processing in a single axon tree
An elegant theory of the single neuron as a multiplexing temporal processing
element has been proposed which utilizes the temporal properties of axonal
conduuction to perform temporal analysis on spike trains (Chung et
al 1970; Raymond & Lettvin 1978; Waxman 1978; Pratt 1990; Wasserman
1992) . Since the beginnings of single neuron electrophysiology, it has
been known that not all action potentials travelling down the axon trunk
invade all terminal branches, and that the times between successive discharges
(interspike intervals) can determine whether or not a given action potential
will be propagated down a given axon branch. In some invertebrate motor
systems conduction failures or "blocks" have been demonstrated to play
important functional roles. In the crayfish, spike trains with different
interspike interval compositions travel down different branches of an axon
tree to independently control different muscles of the claw (Bittner
1968; Perkell & Bullock 1968) .
Conduction blocks are related to the time that it takes for thresholds
to return to normal after an action potential. (Raymond 1979)
found that a many kinds of axons showed triphasic threshold recovery curves
in which each action potential is followed in turn by a refractory phase,
a "superexcitability phase", and a depression phase. During superexcitability,
the membrane is slightly easier to re-excite than when it is at rest. In
different axons, axon branches, and cell bodies, superexcitability culminates
at different recovery times, milliseconds to seconds after the last action
potential. The axon is therefore more sensitive to spike trains with intervals
that coincide with the superexcitability peak, selecting particular temporal
patterns from incoming spike trains to be propagated further on. Different
axon branches with different threshold recovery time courses could "parse"
incoming spike trains in different ways, so that particular interspike
interval distributions could excite different sets of postsynaptic neurons
(or muscles, as in the crayfish). The timing and strength of superexcitability
phases in axon branches have also been observed to be activity-dependent
and independently modifiable (Carley & Raymond, 1987), so that the
effective connectivity between neurons might be adaptively modified by
mechanisms which do not directly involve changes in synaptic efficacy.
Artificial neural networks using pulse-interval temporal parsing trees
based on these concepts have been investigated by (Pratt 1990) .
Since general anesthetics disrupt the superexcitable recovery phase,
it is conceivable that their concomittant effects on consciousness might
be due to the removal of conduction blocks which normally play the role
of "decoding" temporal patterns (Butterworth IV et al 1989)
. Thus when general anesthetics are applied, membrane tunings are lost,
the temporal coherence of neural activity is disrupted, and the functional
integrity of the network is destroyed.
7. Temporal autocorrelation architectures
Perhaps the best articulated neural architecture for processing temporally structured information is still J. C.R. Licklider's autocorrelation-based periodicity-to-place scheme ( (Licklider 1951; Licklider 1959) ). This architecture was originally developed to account for periodicity pitch in the auditory system, and forms the basis for a number of current autocorrelation-based models (Meddis & Hewitt 1991b; Slaney & Lyon 1991; Slaney & Lyon 1993) , physiological studies (Cariani & Delgutte 1992b) , and analog VLSI implementations for pitch (Lazzaro & Mead 1989) . The scheme utilizes the temporal structuring of discharges present in the auditory nerve by performing an autocorrelation analysis in each channel. In Licklider's formulation this was realized through conduction delays and coincidence detectors, but it could also be achieved using cellular "intrinsic oscillations," where cells discharge more frequently and/or more coherently in response to particular stimulus periodicities (Møller 1974; Frisina et al 1990; Kim et al 1990) . Within each frequency channel is a set of delays corresponding to a set of "periodicity" channels. This creates a two-dimensional periodicity vs. "place" map. By summing the outputs from corresponding periodicity channels in different frequency bands, the pitch of a stimulus could be computed by taking the periodicity band with the highest summed activity. This processing architecture thus has the advantage of readily explaining periodicity pitch, the phase insensitivity of the auditory system, and the fusion of place and temporal representations into a unified percept (Licklider 1951; Licklider 1954; Goldstein et al 1959; Kiang & Goldstein 1959; Licklider 1959; Simmons 1990; Simmons et al 1990) .
On the other hand, such a network requires specific anatomical structures -- precisely tuned delays or cellular oscillations ranging from 2-15 msec. Unfortunately, no obvious anatomical structures subserving delay lines (like those found in the brainstem for the computation of interaural time differences) have been found. Many cells with varied "intrinsic oscillations" are present in the cochlear nucleus, but their tunings are relatively broad (an octave or more). This does not correspond well with the stability and accuracy of periodicity pitch judgements (with errors of a few percent). In contrast, as in the auditory nerve these cochlear nucleus units typically show many interspike intervals corresponding precisely to the fundamental period of the stimulus.Thus at the level of the cochlear nucleus, an interval code still appears more promising for periodicity pitch than one based on average rates and periodicity detectors.
If a temporal-to-place transformation were taking place, one would also expect such maps to be present at higher stations. While there is some possibility that delay maps of bats ( (Simmons 1990; Simmons et al 1990; Suga 1990) ) might be homologues to periodicity maps in other mammals, no maps spanning the full range of periodicity pitches (50-500 Hz) have been found thus far (Schwartz & Tomlinson 1990; Sheich 1991) .
8. Temporal
cross-correlation architectures
From an anatomical and physiological point of view, the basic features
of Braitenberg's time-based architecture for computing auto- and cross-correlations
in the cerebellum (Braitenberg 1961; Braitenberg 1967; Freeman &
Nicholson 1970) might be well suited for transposition to the cerebral
cortex. Here, pyramidal cells perform the role of detecting coincidences
between direct thalamic inputs and indirect (delayed) inputs from local
interneurons, association fibers, and commissural fibers (Figure 9A).
These latter inputs come from the massive set of horizontal fiber systems
that make up the superficial layers of the cortex. Depending upon the relative
directions of spike train propagation, an array of coincidence elements
can compute correlations or convolutions (Figure 9B) (Longuet-Higgins
1989) . Many relative delays could be supplied by many mechanisms: 1) differences
in the cortical distances between two pyramidal cells, 2) differences in
conduction velocity of horizontal fibers 3) multiple synaptic delays 4)
reverberating loops of different lengths (a delay and its multiples), and
5) tuned intrinsic recovery kinetics of pyramidal cells.
The many intra- and inter-channel delays in the net permit a coincidence detector at each delay node to compute the auto- or cross-correlation term for a specific lag. Thus a population of coincidence elements embedded in a system of relative delays can compute global auto- and cross-correlation functions. Local inhibitory neurons would play the role of penalizing non-coincidences, so that unstructured inputs tend to inhibit the cell (Figure 10, top). Horizontal fiber systems of different orientations Additional connectivities between different regions having longer delays might be supplied by dispersive, reciprocal, cortico-cortico and cortico-thalamic links.
Since the output of each coincidence detector also has temporal structure, global cross-correlation operations could be iterated by passing the results of one layer of coincidence detections down (or up) to another layer (Figure 11, cf. (Travis 1988) ). Such iterations could provide a temporal sieve through which predominant periodicities could be extracted. Various degrees of global interaction can also be implemented. Depending upon the length and distribution of synapses on horizontal fibers, local cross-correlations could be computed with short systems of horizontal fibers, while more global cross-correlations could be computed with longer systems having more far-flung synapses.
In audition, a temporal cross-correlation network would be capable of computing a global correlation function similar to the population autocorrelation functions which can support the discrimination of periodicity pitch and vowel identity. Many auditory form invariances which are related to frequency ratios (the octave, chords, periodicity pitch, musical consonance) are directly explained by the inherent harmonic structure of interspike intervals. Pairs of spike trains containing intervals related by simple, low integer ratios (i.e. 1:2 octave, 2:3 fifth, 3:4 fourth) will be more highly correlated than those whose intervals have some other relationship. If the delays involved are long enough, rhythm and higher order temporal structure can be analyzed with the same correlative operations that would be utilized for periodicity pitch (Boomsliter & Creel 1962) . The analysis of longer-term temporal patterns would open the door to the analysis of the slower time patterns associated the perceptual qualities of other sensory modalities: colors, tastes, smells, pains, and vibratory textures. Longer reverberatory times would also allow slower temporal patterns to circulate in the network, to be correlated with those which are entering at any given moment. Memory-facilitated operations would thus consist of the "broadcast" of characteristic temporal sequences throughout various networks, thereby increasing the correlations of incoming temporal patterns that were similar in some respect to the memory generated sequence.
Temporal cross-correlational operations could conceivably also play a role in vision. Texture discrimination is similar to the discrimination of different musical chords in that both can be described in terms of either power spectra (of spatial frequencies) or autocorrelation functions (of spatial intervals).
"A current interpretation of the role of frequency channels in vision is that local, but not global, Fourier analysis is performed. On this view patches of the visual image are analyzed into about half a dozen frequency bands at about twenty different orientations. There would be several thousand such patches in the whole visual field, and they would subtend a fraction of a degree in the fovea, and several degrees in the periphery. The result of the analysis would correspond to coefficients of some hundred sinusoids and cosinusoids of differing frequency and orientation for each patch, and the range of frequencies covered would vary with eccentricity and size of the patch. This scheme is a tentative one, but it is consistent with much of the psychophysical and neurophysiological evidence....In vision, the comparable advantage may be that the spatial frequency components represent a description of 'texture' which applies to the whole of each patch. This would be a step beyond a point-by-point description, just as the cochlea goes beyond a moment-by-moment of sound pressure." (Woodhouse & Barlow 1982) .
An autocorrelation-based alternative to an array of spatial frequency detectors is to represent the distributions of spatial intervals in various directions for each retinotopic point. If one can compute a local spatial interval distribution (spatial autocorrelation function) for several orientations, then one has a processing scheme which explains a large part of the psychophysics of the discrimination of texture and the recognition of dotted forms (Uttal 1975; Uttal 1988) . However, spatial distances can be transformed into temporal patterns by either a scanning process (Pitts & McCulloch 1947) , by saccades (Reitboeck et al 1988) , or by propagation through horizontal fiber systems which cross retinotopic maps at various angles (Figure 10, bottom) (Pabst et al 1989) . Lateral inhibitory connections at lower stages of visual processing could also serve the same purpose. Thus instead of arrays of spatial frequency detectors (which, like the "periodicity detectors" in the cochlear nucleus have overly broad tuning) the spatial intervals might be encoded in the time patterns of discharges at each retinotopic point.
One advantage of this strategy is that the harmonic structure of the resulting time intervals can then be used to compute ratios of spatial intervals. While musical chords are perceptually characterized by ratios of sound frequencies and remain similar when all frequencies are shifted upward by a constant factor, textures are perceptually characterized by ratios of spatial frequencies and remain recognizably similar under different magnifications. Invariance of form under different magnifications is of obvious importance to the recognition of objects which are seen at different distances. In this context, it is of interest to note that in auditory frequency maps, being roughly logarithmic, a constant distance or conduction time corresponds to a constant frequency ratio, and in the visual cortex, being also roughly logarithmic (Schwartz 1980; De Valois 1990) , a constant distance or conduction time corresponds to a constant spatial ratio. This conceptual ground, concerning perceptual invariances and "the exchangeability of time and space," was, of course, originally covered long ago (Pitts & McCulloch 1947) : "Octaves span equal cortical distances, as on the keyboard of a piano.").
Two existing texture discrimination models (Reitboeck et al 1988;
Pabst et al 1989) convert spatial intervals into time (interspike)
intervals. These models use precise neural synchronicities to bind perceptual
elements together into wholes (so as to explain the findings of Gray &
Singer (Singer 1990)), but absolute synchronicities may not be necessary
if local interval distributions can be cross-correlated in networks with
many possible delays. Since spatial autocorrelation functions could be
directly implemented in running interspike interval distributions of each
cortical patch, spatial frequency representations potentially subserving
many form invariances (e.g. translation, rotation, magnification)
(Reitboeck & Altmann 1984; Gardenier et al 1986; Vol et al
1989) may be realizable in the time domain by asynchronous delay/coincidence
mechanisms.
The asynchronous neural delay nets proposed here could perform discriminative functions by selectively strengthening synapses representing different sets of relative lags. In such a scheme adaptive synaptic modification effectively chooses which delays (hence, which correlations) are relevant to detect a given temporal pattern. In this context cortical "assimilation" of stimulus rhythms during conditioning (John 1967) may reflect an ongoing strengthening of specific sets of interneural delays in the wake of a temporally patterned input. Since characteristic temporal patterns are the output of such an array, synaptic modification in coincidence nets can also serve to adaptively generate temporally structured outputs. Since the hippocampus and the cerebellum are also organized along the same general cortical plan (horizontal fiber systems, principal cells with oriented dendritic trees, local inhibitory interneurons), analogous asynchronous adaptive timing operations could also perhaps be envisioned for those structures as well.
Ultimately, one would want to embed adaptive timing nets tuned for particular sensory tasks in a larger framework in which all specialized nets are connected to each other via recurrent pathways. Cortical anatomy shows a plentiful abundance of such recurrences (Ramon y Cajal 1894/1990; McCulloch 1947) . Each recurrent pathway has its own reverberation time, and cortical physiology shows an abundance of reverberations and slow oscillations (Chang 1959; Gerard 1959; Walter 1959b; Walter 1959a; Basar 1990) . The particular reverberation times between a particular cortical region and each of all other regions might produce characteristic sequences of returning signals. A temporally-structured message is broadcast to other nodes, facilitating the processing of similarly structured temporal patterns in the other nodes, and evoking a response from every other node which has its own characteristic pattern and return time. By this organization, complex, global resonances would be set up in the network which could be switched, depending upon the history of the network and its inputs (Greene 1962) . It is then not such a leap to connect this perspective with theories of interacting cognitive nodes in which each node has its own time course of activation and extinction (MacKay 1987) .The notion of dynamic organization as a set of global resonance patterns is an old one common to both theories of life and of neural networks: (Lotka 1924/1956; McCulloch & Pitts 1943; McCulloch 1969; Eigen 1974) ) (Rashevsky 1948; Hebb 1949; Maturana 1970; Katchalsky et al 1972; Maturana & Varela 1973; Powers 1973; Varela 1979; Pattee 1982; Kampis 1991b; Rosen 1991) .
In the words of Ernst Mach:
"What is true of the pendulum is true of every
vibrating body. A tuning fork, when it sounds, also vibrates. It vibrates
more rapidly when its sound is higher; more slowly when it is deeper. The
standard A of our musical scale is produced by about four-hundred and fifty
vibrations per second.
"...We strike as many [differently tuned]
forks as we will, the fork tuned to A is perfectly indifferent to their
notes; it is deaf, in fact, to all except its own; and if you strike three,
four, five, or any number whatsoever, of forks all at the same time, so
as to make the shocks which come from them ever so great, the A fork will
not join in the vibrations unless another fork A in the collection struck.
It picks out, in other words, from all the notes sounded, that which accords
with it.
"The same is true of all bodies which can
yield notes. Tumblers resound when a piano is played, on the striking of
certain notes, and so do window panes. Nor is the phenomenon without analogy
in different provinces. Take a dog that answers to the name "Nero." He
lies under your table. You speak of Domitian, Vespasian, and Marcus Aurelius
Antonius, you call upon all the Roman Emperors that occur to you, but the
dog does not stir, although a slight tremor of his ear tells you of a faint
response of his consciousness. But the moment you call "Nero" he jumps
joyfully towards you. The tuning fork is like
your dog. It answers to the name A." (Mach
1865)
Acknowledgments
The opinions expressed herein are solely my own. I would like to thank Nelson Kiang for many insight-provoking discussions and for the use of his excellent library and Professor Frederick Werner for his kind encouragement. The auditory experiments cited were supported by N.I.H. DC00038 and DC00119.