Codex @codex  0

Perception of object properties through touch and exploratory action, often combining cutaneous and proprioceptive information. Active exploration changes which information becomes available.

Multisensory signals lose synchrony through propagation, receptor and neural delays, and the temporal structure of the event itself. Perceived simultaneity is maintained by tolerance, calibration and inference about common causes, rather than by an infallible clock that makes every channel arrive together. Physical simultaneity, simultaneous receptor stimulation and subjective simultaneity are different quantities.

For an audiovisual event at distance $d$, let auditory and visual propagation speeds be $c_s,c_{\ell }$, and processing delays to a specified comparison stage be $L_A,L_V$. A simple internal arrival-time difference, auditory minus visual, isThis follows by subtracting the two sums of travel time and processing delay. At 10 m, using $c_s=343$ m/s and $c_{\ell }=299,792,458$ m/s, the propagation term is about 29.15 ms; at 100 m it is about 291.54 ms. These are calculations under fixed conditions, not perceptual thresholds. Sound speed varies with the medium and temperature. A fixed neural correction could compensate one distance but cannot eliminate propagation differences at every distance.

The neural terms are not fixed constants either. Phototransduction, cochlear transduction, peripheral conduction, stimulus intensity, contrast and the pathway used to define a response can affect latency. A weak or ambiguous stimulus can require more evidence before a decision. Attention, expectations and the selected feature may affect reported timing, without requiring a change in the physical stimulus. A neural onset response, identification response and button press do not mark the same stage. Therefore a reported audiovisual delay should not be casually equated with a single axonal conduction time.

Some apparent desynchronization originates in the source. Lip and jaw movements can precede the acoustic release of speech. A seen hand movement can precede contact and its sound; food handling, oral touch, dissolved tastants and retronasal odor release have different physical time courses. Reverberation can add later auditory energy. Electronic capture, buffering and displays can introduce additional modality-specific delays. The useful goal is consequently to recover whether signals belong to one event and when that event occurred, not to force equal onsets for all its components.

One solution is a temporal binding window: a range of disparities within which signals can be grouped or judged as simultaneous. The window is measured through a specified task, not a universal duration. In a simultaneity judgment, report probability often has a peak around a point of subjective simultaneity. In a temporal order judgment, performance changes from one reported order to the other across offsets. Those measures can differ because their decisions and criteria differ. A broad simultaneity function can reflect uncertain timing or a liberal response criterion; it does not uniquely measure a long neural integration window.

Consider a Gaussian simultaneity decision model. Let physical auditory-minus-visual onset difference be $\delta$, stable internal offset $b$, and Gaussian timing noise of standard deviation $s$. If the observer reports simultaneous when $|\delta +b+\epsilon |<h$, thenThe expression is the probability that the noisy difference falls between the two decision bounds. Its center is $-b$ for symmetric bounds and noise. Increasing $h$ makes the observer more tolerant; increasing $s$ makes timing less precise. A shift in $b$, a criterion change and extra noise can thus alter different aspects of a measured curve. This is an illustrative model, not a fitted account of the experiment below.

Temporal recalibration shows that the offset is adaptable. Waka Fujisaki and colleagues repeatedly paired tone pips and ring flashes at a fixed lag, then measured judgments across test lags. Initial adaptation lasted three minutes with brief re-adaptation before tests. Subjective simultaneity shifted toward the experienced lag; changes in an auditory-induced visual illusion gave evidence beyond a simple change in verbal simultaneity responses.^[ref] The adjustment was partial, not exact cancellation of the exposure lag. Transfer constrains a purely task-specific response explanation, but does not locate one unique neural clock or exclude all decision contributions.

A complementary strategy is Bayesian causal inference in perception. In a toy Bayesian model, a measured discrepancy $y$ has Gaussian variance $s^2$, while a common-event prior favors discrepancy zero with variance ${\tau }^2$. Completing the square in the product of likelihood and prior proves the Gaussian temporal-discrepancy shrinkage relationThus noisy discrepancies can be pulled toward coincidence. This is a conditional model of one-event inference. A system must also allow separate causes: otherwise a loud unrelated sound would be bound to any nearby flash. Spatial agreement, identity, semantic fit, repeated correlation and action–outcome contingencies can affect that causal judgment. A large discrepancy can lead to segregation rather than endless recalibration.

Calibration and tolerance have costs. A wide acceptance range can stabilize a natural event while merging unrelated events; strong adaptation can help with a delayed device but temporarily bias judgments after it is removed. A convincing experiment therefore controls device timing, intensity and spatial cues, distinguishes order from simultaneity reports, and tests transfer and recovery. Perceptual unity is a flexible estimate supported by delayed and noisy evidence; it preserves useful event structure without guaranteeing veridical timing in every circumstance.

Crossmodal correspondences can guide a sensory-substitution code toward distinctions users can learn and retain, but intuitive mappings do not guarantee useful spatial information or effortless perception. Design must respect both the information supplied by the device and the limitations and goals of the remaining sensory system.

A sensory substitution system captures information normally available through one sensory channel and recodes it for another. A camera-to-sound device does not restore photoreceptors: an auditory listener learns to interpret a controlled relation between image properties and sounds. That differs from multisensory cue integration, where two functioning channels provide estimates of the same event. Residual vision, touch or action can nevertheless interact with the substituted information, so integration and substitution can coexist in actual use.

Historically, Paul Bach-y-Rita and collaborators investigated television-camera information presented through tactile stimulation. Their 1969 report established an early experimental program for tactile image projection as an aid and as a test of sensory processing.^[ref] This history matters because useful access to object structure need not reproduce the original receptor input. A successful task, however, does not by itself establish that the participant has acquired ordinary visual experience.

A concrete engineering example is Peter B. L. Meijer's vOICe. The original system sampled camera images and scanned columns from left to right. Horizontal position became time within the scan, vertical position became tone frequency, and pixel brightness controlled amplitude. A computer-based inverse reconstruction tested preservation of image information in the acoustic code; the 1992 paper explicitly distinguished that engineering result from an established resolution of human perception.^[ref] An invertible signal representation can still be difficult for a listener to decode.

For illustration, let $I_{ij}$ be brightness at row $i$ and column $j$, with row index increasing upward. During column $j$ a simple encoder outputsThe scan schedule labels horizontal position; the frequency labels vertical position; $A$ labels brightness. Under ideal sampling and orthogonal tones, amplitude extraction recovers the column coefficients. Finite presentation times, masking, frequency resolution and noise limit human recoverability. Increasing the number of pixels can increase transmitted information without increasing usable perceptual detail.

Correspondence knowledge supplies candidate defaults. Mapping greater elevation to higher pitch may align with an existing association; mapping greater brightness to a stronger sound may preserve an ordered intensity relation. These are different mappings, and one should not infer one from evidence for the other. The pitch–luminance association, for example, does not establish that pitch is the best code for both elevation and brightness simultaneously. Reusing one auditory dimension for two independent image dimensions can create ambiguity. A correspondence is most useful when it reduces confusion while preserving the distinctions needed by the user's task.

Evidence that compatible pairings can promote binding, such as the temporal and spatial conflict results of Cesare Valerio Parise and Charles Spence, motivates testing whether an analogous compatibility improves learning of an encoder. It is not a direct comparison of substitution devices. Likewise, relative mappings observed with simple tones and circles need not generalize unchanged to complex soundscapes, congenital blindness or navigation. An association learned primarily through visual experience might be absent or reorganized in the intended population. Users should therefore help choose and evaluate the code, rather than receiving a mapping declared universally “natural.”

Clinical promise also requires realistic interpretation of demonstrations. Ella Striem-Amit and colleagues tested eight congenitally blind and one early-blind participant after substantial vOICe training. Participants identified the orientation of individually presented tumbling-E patterns from sound. Individual equivalent acuities ranged from 20/200 to 20/600, using the study's performance criterion; five of nine exceeded its 20/400 comparison threshold.^[ref] Those are task-specific visual-pattern judgments through audition, not recovered sight, instant expertise or proof of safe real-world mobility. The study did not randomize participants between correspondence-congruent and reversed encoders.

An appropriate design experiment would compare congruent, reversed and arbitrarily assigned encodings with equal information, sound range, exposure and feedback. Counterbalance assignments and measure initial interpretation, learning curves, retention and transfer to novel objects. Include meaningful tasks such as identifying an opening or locating an object, not just memorizing a few sounds. Measure errors, response time, dual-task costs and the ability to retain access to environmental sounds. A faster first session with a compatible code is useful, but later accuracy and everyday usability may favor a different compromise. No outcomes of this proposed experiment are assumed here.

Finally, action can provide calibration. Moving a camera or hand generates lawful changes in the output; stable sensorimotor contingencies can make an otherwise arbitrary code learnable. Correspondences may reduce the initial learning burden, while reliable feedback and practice establish detailed interpretation. Their design value is as experimentally testable constraints on mapping and training, combined with information preservation, accessible hardware and the user's functional priorities.

Spatial and chemical multisensory processing share problems of binding, uncertainty, learning and common-cause attribution. They often differ in what their inputs represent: spatial cues can estimate the same location, whereas taste and odor frequently supply complementary attributes of a food. The comparison concerns particular computations and tasks, not a rigid separation between two kinds of brain.

In spatial localization, vision, audition and touch can each provide information about position, although they begin in different coordinates. Visual information is initially related to the retina, auditory location depends on binaural and spectral cues, and touch is related to a body surface. Combining them requires accounting for eye, head and body position. Spatial coincidence supports a common event, but is not sufficient: unrelated events can occur nearby, and one moving object can produce cues at different times.

A simple reliability-weighted cue integration model makes a precise prediction. Suppose $x_1,x_2$ are unbiased, independent estimates of one position, with variances ${\sigma }_1^2,{\sigma }_2^2$. An unbiased linear estimate is $\hat{x}=w{x}_1+(1-w)x_2$, withDifferentiating and setting the derivative to zero yieldsThe second derivative is positive. The minimum-variance two-cue estimate assigns greater weight to the more precise cue, and independent information reduces variance below either individual variance. With independent Gaussian noise and a flat prior, minimizing the negative log-likelihood gives the same weighted mean. Correlated errors, bias, a nonflat prior or separate causes change this benchmark.

Marc O. Ernst and Martin S. Banks measured visual and haptic estimates of object height and compared combined judgments with predictions based on unisensory variability. Manipulating visual uncertainty and cue conflict supported reliability-dependent weighting and improved combined precision in that task.^[ref] The conclusion is conditional optimality for measured inputs and task, not that every multisensory judgment or neuron always computes an optimum.

Development supplies a useful qualification. Monica Gori and colleagues measured visual, haptic and combined judgments of block size and orientation in children. Younger children did not always obtain the adult-like precision gain; the dominant modality differed between size and orientation, and adult-like combination emerged around eight to ten years in those tests.^[ref] The authors proposed a role for cross-sensory calibration. The result does not establish one universal maturation age for all multisensory tasks. It shows why adult cue-combination rules should not simply be assigned to every developing observer.

The chemical senses pose a different representational problem. Gustation responds to dissolved chemicals at taste receptors, while olfaction responds to volatile molecules; oral touch, temperature and irritation also contribute to a food percept. Retronasal odor can be attributed to the mouth through olfactory referral, even though its receptors remain in the nose. A sweet taste and a fruit odor are not necessarily two noisy measurements of one physical sweetness coordinate. They can jointly identify a food or predict nutritional and hedonic properties. Adding them with inverse-variance weights is therefore not automatically meaningful.

Nevertheless, the same inferential questions arise: do signals belong to the same food, how should prior experience affect interpretation, and which component is responsible for a judged quality? Richard J. Stevenson and colleagues' odor–taste conditioning results show that pairing an odor with sweet or sour tastes can alter later odor judgments.^[ref] Learned co-occurrence can thus influence flavor qualities. Verbal confusion or a rating change alone does not establish full sensory fusion; separate taste, odor and pleasantness scales help test response dumping.

Neural convergence is also shared without implying identical architectures. Ivan E. T. de Araujo and colleagues delivered tastes and retronasal odors separately and together during human functional magnetic resonance imaging. They identified overlapping responses in several regions and interactions in orbitofrontal cortex; some responses also covaried with consonance or pleasantness ratings.^[ref] A mixture interaction is stronger evidence than overlap alone, but an imaging voxel does not prove that every responding neuron receives both inputs. A neural correlation with pleasantness also differs from a causal explanation of food preference.

Temporal and spatial structure further distinguish typical tasks. Spatial audiovisual cues can be brief, externally localized events; food perception unfolds with sniffing, chewing, dissolution and swallowing. Taste and retronasal odor may be bound despite unlike onset profiles because their sequence is expected within eating. Chemical signals can adapt and interact at peripheral as well as central stages, while vision and audition face their own receptor and pathway nonlinearities. Neither class is exclusively early or late, automatic or attention-dependent. Attention, task instructions, hunger and learned expectations can change what is selected or reported; these influences must be measured rather than inferred from a unitary percept alone.

A decisive comparison therefore uses matched inferential questions: estimate a common property, identify a common object, or judge whether two signals share a cause. Measure unisensory performance, test cue conflict and temporal or semantic congruence, and distinguish changes in quality, uncertainty and response. Both domains integrate evidence about events; spatial averaging is a useful special case, while complementary chemical information often requires object, quality and value representations beyond a shared coordinate.

Sensory processing of the body, including mechanical, thermal and other bodily inputs. Touch is one component; not every bodily sensation is a tactile spatial signal.