Is It Worth Listening When We Have the Transcript?

Words, voices, and the counterfactual that an earnings call cannot give us.

An earnings call can last an hour. Reading its transcript is faster. Searching it is easier. A language model can summarize it in seconds. Why should an analyst, an investor, or a machine spend time listening to the recording?

The intuitive answer is that the transcript tells us what was said, while the recording also tells us how it was said. This sounds obvious, but it is not clear that this additional information is useful. What exactly is it? Can we measure it? Does it alter the judgment of a human or a machine? Or does an audio model simply exploit the speaker, the microphone, and other accidental clues, as in the now-classic, deliberately constructed example of a wolf-versus-husky classifier that relied heavily on snow in the background (Ribeiro, Singh and Guestrin, KDD 2016)?

This is Post 1 of the series introduced in What Does an Earnings-Call Transcript Leave Out? A Research Roadmap. Before discussing data cleaning or comparing voice-cloning models, I want to define the question that the experiments are intended to answer.

N.B. This is independent academic research conducted with my colleague Hamdan Al Ahbabi in the context of his PhD thesis on multimodal AI at Khalifa University, under my co-supervision. It is presented for educational purposes and should not be interpreted as investment advice. The research is ongoing.

Table of Contents

1. What disappears in a transcript?
2. A seven-second experiment
3. Four meanings of “audio adds information”
4. Why audio minus text is not delivery
5. The confounders
6. The counterfactual we would like to hear
7. Where voice cloning enters
8. What would count as evidence?

What disappears in a transcript?

Consider an executive answering a difficult question about declining margins. The written answer may appear reassuring. In the recording, the same answer may sound hesitant, impatient, carefully rehearsed, or genuinely confident.

The transcript retains the words. The waveform contains at least five partly entangled components:

1. Words: the linguistic content of the answer.
2. Speaker identity: the person’s voice, accent, age-related characteristics and habitual speaking patterns.
3. Delivery: pitch, rhythm, pauses, emphasis, loudness, fluency, affect and conversational stance.
4. Interaction: interruptions, turn-taking, latency, overlap, and the way an answer relates to the preceding question.
5. Recording channel: microphone, telephone line, compression, room noise and editing.

Only the first component survives reasonably well in a conventional transcript. Even then, punctuation, disfluencies and transcription choices can change how the words are read.

It is tempting to call everything else “tone”. That label is too broad. A low pitch may be a stable property of a speaker rather than evidence of confidence. A long pause may express hesitation, but it may also come from network latency or a turn-taking mistake. A rough signal may reflect concern, or simply a bad conference-call codec.

This distinction is especially important for modern speech embeddings. A single vector can contain information about the words, the speaker, the delivery and the channel at the same time. A classifier may be accurate without using the component that we intended to study.

A seven-second experiment

Let us make the question concrete with one authentic earnings-call answer.

An analyst proposed a detailed interpretation of the company’s margin guidance and supply-chain costs. He ended with: “Is that a fair way to maybe look at that?”

The executive replied:

Certainly a creative way to put me in the corner, Kurt. I’m not going to fundamentally disagree with your statement.

Before listening, form your own impression of the answer. The controls below do not submit or record anything. They are simply a way to make your initial reading explicit.

Now open the box and listen to the complete seven-second answer.

Did the voice make the answer sound more amused, more confrontational, more guarded, or more cooperative than it appeared on the page? Did your assessment of responsiveness change, even though every word was already visible?

This clip is derived from the public FinCall-Surprise corpus, whose official repository distributes the corpus under the Apache License 2.0. We extracted and resampled this seven-second excerpt.

Your reaction is not an experimental result. This deliberately chosen example is meant only to make the research question tangible: the same words may leave a different impression once their delivery is heard. But it is an illustration, not an experiment. The clip was selected because it is interesting, everyone reads the transcript before hearing the recording, no one is randomly assigned to a text or audio condition, and the page records no responses. It cannot tell us how often listening changes an impression, by how much, or whether the effect generalizes to other answers and listeners.

Four meanings of “audio adds information”

The sentence “audio contains information beyond text” can refer to several different claims. They require different experiments.

The first question is useful, but it is the weakest interpretation. If an audio model predicts an outcome better than a text model, the improvement could come from delivery. It could also come from recognizing a company executive, a particular recording setup, the location of an utterance in the call, or errors in the transcript.

The second question motivated our earlier work on disentangling speech embeddings. We regressed audio representations on corresponding text representations and studied the residual. This can reduce linguistic information, but it does not magically produce a pure measure of delivery. What remains can still mix microphone, telephone-line, compression and recording-quality effects with apparent age, perceived gender, accent and other habitual characteristics of the speaker. Even a pause is ambiguous: it may reflect hesitation, turn-taking or network latency. The result also depends on the audio encoder, the text representation and the function used to remove their shared information. Linguistic information may remain, particularly in nonlinear form.

The third experiment answers an important practical question. If people make a different judgment after hearing the authentic answer rather than reading it, then the choice of medium matters. However, listening changes several things at once. It introduces the speaker, timing, interaction, channel and the greater time and attention required by sequential speech. It estimates the effect of the complete audio experience, not delivery in isolation.

Our main research question is therefore the fourth one.

Why audio minus text is not delivery

Suppose a text model assigns an uncertainty score of 0.30 to an answer, while an audio model assigns 0.55. It is tempting to write:

0.55 - 0.30 = 0.25 of vocal uncertainty

This subtraction has no such interpretation.

The models may have different architectures, training datasets, calibration and error patterns. The audio model may transcribe the words internally, but not in the same way as the text model. Its score may react to noise or speaker identity. The two numbers need not even use the same effective scale.

The comparison can show that one input and model combination behaves differently from another. It cannot identify which part of the waveform caused the difference. The difference is therefore only an unexplained remainder. Delivery may contribute to it, but so may speaker identity, recording artifacts or differences between the two models. Calling the entire remainder “paralinguistic signal” merely renames this mixture; it does not tell us what produced it.

This is analogous to removing the price of an apple from the price of a fruit basket and calling the remainder “the price of the oranges.” The basket contains more than apples and oranges, and the components may not have been priced in the same way.

The confounders

Earlier studies give good reasons to take vocal information seriously. Mayew and Venkatachalam found that measures of managerial affect in earnings-call audio were associated with contemporaneous returns and future firm performance, incrementally to quantitative news and linguistic content (The Power of Voice, 2012). More recent work studies the economic consequences of acoustic comprehensibility and reports that managers’ vocal delivery quality deteriorates with some forms of negative or transitory news (Baik, Kim, Kim and Yoon).

These are interesting associations. They do not, by themselves, establish that changing delivery would change a listener’s conclusion.

Suppose that a company has received bad news. The news can affect the words in the answer, the executive’s delivery and the market reaction. A difficult analyst question can affect both the content and the delivery of the response. Speaker seniority, native language, preparation, fatigue and familiarity with the analyst may also influence how the answer sounds and how it is interpreted.

business situation ──→ answer words ─────────────┐
        │                                         ├──→ listener judgment
        └────────────→ delivery ──→ recording ───┘

question and context ─→ words and delivery
speaker and accent ───→ delivery and judgment
channel and noise ────→ measured recording

If negative news produces both cautious words and a strained voice, the two signals are not independent observations. If experienced executives speak more smoothly and are also trusted more, fluency and credibility share a cause. If remote participants have worse connections and answer different types of questions, channel quality is not random either.

Regression controls can reduce some of these problems. They cannot guarantee that every relevant cause has been observed and measured correctly. This is why our broad question gradually becomes a causal one.

The counterfactual we would like to hear

For one real answer, we would like two versions:

same analyst question
same answer words
same apparent executive
same recording quality
different delivery

We would then randomly assign listeners to the versions and compare their judgments. Did one version sound more responsive? More credible? More evasive? Did it change the listener’s interpretation of the same words?

The difficulty is that the original earnings call contains only one realized delivery. We cannot ask the executive to return to the past and give exactly the same answer under a different vocal condition. This missing version is the counterfactual.

A simpler listening-versus-reading experiment does not create it. That design compares the complete authentic recording with text on a page. It remains worth doing, because it tests whether the medium matters in practice. It does not tell us which property of the recording produced the difference.

Where voice cloning enters

Voice cloning is often presented as a horse race: which model sounds most natural, or which clone is hardest to distinguish from a recording? We will run that comparison because it is necessary to know whether any model is good enough for the next experiment.

The main purpose is different. Provided that the horse race identifies a model that preserves the words and the apparent speaker, controllable voice cloning may help us approximate the missing counterfactual. We can synthesize the same answer several times while requesting, for example, a more assured or a more tentative delivery.

This does not make the causal problem disappear. Synthesis can introduce new confounders:

• The “tentative” version may also sound less natural.
• The “concerned” version may alter the apparent age or identity of the speaker.
• A style instruction may change the words, omit a number, or introduce a second voice.
• One condition may contain more artifacts or a different loudness.
• The model may ignore the instruction completely.

If the requested delivery changes together with naturalness, identity or content, we do not have a clean intervention. The manipulation must therefore pass three levels of checks before we study its effect on judgment:

1. Did the model reproduce the exact answer with one stable speaker?
2. Did it preserve the apparent identity and acceptable naturalness?
3. Did independent listeners actually perceive the requested delivery change?

Only after these checks can a randomized listener study address the consequence of delivery. Voice cloning is thus both a model to benchmark and, if it proves good enough, a tool for constructing the experiment.

What would count as evidence?

At this stage, several outcomes remain possible.

First, audio may improve prediction over text, while the source of that gain remains unclear. This would establish practical value, but not a causal effect of delivery.

Second, listeners may distinguish vocal styles without changing their judgment of the answer. In that case, delivery exists and is perceptible, but the transcript may be sufficient for the particular judgment we tested.

Third, a controlled change in delivery may alter perceived responsiveness, credibility or evasiveness while the words and apparent speaker remain stable. That would provide direct evidence that the recording carries a consequential signal that the transcript cannot represent.

Finally, the cloning instrument may fail. If it changes identity, damages the content, or produces styles that independent listeners cannot recognize, then we have learned something about current voice-cloning technology, but not about the causal effect of vocal delivery.

None of these results would make audio a lie detector. Perceived hesitation is not proof of deception. Perceived guardedness is not evidence that a statement is false. The proposed experiment concerns how delivery affects interpretation, not whether we can infer a speaker’s private mental state.

Before we can run it, we need a much more pedestrian object: a clean reference clip containing one executive, a correctly aligned analyst question, and a complete answer spoken by that same executive. In the next post, I will explain why constructing a dataset of reliable audio triplets proved surprisingly expensive, and why its first version had to be discarded.