Building a Reliable Audio Triplet Is Surprisingly Expensive

Before testing voice cloning, we needed a clean dataset: reference and answer clips from the same speaker.

We first tried to build it automatically. An AI agent used transcript parsing, Whisper alignment, fuzzy text matching, speaker encoders and other audio checks to select and cut the triplets. The agent produced a complete dataset, but when we listened to the clips, its quality was poor. Some words were cut, some clips contained two speakers, and some references and answers came from different executives. We could not use it.

We kept the automatic pipeline for searching, alignment and cutting, and added a final human review. We completed 186 triplet forms, each with a reference and an answer: 372 audio clips reviewed manually, often several times. This took roughly 20 hours of focused listening, mostly in long sessions over several weekends. Keeping full attention on every boundary, word and speaker for that long was probably the hardest part.

This post explains how we combined the automatic tools with human review to produce 100 clean triplets for our voice-cloning experiments.

This is Post 2 of the series introduced in What Does an Earnings-Call Transcript Leave Out? A Research Roadmap. In Post 1, I discussed what may be present in the recording but absent from the transcript. Here, we focus on building the data.

N.B. This is independent academic research conducted with my colleague Hamdan Al Ahbabi in the context of his PhD thesis on multimodal AI at Khalifa University, under my co-supervision. It is presented for educational purposes and should not be interpreted as investment advice. The research is ongoing.

What is an audio triplet?

For each earnings call, we need:

1. A voice reference from the executive’s prepared remarks. It must contain complete sentences spoken by one person. The voice-cloning model uses this clip as its example of the speaker’s voice.
2. The analyst’s question later in the call.
3. The authentic answer from the Q&A, spoken by the same executive.

prepared remarks                         question-and-answer session

one executive ──► voice reference       analyst ──► question
       │                                             │
       │                                     same executive
       └─────────────────────────────────────► authentic answer

voice reference + answer transcript ──► cloned version of the answer
authentic answer vs. cloned answer   ──► comparison

If two people speak in the reference, which voice should the model clone? If another executive gives the answer, the comparison does not make sense. The two clips must contain the same person.

Why the transcript is not enough

We started from the FinCall-Surprise corpus, which contains transcripts and audio for earnings calls from 2019 to 2021.

A transcript row labelled Executives does not say which executive is speaking. The CFO may present the financial results and the CEO may answer the analyst. Both rows look valid in the transcript, but the voices are different.

Timing causes another type of error. The transcript separates turns but does not give the exact start and end of each word. A clip can miss its first sound, stop before the last word is complete, or include the next speaker’s “thank you.”

We also found disagreements between the transcript and the recording: a name, a short phrase or a number. A clip containing only part of a spoken year is not correct, even if we can guess the year from the sentence.

We also found speaker handoffs inside the prepared remarks, analysts saying “right” or laughing inside an answer, and recordings for which changes in volume or channel made the same-speaker decision difficult.

Clipped words and extra speakers were usually easy to hear. Speaker similarity was sometimes less obvious, especially when the channel, volume or speaking style changed. A table of transcript rows cannot resolve these audio problems.

Tools used

Out of 2,688 transcript calls, 2,671 have a matching MP3. We excluded the other 17 because of a problem in the source metadata.

We parsed the prepared remarks and Q&A, then grouped each analyst question with the following executive answer. Basic text and length rules found 8,367 possible exchanges from 1,894 calls. At that stage, they were transcript rows that might yield usable audio. We created review forms for a much smaller set after alignment, cutting and automatic checks.

We used:

The first four tools turn a full call into candidate clips. Whisper provides approximate word positions, difflib aligns them with the source transcript, and librosa and SoundFile cut the WAV files. If a boundary is wrong, we go back to the MP3 and use the neighbouring aligned words to make a new cut. A fixed amount of extra padding was not reliable enough.

The hashes tie each review to the exact files that were played. The other models provide useful checks, but none can approve a triplet: Whisper does not identify the executive, the speaker models do not check the words, and a high speaker score can still hide a brief interjection.

What the automatic checks missed

Two small audits make this concrete.

• In the first review of 42 reference candidates, cutting exactly at Whisper’s last-word timestamp truncated the final word in 16 clips.
• After we improved the pipeline, we compared the combined automatic screen with the final nine-gate decision on 25 triplets. They agreed on 21 cases, or 84%. The automatic screen passed one triplet that the human review rejected, and rejected three that the human review accepted.

These are audits of our pipeline on small samples, not general benchmarks of Whisper or the speaker models. The automatic checks were useful for finding and ranking candidates. They were not accurate enough to decide what entered the dataset.

Human review

For every candidate sent to human review, we listened to the reference and answer and answered nine questions:

1. Does the reference contain complete sentences?
2. Does the reference contain exactly one speaker?
3. Does the displayed reference text match its audio?
4. Does the answer contain complete sentences?
5. Does the answer contain exactly one speaker?
6. Does the displayed answer text match its audio?
7. Are the reference and answer spoken by the same person?
8. Do the analyst question and executive answer belong together?
9. Is the complete source pair usable?

Every answer must be yes. An unsure answer also excludes the triplet from Core-100. We kept the rejected and uncertain cases because they showed us where the automatic pipeline failed. Recurring errors guided the next rounds of alignment, cutting and screening. If we cannot confirm that the same person speaks in both clips, we do not use the triplet.

The questions are simple. Answering them carefully takes time. Comparing the two voices may require several replays, especially when the channel, volume, accent or speaking style changes between the prepared remarks and Q&A. The beginning and end of each clip also need particular attention.

A simplified version of the form used for this review is shown below, loaded with ECV-0001 from the public dataset. You can try the review yourself. The form does not save or send any response.

It looks quick. For a clean case, my estimate is around three to five minutes. A difficult case can take longer: replay both voices, inspect the first and last words, compare the transcript, and sometimes return to a new cut.

Across the different rounds, we completed the form for 186 candidate triplets, or 372 manually reviewed audio clips.

I performed all source-quality checks myself. The task was to decide whether each triplet was usable, not to compare how different listeners perceived the voices.

How much time?

We did not run a stopwatch, so this remains a coarse estimate. At three to five minutes per form, 186 forms already imply roughly 9 to 16 hours for clean cases. Difficult cases take longer. I therefore estimate roughly 20 hours of focused listening and form filling, spread across several weekends.

This estimate covers the listening time only. It leaves out the protocol, debugging, model runs, data transfers and the work required to rebuild the dataset.

Outcome: EarningsCallVoice Core-100

The result is EarningsCallVoice Core-100:

• 100 triplets from 100 distinct earnings calls;
• one prepared-remarks reference and one authentic Q&A answer per triplet;
• the same executive in both clips;
• no cloned or synthetic speech;
• all nine human checks passed.

We fixed the Core-100 selection before running the voice-cloning experiments. We did not select examples because they produced good clones, supported our hypothesis or would look good in the paper. The exact selection rule and item mapping are in the dataset’s selection manifest.

Downloading the dataset

You can load Core-100 using Hugging Face Datasets:

pip install "datasets[audio]"

from datasets import load_dataset

dataset = load_dataset(
    "gmarti/EarningsCallVoice",
    split="core",
    revision="v1.0.0",
)
print(dataset)

Let’s look at the first item:

from IPython.display import Audio, display

example = dataset[0]

print("Item:", example["item_id"])
print("Reference:", example["reference_text"])
print("Question:", example["question_text"])
print("Answer:", example["answer_text"])

reference = example["reference_audio"].get_all_samples()
answer = example["answer_audio"].get_all_samples()

print("Reference audio")
display(Audio(reference.data.squeeze().numpy(),
              rate=reference.sample_rate))

print("Authentic answer audio")
display(Audio(answer.data.squeeze().numpy(),
              rate=answer.sample_rate))

The code selects ECV-0001, the same public triplet used in the review form above. The form shows the values printed by the code and lets you play both audio clips inline.

The dataset also contains durations, source identifiers, SHA-256 hashes and the selection record. The license and responsible-use notes are in the dataset card.

What’s next?

Core-100 is small and selected. It is not meant to represent every earnings call. Its purpose is more practical: every voice-cloning model receives the same clean reference and the same answer text, and we have the authentic answer for comparison.

In the next post, we can finally compare the models. Did the model say the correct words? Did it keep one stable voice? Does that voice sound like the reference executive? Does the result sound natural? These questions are basic, and a model has to pass them before we use it for the later experiment on vocal delivery.