Speech

Speech is a community skill for integrating text-to-speech and speech-to-text capabilities into applications, covering voice synthesis, audio transcription, speaker detection, real-time processing pipelines, and audio content generation for voice-enabled experiences.

What Is This?

Overview

Speech provides integration patterns for both text-to-speech synthesis and speech-to-text transcription services. It covers API configuration for voice generation, audio format handling, streaming speech output, batch transcription processing, speaker diarization, and language detection. The skill enables developers to add voice interaction capabilities to applications using cloud-based speech services or local inference models.

Who Should Use This

This skill serves developers building voice-enabled interfaces for applications, teams creating audio content from text at scale, and engineers integrating speech transcription into meeting notes, customer support analysis, or accessibility features.

Why Use It?

Problems It Solves

Implementing speech capabilities from scratch requires deep audio processing expertise and significant computational resources. Different speech APIs have incompatible interfaces that make switching providers costly. Audio format conversion between recording sources and API requirements adds preprocessing complexity. Real-time speech processing requires streaming architectures that differ from standard request and response patterns.

Core Highlights

Text-to-speech synthesis generates natural-sounding audio from text with configurable voice, speed, and pitch parameters. Speech-to-text transcription converts audio files and streams into text with timestamps and confidence scores. Speaker diarization identifies who spoke which segments in multi-speaker recordings. Streaming support processes audio in real time for live transcription and voice response scenarios.

How to Use It?

Basic Usage

import httpx
from pathlib import Path
from dataclasses import dataclass

@dataclass
class TTSRequest:
    text: str
    voice: str = "alloy"
    speed: float = 1.0
    output_format: str = "mp3"

class SpeechClient:
    def __init__(self, api_key: str):
        self.client = httpx.Client(
            headers={"Authorization": f"Bearer {api_key}"},
            timeout=60.0
        )

    def synthesize(self, request: TTSRequest, output_path: str) -> Path:
        resp = self.client.post(
            "https://api.openai.com/v1/audio/speech",
            json={
                "model": "tts-1", "input": request.text,
                "voice": request.voice, "speed": request.speed,
                "response_format": request.output_format
            }
        )
        resp.raise_for_status()
        path = Path(output_path)
        path.write_bytes(resp.content)
        return path

    def transcribe(self, audio_path: str, language: str = "en") -> dict:
        with open(audio_path, "rb") as f:
            resp = self.client.post(
                "https://api.openai.com/v1/audio/transcriptions",
                data={"model": "whisper-1", "language": language},
                files={"file": f}
            )
        return resp.json()

Real-World Examples

class AudioContentPipeline:
    def __init__(self, client: SpeechClient, output_dir: str):
        self.client = client
        self.output_dir = Path(output_dir)
        self.output_dir.mkdir(exist_ok=True)

    def text_to_audiobook(self, chapters: list[dict]) -> list[Path]:
        paths = []
        for ch in chapters:
            request = TTSRequest(
                text=ch["content"], voice=ch.get("voice", "nova"),
                speed=ch.get("speed", 0.9)
            )
            path = str(self.output_dir / f"chapter_{ch['number']}.mp3")
            self.client.synthesize(request, path)
            paths.append(Path(path))
        return paths

    def batch_transcribe(self, audio_files: list[str]) -> list[dict]:
        results = []
        for audio in audio_files:
            transcript = self.client.transcribe(audio)
            results.append({
                "file": audio,
                "text": transcript["text"],
                "duration": transcript.get("duration")
            })
        return results

Advanced Tips

Split long texts at sentence boundaries before synthesis to avoid quality degradation on very long inputs. Use the verbose JSON response format for transcription to get word-level timestamps useful for subtitle generation. Cache synthesized audio for repeated content to avoid unnecessary API calls and costs.

When to Use It?

Use Cases

Generate audio versions of articles and documentation for accessibility and audio consumption. Transcribe meeting recordings into searchable text with speaker attribution. Build voice interfaces for chatbots that respond with natural speech output.

Related Topics

Audio processing libraries, speech recognition models, voice assistant architecture, WebRTC for real-time audio streaming, and subtitle generation from transcripts.

Important Notes

Requirements

API credentials for a speech service provider such as OpenAI, Google Cloud Speech, or Azure Speech. Audio files in supported formats for transcription inputs. Sufficient storage for generated audio output files.

Usage Recommendations

Do: choose appropriate voice models for the content type and audience. Validate transcription accuracy for critical content before using results downstream. Handle audio format conversion before API submission to match expected input formats.

Don't: send extremely long text blocks as single synthesis requests, which may time out or degrade quality. Assume perfect transcription accuracy for noisy or multi-speaker audio without review. Store audio files without compression when storage costs are a concern.

Limitations

Speech synthesis quality varies across languages and voice options. Transcription accuracy decreases significantly with background noise, accents, or domain-specific terminology. Real-time streaming adds latency requirements that may not be met by all speech API providers. Custom vocabulary support for domain-specific terms varies by provider and may require additional configuration or fine-tuning.