System Context and the Core Engineering Problem
// Why traditional telephony architectures collapse under automated AI calling workloads
High-volume outbound telephony systems live in an inherently hostile networking environment. When placing thousands of automated voice calls per hour, network carriers, telecom gateways (Twilio, Telnyx), and remote human phones produce asynchronous, out-of-order event bursts.
In our system, two separate pipelines operate concurrently on the same lead: an ingestion and pacing pipeline running on AWS serverless event buses, and a real-time bidirectional audio bridge streaming raw 8kHz PCM audio directly between the telephone handset and an LLM-based voice model. If the state machine handling these events is not rigorously coordinated, race conditions cause catastrophic failures: dual dialing (two AI agents calling the same property owner simultaneously), ghost leases, and carrier spam labeling.
This article breaks down how we solved these concurrency bottlenecks, structured the codebase across 11 microservices, configured Infrastructure as Code via AWS SAM, and calibrated memory and timeout allocations to guarantee sub-second conversational latency.
- ▪Zero duplicate calls guaranteed by transactional PostgreSQL row-level advisory locks paired with TTL-driven DynamoDB sweep locks.
- ▪Sub-second voice response achieved by opening WebSocket audio bridges in parallel with outbound ringing, buffering the initial greeting before pickup.
- ▪Predictable pacing enforced by a continuous SQS metronome that throttles dispatch to match telecom carrier rate limits and regulatory windows.
The Core Service Highway: Decoupling Microservices from Cascading Teardowns
// Why direct template references create cloud deployment deadlocks and how a central highway pattern solves it
In distributed serverless architectures, a common early design mistake is allowing microservices to communicate through direct CloudFormation template references or cross-stack exports (!GetAtt and !ImportValue). In early iterations, when Service A (such as the Webhook Receiver) directly referenced outputs from Service B (such as the Pacing Metronome or State Arbiter), CloudFormation created rigid, unbreakable dependency locks.
This coupling turned deployment into a nightmare. If an engineer needed to update, roll back, or take down a single microservice, AWS CloudFormation rejected the action with export-in-use errors. Taking down or fixing one service required taking down multiple interconnected services across the stack.
To permanently eliminate this deadlock, we re-architected the system around the Core Service Highway pattern. The Core Service acts as the centralized highway of the entire repository. It encapsulates all shared contracts, Pydantic schemas, database connection pool factories (AWS RDS Proxy), centralized JSON logging, and standard event bus topics.
Every individual microservice (the leaves) connects directly to the Core Service Highway, but leaves never import, reference, or know about each other. The operational benefit is massive: you can deploy, update, or completely tear down the Webhook Receiver or Audio Bridge in seconds without affecting or taking down any peer service. The only architectural tradeoff is that modifying the Core Service itself is a foundation event that requires coordinated testing, but because business logic lives in the leaves, 99% of deployments remain completely autonomous.
- ▪Zero cross-service CloudFormation deadlocks: Independent stacks deploy and destroy cleanly without cascading export-in-use errors.
- ▪Centralized highway contract: All microservices share uniform Pydantic event schemas and database pooling through the core layer.
- ▪Isolated blast radius: A failure or rollback in one leaf service cannot bring down or block updates to adjacent services.
Codebase Topology, Parser-Executor Pattern, and Mirrored Tests
// Strict separation of concerns with isolated parser adapters, domain executors, and 1:1 mirrored test suites
The repository layout reflects our Core Service Highway and microservice isolation rules. Rather than an unstructured directory of scripts, each microservice follows a strict internal Parser-Executor-Handler pattern to separate network protocol adapters from pure business logic.
Within each microservice: parser.py ingests raw external wire data (such as carrier webhook payloads, SQS messages, or WebSocket audio frames) and validates it against Core schemas; executor.py contains the pure business rules and state transitions with zero AWS Lambda or HTTP dependencies; and handler.py is a thin 25-line glue file linking the two.
Crucially, testing is governed by a strict 1:1 mirroring pattern. Inside each microservice, the tests/ folder mirrors the exact file layout of the parent directory (excluding the tests directory itself). Every source script has its own isolated mirrored test script (test_parser.py, test_executor.py, test_handler.py). This prevents test state bleeding and immediately pinpoints which layer failed during CI.
- 📁 wholesale-automation/# Root project repository
- 📁 core/# The Core Service Highway: shared contracts, DB pools, and event schemas
- 📁 contracts/
- 📄 events.py# Pydantic schemas for event buses
- 📄 telephony.py# Call state models and priority enums
- 📁 database/
- 📄 connection.py# PostgreSQL RDS Proxy pool management
- 📄 advisory.py# Transactional advisory locking helpers
- 📁 telemetry/
- 📄 logger.py# Structured JSON logging for Datadog / CloudWatch
- 📁 tests/# 1:1 mirrored tests for core foundation modules
- 📄 test_events.py
- 📄 test_connection.py
- 📄 test_advisory.py
- 📁 services/# Independent microservices (each deployable as an autonomous Lambda)
- 📁 webhook_receiver/# Ultra-fast endpoint answering Twilio status callbacks
- 📄 handler.py# Lambda entrypoint (under 25 lines)
- 📄 parser.py# Sanitizes carrier headers & validates signatures
- 📄 executor.py# Enqueues raw callbacks to SQS highway
- 📄 requirements.txt
- 📁 tests/# 1:1 mirrored test scripts
- 📄 test_parser.py
- 📄 test_executor.py
- 📄 test_handler.py
- 📁 pacing_metronome/# Dispatches calls in precise intervals to respect telecom rate limits
- 📄 handler.py
- 📄 parser.py
- 📄 executor.py
- 📁 tests/
- 📄 test_parser.py
- 📄 test_executor.py
- 📁 audio_bridge/# Real-time WebSocket server bridging Twilio Media Streams to Hume AI
- 📄 handler.py
- 📄 bridge.py
- 📄 audio_buffer.py
- 📄 vad.py# Voice activity detector
- 📁 tests/
- 📄 test_bridge.py
- 📄 test_audio_buffer.py
- 📄 test_vad.py
- 📁 state_arbiter/# PostgreSQL advisory locking and lead state updates
- 📄 handler.py
- 📄 parser.py
- 📄 executor.py
- 📁 tests/
- 📄 test_parser.py
- 📄 test_executor.py
- 📁 batch_analytics/# Post-call batch semantic extraction and seller motivation scoring
- 📄 handler.py# Triggered on queue batch size or cron
- 📄 parser.py# Extracts transcript chunks and metadata
- 📄 semantic_analyzer.py# LLM structured parameter extraction
- 📁 tests/
- 📄 test_parser.py
- 📄 test_semantic_analyzer.py
- 📁 infrastructure/# Infrastructure as Code templates and environment configs
- 📄 template.yaml# AWS SAM template defining all cloud resources
- 📄 samconfig.toml# Deployment parameters for staging and prod
- 📁 tests/# Cross-service end-to-end integration and mock telecom servers
- 📄 e2e_call_lifecycle.py
- 📄 mock_twilio_server.py# Simulates Twilio Media Streams locally
Infrastructure as Code: The AWS SAM Blueprint
// Declarative resource definition with dead-letter queues, event buses, and managed networking
All AWS resources are provisioned declaratively via AWS Serverless Application Model (SAM). We avoid clicking in the AWS Management Console to ensure environment parity between staging and production.
A key architectural pattern here is the SQS Dead-Letter Queue (DLQ) topology. Every critical message bus has a dedicated redrive policy. If a lead update fails after 3 retries, it is routed to a dead-letter queue with alerting, preventing unhandled exceptions from blocking the pacing pipeline.
Below is a production-hardened excerpt of the SAM template defining our state bus, metronome queue, and telephony webhook handler:
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Wholesale Automation Telephony and Distributed State Engine
Globals:
Function:
Runtime: python3.11
Architectures:
- arm64 # Graviton2: 20% lower cost, 15% lower compute latency
Timeout: 10
MemorySize: 512
Environment:
Variables:
STAGE: !Ref Stage
LOG_LEVEL: INFO
DB_SECRET_ARN: !Ref DatabaseSecretArn
Parameters:
Stage:
Type: String
Default: staging
AllowedValues: [staging, production]
DatabaseSecretArn:
Type: String
Description: AWS Secrets Manager ARN containing PostgreSQL credentials
Resources:
# 1. Pacing SQS Queue with Dead-Letter Queue
MetronomeDLQ:
Type: AWS::SQS::Queue
Properties:
QueueName: !Sub "${Stage}-wholesale-metronome-dlq"
MessageRetentionPeriod: 1209600 # 14 days
MetronomeQueue:
Type: AWS::SQS::Queue
Properties:
QueueName: !Sub "${Stage}-wholesale-metronome"
VisibilityTimeout: 60 # 6x the Lambda timeout of 10s
RedrivePolicy:
deadLetterTargetArn: !GetAtt MetronomeDLQ.Arn
maxReceiveCount: 3
# 2. Ultra-Fast Webhook Receiver (Twilio Status Callbacks)
TwilioWebhookFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri: services/webhook_receiver/
Handler: handler.lambda_handler
MemorySize: 256 # Lightweight string validation and queuing
Timeout: 5 # Fast return to avoid Twilio callback timeout
Events:
TwilioHttp:
Type: HttpApi
Properties:
Path: /webhooks/twilio
Method: post
# 3. Audio Bridge Worker (Real-time WebSockets & VAD)
AudioBridgeFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri: services/audio_bridge/
Handler: handler.lambda_handler
MemorySize: 1769 # Allocated exactly 1 full dedicated vCPU
Timeout: 300 # 5-minute maximum call duration
Policies:
- AWSLambdaBasicExecutionRole
- SecretsManagerReadWriteService Sizing: The Mathematics of Memory and Latency
// Why allocating 1769 MB Lambda memory is the secret to zero-jitter real-time audio
A common mistake in serverless architectures is setting memory to the minimum (128 MB or 256 MB) to save cost. In real-time audio and high-concurrency systems, this causes subtle, crippling performance degradation.
In AWS Lambda, CPU performance and network bandwidth scale proportionally with memory allocation. At 1769 MB of RAM, AWS Lambda provisions exactly 1 full dedicated virtual CPU (vCPU). Below 1769 MB, the function shares fractional CPU time slices, which introduces periodic micro-pauses during garbage collection and Python GIL operations. For a REST API, a 40ms pause is unnoticeable; for an 8kHz live audio stream, a 40ms pause sounds like a stuttering, broken robot.
Here is the exact sizing matrix we calibrated across all system components:
Audio Bridge Worker
Eliminates CPU throttling and GC pauses during live audio buffer processing and VAD computation.
Twilio Webhook Receiver
Lightweight request parsing. Must return 200 OK within 300ms so Twilio does not trigger automatic retries.
Pacing Metronome Dispatcher
Single worker running at continuous cadence to enforce strict telecom regulatory pacing without clock drift.
State Arbiter & Lock Release
Needs persistent PostgreSQL connection pool (RDS Proxy) to complete transactions without connection exhaust.
Batch Analytics Worker
Processes 25 call transcripts per batch; higher memory provides faster JSON parsing and handles large transcript token payloads.
Lead Ingestion & Normalizer
Processes large batches of property records, running phone formatting and deduplication regexes over 10k items.
Distributed Concurrency: The Advisory Lock Pattern
// How we solved out-of-order telecom webhooks using transactional PostgreSQL locks
When an outbound call is placed, up to 5 events can arrive simultaneously: the carrier status update ("ringing"), the Twilio stream connection ("media-ready"), an owner hung-up event, and an internal pacing timeout. Traditional updates like UPDATE leads SET status = ... result in whichever query lands last overwriting truth.
To solve this, we implemented a dual-locking strategy: DynamoDB with Time-To-Live (TTL) acts as an ephemeral barrier, while PostgreSQL Advisory Locks provide serialized, transactional state validation.
Here is the core database query executed by the State Arbiter before any mutation is permitted:
import psycopg2
from contextlib import contextmanager
@contextmanager
def acquire_lead_lock(cursor, lead_id: int):
"""
Acquires an application-level transactional advisory lock on a lead ID.
If another worker is currently mutating this lead, the query will wait
until the transaction completes, guaranteeing serialized state transitions.
"""
try:
# pg_advisory_xact_lock automatically releases when the transaction commits or rolls back
cursor.execute("SELECT pg_advisory_xact_lock(%s);", (lead_id,))
yield
finally:
pass
def update_call_outcome(conn, lead_id: int, new_status: str, call_duration_sec: int):
STATUS_PRIORITY = {
'PENDING': 0,
'DISPATCHED': 1,
'RINGING': 2,
'CONNECTED': 3,
'QUALIFIED': 4,
'COMPLETED': 4,
'FAILED': 4
}
with conn.cursor() as cursor:
with acquire_lead_lock(cursor, lead_id):
cursor.execute("SELECT status FROM leads WHERE id = %s FOR UPDATE;", (lead_id,))
current = cursor.fetchone()
if not current:
return False
current_status = current[0]
# Prevent out-of-order carrier packets from downgrading final states
if STATUS_PRIORITY.get(new_status, 0) < STATUS_PRIORITY.get(current_status, 0):
# Log stale event and discard safely
return False
cursor.execute("""
UPDATE leads
SET status = %s,
duration = duration + %s,
updated_at = NOW()
WHERE id = %s;
""", (new_status, call_duration_sec, lead_id))
conn.commit()
return TruePost-Call Intelligence: Batch Semantic Data Analysis and Extraction
// How we analyze call recordings and transcripts in asynchronous batches to extract seller motivation and save costs
A critical architectural lesson learned from early production runs is that heavy semantic analysis should never happen during the live call. Attempting to extract seller motivations, distress classifications, property condition details, and price spreads synchronously while streaming audio adds latency to the voice bridge and bloats real-time LLM inference costs.
Instead, our pipeline cleanly separates live conversation from post-call semantic intelligence. The moment a call reaches a terminal state (Completed, Qualified, or Failed), the Audio Bridge publishes a lightweight CallCompleted event to AWS EventBridge containing the call duration, audio recording S3 URI, and the raw turn-by-turn conversation transcript.
An asynchronous SQS queue buffers these events, triggering the Batch Analytics Worker. Operating on micro-batches of up to 25 calls, the worker passes the sanitized conversation transcripts through an LLM parser constrained by strict Pydantic schemas. This batching strategy reduces API round-trips by over 70% and achieves substantial token volume discounts compared to streaming per-turn extraction.
The semantic analyzer extracts five vital dimensions: a motivation score from 1 to 10, categorized distress signals (such as tax liens, probate, or urgent relocation), property repair tier, key seller objections, and an automated CRM deal-stage recommendation. Once validated, these structured insights are committed to PostgreSQL, triggering automated SMS follow-ups or notifying acquisitions managers in real time.
- ▪Zero latency penalty: Audio bridging workers remain dedicated exclusively to sub-second audio packet forwarding without running heavy evaluation models.
- ▪70% token cost reduction: Grouping multiple transcripts into batched structured extraction calls drastically amortizes system prompt overhead.
- ▪Strict Pydantic schema validation: Output is guaranteed to match expected data types before committing to database tables or triggering CRM webhooks.
- ▪Automated deal triage: High-motivation leads (score 8+) immediately ping acquisitions specialists via Slack and SMS, while low-urgency leads route to long-term automated drip campaigns.
from typing import List, Optional
from pydantic import BaseModel, Field
class CallSellerInsights(BaseModel):
call_sid: str
motivation_score: int = Field(ge=1, le=10, description="Urgency rating: 1 is passive, 10 is immediate distress")
distress_category: Optional[str] = Field(None, description="E.g., tax_lien, probate, code_violation, divorce, relocation")
property_condition: str = Field(description="turnkey, cosmetic_only, heavy_rehab, teardown")
asking_price: Optional[float] = None
tax_assessment_delta_pct: Optional[float] = None
primary_objections: List[str] = Field(default_factory=list)
crm_action: str = Field(description="escalate_to_buyer, nurture_60_days, dead_lead")
def analyze_call_batch(transcripts: List[dict]) -> List[CallSellerInsights]:
"""
Executes asynchronous batch semantic analysis on completed call transcripts.
Separates heavy cognitive evaluation from the sub-second live audio bridge,
reducing token overhead and eliminating runtime latency.
"""
sanitized_inputs = [
{"sid": t["call_sid"], "conversation": t["transcript_text"]}
for t in transcripts
]
# In production, invokes structured LLM extraction with JSON schema constraints
# and returns validated CallSellerInsights models ready for database insertion
return run_structured_llm_extraction(sanitized_inputs, schema=CallSellerInsights)DevOps, CI/CD, and Automated Testing
// Continuous delivery with automated canary validations and mock telecom servers
Deploying changes to a live voice system requires extreme caution. A broken deploy can burn hundreds of phone numbers or disrupt calls with live clients.
Our CI/CD pipeline runs on GitHub Actions with a multi-stage validation pattern:
1. Linting and static analysis (Ruff + Mypy strict type-checking).
2. Unit tests with mocked AWS services using moto.
3. Integration tests using a custom local mock server (mock_twilio_server.py) that opens real WebSocket connections and streams synthetic μ-law audio bursts to verify the audio bridge.
4. Staging deployment via AWS SAM, followed by an automated synthetic test call to an internal verified number.
5. Manual promotion to production with automated rollback if CloudWatch alarm 5XX errors exceed 1% over a 5-minute window.
name: Telephony Pipeline CI/CD
on:
push:
branches: [main, staging]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.11'
cache: 'pip'
- name: Install dependencies
run: pip install -r requirements-dev.txt
- name: Run Ruff Linter & Mypy
run: |
ruff check .
mypy core/ services/
- name: Run Unit & Integration Tests
env:
AWS_DEFAULT_REGION: us-east-1
run: pytest services/ core/ tests/ -v --cov=services
deploy:
needs: test
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: aws-actions/setup-sam@v2
- uses: aws-actions/configure-aws-credentials@v4
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: us-east-1
- name: SAM Build
run: sam build --use-container
- name: SAM Deploy to Staging
if: github.ref == 'refs/heads/staging'
run: |
sam deploy --config-env staging --no-confirm-changeset --no-fail-on-empty-changeset
- name: SAM Deploy to Production
if: github.ref == 'refs/heads/main'
run: |
sam deploy --config-env production --no-confirm-changeset --no-fail-on-empty-changesetArchitecture Takeaways for Engineering Teams
// Actionable design principles for building robust real-time cloud systems
If you are designing a high-concurrency cloud telephony system or real-time voice interface, keep these four foundational lessons in mind:
- ▪Decouple microservices via a Core Service Highway: Never let CloudFormation microservice stacks reference each other directly with cross-stack exports, or taking down one service will lock down the entire repository.
- ▪Always size audio processing Lambdas to at least 1769 MB: The cost increase is negligible compared to the loss of client conversions caused by GC-induced audio stuttering.
- ▪Never run heavy semantic analysis on live voice streams: Buffer transcripts and process them in asynchronous micro-batches to save 70% in token costs and protect conversational latency.
- ▪Enforce 1:1 mirrored test structures: Keep parser adapters strictly separated from domain execution logic, and mirror every source file with an isolated test file to prevent test bleeding.
Part 2: Lead Ingestion Pipelines, Geographic Pacing, and 3-Tier Waterfall Resilience
Designing or stabilizing a high-concurrency cloud telephony system?
We specialize in resilient cloud systems, real-time voice latency elimination, and bulletproof state synchronization under heavy production load.