~/douglas-montanus
← All posts

Building an AI Phone Receptionist with Gemini Live, LiveKit, and Google Cloud

October 21, 2026 · 11 min read

AIVoice AIGoogle CloudLiveKitGemini

Small contractors have a phone problem. The calls that matter, like a homeowner with water coming through the ceiling, arrive mixed in with a steady stream of people selling SEO, "Google listing verification", lead lists, and merchant processing. Every one of those interrupts someone on a roof. Miss a real customer and they call the next roofer on the list.

So I built an AI receptionist for RL Spencer Roofing. It answers every call, figures out within a sentence or two whether the caller actually needs roofing, shutters, gutters, or lanai and screen work, and then does one of two things. Real customers get connected live to a person, or booked for a visit if nobody picks up. Everyone else gets a polite "we'll have someone call you back" and never reaches a human.

This post walks through how it's built, the architecture decisions, and the handful of things that bit me along the way.

The architecture

Caller
  │  PSTN
  ▼
Telnyx (SIP trunk)
  │  SIP + RTP
  ▼
Google Compute Engine VM (us-central1)
  ├─ LiveKit SIP gateway      turns the phone call into a WebRTC participant
  ├─ LiveKit server           one room per call
  ├─ Redis
  └─ Agent worker (Python)    joins the room, owns the conversation
        │  WebSocket (bidirectional audio)
        ▼
  Vertex AI: Gemini Live native-audio model (speech in, speech out)

After the call:
  Firestore (lead + transcript) → email summary → CRM contact
  Google Calendar (appointments)

Three pieces do the heavy lifting:

  • Telnyx provides the phone number and delivers the call over SIP to my server's IP.
  • LiveKit (self-hosted) bridges SIP into WebRTC rooms and gives me a solid agent framework for turn-taking, tool calls, and room management.
  • Gemini Live on Vertex AI does the listening, thinking, and talking in one model.

Why speech-to-speech instead of a pipeline

The classic voice-bot design is a chain: speech-to-text, then an LLM, then text-to-speech. Every hop adds latency, and each component makes its own guess about when the caller has stopped talking. It works, but it sounds like talking to a bot. There are awkward pauses, and it talks over you when you interrupt.

Gemini's native-audio Live model takes raw audio in and produces audio out over a single WebSocket. Turn detection and barge-in (the caller interrupting mid-sentence) are handled by the model itself. In practice that's the difference between a caller saying "is this a person?" and a caller just talking. It also returns text transcripts of both sides, so I still get a written record of every call without running a separate transcription service.

With LiveKit's agent framework, wiring the model in is short:

session = AgentSession(
    llm=google.realtime.RealtimeModel(
        model="gemini-live-2.5-flash-native-audio",
        voice="Gacrux",
        vertexai=True,  # project and region come from the environment
        temperature=0.6,
    ),
)
await session.start(agent=Receptionist(call_ref, caller, background), room=ctx.room)
await session.generate_reply(instructions="Greet the caller now, following the FIRST TURN INSTRUCTION.")

There's no VAD plugin, no STT, and no TTS in that block. The realtime model replaces all three.

Why one VM instead of serverless

My first instinct was Cloud Run for the agent, but voice agents are a poor fit for request-scoped serverless. A worker holds a long-lived connection to the LiveKit server and streams audio continuously for the length of every call. CPU throttling between requests and scale-to-zero cold starts both turn directly into dead air on the phone.

Instead, everything runs on one Compute Engine VM with Docker Compose and host networking: the SIP gateway, LiveKit, Redis, and the agent. Agent-to-LiveKit traffic never leaves localhost, and the VM sits in the same region as the Vertex AI endpoint. A static IP gives Telnyx a stable target, and the firewall only accepts SIP signaling from Telnyx's published signaling IPs, so random SIP scanners never reach the gateway.

The prompt is the product

The code is mostly plumbing. The behavior that makes this useful lives in the system prompt, and that's where most of my iteration went.

Screening first. The prompt lists every way a homeowner might describe a job we do: "water stain on the ceiling", "pool cage", "rescreen", "roll-down shutters", "downspouts", "blown-off shingles", "insurance claim", and dozens more. The rule is simple. The moment the caller mentions any of them, stop asking questions and move toward a person. If it's unclear, ask once: "What can we help you with today?"

Marketers get a dead end. Anyone selling marketing, listings, leads, financing, software, or supplies, plus anyone asking for "the owner" without an actual roofing need, gets the same treatment. She asks who they're with, says "I'll pass that along and we'll have someone call you back", logs it, and hangs up. An explicit rule says these callers never get transferred, no matter how they phrase it. Persistent salespeople are very good at "can I just speak to the owner real quick?"

Contact details gently, in order. We already have the caller's number from caller ID, so she just confirms it's the best one to reach them. Then she asks for email and, last, the property address. The address is framed as a favor to the caller ("if you don't mind sharing the address, we can figure out the soonest time someone can get out there based on where our crews are that day"), and she drops it if they hesitate. Nobody likes being interrogated by a phone bot.

Personality is prompt plus voice. The owner wanted a down-to-earth, slightly raspy Southern woman. Gemini doesn't offer accents as a setting, but combining one of its mature female voices with a short persona section in the prompt (relaxed drawl, the occasional "hon" or "y'all", but stay professional) gets surprisingly close. The prompt also tells her how to pronounce the company name, because "RL" kept coming out as a word.

Giving her hands: function tools

The model talks to the outside world through four tools:

  • record_lead: saves name, need, urgency, and any contact details to Firestore. It can be called repeatedly, and later calls merge with earlier ones.
  • transfer_to_agent: rings a real person and connects the caller.
  • book_appointment: puts a visit on the company Google Calendar.
  • end_call: hangs up.

Warm transfer, not cold transfer

The first version used a SIP REFER, which hands the call back to the carrier to forward. It's the simplest thing that works, but once the carrier takes the call, the agent can't get it back. If the owner didn't pick up, the customer went to his voicemail.

The current version keeps the caller in the LiveKit room with hold music and dials the owner into the same room as a second SIP participant, with a ring timeout:

session.input.set_audio_enabled(False)
hold = self._background.play(BuiltinAudioClip.HOLD_MUSIC, loop=True)
try:
    await job_ctx.api.sip.create_sip_participant(
        api.CreateSIPParticipantRequest(
            sip_trunk_id=OUTBOUND_TRUNK_ID,
            sip_call_to=TRANSFER_NUMBER,
            room_name=job_ctx.room.name,
            participant_identity="rep",
            wait_until_answered=True,
            ringing_timeout=Duration(seconds=TRANSFER_RING_SECONDS),
        )
    )
    answered = True
except Exception:
    answered = False
finally:
    hold.stop()

If he answers, both humans hear each other directly, the agent goes silent, and the room is torn down when either side hangs up. If he doesn't, she comes back on the line, apologizes that the sales reps are busy, and offers a visit instead.

One catch: the phone network can't tell the difference between a person answering and voicemail answering. To keep the owner's voicemail from swallowing customers, the ring timeout is set just under his carrier's voicemail pickup time. A "press 1 to accept" screen would be more robust, and it's on the list.

Booking against a real calendar

When nobody picks up, she offers two slots: tomorrow or the next day, at the first free hour among 4, 5, 6, and 7 PM. If a day is fully booked, it's skipped. The agent checks the calendar in the background while the greeting plays (the offer comes several turns later, so there's plenty of time), then updates the model's instructions with the real options. Right before booking, it checks again, in case someone else grabbed the slot mid-call.

Getting Calendar access from a VM had a subtle wrinkle. Compute Engine VMs have access scopes, and mine stops at cloud-platform, which doesn't cover the Calendar API. Rather than stopping the VM to change scopes or shipping a key file, the agent has its service account mint itself a Calendar-scoped token through the IAM Credentials API:

source, _ = google.auth.default()
creds = impersonated_credentials.Credentials(
    source_credentials=source,
    target_principal=CALENDAR_SA,  # the VM's own service account
    target_scopes=["https://www.googleapis.com/auth/calendar.events"],
)

That requires granting the service account Token Creator on itself, and sharing the company calendar with the service account's email. No keys on disk.

After the call

When the call ends, a shutdown hook:

  1. Writes the full transcript to Firestore next to the lead.
  2. Emails a summary to the office: who called, what they need, urgency, any appointment, whether the transfer connected, and the transcript.
  3. Pushes real customers into the CRM through the same endpoint the company website's contact form uses. Marketers are filtered out before they ever reach it.

The office ends up with a CRM contact, an email, and a calendar event. Nobody has to retype anything from a voicemail.

Things that bit me

Most of the time on this project went into small, specific failures rather than the big architecture. A few worth sharing:

  • Tools fired before the sentence finished. Gemini sometimes emits the end_call or transfer tool call before, or while, it speaks the sentence that goes with it. Acting immediately cut her off mid-word. The fix is to wait until the agent has actually been quiet for a beat. Hang-up also runs in the background, so a goodbye spoken after the tool call still plays.
async def wait_until_quiet(session, quiet: float, timeout: float) -> None:
    loop = asyncio.get_running_loop()
    deadline = loop.time() + timeout
    quiet_since = loop.time()
    while loop.time() < deadline:
        if session.agent_state == "speaking":
            quiet_since = loop.time()
        elif loop.time() - quiet_since >= quiet:
            return
        await asyncio.sleep(0.1)
  • Gemini rejects integer enums. I changed a tool parameter to Literal[1, 2], and every call immediately crashed at session start, because Gemini's function schemas only accept string enums. Literal["1", "2"] fixed it. The lesson: run your tool definitions through the provider's schema validation as part of every deploy, not after a customer hears silence.
  • The default Firestore database was in Datastore mode. It was an older GCP project, so the Firestore API refused to write to (default). Creating a separately named Native-mode database for this app sidestepped it.
  • The model invents placeholders. When a caller never gave a name, Gemini filled the field with "(caller did not provide name)", which would have become a CRM contact. Making the field optional, and saying in its description to leave it empty rather than write a placeholder, fixed it. A defensive check in the CRM push catches anything that slips through.
  • Phone-quality transcription is rough on short answers. The model's own lines are always exact, but an 8 kHz "yes" occasionally transcribes as nonsense. It doesn't affect the conversation, but don't treat the caller side of the transcript as a legal record.
  • Carrier configuration matters as much as code. The trunk's number format has to match what LiveKit expects (E.164 with the plus sign). The first transfer attempt failed until the SIP connection had an outbound voice profile. IAM grants can take a few minutes to propagate before they work.

What it costs

For a small business, the economics are the point. The always-on VM is the bulk of the fixed cost, about $100 a month at the size I started with, and a smaller instance handles several concurrent calls for roughly half that. Gemini Live is billed per audio token and comes out to a few cents per minute of conversation, so a typical two-to-three-minute call costs well under a dime. Firestore stays in the free tier. Telnyx adds a dollar or so for the number plus fractions of a cent per minute. That's cheaper than an answering service, and it doesn't put marketers through.

What's next

The next steps are a "press 1 to accept" screen on transfers, a small portal for browsing calls and transcripts, and tighter CRM integration so appointments and follow-ups live in one place. But the core is already doing its job: real customers reach a person or get on the calendar, and nobody on a roof gets interrupted by a sales pitch.