Inside an AI Receptionist: The Architecture Behind Answering Every Call

Every service business has the same phone problem. The calls that pay the bills, like a homeowner with a leak or a customer ready for a new roof, are buried in a steady stream of sales pitches, robocalls, and "quick questions for the owner." When the crew is on a job site, the phone either interrupts real work or goes to voicemail, and a customer who reaches voicemail usually calls the next company on the list.

I built an AI receptionist for RL Spencer Roofing to solve exactly that. It answers every call, in a natural voice, at any hour. This post covers what it does and how it's put together, at the level of architecture and design decisions. If you want the code-level details, I wrote those up separately in Building an AI Phone Receptionist with Gemini Live, LiveKit, and Google Cloud.

What happens on a call

From the caller's side, it's a short, friendly conversation:

1. She answers immediately with the company greeting and asks for the caller's name and what they need.
2. She works out who's calling. If the caller mentions anything the company does (roofing, a leak, storm damage, shutters, gutters, lanai and screen work), they're treated as a customer right away. If it's a marketer, vendor, or solicitor, they're politely told someone will call them back, and they never reach a person.
3. Customers get connected to a person. She puts the caller on hold and rings the sales team. If someone answers, the caller is connected live.
4. If nobody picks up, she books a visit. She comes back on the line, apologizes, and offers the next open appointment slots, checked against the company's real calendar.
5. She gathers details without pushing. She confirms the callback number, asks for an email, and only then, gently and optionally, the property address.
6. The office gets everything automatically. When the call ends, the lead appears in the CRM, the appointment is on the calendar, and a summary with the full transcript lands in the office inbox.

The owner's phone only rings for people who actually want to hire the company.

The architecture at a glance

  Customer's phone
        │
        ▼
  Phone carrier (SIP trunk)
        │
        ▼
 ┌────────────────────── Google Cloud ──────────────────────┐
 │                                                          │
 │   Call gateway ──► Conversation agent ◄──► Gemini Live   │
 │                        │                  (speech AI)    │
 │                        │                                 │
 │          ┌─────────────┼──────────────┐                  │
 │          ▼             ▼              ▼                  │
 │    Call records    Google         Live transfer          │
 │    (Firestore)     Calendar       to sales team          │
 │                                                          │
 └──────────────────────────┬───────────────────────────────┘
                            │ after each call
                            ▼
                Email summary  +  CRM contact

There are four layers, each with one job:

• Telephony. A SIP trunk from a phone carrier delivers calls to the system. The business keeps its own number, and the carrier can be swapped without touching anything else.
• Real-time media. An open-source media server (LiveKit) turns each phone call into a live audio session the agent can join, and handles hold music and live transfers.
• Intelligence. Google's Gemini Live model listens and speaks directly. It decides what to say and when to take an action.
• Business systems. Google Calendar, the existing CRM, email, and a database of call records. These are the systems the business already relies on.

Five design decisions that matter

1. Speech-to-speech, not a chain of services

Most voice bots are a chain: transcribe the caller, send the text to a language model, then convert the reply back to speech. Every link adds delay, and the result feels robotic: long pauses, and talking over the caller when they interrupt.

This system uses a model that hears audio and speaks audio directly. It handles natural turn-taking and interruptions the way a person would. For a business, that's the difference between callers hanging up on "the robot" and callers simply having a conversation.

2. The AI decides, the code acts

The language model never touches the calendar, the CRM, or the phone system directly. It can only request a small set of well-defined actions: save the lead, transfer the call, book an appointment, hang up. Regular code carries each one out and enforces the rules:

• Marketers are filtered out before anything reaches the CRM.
• Appointment slots are re-checked against the calendar right before booking, so two callers can't be booked into the same time.
• Transfers have a ring limit, so a customer is never left holding indefinitely.

That split is what makes an AI system trustworthy in production. The model brings the conversation skills, and deterministic code owns everything with consequences.

3. Everything that touches audio lives together

All the real-time pieces (the call gateway, the media server, and the agent) run together on one server, in the same Google Cloud region as the AI model. Audio never takes an unnecessary trip across the internet, which keeps the conversation responsive.

I deliberately didn't use a "serverless" platform here. Serverless is excellent for websites and APIs that handle short requests, but a phone call is a continuous, minutes-long audio stream. Spin-up delays or throttling turn directly into dead air. Matching the infrastructure to the shape of the workload is one of the most important calls in any architecture, and here it pointed to a single, always-on server.

4. Integrate with what the business already uses

The business didn't need new software. The receptionist books into the Google Workspace calendar the team already checks. It pushes leads into the CRM through the same intake the company website's contact form uses. It sends summaries to the office email. Nobody had to learn a new dashboard, and the AI fits into the existing workflow instead of creating a parallel one.

5. Fail gracefully, never drop the caller

Every outside dependency can fail, so the design assumes it will:

• If the database is unavailable, the call continues and the failure is logged.
• If the calendar can't be reached, she still takes the appointment request and tells the caller the team will confirm.
• If nobody answers a transfer, the caller gets an appointment offer instead of a dead end.

The rule is simple: a back-office hiccup should never be the customer's problem.

Security and data ownership

Phone systems attract a lot of automated abuse, so the call gateway only accepts traffic from the phone carrier's published addresses. The system authenticates to Google services with short-lived credentials tied to its own service identity, rather than long-lived keys stored on disk. Call records, transcripts, and leads all live in the business's own Google Cloud project. The business owns its data, not a third-party answering service.

What it costs to run

For a small business, the economics are part of the design. The always-on server is the main fixed cost, roughly $50 to $100 a month depending on size. The AI conversation itself costs a few cents per minute, so a typical call costs well under a dime. Call records fit comfortably within the database's free tier, and the phone number adds a dollar or two a month. That's less than a traditional answering service, it answers instantly at any hour, and it keeps the sales pitches away from the owner.

It adapts to other businesses

Nothing here is specific to roofing except the conversation design. The services she listens for, the questions she asks, the transfer rules, the appointment windows, and where leads go are all configuration. The same architecture fits HVAC, plumbing, landscaping, pest control, medical and dental front desks, property management, or any business where the phone is the front door and missed calls are missed revenue.

How I approach projects like this

This project reflects how I like to build systems:

• End to end. From the phone carrier and real-time audio, through the AI and its guardrails, to the CRM and calendar on the other side. The value is in the whole flow working, not any one component.
• Constraint-driven architecture. Latency, reliability, and cost requirements decided the design, not trends or habit.
• AI with guardrails. Use the model for what it's genuinely good at (natural conversation) and keep anything with real-world consequences in code that behaves the same way every time.
• Built for the people who use it. The best measure of this system is that the office staff didn't have to change how they work. Leads, appointments, and summaries just show up.

If you run a business where missed calls cost you customers, or you're building a team that needs someone who can take an AI system from idea to production, I'd be glad to talk. Reach me at douglas@montanus.biz.