Building Voice-Powered AI Agents — Jason Michael Perry

Howdy 👋🏾. Can you believe it’s December already? I hope you had a wonderful Thanksgiving! Over the break, I finally got to play with ElevenLabs’ latest offering—a platform that makes building conversational AI agents remarkably simple.

As a reader, you may remember that I built an AI replica of myself and interviewed it, but I was not fond of the lag between messages. ElevenLabs’ new solution solves that problem with seamless integration across AI models, custom voices, and an experience as responsive as Google’s Gemini or OpenAI’s Advanced Voice Mode.

What’s even more exciting is how easy it is to implement these conversational AI agents in practical settings. ElevenLabs’ implementation is so straightforward that I can see restaurants and retail outlets creating AI bots to answer questions about their menu, opening times, reservations, or even inventory status—all with a human-like conversational quality.

The platform also supports RAG operations, like function calling, enabling you to create agents that perform tasks such as scheduling, providing real-time updates, or even handling outbound SDR-style sales calls. And, of course, ElevenLabs shines with its library of thousands of voices, including a few celebrity options. Its voice-cloning feature also allows you to replicate your voice or others, which I’ve previously used to create deepfakes of myself.

In the spirit of Thanksgiving, I built a special AI agent: Jonathan Turkey, the most charismatic turkey you’ll ever chat with with a splash of an accent.<OR> In the spirit of Thanksgiving, meet Jonathan Turkey—the most charming, accent-sporting AI turkey you’ll ever chat with! Creating Jonathan was a breeze—I gave him an introductory prompt, crafted a system prompt describing his personality, and linked him to potential data sources and tools for advanced capabilities.

ElevenLabs lets you choose from several LLMs, including custom models that follow OpenAI’s API structure. This allows you to create a pre-trained model as the conversational base for your application, offering endless possibilities for integrating unique capabilities.


Once you’ve done this, you can export your agent as embeddable HTML for your website. You can even check out Jonathan Turkey on my site. Go ahead and chat with him—I’ll wait!

One feature I’m particularly excited about is the integration with Twilio. It enables inbound and outbound phone calls with a conversational AI agent powered by your custom LLM. Imagine an AI that sounds human, understands tasks, and can make phone calls on your behalf. Tools like this make it easier than ever to create conversational AI agents. However, they also highlight the importance of high-quality data and APIs. Without fast, reliable access to your proprietary data, even the best AI tools will fall short.

Now my thoughts on tech & things:

🚗 Charging Made Easy
Starting in 2025, EV chargers will be able to detect your car and handle payments automatically, eliminating the need for apps or accounts. This universal “plug-and-charge” system will make charging as seamless as Tesla’s Supercharger network. Read more.

🎄OpenAI’s 12 Days of Shipmas
OpenAI is releasing new products daily for its “12 Days of Shipmas” event, and AGI feels closer than ever. Sora, the company’s video AI tool, and an enhanced reasoning model are just the start. Read more.

🚀 AWS re:Invent Brings Nova AI Models
AWS re:Invent kicks off with Nova, a new family of AI models that leap ahead of Titan, and big updates to developer tools. Stay tuned for more highlights! Read more.


Many businesses realize that while ChatGPT, Claude, and other LLMs are powerful, the real key to success lies in the quality of your data. The AI part of the equation is quickly becoming commoditized. To make AI truly useful, you need organized, accessible data that can power RAG operations in milliseconds. Without this, your conversational systems will struggle to deliver the responsiveness users expect.

I’ve said it before: winning the AI race isn’t about having the most powerful model. It’s about owning and organizing your data now. Solve the data problem, and you’ll unlock the full potential of tools like ElevenLabs. I cover this topic in depth in my book, The AI Evolution. The ebook is available for preorder now and will ship in early 2025.

Next week, I’m speaking at The AI Summit New York. If you still need a ticket, use my promo code SPKRJasonMP20OFF for 20% off. After that, I’m heading to Las Vegas for CES. Let me know if you’ll be there—I’d love to connect!

-jason

p.s. If you’re curious about talking AI agents, Joanna Stern of the Wall Street Journal recently went camping with some to test their friendliness. It’s a fun and insightful watch!