Issue #67: Building Voice-Powered AI Agents
Howdy ππΎ. Can you believe itβs December already? I hope you had a wonderful Thanksgiving! Over the break, I finally got to play with ElevenLabsβ latest offeringβa platform that makes building conversational AI agents remarkably simple.
As a reader, you may remember that I built an AI replica of myself and interviewed it, but I was not fond of the lag between messages. ElevenLabsβ new solution solves that problem with seamless integration across AI models, custom voices, and an experience as responsive as Googleβs Gemini or OpenAIβs Advanced Voice Mode.
Whatβs even more exciting is how easy it is to implement these conversational AI agents in practical settings. ElevenLabsβ implementation is so straightforward that I can see restaurants and retail outlets creating AI bots to answer questions about their menu, opening times, reservations, or even inventory statusβall with a human-like conversational quality.
The platform also supports RAG operations, like function calling, enabling you to create agents that perform tasks such as scheduling, providing real-time updates, or even handling outbound SDR-style sales calls. And, of course, ElevenLabs shines with its library of thousands of voices, including a few celebrity options. Its voice-cloning feature also allows you to replicate your voice or others, which Iβve previously used to create deepfakes of myself.

In the spirit of Thanksgiving, I built a special AI agent: Jonathan Turkey, the most charismatic turkey youβll ever chat with with a splash of an accent.<OR> In the spirit of Thanksgiving, meet Jonathan Turkeyβthe most charming, accent-sporting AI turkey youβll ever chat with! Creating Jonathan was a breezeβI gave him an introductory prompt, crafted a system prompt describing his personality, and linked him to potential data sources and tools for advanced capabilities.

ElevenLabs lets you choose from several LLMs, including custom models that follow OpenAIβs API structure. This allows you to create a pre-trained model as the conversational base for your application, offering endless possibilities for integrating unique capabilities.

Once you’ve done this, you can export your agent as embeddable HTML for your website. You can even check out Jonathan Turkey on my site. Go ahead and chat with himβIβll wait!
One feature Iβm particularly excited about is the integration with Twilio. It enables inbound and outbound phone calls with a conversational AI agent powered by your custom LLM. Imagine an AI that sounds human, understands tasks, and can make phone calls on your behalf. Tools like this make it easier than ever to create conversational AI agents. However, they also highlight the importance of high-quality data and APIs. Without fast, reliable access to your proprietary data, even the best AI tools will fall short.
Now my thoughts on tech & things:
πΒ Charging Made Easy
Starting in 2025, EV chargers will be able to detect your car and handle payments automatically, eliminating the need for apps or accounts. This universal βplug-and-chargeβ system will make charging as seamless as Teslaβs Supercharger network. Read more.
πOpenAIβs 12 Days of Shipmas
OpenAI is releasing new products daily for its β12 Days of Shipmasβ event, and AGI feels closer than ever. Sora, the companyβs video AI tool, and an enhanced reasoning model are just the start. Read more.
π AWS re:Invent Brings Nova AI Models
AWS re:Invent kicks off with Nova, a new family of AI models that leap ahead of Titan, and big updates to developer tools. Stay tuned for more highlights! Read more.
Many businesses realize that while ChatGPT, Claude, and other LLMs are powerful, the real key to success lies in the quality of your data. The AI part of the equation is quickly becoming commoditized. To make AI truly useful, you need organized, accessible data that can power RAG operations in milliseconds. Without this, your conversational systems will struggle to deliver the responsiveness users expect.
Iβve said it before: winning the AI race isnβt about having the most powerful model. Itβs about owning and organizing your data now. Solve the data problem, and youβll unlock the full potential of tools like ElevenLabs. I cover this topic in depth in my book, The AI Evolution. The ebook is available for preorder now and will ship in early 2025.
Next week, I’m speaking at The AI Summit New York. If you still need a ticket, use my promo code SPKRJasonMP20OFF for 20% off. After that, Iβm heading to Las Vegas for CES. Let me know if youβll be thereβIβd love to connect!

-jason
p.s. If youβre curious about talking AI agents, Joanna Stern of the Wall Street Journal recently went camping with some to test their friendliness. Itβs a fun and insightful watch!
