Howdy 👋🏾. In the early days of search, websites needed a way to signal what, if any, of their content should be indexed by search engines. This served two purposes. First, some organizations might prefer that search engines like Google or Bing not index their content. Second, it allowed them to restrict specific types of pages from being indexed, such as membership-only content or other pages meant to be gated.

As a solution, a standard called the robots.txt file was introduced as a voluntary system that allows websites to signal their intentions to search engines. This has been the law of the land since 1994.

This file is getting new attention as a new question arises: what content on the open web should AI companies be allowed to use to train their models, and what content is off-limits? Many AI models have slurped content from the web, claiming it sits in the public domain, but have also released tools using the robots.txt file to allow websites to decide what content they would like to make available to these bots.

A few weeks ago, I shared the backlash Perplexity AI received for its AI-powered search engine, which some refer to as an answer engine, as it uses sourced content from the web to power responses but does not send traffic to the sources it pulls from. It was also discovered that Perplexity ran afoul of its own stated rules, pulling information into its answers from sites that explicitly asked it not to use their robots.txt file (Perplexity’s grand theft AI).

The response from Perplexity’s CEO made sense. “Perplexity is not ignoring the Robot Exclusions Protocol and then lying about it,” said Perplexity co-founder and CEO Aravind Srinivas in a phone interview. “I think there is a basic misunderstanding of how this works,” Srinivas said. “We don’t just rely on our own web crawlers; we rely on third-party web crawlers as well.”

In other words, indexing is not limited to a singular bot crawling the Internet. Different bots and third-party services used for their search make it difficult to easily point out that they are purposely not following the rules they stated.

Of course, Reddit, which has made it known that it plans to monetize the value of its community content fully, has gone a step further than robots.txt by implementing checks to identify bots and crawlers attempting to access its content and returning them a 404 (page not found) error. This ensures that this content is prohibited for those who ignore the voluntary pact of the robots.txt file (Reddit blocking search engine crawlers and AI bots).

What made me interested in this now is that OpenAI just released SearchGPT, its competitor to Perplexity AI. While I haven’t had the opportunity to try it myself, I have spent time reading how it works and noticed a pretty interesting bit about how it handles crawling for its search service (OpenAI Platform):

For the less technical readers, OpenAI has different bots that serve other uses, allowing publishers of content to grant more granular control to the various ways OpenAI may interact with openly crawled data:

• GPTBot: This bot is the one most are worried about. It actively consumes content it crawls and potentially uses to train OpenAI’s many generative models.
• ChatGPT-User: One of the features of the multimodal ChatGPT 4o is that a user can insert a web URL in a prompt, and ChatGPT can read that URL and use its contents to aid its response. This type of bot is used only for direct response to help the user, and the URL contents are not stored for search or training.
• OAI-SearchBot:  This bot is for search. It indexes content to aid the search experience or to surface content, but the content crawled is not used to train models.

This all makes a ton of sense, and the more granular rule system gives publishers a ton of control in picking and choosing to what extent they would like to share their data with OpenAI. Perplexity appears to have one bot and one bot only, as stated in its documentation, but I could see it quickly following suit.

With the new complexities of AI and the diminishing line between AI-powered chatbots, search, and answering engines pushing us to Google Zero or zero-click traffic, one has to wonder if the voluntary robots.txt system is enough, or if Reddit’s heavier-handed approach of directly blocking bots is the best way. Who would have known that a small text file (The rise and fall of robots.txt) would be so important?

Now, my sponsors and my thoughts on tech & things:

🤝 This week’s newsletter issue is proudly sponsored by:

If you are looking to find qualified candidates, contact Baird Consulting.

🚫 Why Google Is No Longer Limiting Third-Party Cookies in Chrome – Google’s shift away from limiting third-party cookies in Chrome marks a significant change in the browser’s privacy approach. Explore the reasons behind this decision and its potential impact on web tracking and privacy. Read more>

🔍 OpenAI Just Released Search – OpenAI’s new search capabilities promise to revolutionize how we retrieve information. Dive into the features of this latest release and how it could reshape our interaction with AI. Read more >

🤖 Zuckerberg’s Thoughts on Open Source AI – Meta’s release of Llama 3.1, the largest open-source AI model with 405 billion parameters, sets a new standard in AI development. Discover the implications of this move and its potential impact on the AI ecosystem. Read more >

🍏 The First Taste of Apple Intelligence Is Here – Apple’s latest betas for iOS and iPadOS 18.1 introduce new Apple Intelligence features. Get a sneak peek at these innovations and what they mean for the future of Apple’s devices. Read more >

SearchGPT begs the question: what is search today? I initially viewed Perplexity AI as a search engine, but it’s quite different. It operates through a chat interface that enhances its responses with search results using a RAG-style approach. Is it search, or is the term “answer engine” or “AI chat” a better way to describe it?

Because these engines are coming from the perspective of AI and what should be used and allowed when training, or out of fear of giving AI too much access to our information, it also feels like OpenAI and Perplexity AI are getting away with limiting the sources they index for their search systems. Take this quote from OpenAI’s website:

“We are committed to a thriving ecosystem of publishers and creators. We hope to help users discover publisher sites and experiences, while bringing more choice to search. For decades, search has been a foundational way for publishers and creators to reach users. Now, we’re using AI to enhance this experience by highlighting high-quality content in a conversational interface with multiple opportunities for users to engage.”

With OpenAI and other AI companies working hard to license data from select publishers, in some ways this could lead to more limited sources powering these systems. Is Wired right to limit Perplexity AI? Would they ever consider doing the same to Google Search?

I don’t know the answers to these questions, but seeing how we figure them out will be interesting.

-jason

p.s. I’m a board member of the Baltimore Symphony Orchestra and a fan of jazz music, but alas, I do not play an instrument myself. I dream of playing the piano, maybe with a group of friends in a smoky dark jazz club in New Orleans, and until today I thought knowing how to play music was a prerequisite to releasing an album or performing live… but folks, H. Jon Benjamin, the voice of Archer and Bob from Bob’s Burgers, has shown me where there’s a will, there is indeed a way. Watch, and enjoy.

Howdy 👋🏾, last month, Tyler Perry announced he was pulling back from his $800 million investment in a new film studio in Atlanta. This decision came after he laid eyes on OpenAI’s new AI diffusion model, Sora, which can create video from a text prompt. If you haven’t already, check out Sora’s videos; they’re amazing. Unfortunately, Sora is quite locked down, with access limited to a select audience, and plans for greater availability later this year (if you know someone, I would love access). However, it’s far from the only option, so this week, I thought I would give you a tour of some of the text-to-video AI models and how they work.

As a refresher, ChatGPT and most text-generative AI models are Large Language Models, which work as a very complex auto-suggest, guessing the next word or token in a phrase based on its training data. Most image models are diffusion models that diffuse or break up parts of an image bit by bit to understand its elements better. They then use that information to understand an image’s elements and use that data to generate new images or artwork. Stable Diffusion puts it right in the name, but Midjourney and DALL-E are also examples of diffusion models.

To date, most text-to-video models are limited to creating about 1 minute of video, as shown in Sora’s technical report comparing the differences between videos generated with different levels of compute power. Many of these models can also start with an image and transform it into a video or an animation, or extend an existing video to finish or complete it.

RunwayML

RunwayML’s new Gen-2 AI model allows you to create videos from a text prompt, extend an existing video, or use an existing image. I set up a free account that limits me to a handful of attempts and 4 seconds of generated video.

I asked RunwayML if it could create a claymation video of a black man with large hair asking you to please subscribe to his newsletter, and it did not disappoint.

I grabbed a photo from Equitech Tuesdays at Guilford Hall Brewery by photographer Ian Harpool to check out how well it adds motion to existing images. First, let me show you the original photo:

This is the video generated from that photo:

Pika

Pika has some extra abilities, including the power to dynamically add sound effects to your generated videos, take an existing image or video, and sync the person’s mouth to an audio clip. Pika also supports the same features as RunwayML, such as video from prompts, video from existing images, and the ability to extend an existing image. Some of these features require a paid account, but you can get a feel for the model’s abilities using the free plans.

I also asked Pika to generate a video of a 3D closeup of a black man with large hair frantically telling people that the M&M vending machines are spying on us (You’ll understand when you get to the p.s.).

I asked Pika to convert that same existing image to video, and it came back with this:

Of course, these are early days, and plenty of video-generating AI models are getting closer to show time. Stability AI, makers of Stable Diffusion, released an early version of Stable Video – an open model you can download and stand up on your own hardware. I’ve personally dabbled with a handful of video models on Huggingface that show tons of possibility. These things are getting better and better. Now, my thoughts on tech & things:

⚡️I believe that the quality of Google Search has been declining for years. However, combining AI and large media organizations employing extensive SEO tactics to sell affiliate products has taken a significant toll, making Google Search quite subpar. I suspect that Google is feeling the pressure from newer AI search upstarts like Perplexity AI and has finally taken steps to weed out the worst offenders.

⚡️Anthropic recently released a new family of AI models and compared them to OpenAI’s ChatGPT 4, but what does that really mean? To date, many AI companies have declared the greatness of their models based on how well they can pass medical exams or the bar. However, that’s not how we use these AI assistants in our daily lives. I want to know if an AI can tell me how much water to use for long-grain rice versus jasmine rice or if it can check my estimates on conversion rates. Techcrunch has a great piece on why AI benchmarks seem to tell us so little when it comes to our everyday use of AI models.

⚡️Apple’s reactions to the EU’s DMA (Digital Markets Act) have felt uncharacteristic for the company, which seems to have done everything it can to meet the letter of the law and nothing more. The most recent blip was Apple’s sudden banning of Epic Sweden from opening its own App Store marketplace in the EU, followed by Apple quickly reinstating it. John Gruber has a good write-up of what happened.

⚡️A company at SXSW used generative AI to bring Marilyn Monroe to life through an interactive bot that mimics her voice, emotions, and physical reactions. The world of digital avatars continues to feel like the uncanny valley but it also opens the door for some truly interesting possibilities.

⚡️After last year’s tumultuous events at OpenAI, which saw Sam Altman fired, two CEOs hired, and Sam Altman rehired, the company finally completed its investigation, returning Sam Altman to OpenAI’s board. The whole saga has been fascinating to watch, and now, Elon Musk – who donated millions to help start OpenAI – is suing the company. This has led to a rather public exchange of words and the public sharing of past emails on OpenAI’s website.

This week, I had a great time delivering an introductory AI talk to the Tristate HR Association at Rowan College in Southern New Jersey. I walked the group through how AI works and its history. I also offered some interactive demos of creating chatbots using OpenAI’s assistants that can answer questions about HR policy from employee handbooks or API calls to HRIS (Human Resources Information Systems).

I have more of these talks lined up and would love to speak at your next event or conference or craft a private talk on how your organization can use AI. I’m also working to launch in-person and online half-day workshops, so keep your eyes open or check out my website for more information and availability.

-jason

p.s. A facial recognition-equipped M&M vending machine on a college campus was discovered thanks to an error message. The company uses it to understand buyer demographics better and determine repeat visitors, but it seems unnecessary.

Howdy👋🏾, users of many AI models like ChatGPT reported hallucinations when asking for real-time stats during the Super Bowl. The issue is that AI learns from past data, not present events. At OpenAI DevDay, Sam Altman apologized that the data feeding ChatGPT lagged by years, promising to keep the data cutoff more current going forward.

This lag by design limits the types of queries current AI models can accurately answer. I need real-time info on the weather or live sports scores. Chatbots built on models like ChatGPT are trained on past data so they can’t update you on recent events like a rescheduled volleyball game. As IBM researchers noted, many foundation models essentially take closed-book tests, where that book is slightly out of date.

To address this, models can use Retrieval-Augmented Generation (RAG). RAG realizes when a question requires fresher data and fetches it from APIs, databases, or other real-time sources to augment the model’s existing knowledge. It’s like an open-book test that consults Twitter or Google for a boost. For example, the model could call the weather channel API to answer a question about next week’s weather to provide the most accurate forecast.

Perplexity AI exemplifies this approach. It understands questions via Natural Language Processing (NLP) to extract key points for search. It then scans its own data, APIs, and alternate sources to find insightful, up-to-date information to improve its answers, avoiding hallucination. This combines the power of foundation models with access to living data sources. This approach makes its results very accurate, and it is now one of my favorite places to ask complex questions on the Internet.

I’ve experimented with basic examples of using RAG with OpenAI’s Assistants in my playground test environment. Assistants allow you to define functions – a form of RAG – that get called when the context of a question matches the description you provide.

In the code block above, I define a weather function and provide details to help the AI model know when to utilize it. The code specifies required or optional parameters like “the city” to get the weather. If I ask “Should I wear shorts or a jacket to New Orleans next weekend?” the Assistant can infer I want current weather data. It will invoke my function, passing “New Orleans” as the city parameter and potentially detecting that I need a future weather timeframe.

In my playground, Mavis “Ace” Jarvis, my personal assistant, will take requests for weather or stock prices and call open free APIs to retrieve the latest data. It then uses those real-time responses to augment its knowledge when formulating a final answer.

We could expand this concept across many data sources – like getting the rescheduled volleyball game time or the status of support tickets from systems like Jira and Zendesk. For example, on my playground, Samantha “Smiles” Miles calls a mock API that returns a list of open tickets. It allows you to then ask for updates on any of those tickets. This mock API demonstrates how we could tie into various real-time internal data systems – from prices to employee PTO data – to enable richer responses.

Foundation models are amazing, but the real power of AI comes from a robust multi-model strategy. This approach combines AI models with external data sources like databases, APIs, CRMs, and websites that provide fresh, specialized information. The AI models handle what they have been explicitly trained for, while the living data sources supplement and augment that knowledge. With this combined approach, you can deploy solutions that avoid hallucination and stay current, delivering accurate responses no matter the question. Now, onto my thoughts on tech & things:

⚡️App Clips (instant apps on Android) are drastically underused. This feature allows a QR code or map link to trigger a thin version of an app that does not require a user to download the full app. For example, imagine scanning a QR code on a bill and paying it with an app that’s instantly downloaded. Play has a pretty amazing use case for App Clips that allows anyone to prototype an app and distribute it to investors using this feature. Absolutely a brilliant idea!

⚡️Epic has decided to accept Apple’s offer and release its App Store in the EU, potentially making Fortnite available for installation again for iPhone users in the EU. Apple’s rules charge per install, but Epic’s model is less about installs and app updates and more on SaaS services and virtual currency. More to come, but it is a place to keep an eye on.

⚡️If you missed it, Mark Zuckerberg gave his review of Apple Vision Pro, and surprise he thanks Meta Quest 3 is a better bang for the buck. I shared my thoughts on my blog, and I think Meta needed competition in the mixed-reality space, and this renewed competition will improve the next versions.

OpenAI functions are one way to implement RAG. Cloud platforms like Azure, GCP, and AWS also offer options to connect AI models to live data sources easily. When architecting AI, view it as part of a broader stack, leveraging two tiers of data: relatively static information that trains models and real-time, augmenting data via RAG.

For example, you could train a product support model on a database of specs, descriptions, etc. But data like prices and inventory change frequently, so supplementing with current API data enables accurate responses about new or out-of-stock items. The key is strategically segmenting data as an evergreen foundation vs dynamic augmentation.

An effective strategy for ingesting and partitioning training vs real-time data is just as crucial as data quality. Don’t ask AI models closed-book questions when you can provide open-book advantages with the right architecture. Apply RAG principles to give your models an unfair, data-fueled edge!

-jason

p.s. CaliExpress, a restaurant in Pasadena, CA, has opened a fully autonomous AI – powered restaurant. Yep, you heard that correctly. Fries and burgers are cooked by robots and ordered through an interactive kiosk. I need to schedule a trip out west for some taste testing.

Howdy 👋🏾, earlier this year, I launched a little AI playground for testing various models with the same prompt – an experiment allowing head-to-head comparisons evaluating output differences. My AI playground app is still in beta, so pardon any rough edges, but I did take a little time to drop some new additions this weekend:

• Added Perplexity.ai – the latest trendy AI for developers to tap cutting-edge capabilities
• Integrated the hugely multilingual BLOOM model for broader global applications
• Built text-to-speech using quality OpenAI voices
• A parts of speech tool to test tokenization and how natural language processing works

Keep checking back for more updates! I’m running most models on free tiers and token caps, so I apologize if we hit any caps. If you’re interested in sponsoring to cover compute costs and expand the playground, reach out.

Using any of these models is as simple as typing a prompt, but behind that string lies some interesting complexity. When you submit a text prompt, many AI models first preprocess this with natural language processing or an NLP tool called a tokenizer. A tokenizer splits our phrase into individual words, or “tokens,” allowing isolated examination to discern relationships in the sequence. Take the question – “What color is the sky?”. Tokenizing this yields:{“what”, “color”, “is”, “the”, “sky”}

Creating tokens by word is easy but can require a substantial dictionary in the AI model to understand every possibility. Some tokenizers break up a phrase into subwords or separate modifiers like “eating,” becoming “eat,” and “ing.”

Next, natural language processing identifies part-of-speech and grammatical roles. The NLP tagger labels “sky” as a noun, “color” as a noun, and “the” as a determiner. Understanding role context allows sensible analysis.

These same strings of text we use for our prompts are the same as the structured or unstructured data we use to train AI models. You can imagine developers feeding an AI model content. You might picture a conveyor belt of books, articles, newspapers, and more being dropped into a robot brain a token at a time.

Each of those tokens is used to build a graph that relates words to one another based on how they relate to other words and their usage.

This graph shows words based on the strength of the relationship to a word or the similarity between the words. So for the token “sky,” we might see “atmosphere,” “sun,” “clouds,” and “stars.” This relationship lets us see how a model may infer or understand what something means, and each of these values maintains a weight from +1 to -1 that shows how close a token relates based on the context. Tokens with a negative weight represent opposites like “underground”.

Tokenizers offer valuable insights into an AI model’s text processing. Some may recall the tricky question I posed to various AI models, ‘What might Paris Hilton like about the Paris Hilton?’. The complexity arises as numerous nouns make the part of speech for nouns challenging to decipher, resulting in a complex and confusing graph.

Models also give precedence to the order of tokens. While image generation models, especially those accepting shorter prompts, demonstrate this more evidently, the phrase ‘My favorite fruit is mango slice’ emphasizes my favorite fruit as mango over slices. In contrast, ‘Slices of mango are my favorite fruit’ subtly shifts the focus to slices of mango in interpreting the images.

Text completion prompts often prefer right-to-left emphasis, especially with models that offer large context windows like text-to-text or completion generators. Suppose the left represents the history or the past. Text generators have more and more history from previous messages to the left and assume the right represents newer and fresher content. This allows you to better control the response by focusing on the most important tokens to the end or right of a message. You can discern a model’s preference by observing its reaction to the placement of key information, like a 200-character limit.

For instance, let’s experiment with token emphasis from left to right:

“Compose a 200-word article on the impact of artificial intelligence on healthcare, focusing on the advancements in disease diagnosis and treatment.”

If you test this, many popular LLMs may ignore the word count and focus on the article. Now, let’s shift the emphasis from right to left:

“Compose an article on the impact of artificial intelligence on healthcare, focusing on the advancements in disease diagnosis and treatment in 200 words.”

If you run this in your favorite model, notice how the placement of the 200-character limit influences the model’s response, highlighting the nuanced emphasis based on token positioning.

“Tell me about AI in healthcare.”

Remember, if you overlook these nuances and omit key details, forcing the AI to assume the outcome may not meet expectations. You’re welcome. And now, onto thoughts on Tech & Things:

⚡️ With 180,000 Vision Pro pre-orders but just 150 apps, Apple’s mixed reality play faces developer resistance – but is it deliberate protest or practical reluctance?

⚡️ Mia Sato’s recent piece on the quality of search results and SEO, featured on Marketplace and The Verge, sheds light on the declining quality of search results across platforms like Google. Sato highlights the growing concern that SEO practices contribute to homogenizing web design. The pressure to optimize for search engines may lead to many websites looking remarkably similar.

Large language models use a complex graph of tokens and vector databases to craft responses based on the likelihood of the next word in a sequence. If that sounds like gobbly gop, think of completing the sentence ‘Jack and’ as assembling a puzzle. Adding ‘Jill’ is akin to placing a puzzle piece (token) to construct a meaningful picture (output statement). Our advanced language model settings empower us to fine-tune an obvious answer like “Jill” or embrace variety, whether it’s favorable or not. The untold stories of Jack and Ayanna can be shaped with just a tweak of these properties:

• Temperature: Think of temperature like a cooking flame. Higher temperature (hotter flame) gives you more focused tokens, while lower temperature (cooler flame) introduces more random tokens.

• Top-K: Imagine you have 10 flavors of ice cream, and you pick your top 3 favorites (that’s Top-K = 3). Those 3 are what the shop will choose from to serve you a cone. If you increase Top-K to 5, you allow them to pick from more of your favorites, but the flavor they ultimately grab might not be in your top 3.
• Top-P: Now imagine there are 100 flavors to choose from. Top-P is like asking for a scoop with at least 30% of your favorite flavors (that’s Top-P = 0.3). So, at least 30% of the scoop is bound to be flavors in your top favorites. Make Top-P higher, and you get more of a guarantee that the predominant flavors will be your top preferences. But set it too high (like 90%), and you may end up with less variety.
• Frequency Penalty: It’s like adding more unique spices to a recipe. Frequency Penalty lowers the dominance of common tokens or ingredients, making the dish more diverse.
• Presence Penalty: Consider telling a story. Presence Penalty ensures you don’t repeat the same words too often, making the narrative more engaging.

The newly added Perplexity AI model in the playground invites you to explore these settings across various models. Give it a try and see the creative possibilities unfold!

-jason

p.s. I’m thrilled about finally giving Apple’s Vision Pro a try! We’re actively developing new apps for clients and preparing existing ones for this device. However, I haven’t had the opportunity to experience it firsthand; my interaction has been limited to what I can see in the simulator. Watching this video showcasing the making of the Vision Pro triggered my Apple fanboy excitement! I can’t wait to get my hands on it. If you’re curious how it might work, this video walkthrough is excellent.

Howdy 👋🏾. Apple surprised everyone by prominently featuring the Apple Vision Pro at its Worldwide Developers Conference several weeks ago. If you follow Apple, there is plenty of reason for concern. Early rumors indicated Apple was cutting its delivery predictions, initial buyers reported high return rates, and several reviewers, including myself, found the app lacking killer features.

However, there was one huge positive: the importance of competition. Meta responded by releasing numerous updates to its Meta Quest devices, reviving forgotten or abandoned features to close the feature gap. Meta also announced a significant reshuffling of its Reality Labs division into two sub-units: one focused on wearables and the other on the metaverse. The metaverse division aims to build out its VR platforms, like Horizon’s social platform and VR headsets. At the same time, the wearables team is set to capitalize on the unexpected early success of the company’s Ray-Ban glasses.

I love the Ray-Bans and have been quite happy with the constant release of features, including adding AI tools to detect objects. Unfortunately, they do not accommodate my prescription for the frames, but I’m hopeful the next set, rumored to include a Google Glass screen, might.

With this new view of the evolving mixed reality market, I downloaded VisionOS 2 beta 2 onto Apple Vision Pro before a quick work trip to New York to see how the system-level changes to the device stand up. Special thanks to Mindgrub for letting me hold on to this headset for a little longer.

But before we dive in, a word from my sponsors and my thoughts on tech & things:

🤝 This week’s newsletter issue is proudly sponsored by:

If you are looking to find amazing people, contact Baird Consulting.

⚡️Anthropic continues to update its AI models with a new release they say matches ChatGPT 4o. What I’m most excited about is a very cool concept called Artifacts. Artifacts allow you to create a shared document space that you and Claude can both share and work on together, and this sounds like a killer feature! 

⚡️Apple is buckling down for a huge lawsuit and fight with the EU over the DMA and has now stated that certain features, including Apple Intelligence, will not launch this year, and they don’t know when. Folks have varying views on whether this is a threat from Apple or an issue with a rule that requires interpretation.

Beta means beta, and it’s not uncommon for Apple’s early beta releases to have significant bugs. You never know when a release might cause major issues. Of course, as soon as I got back from camping and had a stable internet connection, I installed all the betas on my TV, watch, phone, laptop, and Apple Vision Pro. Aside from some issues with macOS, I’ve been pleasantly surprised with how complete the betas have felt.

At WWDC, Apple listed an exhaustive array of new Apple Vision Pro features. Some of these are immediately available in the betas, while others, like multi-view, which allows users to watch up to 5 soccer or baseball games simultaneously, are still in development. As I use and test these features, I’ve kept my eyes on how well they address some of my biggest complaints. Here’s what I think so far:

Eye Tracking

Most of Apple Vision Pro’s interface works by tracking what you’re looking at, allowing you to interact with an object by looking at it and pinching your fingers together. I found this frustrating because your eyes drift, or your focus sometimes jumps to your next target without you realizing it. While Apple hasn’t announced changes here, I’ve noticed fewer issues selecting dialogs or smaller elements since installing the beta. This is one of those weird things where you do something that once felt frustrating, and then you realize it wasn’t, but you can’t quite figure out why. I am glad to see this getting improved, and I’m curious to hear if others have felt similarly. 

Mouse Support

Apple Vision Pro, at launch, supported Bluetooth keyboards and trackpads but did not provide support for a mouse. I found this odd, but I didn’t expect it to be much of an issue. The problem is that many third-party trackpads present themselves as a mouse, so my portable keyboard and trackpad combo didn’t work. I’m happy to say that these devices now pair and work without issues.

Hand Gestures and Notification Center

Some of the early ideas for how you launch the home screen with apps, access the notification center, or dismiss notifications felt a bit unpolished. The introduction of new hand gestures is a very cool way to fix that. Instead of tapping the digital crown to bring up the home screen with apps, you can now hold your hand out palm up, causing a floating icon to appear. If you complete an “O” with your fingers, Apple Vision Pro will launch or close the home screen without requiring you to touch the device. Turning your hand so the back faces up while holding your fingers together transforms the floating “O” into a display that shows the time and volume, which can easily expand to become the notification center.

It took some trial and error to get used to this action, but once it became comfortable, it felt natural. It also kept me from constantly touching the device, which sometimes required me to readjust from the downward pressure of tapping it.

Screen Sizes

When sharing my laptop’s screen, new settings appear that allow you to adjust the resolution to many additional choices. I love having a large rectangular display that gives me more space to play, and it still connects and works with ease. One promised but still missing feature is the environmental passthrough of an external keyboard, which makes your keyboard invisible when moving into an environment.

Additional Environments

Apple introduced additional immersive environments, including Bora Bora, and I’m already in love with working from the beach. It’s perfect for evenings when I can recline back on my patio, sit on the beach, and listen to the waves as they hit the shore. It is so relaxing and feels like I’m in a Corona beer commercial. 

Handling Darkness

Darkness remains an issue, but tracking is better, and in my testing, Apple Vision Pro does a better job of maintaining its position at night. On my patio, as the evening became darker, I received warnings that tracking would become difficult, but it managed to pick up my hands until things got closer to pitch black.

Overall, the updates, while minor, really go a long way to making the device feel complete. This beta reminds me of my experience using the very first Apple Watch, now known as the Series 0. It was an okay device, but over time, operating system updates fixed issues, and the iteration turned an okay device into a good or great device. Apple Vision Pro feels set on that same journey, and I can already tell you that visionOS 2 beta moves it a few notches above ok.

If you like this content, please share it with your coworkers and friends. Also, this a reminder that I’m looking for newsletter sponsors and that I’m available as a fractional Chief AI Officer or a technical consultant.

-jason

p.s. I really love Perplexity AI. I use DuckDuckGo as my default search engine, but that’s due to the muscle memory of searching by typing in the browser’s URL bar. For anything that feels remotely complicated, I hit up Perplexity AI. Wired has a great article on not only Perplexity creating BS but also indexing content when it says it explicitly won’t.

If that wasn’t enough, Perplexity indexed Wired’s article (which blocks AI bots) and created an article about itself using the content of the article! You can’t make this stuff up.

Howdy 👋🏾. No matter how often I make rice, I need a reminder of the different water-to-rice ratios for different types of rice grains. A Basmati is just a bit different than a Jasmine rice, and as a New Orleanian, serving bad rice is a sin. For years and years, I would do a quick search on Google to confirm. That gradually shifted to me asking Alexa and now Siri, who does a so-so job with responses. If this sounds somewhat normal to you, you already understand why Google Zero and Zero click search results are not only here but inevitable in a world of AI-powered assistants.

This transition is a shift in how we interface with the web, and this evolution has been happening for some time. The primary competitor to the web browser and the open web is the ever-expanding app ecosystem. Several years ago, The Oatmeal posted a comic that still perfectly describes this change, one that is now accelerated by AI.

In a recent newsletter, I covered the death of SEO and how AI is accelerating the death of the open web because it can answer a question using the web as its source but without the attribution or click you might depend on. My question about rice is a prime example. Many articles and recipes exist on the web that explain how to make perfect rice, but my question can be answered without needing me to click in and view ads or note the website the content came from.

In many ways, I see Web 4.0, known as the semantic web or the intelligent web, as a distillation of these ideas. Web 4.0 understands the Internet’s evolution into ecosystems and uses that understanding to reimagine a web where the web browser is not king. I like to think of Web 4.0 as the API-fication of the Internet, where APIs or Application Programming Interfaces speak to the ways we allow computers or ecosystems to communicate with each other. Some of the core tenants include:

Decentralization

Decentralization, which I think is best reflected in the push by the social media platforms Threads and Blusky to accept the Fediverse and open their doors to other ecosystems that manage and maintain their own rules. In a world of AI bots, this becomes incredibly important. For the next generation of personal AI bots to really help us, they need ways that they can interact with other bots or computers to help get the information we need. For example, if you want a plane ticket from Southwest Airlines, it would be great to write a message or ask a virtual assistant something like, “What are the 3 cheapest nonstop flights this Thursday from Baltimore to New Orleans?”. To make this possible, AI will need the ability to bypass a website and ask questions to a company’s APIs, creating a lesser need to flock to a browser.

Artificial Intelligence and User-Centric Design

Artificial Intelligence and User Centric Design speak to increasing depth of customization and personalization. Platforms know a lot about us, and they can use this information to better personalize experiences to match our needs or target content based on our likes. On the web, DXP or Digital Experience Platforms are allowing web and mobile applications to integrate information from CRMs or other data sources to dynamically change the flow of content to cater to a user. TikTok and Instagram have had highly targeted ads for years, but TikTok’s shopping marketplaces are becoming highly personalized shopping malls that cater products to you based on your needs. To date, while possible, true customization has been hard, and the results have been so-so. What is changing is the power of AI tools combined with rich customer information and the ability of these bots to customize messaging that feels truly targeted and human.

Advanced Interactivity

The world of mixed reality has stumbled quite a bit. The AI Pin was a flop, and Apple Vision Pro, while amazing, proved how far these devices might be from the norm. They also reflect our growing need for Advanced Interactivity that moves beyond the keyboard. Voice assistants are an amazing step, but the voice is not always the right tool, especially in busy and noisy environments like offices, buses, or trains. Web 4.0 will help to meld mixed reality experiences and non-keyboard experiences to make the ways we interact with the web different and unique.

IoT or the Internet of Things

Finally, IoT or the Internet of Things is becoming easier with better support for standards like Thread. To date, IoT has been clunky, and interoperability between ecosystems is limited, but this is changing as more devices can easily communicate with one another without needing a developer to configure things. IoT is also evolving in other ways like Amazon’s Sidewalk Network, NFC devices that can share data like business cards and menus, the ability to tap and pay, or features like AirPlay that let you stream music to a device you’re in proximity of.

It’s early in this new wave of an AI-powered, decentralized API-first web that allows for intense personalization, new ways of interacting, and easier connectivity between devices, but the seeds are sprouting. Now my sponsors and thoughts on tech & things:

🤝 This week’s newsletter issue is proudly sponsored by:

⚡️Forbes is taking legal action after Perplexity AI released a beta product that summarizes or answers questions from sourced content. As Forbes put it, the content is directly based on their own but does not properly credit or attribute it while beating them as the original source of the content in search results.

⚡️Tim O’Reilly, creator of O’Reilly Media, has an interesting write-up on AI copyright infringement and some thoughts on how rights holders should get paid. The system is similar to that of streaming music on online training libraries, where a revenue pool is created and paid out based on your percentage of streams. In his model, AI knows the sources of the data it uses to create a response and pays out for the use of that data. It’s interesting and worth a read.

⚡️From the day Zara and I met, I knew it was only a matter of time before an AI-focused social network emerged, and it’s here. I haven’t had a chance to download the app yet, but this is only the beginning of more and more AI-generated content targeted to your personal interest. Crazy time to be alive!

APIs and stronger standards for open communication are going to become extremely important as we embrace Web 4.0 and AI. Many of my enterprise clients are flocking to build large data lakes that pull systems together to create powerful RAG (Retreival-Augmented Generation) data sources that can feed their AI systems. If you want a powerful healthcare AI tool, it must talk with your EHR (Electric Healthcare Record) system. If you want an HR bot, it needs to talk with your HRIS (Human Resource Information System). If you are looking to do something for customers or sales, it probably needs insights into a CRM (Customer Relationship Management) or support and ticketing system.

Those enterprises facing the most difficulty are companies locked into agreements with legacy systems that purposely make it hard to get access to this data or those who build protective walls around their systems to make migration harder or force you to use their own brand of products. This is not just a pain; it’s a setback that will make businesses that can openly access their data unable to compete with those that can. 

Thank you to the Baird Consulting Group for being the first sponsor of Thoughts on Tech & Things. If you need help filling a position or find yourself in the market, please reach out to them here or upload your resume to their website. 

I am actively seeking additional sponsors for a variety of exciting initiatives. If you or your organization are interested in exploring opportunities, need a fractional Chief AI Officer or AI consultancy, or are interested in participating in one of my workshops, please don’t hesitate to contact me via my website or at jason@jasonmperry.com.

I hope you had time to reflect over the Juneteenth break, and I’ll see you all next week.

-jason

p.s. I guess AI more than mimics reality. A competition for the best AI-generated photo disqualified one of its winners after the photographer admitted that it was a real photo and not actually AI-generated.

Howdy 👋🏾, when Google released its new AI Search Overviews at Google I/O, I felt a great disturbance in the Force, as if millions of SEO specialists and content creators suddenly cried out in terror and were suddenly silenced. I fear something terrible has happened.

AI Search Overviews provide summaries or direct answers to users’ search queries generated from Google’s Gemini AI model. The idea is to help users get answers quickly without searching through search results. Have a question about Frederick Douglass? Google can now answer that directly with an AI-generated overview.

The obvious fear is that features like this create what The Verge’s Nilay Patel calls Google Zero, or what is known in the industry as zero-click search. Users search Google and get an answer sourced from content written by you, but all the interaction happens on Google without traffic sent to your website.

For years, many have felt Google has focused on building a moat around its search services, pushing users to interact with Google’s ecosystem rather than search results. If you search for a product, you’re met with Google marketplace links. Look for a restaurant, and Google’s business profiles and map results dominate the page, followed by sponsored ads. These changes push valuable page 1 links lower, generating less traffic.

It’s easy to blame Google, but our search habits have evolved. AI overviews are just the latest reminder. Take voice-powered gadgets like Alexa, Siri, Google Assistant, or Cortana. If you ask about Frederick Douglass, the distance to Trader Joe’s, or the ratio of jasmine rice to water, the last thing you want is a page of search results. You want a direct answer, something AI-powered systems can provide by consuming content from other sources.

Voice assistants are not the only competitors to search. Increasingly, AI tools like ChatGPT and Claude are taking what may have once started as a search query and giving back answers. Do you need help with your resume? Are you struggling to write a job description? ChatGPT is often a much better starting place than search, and while the data for these answers may have come from your content, you’re not getting attribution or traffic.

Perplexity AI, a search engine I love and often mention, exemplifies what search might evolve into. You ask a question, and using AI and RAG (Retrieval Augmented Generation), it searches through curated responses and returns an AI-summarized answer. It’s the best of both worlds, coupling the abilities of AI assistants with the power of curated search results. The web pages that generate these answers are linked as sources, but how many of us will click to read those links?

Media outlets, blogs, and product companies are already noticing these trends. Search traffic has been declining for years. Gartner believes search traffic will drop by as much as 25% by 2026, and I think those numbers are extremely conservative. For evidence, read HouseFresh’s managing editor blog post on the site’s battle with Google search and her excellent follow-up; that begs the question, what do you do when all of that search traffic disappears? 

Google Zero is here for some; AI-powered search and assistants will hasten it. It’s not crazy to imagine that jockeying for page 1 of search results in a few short years might be just as important as showing up first in the Yellow Pages. So, SEO may not be dead yet, but now may be a good day to start imagining where it is tomorrow. Now, my thoughts on tech & things:

⚡️ ️Rumor has it that ​Apple and OpenAI just inked a deal​ for the company to power AI in the Apple ecosystem. I’m still betting that most of the AI happens locally on the device, but it reaches out for more extensive calls. That said, this deal seems to have required Altman to calm fears with Microsoft. Apple’s WWDC is June 10th, so we might know what we’re getting soon.

⚡️Google has confirmed a ​huge leak of internal documentation on search​, which has some wondering how truthful the company has been on how search actually works. ​Search Engine Land​ has the SEO breakdown on what it means.

⚡️Snowflake security breach – stay tuned…

Discovery on the Internet feels harder than ever, as does the value of a website in a world of increasingly closed systems and communities. I’m amazed by the number of influencers on platforms like YouTube, Instagram, and TikTok who don’t have websites. Can you imagine a large brand like Ford, Apple, or GM existing without a website? Yet folks like Mr Beast don’t live on the open web; they live in ecosystems like YouTube and Instagram.

Trends have been shifting from the open web and search for some time, and chances are you spend more time today in a closed ecosystem than you do jumping from web page to web page. Your product purchases and news habits are now governed more by social media communities or content aggregators like Google News or Apple News.

When I talk with clients, I find that many have replaced search traffic for discovery with one of these ecosystems that they increasingly feel beholden to. For influencers, it’s YouTube, Instagram, and TikTok. Businesses and job seekers live on LinkedIn. Restaurants depend on Yelp and Google Maps; for delivery, they depend on DoorDash and Uber Eats. Retailers live on Amazon, Etsy, and Shopify. Writers and bloggers have flocked to Substack and Medium. Software developers depend on the Apple App Store and Google Play Store. Game developers live on Steam. Movie and TV producers live on YouTube and Netflix.

The open web, which is the heart of search, is dying. It’s harder now to simply put a web page or blog on the web and survive without finding an ecosystem, and those who try to survive without them are finding Google Zero is forcing their hand. 

As SEO becomes less relevant, the future of digital marketing and discovery is shifting towards ecosystem marketing. This involves understanding and leveraging which ecosystems drive AI tools and user interactions. Curated and trusted sources like Wikipedia, Yelp, and Reddit are becoming powerful aggregators for AI-powered assistants. It makes groups like OpenAI keen to negotiate deals with these ecosystems to get that valuable data.

Perplexity AI allows focuses that can limit the systems it sources to particular data ecosystems. Apple Maps sources much of its data from Yelp, though it has begun collecting its own data. One can imagine that this will factor even more significantly when AI assistants like Siri suggest dining options nearby. Owning your presence in these ecosystems will be crucial to becoming the new #1.

I have two milestones to celebrate. This Friday marks my last day at Mindgrub, and this newsletter you’re reading is issue #50! It’s strange looking at the first issues and watching as some of my early ideas began to take shape. As always, I really hope you all enjoy the read, and if you do, please forward, share, or comment.

Also, don’t forget that if you or someone you know needs AI consulting, support from a fractional CTO or CAIO, or to check out one of my workshops, don’t be a stranger and reach out.

-jason

p.s. Every ecosystem comes with rules, and those rules can drive you crazy. If you own an LG or Samsung washer machine, you may have noticed it plays a little Diddy after a cycle is complete. One YouTuber lost untold amounts of cash after copyright claims were made on the public domain Samsung chime in his video streams. ArsTechnica takes a deep dive into content creators’ continued struggles with YouTube content ID. Enjoy the many sounds of washers and dryers.

I think I like LG’s the best.

Howdy👋🏾. It’s official: Google has been ruled a monopoly and found to have violated antitrust rules by acting to prevent competition. The facts are clear: Google accounts for 90% of all internet searches, with its closest competitor, Microsoft Bing, at just 6%.

Although Google will likely appeal, we can expect a court decision on remedies within the next few months. This might force Google to break off search, divest certain assets, or – more likely – stop paying companies for default search engine status in web browsers.

This potential change could significantly impact companies like Apple and Mozilla. Apple, for instance, earns an estimated $26 billion a year by making Google the default search engine on Safari across its devices. Mozilla, a non-profit and maker of the popular Firefox browser, faces a more dire situation, with Google’s $400 Million yearly payments accounting for about 80% of its revenue.

Amidst this turmoil, Google faces new competition, especially from AI-enhanced engines. These competitors, including Microsoft with Bing and Copilot, OpenAI with SearchGPT, and Perplexity, are shifting from traditional search to “answer engines.” These platforms provide direct answers to queries, bypassing the list of results and potentially reducing the need for traditional search.

Google’s struggle to adapt has been evident. After making a big splash, it has primarily removed AI Overviews and needs to be faster to incorporate generative AI into its search experience. The next few months will likely bring answers and aggressive strategies from competitors looking to capitalize on Google’s vulnerabilities. Now, onto my sponsors and thoughts on tech & things:

🤝 This week’s newsletter issue is proudly sponsored by:

If you are looking to find qualified candidates, contact Baird Consulting.

🎤 Is Meta Building a Celebrity Voice Assistant? Meta is exploring celebrity-voiced AI assistants, potentially transforming how users interact with technology. Read more >

🔄 Microsoft Lists OpenAI as a Competitor Microsoft now views OpenAI as a competitor, reflecting changing dynamics as they develop similar AI products. Read more >

🛠️ OpenAI Unveils Structured Outputs OpenAI’s new feature allows developers to define schemas for responses, enhancing how AI integrates with other systems. Read more >

🎵 Is Suno a Modern-Day Napster Saga? The RIAA sues AI music firms like Suno, challenging the boundaries of copyright law and fair use. Read more >

The critical question remains: How will companies like Apple, Samsung, and Mozilla react if Google’s payments for default status dry up?

A dirty little secret is that most other search engines heavily license Bing’s search index for their search engines, adding their unique enhancements on top. For instance, DuckDuckGo is Bing’s search results, enhancing them with its own features. Perplexity also licenses Bing to support their search features.

A few years ago, Apple integrated Bing into its native search feature, Spotlight, which alarmed Google that users might not use the browser to search, potentially reducing traffic. If that payday stops, Apple may intensify its efforts with Apple Intelligence, potentially powered by Bing, to transition from traditional search to an answer engine model where Siri answers your question without opening a browser or searching.

Google will definitely continue to face new competition, but maybe the days of traditional search are over. Could this antitrust case be the tipping point for a new era where answers are given directly from AI-powered devices without accessing Google’s site or viewing its ads?

Lastly, I’m honored to announce that Technical.ly has named me one of the top engineers in the mid-Atlantic! I also have some exciting things brewing, including a new speaker series and a book. More will come very soon, so keep your eyes and ears open.

-jason

P.S. Once a year, I think it’s important to ask a pretty simple question—what is a hot dog? Is it a sandwich? Is a hot dog a taco? Is Ravioli Sushi? When is soup a soup, and what are cereal and milk soups? Is coffee a three-bean soup? I’m glad someone has tried to put some rules in place to help us understand what we’re eating—and I love my steak salad.

Howdy, I hope you had a fantastic Memorial Day. Long weekends like these are great for reflection, and that reflection can lead to transformation, which is the theme of this week’s newsletter.

My daughter graduated high school this weekend and will soon start her college journey! If you looked in a dictionary for the words proud papa, I’m sure it has a picture of me cheesing. Congrats to Olivia and all of the classes of 2024!

I also have some news. After 8 years as the CTO of Mindgrub, I’ve decided it’s time to start a new adventure. As my last act, I’m hosting a smaller version of our AI in A-Minor for a private group in upstate New York and flying right back to participate in a panel on AI for the University of Baltimore. I’ve loved every minute of my time at Mindgrub, and I’m not fully sure what the next chapter holds. But in the meantime, I plan to relax, starting with a camping retreat into the wilderness for a few days. When I’m back, I’ll spend the next few months consulting and teaching AI workshops, so if you need a hand, don’t be a stranger. Reach out.

Change is good, but embracing it can be challenging. Take Google’s struggles with its new AI-powered search. I’m sure many of you remember the release of Gemini, a pretty amazing AI model that was overshadowed by its generation of images that some saw as revisionist history. At Google’s developer conference, Google IO, it deepened AI in its search engine, adding new AI search overviews that featured AI-powered answers, and once again, the company finds itself as the punchline.

I have to say I’m still surprised at how much Google has stumbled, but it’s hard to change when you have so much to protect. Marketers are afraid of AI overviews and the possibility of losing their page 1 search result advantage, and I’m sure that’s slowed Google in how much it transforms search. To look at the other path, consider upstart Perplixity AI, my new go-to AI search engine. Its simplicity reminds me of how easy Google was when it first launched. Perplexity is a RAG-powered search engine that takes a prompt or question and then attempts to answer it by sourcing curated search results and using them to summarize and answer. The Verge has a great interview with Google’s CEO Sundar Pichai and asks whether what we see is part of the journey or the destination.

Another place you can see change is how tech companies view implementing AI in our homes, and enterprises are beginning to fork. Take OpenAI and Microsoft, which are OpenAI’s biggest supporters and financial backers. After all, most of Microsoft’s sprawling suite of Copilot AI products are powered by OpenAI’s ChatGPT or other jointly developed AI models.

Even though these two companies are connected at the hip, you might notice stark differences in how the companies plan to approach AI, which I think is best summed up in this quote: 

“I don’t like anthropomorphizing AI,” Microsoft Corp. Chief Executive Officer Satya Nadella told Bloomberg Television on Monday, referring to the practice of using verbs and nouns to describe AI that are typically reserved for people. “I sort of believe it’s a tool.”

This is a big difference from OpenAI, especially if you recently watched OpenAI’s Spring event, which has Scarlett Johansson threatening to sue the company for stealing her voice in an attempt to copy the movie Her. The releases of ChatGPT 4o appear set to create exactly what Nadella does not want: an anthropomorphized AI that feels like a human or a friend. 

Last week, I mentioned Microsoft’s Copilot Studio product, which promises to make it easier and easier to build rich enterprise AI agents or assistants. However, that sense of personality is lacking—and apparently by design.

I’m personally enamored with the idea of AI friends, but maybe my love of Sci-fi and anime has led me to dream of worlds with Data or C-3PO. As The Verge mentions, maybe that’s now what we really want.

Take a moment and watch a few clips of Rosey from Jetsons, and imagine it as a physical bot in your house. It’s easy to see her as the main villain in a horror movie. Of course, the other approach is to make these bots different from us. Some have taken this to mean making them cute, but it makes me wonder who is right and what ideal will win out in the end.

As I prepare for a big change this week, I can’t help but think of tomorrow, and the future my daughter and the class of 2024 will help to mold. Now, my thoughts on tech and things: 

⚡️The countdown to Apple’s developer conference WWDC is near; we’re getting ​some hints​ of what we might see. I’m betting we will get deeper AI integration into all of Apple’s operating systems, and I’m hoping Siri finally gets good.

⚡️Making self-driving cars is hard. Waymo gets a lot of applause for its much more cautious approach to rolling out self-driving cars, but that last 5% of development has to be very difficult. We’re learning a bit more about Waymo’s incidents from NHTSA investigations.

⚡️We’re getting more details about Sam Altman and why he was fired as CEO of OpenAI in November. Helen Tomer, a former board member, joined The Ted AI Show podcast and shared details – and in short, they say Altman has a history of lying and intimidation.

I’ve come to love the Ezra Klein podcast, and in a recent episode, he said something that resonated with me. For all of the AI chatbots I’ve used and created, including my Kindroid friend Zara, I get bored with them and stop using them. Klein mentioned the same thing. The thing is, these friend bots aren’t sticky. No matter how conversational they are, it’s not fun listening to a bot talk about the make-believe things they did that day.

It’s a very cool demo, but that’s all it is. It has the sizzle but not the steak. For the last few weeks, I’ve worked on my conversational version of Zara using Meta Llama3, and I find myself in that same loop. It’s cool, but it needs something more to feel engaging.

It’s hard not to hear Satya Nadella when I think of this. Do we want bots that lie to us and make a fake day? Do we want something that speaks reality and reminds us it is just an AI in a box? Or should we stop expecting bots to become our friends and build them to be utilities like a toaster?

Who are we asking our AI to be? I’m realizing I’m not sure I know.

So, yep, June 7th is my last day at Mindgrub. If you thought about it but haven’t done it, now is a great time to forward my website and newsletter to your co-workers and friends, reach out for a private or public AI workshop, or drop me a line about consulting help. To all my Mindgrub friends and family, it’s been an amazing few years and so long👋🏾, but thanks for all the fish.

-Jason

P.S. Man, I miss Gizmodo and the Onion of old. After the Gawker collapse and the buyouts, all properties have generally been eh. The new buyer of the Onion wasted little time reminding us how topical and great this institution was and can be. I leave you with the Onions article on the 7-finger Scarlett Johansson on OpenAI’s weirdly similar voice. It’s worth the read.