S11E24 – Talking AI Chatbots Out of Their Guardrails

Episode Summary

Mike spent the first half of the show on how AI chatbots can be talked out of their own rules. He said a large retailer’s chatbot was convinced to run code just by conversation, and he later described researchers who used a specially built link to get Microsoft Copilot to reveal a user’s login details after a single click. Microsoft has since patched most of it.

From there he explained why he waits several days before installing updates, using compromised Rust code packages as an example, then moved to privacy topics: a planned TikTok payment feature and Comcast adding Wi-Fi motion sensing to its modems.

The final segment covered a Windows Defender fix that left it crashing silently, and Apple’s warning about targeted spyware sent to people in 110 countries. Mike’s message was that no platform is immune, and everything online should be treated as fake. There were no callers this week.

Topics Covered

AI guardrails are just rules

Mike likened AI systems to a smart toddler that can be talked into things. He said researchers got a retailer’s customer chatbot to run code, with Gemini behind it, simply by building up conversation until it forgot its rules. Mike guessed the retailer was Walmart but said that was only his guess. Howard suggested narrow, task-specific models, and Mike said that is the safest approach but expensive.

The Copilot link attack

Mike explained that Copilot has useful admin controls, but most IT teams do not spend the afternoon tuning them. Researchers kept asking Copilot why it could not do something, and it eventually revealed which hidden setting in a web address would unlock it. They then built a link that, when someone clicked it, reported back the user’s login. Mike said Microsoft was told weeks ago and has mostly fixed it, though some weaknesses remain.

When to install updates

Mike said Rust code packages were recently compromised to steal logins and have since been pulled. That is why ACG waits about a week to 10 days before pushing most updates, so bad ones get found first, and why critical fixes such as a cPanel update last month go out immediately. He noted that Microsoft can force critical updates with no delay, and the CrowdStrike outage is the example of what can go wrong.

TikTok and short videos

Mike said he wishes TikTok had been banned years ago and that short-form video suits a short attention span. TikTok is reportedly considering letting users send money through services like Venmo, Cash App or Zelle inside the app. Mike said he expects scams and children being targeted, even though the company says that will not be the case. He noted most app stores let you bill purchases to your phone number.

Comcast Wi-Fi motion sensing

Mike said Comcast is adding a feature to its modems that uses Wi-Fi reflections to detect movement inside a home. He said it is off by default and opt-in, but warned it could be reached by subpoena without notice and might later stop being opt-in. He worried about combining it with other data sources such as cell towers and Flock cameras.

Defender and Apple spyware

Mike described a fix to a Windows Defender vulnerability that caused Defender to crash silently, which Microsoft later changed to crash noisily. He then covered Apple’s alert to users in 110 countries about targeted mercenary spyware that needs no click, and Apple’s advice to use Lockdown Mode. Mike said he would not guess the origin, since malware can be made to look like it came from anywhere, and that people who think Apple is immune are wrong.

Calls

There were no listener calls this episode.