AKA Microsoft “Recall” or “AI Explorer”
If you’re a privacy enthusiast, or just a generally conscious user of the internet, you’ve probably seen headlines recently about Microsoft’s plans to leverage AI to basically keep a record of everything you do on your PC. Essentially what it will do, if/when enabled, is take regular screenshots/snapshots of everything you do every few seconds, right down to helping you find individual slides in a presentation or transcribing meetings. At first glance, this sounds like a privacy nightmare, so let’s break down how I think it could be both good, and bad, and what the most likely outcome of it will be.
The Good
Let’s start with the good. If the data is properly secured to prevent unauthorized access, this could be a really useful tool, especially for folks who stay busy and have a lot to keep up with. It would be nice to be able to ask Copilot something like, “Who did I talk to in the meeting about topic X last month” and have it bring up a written transcript of the meeting and perhaps some screenshots of the video call. The data is supposedly stored completely locally, so it may not be particularly useful for referencing work you’ve done on other machines, though I suppose since the data is encrypted for your Microsoft account, it is possible that they could store encrypted metadata and if you ask a query about something not on your device, perhaps it could answer with something like, “It looks like you spoke to person X on InsertDateHere. The slides you worked on together are stored on Computer1.” Perhaps if the file is synced to your OneDrive it could still access the file for you on devices other than the one where you originally worked on it. Keeping this secure between multiple interconnected devices is a whole other can of worms, and as of right now results are only valid when you’re searching on the same machine work was done on. I’m just brain storming about how things could work in the long term.
That, however, is pretty much the end of the the “good”. If properly implemented, it could really help people stay productive and help improve Windows search. However, I have plenty of concerns.
The Bad
First off, let’s not forget that Microsoft tried something kind of like this years ago with the Windows 10 timeline feature. The point then was that you could be working on something on your Windows PC, and you could pick that up anywhere else that you were signed into your Microsoft account whether that be other PCs, phones, etc. They even had a digital assistant, Cortana, that for a time at least, was quite good. I remember for a few years I tried to go full Microsoft with my technology. I used Windows 10 on my PCs, I used Cortana quite a lot and I even carried a Windows Phone for several years, and enjoyed it. It was legitimately useful and I even bragged to others how great it was. I had a Lumia 640 for a while and eventually upgraded to an Alcatel Idol 4S, both Windows phones. I had a desktop microphone hooked to my PC and remember just how cool it was that I could speak up to my desktop PC and say, “Hey Cortana, when I’m near the Dollar General, remind me to stop and pick up item X”, and my Windows Phone would actually do it; in real time it would pop up when I drove by the dollar store and tell me to grab that item. For a brief moment in history it was amazing. But, like anything that isn’t Office, XBox or Windows, Microsoft abandoned all of it. They killed the Windows 10 timeline feature, they killed Cortana and they abandoned Windows Phone. Later in its useful life my Windows 10 phone seemed like it became more unstable and more unusable with every system update, to the point that I could barely make regular phone calls without the device just randomly overheating and freezing up. Microsoft even abandoned all their first party apps on Windows phone before they stopped updating Windows phone itself. Microsoft has a history of experimenting on its users with half baked plans that it’s not entirely committed to, axing it at the first sign of financial trouble and abandoning the users who actually gave their tech a chance. Even IF I was still a Windows user, and IF I trusted them to safeguard the information stored by Recall, I still don’t think I could bring myself to trust them to support the product in the long term. I couldn’t bring myself to use and become dependent on a new product of theirs; I would be constantly watching for the announcement of its discontinuation. Just look at what they’re doing to XBox right now. They went out and spent a boatload of money on a bunch of game studios they were ill prepared to manage, and they just recently shut down 4 of those studios. There’s even a website called “Killed by Microsoft” that documents their history of excess and mismanagement.
My next concern is how Microsoft will roll this product out. I’m not morally opposed to this sort of session recording and note taking in and of itself, if the user is well informed and consents to the recording. There has been been a third party application you could install on Mac OS to accomplish the same thing for quite a while now. The difference however is that if you use that app, you’re very obviously consenting to the behavior because you had to seek out that application and set it up. With this feature being a core part of the operating system, I suspect a few things to be true about the rollout. These points are pure speculation, and I stand to be corrected when the final version releases, but this is how I “think” a few things will probably work.
- It will probably be “opt-out” instead of “opt-in”. In other words, it’ll be on by default and you’ll have to choose to disable it. Microsoft has a history of this and other dark patterns when it comes to features they roll out. They turn it on by default and even if they give you a clear option to disable it when the feature is added, the option to do so is often subdued or made less prominent than the option to leave it enabled. They know that most users will take the path of least resistance to get back to work, so turning it on by default and subduing the option to disable it will ensure maximum user participation in the new feature. Notice how in the screenshot below, during the normal installation process for Windows 11, the default, highlighted option to “Continue” is actually an upsell that begins the process of signing you up for an upgraded, paid subscription to Microsoft 365. This happens during the install and setup process for an operating system that you already paid for.
- Even if you choose to opt-out, they’ll probably regularly “reset” that option when there’s a Windows update. So users who opt-out will have to babysit that setting to make sure it doesn’t turn itself back on without their permission. If there’s an option to uninstall it, it will get magically reinstalled from time to time as part of your regular Windows updates.
- While the “contents” of your files may stay local and encrypted, metadata from those files could be collected and analyzed. The screenshots may be analyzed for questionable material. There will probably be some mechanism for Microsoft to collect something they can use to target ads and monetize the feature. This is the company that puts ads in the start menu of a paid operating system, who automatically installs “recommended” third party apps in the background without your permission. This is the same company that has spent billions investing in and partnering with OpenAI. They’re not just going to give this feature away. They will benefit from this in some way, and since its entire purpose seems to be bulk data collection, that will probably be where the money comes from. They might not share your actual documents or photos with third parties, but that’s not to say they won’t leverage your NPU cores and the local AI engine to analyze them, figure out what’s in them, and then use that data to target ads, report you to authorities, etc.
Potential Ramifications
Because of this, I suspect that it’ll only be a matter of time before this feature is abused by a bad actor. Some government agency or big corporation will get breached because somebody will forget that Recall was enabled, and probably thru some kind of malware running under the user’s account (and therefore with access to the encryption keys) will have screenshots and a transcript of the meeting leaked. Microsoft can say the words “private” and “encrypted” all they want, but in order for data to be stored and recalled, the encryption keys have to be stored somewhere during runtime. Take my personal home server. The storage drives in it are encrypted. I could take one out, hand it to you and make a pretty solid bet you’d never get anything useful off it. But, while the server is booted and running, the drives are “unlocked” and mounted as if they were a regular folder. They have to be or the other software wouldn’t be able to access the files on them. So even though the drives are “encrypted”, that doesn’t stop malware running on the server from accessing the data on them.
And while we’re worrying about the big players like defense departments, let’s not forget to mention the individuals that will get extorted by scammers who steal financial information from screenshots of their online banking sessions, or pull snapshots of them browsing adult websites and threaten to send them to all their social media friends and things of that nature, many of whom may not even realize that Recall was running on their system in the first place. Let’s also not forget the people who will haul their computers down to Geek Squad or their local tech guy wondering why their boot drive is full, only to find that there were tens of gigabytes of Recall snapshots eating up their storage space. Or what about puclibly accessible computers like what you might use in a library or an internet cafe. What assurances do people have that these public machines aren’t recording everything everybody does on them without their knowledge, all under a single user account, for whatever malware they contract to scoop up and ship off, or for the next person who sits down to just open up and look at?
Conclusion
All in all, it seems like a privacy nightmare. This seems like a prime target for hackers. You can bet money that there are already bad actors enrolled in the Windows Insider program just so they can spend time analyzing how new features like this work so that when they roll out to the general public they already have attacks ready to deploy. The Windows Insider program isn’t some exclusive club or anything; anybody can turn on the toggle for no extra charge.
I sincerely hope that Microsoft proves all of my concerns wrong because even though I am personally no longer one of their customers, hundreds of millions of other people are, including our own government, so anything Microsoft does needs to be done with care, precision and respect for their users. I hope they make the security of this feature a top priority. I hope they make it opt-in instead of opt-out, and possibly even offer the option of removing or uninstalling the feature altogether. I hope they make it very clear to users exactly what this feature does and how to use and manage it later. And once it rolls out to the general public, I hope they don’t abandon it 3 years from now after people have gotten used to using it.
So that’s my take. I can see how a tool like this could be useful, and I sincerely hope that Microsoft takes their responsibility to user security and privacy very seriously with a product like this. My biggest concerns boil down to the fact that Microsoft is the company behind it.
Shameless Personal Plug
Writing blogs like this takes time and effort. If you have found value in my work or if it has benefited you in any way, I would appreciate it if you would consider donating to help compensate me for my time. There are several ways you can do this, so here they are in no particular order.
PayPal: https://paypal.me/gerowen
LiberaPay: https://liberapay.com/gerowen
Bitcoin (BTC):
bc1q86c5j7wvf6cw78tf8x3szxy5gnxg4gj8mw4sy2
Monero (XMR):
42ho3m9tJsobZwQDsFTk92ENdWAYk2zL8Qp42m7pKmfWE7jzei7Fwrs87MMXUTCVifjZZiStt3E7c5tmYa9qNxAf3MbY7rD
Gerowen's Corner 