Google is Developing AI that Can Take Over Chrome to Help You Buy Things, Do Research

This week Google-backed Anthropic announced its upgraded AI model Claude 3.5 Sonnet could "perform tasks like navigating web browsers, filling forms, and manipulating data."

Now Google plans something similar for Chrome, reports 9to5Linux.com: According to The Information, Google is "developing artificial intelligence that takes over a person's web browser to complete tasks such as gathering research, purchasing a product or booking a flight."

"Project Jarvis" — in a nod to J.A.R.V.I.S. in Iron Man — would operate in Google Chrome and is a consumer-facing (rather than enterprise) feature to "automate everyday, web-based tasks." The article doesn't specify whether this would be for mobile or desktop... Given a command/action, Jarvis works by taking "frequent screenshots of what's on their computer screen, and interpreting the shots before taking actions like clicking on a button or typing into a text field."
The Information reports that Google "plans to preview the product, also known as a computer-using agent, as early as December alongside the release of its next flagship Gemini large language model, which would help power the product, two of the people said."

In fairness

[quonset]

i can see this being extremely useful for handicapped people or the elderly. Assuming it does exactly what they ask, this would make their life better and easier.

Of course, this will also be a boon for kids who want to do their "research".

Re:

[ArmoredDragon]

I think it would mostly be a boon for spam botnets. The first thing I think would happen here is it would be captcha'd to the point of being basically useless, either that or literally everything you do with this would require an active login to a google account to do anything more than jack shit.

I think there would be more to gain with having more standardization of form fields baked into web standards than doing crap like this, that way existing tools can more accurately autofill forms. Probably distill s

Re:In fairness [I want to laugh at AI]

[shanen]

I think you're trying to make an important point with your FP, but you don't make it well and your vacuous Subject in this situation mostly suggests you weren't even sure what point you were trying to make.

Do you make the usual pleading of "But I was in a hurry to capture FP"?

The story does have high potential for Funny, but I'm not expecting much of today's Slashdot. Perhaps we could ask Claude for some humorous contributions? Have to put a governor on it, however, or Claude (and his AI friends) will flood

Re:

[Big Hairy Gorilla]

This is the same argument as "think of the children".. an emotional appeal on behalf of a marginalized group in society.

However, I think you are correct in saying that there are, or could be, SOME legitimate use cases.

I have heard, much to my annoyance, articles on pubic radio here in Canada (CBC), doing personal interest stories on elderly and handicapped people who find solace in having audio books read to them by Alexa. The issue I have is that the article never points out or explains anything about what

Just like "agents" from 10-15 years ago

[iAmWaySmarterThanYou]

When was it that "agents" were the cool new thing that would shop and search for us and just return the results it knew we wanted?

I find AI is ok for math questions and factual data it is blindly spitting out from someone else's database but when I've asked any of the AI that have current web access to shop or do anything useful for me they fail. The results are worse than just typing in "price of best ice cream machine for home" into google and skimming the first few result summaries.

For those following a

Re:

[iAmWaySmarterThanYou]

You know what you need to do if you want to get rid of me short or long term.

Help me get that ice cream maker as specified and I'm going away for at least a week and if you're lucky, forever.

Now is your chance.

Not in my world it ain't

[Rosco P. Coltrane]

Google's not taking over anything on my computers.

In fact, I work my butt off to keep Google out of my life as much as humanly possible - which is unfortunately very difficult since Google managed to insert themselves into everything and giving them the boot completely means no working internet.

Fuck control-freak monopolies.

Re:

[markdavis]

>"Google's not taking over anything on my computers."

Then you should have less to worry about, because you are using Firefox and this is about AI taking over Chrome.

>"is unfortunately very difficult since Google managed to insert themselves into everything and giving them the boot completely means no working internet."

I don't use Google's products/services/sites for anything on my desktop, and the internet seems to work just fine for me :)

Re:

[Rosco P. Coltrane]

Your browser silently hit many Google servers whether you want it or not anything you visit almost any site on the internet.

Re:

[ChatHuant]

I don't use Google's products/services/sites for anything on my desktop

Not sure what are your criteria for "use Google", but you don't need to call Google directly or install any of their products on your desktop in order to interact with them. Many web pages link to Google properties, like google-analytics or gstatic, some public wifi networks use Google's DNS, Google maps are used/embedded in many applications or sites, and so on. Even ignoring this and even if you're not online at all Google still gets tracking data about you: they buy the credit card transactions you make

Re:

[Entrope]

Courts have ruled that businesses can be held to promises made by their AI agents. I expect the same will be true of consumer-side agents, at least until a scummy business or advertiser figures out a way to replicate https://xkcd.com/1807 [xkcd.com] with this kind of bot.

Ooh look someone copied Microsoft's dystopian idea

[Rosco P. Coltrane]

frequent screenshots of what's on their computer screen

Recall much [tomsguide.com]?

Honestly at this point, it's fair to say dig data is out of control and becoming a seriously aggressive threat to privacy.

Re:

[fahrbot-bot]

frequent screenshots of what's on their computer screen

Recall much [tomsguide.com]?

Honestly at this point, it's fair to say dig data is out of control and becoming a seriously aggressive threat to privacy.

Made me think of this as well; you beat me to it.

How does it differentiate between contexts?

[dfghjk]

'Jarvis works by taking "frequent screenshots of what's on their computer screen, and interpreting the shots before taking actions like clicking on a button or typing into a text field."'

So how does it differentiate between contexts? Or does it just assume that a user only cares about, and works on, only one thing? Since when does ANY desktop user only have one topic on their screen ever?

Also, as this tool drives the computer to "perform actions" on behalf of the user, how do individual users remain differentiated? Increasingly, all users will become homogenized into buckets according to actions Google decides to perform. This is already a fundamental problem with "the algorithm" and it is a major issue for society, this would turbocharge it.

We know the answers already. Google doesn't care about context, it cares about learning more and more about individual users. Google isn't interested in separating your porn browsing from your work research and it doesn't care about any inadvertent cross pollination of those activities. Google cares about monetizing you, not providing you useful tools. The headline here should be that Google plans to take continuous screenshots of usage and make it available for their own use. And don't forget, part of Google's use is government use, Google already reports your computer use to the government under the "right" circumstances. If you aren't convinced that Google is the enemy, what will it take?

Re:

[Calydor]

Hell, let's break it down more. How does it differentiate between browsing Amazon for Christmas gift ideas (which I reckon a lot of us are getting close to) and browsing Amazon to actually BUY Christmas gifts? Is it just going to indiscriminately buy any product you open up to check the details? And are you going to actually be on the hook for the bill?

No

[GrahamJ]

I wonâ(TM)t use Chrome due to manifest v3

Why?

[PPH]

Why even run Chrome, or any web browser for that matter? The AI should be able to determine what it is that I should be buying, place the order and submit my credit card information.

Life would be so much simpler if we would all consume as we are expected to and stop trying to make our own decisions.

I would love to have ...

[MpVpRb]

... an excellent AI purchasing agent that accurately managed my inventory and ordering.
Maybe someday in the future, this will be available, but today's AI will ineptly suggest useless crap you don't need.
Meanwhile, I read articles where "New machine learning model quickly and accurately predicts dielectric function". This is the direction that excites me about AI. Unfortunately, today's mass market AI seems focused on generating crap and now, selling crap

Google missed the memo that said ...

[fahrbot-bot]

... Don't use the words "AI' and "take over" in the same sentence.

[Filed under, "What were they thinking?"]

WTF?

[jvkjvk]

'Jarvis works by taking "frequent screenshots of what's on their computer screen, and interpreting the shots before taking actions like clicking on a button or typing into a text field."'

So, when does the first automated install of malware by Jarvis occur in the wild*? (*only if this thing gets into real production use)

Anyone set up a pool yet?