Google is Developing AI that Can Take Over Chrome to Help You Buy Things, Do Research (9to5google.com) 57
This week Google-backed Anthropic announced its upgraded AI model Claude 3.5 Sonnet could "perform tasks like navigating web browsers, filling forms, and manipulating data."
Now Google plans something similar for Chrome, reports 9to5Linux.com: According to The Information, Google is "developing artificial intelligence that takes over a person's web browser to complete tasks such as gathering research, purchasing a product or booking a flight."
"Project Jarvis" — in a nod to J.A.R.V.I.S. in Iron Man — would operate in Google Chrome and is a consumer-facing (rather than enterprise) feature to "automate everyday, web-based tasks." The article doesn't specify whether this would be for mobile or desktop... Given a command/action, Jarvis works by taking "frequent screenshots of what's on their computer screen, and interpreting the shots before taking actions like clicking on a button or typing into a text field."
The Information reports that Google "plans to preview the product, also known as a computer-using agent, as early as December alongside the release of its next flagship Gemini large language model, which would help power the product, two of the people said."
Now Google plans something similar for Chrome, reports 9to5Linux.com: According to The Information, Google is "developing artificial intelligence that takes over a person's web browser to complete tasks such as gathering research, purchasing a product or booking a flight."
"Project Jarvis" — in a nod to J.A.R.V.I.S. in Iron Man — would operate in Google Chrome and is a consumer-facing (rather than enterprise) feature to "automate everyday, web-based tasks." The article doesn't specify whether this would be for mobile or desktop... Given a command/action, Jarvis works by taking "frequent screenshots of what's on their computer screen, and interpreting the shots before taking actions like clicking on a button or typing into a text field."
The Information reports that Google "plans to preview the product, also known as a computer-using agent, as early as December alongside the release of its next flagship Gemini large language model, which would help power the product, two of the people said."
In fairness (Score:1)
i can see this being extremely useful for handicapped people or the elderly. Assuming it does exactly what they ask, this would make their life better and easier.
Of course, this will also be a boon for kids who want to do their "research".
Re:In fairness (Score:5, Interesting)
I think it would mostly be a boon for spam botnets. The first thing I think would happen here is it would be captcha'd to the point of being basically useless, either that or literally everything you do with this would require an active login to a google account to do anything more than jack shit.
I think there would be more to gain with having more standardization of form fields baked into web standards than doing crap like this, that way existing tools can more accurately autofill forms. Probably distill something off of x.500.
Re:In fairness [I want to laugh at AI] (Score:1)
I think you're trying to make an important point with your FP, but you don't make it well and your vacuous Subject in this situation mostly suggests you weren't even sure what point you were trying to make.
Do you make the usual pleading of "But I was in a hurry to capture FP"?
The story does have high potential for Funny, but I'm not expecting much of today's Slashdot. Perhaps we could ask Claude for some humorous contributions? Have to put a governor on it, however, or Claude (and his AI friends) will flood
Re: In fairness [I want to laugh at AI] (Score:1)
Re: (Score:2)
However, I think you are correct in saying that there are, or could be, SOME legitimate use cases.
I have heard, much to my annoyance, articles on pubic radio here in Canada (CBC), doing personal interest stories on elderly and handicapped people who find solace in having audio books read to them by Alexa. The issue I have is that the article never points out or explains anything about what
Just like "agents" from 10-15 years ago (Score:1)
When was it that "agents" were the cool new thing that would shop and search for us and just return the results it knew we wanted?
I find AI is ok for math questions and factual data it is blindly spitting out from someone else's database but when I've asked any of the AI that have current web access to shop or do anything useful for me they fail. The results are worse than just typing in "price of best ice cream machine for home" into google and skimming the first few result summaries.
For those following a
Re: (Score:1)
You know what you need to do if you want to get rid of me short or long term.
Help me get that ice cream maker as specified and I'm going away for at least a week and if you're lucky, forever.
Now is your chance.
Re: (Score:2)
AI, okay for math questions and factual data? Are you joking? AI "hallucinations" are well documented, and LLM accuracy with integer arithmetic is pretty poor.
Re: (Score:2)
Worked for me so far. For example last night we were discussing height/sex in US vs China. The AI gave me an answer that perfectly matched several sites. The decimal place match was such that the AI almost certainly was pulling from one of those sites.
A few weeks ago there was an article here about one of the South Pole glaciers melting and that it would cause sea levels to rise by $x. So I started looking up shit like surface area of the earth and so on and did the calculations and came up with a numbe
Re: (Score:2)
I still can't get any AI to successfully write a BaSH script that removes spaces from filenames, recursively. Seems like a very base-level "intelligence test" for an AI, and it fails every time.
Re: (Score:2)
Hewlett-Packard has shipped you some more ink. Whether you needed it or not.
Re: (Score:2)
They want to make sure my (as yet unborn) grandchildren have plenty on hand.
Re: (Score:2)
As an ice cream fan, I wish you luck in your quest, lol
If you're willing to go the used route there appear to be a few of them on ebay, some of them stating they're in pristine/like new condition in the $460 and up range.
As for new, I see the ones you mention on Ali Express, but I'm not sure if they're fake or not. They're anywhere from ~$450 to $540 with free shipping. What is it that leads you to think that the Ali Express machines are fake? They look legit but I'm no expert on ice cream machines. (??)
$53
Re: (Score:2)
On Ali Express I've ordered from 3 (allegedly) different companies who all came back a few days later to say they don't have it in their warehouse and that I should cancel my order. My credit card charges were properly reversed in all 3 cases but it was a waste of time.
So I sent messages through the chat system to 2 others there to ask if they actually have it on hand, new, in warehouse, ready to ship. No clear answers. One clearly had a language issue and sent a copy paste generic reply. The other said
Re: (Score:2)
Of course I can just go buy one. That's hardly the point and there's no fun in that. I get great pleasure finding the special hard to find deals. I also negotiate at flea markets for things I don't even want that cost less than the drink I had at lunch. Because I enjoy the process.
At conventions my friends all ask me to buy the stuff they want from the vendor room for them. "Ok, sure, what's he asking for it? 40? Ok, how much you willing to pay? 20? Ok, gimme a 20, and don't hover, I'll come find you
Re: (Score:2)
Of course I can just go buy one. That's hardly the point and there's no fun in that. I get great pleasure finding the special hard to find deals.
Yep, I get that; I'm similar in that respect. .
I want to 'win' my ice cream machine and I'm ok with having SD trolls help me do it. Any clown can run a credit card. What's the fun in that?
Exactly. Sure, I can whip out a card and just buy whatever it is, but where's the sport in that? I like to dig around and find a deal on the thing I want. It's way more satisfying if I can find a screamin' deal on it and snag it. It's great to get the thing and get it at a really good price. And I'm patient enough so I can usually wait until I find a deal that pushes my "GET IT" button.
Not in my world it ain't (Score:3)
Google's not taking over anything on my computers.
In fact, I work my butt off to keep Google out of my life as much as humanly possible - which is unfortunately very difficult since Google managed to insert themselves into everything and giving them the boot completely means no working internet.
Fuck control-freak monopolies.
Re: (Score:2)
>"Google's not taking over anything on my computers."
Then you should have less to worry about, because you are using Firefox and this is about AI taking over Chrome.
>"is unfortunately very difficult since Google managed to insert themselves into everything and giving them the boot completely means no working internet."
I don't use Google's products/services/sites for anything on my desktop, and the internet seems to work just fine for me :)
Re:Not in my world it ain't (Score:4, Informative)
Your browser silently hit many Google servers whether you want it or not anything you visit almost any site on the internet.
Re: (Score:2)
>"Your browser silently hit many Google servers whether you want it or not anything you visit almost any site on the internet."
Well, yeah. It is impossible to avoid ALL second/third-party interaction with Google (fonts, javascript CDS, etc), but I am using UBO which blocks a lot of the worst stuff. I don't use their DNS, browsers, services, logins, etc. I am probably as much off their radar as it is possible for most anyone to be while being online.
Re: (Score:2)
I don't use Google's products/services/sites for anything on my desktop
Not sure what are your criteria for "use Google", but you don't need to call Google directly or install any of their products on your desktop in order to interact with them. Many web pages link to Google properties, like google-analytics or gstatic, some public wifi networks use Google's DNS, Google maps are used/embedded in many applications or sites, and so on. Even ignoring this and even if you're not online at all Google still gets tracking data about you: they buy the credit card transactions you make
Re: (Score:2)
Courts have ruled that businesses can be held to promises made by their AI agents. I expect the same will be true of consumer-side agents, at least until a scummy business or advertiser figures out a way to replicate https://xkcd.com/1807 [xkcd.com] with this kind of bot.
Re: (Score:2)
Ooh look someone copied Microsoft's dystopian idea (Score:2)
frequent screenshots of what's on their computer screen
Recall much [tomsguide.com]?
Honestly at this point, it's fair to say dig data is out of control and becoming a seriously aggressive threat to privacy.
Re: (Score:2)
frequent screenshots of what's on their computer screen
Recall much [tomsguide.com]?
Honestly at this point, it's fair to say dig data is out of control and becoming a seriously aggressive threat to privacy.
Made me think of this as well; you beat me to it.
How does it differentiate between contexts? (Score:5, Informative)
'Jarvis works by taking "frequent screenshots of what's on their computer screen, and interpreting the shots before taking actions like clicking on a button or typing into a text field."'
So how does it differentiate between contexts? Or does it just assume that a user only cares about, and works on, only one thing? Since when does ANY desktop user only have one topic on their screen ever?
Also, as this tool drives the computer to "perform actions" on behalf of the user, how do individual users remain differentiated? Increasingly, all users will become homogenized into buckets according to actions Google decides to perform. This is already a fundamental problem with "the algorithm" and it is a major issue for society, this would turbocharge it.
We know the answers already. Google doesn't care about context, it cares about learning more and more about individual users. Google isn't interested in separating your porn browsing from your work research and it doesn't care about any inadvertent cross pollination of those activities. Google cares about monetizing you, not providing you useful tools. The headline here should be that Google plans to take continuous screenshots of usage and make it available for their own use. And don't forget, part of Google's use is government use, Google already reports your computer use to the government under the "right" circumstances. If you aren't convinced that Google is the enemy, what will it take?
Re: (Score:2)
Hell, let's break it down more. How does it differentiate between browsing Amazon for Christmas gift ideas (which I reckon a lot of us are getting close to) and browsing Amazon to actually BUY Christmas gifts? Is it just going to indiscriminately buy any product you open up to check the details? And are you going to actually be on the hook for the bill?
No (Score:2, Offtopic)
I wonâ(TM)t use Chrome due to manifest v3
Why? (Score:2)
Why even run Chrome, or any web browser for that matter? The AI should be able to determine what it is that I should be buying, place the order and submit my credit card information.
Life would be so much simpler if we would all consume as we are expected to and stop trying to make our own decisions.
I would love to have ... (Score:3)
... an excellent AI purchasing agent that accurately managed my inventory and ordering.
Maybe someday in the future, this will be available, but today's AI will ineptly suggest useless crap you don't need.
Meanwhile, I read articles where "New machine learning model quickly and accurately predicts dielectric function". This is the direction that excites me about AI. Unfortunately, today's mass market AI seems focused on generating crap and now, selling crap
Re: (Score:2)
... an excellent AI purchasing agent that accurately managed my inventory and ordering.
If you hadn't abused the subject line...
But my point is that if another human is providing that 'automated' service to you, you can be certain that that humans intentions will be revealed in the automation.
Long story short, there is no assistant that you will ever have that will work strictly towards YOUR goals unless you write it yourself.
Google missed the memo that said ... (Score:5, Funny)
[Filed under, "What were they thinking?"]
Re: (Score:2)
That's the only Funny in the "matured" discussion? Not such a bad joke, but I think the story had rather more potential for humor.
If Slashdot was still "significant" enough to justify some sort of research, I think one interesting topic would be the effects of FPs on the discussion. On one level, it would seem rather straightforward. Some of the categories of FPs are easy to recognize. Main features for such FP analysis might be length of FP and quality of Subject, especially for topical versus vacuous. Som
WTF? (Score:2)
'Jarvis works by taking "frequent screenshots of what's on their computer screen, and interpreting the shots before taking actions like clicking on a button or typing into a text field."'
So, when does the first automated install of malware by Jarvis occur in the wild*? (*only if this thing gets into real production use)
Anyone set up a pool yet?
Two obvious problems here (Score:4, Interesting)
Problem 1: The Alexa Problem. People do not _WANT_ an external entity to be their purchasing concierge. At least, not regular consumers. Sure, if you're too rich and busy to comparison shop and you just want "Gucci T-shirt, blue, XXL, same day delivery" concierge shopping is great. But real people want to price compare. Google's agent will not - for various reasons (see #2 below) do the same sorts of comparisons a real person would do.
Problem 2: Gaming. This just becomes another front in the SEO war, because a) Google will establish preferential relationships, b) they don't necessarily have the entire purchasing spectrum in their databases even assuming a completely level playing field [which it will never, ever be, btw], and perhaps most importantly c) this merely spawns another arms race of tools, techniques, botnets, device farms and other weapons that artificially raise my product's ranking above yours.
tl;dr - Pre-doomed.
Re: (Score:2)
Eg-ads! (Score:1)
Google Chrome:
"We have ads!"
"We have AI ads that help you find ads!"
"We have AI ads that force you to see ads even if you block ads!"
"We have ads about ads that help you find ads inside ads!"
570 billion planets (Score:2)
"Google is Developing AI that Can Take Over Chrome to Help You Buy Things"
There are 570 billion planets out there and I had to end up on the one where Google's AI is helping me buy things. FML
A target for scammers (Score:2)
Yeah, Google doesn't want some corporation's on-retainer lawyers on their doorstep because their AI autonomously purchased $20 million of care-bears. It's better to leech from a working-class that can't pay for 10 years of court-room name-calling.
A software agent with one's personal details: It's a valuable target for scammers. How will JARVIS know it isn't pointed at a fake Western Union web-page? Will it check encryption certificates for a recognized owner? For this not to be a phishing free-for-all,
Who Pays? (Score:2)
Honey, did you order two dozen antelope?
I didn't need another reason to avoid Chrome... (Score:1)
I didn't need yet another reason to avoid Chrome, but Google thoughtfully gave me one anyways.
Web extension support (Score:2)
Hey, let's cripple the web extensions API so it can't do anything useful anymore, like ad blocking.
Oh, hey... now let's let AI take over the browser so pesky end-users don't have to make their own decisions!
Plausible deniability (Score:2)
"Google is 'developing artificial intelligence that takes over a person's web browser to complete tasks such as gathering research, purchasing a product or booking a flight.'"
Does AI count as a legal agent that can legally make decisions on your behalf? Like an agent with power of attorney privileges? If not, then wouldn't decisions made by such an agent by legally deniable? I.e., legally plausible deniability.
Other other hand, if the AI does have power of attorney privileges, then ... lots of really bad
Re: (Score:2)
As such it is a tool, but only a tool would use it to automate spending their own money.
Read my mind (Score:2)
AI, read my mind
Another reason NOT to use Chrome.... (Score:2)
Ah, yes, just what I'm missing. (Score:2)
Living isn't quite expensive enough right now. What I really need is Google's agent deciding when I need to buy things and doing it for me. I'll just pay the bill and never complain, because that's what we're supposed to do, right? The corporate world knows better than I do what I need to buy. Be a good consumer, let Google decide for you!
GTFO. Automation may be a good thing in some contexts, but "buying things for you" is not an area I want Google anywhere near. Piss off.
"help" me? (Score:2)
Really? So it will save credit card info, and buy things for me... that it decided I needed? When do we get to click "you will agree to pay for whatever the system buys, whether you wanted it or not, whether you can afford it or not""?