'Make It Real' AI Prototype Turns Drawings Into Working Software (arstechnica.com) 50
An anonymous reader quotes a report from Ars Technica: On Wednesday, a collaborative whiteboard app maker called "tldraw" made waves online by releasing a prototype of a feature called "Make it Real" that lets users draw an image of software and bring it to life using AI. The feature uses OpenAI's GPT-4V API to visually interpret a vector drawing into functioning Tailwind CSS and JavaScript web code that can replicate user interfaces or even create simple implementations of games like Breakout. "I think I need to go lie down," posted designer Kevin Cannon at the start of a viral X thread that featured the creation of functioning sliders that rotate objects on screen, an interface for changing object colors, and a working game of tic-tac-toe. Soon, others followed with demonstrations of drawing a clone of Breakout, creating a working dial clock that ticks, drawing the snake game, making a Pong game, interpreting a visual state chart, and much more.
Tldraw, developed by Steve Ruiz in London, is an open source collaborative whiteboard tool. It offers a basic infinite canvas for drawing, text, and media without requiring a login. Launched in 2021, the project received $2.7 million in seed funding and is supported by GitHub sponsors. When The GPT-4V API launched recently, Ruiz integrated a design prototype called "draw-a-ui" created by Sawyer Hood to bring the AI-powered functionality into tldraw. GPT-4V is a version of OpenAI's large language model that can interpret visual images and use them as prompts. As AI expert Simon Willison explains on X, Make it Real works by "generating a base64 encoded PNG of the drawn components, then passing that to GPT-4 Vision" with a system prompt and instructions to turn the image into a file using Tailwind. You can experiment with a live demo of Make It Real online. However, running it requires providing an API key from OpenAI, which is a security risk.
Tldraw, developed by Steve Ruiz in London, is an open source collaborative whiteboard tool. It offers a basic infinite canvas for drawing, text, and media without requiring a login. Launched in 2021, the project received $2.7 million in seed funding and is supported by GitHub sponsors. When The GPT-4V API launched recently, Ruiz integrated a design prototype called "draw-a-ui" created by Sawyer Hood to bring the AI-powered functionality into tldraw. GPT-4V is a version of OpenAI's large language model that can interpret visual images and use them as prompts. As AI expert Simon Willison explains on X, Make it Real works by "generating a base64 encoded PNG of the drawn components, then passing that to GPT-4 Vision" with a system prompt and instructions to turn the image into a file using Tailwind. You can experiment with a live demo of Make It Real online. However, running it requires providing an API key from OpenAI, which is a security risk.
Define working (Score:4)
The truth of the headline strongly depends on how "working" is defined, and I suppose who defines it...
Honestly, "working" and "software" aren't normally associated in my mind. I can't think of anything that I use daily which is less reliable than some of the software I have to use.
Re:Define working (Score:5, Funny)
Don't worry. By tomorrow morning there will be a whole industry backing this piece of software and lauding it as the next replacement for basic competence.
Re: (Score:2)
It is even worse, I read about it yesterday on slashdot, which was literally linking to a literary magazine of dubious quality.
Re: (Score:2)
Don't worry. By tomorrow morning there will be a whole industry backing this piece of software and lauding it as the next replacement for basic competence.
Of course there will! After all, it just takes a start block, leading to the magic block, which leads to the end profit block. I have no idea why someone didn’t think of this 50 years ago.
This stuff is still going to devour jobs (Score:3)
The real change here is that a ton of shit old Mr pointy haired didn't think could be automated but *could* he now believes can be. Yes, we'll all have a good laugh at our boss calling a perl script "AI".... right before he fires us.
And if you don't think you can be replaced by a perl script, well, go away or "I shal
shell, not Perl (Score:2)
If you quote a meme, do it correctly: It's not about a Perl script but a shell script.
Re: (Score:3)
> I can't think of anything that I use daily which is less reliable than some of the software I have to use.
I always think of if Microsoft Windows was a car they would have been sued out of existence by all the accident victims. I mean, imagine if your car had an ignition switch where any key could turn it on, the steering wheel became inverted at arbitrary intervals, and the tires were made of paper. No person in their right mind would buy such a car. But somehow everyone accepted such a shoddy product
Re: Define working (Score:4, Funny)
Re: (Score:2)
Why would anyone in their right mind buy an airplane that goes down when you tell it to go up?
Re: (Score:2)
I always think of if Microsoft Windows was a car they would have been sued out of existence by all the accident victims. I mean, imagine if your car had an ignition switch where any key could turn it on, the steering wheel became inverted at arbitrary intervals, and the tires were made of paper. No person in their right mind would buy such a car. But somehow everyone accepted such a shoddy product from a software company. With the added benefit that through no fault of your own your bank account could be depleted if someone stole your car.
Yes, but look at it from the bright side. You could fix most problems with your car by just closing all the windows and restarting. Though, a BSOD on the highway would be literal...
Re: (Score:2)
Re: (Score:2)
But somehow everyone accepted such a shoddy product from a software company.
1. Nobody dies when your PC does something inappropriate.
2. What was the alternative? There's Mac, which is much more expensive, and Linux, which... well I guess the year of the Linux desktop is coming any day now.
Re: (Score:2)
This is a wonderful tech demo.
It's called Visual Basic Classic (Score:4, Interesting)
You "drew" by dragging and dropping widgets from the toolbox into the form designer. Press the "run" button and you have a form. Variations of the idea from other vendors did automatic data hookups also.
It all fell out of favor because it didn't violate YAGNI enough like modern bloated crap does. Rocket surgery stacks are job security, so professional bloaters scare customers into wanting potential features using reverse YAGNI logic. "Trust me, your lovely little company may easily need web scale and internationalization soon because you are such wonderful shiny skilled managers!" Gag me.
Re: (Score:1)
Re: (Score:1)
Re Dreamweaver, I didn't like the product myself, but it usually got the job done quickly as long as you didn't want some screwy eye-candy. For intranet, it was usually good enough. It just couldn't keep up with eye-candy fads on the internet.
Tried it with a Picasso painting (Score:4, Funny)
Ran the resulting "working software" on my computer, still cleaning up bits of PCB, cabling and what used to be a cool case.
Re: (Score:2)
PLEASE OH PLEASE MOD THIS UP.
And I would like to join your rag-tag crew of product testers.
Re: (Score:2)
Re: (Score:1, Offtopic)
Drew some stars and a citizen (Score:4, Funny)
I only see a text in the middle saying : need more funding to complete this game.
Need more money to complete... (Score:4, Funny)
Wait... half finished, and demanding more money... Oh fuck. It can replace consultants and contractors completely now.
Re: (Score:2)
Wow, I didn't realize AI was that far along.
How do people like this sleep at night? (Score:2)
2.7 million for this? How?
I couldn't do it. I'd either start laughing at the absurdity of it all or confess from the guilt.
Re: (Score:1)
See kids, how jealousy rears its ugly head?
Re: (Score:3)
Re: (Score:2)
Many programmers can't even come near properly made software.
Properly made software takes a long time, and the people paying for software don't like long development times. This is the very reason we've tried for decades to make tools to reduce development time.
I'm of the opinion, based on decades of experience, that most programmers are capable of properly making software, but the market doesn't reward it.
Re: (Score:2)
Re: (Score:2)
IIUC, what it's doing is building a working GUI interface. It doesn't do anything else. (That's all you need for a really simple blocks game.)
The thing is, a lot of design is just for that "simple interface" garbage.
OTOH, you probably don't have a huge amount of control over what that GUI looks like. (Well, colors possibly, and perhaps texture.) I kept crashing KDesigner because I wanted things it couldn't handle, but which C++ plus Qt could handle just fine.
Errrm, ... Tailwind?!? Why that? (Score:3)
Tailwind CSS is specifically designed for humans to push out clean implemented designs quickly. That's why its utility approach has CSS-Nazis foam at the mouth and go into a hissy fit whenever someone even mentions Tailwind. Discussions about utility CSS for humans aside (I personally do see their point), it makes no sense to have bots use this approach
If you're going to have an AI implement the design you might as well generate the actual CSS right away without any framework or intermediary tooling. Obviously.
Re: (Score:2)
AI doesn't worry ... (Score:2)
... about anything. And appreciate CSS classes with appropriate naming are way easier to understand than generic utility CSS microclasses and way more readable because they eliminate the significant bloat that comes with tools like Tailwind.
Re: (Score:2)
I don't think this AI is trained at all? It sounds like they just plug in ChatGPT.
Re: (Score:3)
What is the quality of the produced code though?
How maintainable by humans is it, or are you stuck having to ask the AI to make any changes?
How robust is the software, is it full of SQL injection attacks or trivial code editing hacks?
UML ? (Score:5, Insightful)
Re: (Score:3)
Aww, come on, using UML (or any other similar diagram spec) to represent the general architecture or a specific hairy piece of a program or algorithm visually can be quite useful. I prefer pictures to long-winded specifications when reasonable.
It ain't a silver bullet, but it has its uses.
And nobody forces you to buy the whole toolkit :)
Re: (Score:2)
Re: (Score:2)
And before UML there was Structured Analysis / Structured Design, and before SA/SD there was the humble flowchart. People have been trying to take programmers out of the programming loop for well over 50 years now. "Just sketch out your spec and it runs! So simple even a marketeer can do it!" Except that it always turns out that the only thing detailed enough to produce non-trivial working code is... code. Whether that code is assembly language or Rust, written on a keyboard or by dragging and dropping el
Took long enough (Score:1)
Time to bring back those CRUD tables and create a new version of knowledgeware.
Re: (Score:2)
Do programmers often underestimate time needed to code anything?
Re: (Score:2)
VisualBasic Forms (Score:2)
Has everyone forgotten VisualBasic Forms? Literally the same thing (draw UI, get application) with a little more work (you have to hook up buttons and things yourself).
Sure used a LOT less energy.
Re: (Score:2)
I know you can do similar things with "web technologies" these days, but it's always been an ugly workaround around a protocol that was NOT designed for this (html) and where you only see the visual result of what you're doing when you run the page into a server.
WYSIWYG is *not* evil! (Score:2)
HTML/DOM is too inconsistent to get WYSIWYG. It's why it's impossible to build a PDF renderer in JavaScript + DOM unless you emulate at the pixel level, essentially reinventing an OS from scratch, an "abstraction inversion".
And it's not true that W
Re: (Score:1)
> Difference is, you have to have someone sit down at an IDE to design the program.
That's how most web dev is done anyhow, it's just that we have less control over the UI than non-web. With web you "request" the UI framework and/or DOM to do something, and hope it understands you properly, then turn purple tweaking it until it does. Web UI fiddle-waste is a biggest time drain in IT.
I wish I specialized in Delphi/Lazarus in hindsight. Web UI "standards" suck rotting eggs on a good day. At the time I hoped
Saw another GPT-4V use (Score:2)