'Make It Real' AI Prototype Turns Drawings Into Working Software
-
wrote on Nov 17, 2023, 5:33 AM last edited by
_On Wednesday, a collaborative whiteboard app maker called "tldraw" made waves online by releasing a prototype of a feature called "Make it Real" that lets users draw an image of software and bring it to life using AI . The feature uses OpenAI's GPT-4V API to visually interpret a vector drawing into functioning Tailwind CSS and JavaScript web code that can replicate user interfaces or even create simple implementations of games like Breakout. "I think I need to go lie down," posted designer Kevin Cannon at the start of a viral X thread that featured the creation of functioning sliders that rotate objects on screen, an interface for changing object colors , and a working game of tic-tac-toe . Soon, others followed with demonstrations of drawing a clone of Breakout , creating a working dial clock that ticks, drawing the snake game , making a Pong game , interpreting a visual state chart , and much more .
Tldraw, developed by Steve Ruiz in London, is an open source collaborative whiteboard tool. It offers a basic infinite canvas for drawing, text, and media without requiring a login. Launched in 2021, the project received $2.7 million in seed funding and is supported by GitHub sponsors. When The GPT-4V API launched recently, Ruiz integrated a design prototype called "draw-a-ui " created by Sawyer Hood to bring the AI-powered functionality into tldraw. GPT-4V is a version of OpenAI's large language model that can interpret visual images and use them as prompts. As AI expert Simon Willison explains on X, Make it Real works by "generating a base64 encoded PNG of the drawn components, then passing that to GPT-4 Vision" with a system prompt and instructions to turn the image into a file using Tailwind._
You can experiment with a live demo of Make It Real online. However, running it requires providing an API key from OpenAI, which is a security risk.
-
wrote on Nov 17, 2023, 11:55 AM last edited by jdaviescoates Nov 17, 2023, 12:00 PM
Sadly the self hosted version of TLDraw is fairly useless as you can't collaborate with others nor share any boards.
-
wrote on Nov 17, 2023, 12:50 PM last edited by
that is amazing