Blog
Design ArenaAI benchmark human preferencenon-verifiable domains

What Is Design Arena? How AI Creative Benchmarks Work

Design Arena turns what millions of users pick on websites, images and video into a benchmark for subjective AI work. How it works and why labs care.

Portrait of Luis Chavez-Mattos
Edited by Luis Chavez-Mattos, Director, Product··RSS

For Design, Marketing and Engineering

Overhead view of a dense grid of identical blank matte website mockup cards, with one red card slightly raised and casting a tiny hard shadow.

What is Design Arena?

Design Arena is a product where people use AI models to make creative things, such as websites, images and videos, and where their choices are turned into a benchmark. A user types a request, gets outputs from different models, and keeps what they like. Behind the scenes, those choices and edits become data on which models people actually prefer. Its CEO, Grace, describes the goal as measuring how good frontier models are in areas that are hard to verify, and then helping AI labs improve their models there.

TL;DR

  • Design Arena is a product first and a benchmark second. People come to build websites, images and videos, and their behavior becomes the measurement.
  • It targets non-verifiable domains, meaning tasks like design where no answer key says what "good" is.
  • The benchmark relies on population-level signal. One person's taste, or one expert's, can't tell a lab whether a model is failing for a whole audience.
  • Design Arena's CEO says stated preferences drift over time. What people say they want in one quarter differs from a few quarters later, and some labs try to predict that drift.
  • Different labs aim at different audiences, so "better" design can mean different things depending on who the model is for.
  • The company describes itself as a possible marketplace for specific intelligence, where many specialized models are chosen for particular jobs.
  • For you, the practical lesson is that leaderboards of general intelligence say little about creative quality. Test models on your own work.

Why is it hard to benchmark subjective AI output?

Most well-known AI benchmarks test things with a right answer: a math problem, a chemistry question, a block of code that passes or fails. These are called verifiable domains. A model can be scored automatically, and it can be trained against that score.

Design is different. No answer key says a landing page looks good, that a logo feels right, or that a video is compelling. Grace described the conventional wisdom from a year or so ago: models would keep improving at math, chemistry and physics, and would struggle to improve in non-verifiable domains like design or game development. Building a training environment for "this looks good" is hard because nothing automatically says what good is.

She disagreed with that conclusion. People do get better at subjective skills. A game developer with three years of experience is clearly more skilled than a beginner. So there must be a signal to learn from. It is just learned differently than a math answer. Her argument was that progress was slow partly because nobody had a clear target for what "better" means in these areas.

Verifiable domainsNon-verifiable domains
ExamplesMath, physics, chemistryWeb design, images, video, games
How "good" is decidedA correct answer existsDepends on who is looking
Who can judgeAnyone with the answer keyA large population of users
Typical measurementPass or failPreference and behavior at scale

How does Design Arena turn product usage into a benchmark?

The company started by building a game engine. About a year and a half before the interview, the best models could only output a single long HTML file at a time, and there were no coding agents as capable as today's. The models could make games that worked, but the games weren't fun. They lacked good design, storylines and anything that kept a player coming back.

That raised a question: is there a real signal for whether a game is good? There was. Retention, meaning how many people keep playing, is a ground truth of sorts (Grace noted that Steam uses something similar). From that came the idea of a two-sided product. Humans get access to powerful AI to make things they love. The AI, in turn, gets access to humans, so it can learn what people love at scale.

The product then expanded from games to websites, images and video. Grace said it has grown to over 7 million users across 192 countries.

The mechanics work like this:

  1. A user arrives with a real task, such as making a website, an image or a video.
  2. They get multiple outputs from different models.
  3. They choose, request edits, and say what they like or dislike.
  4. Those interactions are converted into signals about whether the user enjoyed each model's output.
  5. The aggregate becomes a real-world evaluation of the models.

Grace compared this to a product manager. A PM watches what users do in a product and tells engineers to fix specific things. Design Arena watches what users do with models and tells model teams what to fix.

Why show people multiple options instead of asking for ratings?

Because asking people to rate things makes the product worse, and people stop using it. Grace was direct that you can't degrade the user experience by asking for labels. The labeling has to be built into something people want to do anyway.

Her comparison was a designer decorating your room. A good designer brings several mockups, because seeing the outputs helps you understand what you want. Choosing among options is a natural part of a visual project. So the same behavior that improves the experience also produces the data. PMs do something similar when they track key buttons rather than interrupting users to ask whether they are having a good time.

Why do labs need population-level data?

Grace's argument is that you can't improve a model until you know where it breaks. In subjective areas, you can't find those failures from one person's opinion. If you prefer X to Y, that doesn't tell you whether a whole audience, or the particular group a lab is targeting, agrees.

She used TikTok as the example. TikTok has no single person who knows what makes a good video. It has built a system, an algorithm plus enormous amounts of behavioral data, that predicts how a video will perform. Each viewer is effectively the verifier, and how long they stay is the signal. It also isn't a "house of high-taste individuals," she said, because what counts as high taste in one cohort may be low taste in another cohort that matters just as much.

That is why a single expert panel is a weak verifier for design. The useful eval is a large population making real choices.

What are stated and revealed preferences, and why do they drift?

Grace separated two things:

  • Stated preferences: what people say they want.
  • Revealed preferences: what they ultimately choose.

Even looking only at stated preferences, she said, the data shifts over time. What people asked for in the first quarter of a year looks quite different from the third or fourth quarter. Because Design Arena sees this data at scale, she argued, it gets an early view of where things are heading, roughly three to six months out, rather than just reacting to today.

She said some frontier labs use this drift to improve their models. They try to predict what people will want next quarter and train for it. This should be read as the CEO's description of lab behavior, not a documented industry standard. But the idea is useful for anyone: tastes in design change, and a model tuned to last year's look can feel dated.

Do all AI labs optimize for the same thing?

No. Grace said labs have different target demographics. A large frontier lab might want to appeal to a different audience than a small open-source lab. For subjective work, "better" is different in each case, so different post-training is needed.

Her example: if you want to serve users in Asia, your idea of how a user interface should look may differ from the one you'd use to serve North America. She compared the situation to the advertising world, where the right output depends on the task and on who is asking.

What is the "marketplace for specific intelligence"?

This is Grace's view of where the AI industry may go. She cited a claim from a customer-support AI company that about 70% of its model usage is open-source models. That makes sense to her: answering a phone and saying "Hi, how are you? Can I start an account for you?" doesn't need the most advanced model. A cheap, fast one can do it.

From that she drew a bigger idea. The frontier of intelligence may look less like one giant model and more like a system that chains together many specialized components. If those components come from a mix of providers (open source, application companies, newer labs and frontier labs), then someone needs to say which one is best for which job and which audience. She said her company sees itself as a marketplace of that kind. She also named conditions that would affect this: how easy it is to post-train a model for one capability, whether anyone can build a defensible advantage there, and how cheap general intelligence becomes.

Treat this as a stated vision, not a product you can buy today.

What does this mean for you?

If you use AI for creative work. A model that ranks high on a math or coding leaderboard may not produce the best slide layout, product image or landing page. Generate the same request with two or three models and compare. Keep notes on which one you picked and why.

If you manage a team that uses AI. Don't decide on one tool because of a general reputation. Ask your team to run side-by-side tests on real tasks, such as a customer email, a brochure image or a campaign page, and let a group choose. One person's taste is a weak signal. Several people choosing is stronger.

If you buy AI tools. Ask vendors what audience their model was tuned for. A design tool built for one market may feel off in another.

If you're curious about where things are heading. Expect more specialized models for specific jobs, and more attention to measuring taste, not just correctness.

Can you build your own small preference test?

Yes, and you can do it for your own team. The Design Arena idea scales down: show several outputs, record which one people choose. Remy can build a small internal app like this from a plain-English description. It could take a prompt, run it through several of the AI models Remy has access to, show the outputs side by side, and log each person's pick and comment in a database. That gives a marketing or design team its own record of which model works best for its tasks, instead of a shared spreadsheet and scattered opinions.

Frequently Asked Questions

4 questions

Is Design Arena a website I can use?

According to the interview, yes. People use it to build websites, images and videos with AI models, and that usage is what generates the benchmark data. The CEO described it as one of the default places people go to make creative things with models.

What is a non-verifiable domain?

It is a task with no automatic way to check the answer. Math has a correct result. A website design or a video doesn't. Whether it is good depends on the people looking at it, which is why measuring it takes data from many users.

Why can't a panel of design experts judge AI output?

Grace argued that one expert, or a small group, can't tell you whether a whole audience would agree. What counts as good taste in one group may be poor taste in another. She said you need population-level data to find where a model fails for the group it is meant to serve.

Does the benchmark prove one model is best?

Not in general. The CEO's own argument is that "best" depends on the task and the audience. She called her company's benchmark the most accurate in the world for creative domains, but that is the company's claim, and I haven't seen independent verification of it.

How we source, write, and correct our articles. Editorial standards

Related articles

Straight to your inbox.
New pieces as they publish. No noise.