Abstract
Large language models (LLMs) can help writers build story worlds by generating world elements, such as factions, characters, and locations. However, making sense of many generated elements can be overwhelming. Moreover, if the user wants to precisely control aspects of generated elements that are difficult to specify verbally, prompting alone may be insufficient. We introduce Patchview, a customizable LLM-powered system that visually aids worldbuilding by allowing users to interact with story concepts and elements through the physical metaphor of magnets and dust. Elements in Patchview are visually dragged closer to concepts with high relevance, facilitating sensemaking. The user can also steer the generation with verbally elusive concepts by indicating the desired position of the element between concepts. When the user disagrees with the LLM's visualization and generation, they can correct those by repositioning the element. These corrections can be used to align the LLM's future behaviors to the user's perception. With a user study, we show that Patchview supports the sensemaking of world elements and steering of element generation, facilitating exploration during the worldbuilding process. Patchview provides insights on how customizable visual representation can help sensemake, steer, and align generative AI model behaviors with the user's intentions.
Project video
Core concept
1. sensemake, 2. steer, and 3. correct AI generations with
💨 Dust and 🧲 Magnet visualization.
That is, when using AI to generate story world elements (e.g., 👥 characters, 📦 props, 📍 places)
to sensemake AI-generated world elements,
the user can first form a magnet space of the concept
(Here, with concepts about factions)
and add world elements to organize them according to the concepts of interest
AI will automatically do positioning of these elements on behalf of the user!
For example,
here, Sir Pouncealot Whiskerton is a spymaster of Scratching Post Clan,
and hence is positioned close to the magnet of the Clan.
The user can
The user also steer AI generation in this magnet space,
simply by placing a marker that indicates "generate element here."
See how Patchview generates an element with steering input!
When AI makes errors for visualizing and generating elements,
The user can correct these in the magnet space.
Simply,
they can reposition the element when AI put it in a wrong place,
or ask AI to rewrite it to be placed in the new position
when the user thinks the element should be revised.
Patchview uses these corrections to align its future behaviors to the user's expectations,
by providing them as examples in the LLM prompts.
Examples from the User Study
We conducted a user study to see how people use Patchview. Below are some examples.
P4 created characters according to their physical power, magical prowess, and goodness-evilness.
P6 organized and generated events along the timeline, with the start and the end magnet being the concept magnet.
P4 generated characters considering four factions they can belong to.