Brick Heart — Research Directions

Why LEGO?

LEGO construction is constrained:

• a finite vocabulary of parts
• discrete dimensions
• regular spatial relationships
• known connection mechanisms
• discrete orientations
• machine-verifiable placement and connection rules

This makes it possible to reason about construction much more precisely than arbitrary 3D modelling.

LDraw provides an open, established representation of LEGO-compatible geometry and can be rendered by existing tools.

Spatial programming

A construction could be represented through operations such as:

SELECT brick_12
ROTATE 90°
MOVE 2 STUDS
CONNECT TO brick_7

The underlying system would maintain a semantic scene rather than merely manipulating screen coordinates.

This makes objects and relationships directly addressable:

"Blue 2×4 brick, connected to the red brick, with three available connection points."

That semantic representation can then be exposed through visual, textual, audio or Braille interfaces.

Accessibility

A major motivation is that conventional 3D interfaces are fundamentally visual and mouse-oriented.

Brick Heart could instead allow a voice user, switch user, touch-screen user and conventional mouse user to operate on the same semantic construction model.

For example:

Brick 12
  ├── Move
  ├── Rotate
  ├── Delete
  └── Connection 1
      Connection 2
      Connection 3

A switch user could sequentially navigate these options; a voice user could issue a semantic command; a screen reader could describe the scene.

This builds on the broader research direction represented by accessible 3D modelling projects such as shapeCAD and A11yShape, but uses LEGO-like construction as a much more constrained domain.

AI and LLMs

An LLM could operate at the level of construction planning, rather than generating arbitrary geometry.

For example:

"Build a bus using 20 red 2×4 bricks, 10 black 1×2 bricks and four wheels."

The model generates a spatial program. A deterministic environment then checks each operation:

VALID
CONNECTED
COLLISION
INVALID ROTATION
PART NOT AVAILABLE

The model can use this feedback to revise its construction. This produces an interactive agent environment, rather than simply asking an LLM to generate a 3D file.

Why not just use Blender, Unreal or Minecraft?

Modern LLMs can already generate substantial Blender scenes and Unreal projects, while Minecraft already occupies much of the programmable spatial-construction space. Minecraft Education supports programmable agents, and research benchmarks such as MineAnyBuild use Minecraft to evaluate spatial planning and construction.

The potential distinction for Brick Heart is component assembly rather than generic voxel construction.

Minecraft primarily asks:

Where should this block go?

Brick Heart can ask:

How do these discrete manufactured components connect, align, support and constrain one another?

This potentially places the system between Minecraft-style construction and real-world robotic assembly.

The benchmark opportunity

The strongest research direction may be to make Brick Heart an LLM/spatial-agent benchmark.

Rather than evaluating whether a generated image looks like a bus, the environment can evaluate exact properties:

• all components connected
• correct orientations
• valid connections
• no collisions
• parts inventory respected
• symmetry satisfied
• construction constraints satisfied
• number of actions
• recovery from invalid actions

A task could therefore specify:

Inventory:
8 × 2×4
4 × 2×2
8 × 1×2
4 wheels

Requirements:
4 wheels
2 axles
bilateral symmetry
all components connected
maximum 20 parts

The result is an executable spatial task with objective evaluation.

Existing work such as LEGO-Puzzles already demonstrates that LEGO is a difficult domain for multimodal spatial reasoning. The opportunity would be to go beyond perception/question answering toward interactive construction and assembly.

Proposed research framing

A useful central question is:

Can a common semantic spatial-programming representation support both accessible human interaction and machine spatial reasoning within a discrete 3D construction environment?

This leads to three broad research questions:

1. Human interaction: Can people with different abilities and input modalities construct and understand spatial models?
2. Representation: Does a semantic spatial representation make 3D construction more accessible and precise?
3. AI: Can LLM agents use the same representation to plan, execute and repair spatial constructions?

LEGO is the experimental medium, not necessarily the final application domain.