Learning reliable plans from images
A neuro-symbolic framework that learns executable PDDL action models from raw visual traces—even when observations are noisy or incomplete.
The challenge
Classical planners require symbolic descriptions of states and actions, normally written by experts. Earlier image-based systems could learn such models without supervision, but their generated plans could contain visual hallucinations or actions disconnected from what an agent can actually execute.
The approach
- 01Grounded learned transitions in real high-level agent actions through a transition-identification mechanism.
- 02Encoded raw images into discrete latent states and learned the preconditions and effects required to translate transitions into PDDL.
- 03Evaluated the resulting models on Towers of Hanoi, Blocksworld and Sokoban under complete, partial and noisy visual data.
- 04Tested executability with the Fast Downward classical planner and studied robustness to transition-identifier errors.
The outcome
R-Latplan produced action models linked to executable agent capabilities and enabled optimal planning across the evaluated complete visual domains, including noisy observations. The work was published at the 36th IEEE International Conference on Tools with Artificial Intelligence.
Why reliability matters
A plan must be executable—not merely plausible.
