SPECTRE: Skeleton Priority Estimation Conditioned on TAMP Refinement Evidence

Anonymous Authors
Under review at ICRA 2027
Supplementary Material
Initial scene with the goal of restocking the shelf; candidate plans with their initial ranking; a failure observation where placement fails with a wrist collision; and the successful plan in which all picks and places succeed.

SPECTRE uses refinement feedback to guide plan selection in a restocking task. The robot must place tall and short cans on a shelf (left). Refinement of the initially highest-ranked plan fails because the arm lacks clearance to place a tall can on the top shelf (top right). SPECTRE uses this feedback to re-rank the remaining candidates (center) and selects a feasible plan that places tall cans on the bottom shelf (bottom right).

Real-World Validation

We demonstrate SPECTRE in a real-to-sim-to-real restocking task. The robot is a TidyBot++ mobile manipulator with a seven-degree-of-freedom Kinova Gen3 arm and a Robotiq 2F-85 gripper. SPECTRE uses failure evidence during simulated refinement to re-rank the candidates, and the robot executes the resulting plan. (Video playback at 15× speed.)

Abstract

Refining abstract plans into executable robot motions is a computational bottleneck in task and motion planning (TAMP). Learned plan rankers aim to reduce failed refinement attempts by predicting which plans will succeed. However, using only the initial scene or refinement success statistics leaves out information computed during failed attempts, such as which objects obstructed an action. We present SPECTRE, a learned plan ranker that uses this evidence to re-rank the remaining candidates after each failed refinement. The ranker learns how structured failure records and derived features should adjust its initial predictions from the scene and goal. We use the same record fields and derived features across all tested domains. We evaluate SPECTRE on three TAMP domains against a classical planner, two learned methods, and a vision-language model planner. SPECTRE reduces mean wall-clock time to the first feasible plan by 17–59% relative to the fastest baseline in each domain. It also requires fewer failed refinement attempts on average than the learned baselines on harder problems excluded from training. Ablations show that using failure feedback reduces the mean number of failed refinement attempts by 57–60% relative to the same ranker without feedback. Our results show that information computed during refinement is useful for learning to guide task and motion planning.

Approach

For each problem, we use top-k planning to generate a fixed pool of K goal-reaching skeletons. SPECTRE initially scores these candidates from the scene and goal, with an empty failure history. It passes the highest-scoring skeleton to the refiner. On success, the planner returns the solution. After a failed attempt, the refiner returns failure records. SPECTRE stores the attempted skeleton with these records in the history. SPECTRE re-scores the untried candidates using the scene, goal, pool, and failure history and refines the highest-scoring one. It repeats this process until it finds a solution, exhausts the pool, or reaches the computation budget.

SPECTRE architecture. Scene tokens and a candidate embedding feed a static channel with cross-attention and a static head, trained in stage 1 and then frozen. Failure-history record tokens and the compiled features coverage, waste, blame, and Jaccard feed an evidence channel with cross-attention and an adaptive head, trained in stage 2. The two scores are combined through a gate to rank plans; the refiner logs a record per rejected sample and appends it to the failure history.

SPECTRE architecture. A candidate’s score sums a static channel that accounts for the plan, scene, and goal (trained on Stage 1, then frozen) and a failure-based evidence channel that reads the record tokens and compiled scalars (Stage 2). After each failed refinement, the new evidence is appended, and the remaining pool is re-scored.

Results

Environments

We evaluate the planners in three manipulation environments with different sources of refinement failure.

StickButton2D (SB2D)

A robot must press buttons in two regions, using a stick to reach one region. Refinement can fail because a press is out of reach or would trigger an unintended button. Problems contain 1, 2, 3, or 5 buttons.

Drawer Decluttering 2D (DD2D)

The robot must retrieve a target from a cluttered drawer, staging blocking objects on a small tray when needed. Refinement can fail if blockers remain in the target’s grasp corridor or cannot be placed on the tray. Problems require staging 0 to 3 blockers.

Restock3D

The robot must pack objects of varying widths and heights onto two shelf levels with different clearances. Refinement can fail if an object or its required grasp lacks clearance, or if too many objects are assigned to one level. Problems contain 6 to 9 objects.

We compare the planners on held-out problems at all difficulty levels. SPECTRE has the lowest mean failed-refinement count and runtime in each environment.

Failed refinements before first success on StickButton2D, DD2D, and Restock3D for Base, SPECTRE, PIGINet, LAZY, and VLMPlan, shown as strip plots with mean bars and confidence intervals. Wall-clock time to first success on StickButton2D, DD2D, and Restock3D for Base, SPECTRE, PIGINet, LAZY, and VLMPlan, with bars broken down into abstract plan generation, inference, and refinement time.

Failed refinements before first success (top) and wall-clock time to first success (bottom), per environment, for Base, SPECTRE, PIGINet, LAZY, and VLMPlan. Wall-clock bars are broken down into abstract plan generation, inference, and refinement time. The ratio above each bar is that method’s mean divided by SPECTRE’s mean in the same panel.