One Shot
DistillingSol 6.1 into Qwen and Flint for TerminalBench Drake Robotics task.

Sol 6.1 solves Terminal-Bench's biped-contact-dynamics task on creating biped robot that walks and jumps. It uses Drake, a robotics library I know well enough to understand if the resulting python code is any good.

The assignment sounds approachable: make a two-legged robot walk, jump, and run. The deliverable is a Python generator that reads a configuration and writes three NumPy archives. Each archive contains time, robot state, actuator torques, foot-contact forces, and the contact mode. The difficulty is making those quantities agree. An airborne robot should actually leave the ground, etc. Furthermore, the solution should be robust to a hidden example (movement different from walk and jump that the evaluator runs).

However, when I opened the Sol solution code in PyCharm, the solution is dense python file, that ignores drake libraries and used Scipy to solve things from first principles. Basically unmaintainable code that passes benchmark.

However, we have a human generated solution as part of the benchmark. What if I ask Sol to compare the human solution, its own solution, and come up with better benchmark description. For example, since there is no web access, the benchmark should include some drake samples and documentation. Once we can transfer human style to Sol, we can transfer a single Sol solution to Flint and Qwen and have local models solve these tasks as well. Currently the local models produce valid code that fails physics checks.

I'm calling this “one shot distilling”. I am transferring guidance through the task description and its supporting material. I am not updating weights or claiming to have reproduced Sol's capabilities in a smaller model, I am also not giving away the solution, just telling Sol to make a human friendly solution.

Here is a small excerpt from the first solution:

class Generator:
    def __init__(self, path):
        self.p = MultibodyPlant(0.0);
        Parser(self.p).AddModels(str(path));
        self.p.WeldFrames(self.p.world_frame(), self.p.GetBodyByName('base').body_frame(), RigidTransform.Identity());
        self.p.Finalize();
        self.c = self.p.CreateDefaultContext()
        self.B = self.p.MakeActuationMatrix();

From passing to readable

The naming is only the visible part of the problem. It did use Drake for the parsed model and final inverse-dynamics calculations. But much of the numerical machinery sat outside Drake's modeling and optimization interfaces, and the code compressed several responsibilities into one place.

The task also comes with a human-provided reference solution. For example, the human written code is:

class DrakeBiped:
    def __init__(self, model_path: Path):
        builder = DiagramBuilder()
        self.plant = builder.AddSystem(MultibodyPlant(0.0))
        Parser(self.plant).AddModels(str(model_path))
        self.plant.WeldFrames(
            self.plant.world_frame(),
            self.plant.GetBodyByName("base").body_frame(),
            RigidTransform.Identity(),
        )

Like calling plant a plant instead of p makes the code a lot more readable.

Sol generated the following instruction block, including the offline documentation location:

## Required implementation architecture and style


Write readable, maintainable PyDrake code following the bundled examples
in /app/examples/drake/ and Drake's Python style guide.


### Model abstraction


Implement a class named DrakeBiped whose responsibility is access to the
robot model and its kinematics and dynamics.


- Construct a MultibodyPlant from config["model_path"], apply the model's
  required frame weld, finalize it, and create its Context.
- Use descriptive attributes such as self.plant, self.context, and
  self.actuation_matrix.


... 


Offline Drake documentation is available at /app/drakedocs/.
Read index.html for navigation.
Consult the installed package's help() and docstrings to confirm
signatures when documentation and runtime behavior differ.

The examples came from Drake's own multibody simulation and mathematical programming tutorials, together with its multibody optimization example. These teach API usage; they do not contain this biped task's answer.

The next Sol solution was much closer to code I wanted to work with. It separated model access into DrakeBiped, used descriptive attributes, and exposed methods whose units and array shapes were documented. For example, the inverse-dynamics method was explicit:

def inverse_dynamics(
    self, q: np.ndarray, v: np.ndarray, acceleration: np.ndarray
) -> np.ndarray:
    """Return M a + bias minus gravity and damping, shape (7,)."""
    self.set_state(q, v)
    self.external_forces.SetZero()
    self.plant.CalcForceElementsContribution(
        self.context, self.external_forces
    )
    return self.plant.CalcInverseDynamics(
        self.context, acceleration, self.external_forces
    )

That is a much nicer object to inspect in a debugger than a large fitting routine with model state hidden behind self.p and self.c.

The solution now included also included an independent validation script and a README. The README explained how to run the generator and the local validator. This becomes important with Flint and Qwen. Those models could not come up with the physics validator script themselves, however, including this validation script in the benchmark allowed them to solve it.

Asking for innovation

I tried to push the instructions a little further by adding innovation request to the prompt:

Don't be afraid to innovate control algorithms, for example an L1 total variation type smoothing to make the algorithm more robust. Mark all innovations with a comment. Also in your python code write descriptive comments on each function and complicated algorithm. Assume user is not too familiar with robots and drake and needs some handholding.

I had something like Boyd-style convex total-variation regularization in mind: penalize changes between neighboring force or torque samples while retaining the physical constraints. An L1 penalty can favor piecewise-constant signals and preserve transitions. However, Sol only provided one “innovation” comment in the result, and the comment is not clear. Either the models cannot really innovate or we need a better prompt.

"""Fit a constant hip offset to reduce single-contact moment error.


Innovation: a one-parameter centroidal fit accounts for leg mass and
swing inertia without slowing forward travel. Two model evaluations
give a local sensitivity; an unconstrained linear least-squares step
centers the required ground-reaction line on the stance foot. Final
forces and actuator limits are still handled by the constrained QP.
"""

The local-model test

With the task package improved, it was time to try Recursive's Flint, a local coding model positioned alongside the Qwen models I wanted to compare. Its model card describes a mixture-of-experts model with 39.8 billion total parameters and about 2.5 billion active per token. It can run at over 100 tokens per second on an M5 Macbook pro.

The first Flint run produced solution files. It did not pass the evaluation. The saved report records failures on the visible trajectories and both hidden configurations.

My approximate end-to-end timings were:

When the physics fails

I then gave Sol both submissions and asked it to explain the gap. Its diagnosis was specific:

Flint failed because its dynamics calculation omitted gravity and applied contact forces in the wrong direction, while its trajectory construction never enforced grounded, stationary stance feet. Its own validation reused those errors and reported a misleading near-zero residual.

That last sentence explains why “check your work” is such an incomplete instruction. If the generator and its checker share the same wrong equations, they can agree beautifully while the robot violates the physics.

The task already told Flint to keep stance feet stationary, include full dynamics, respect friction limits, and validate its output. Sol's analysis pointed out that repeating those requirements would probably achieve little. The higher-value addition was a public numerical checker that the model could actually run.

Here is the recommendation:

The highest-value change is to ship a public numerical checker and require its output. It should load the exported NPZ files, construct its own plant, and report every failed metric instead of stopping at the first assertion. Keep hidden configurations private; exposing the acceptance calculations does not expose a trajectory solution.

This is a more useful artifact to transfer between models than another paragraph asking them to be careful. The checker can tell Flint which physical requirements its current files violate. Reporting all failed metrics also gives it a fuller picture than stopping at the first assertion and making it discover the remaining failures one run at a time.

A checker worth transferring

Sol proposed this addition:

## Numerical development and validation


Establish physical feasibility before completing the implementation-style
review. Run the supplied public checker:


python /app/check_trajectories.py \
  --config /app/configs/visible.json --output /app/results

Sol managed to generate a useful validator for its solution, I am including this in the benchmark and will release the results of running Flint and Qwen with different quants in Part 2, including what would be the best local model to run.

The original Sol run established that the problem was solvable. Adding examples, documentation, and explicit implementation expectations made its answer much more useful to me. The first Flint run established that those improvements did not automatically transfer into a passing local solution, however, I think it is still possible to still run local models with more explicit instructions. Goal is to have Flint help me write some proprietary robot stuff that I don't want to send to OpenAI, so faking the benchmark a bit by making it easier is fine.