FOVI-Dex

Foveation enables efficient perceptive learning
of dexterous manipulation in simulation

No human demonstrations. No state-based policy pretraining or distillation.
Learn in simulation. Transfer zero-shot to the real robot.

Nicholas M. Blauch^†Nicolas Moenne-LoccozJayjun LeePatrick YinAsif RanaMohak BhardwajJessica YinSam MadyMarilyn ReebKarl Van WykNathan RatliffJae-Hyun JungRuth RosenholtzAnkur HandaGavriel State

Dex Manipulation Workshop · CoRL 202610th Conference on Robot Learning (CoRL 2026), Austin TX, USA.

Paper (PDF) ↗Explore the idea ↓ Watch the film

The idea, the learning, and the real robot. Download video ↗ Transcript ↗

Foveation enables efficient perceptive RL

Rendering and encoding dense images consume resources that could support more parallel learning. Foveation concentrates detail around gaze in a compact image while retaining peripheral context.

CMF a (°) 3.483

Uniform fixed high-resolution300 × 480

Foveated active128 × 128

Drag on the left image to move gaze. The left raster matches the high-resolution model; the active view samples the full source image. This illustrates camera sampling on a static simulation scene. Training renders each observation directly.

Same ≈45 GB budgetHigh resolution300 × 480 · uniform fixedCompact observation128 × 128 · foveated active

Environments / GPU2081 2486.0× as many

Total training FPS8 L40S GPUs≈4 685≈20 7604.4× the throughput

Visual tokens / frame570648.9× fewer

Throughput: full training runs on eight L40S GPUs, averaged over three seeds. Environment allocations use the same approximate 45 GB process-memory budget from single-Blackwell calibration.

More learning from the same compute budget

Foveated observations preserve detail where the robot looks, while leaving more compute for parallel learning. Compare the models on five-object clutter over the same 72 hours of training.

Foveated view centered on the sneaker, with enlarged detail near fixation and compressed peripheral context — The same five-object scene through each model’s camera. Active views fixate the sneaker for this illustration; these are native camera images, not policy rollouts. Dimensions are height × width.

Uniform low-resolution view aimed at the same sneaker; the target occupies fewer pixels — The same five-object scene through each model’s camera. Active views fixate the sneaker for this illustration; these are native camera images, not policy rollouts. Dimensions are height × width.

8 GPUs4 GPUs

Inspect at —

Zero-shot sim2real transfer

78%

Real-world success with foveated active vision.

The policy must distinguish a boot, laced boot, and sneaker among clutter, then grasp the requested object. It transfers directly from simulation, without real-world policy fine-tuning.

26%Uniform active

24%Uniform fixed, high-res

Trial success: foveated active 87.6% in full simulation, 90% in matched simulation, 78% on the robot; uniform active 53.75%, 52%, 26%; uniform fixed high-resolution 64.2%, 64%, 24%. — Foveated active vision leads in both domains. All models show a sim-to-real gap.

Per model: 2,000 simulation trials and 50 reviewed robot trials. “Sim (50)” uses the same planned layouts and targets as the robot trials and is a subset of the full simulation set. These are trial-success rates from one completed checkpoint per model, distinct from the training curves above.

Simulation Success

Reality Success

Ten foveated-policy layouts, with two trials per target and 8/10 robot successes. Simulations are fresh deployment reruns that stop on success, failure, or the time limit; they are separate from the scored benchmark above. Clips keep their native timing; the shorter clip holds its final frame.

Balance detail with context

We vary how strongly the image concentrates detail around gaze while keeping the compact image size, encoder, and training allocation fixed.

Intermediate foveation outperforms both the strongest concentration and nearly uniform sampling. The main model sits in this effective region; overlapping seed ranges do not establish a precise optimum.

Stronger foveationNearly uniform

Intermediate foveation reaches about 60% best ISR; the main-paper setting reaches about 62%. The strongest and near-uniform settings reach about 52–53%. Individual seeds and their full ranges are shown. — Connected diamonds: three-seed means, including the main model. Bars: full seed ranges. Small points: individual seeds.

Five-object training on four GPUs, compared over the same 72 hours. Each seed contributes its best recorded success-region occupancy (ISR) at the final curriculum level. ISR measures time in the success region, not episode success.

Rendering and policy architecture

Render where the robot looks

We use OmniLensDistortionLutAPI to attach forward and inverse lookup textures to the RTX camera. RTX traces rays directly into a compact foveated image. Gaze moves the high-detail center while retaining the wider scene. On the robot, software samples the same view from a fixed fisheye camera.

Learn from vision, language, and touch

A frozen, foveation-adapted visual encoder feeds a recurrent policy alongside the instruction, tactile feedback, and robot state. Reinforcement learning trains the motor policy, with supervised gaze learning and a privileged critic used only during training.

Paper architecture: direct foveated sensing and frozen visual features feed vision-language cross-attention, then a shared LSTM with tactile and proprioceptive inputs. Separate training-only critic and gaze teacher provide supervision. — Figure 1 · The paper’s complete architecture. Click to enlarge. Dashed orange paths are training-only supervision.

Explore the rendering geometry

Rotate gaze 0° Camera tilt 0° Apply camera bounds mask

01 / Forward LUT128 × 128 · RGB directions

Color encodes direction, not scene color.Texture coordinates → camera-space ray

02 / Ray → sceneRotate observation camera · drag or arrows

Selected ray Sparse subset of the ray bundleBoth LUTs stay fixed when gaze rotates.

03 / Foveated output128 × 128 · WebGL rendering

Circular + calibrated-camera masksSame ray samples, fixed transpose / flip

Output pixel · column, row—

Forward lookup · u, v—

Camera ray · x, y, z—

Selected sample—

The second LUT512 × 512 · RG image coordinates

Each position represents a direction.
Its stored color represents an image location.

Reverse lookup: direction to pixel

Tracing starts with a pixel and asks where it looks. Operations such as motion-vector computation ask the reverse: where does this direction land in the image?

We analytically invert the angular mapping and CMF warp, then sample that inverse into a second texture. Its square domain is an octahedral encoding of directions over the whole sphere. Its two stored values are the corresponding forward-texture coordinates.

Selected ray → direction address —Inverse LUT → recovered u, v —

Try it Move gaze: the rendered scene changes, but both lookup textures and this camera-space correspondence remain fixed. Switch to uniform sampling: both maps change together.

Interactive WebGL illustration of the captured five-object scene, Rizon arm and Sharpa hand. Joint positions and camera calibration match the film’s RTX capture; this browser view uses lit meshes and a cube-camera ray lookup, following the Alpha/NIST viewer. It omits the lab background; the film shows native RTX output.

FOVI-DexFoveation enables efficient perceptive learning of dexterous manipulation in simulation