Solving Vision Tasks with Simple Photoreceptors Instead of Cameras (2024)

3.1 Computational Model of a Photoreceptor

We define a photoreceptor (PR) as a visual sensor located at a specific point in space that integrates all incoming light from a specific field of view.Unlike complex, high-resolution lens-based sensors, a PR does not produce a high-resolution image but only provides a low-dimensional signal of the average light intensity (for each color channel).An analogy from nature is a single photoreceptor placed into a pigment tube, allowing light only from a given direction and field of view.In practice, such a sensor can be realized with a photodiode[12] by restricting its field of view using a casing that prevents it from receiving the light from the entire scene.The lensless compound eye realized by Kogos et al.[23] is analogous to a grid of PR sensors introduced below.Note that our framework, including the design optimization method in Sec.5, is agnostic to this particular choice of a simple visual sensor and can work with any other sensor that can be implemented in a simulator (e.g., see an optimized camera design in Tab.1).

Computationally, we implement this definition of the PR sensor as an averaging of the pixel values of a pinhole camera image.This approach allows us to model PRs in any simulator that provides a rendering engine without additional implementation costs.To read the signal $x$ of a PR with given extrinsic (position and orientation) and intrinsic (the field of view) parameters, we spawn a camera with the same parameters and render its corresponding image view $I\in\mathbb{R}^{3\times H\times W}$.Then, we average it’s signal along the spatial dimension to get the final value $x^{c}=\frac{1}{HW}\sum_{i,j}I^{c}_{p},\,x\in\mathbb{R}^{3}$, where $c\in\{1,2,3\}$ each stands for a channel and $p$ for the spatial pixel coordinate.In addition to a single PR, we also consider simple sensors using a grid of PRs of size $B\times B$ for low $B$ ($\leq 8$, in our experiments) that share the same position but have different adjacent fields of view.We implement such a grid by splitting the image into $B^{2}$ patches and averaging each of them spatially.See Fig.2-Center for the visualization.A single grid sensor enables the extraction of useful information (e.g., direction of motion) and makes a useful building block for a simple visual system.Moreover, using such a grid instead of $B^{2}$ independent photoreceptors results in a significant computational improvement when training in simulation, as it requires rendering only a single image instead of $B^{2}$ images.

3.2 Design of Visual Sensors

Design Space.We associate each sensor with its 7-dimensional design parameter vector $\theta_{i}=[\mathbf{x}_{i},\mathbf{y}_{i},\mathbf{z}_{i},\mathrm{yaw}_{i},$, where $(\mathbf{x}_{i},\mathbf{y}_{i},\mathbf{z}_{i})\in\mathbb{R}^{3}$ is the position in space, which we constrain to be on the agent’s body, $(\mathrm{yaw}_{i},\mathrm{pitch}_{i},\mathrm{roll}_{i})\in[0,2\pi]^{2}$ is the orientation, and $\mathrm{fov}_{i}\in[0,180]$ is the field of view.See Fig.2-Left for the visualization.In our experiments, we use $K\in\{2,4,8\}$ sensors and explore sensors represented by a single PR (a $1\times 1$ grid) and a grid of PRs of sizes $4\times 4$ and $8\times 8$.This results in a total of $KB^{2}$ PRs (ranging from 2 to 256 in our experiments) with the visual observation represented as $\{x_{kj}\}_{k,j=1,1}^{K,B^{2}}$.We also explore different designs for a camera sensor, in which case we have a single camera sensor and change its vector of parameters $\theta$.

4.1 Experimental Setting

We perform our experiments with the following active vision tasks.First, we consider two visual navigation tasks using the Habitat[54] simulator with 3D scans of real apartments from the Matterport3D[4] dataset.Our second set of tasks are continuous control tasks from the DeepMind Control (DMC) Suite[57], which we attempt to solve solely from the vision signal.Below, we provide a brief description of the experimental setting.For more detailed information, please refer toSupplementary Material.

Reinforcement learning background.We consider solving the active visual tasks as the decision-making processes using reinforcement learning in partially observable Markov decision processes (POMDP).At a state $s_{t}$, the agent receives an observation $o_{t}$ which cannot precisely determine the underlying state $s_{t}$.Then, the agent applies an action $a_{t}$, transits to the next state $s_{t+1}$, and receives a reward $r_{t}$.Let $\tau$ be the trajectory rollout provided to the agent by iterating the steps above, i.e., $\tau_{t}=(o_{t},a_{t},r_{t},o_{t+1},\cdots)$.Assume the agent computes the action $a_{t}$ with a control policy $\pi$, i.e., $a_{t}\sim\pi(\cdot|o_{t})$.We find the optimal control policy $\pi^{\star}$ by optimizing the expected return $\mathbb{E}_{\tau_{t}\sim\pi}[R(\tau_{t})]$, where the return is defined as $R(\tau_{t})=\Sigma_{i=0}\gamma^{i}r_{t+i}$ and $\gamma$ is the discount factor. In our experiments, we use Proximal Policy Optimization (PPO)[44] to optimize the control policy $\pi$.

Navigation in Habitat. We train navigation agents for PointGoalNav and TargetNav tasks in 3D replicas of real houses from the Matterport3D dataset[4] using the Habitat simulator [54].In PointGoalNav, the agent is randomly spawned in an environment and needs to navigate to a target coordinate.The agent observes an egocentric RGB view and its current position (coordinate) and orientation through the GPS+Compass sensor. We measure the performance of an agent using ‘SPL’ (Success weighted by Path Length)[1], which quantifies the performance relative to the optimal trajectory.In TargetNav, the agent is also equipped with the egocentric RGB sensor and the GPS+Compass sensor.In contrast to PointGoalNav, the agent does not receive a target coordinate but is asked to navigate to a green sphere that is randomly placed in the house at a height of 1.5m from the floor. This task, therefore, requires exploration and target identification by design. We therefore measure the performance of an agent using the success rate; that is, whether or not it finds the target sphere.

Continuous Control in DMC. We train continuous control agents using the MuJoCo simulator[58] on the DeepMind Control (DMC) benchmark[57].DMC provides a variety of continuous control tasks, including reaching, manipulation, locomotion, etc.In our context, we focus on learning the control policy that receives only visual information from either photoreceptors or a camera.We consider using the following six tasks: Reacher:Hard, Walker:Stand, Walker:Walk, Walker:Run, Finger:Spin, and Finger:Turn Easy (see Sec.E.3 and the original DMC video¹¹1https://www.youtube.com/watch?v=rAai4QzcYbs for a more detailed description of tasks.)

Architecture.Fig.2-right illustrates our control policy architecture.We model the policy $\pi_{w}(a_{t+1}\,|\,o_{t})$ using a simple three-layer Transformer[61] backbone that encodes the current observation $o_{t}$ from a visual sensor, camera or PRs, and GPS+Compass for navigation, and a small MLP that predicts the next action $a_{t+1}$.For the PR-based policy, we use $\{[p_{kj},\theta_{k},e_{j}]\}_{kj}$ as input tokens, where $\theta_{k}$ is the design vector of the $k^{\text{th}}$ grid sensor, and $e_{j}$ is the trainable positional embedding of the $j^{\text{th}}$ PR in the grid.For the camera-based policy, similar to ViT [11], we split the input image into 16x16 patches $\{x_{ij}\}$, flatten them, and add positional embeddings and the design vector $\theta$ to construct the final input tokens for the encoder: $\{[\overline{x}_{ij},\theta,e_{ij}]\}_{ij}$.Since we use a single camera, one can omit the design embedding $\theta$, but we keep it for consistency and as we use it in the design optimization method described in Sec.5.2.

4.2 Photoreceptors Achieve Performance Close to a Camera

Visual Navigation.Fig.3 shows that for both PointGoalNav and TargetNav tasks, even a simple visual sensor consisting of two 4$\times$4 PR grids (32 PRs) provides useful information, allowing the corresponding agent to outperform significantly an intelligent blind agent without a visual signal.The PR agents match the performance of the camera agent using the same shallow 3-layer Transformer encoder as for PRs, while having a visual signal bandwidth of only $\approx 1\%$ of that of the 128$\times$128 camera sensor.When compared to the camera agent using the ResNet-50 encoder, a default choice inthe literature (“gold standard”) for a fair comparison, PR agents still perform reasonably well. Fig.4 shows visualizations of the best-performing photoreceptor designs for both PointGoalNav and TargetNav tasks.

Fig.5 further demonstrates exemplar trajectories for each type of agent for each task.In the PointGoalNav task, where the target position is known and the main challenge is to navigate to it efficiently, we find that the PR agent follows more optimal trajectories (as measured by SPL) and effectively uses the visual signal to avoid collisions similar to the camera agent.In the TargetNav task, on the other hand, the target’s position is unknown and can only be identified visually, and it is important to explore a scene efficiently.Fig.5-right shows that both the PR and camera agents explore the scene much more efficiently, achieving a much higher success rate compared to the blind agent.

Continuous Control in DMC.Fig.6demonstrates that for most of the considered tasks, an agent with a few $1\times$1 photoreceptors significantly outperforms the blind agent and performs closely to the camera agent.Adding more sensors or increasing the grid sizes further improves the performance in most cases.However, we find that having a higher-resolution signal leads to a performance drop in some cases.We conjecture that the main reason is the suboptimal design.Indeed, as we find inSec.5, the design optimization algorithm fails to find an optimal design in some cases.

We note that, in general, the camera agent can achieve higher performance by utilizing, for example, more specialized and tuned reinforcement learning algorithms[71], data augmentation techniques[29], and/or training for more steps[56].In contrast, we employ a standard PPO[44] algorithm with a relatively short number of steps compared to[56] (5x).Note, however, that PR agents achieve reasonably good performance even compared to the maximum possible reward of 1000.In addition, we find that, for example, a longer training of the $K=4,1\times 1$ PR design leads to an improved reward of 930 compared to the original 605, suggesting that the performance of PR agents can also be improved further by specializing the learning algorithm or longer training.

5.1 Design is Important for the Effectiveness of Photoreceptors

How does the design of photoreceptorsinfluence the final performance of a control policy?Fig.7 shows that the performance of a poor design can drop drastically compared to the best design for the corresponding task.We find that some designs result in performance similar to that of a blind agent, suggesting that the visual signal does not provide any useful information.These results signify the importance of a design optimization algorithm to find good-performing designs automatically.

5.2 Computational Design via Joint Optimization

The design $\theta$ of the visual sensor(s), either PRs or camera, defines what observation the agent receives at each step, i.e., $o_{t}\triangleq o_{t}(\theta)$ and what design-specific control policy $\pi_{w}\triangleq\pi_{w}^{\theta}$ with what performance will be learned.To find the best design $\theta^{*}$, one would need to find the design that leads to training the best-performing design-specific control policy, resulting in a bi-level optimization problem$\max_{\theta}\,\max_{w}\mathbb{E}_{\tau\sim\pi_{w}}R(\tau)$.However, training the design-specific control policy $\pi_{w}^{\theta}$ in an inner loop for every new design would make this process prohibitively expensive.

Similar to [73], we, instead, cast this problem as joint optimization and amortize the costs of training multiple design-specific policies by training a single “generalist” policy that implements control for different designs.

Design Policy.We model the design policy as a Gaussian distribution over the design parameters $\pi_{\phi}(\theta)=\mathcal{N}(\theta\,|\,\mu,\mathrm{diag}(\sigma))$, where $\phi=(\mu,\sigma),\,\mu,\sigma\in\mathbb{R}^{K\times 7}$.After training, we use $\theta^{*}=\mu$ as the final optimal design.Note that while the distribution models each sensor independently, the final design of each sensor is informed of each other by virtue of being optimized together.

Generalist Control Policy.Training the generalist policy allows, in principle, to amortize the costs of training design-specific policies by “knowledge” and parameter sharing.Modeling and training such a policy that implements control for all possible designs can still have high memory and computing costs.Note, however, that Eq.1 only needs the policy that implements control for (likely) samples $\tilde{\theta}$ from the current design policy to provide it with a local direction for the improvement.Thus, we only need to train a local generalist policy and control the locality by the variance of the design policy, which in the limit of low variance allows approximating such a control policy with a linear dependency on design[32, 31].In practice, we initialize the variance $\sigma$ to allow training the local generalist policy that performs similarly to design-specific policies and, thus, provides a good signal for a design update.

5.3 Design Optimization Experiments

In this section, we demonstrate the effectiveness of the proposed design optimization method.We show that it can improve the performance of the initial design for both photoreceptors and camera sensors.

Design optimization improves the performance upon the initial design guess.Fig.9 shows that in the majority of cases, the design optimization method improves the initial random design.For DMC, we run the design optimization from two random initializations for each setting (task and $K$) and find that while it improves their performance, the best-performing computational design depends on the initialization (e.g., reaching the reward of 375 and 562 for two initializations for the Walker:Walk tasks and $K=2$).This suggests that the optimization landscape might contain multiple local optima, and improving the exploration abilities of the design optimization method is an important research direction for finding the best-performing designs.

Design optimization improves the “default” intuitive camera design.We also apply the design optimization to explore if we can improve the intuitive camera design used by default in the Habitat AI[54] simulator.Tab.1 shows that the agent using a computationally designed camera outperforms the one using the default intuitive design in both navigation tasks.

5.4 Intuitive Designs

In this section, we explore the effectiveness of human intuition in engineering well-performing designs of simple photoreceptor sensors.Since there is no single obvious way to design such visual sensors, we conducted a human survey to collect intuitive designs.We ask participants to design the design parameters of visual sensors in our defined design space.We ask participants to find the photoreceptors design parameters for a given morphology and task.We collect eight designs for the TargetNav visual navigation task and six designs for the Walker agent in DMC and evaluate them on Walk and Stand tasks.We provide a more detailed description of the survey setting in AppendixF.

Fig.10 shows that the best human intuition can provide well-performing designs, and computational design is among the best designs (or the best one) in most cases.We also find a high variance in the performance of different intuitive designs in all settings, signifying the importance of a computational approach to visual design.

5.5 Do designs transfer between tasks?

Optimizing a design for a given agent and task and deploying it can be a time-consuming process.One would wish to have a visual sensor design that can be optimized and deployed once and recycled for different downstream applications of the same robot without needing to repeat the process.We compare the performance of different designs we collect in this work (random, intuitive, and optimized) on two pairs of tasks for the same agent morphology.Fig.11 shows that a general trend suggests that one can optimize the design for one task and recycle it for another one.However, there are some designs that can underperform when transferred, especially for the Walker agent.This means that to find a transferable design during design optimization on one task, some form of regularization needs to be included in addition to the performance only to avoid such cases.

5.6 Evaluation in the Real World

To evaluate generalization and ensure that the strong performance of photoreceptors is not confined to simulators, we conducted the target navigation experiment (without access to GPS+Compass sensor) in a real-world setting.

We deployed a control policy using 64 PRs (less than 1% of the camera resolution) on a real robot as shown in Fig.12. It demonstrates impressive performance, successfully navigating to the target ball in an unknown room with no real-world training, relying solely on the low-resolution visual signal. The results can be seen at https://visual-morphology.epfl.ch/#real-world.

B.1 Does the task affect the computationally obtained design?

Fig.14 shows that even with the same initial random design, the proposed design optimization method converges to different designs for different tasks, namely, PointGoalNav and TargetNav.

B.2 Design optimisation method (computational design) uses available sensors well

We run a design ablation to show that the proposed design optimization method is optimizing the placement of sensors to maximize performance. We choose the simplest setting of K=2 grids of 4$\times$4 photoreceptors in the PointGoalNavigation setting. From the computational design with K=2, we create two designs of K=1 by picking one of the two sensor grids in the computational design. In Fig.15, we show a comparison between the original computational design and the ablated designs, showing that the design optimization utilizes the placement of the additional sensor grid effectively as neither of the two sensor grids alone performs well but together the performance is significantly boosted.

B.3 Photoreceptor-based agents can do Target Detection

In the TargetNav task, to confirm that photoreceptors can perform target detection, i.e., identify the green sphere and move towards it, we test the behavior and performance of the trained PR agent with a transparent sphere as the target instead of the green sphere. The comparison between an environment with the green sphere and the transparent sphere is shown in Fig.16. Fig.17 shows trajectory visualizations comparing the two settings: one with a green target sphere and the other with a transparent sphere in otherwise identical episodes. We see that initially, in both cases, the PR agent follows the same trajectory. In the episode with the green target, the PR agent is able to recognize it and move towards it, while in the episode with the transparent target, the agent does not see it (as expected) and continues searching for it. For a quantitative comparison, to demonstrate that the PR agent is indeed performing target detection and moving towards it to achieve success, rather than only conducting efficient exploration, we compare the agent’s success rate in both target settings. Tab.2 shows that the PR agent is indeed performing target detection, resulting in a much higher success rate.

B.4 Comparing importance of the different design space variables

We run additional experiments in order to compare the importance of the different design variables in the design space defined in the main manuscript, i.e., $[x_{i},y_{i},z_{i},\mathrm{yaw}_{i},\mathrm{pitch}_{i},\mathrm{fov}_{i}]$. For measuring the importance of a specific design variable, $x_{i}$ for example, starting from the computational design, for all PR grids, we set all design variables except $x_{i}$ to their initial values (before optimization). This comparison between different design axes is shown in Fig.18 for K=2 grids of 8$\times$8 PRs in PointGoalNavigation, which shows that the height $y_{i}$ and the pitch $\mathrm{pitch}_{i}$ design variables are the most important. We also show visualizations for each of these design change in Fig.20.

B.5 Distorting the computational design to analyse the success of design optimisation

C.1 Computational vs Random Design Visualisations

In Fig.21 and Fig.22, we show the initial random design and the corresponding computational design obtained using the proposed design optimization method for DeepMindControl and Fig.23 shows the same for the PointGoalNav task. The figures also show the improved reward corresponding to the computational design.

C.2 Intuitive Design Visualisations

Fig.24, shows visualizations for some of the intuitive designs collected using the survey described in AppendixF and their corresponding performance on the TargetNav task. This shows that the variance in the performance of intuitive designs is high as well.

E.1 PointGoal Navigation Setting

In PointGoalNav, the agent is randomly initialized in an environment and asked to navigate to a target point given relative to the start state.The episode ends if the agent calls the stop action. It succeeds in the episode if it stops within a 0.2-meter radius of the target point.

Observation Space.The agent has access to an idealized GPS+Compass input that provides its current position and rotation relative to the starting state.It also receives the relative position of the target point.In addition, the agent observes egocentric RGB views through its photoreceptors.

Action Space.The agent can execute 4 actions: move$\_$forward (0.25m), turn$\_$left (30^∘), turn$\_$right (30^∘), and stop.

Dataset. In PointGoalNav, we use the Matterport3D dataset [4] for training and testing.The training data (train split) contains 61 scenes with around 80k episodes per scene.The testing data (test split) contains 18 scenes unseen during training with 56 episodes per scene (in total 1008 episodes).

Evaluation Process.We evaluate our agent based on the Success weighted by Path Length (SPL) metric[1].One episode is counted as success only when the agent takes the stop action within 0.2 meters of the goal position within 500 steps.The SPL reported is the average across all episodes in the test split.

E.2 Target Navigation Setting

In TargetNav, the agent is spawned at a random ground location in an environment and needs to navigate to a target green sphere placed randomly in the scene.The radius of the target sphere is 0.5 m.The agent succeeds in the episode if it enters a circle of 0.8-meter radius around the target sphere center.

Observation Space.The agent receives its current position and rotation relative to the starting point and orientation from an idealized GPS+Compass sensor.Besides, the agent also observes egocentric RGB views through its photoreceptors.Compared to PointGoalNav, the agent does not receive the target position information.Thus, TargetNav focuses more on evaluating the agent’s ability to explore the environment.

Action Space.The agent has access to 3 actions: move$\_$forward (0.25m), turn$\_$left (30^∘), and turn$\_$right (30^∘).

Dataset. In TargetNav, we construct our training and testing data from the PointGoalNav dataset in Matterport3D scenes [4].We randomly sample 10 scenes from the train split of the PointGoalNav setting.We use the same 18 test scenes as in the test split of the PointGoalNav dataset.We use all the episodes of the PointGoalNav dataset for the scenes chosen above.For each episode, we add the green sphere at a height of 1.5 meters above the ground at the goal position.

Reward and Training Process.TargetNav uses the same reward design and training process as in the PointGoalNav setting described in Sec.E.1.

Evaluation Process.In TargetNav, we evaluate our agent based on Success.One episode is successful only when the agent enters the circle of 0.8-meter radius around the target sphere center within 1500 steps.We do not require the agent to call the stop action because we want to focus more on evaluating the PR agent’s exploration ability using its onboard PRs.The Success reported is the average across all episodes in the testing data.

E.3 Continuous Control in DeepMind Control Suite

Observation Space.The agent only receives egocentric views from its onboard photoreceptors.Since visual observation does not provide full information about the state (e.g., velocities), we use the standard practice of stacking three consecutive frames and using them as input to the control policy.

Training Process.To maintain consistency with the navigation experiments, we use the PPO[44] learning algorithm with the following hyperparameters.We use $\gamma=0.99$ for reward discounting, GAE $\lambda=0.95$, and $\epsilon=0.2$ for the PPO clipping loss.We train the control policies using half of a V100 or A100 GPU.During training, we have 10 parallel environments while each environment collects 10000 steps of experience per rollout.Here each rollout collection step is equivalent to 0.1M simulation steps.We split the collected rollouts into mini-batches of 1000 and perform 4 epochs of PPO updates.We show the specific number of simulation steps for each task inTab.3.We use Adam[22] optimization method with a learning rate $0.0001$.

E.4 Design Optimization

Navigation Tasks. In PointGoalNav and TargetNav, we use a Gaussian distribution as the design policy $\pi_{\phi}(\theta)=\mathcal{N}(\theta\,|\,\mu,\mathrm{diag}(\sigma))$, where $\phi=(\mu,\sigma),\,\mu,\sigma\in\mathbb{R}^{K\times 7}$ is the mean and standard deviation, $\theta$ is the design parameter, and $K$ is the number of PRs.We initialize the mean to be a zero vector $\mu=\mathbb{0}^{K\times 7}$ and set the initial standard deviation to be 0.2, i.e., $\sigma=0.2\times\mathbb{1}^{K\times 7}$. We separate the design optimization into two stages: Frozen Stage and Update Stage.

Frozen Stage: In this phase, the design policy is “frozen”, and we only train the control policy to act as a local generalist. At the beginning of each episode, the design parameter $\theta$ is sampled from the frozen design policy, $\theta\sim\pi_{\phi}(\cdot)$, thereby altering the robot design. As outlined in Section 5.2 of the main paper, the local generalist policy is optimized to manage control within a specific range of design parameters centered around the mean $\mu$. The scope of this range is determined by the standard deviation $\sigma$; a larger $\sigma$ allows the policy to handle a wider variety of design parameters, while a smaller $\sigma$ limits it to a narrower range. During this stage, the control policy undergoes training for 20k rollout collection steps (153M simulation steps).

Update Stage: During this phase, both the design policy and the control policy undergo training simultaneously. We initiate updates to the design policy every 100 rollout collection steps (6M simulation steps) following each update of the control policy every 400 rollout collection steps (3.1M simulation steps).

When updating the design policy, we maintain the control policy in a frozen state. The objective is to align the design policy with the distribution of returns across the design parameter space. Instead of using returns as the primary objective, which tends to favor longer episodes due to accumulated rewards, we adopt SoftSPL (Soft Success Weighted by Path Length) [10] as the objective function. SoftSPL balances episode efficiency and success more effectively by considering the minimum distance achieved to the target, thus providing a denser and smoother reward landscape for optimizing the design policy.

Concurrently, when updating the control policy, we freeze the design policy. This approach ensures that the control policy adapts to manage the agent within the local parameters defined by the updated design policy. Each rollout consists of 64 steps to facilitate more frequent updates of the policies.

This dual updating strategy allows for comprehensive refinement of both the design and control policies, ensuring robust performance across various task scenarios.

DeepMind Control Suite.We use the same Gaussian design policy as in navigation setting. The total number of simulation steps dedicated to design optimization for each task is detailed in Tab.3. During the Update Stage, we utilize return as the objective function for both the control and design policies. To adapt the control policy to the changing design policy, we update the design policy every single rollout collection step (0.1M simulation steps), following each training session of the control policy every 8 rollout collection steps (0.8 million simulation steps).

E.5 Network Architecture

Fig.26 provides a detailed architecture of the transformer encoder used in both settings, i.e. Navigation and DMC. Each photoreceptor token consists of the RGB triple, the position embedding based on the position of the photoreceptor on the grid and the design parameters of the grid. This forms the encoder input to the control policy $\pi$.In the PointGoalNav and TargetNav navigation tasks, the policy is a 2-layer LSTM, while in DeepMindControl (DMC), we stack the last 3 frames’ encodings as input to the policy.

F.1 Visual Navigation Tasks

To compare the performance of our design optimization algorithm with that of human engineers, we designed and conducted a survey. The survey asked participants to optimize the position, orientation, and field of view (FoV) of visual sensors on a robot, aiming to enable it to complete the TargetNav task as quickly as possible.

F.2 Continuous Control Tasks from the DMC Suite

The DMC benchmark uses the MuJoCo simulator[58], which is challenging to deploy within a browser due to the specialized format used to define scenes and the agent’s morphology. Therefore, we ask participants to sketch their placement design based on a rendered image depicting the agent’s morphology and the environment. Given a sketch, we implement the design inside the simulator, show it to the participant, and update it based on their feedback until convergence, i.e., when the participant agrees that the design corresponds to their intended placement.

Due to the demanding nature of this process, we collect designs for only two tasks within one domain from six participants for continuous control tasks. Each participant provides one design for both tasks (Walker and Walker), which share the Walker domain. We provide a description of these two tasks similar to that in the original paper introducing the DMC benchmark[57].