VirtualHome

About
VirtualHome is a multi-agent platform to simulate activities in a household. Agents are represented as humanoid avatars, which can interact with the environment through high-level instructions. You can use VirtualHome to render videos of human activities, or train agents to perform complex tasks. VirtualHome also includes a Knowledge Base, providing instructions to perform a large set of activities.

Try the Simulator Check out the Docs Explore RobotHow

Features

Interactive Objects

You can move and change the state of the objects in the environment. The environment can be programatically generated.

Humanoid Agents

We provide multiple human avatars as agents, with Path-planning, IK and Animations to generate realistic actions.

Multi-Modal

You can record activities across different modalities and using multiple camera views simulatenously.

Multi-Agent

You can add multiple agents in the environment, and control them concurrently.

Activity Knowledge Base

We have collected a knowledge base of activities people do at home. For every activity, we have descriptions of different ways to perform it, and a program that describes how to execute it. The knowledge base also contains information about states of objects in the environment, and typical locations of those objects.

Check out RobotHow for an overview of the knowledge base of activity programs.

VirtualHome in Research

Here is some of the research using VirtualHome

VirtualHome: Simulating Household Activities via Programs

Presents VirtualHome and proposes a model to predict programs from videos or descriptions. The predicted program is finetuned using RL to be executable in the simulator.

X. Puig*, K. Ra*, M. Boben*, J. Li, T. Wang, S. Fidler, A. Torralba.
Link paper
In Proc. Computer Vision and Pattern Recognition (CVPR), 2018.

Synthesizing Environment-Aware Activities via Activity Sketches

Represents activities via Sketches. Proposes a model to generate programs from sketches such that they are consistent with a target environment.

A. Liao*, X. Puig*, M. Boben, A. Torralba, S.Fidler.
Link paper
In Proc. Computer Vision and Pattern Recognition (CVPR), 2019.

Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration

Presents a challenge where an agent has to infer a human-like agent's goals, and find the best way to achieve the goals efficiently.

X. Puig, T. Shu, S. Li, Z. Wang, A. Liao, J. Tenenbaum, S. Fidler, A. Torralba.
Link paper
In Proc. International Conference on Learning Representations

Team