tianshou.env¶
VectorEnv¶
BaseVectorEnv¶
-
class
tianshou.env.
BaseVectorEnv
(env_fns: List[Callable[], gym.core.Env]], worker_fn: Callable[[Callable[], gym.core.Env]], tianshou.env.worker.base.EnvWorker], wait_num: Optional[int] = None, timeout: Optional[float] = None, norm_obs: bool = False, obs_rms: Optional[tianshou.utils.statistics.RunningMeanStd] = None, update_obs_rms: bool = True)[source]¶ Bases:
object
Base class for vectorized environments wrapper.
Usage:
env_num = 8 envs = DummyVectorEnv([lambda: gym.make(task) for _ in range(env_num)]) assert len(envs) == env_num
It accepts a list of environment generators. In other words, an environment generator
efn
of a specific task means thatefn()
returns the environment of the given task, for example,gym.make(task)
.All of the VectorEnv must inherit
BaseVectorEnv
. Here are some other usages:envs.seed(2) # which is equal to the next line envs.seed([2, 3, 4, 5, 6, 7, 8, 9]) # set specific seed for each env obs = envs.reset() # reset all environments obs = envs.reset([0, 5, 7]) # reset 3 specific environments obs, rew, done, info = envs.step([1] * 8) # step synchronously envs.render() # render all environments envs.close() # close all environments
Warning
If you use your own environment, please make sure the
seed
method is set up properly, e.g.,def seed(self, seed): np.random.seed(seed)
Otherwise, the outputs of these envs may be the same with each other.
- Parameters
env_fns – a list of callable envs,
env_fns[i]()
generates the i-th env.worker_fn – a callable worker,
worker_fn(env_fns[i])
generates a worker which contains the i-th env.wait_num (int) – use in asynchronous simulation if the time cost of
env.step
varies with time and synchronously waiting for all environments to finish a step is time-wasting. In that case, we can return whenwait_num
environments finish a step and keep on simulation in these environments. IfNone
, asynchronous simulation is disabled; else,1 <= wait_num <= env_num
.timeout (float) – use in asynchronous simulation same as above, in each vectorized step it only deal with those environments spending time within
timeout
seconds.norm_obs (bool) – Whether to track mean/std of data and normalize observation on return. For now, observation normalization only support observation of type np.ndarray.
obs_rms – class to track mean&std of observation. If not given, it will initialize a new one. Usually in envs that is used to evaluate algorithm, obs_rms should be passed in. Default to None.
update_obs_rms (bool) – Whether to update obs_rms. Default to True.
-
get_env_attr
(key: str, id: Optional[Union[int, List[int], numpy.ndarray]] = None) → List[Any][source]¶ Get an attribute from the underlying environments.
If id is an int, retrieve the attribute denoted by key from the environment underlying the worker at index id. The result is returned as a list with one element. Otherwise, retrieve the attribute for all workers at indices id and return a list that is ordered correspondingly to id.
- Parameters
key (str) – The key of the desired attribute.
id – Indice(s) of the desired worker(s). Default to None for all env_id.
- Return list
The list of environment attributes.
-
set_env_attr
(key: str, value: Any, id: Optional[Union[int, List[int], numpy.ndarray]] = None) → None[source]¶ Set an attribute in the underlying environments.
If id is an int, set the attribute denoted by key from the environment underlying the worker at index id to value. Otherwise, set the attribute for all workers at indices id.
- Parameters
key (str) – The key of the desired attribute.
value (Any) – The new value of the attribute.
id – Indice(s) of the desired worker(s). Default to None for all env_id.
-
reset
(id: Optional[Union[int, List[int], numpy.ndarray]] = None) → numpy.ndarray[source]¶ Reset the state of some envs and return initial observations.
If id is None, reset the state of all the environments and return initial observations, otherwise reset the specific environments with the given id, either an int or a list.
-
step
(action: numpy.ndarray, id: Optional[Union[int, List[int], numpy.ndarray]] = None) → Tuple[numpy.ndarray, numpy.ndarray, numpy.ndarray, numpy.ndarray][source]¶ Run one timestep of some environments’ dynamics.
If id is None, run one timestep of all the environments’ dynamics; otherwise run one timestep for some environments with given id, either an int or a list. When the end of episode is reached, you are responsible for calling reset(id) to reset this environment’s state.
Accept a batch of action and return a tuple (batch_obs, batch_rew, batch_done, batch_info) in numpy format.
- Parameters
action (numpy.ndarray) – a batch of action provided by the agent.
- Returns
A tuple including four items:
obs
a numpy.ndarray, the agent’s observation of current environmentsrew
a numpy.ndarray, the amount of rewards returned after previous actionsdone
a numpy.ndarray, whether these episodes have ended, in which case further step() calls will return undefined resultsinfo
a numpy.ndarray, contains auxiliary diagnostic information (helpful for debugging, and sometimes learning)
For the async simulation:
Provide the given action to the environments. The action sequence should correspond to the
id
argument, and theid
argument should be a subset of theenv_id
in the last returnedinfo
(initially they are env_ids of all the environments). If action is None, fetch unfinished step() calls instead.
-
seed
(seed: Optional[Union[int, List[int]]] = None) → List[Optional[List[int]]][source]¶ Set the seed for all environments.
Accept
None
, an int (which will extendi
to[i, i + 1, i + 2, ...]
) or a list.- Returns
The list of seeds used in this env’s random number generators. The first value in the list should be the “main” seed, or the value which a reproducer pass to “seed”.
DummyVectorEnv¶
-
class
tianshou.env.
DummyVectorEnv
(env_fns: List[Callable[], gym.core.Env]], **kwargs: Any)[source]¶ Bases:
tianshou.env.venvs.BaseVectorEnv
Dummy vectorized environment wrapper, implemented in for-loop.
See also
Please refer to
BaseVectorEnv
for other APIs’ usage.
SubprocVectorEnv¶
-
class
tianshou.env.
SubprocVectorEnv
(env_fns: List[Callable[], gym.core.Env]], **kwargs: Any)[source]¶ Bases:
tianshou.env.venvs.BaseVectorEnv
Vectorized environment wrapper based on subprocess.
See also
Please refer to
BaseVectorEnv
for other APIs’ usage.
ShmemVectorEnv¶
-
class
tianshou.env.
ShmemVectorEnv
(env_fns: List[Callable[], gym.core.Env]], **kwargs: Any)[source]¶ Bases:
tianshou.env.venvs.BaseVectorEnv
Optimized SubprocVectorEnv with shared buffers to exchange observations.
ShmemVectorEnv has exactly the same API as SubprocVectorEnv.
See also
Please refer to
BaseVectorEnv
for other APIs’ usage.
RayVectorEnv¶
-
class
tianshou.env.
RayVectorEnv
(env_fns: List[Callable[], gym.core.Env]], **kwargs: Any)[source]¶ Bases:
tianshou.env.venvs.BaseVectorEnv
Vectorized environment wrapper based on ray.
This is a choice to run distributed environments in a cluster.
See also
Please refer to
BaseVectorEnv
for other APIs’ usage.
Worker¶
EnvWorker¶
-
class
tianshou.env.worker.
EnvWorker
(env_fn: Callable[], gym.core.Env])[source]¶ Bases:
abc.ABC
An abstract worker for an environment.
-
send
(action: Optional[numpy.ndarray]) → None[source]¶ Send action signal to low-level worker.
When action is None, it indicates sending “reset” signal; otherwise it indicates “step” signal. The paired return value from “recv” function is determined by such kind of different signal.
-
recv
() → Union[Tuple[numpy.ndarray, numpy.ndarray, numpy.ndarray, numpy.ndarray], numpy.ndarray][source]¶ Receive result from low-level worker.
If the last “send” function sends a NULL action, it only returns a single observation; otherwise it returns a tuple of (obs, rew, done, info).
-
step
(action: numpy.ndarray) → Tuple[numpy.ndarray, numpy.ndarray, numpy.ndarray, numpy.ndarray][source]¶ Perform one timestep of the environment’s dynamic.
“send” and “recv” are coupled in sync simulation, so users only call “step” function. But they can be called separately in async simulation, i.e. someone calls “send” first, and calls “recv” later.
-
DummyEnvWorker¶
-
class
tianshou.env.worker.
DummyEnvWorker
(env_fn: Callable[], gym.core.Env])[source]¶ Bases:
tianshou.env.worker.base.EnvWorker
Dummy worker used in sequential vector environments.
-
static
wait
(workers: List[tianshou.env.worker.dummy.DummyEnvWorker], wait_num: int, timeout: Optional[float] = None) → List[tianshou.env.worker.dummy.DummyEnvWorker][source]¶ Given a list of workers, return those ready ones.
-
static
SubprocEnvWorker¶
-
class
tianshou.env.worker.
SubprocEnvWorker
(env_fn: Callable[], gym.core.Env], share_memory: bool = False)[source]¶ Bases:
tianshou.env.worker.base.EnvWorker
Subprocess worker used in SubprocVectorEnv and ShmemVectorEnv.
-
static
wait
(workers: List[tianshou.env.worker.subproc.SubprocEnvWorker], wait_num: int, timeout: Optional[float] = None) → List[tianshou.env.worker.subproc.SubprocEnvWorker][source]¶ Given a list of workers, return those ready ones.
-
send
(action: Optional[numpy.ndarray]) → None[source]¶ Send action signal to low-level worker.
When action is None, it indicates sending “reset” signal; otherwise it indicates “step” signal. The paired return value from “recv” function is determined by such kind of different signal.
-
static
RayEnvWorker¶
-
class
tianshou.env.worker.
RayEnvWorker
(env_fn: Callable[], gym.core.Env])[source]¶ Bases:
tianshou.env.worker.base.EnvWorker
Ray worker used in RayVectorEnv.
-
static
wait
(workers: List[tianshou.env.worker.ray.RayEnvWorker], wait_num: int, timeout: Optional[float] = None) → List[tianshou.env.worker.ray.RayEnvWorker][source]¶ Given a list of workers, return those ready ones.
-
send
(action: Optional[numpy.ndarray]) → None[source]¶ Send action signal to low-level worker.
When action is None, it indicates sending “reset” signal; otherwise it indicates “step” signal. The paired return value from “recv” function is determined by such kind of different signal.
-
static