tianshou.trainer¶

On-policy¶

class tianshou.trainer.OnpolicyTrainer(policy: ~tianshou.policy.base.BasePolicy, train_collector: ~tianshou.data.collector.Collector, test_collector: ~typing.Optional[~tianshou.data.collector.Collector], max_epoch: int, step_per_epoch: int, repeat_per_collect: int, episode_per_test: int, batch_size: int, step_per_collect: ~typing.Optional[int] = None, episode_per_collect: ~typing.Optional[int] = None, train_fn: ~typing.Optional[~typing.Callable[[int, int], None]] = None, test_fn: ~typing.Optional[~typing.Callable[[int, ~typing.Optional[int]], None]] = None, stop_fn: ~typing.Optional[~typing.Callable[[float], bool]] = None, save_best_fn: ~typing.Optional[~typing.Callable[[~tianshou.policy.base.BasePolicy], None]] = None, save_checkpoint_fn: ~typing.Optional[~typing.Callable[[int, int, int], str]] = None, resume_from_log: bool = False, reward_metric: ~typing.Optional[~typing.Callable[[~numpy.ndarray], ~numpy.ndarray]] = None, logger: ~tianshou.utils.logger.base.BaseLogger = <tianshou.utils.logger.base.LazyLogger object>, verbose: bool = True, show_progress: bool = True, test_in_train: bool = True, **kwargs: ~typing.Any)[source]¶

Bases: BaseTrainer

An iterator class for onpolicy trainer procedure.

Returns an iterator that yields a 3-tuple (epoch, stats, info) of train results on every epoch.

The “step” in onpolicy trainer means an environment step (a.k.a. transition).

Example usage:
trainer = OnpolicyTrainer(...)
for epoch, epoch_stat, info in trainer:
    print("Epoch:", epoch)
    print(epoch_stat)
    print(info)
    do_something_with_policy()
    query_something_about_policy()
    make_a_plot_with(epoch_stat)
    display(info)
epoch int: the epoch number

epoch_stat dict: a large collection of metrics of the current epoch

info dict: result returned from gather_info()

You can even iterate on several trainers at the same time:
trainer1 = OnpolicyTrainer(...)
trainer2 = OnpolicyTrainer(...)
for result1, result2, ... in zip(trainer1, trainer2, ...):
    compare_results(result1, result2, ...)

Parameters

policy – an instance of the BasePolicy class.
train_collector (Collector) – the collector used for training.
test_collector (Collector) – the collector used for testing. If it’s None, then no testing will be performed.
max_epoch (int) – the maximum number of epochs for training. The training process might be finished before reaching max_epoch if stop_fn is set.
step_per_epoch (int) – the number of transitions collected per epoch.
repeat_per_collect (int) – the number of repeat time for policy learning, for example, set it to 2 means the policy needs to learn each given batch data twice.
episode_per_test (int) – the number of episodes for one policy evaluation.
batch_size (int) – the batch size of sample data, which is going to feed in the policy network.
step_per_collect (int) – the number of transitions the collector would collect before the network update, i.e., trainer will collect “step_per_collect” transitions and do some policy network update repeatedly in each epoch.
episode_per_collect (int) – the number of episodes the collector would collect before the network update, i.e., trainer will collect “episode_per_collect” episodes and do some policy network update repeatedly in each epoch.
train_fn (function) – a hook called at the beginning of training in each epoch. It can be used to perform custom additional operations, with the signature f(num_epoch: int, step_idx: int) -> None.
test_fn (function) – a hook called at the beginning of testing in each epoch. It can be used to perform custom additional operations, with the signature f(num_epoch: int, step_idx: int) -> None.
save_best_fn (function) – a hook called when the undiscounted average mean reward in evaluation phase gets better, with the signature f(policy: BasePolicy) -> None. It was save_fn previously.
save_checkpoint_fn (function) – a function to save training process and return the saved checkpoint path, with the signature f(epoch: int, env_step: int, gradient_step: int) -> str; you can save whatever you want.
resume_from_log (bool) – resume env_step/gradient_step and other metadata from existing tensorboard log. Default to False.
stop_fn (function) – a function with signature f(mean_rewards: float) -> bool, receives the average undiscounted returns of the testing result, returns a boolean which indicates whether reaching the goal.
reward_metric (function) – a function with signature f(rewards: np.ndarray with shape (num_episode, agent_num)) -> np.ndarray with shape (num_episode,), used in multi-agent RL. We need to return a single scalar for each episode’s result to monitor training in the multi-agent RL setting. This function specifies what is the desired metric, e.g., the reward of agent 1 or the average reward over all agents.
logger (BaseLogger) – A logger that logs statistics during training/testing/updating. Default to a logger that doesn’t log anything.
verbose (bool) – whether to print the information. Default to True.
show_progress (bool) – whether to display a progress bar when training. Default to True.
test_in_train (bool) – whether to test in the training phase. Default to True.

Note

Only either one of step_per_collect and episode_per_collect can be specified.

policy_update_fn(data: Dict[str, Any], result: Optional[Dict[str, Any]] = None) → None[source]¶: Perform one on-policy update.

tianshou.trainer.onpolicy_trainer(*args, **kwargs) → Dict[str, Union[float, str]][source]¶

Wrapper for OnpolicyTrainer run method.

It is identical to OnpolicyTrainer(...).run().

Returns: See gather_info().

tianshou.trainer.onpolicy_trainer_iter¶: alias of OnpolicyTrainer

Off-policy¶

class tianshou.trainer.OffpolicyTrainer(policy: ~tianshou.policy.base.BasePolicy, train_collector: ~tianshou.data.collector.Collector, test_collector: ~typing.Optional[~tianshou.data.collector.Collector], max_epoch: int, step_per_epoch: int, step_per_collect: int, episode_per_test: int, batch_size: int, update_per_step: ~typing.Union[int, float] = 1, train_fn: ~typing.Optional[~typing.Callable[[int, int], None]] = None, test_fn: ~typing.Optional[~typing.Callable[[int, ~typing.Optional[int]], None]] = None, stop_fn: ~typing.Optional[~typing.Callable[[float], bool]] = None, save_best_fn: ~typing.Optional[~typing.Callable[[~tianshou.policy.base.BasePolicy], None]] = None, save_checkpoint_fn: ~typing.Optional[~typing.Callable[[int, int, int], str]] = None, resume_from_log: bool = False, reward_metric: ~typing.Optional[~typing.Callable[[~numpy.ndarray], ~numpy.ndarray]] = None, logger: ~tianshou.utils.logger.base.BaseLogger = <tianshou.utils.logger.base.LazyLogger object>, verbose: bool = True, show_progress: bool = True, test_in_train: bool = True, **kwargs: ~typing.Any)[source]¶

Bases: BaseTrainer

An iterator class for offpolicy trainer procedure.

Returns an iterator that yields a 3-tuple (epoch, stats, info) of train results on every epoch.

The “step” in offpolicy trainer means an environment step (a.k.a. transition).

Example usage:
trainer = OffpolicyTrainer(...)
for epoch, epoch_stat, info in trainer:
    print("Epoch:", epoch)
    print(epoch_stat)
    print(info)
    do_something_with_policy()
    query_something_about_policy()
    make_a_plot_with(epoch_stat)
    display(info)
epoch int: the epoch number

epoch_stat dict: a large collection of metrics of the current epoch

info dict: result returned from gather_info()

You can even iterate on several trainers at the same time:
trainer1 = OffpolicyTrainer(...)
trainer2 = OffpolicyTrainer(...)
for result1, result2, ... in zip(trainer1, trainer2, ...):
    compare_results(result1, result2, ...)

Parameters

policy – an instance of the BasePolicy class.
train_collector (Collector) – the collector used for training.
test_collector (Collector) – the collector used for testing. If it’s None, then no testing will be performed.
max_epoch (int) – the maximum number of epochs for training. The training process might be finished before reaching max_epoch if stop_fn is set.
step_per_epoch (int) – the number of transitions collected per epoch.
step_per_collect (int) – the number of transitions the collector would collect before the network update, i.e., trainer will collect “step_per_collect” transitions and do some policy network update repeatedly in each epoch.
episode_per_test – the number of episodes for one policy evaluation.
batch_size (int) – the batch size of sample data, which is going to feed in the policy network.
update_per_step (int/float) – the number of times the policy network would be updated per transition after (step_per_collect) transitions are collected, e.g., if update_per_step set to 0.3, and step_per_collect is 256 , policy will be updated round(256 * 0.3 = 76.8) = 77 times after 256 transitions are collected by the collector. Default to 1.
train_fn (function) – a hook called at the beginning of training in each epoch. It can be used to perform custom additional operations, with the signature f(num_epoch: int, step_idx: int) -> None.
test_fn (function) – a hook called at the beginning of testing in each epoch. It can be used to perform custom additional operations, with the signature f(num_epoch: int, step_idx: int) -> None.
save_best_fn (function) – a hook called when the undiscounted average mean reward in evaluation phase gets better, with the signature f(policy: BasePolicy) -> None. It was save_fn previously.
save_checkpoint_fn (function) – a function to save training process and return the saved checkpoint path, with the signature f(epoch: int, env_step: int, gradient_step: int) -> str; you can save whatever you want.
resume_from_log (bool) – resume env_step/gradient_step and other metadata from existing tensorboard log. Default to False.
stop_fn (function) – a function with signature f(mean_rewards: float) -> bool, receives the average undiscounted returns of the testing result, returns a boolean which indicates whether reaching the goal.
reward_metric (function) – a function with signature f(rewards: np.ndarray with shape (num_episode, agent_num)) -> np.ndarray with shape (num_episode,), used in multi-agent RL. We need to return a single scalar for each episode’s result to monitor training in the multi-agent RL setting. This function specifies what is the desired metric, e.g., the reward of agent 1 or the average reward over all agents.
logger (BaseLogger) – A logger that logs statistics during training/testing/updating. Default to a logger that doesn’t log anything.
verbose (bool) – whether to print the information. Default to True.
show_progress (bool) – whether to display a progress bar when training. Default to True.
test_in_train (bool) – whether to test in the training phase. Default to True.

policy_update_fn(data: Dict[str, Any], result: Dict[str, Any]) → None[source]¶: Perform off-policy updates.

tianshou.trainer.offpolicy_trainer(*args, **kwargs) → Dict[str, Union[float, str]][source]¶

Wrapper for OffPolicyTrainer run method.

It is identical to OffpolicyTrainer(...).run().

Returns: See gather_info().

tianshou.trainer.offpolicy_trainer_iter¶: alias of OffpolicyTrainer

Offline¶

class tianshou.trainer.OfflineTrainer(policy: ~tianshou.policy.base.BasePolicy, buffer: ~tianshou.data.buffer.base.ReplayBuffer, test_collector: ~typing.Optional[~tianshou.data.collector.Collector], max_epoch: int, update_per_epoch: int, episode_per_test: int, batch_size: int, test_fn: ~typing.Optional[~typing.Callable[[int, ~typing.Optional[int]], None]] = None, stop_fn: ~typing.Optional[~typing.Callable[[float], bool]] = None, save_best_fn: ~typing.Optional[~typing.Callable[[~tianshou.policy.base.BasePolicy], None]] = None, save_checkpoint_fn: ~typing.Optional[~typing.Callable[[int, int, int], str]] = None, resume_from_log: bool = False, reward_metric: ~typing.Optional[~typing.Callable[[~numpy.ndarray], ~numpy.ndarray]] = None, logger: ~tianshou.utils.logger.base.BaseLogger = <tianshou.utils.logger.base.LazyLogger object>, verbose: bool = True, show_progress: bool = True, **kwargs: ~typing.Any)[source]¶

Bases: BaseTrainer

An iterator class for offline trainer procedure.

Returns an iterator that yields a 3-tuple (epoch, stats, info) of train results on every epoch.

The “step” in offline trainer means a gradient step.

Example usage:
trainer = OfflineTrainer(...)
for epoch, epoch_stat, info in trainer:
    print("Epoch:", epoch)
    print(epoch_stat)
    print(info)
    do_something_with_policy()
    query_something_about_policy()
    make_a_plot_with(epoch_stat)
    display(info)
epoch int: the epoch number

epoch_stat dict: a large collection of metrics of the current epoch

info dict: result returned from gather_info()

You can even iterate on several trainers at the same time:
trainer1 = OfflineTrainer(...)
trainer2 = OfflineTrainer(...)
for result1, result2, ... in zip(trainer1, trainer2, ...):
    compare_results(result1, result2, ...)

Parameters

policy – an instance of the BasePolicy class.
buffer – an instance of the ReplayBuffer class. This buffer must be populated with experiences for offline RL.
test_collector (Collector) – the collector used for testing. If it’s None, then no testing will be performed.
max_epoch (int) – the maximum number of epochs for training. The training process might be finished before reaching max_epoch if stop_fn is set.
update_per_epoch (int) – the number of policy network updates, so-called gradient steps, per epoch.
episode_per_test – the number of episodes for one policy evaluation.
batch_size (int) – the batch size of sample data, which is going to feed in the policy network.
test_fn (function) – a hook called at the beginning of testing in each epoch. It can be used to perform custom additional operations, with the signature f(num_epoch: int, step_idx: int) -> None.
save_best_fn (function) – a hook called when the undiscounted average mean reward in evaluation phase gets better, with the signature f(policy: BasePolicy) -> None. It was save_fn previously.
save_checkpoint_fn (function) – a function to save training process and return the saved checkpoint path, with the signature f(epoch: int, env_step: int, gradient_step: int) -> str; you can save whatever you want. Because offline-RL doesn’t have env_step, the env_step is always 0 here.
resume_from_log (bool) – resume gradient_step and other metadata from existing tensorboard log. Default to False.
stop_fn (function) – a function with signature f(mean_rewards: float) -> bool, receives the average undiscounted returns of the testing result, returns a boolean which indicates whether reaching the goal.
reward_metric (function) – a function with signature f(rewards: np.ndarray with shape (num_episode, agent_num)) -> np.ndarray with shape (num_episode,), used in multi-agent RL. We need to return a single scalar for each episode’s result to monitor training in the multi-agent RL setting. This function specifies what is the desired metric, e.g., the reward of agent 1 or the average reward over all agents.
logger (BaseLogger) – A logger that logs statistics during updating/testing. Default to a logger that doesn’t log anything.
verbose (bool) – whether to print the information. Default to True.
show_progress (bool) – whether to display a progress bar when training. Default to True.

policy_update_fn(data: Dict[str, Any], result: Optional[Dict[str, Any]] = None) → None[source]¶: Perform one off-line policy update.

tianshou.trainer.offline_trainer(*args, **kwargs) → Dict[str, Union[float, str]][source]¶

Wrapper for offline_trainer run method.

It is identical to OfflineTrainer(...).run().

Returns: See gather_info().

tianshou.trainer.offline_trainer_iter¶: alias of OfflineTrainer

utils¶

tianshou.trainer.test_episode(policy: BasePolicy, collector: Collector, test_fn: Optional[Callable[[int, Optional[int]], None]], epoch: int, n_episode: int, logger: Optional[BaseLogger] = None, global_step: Optional[int] = None, reward_metric: Optional[Callable[[ndarray], ndarray]] = None) → Dict[str, Any][source]¶: A simple wrapper of testing policy in collector.

tianshou.trainer.gather_info(start_time: float, train_collector: Optional[Collector], test_collector: Optional[Collector], best_reward: float, best_reward_std: float) → Dict[str, Union[float, str]][source]¶

A simple wrapper of gathering information from collectors.

Returns

A dictionary with the following keys:

train_step the total collected step of training collector;
train_episode the total collected episode of training collector;
train_time/collector the time for collecting transitions in the training collector;
train_time/model the time for training models;
train_speed the speed of training (env_step per second);
test_step the total collected step of test collector;
test_episode the total collected episode of test collector;
test_time the time for testing;
test_speed the speed of testing (env_step per second);
best_reward the best reward over the test results;
duration the total elapsed time.