continuous#

Source code: tianshou/utils/net/continuous.py

class Actor(preprocess_net: Module, action_shape: Sequence[int] | int | int64, hidden_sizes: Sequence[int] = (), max_action: float = 1.0, device: str | int | device = 'cpu', preprocess_net_output_dim: int | None = None)[source]#

Simple actor network.

It will create an actor operated in continuous action space with structure of preprocess_net —> action_shape.

Parameters:

  • preprocess_net – a self-defined preprocess_net which output a flattened hidden state.

  • action_shape – a sequence of int for the shape of action.

  • hidden_sizes – a sequence of int for constructing the MLP after preprocess_net. Default to empty sequence (where the MLP now contains only a single linear layer).

  • max_action – the scale for the final action logits. Default to 1.

  • preprocess_net_output_dim – the output dimension of preprocess_net.

For advanced usage (how to customize the network), please refer to Build the Network.

See also

Please refer to Net as an instance of how preprocess_net is suggested to be defined.

forward(obs: ndarray | Tensor, state: Any = None, info: dict[str, Any] | None = None) → tuple[Tensor, Any][source]#

Mapping: obs -> logits -> action.

get_output_dim() → int[source]#
get_preprocess_net() → Module[source]#
class ActorProb(preprocess_net: Module, action_shape: Sequence[int] | int | int64, hidden_sizes: Sequence[int] = (), max_action: float = 1.0, device: str | int | device = 'cpu', unbounded: bool = False, conditioned_sigma: bool = False, preprocess_net_output_dim: int | None = None)[source]#

Simple actor network (output with a Gauss distribution).

Parameters:

  • preprocess_net – a self-defined preprocess_net which output a flattened hidden state.

  • action_shape – a sequence of int for the shape of action.

  • hidden_sizes – a sequence of int for constructing the MLP after preprocess_net. Default to empty sequence (where the MLP now contains only a single linear layer).

  • max_action – the scale for the final action logits. Default to 1.

  • unbounded – whether to apply tanh activation on final logits. Default to False.

  • conditioned_sigma – True when sigma is calculated from the input, False when sigma is an independent parameter. Default to False.

  • preprocess_net_output_dim – the output dimension of preprocess_net.

For advanced usage (how to customize the network), please refer to Build the Network.

See also

Please refer to Net as an instance of how preprocess_net is suggested to be defined.

forward(obs: ndarray | Tensor, state: Any = None, info: dict[str, Any] | None = None) → tuple[tuple[Tensor, Tensor], Any][source]#

Mapping: obs -> logits -> (mu, sigma).

get_output_dim() → int[source]#
get_preprocess_net() → Module[source]#
class Critic(preprocess_net: ~torch.nn.modules.module.Module, hidden_sizes: ~collections.abc.Sequence[int] = (), device: str | int | ~torch.device = 'cpu', preprocess_net_output_dim: int | None = None, linear_layer: ~collections.abc.Callable[[int, int], ~torch.nn.modules.module.Module] = <class 'torch.nn.modules.linear.Linear'>, flatten_input: bool = True)[source]#

Simple critic network.

It will create an actor operated in continuous action space with structure of preprocess_net —> 1(q value).

Parameters:

  • preprocess_net – a self-defined preprocess_net which output a flattened hidden state.

  • hidden_sizes – a sequence of int for constructing the MLP after preprocess_net. Default to empty sequence (where the MLP now contains only a single linear layer).

  • preprocess_net_output_dim – the output dimension of preprocess_net.

  • linear_layer – use this module as linear layer. Default to nn.Linear.

  • flatten_input – whether to flatten input data for the last layer. Default to True.

For advanced usage (how to customize the network), please refer to Build the Network.

See also

Please refer to Net as an instance of how preprocess_net is suggested to be defined.

forward(obs: ndarray | Tensor, act: ndarray | Tensor | None = None, info: dict[str, Any] | None = None) → Tensor[source]#

Mapping: (s, a) -> logits -> Q(s, a).

class Perturbation(preprocess_net: Module, max_action: float, device: str | int | device = 'cpu', phi: float = 0.05)[source]#

Implementation of perturbation network in BCQ algorithm.

Given a state and action, it can generate perturbed action.

Parameters:

  • preprocess_net – a self-defined preprocess_net which output a flattened hidden state.

  • max_action – the maximum value of each dimension of action.

  • device – which device to create this model on. Default to cpu.

  • phi – max perturbation parameter for BCQ. Default to 0.05.

For advanced usage (how to customize the network), please refer to Build the Network.

See also

You can refer to examples/offline/offline_bcq.py to see how to use it.

forward(state: Tensor, action: Tensor) → Tensor[source]#

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class RecurrentActorProb(layer_num: int, state_shape: Sequence[int], action_shape: Sequence[int], hidden_layer_size: int = 128, max_action: float = 1.0, device: str | int | device = 'cpu', unbounded: bool = False, conditioned_sigma: bool = False)[source]#

Recurrent version of ActorProb.

For advanced usage (how to customize the network), please refer to Build the Network.

forward(obs: ndarray | Tensor, state: dict[str, Tensor] | None = None, info: dict[str, Any] | None = None) → tuple[tuple[Tensor, Tensor], dict[str, Tensor]][source]#

Almost the same as Recurrent.

class RecurrentCritic(layer_num: int, state_shape: Sequence[int], action_shape: Sequence[int] = [0], device: str | int | device = 'cpu', hidden_layer_size: int = 128)[source]#

Recurrent version of Critic.

For advanced usage (how to customize the network), please refer to Build the Network.

forward(obs: ndarray | Tensor, act: ndarray | Tensor | None = None, info: dict[str, Any] | None = None) → Tensor[source]#

Almost the same as Recurrent.

class VAE(encoder: Module, decoder: Module, hidden_dim: int, latent_dim: int, max_action: float, device: str | device = 'cpu')[source]#

Implementation of VAE.

It models the distribution of action. Given a state, it can generate actions similar to those in batch. It is used in BCQ algorithm.

Parameters:

  • encoder – the encoder in VAE. Its input_dim must be state_dim + action_dim, and output_dim must be hidden_dim.

  • decoder – the decoder in VAE. Its input_dim must be state_dim + latent_dim, and output_dim must be action_dim.

  • hidden_dim – the size of the last linear-layer in encoder.

  • latent_dim – the size of latent layer.

  • max_action – the maximum value of each dimension of action.

  • device – which device to create this model on. Default to “cpu”.

For advanced usage (how to customize the network), please refer to Build the Network.

See also

You can refer to examples/offline/offline_bcq.py to see how to use it.

decode(state: Tensor, latent_z: Tensor | None = None) → Tensor[source]#
forward(state: Tensor, action: Tensor) → tuple[Tensor, Tensor, Tensor][source]#

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.