Observation and Action Spaces¶

This is an introduction to the three standard observations and the action space of Flatland.

Action Space¶

Flatland is a railway simulation. Thus the actions of an agent are strongly limited to the railway network. This means that in many cases not all actions are valid. The possible actions of an agent are

0 Do Nothing: If the agent is moving it continues moving, if it is stopped it stays stopped
1 Deviate Left: If the agent is at a switch with a transition to its left, the agent will chose th eleft path. Otherwise the action has no effect. If the agent is stopped, this action will start agent movement again if allowed by the transitions.
2 Go Forward: This action will start the agent when stopped. This will move the agent forward and chose the go straight direction at switches.
3 Deviate Right: Exactly the same as deviate left but for right turns.
4 Stop: This action causes the agent to stop.

Observation Spaces¶

In the Flatland environment we have included three basic observations to get started. The figure below illustrates the observation range of the different basic observation: Global, Local Grid and Local Tree.

Global Observation¶

Gives a global observation of the entire rail environment.

The observation is composed of the following elements:

transition map array with dimensions (env.height, env.width, 16), assuming 16 bits encoding of transitions.
Two 2D arrays (map_height, map_width, 2) containing respectively the position of the given agent target and the positions of the other agents’ targets.
A 3D array (map_height, map_width, 8) with the first 4 channels containing the one hot encoding of the direction of the given agent and the second 4 channels containing the positions of the other agents at their position coordinates.

We encourage you to enhance this observation with any layer you think might help solve the problem. It would also be possible to construct a global observation for a super agent that controls all agents at once.

Local Grid Observation¶

Gives a local observation of the rail environment around the agent. The observation is composed of the following elements:

transition map array of the local environment around the given agent, with dimensions (2*view_radius + 1, 2*view_radius + 1, 16), assuming 16 bits encoding of transitions.
Two 2D arrays (2*view_radius + 1, 2*view_radius + 1, 2) containing respectively, if they are in the agent’s vision range, its target position, the positions of the other targets.
A 3D array (2*view_radius + 1, 2*view_radius + 1, 4) containing the one hot encoding of directions of the other agents at their position coordinates, if they are in the agent’s vision range.
A 4 elements array with one hot encoding of the direction.

Be aware that this observation does not contain any clues about target location if target is out of range. Thus navigation on maps where the radius of the observation does not guarantee a visible target at all times will become very difficult. We encourage you to come up with creative ways to overcome this problem. In the tree observation below we introduce the concept of distance maps.

Tree Observation¶

The tree observation is built by exploiting the graph structure of the railway network. The observation is generated by spanning a 4 branched tree from the current position of the agent. Each branch follows the allowed transitions (backward branch only allowed at dead-ends) until a cell with multiple allowed transitions is reached. Here the information gathered along the branch is stored as a node in the tree. The figure below illustrates how the tree observation is built:

From Agent location probe all 4 directions (L:Blue, F:Green, R:Purple, B:Red) starting with left and start branches when transition is allowed.
1. For each branch walk along the allowed transition until you reach a dead-end, switch or the target destination.
2. Create a node and fill in the node information as stated below.
3. If max depth of tree is not reached and there are possible transitions, start new branches and repeat the steps above.
Fill up all non existing branches with -infinity such that tree size is invariant to the number of possible transitions at branching points.

Note that we always start with the left branch according to the agent orientation. Thus the tree observation is independent of the NESW orientation of cells, and only considers the transitions relative to the agent’s orientation.

The colors in the figure bellow illustrate what branch the cell belongs to. If there are multiple colors in a cell, this cell is visited by different branches of the tree observation. The right side of the figure shows the resulting tree of the railway network on the left. Cross means no branch was built. If a node has no children it was either a terminal node (dead-end, max depth reached or no transition possible). A circle indicates a node filled with the corresponding information stated below in Node Information.

Node Information¶

Each node is filled with information gathered along the path to the node. Currently each node contains 9 features:

1: if own target lies on the explored branch the current distance from the agent in number of cells is stored.
2: if another agent’s target is detected, the distance in number of cells from the current agent position is stored.
3: if another agent is detected, the distance in number of cells from the current agent position is stored.
4: possible conflict detected (This only works when we use a predictor and will not be important in this tutorial)
5: if an unusable switch (for the agent) is detected we store the distance. An unusable switch is a switch where the agent does not have any choice of path, but other agents coming from different directions might.
6: This feature stores the distance (in number of cells) to the next node (e.g. switch or target or dead-end)
7: minimum remaining travel distance from this node to the agent’s target given the direction of the agent if this path is chosen
8: agent in the same direction found on path to node
- n = number of agents present in the same direction (possible future use: number of other agents in the same direction in this branch)
- 0 = no agent present in the same direction
9: agent in the opposite direction on path to node
- n = number of agents present in the opposite direction to the observing agent
- 0 = no agent present in other direction to the observing agent