Public Interfaces

ROSBridgeLib

We use the following template from github for the ROSBridge: https://github.com/michaeljenkin/unityros.

Basically the ROSBridge consists of three different parts:

  1. ROSBridgeWebSocketConnection
  2. ROSBridgeMsg
  3. ROSBridge Actor aka Subscriber, Publisher and Service

ROSBridgeWebSocketConnection

class ROSBridgeLib::ROSBridgeWebSocketConnection

This class handles the connection with the external ROS world, deserializing json messages into appropriate instances of packets and messages.

This class also provides a mechanism for having the callback’s exectued on the rendering thread. (Remember, Unity has a single rendering thread, so we want to do all of the communications stuff away from that.

The one other clever thing that is done here is that we only keep 1 (the most recent!) copy of each message type that comes along.

Version History 3.1 - changed methods to start with an upper case letter to be more consistent with c# style. 3.0 - modification from hand crafted version 2.0

Author
Michael Jenkin, Robert Codd-Downey and Andrew Speers
Version
3.1

Public Functions

ROSBridgeLib.ROSBridgeWebSocketConnection.ROSBridgeWebSocketConnection(string host, int port)

Make a connection to a host/port.

This does not actually start the connection, use Connect to do that.

void ROSBridgeLib.ROSBridgeWebSocketConnection.AddServiceResponse(Type serviceResponse)

Add a service response callback to this connection.

void ROSBridgeLib.ROSBridgeWebSocketConnection.RemoveServiceResponse(Type serviceResponse)

Not Implemented!

void ROSBridgeLib.ROSBridgeWebSocketConnection.AddSubscriber(Type subscriber)

Add a subscriber callback to this connection.

There can be many subscribers.

void ROSBridgeLib.ROSBridgeWebSocketConnection.RemoveSubscriber(Type subscriber)

Removes a subscriber and disconnects him if the connection is already running.

void ROSBridgeLib.ROSBridgeWebSocketConnection.AddPublisher(Type publisher)

Add a publisher to this connection.

There can be many publishers.

void ROSBridgeLib.ROSBridgeWebSocketConnection.RemovePublisher(Type publisher)

Removes a publisher from the connection.

Disconnects the publisher if connection is already running.

void ROSBridgeLib.ROSBridgeWebSocketConnection.Connect()

Connect to the remote ros environment.

void ROSBridgeLib.ROSBridgeWebSocketConnection.Disconnect()

Disconnect from the remote ros environment.

Private Functions

void ROSBridgeLib.ROSBridgeWebSocketConnection.CheckConnection()

Checks if the connection is open and prints a message in that case.

Does not do anything in the other case as Connect() has an exception in this case and it wont be printed anyway.

void ROSBridgeLib.ROSBridgeWebSocketConnection.OnError(object sender, WebSocketSharp.ErrorEventArgs e)

Error handler method which simply prints the error message.

ROSBridgeMsg

class ROSBridgeLib::ROSBridgeMsg

This (mostly empty) class is the parent class for all RosBridgeMsg’s (the actual message) from ROS.

As the message can be empty....

This could be omitted I suppose, but it is retained here as (i) it nicely parallels the ROSBRidgePacket class which encapsulates the top of the ROSBridge messages which are not empty, and (ii) someday ROS may actually define a minimal message.

Version History 3.1 - changed methods to start with an upper case letter to be more consistent with c# style. 3.0 - modification from hand crafted version 2.0

Author
Michael Jenkin, Robert Codd-Downey and Andrew Speers
Version
3.1

Subclassed by ROSBridgeLib.custom_msgs.DebugMsg, ROSBridgeLib.custom_msgs.DurationMsg, ROSBridgeLib.custom_msgs.ErrorMsg, ROSBridgeLib.custom_msgs.ExternalForceMsg, ROSBridgeLib.custom_msgs.ExternalJointMsg, ROSBridgeLib.custom_msgs.FloatArrayMsg, ROSBridgeLib.custom_msgs.ForceMsg, ROSBridgeLib.custom_msgs.InfoMsg, ROSBridgeLib.custom_msgs.LinkMsg, ROSBridgeLib.custom_msgs.ModelMsg, ROSBridgeLib.custom_msgs.PositionCustomMsg, ROSBridgeLib.custom_msgs.RoboyPoseMsg, ROSBridgeLib.custom_msgs.RoboyPositionMsg, ROSBridgeLib.custom_msgs.StringArrayMsg, ROSBridgeLib.custom_msgs.TendonInitializationMsg, ROSBridgeLib.custom_msgs.TendonUpdateMsg, ROSBridgeLib.custom_msgs.WarningMsg, ROSBridgeLib.geometry_msgs.PointMsg, ROSBridgeLib.geometry_msgs.PoseMsg, ROSBridgeLib.geometry_msgs.QuaternionMsg, ROSBridgeLib.geometry_msgs.TwistMsg, ROSBridgeLib.geometry_msgs.Vector3Msg, ROSBridgeLib.sensor_msgs.CompressedImageMsg, ROSBridgeLib.sensor_msgs.ImageMsg, ROSBridgeLib.std_msgs.BoolMsg, ROSBridgeLib.std_msgs.ColorRGBAMsg, ROSBridgeLib.std_msgs.HeaderMsg, ROSBridgeLib.std_msgs.Int32Msg, ROSBridgeLib.std_msgs.Int32MultiArrayMsg, ROSBridgeLib.std_msgs.Int64Msg, ROSBridgeLib.std_msgs.Int64MultiArrayMsg, ROSBridgeLib.std_msgs.Int8Msg, ROSBridgeLib.std_msgs.Int8MultiArrayMsg, ROSBridgeLib.std_msgs.MultiArrayDimensionMsg, ROSBridgeLib.std_msgs.MultiArrayLayoutMsg, ROSBridgeLib.std_msgs.StringMsg, ROSBridgeLib.std_msgs.TimeMsg, ROSBridgeLib.std_msgs.UInt16Msg, ROSBridgeLib.std_msgs.UInt16MultiArrayMsg, ROSBridgeLib.std_msgs.UInt32Msg, ROSBridgeLib.std_msgs.UInt32MultiArrayMsg, ROSBridgeLib.std_msgs.Uint64Msg, ROSBridgeLib.std_msgs.UInt64MultiArrayMsg, ROSBridgeLib.std_msgs.UInt8Msg, ROSBridgeLib.std_msgs.UInt8MultiArrayMsg, ROSBridgeLib.turtlesim.ColorMsg, ROSBridgeLib.turtlesim.PoseMsg, ROSBridgeLib.turtlesim.VelocityMsg

As every type of ROSBridgeMsg should derive from this class, here is an example how an actual implementation looks like.

class ROSBridgeLib::turtlesim::PoseMsg

Define a turtle pose message.

This has been hand-crafted from the corresponding turtle message file.

Version History 3.1 - changed methods to start with an upper case letter to be more consistent with c# style. 3.0 - modification from hand crafted version 2.0

Inherits from ROSBridgeLib.ROSBridgeMsg

Public Functions

ROSBridgeLib.turtlesim.PoseMsg.PoseMsg(JSONNode msg)

This constructor is called when you receive a message from the ROSBridge.

Parameters
  • msg

ROSBridgeLib.turtlesim.PoseMsg.PoseMsg(float x, float y, float theta, float linear_velocity, float angular_velocity)

This constuctor can be used to construct a message in Unity and send it over the ROSBridge.

Parameters
  • x
  • y
  • theta
  • linear_velocity
  • angular_velocity

override string ROSBridgeLib.turtlesim.PoseMsg.ToYAMLString()

You need this function to send a message over the ROSBridge to the desired ROS node as YAML is the standard format for this.

Return

Public Static Functions

static string ROSBridgeLib.turtlesim.PoseMsg.GetMessageType()

This is called when you send the message over the ROSBridge.

It must be equal to the type of the input of the receiving node.

Return

Managers

RoboyManager

class RoboyManager

Roboymanager has the task to adjust roboys state depending on the ROS messages.

In summary it does the following:

-# receive pose messages to adjust roboy pose.
-# subscribe to the external force event and forward the message to the simulation.
-# send a service call for a world reset.
-# FUTURE: receive motor msg and forward it to the according motors.

Inherits from Singleton< RoboyManager >

Public Functions

void RoboyManager.InitializeRoboyParts()

Initializes the roboy parts with a random count of motors => WILL BE CHANGED IN THE FUTURE, for now just a template

void RoboyManager.ReceiveMessage(RoboyPoseMsg msg)

Main function to receive messages from ROSBridge.

Adjusts the roboy pose and the motors values (future).

Parameters
  • msg: JSON msg containing roboy pose.

void RoboyManager.ReceiveExternalForce(RoboyPart roboyPart, Vector3 position, Vector3 force, int duration)

Sends a message to the simulation to apply an external force at a certain position.

Parameters
  • roboyPart: The roboypart where the force should be applied.
  • position: The relative position of the force to the roboypart.
  • force: The direction and the amount of force relative to roboypart.
  • duration: The duration for which the force should be applied.

Property

property RoboyManager::Roboy

Public variable so that all classes can access the roboy object.

property RoboyManager::RoboyParts

Public variable for the dictionary with all roboyparts, used to adjust pose and motor values

Private Functions

void RoboyManager.Awake()

Initialize ROSBridge and roboy parts

void RoboyManager.Update()

Run ROSBridge

void RoboyManager.drawTendons()

Test function to draw tendons.

For now draws only random lines. TEMPLATE!

void RoboyManager.adjustPose(string name, RoboyPoseMsg msg)

Adjusts roboy pose for all parts with the values from the simulation.

Parameters
  • msg: JSON msg containing the roboy pose.

void RoboyManager.getRoboy()

Searches for roboy via the “Roboy” tag.

void RoboyManager.getRoboyParts(string name)

Searches for roboy and all roboy parts.

Private Members

Transform RoboyManager.m_Roboy

Transform of roboy with all roboy parts as child objects

RoboyPoseMsg RoboyManager.m_RoboyPoseMessage

Pose message of roboy in ou r build in class

Dictionary<string, RoboyPart> RoboyManager.m_RoboyParts = new Dictionary<string, RoboyPart>()

Dictionary with all roboyparts, used to adjust pose and motor values

InputManager

class InputManager

InputManager holds a reference of every tool.

On top of that it listens to button events from these tools and forwards touchpad input to the respective classes.

Inherits from Singleton< InputManager >

Public Types

enum TouchpadStatus

Possible touchpad positions.

Values:

Right
Left
Top
Bottom
None

Public Functions

void InputManager.Initialize(List< ControllerTool > toolList)

Initialize all tools.

void InputManager.OnChangeGUITool(object sender, ClickedEventArgs e)

Changes view mode when the user presses the side button on the controller.

Parameters
  • sender
  • e

void InputManager.OnChangeTool(object sender, ClickedEventArgs e)

Changes the tool when the user presses the side button on the controller.

Parameters
  • sender
  • e

void InputManager.GetTouchpadInput(object sender, ClickedEventArgs e)

Retrives the touchpad input of the tool controller and updates the values.

Parameters
  • sender
  • e

Property

property InputManager::GUI_Controller

Public GUIController reference.

property InputManager::View_Controller

Public ViewController reference.

property InputManager::ModelSpawn_Controller

Public ModelSpawnController reference.

property InputManager::Selector_Tool

Public SelectorTool reference.

property InputManager::ShootingTool

Public ShootingTool reference.

property InputManager::TimeTool

Public TimeTool reference.

property InputManager::SelectorTool_TouchpadStatus

Touchpad status of the controller where selector tool is attached to.

property InputManager::GUIController_TouchpadStatus

Touchpad status of the controller where gui controller tool is attached to.

Private Functions

void InputManager.Update()

Calls the ray cast from the selector tool if it is active.

void InputManager.setTools(List< ControllerTool > toolList)

Set all tools depending on their type to the respective variable.

Parameters
  • toolList

IEnumerator InputManager.initControllersCoroutine()

Initializes all controllers and tools.

Return

Private Members

SelectorTool InputManager.m_SelectorTool

Private SelectorTool reference.

Is serialized so it can be dragged in the editor.

ShootingTool InputManager.m_ShootingTool

Private ShootingTool reference.

Is serialized so it can be dragged in the editor.

TimeTool InputManager.m_TimeTool

Private TimeTool reference.

Is serialized so it can be dragged in the editor.

HandTool InputManager.m_HandTool

Private HandTool reference.

Is serialized so it can be dragged in the editor.

GUIController InputManager.m_GUIController

Private GUIController reference.

Is serialized so it can be dragged in the editor.

ViewController InputManager.m_ViewController

Private GUIController reference.

Is serialized so it can be dragged in the editor.

ModelSpawnController InputManager.m_ModelSpawnController

Private ModelSpawn reference.

Is serialized so it can be dragged in the editor.

SelectionWheel InputManager.m_ToolWheel

Selection wheel to select tools.

SelectionWheel InputManager.m_GUIWheel

Selection wheel to select different GUI modes.

bool InputManager.m_Initialized = false

Controllers initialized or not.

ModeManager

class ModeManager

ModeManager holds a reference of every active mode and provides function to switch between them.

This includes:

  • Current tool: ShootingTool, SelectionTool etc.
  • Current view mode: singe vs. comparison
  • Current GUI mode: selection vs. GUI panels
  • Current panel mode: motorforce, motorvoltage etc.

Inherits from Singleton< ModeManager >

Public Types

enum Viewmode

We change between Single view where we can choose only one objet at a time and comparison view with three maximum objects at a time.

Values:

Single
Comparison
enum Panelmode

Describes the different modes for panel visualization.

Values:

Motor_Force
Motor_Voltage
Motor_Current
Energy_Consumption
Tendon_Forces
enum GUIViewerMode

Enum for current GUI mode.

Values:

Selection

Panel of all selectable mesh parts of the current roboy model

MotorValues

Graphs for each motor for each type of information

enum ToolMode

SelectorTool: Select roboy meshes.

ShooterTool: Shoot projectiles at roboy. TimeTool: Reverse/stop time.

Values:

SelectorTool
ShootingTool
TimeTool
HandTool
Undefined

Public Functions

void ModeManager.ChangeViewMode()

Changes between single and comparison view.

void ModeManager.ChangeGUIViewerMode()

Switches between selection and panels GUI mode.

void ModeManager.ChangeToolMode()

Switches between all tools.

void ModeManager.ChangeToolMode(ToolMode mode)

Changes the tool mode based on the enum to the new one and turns off the old tool.

Parameters
  • mode

void ModeManager.ChangePanelModeNext()

Changes the panel mode to the next one based on the order in the enum defintion.

void ModeManager.ChangePanelModePrevious()

Changes the panel mode to the previous one based on the order in the enum defintion.

void ModeManager.ResetPanelMode()

Resets current panel mode to MotorForce.

Property

property ModeManager::CurrentViewmode

Current view mode, READ ONLY.

property ModeManager::CurrentPanelmode

Current panel mode, READ ONLY.

property ModeManager::CurrentGUIViewerMode

Current GUIViewer mode, READ ONLY.

property ModeManager::CurrentToolMode

Current Tool mode, READ ONLY.

property ModeManager::CurrentGUIMode

Current GUI Mode.

READ ONLY.

Private Functions

void ModeManager.changeToolStatus(ToolMode tool, bool state)

Changes the tool based on the enum to the new state.

Parameters
  • tool
  • state

Private Members

Viewmode ModeManager.m_CurrentViewmode = Viewmode.Comparison

Private variable for current view mode.

Panelmode ModeManager.m_CurrentPanelmode = Panelmode.Motor_Force

Private variable for current panel mode.

GUIViewerMode ModeManager.m_CurrentGUIViewerMode = GUIViewerMode.Selection

Private variable for current GUIViewer mode.

SpawnViewerMode ModeManager.m_CurrentSpawnViewerMode = SpawnViewerMode.Insert

Private variable for the current mode of the model spawn controller.

ToolMode ModeManager.m_CurrentToolMode = ToolMode.SelectorTool

Private variable for current Tool mode.

GUIMode ModeManager.m_CurrentGUIMode = GUIMode.GUIViewer

Current view mode of the GUI tools

SelectorManager

class SelectorManager

SelectorManager is responsible to hold references of all selected roboy parts and the corresponding UI elements.

Inherits from Singleton< SelectorManager >

Public Functions

void SelectorManager.AddSelectedObject(SelectableObject obj)

Adds the roboy part to selected objects.

Parameters
  • obj: SelectableObject component of the roboy part.

void SelectorManager.RemoveSelectedObject(SelectableObject obj)

Removes the roboy part from the selected objects.

Parameters
  • obj: SelectableObject component of the roboy part.

void SelectorManager.ResetSelectedObjects()

Resets all roboy parts to default state and empties the selected objects list.

Public Members

int SelectorManager.RoboyUIElementsCount = 13

TEMPORARY VARIABLE TO CHECK HOW MANY UI ELEMENTS ARE INITIALIZED

Property

property SelectorManager::UI_Elements

Property which returns a dictionary of all UI elements in the SelectionPanel.

property SelectorManager::SelectedParts

Reference of all currently selected roboy parts.

property SelectorManager::MaximumSelectableObjects

Integer to switch between single mode selection and normal mode collection.

Private Functions

IEnumerator SelectorManager.Start()

Initializes all variables.

Return

Private Members

Transform SelectorManager.m_Roboy

Transform of roboy model.

List<SelectableObject> SelectorManager.m_RoboyParts = new List<SelectableObject>()

List of SelectableObject components of all roboy parts.

List<SelectableObject> SelectorManager.m_SelectedParts = new List<SelectableObject>()

List of SelectableObject components of all selected parts.

int SelectorManager.m_MaximumSelectableObjects = 3

Maximum cound of selectable objects in multiple selection mode.

int SelectorManager.m_CurrentMaximumSelectedObjects = 3

Current count of maximum selectable objects.

Dictionary<string, GameObject> SelectorManager.m_UI_Elements = new Dictionary<string, GameObject>()

Private reference to all UI elements.

BeRoboyManager

class BeRoboyManager

BeRoboymanager has different tasks to do:

  1. Keep track of user movement and translate roboy when in specific view modes
  2. Convert received images into textures which can then be rendered on screen
  3. Send tracking messages over the rosbridge to gazebo/ real roboy

Inherits from Singleton< BeRoboyManager >

Public Functions

void BeRoboyManager.ReceiveZedMessage(ImageMsg image)

Primary function to receive image (zed) messages from ROSBridge.

Renders the received images.

Parameters
  • msg: JSON msg containing roboy pose.

void BeRoboyManager.ReceiveSimMessage(ImageMsg image)

Primary function to receive image (simulation) messages from ROSBridge.

Renders the received images.

Parameters
  • msg: JSON msg containing roboy pose.

void BeRoboyManager.ReceiveExternalJoint(List< string > jointNames, List< float > angles)

Function to publish ExternalJoint messages via ROS.

Parameters
  • jointNames
  • angles

void BeRoboyManager.ReceivePosition(Vector3 pos)

Function to publish Position messages via ROS.

Parameters
  • pos

Public Members

bool BeRoboyManager.TrackingEnabled = false

Set whether head movement should be tracked or not.

RenderTexture BeRoboyManager.RT_Zed

Reference to the render texture in which the Zed feed gets pushed into.

RenderTexture BeRoboyManager.RT_Simulation

Reference to the render texture in which the Simulation feed gets pushed into.

Private Functions

void BeRoboyManager.Awake()

Initialize textures.

void BeRoboyManager.RefreshZedImage(ImageMsg image)

Renders the received images from the zed camera

Parameters
  • msg: JSON msg containing the roboy pose.

void BeRoboyManager.RefreshSimImage(ImageMsg image)

Renders the received images from the simulation.

Parameters
  • msg: JSON msg containing the roboy pose.

void BeRoboyManager.translateRoboy()

Turn Roboy with the movement of the HMD.

void BeRoboyManager.tryInitializeCamera()

Looking for the main camera in the scene, which can be attached to Roboy.

Private Members

GameObject BeRoboyManager.m_Cam

The HMD main camera.

Texture2D BeRoboyManager.m_TexSim

Texture in which the received simulation images get drawn.

Texture2D BeRoboyManager.m_TexZed

Texture in which the received zed images get drawn.

bool BeRoboyManager.m_CamInitialized = false

Is the main camera initialized or not.

float BeRoboyManager.m_CurrentAngleX = 0.0f

Variable to determine if headset was rotated.

float BeRoboyManager.m_CurrentAngleY = 0.0f

Variable to determine if headset was rotated.

Color [] BeRoboyManager.m_ColorArraySim = new Color[640 * 480]

Color array for the simulation image conversion.

Color [] BeRoboyManager.m_ColorArrayZed = new Color[1280 * 720]

Color array for the zed image conversion.

SteamVR_TrackedObject BeRoboyManager.m_Controller

Reference to the tools controller (right).

ViewSelectionManager

class ViewSelectionManager

ViewSelectionManager handles the transition between various view scenarios.

Inherits from Singleton< ViewSelectionManager >

Public Functions

void ViewSelectionManager.TurnTrackingOn()

Turn head tracking for BeRoboy on.

void ViewSelectionManager.TurnTrackingOff()

Turn head tracking for BeRoboy off.

void ViewSelectionManager.SwitchToSimulationView()

Switches the view to the simulation view.

void ViewSelectionManager.SwitchToZEDView()

Switches the view to the ZED(real roboy camera in the head) view.

void ViewSelectionManager.SwitchToObserverView()

Switches the view to the observer view.

void ViewSelectionManager.SwitchToBeRoboyView()

Switches the view to the beroboy view.

Public Members

Canvas ViewSelectionManager.InstructionCanvas

Reference to the Canvas that is placed on the Camera plane(HMD).

Image ViewSelectionManager.BackgroundImage

Reference to the image where intructive text can be displayed.

RawImage ViewSelectionManager.GazeboImage

Reference to the image where the simulation feed can be displayed.

Image ViewSelectionManager.HtcImage

Reference to the image where the htc feed can be displayed.

RawImage ViewSelectionManager.ZedImage

Reference to the image where the zed feed can be displayed.

GameObject ViewSelectionManager.Cave

Reference to Roboy’s surrounding.

GameObject ViewSelectionManager.Roboy

Reference to Roboy itself.

GameObject ViewSelectionManager.Pointer

Reference to the teleporter ray.