1.

INTRODUCTION

System integration is a challenging problem for any complex system. Various techniques have been developed to simplify this process. Open Architecture of hardware and software components enables adding, upgrading and swapping of components. Further refinement of the concept adds loosely coupled components communicating through well-defined published interfaces. There are currently multiple established and emerging approaches to this within DoD depending on program needs such as: Joint Architecture for Unmanned Systems (JAUS) [2], Future Airborne Capability Environment (FACE) [3], and Vehicle Integration for C4ISR/EW Interoperability (VICTORY) [4]. Within each domain, interoperability is possible due to enforcement of common middleware and data models. While systems developed within each of these environments are Interoperable within that domain, achieving interoperability between them and with existing legacy investments is again a challenge requiring additional testing and verification.

RCTA faces these same issues, with the additional constraint that we wish the researchers to be as productive as possible. Therefore, enforcing common middleware and software frameworks are an impediment as it would take valuable research dollars away from the research. Replacing the researchers preferred environment with a new system also results in loss of time and program dollars while the new system is learned. As a result, research occurs in multiple development environments, using a variety of tools. For example, one of the consortium members is Jet Propulsion Labs (JPL) who in the past transitioned RCTA developments to and from other DoD programs such as Legged Squad Support System. JPL has over the years developed real-time capable design practices and infrastructure to deal with the rigors of other worlds. On the other side of the spectrum are the University researchers who use less structured, non-real-time environments such as Robot Operating System (ROS) [5] and Matlab. To force JPL to use ROS would result in additional time cost when transitioning research, while forcing graduate students into real-time environments adds complexity to immediate task. Additionally, research environment changes based on research area and individual researcher preferences. For example, what is most efficient for Intelligence is not necessarily the best toolbox for Perception. It falls upon Integrated Research aspect of RCTA to find solutions which allow all consortium members to collaborate, integrate, and assess the joint technology in a single platform, and the data logs for evaluation by the Army Research Labs assessment team. Given integrated capability, this team collaborates to develop Integrated Research Assessment (IRA) scenarios consistent with transition to fielded solutions. Data from these IRAs are then reviewed to provide feedback to the consortium on how it is meeting expectations, documents research maturity, and is used to inform government leadership as to the status of the program. There are 4 basic elements to facilitate integration: RFrame, a Common World Model, RIVET simulator, and a common ITAR free Integration Platform.

2.

RFRAME

The first element of the IR approach is the RFrame middleware layer which provides transport agnostic, data centric communications between modular components. RFrame was designed by surveying common interoperability approaches to identify common functionality and take into consideration lessons learned from GDLS’s 23 years of real-time robotics experience. This survey included transports most common within the robotics community and potential transition to DoD programs: RCSLIB[6], DDS[7], JAUS, and ROS. Follow on reviews of LCM[8], MsgPack[8], and Protobuf [10] support the initial survey conclusions. This analysis found the following characteristics:

During the investigation, the following realizations occurred:

RFrame is implemented using C++ as dynamically loadable modules from simple core interfaces and common utilities such as configuration, transparent logging, thread safety, and data structures such as UUID and Sequence. Changing data transport, supporting multiple transports at runtime, upgrading/exchanging functionality, or relocating modules between processing elements is a configuration file change not a code change. Extending existing codebases is minimally invasive as RFrame may be integrated as an I/O library meeting the common open, close, read, write API model. Being transport agnostic enables modules to be integrated side by side with existing components. There is a rich research community and multiple interoperability efforts, each with de-facto, current and emerging standards, built on separate messaging and transport protocols, leaving a fractured landscape for Integrators to navigate. RFrame enables interoperability between new and legacy systems built upon different middleware architectures through the use of a superset Interface Description Language and template based code generation. This role is defined in the FACE standard as a “Paradigm Translator”.

For RCTA, this simply means that RFrame allows each researcher to use their preferred tools as it is simple to import their data, create data sets, and provide unified system logs to the government assessment team. Collaborator data are imported into RFrame XML using simple import conversion scripts (figure 1). These scripts will populate missing fields of the superset IDL if necessary. For example, if the input IDL contains a manually assigned id, it is maintained. If an ID is not provided, one is auto-generated by hashing the name of the imported structure with its namespace. If the IDL requires specialized information for its own protocol, it is also maintained. For example, ROS uses specialized MD5 checksums instead of message id and version numbers. Code generators then produce the serialization libraries for each requested transport protocol, which are dynamically loaded at runtime based on system configuration. As a result, protocols are transparent to the RFrame native modules minimizing extra bridging/interface code. RFrame is therefore data-centric and transport agnostic. The messaging system handles logging of all messaging regardless of transport, as well as data playback in simulation mode by using specialized modules. For example switching logging between disk and database solutions. All messages are transparently logged by default so ensure that all data is available when performing system analysis.

Figure 1:

RFrame code generation imports from multiple specification formats into a common XML based IDL; generates serialization for multiple data format & protocols, enabling a data-centric transport agnostic design strategy which reduces development time.

Structurally, RFrame adopts a layered approach (figure 2). At the lowest level are the Connection Managers such as ROS comms and JAUS comms which handle messaging for a specific protocol. Above the protocols is the Messaging Manager which provides a single interface to the client modules, and handles message routing to/from the Connection Managers and distribution of messages to client callbacks if necessary. RFrame supports both callback and non-callback (“direct”) read/write operations in order to support legacy, current, and future design patterns. Both the Connection Managers and Messaging Manager adopt a common interface API. Client modules are implemented using a MessagingClient, which extends the common interface API to simplify use of these client interfaces by taking advantage of the data introspection capabilities created by the code generator. As mentioned above, each client is implemented as a dynamically loaded module, and runs within a separate thread of execution. To support threading all modules are built around common state machine (construct -> init -> start -> exec main loop “once” -> stop -> shutdown ->destruct). This pattern allows RFrame core to handle complexity for the developer such as thread safety, module loop period, potential I/O deadlocks during startup/shutdown. This design enables an RFrame module to support both hard-real time designs such as the 4D/RCS [11] based GDLS systems, and soft-real time designs such as JAUS or ROS. We are able to interact with non-4D/RCS nodes while maintaining our proven real-time concepts.

Figure 2:

RFrame code generation imports from multiple specification formats into a common XML based IDL; generates serialization for multiple data format & protocols, enabling a data-centric transport agnostic design strategy reducing development time.

3.

WORLD MODEL

Element two, which is central to our Architecture, is the RCTA Common World Model (CWM) shown in Figure 3. The CWM defines and instantiates the Data Model for the RCTA Intelligence Architecture with a data-centric approach providing a common, centralized Intelligent Data Store services. The data model was defined by need to represent 3 layers of information: Metric, Semantic, and Self. Specific data types within these layers were determined by querying RCTA collaborators to determine the best mechanisms to represent the desired data types.

Figure 3:

High level component layout of the world model “Phase 1” [12]. “Phase 2” is currently active [13]

The CWM leverages the transport agnostic capabilities of RFrame to be accessible by the preferred environments of RCTA Collaborators. The modular nature of RFrame makes simplifies integration by providing a stable, consistent mechanism for linking disparate system components to update and extend the CWM. This approach was proven successful during the Capstone Assessment of the previous RCTA program [14, 15, 16, 17]. The assessment covered integration of technology developed over an eight-year research program into autonomous navigation. A week prior to the assessment, a vision based water detector [18] was integrated in four hours. Integration included linking output of the detector into the world model, defining the water types as obstacles in a configuration file, and completing successful live field testing. This integration was considered “low risk” primarily due to the use of the proven world model.

The world model is divided into three main concepts: Metric (sensor data and aggregates), Semantic Objects (class descriptions and instances), and Self Information. An access interface is provided for each of these following the “CRUD” database interface model: Create, Replace, Update, and Delete. Internally, the world model knows how the various data sources inter-relate, and when appropriate propagates changes between the three levels. This is why we refer to the world model as an “Intelligent” Data Store. It is more than just a database, it provides background data aggregation and calculation processes. Self-Information contains data relative to the robot’s “Self”. By encoding current capability, component status, task execution state, and their histories, we track information which enables the robot to reason and adapt its performance using Meta-Cognition and Machine Learning principles. This allows the system to answer questions such as: Where am I? What can I do? What am I commanded to do? and How well am I doing it?

The CWM is fully documented. The API, manual, example client programs, and technical support are provided to the consortium. More detailed information on the CWM may be found in [12][13].

4.

RIVET SIMULATOR

The third basic element is simulation. RCTA has developed a Hardware In The Loop (HITL) simulator called RIVET to meet the needs of RCTA researchers. RIVET is based on a commercially available 3D Gaming engine, and industry standard PhysX high fidelity physics.

Testing with a real robot is required to fully access a robotic capability. While necessary, physical testing is expensive: acquiring robots, maintenance, travel costs, range time and possibility of delays (weather, equipment failures, over headed vehicles, etc). Immature algorithms increase these costs. RIVET provides a highly detailed, capable environment for sensor and algorithm development, integration and assessment using common off the shelf computers. Researchers are able to execute an order of magnitude more runs in the simulator, than would be possible with the real robot. This savings of time directly translates to cost savings, increased productivity, and more refined algorithms ready for system integration. RCTA uses RIVET as a “gateway” to placing research onto the physical platforms, as well as a regression test mechanism for the integrated system.

Documented interfaces to RIVET are available through the use of the Basic Operation Level interface (BOLT) extension library. BOLT provides simple low level control and sensor interfaces to RIVET, allowing researchers to tailor the level of the simulation to their needs. RIVET installation provides a detailed model of the Fort Indiantown Gap CACTF test site, enabling research in a similar environment to real world assessment. Multiple vehicles and robot types are included (Figure 4a). Many sensors are also simulated including millimeter wave RADAR, 2D Ladar, 3D Ladar, monocular and stereo camera. Recently, Rivet has been upgraded to provide simultaneous ground truth for perception algorithms, which enables rapid training of the machine learning based perception algorithms while reducing false classifications due to human error.

Figure 4:

a) Left: Example RIVET environments. Simulated robots include ground, surface, and air vehicles, b) Right: Example of matched ground truth for Semantic Labeling of the scene showing buildings (orange), windows (cyan), vehicle (lime), doorways (yellow), gas pumps (pink) and wall (purple).

5.

INTEGRATION

RCTA as a program recognizes that individual research components can only be fully assessed when integrated onto a robot where they interact with other aspects of the system to create cross-Task capabilities which move beyond the State of the Art. To address the challenges inherent in integration, we have presented a multi-pronged approach of tools: RFrame enables transport agnostic messaging, CWM provides common Data Model and services, and RIVET provides a common cost effective initial integration platform.

For field assessment of integrated solutions, the fourth element supporting the RCTA program provides a common hardware platform. We are in the process of building 8 of these platforms: 3 fully outfitted to be used for program integration and assessment, and 5 basic setups provided to RCTA and ARL collaborators. The base configuration consists of a Clearpath Husky platform [23], Hokuyo line scanner [24], navigation unit, and bumblebee stereo camera, and a mac-mini running platform mobility control software. The full configuration (figure 5) adds a GDLS Adonis hi-dynamic range imager, and a GDLS microLadar 360 Ladar [26], and 3 mac-minis to run researcher software.

Figure 5:

Fully configured research platform.Top shelf: bumblebee stereo (left), GDRS uLadar (center), ADONIS (right). Bottom shelf: gps (left), hokuyo Ladar (center), teleop receiver (right).

We are adopting a “rolling integration” solution. As components are integrated and verified to successfully work together, they are version controlled and released back to the consortium to enable future research. For example, a researcher in path planning algorithms is able to use the latest vetted perception algorithms with RIVET to evaluate their research. A designated regression test SIL is being setup to assist with this purpose.

To date the program has performed 4 Integrated Research Assessments. These assessments are designed and executed by the ARL assessment team and designed to challenge the integrated system in a statistically sound manner. The goal is to find out how the algorithms respond to these scenarios in order to improve the science itself. Positive or negative outcomes during the assessment are simply results providing the researchers with data which allows them to continue improvement