Review of conformal displays: more than a highway in the sky

Niklas Peinecke; Sven Schmerwitz; Hans-Ullrich Döhler; Thomas Lüken

doi:10.1117/1.OE.56.5.051406

6 February 2017 Review of conformal displays: more than a highway in the sky

Niklas Peinecke, Sven Schmerwitz, Hans-Ullrich Döhler, Thomas Lüken

Author Affiliations +

Optical Engineering, Vol. 56, Issue 5, 051406 (February 2017). https://doi.org/10.1117/1.OE.56.5.051406

Abstract

A head-worn display combined with accurate head-tracking allows one to show synthetically generated symbols in a way that they appear as a part of the real world. Depending on the specific research context, different terms have been used for the ability to show display elements as parts of the outside world. These include contact analog, scene linked, augmented reality, and outside conformal. While the famous highway in the sky was one of the first applications in avionics, over the years more and more conformal counterparts have been devised for aircraft-related instruments. Among them are routing information, navigation aids, specialized landing displays, obstacle warnings, drift indicators, and many more. Conformal displays have been developed for more than 40 years. We present a review of some results, as well as look ahead to research trends for the next years. We suggest that naturalism is not the best choice for the design of conformal displays. Instead, more abstract representations often yield better pilot acceptance.

1. Introduction

Usually, a display can only show a two-dimensional (2-D) representation of information that may be three-dimensional (3-D) by nature. To overcome this perspective projections of the respective content are used. For short we will call this perspective display 3D displays. These should not be confused with display technologies such as holographic displays. By the term “conformal display” we denote the ability of a display to represent synthetically generated content as part of the outside world. This technique dates back to at least 1968¹ when the necessary technology, namely see-through-displays and head-tracking devices, was available first. Also, even then the wording “contact analog,” a term closely related to conformal displays, was used. At this time, Dougherty already noted that “head-up and eyeglass displays seem to offer the next step forward in instrument display.”¹

Over the years the available devices have developed: displays were introduced that offer better contrast and color representations.²^,³ Further, a larger field of view was developed: while in 1999 state of the art was a 51-deg horizontal field-of-view display⁴ today 80 deg binocular⁵ and above are possible. With the introduction of electromagnetic sensors head tracking has become noncontact.⁶^–⁸ Although most of these developments date back to at least the early 1980s, mature consumer-market products combining all of these have become available just recently. This interest was further sparked by the advent of “augmented reality (AR),” a further term for combining real-world views with computer-generated objects coined in 1992.⁹

Helmet-mounted displays (HMD) have been demonstrated to show several advantages. This includes decreased scanning times between instrument information and outside scene, reduced visual reaccommodation, increased freedom in movement compared with head-up displays (HUD) and the possibility to depict conformal information.¹⁰ Further, 3-D displays have been considered in the field of synthetic vision displays for many years now. Thereby several human factors aspects have to be considered within the process of design and evaluation. Patterson¹¹ provides a review of perceptional and human factors issues associated with 3-D displays. As a major advantage conformal symbology allows the integration of information in the mental model of the outside scene.¹² This was found to provide benefits regarding guidance and navigation as well as reduction of attentional capture.¹³ The authors further point out the importance of scene-linking for nap-of-the-earth helicopter operations and low visibility approaches. Recently, implementations have been proposed that minimize the amount of overall clutter for the pilot.¹⁴

2. History of Conformal Displays at DLR

DLR’s Institute of Flight Guidance has investigated enhanced synthetic vision systems (ESVS) displays for flight guidance application for more than 17 years.¹⁵ One of the core components is its in-house developed test bed for ESVS applications.¹⁶ These have been examined in head-down, as well as head-up¹⁷ and head-worn implementations. One of the earlier devices that was used for this was the Microvision Nomad HMD. Starting in 2012, DLR has begun to develop dedicated symbols for conformal displays.¹⁸ Starting from a head-down only full-color representation of unclassified obstacles for brownout flight and landing [see Fig. 1(a)].

Fig. 1

“Manhattan” obstacle symbology for degraded visual environment: (a) head down, (b) HMD.

The symbology was later adapted to be shown in a head-tracked HMD fully conformal [see Fig. 1(b)].¹⁹ These early works were carried out within DLR’s project ALLFlight (Assisted Low-Level Flight and Landing on Unprepared Landing Sites), an initiative to mitigate effects of brownout and related degraded visual environments on helicopter operations. First results were published in 2012.²⁰ Furthermore, ALLFlight was the institutes’s first project to feature a military grade see-through display with head-tracking, namely Elbit’s JedEye™ helmet (see Fig. 2).

Fig. 2

Wide-field-of-view helmet-mounted display system.

One of the lessons learned from using this display was that military-grade solutions usually require large efforts for integration and maintenance in an always changing experimentation environment. Especially components such as the magnetic head tracker unit (MTU) have to be recalibrated in case of changes to cockpit geometry or electromagnetic conditions see (Sec. 3 for details). For a research helicopter, this can become expensive and time-consuming. Consequently, drop-in solutions for rapid prototypes and in-between developments were sought. The institute had already gathered experience with consumer-market VR glasses, namely the NVIS cybergoggle nVisor SX [see Fig. 3(a)].²¹ It was decided that for a faster shared development a set of easier to use consumer-market goggles should be used. As a first step, DLR bought an Oculus Rift DK2 [see Fig. 3(b)], later an Oculus Rift consumer edition. These sets were successfully used in human-factors evaluations for the new symbology to be developed.²² However, both of these sets are opaque virtual reality glasses. Thus, a see-through device (Metavision Meta2) was bought, although this is not yet in productive use. A further advantage of this device is its full-color display. Nevertheless, its use still has to be proven in real applications.

Fig. 3

VR helmet-mounted display systems: (a) NVIS cybergoggle nVisor SX, (b) Oculus Rift DK2.

3. Hardware Integration

At DLR mainly two research facilities have been used for HMD research: the research helicopter advanced control technology flying helicopter simulator (ACT/FHS), a Eurocopter EC135, and the Generic Cockpit Simulator GECO. DLR has integrated a wide field of view binocular helmet-mounted display system into both. The main electronics of the HMD system consists of three boxes: the aircraft fixed magnetic MTU, the JedEye display unit (JDU) for transferring the visual image to the helmet display, and the JedEye system display unit for producing the image and for interfacing with other on-board components (see Fig. 4). The front-end of the system consists of a transparent visor. Its monochrome green binocular projection system comprises two image projectors, each with a resolution of $1920 \times 1200 pixels$ . The optics consists of the projection lenses and a transparent dichroic spherical mirror (visor) for each eye. They offer a field of view of $\sim 80 \deg \times 40 \deg$ . Together with a magnetic high-precision tracker, the system yields an unlimited field of regard, that is, $- 180 \deg$ to 180 deg in azimuth and $- 90 \deg$ to 90 deg in elevation.

Fig. 4

Functional integration of HMD system into environment.

To find the best location of the magnetic tracker with a minimum of electromagnetic influences, the motion box of the pilot’s head has to be measured using a 3-D scanning robot. The purpose is to calibrate the systematic error of the entire tracking system inside the motion box. This requires a perfect zero degree leveling of the platform that has to be constructed for each environment the tracker is to be integrated into. In our case, these were both the ACT/FHS [Fig. 5(a)] and the GECO [Fig. 5(a)]. The tracker was temporarily attached to different possible mounting positions in order to find the best fitting. The final position has been established as a permanent installation of the MTU.

Fig. 5

Magnetic survey of the (a) ACT/FHS and (b) GECO.

To align the system’s optical axis with the aircraft axis, a boresight reference unit (BRU) is used that provides an aircraft aligned optical reference beam. Before flight, pilots have to align a marking on the HMD with the reference beam. This method allows pilots to align the system with different body sizes in terms of the 0-deg line of sight (LOS). This ensures an alignment of the helmet with the extension of the longitudinal axis of the helicopter. It requires a very precise measurement of the LOS in conformance with the head movements of the pilot with minimal latency between LOS measurement and image presentation. Otherwise the required handling qualities could not be guaranteed. If conformal symbology does not correspond to the real world, an increasing irritation and possible sickness of the pilot can occur after a few minutes. An installation of a BRU in the GECO was not necessary, since the alignment process can be started by focusing the 0-deg coordinate on an existing virtual reference grid.

4. Conformal Display Symbologies

The idea of a conformal display symbology is to show synthetically generated symbols in spatial relation to real world objects or their positions. This way, the user can intuitively perceive additional information about these objects and gain a better situational awareness. This is important in situations with degraded vision, for example, dust, fog, mist, or glare. Further, such displays can help in scenarios that require a rapid understanding of a situation, for example, helicopter or airplane landing in unprepared areas.

Different terms have been defined for the concept of conformal displays. Dougherty¹ used the term “contact analog,” describing a display that mimics the real-world appearance and placement of objects as close as possible.

Johnson and Jones²³ state that contact analog displays are “difficult to generate” and “have low information content.” Instead they prefer the term “integrated display” for a display type containing symbols that are associated with certain ground locations, for example, the touch-down point. This shows that at that time no real consent was existing about the terminology describing a display that combines real sensor data with synthetic symbols.

This changed with the emergence of virtual reality techniques. Caudell and Mizell⁹ introduced the term AR for a mixture of real-world perceptions with computer-generated synthetic symbols. Additionally to imagery, AR may also encompass sound and tactile information.

Therefore, we prefer the term conformal display. This concept has been in use since at least 1992.²⁴ The term emphasizes the alignment of symbols with the outside scenery.

4.1.

Tunnel Displays

The best known example of a conformal symbol is the concept known as “highway in the sky” or “tunnel in the sky.” This concept is known since at least 1979²⁵ although similar displays predate this description.²⁶^,²⁷ Probably the first animated illustration of this idea can be seen in Stanley Kubrick’s “2001—A Space Odyssey” (1968) wherein a tunnel display is used during a docking maneuver at a space station. A highway or tunnel-in-the-sky is composed of discrete “gates” floating in midair that the pilot has to steer through in order to follow a preplanned trajectory. Various implementations exist that differ by the shapes of the gates and the way these are interconnected. Tunnels have been used for several purposes, for example, curved approaches,²⁸ following noise abatement flight procedures, and emission control.²⁹

There is evidence that tunnel displays lead to a better overall pilot performance.³⁰ Highway-in-the-sky displays allow pilots to follow even difficult trajectories. However, these displays may bind the pilots’ attention sacrificing other tasks. This is true, even more when the display is implemented head-down. Thus, an HUD is considered useful. The pathway helps mitigating the perceptual segregation between the static near domain and the dynamic far domain and hence might improve attention switching between both sources. Generally, head-up implementations combine the advantages of both—tunnel displays and HUD.³¹ To overcome the perceptual near-to-far domain disconnect, alphanumeric symbols can be attached to the pathway. This HUD design concept is called scene linking. A scene-linked pathway-predictor was implemented on a monocular retinal scanning head-mounted device in combination with an optical head tracker. The evaluation comprises low-fidelity part-task simulations, high-fidelity simulator runs, and flight trials. Where laboratory experiments found evidence in favor of scene-linked pathway HUDs or HMDs, the real flight tests could not fully support this display concept. Even so, in all studies evidence has been found that the head-up pathway concept could be superior to previous, nonconformal head-up solutions.¹⁷

One of the most recent implementations of a tunnel-in-the-sky has been tested on a fully head-tracked HMD³² (see Fig. 6).

Fig. 6

HMD implementation of tunnel-in-the-sky.

4.2.

Landing Displays

Tunnel displays are relatively referenced to precalculated points of empty space in midair. Thus, their requirements in terms of accuracy of location are defined by the navigation solution of the aircraft. Other types of symbols need to be attached to real-world visible landmarks, making a precise navigation even more important. An example of such a display is the helicopter landing display. The purpose of a landing display is to help the pilot in finding the exact location and orientation of a previously identified landing spot. This can be, for example, a concrete helipad or an unprepared clearing in a forest. Depending on the actual type of the helicopter and its mission, additional information may be important, for example, terrain slope, wind direction, and drift relative to ground.

Different companies and research groups have proposed designs for landing displays, for example, DeVila and JedEye.³³ Among these, the US-Army’s BOSS (see Fig. 7) has become a baseline and de facto standard.

Fig. 7

(a) BOSS and (b) DEVILA landing displays.

Other than tunnel displays, HMD landing displays do not seem to support the pilot’s performance positively.³³

At DLR there is an ongoing initiative to investigate the integration of further information in a conformal way. For example, a drift indicator has been devised and tested recently.³⁴ More details on this can be found in Schmerwitz et al.³⁵

4.3.

Obstacle Symbols

A main area of research interest is the development of obstacle symbols. Naturally, obstacles are a major hazard, especially to helicopters. Therefore, their representation to the pilot—just after detection—is safety-critical. Similar to landing zone visualizations, obstacle symbols have to be placed very accurately at their real-world counterpart. Over the years, DLR has investigated three classes of obstacle symbols: volumetric box-shaped symbols (“Manhattan” blocks), simplified naturalistic representations, and glassdome symbols.

4.3.1.

Manhattan blocks and octree blocks

The first implementation was volumetric box-shaped symbols that were called “Manhattan” metaphor.¹⁸ These were primarily meant to represent locations of raw, unclassified point data, for example, from a laser scanner or similar sensor devices. As a preprocess, obstacle data have to be separated from terrain data. Terrain data can then be shown as a regular ground grid, while Manhattan boxes can represent the bounding volumes of clustered obstacle points. This is useful in case there is no detailed knowledge about the nature of the individual obstacle (see Fig. 1). For this, the terrain is divided into a regular grid of fixed dimensions. All detected sensor data within a grid cell are represented by a ground-based column. The height of the column corresponds to the highest measured data point in that grid cell (see Fig. 8). Since all obstacles displayed are ground based, the pilot cannot make use of form cues. Structures that are not attached to the ground appear grounded. For example, a bridge is shown closed and can no more be recognized as a bridge.

Fig. 8

Principle of the Manhattan blocks.

This consideration lead to the introduction of the octree blocks representation. In this case, terrain is shown as before but obstacles are represented as freely placed cubes of varying sizes. For this, the bounding volume of a cluster of measured points is subdivided recursively into cubes of equal size until either each point is exclusively contained within a cube, or the cubes reach a minimum size. The points are then represented by all occupied cubes. Figure 9 shows this principle in a 2-D variant: As more data points become available, the bounding box is divided into smaller blocks, and only occupied blocks are being drawn. This display requires more computational power from the display system. It allows the pilot to make use of simple form cues. For example, bridges, cranes, and poles can be recognized as such.

Fig. 9

Principle of the octree blocks.

In order to evaluate the different types of obstacle displays, a reaction time experiment was conducted.¹⁹ The three variants of the display—Manhattan, octree, and a baseline terrain-only representation—were explained in detail and sample screenshots were shown in order to familiarize the test person with the setup. After starting the test software, the test subjects pressed a key to start the experiment. For each obstacle display, 20 situations were presented, resulting in a total of 60 test runs. Test subjects completed all test situations with one display before starting the same 20 tests with a different display. The test person was sitting in front of a standard desktop PC with two monitors and a keyboard. On the left monitor two images were displayed, labeled with letters “A” and “B.” These images were taken during a real flight test using the onboard electro-optical camera. On the right monitor, the obstacle display was shown, representing a 3-D reconstruction of sensor data recorded during the same flight test. The sensor data belonged to either of the two images on the left monitor. While the images were still, the obstacle display was showing a 5-s dynamic scene, that is, the virtual helicopter was moving during display and the test person could see the reconstruction appearing as more sensor data became available. The test person then had to decide to which of the two images the data shown in the obstacle display belonged. Pilots were instructed to evaluate the displays as if they were using them operationally rather than in an experimental setup. Twelve male pilots and one trainee pilot participated in the experiment. Participants showed the highest level of percent correct responses with 83.85% for the Manhattan display. 76.54% correct responses were given with the octree display and 66.54% with the Terrain display. Further, we performed an analysis of variance only including the reaction times of correct responses to evaluate whether there are speed differences between the displays when actually making the right decision. No significant results were found, that is display type did not seem to influence reaction time for correct responses. Results of a final questionnaire show that pilots want to use the octree and the Manhattan display significantly more in a degraded visual environment than the plain Terrain display. Twelve of 13 pilots rated octree as their favorite display, one pilot rated Manhattan as his favorite and no one preferred the Terrain depiction.

In a second experiment, it was tested if pilots prefer transparent Manhattan block representations (wire-frame) over opaque Manhattan block representations or vice versa.³⁶ Tests included a primary task in landmark detection and a secondary task—checking aircraft parameters—to measure the overall workload. Both seem to perform well in landmark detection. However, although pilots rate the wire-frame display rather negatively this does not show in the objective results regarding the landmark detection. The result is consistent with pilot comments that the opaque design required the smallest amount of concentration and left more attentional capacity for the secondary tasks. The authors conclude that the main reason for the result can be explained by the different levels of clutter. In line with the subjective statements, the wire-frame design produces the highest level of clutter since objects cannot be easily distinguished from terrain. A high level of clutter seems to require and capture a higher amount of attentional capacity on the display. Thus, clutter does seem to affect the information search and the ability to respond correctly. The opaque display received the highest subjective ratings and the best measured results on both detections of obstacles and the ability to perform extra tasks.

4.3.2.

Naturalistic symbols

A second option was to use simplified but naturalistic representations of single obstacles.⁵ These are meant to be used in cases of known obstacle locations and classes from a reliable source, for example, an airport obstacle database. In this display, symbols are shown at the exact locations of the real obstacles, sometimes replacing or obscuring these, see Fig. 10. Therefore, the primary use of this display is in degraded visual conditions when there are few useful outside cues left. Further, the naturalistic representation suggests that information about, for example, orientation of the turret and the rotor blades are correct, even if such information unknown. This can be potentially dangerous if the pilot relies on this information.

Fig. 10

HMD obstacle representation with naturalistic symbols.

The results of a human-factor evaluation show conformal symbology advantages for both flight path tracking and landing precision. Subjective ratings of the HMD symbology designs were predominantly positive: 15 out of 18 pilots reported that the HMD provided them better support than the standard instrumentation and that performing the scenarios with the HMD was easier compared with the baseline condition.⁵

4.3.3.

Glass domes

Helicopter emergency missions data from existing databases containing terrain elevations, possible obstacles, and water bodies can be included to warn the pilot about landing risks. These preexisting data can be combined with data collected on-the-fly by on-board sensors like Ladar or radar. One of the main challenges is then to represent the multitude of data to the pilot in a way that enables him to operate the helicopter safely, instead of overloading him with unnecessary information. This is achieved using a combination of a head-tracked HMD and special nonintrusive symbologies that provide just the essential information without cluttering the display or blocking the pilot’s view on important outside details.

DLR started developing a full-scale helicopter sensor solution combined with HMD capabilities within project ALLFlight.²⁰ In a number of improvement steps,⁵^,¹⁹ a noncluttering, low intrusion symbology to be used with conformal displays was developed. This effort has been transferred into the follow-up project AllInFlight (assisted low-level flight using in-flight simulation capability). The current symbol set is called “glass dome.”²²^,³⁷ It is meant to serve as a universal base implementation that can be easily extended to embrace more classes of obstacles if desired. This latest addition to the experimentation setup is a combination of a simplified bounding volume representation with a class icon. The main advantage is that the glass domes usually do not obscure large parts of the real-world scenery. Further, their small but a legible size adjustable icons allow a safe identification of obstacle classes for the pilot (see Fig. 11).

Fig. 11

HMD obstacle representation with glass domes.

The main use of this symbol set is when there are reliable obstacle data, for example, from off-line databases or certified sensor chains while it is important to keep visual contact with outside landmarks. To keep display clutter at a minimum, it was implemented that groups of equal symbols could be clustered together in a common glass dome symbol if they are placed sufficiently close together. Such cluster symbols would appear in place of the symbol groups at larger distances (see Fig. 11). At closer approach, these clusters are replaced by the individual symbols. To find an optimal balance between detailed information and low display clutter, two variants of this strategy were evaluated: unclustering at 3 km distance and unclustering at 2 km distance.

A first evaluation of the display was carried out. Only four pilots participated in the study. Thus, it should be repeated to gain relevant data. In the evaluation, the pilots considered the display to be helpful. The legibility and acuity of the symbology and its elements were generally rated rather favorably. However, the clustering feature was seen critically in good weather conditions. One reason might be the evidence of large and prominent cluster symbols, causing the illusion of reversed spacial order, with larger clusters appearing closer than they actually are. This is especially true in adverse weather. One pilot mentioned that in mist with no obvious ground reference some symbols appear to be floating above ground. Furthermore, the late unclustering variation at 2 km distance was seen unfavorable, since the presence of both—clustered and unclustered symbols—was seen to introduce clutter and misinterpretations on the display. Figure 12 shows the questions and the pilots’ ratings on a scale from 1 (strongly disagree) to 5 (strongly agree). Results are shown for either single symbols or clustered variants (3 or 2 km) with good or poor visual conditions.

Fig. 12

Poststudy questionnaire for glass dome symbology.

Icon display was judged as not helpful because of legibility problems. However, these could be overcome, for example, by a fixed minimum icon size.²²

4.3.4.

Safety line

A different approach to obstacle warning and avoidance represents the safety line. A safety line is a function $s$ that generates for each possible azimuth direction $α$ an elevation angle $s (α)$ , representing the minimum safe angle to avoid all known obstacles or terrain. $s (α)$ is calculated based on:

• terrain elevation,
• database obstacles,
• sensor detected obstacles,
• current aircraft state,
• known aircraft performance parameters.

This ensures that the angle $s$ always leads to a safe position that can be reached by the helicopter given its current position and state.

Safety lines can be integrated into several instruments, including primary flight display (PFD), head-down synthetic vision display, as well as HUD and HMD displays. Usually, the safety line would be represented as a silhouette conformal to terrain and obstacles.

Figure 13 shows a concept drawing superimposed on a simulator view. The amber line represents the safe escape angle based on current aircraft state, performance parameters, and the detected obstacles. Safety lines have been implemented in a number of current displays and obstacle warning systems.¹⁴^,³⁸

Fig. 13

Concept of a safety line: the amber line represents a safe escape elevation angle.

4.4.

Virtual Flight Decks

Virtualization describes the technique of replacing components in a process by simulated counterparts in a virtual reality environment. For example, instead of building a clay model a designer can construct a CAD model and view it using VR glasses. Virtualization has found applications in all areas of design, manufacturing, and user interaction. Thus, an aircraft piloting itself is likely to be virtualized soon. Replacing the physical cockpit instruments with VR goggles and a set of software simulations exposes different advantages and drawbacks. DLR has been exploring these questions since 2015.³⁹ There are two approaches of virtual cockpit instrumentation:

1. Simply display a virtual version of a traditional flight deck instrument, for example, a PFD at the location where one would expect the real instrument (see Fig. 14).
2. Design a new set of virtual instruments that allow the pilot to carry out all his tasks more efficiently, if possible.

Fig. 14

Virtual flight deck instruments as seen through the HMD.

The first approach is straightforward in implementation, in that one only has to provide virtual instruments that exist from earlier “glass cockpit” designs. The second approach can take benefit from developing conformal HUD and HMD symbologies. Today, more pilots are familiar with such displays. Thus, future concepts may differ from traditional patterns. Human factors evaluations show that pilots more and more learn to navigate in virtual worlds. Further, both approaches can be combined to allow a fully configurable pilot workplace that can be adapted to different aircraft and mission requirements.

A major challenge is the question how to interact properly with pure virtual environments. Usually, tactile and haptic inputs are not present with a virtual display, and even if they are, then to a much lesser degree than in a real environment.⁴⁰ In addition, interactive techniques such as touch displays, vibrotactile actuators, optical head-, hand-, and eye-tracking provide integration problems in a cockpit that itself oscillates, moves, rotates, and is limited in space. Consumer market technologies are often adapted to inside situations without problems of vibration or difficult lighting. Therefore, flight-ready equipment has to be selected carefully, or has to be adapted to the cockpit environment.

Both approaches have to deal with the problems of safety and security that arise from the fact that, for example, the failure of a single component—in this case the VR headset—disables all virtual instruments. This is not just a safety but also a security issue, as a potential attacker could especially target the VR glasses, their connections to the avionics, or the aircraft’s sensors. On the other hand, virtual instruments solve some traditional safety and security issues: They allow the aircraft designers to place the cockpit at a safer position inside the aircraft—or even outside the aircraft. Here, the concept of a virtual flight deck closes the gap between already existing glass cockpits and future concepts like the single pilot cockpit, remote copilots, and even remotely piloted aircraft.

5. Conclusion

We have described the development of conformal displays in head-worn and helmet mounted displays during the last half century. Special focus was on DLR’s own contributions in the last 17 years. We have shown that conformal symbologies have developed from simple highway-in-the-sky applications to a broad variation of nearly all cockpit-related instrumentation. With the availability of better technology for display and head-tracking conformal displays have become more useful in all stages of flight and mission support. The authors expect these applications to be widely accepted in military as well as civil aeronautics in the forthcoming years.

Although the main focus of conformal symbologies is still their application in see-through displays, opaque virtual reality displays will probably play a major role in the near future. Their acceptance will be driven by parallel developments in consumer electronics and entertainment industry.

Recent results have shown that DVE operations can strongly benefit from applying carefully designed conformal displays. The central point for the authors is that a conformal display should not merely imitate reality. Often, a too naturalistic representation may mislead the pilot to false assumptions. Instead, it shows that a good symbol is a symbol that just provides the relevant information without adding further clutter to the display. The drift line indicator and the glass dome can serve as examples for this principle: reduce the information shown to just the relevant features. Further more, present the data in the most intuitive way. If these goals can be reached, pilots are likely to accept conformal displays, and they will expect a safety benefit for their missions.

Acknowledgments

The work presented in this paper was partially developed in the DLR internal project AllInFlight and was partly sponsored by the German Department of Defense (DOD) within the project “Increase of All-Weather Capability by Developing a Flight Management System with the Use of a Helmet-Mounted System (HMD).”

References

1.

D. Dougherty, “Advances in V/STOL cockpit information,” in The Avionics Panel XVIth Technical Symp.: Problems of the Cockpit Environment, 15 (1968). Google Scholar

2.

D. A. Bosserman, “High luminance miniature display,” US Patent 4,361,384 (1982).

3.

H. S. Cole, “Transflective liquid crystal display,” US Patent 4,398,805 (1983).

4.

S. Yamazaki et al., “Thin wide-field-of-view HMD with free-form-surface prism and applications,” Proc. SPIE, 3639 453 –462 (1999). http://dx.doi.org/10.1117/12.349411 PSISDG 0277-786X Google Scholar

5.

N. Peinecke et al., “An evaluation environment for a helmet-mounted synthetic degraded visual environment display,” in IEEE/AIAA 33rd Digital Avionics Systems Conf. (DASC ‘14), (2014). Google Scholar

6.

H. P. Kalmus, “Direction indicator,” US Patent 3,121,228 (1964).

7.

G. Board and J. Scully, “Communicating with computers in human terms,” in Proc. of Graphics Interface, 329 –335 (1982). Google Scholar

8.

J. H. Allen and R. C. Hebb, “Helmet mounted display feasibility model,” (1983). Google Scholar

9.

T. P. Caudell and D. W. Mizell, “Augmented reality: an application of heads-up display technology to manual manufacturing processes,” in Proc. of the Twenty-Fifth Hawaii Int. Conf. on System Sciences, 659 –669 (1992). http://dx.doi.org/10.1109/HICSS.1992.183317 Google Scholar

10.

M. Yeh, C. D. Wickens and F. Seagull, “Effects of frame of reference and viewing condition on attentional issues with helmet mounted displays,” (1998). Google Scholar

11.

R. Patterson, “Human factors of 3-D displays,” J. Soc. Inf. Display, 15 (11), 861 –871 (2007). http://dx.doi.org/10.1889/1.2812986 JSIDE8 0734-1768 Google Scholar

12.

C. Wickens, “Aviation displays,” Principles and Practice of Aviation Psychology, 147 –200 Lawrence Erlbaum Associates, Mahwah, New Jersey (2003). Google Scholar

13.

R. S. McCann and D. C. Foyle, “Scene-linked symbology to improve situation awareness,” in AGARD Conf. Proc., 1 –11 (1996). Google Scholar

14.

F. Viertler, C. Krammer and M. Hajek, “Analyzing visual clutter of 3D-conformal HMD solutions for rotorcraft pilots in degraded visual environment,” in 41st European Rotorcraft Forum, (2015). Google Scholar

15.

B. Korn, H.-U. Doehler and P. Hecker, “MMW radar data processing for enhanced vision,” Proc. SPIE, 3691 29 –38 (1999). http://dx.doi.org/10.1117/12.354412 PSISDG 0277-786X Google Scholar

16.

H.-U. Döhler and N. Peinecke, “An evaluation test bed for enhanced vision,” Proc. SPIE, 7689 768907 (2010). http://dx.doi.org/10.1117/12.850716 PSISDG 0277-786X Google Scholar

17.

B. Korn et al., “Combining enhanced and synthetic vision for autonomous all-weather approach and landing,” Int. J. Aviat. Psychol., 19 (1), 49 –75 (2009). http://dx.doi.org/10.1080/10508410802597408 Google Scholar

18.

N. Peinecke et al., “Developing an obstacle display for helicopter brownout situations,” Proc. SPIE, 8360 83600R (2012). http://dx.doi.org/10.1117/12.919081 PSISDG 0277-786X Google Scholar

19.

N. Peinecke and P. Knabl, “Design considerations for a helmet-mounted synthetic degraded visual environment display,” in IEEE/AIAA 31st Digital Avionics Systems Conf. (DASC ‘12), (2012). http://dx.doi.org/10.1109/DASC.2012.6382300 Google Scholar

20.

T. Lueken et al., “ALLFlight: tackling the brownout problem,” CEAS Aeronaut. J., 3 1 –15 (2012). http://dx.doi.org/10.1007/s13272-011-0027-3 Google Scholar

21.

N. Peinecke and S. Schmerwitz, “Correcting hyperstereopsis in a helmet mounted vision system,” Proc. SPIE, 7689 76890B (2010). http://dx.doi.org/10.1117/12.849625 PSISDG 0277-786X Google Scholar

22.

N. Peinecke et al., “The glass dome: low-occlusion obstacle symbols for conformal displays,” Proc. SPIE, 9839 983907 (2016). http://dx.doi.org/10.1117/12.2223597 PSISDG 0277-786X Google Scholar

23.

D. Johnson and R. W. Jones, “Head-up display of approach information,” in The Avionics Panel XVIth Technical Symp.: Problems of the Cockpit Environment, (1968). Google Scholar

24.

D. R. Jones, T. S. Abbott and II J. R. Burley, “Evaluation of conformal and body-axis attitude information for spatial awareness,” Proc. SPIE, 1695 146 –153 (1992). http://dx.doi.org/10.1117/12.131959 PSISDG 0277-786X Google Scholar

25.

J. J. Hatfield, J. B. Robertson and V. M. Batson, “Advanced crew station concepts, displays, and input/output technology for civil aircraft of the future,” in 3rd Digital Avionics Systems Conf., (1979). Google Scholar

26.

W. C. Blair and D. W. Plath, “Submarine control with the combined instrument panel and a contact analog-roadway display,” J. Eng. Psychol., 1 (2), 68 –81 (1962). Google Scholar

27.

T. B. Sheridan, H. M. Paynter and S. A. Coons, “Some novel display techniques for driving simulation,” IEEE Trans. Hum. Factors Electron., HFE-5 29 –32 (1964). http://dx.doi.org/10.1109/THFE.1964.231652 Google Scholar

28.

T. Lüken and B. Korn, “Tunnel in the sky guidance for helicopter IFR steep and curved approaches,” in 9th ONERA-DLR Aerospace Symp., (2008). Google Scholar

29.

B. Gursky, W. Olsman and N. Peinecke, “Development of a tunnel-in-the-sky display for helicopter noise abatement procedures,” CEAS Aeronaut. J., 5 (2), 199 –208 (2014). http://dx.doi.org/10.1007/s13272-014-0100-9 Google Scholar

30.

B. Lorenz, H. Többen and S. Schmerwitz, “Human performance evaluation of a pathway HMD,” Proc. SPIE, 5802 166 –176 (2005). http://dx.doi.org/10.1117/12.603280 PSISDG 0277-786X Google Scholar

31.

S. Fadden, P. M. Ververs and C. D. Wickens, “Pathway HUDs: are they viable?,” Hum. Factors, 43 (2), 173 –193 (2001). http://dx.doi.org/10.1518/001872001775900841 Google Scholar

32.

T. Lueken, H.-U. Doehler and S. Schmerwitz, “Helmet mounted display supporting helicopter missions during en route flight and landing,” Proc. SPIE, 9839 98390A (2016). http://dx.doi.org/10.1117/12.2225440 PSISDG 0277-786X Google Scholar

33.

H.-U. Doehler et al., “Evaluation of DVE landing display formats,” Proc. SPIE, 8360 83600P (2012). http://dx.doi.org/10.1117/12.922459 PSISDG 0277-786X Google Scholar

34.

S. Schmerwitz et al., “Drift indication for helicopter approach and landing,” Proc. SPIE, 9471 94710D (2015). http://dx.doi.org/10.1117/12.2177816 PSISDG 0277-786X Google Scholar

35.

S. Schmerwitz et al., “Conformal display: human factors analysis of innovative landing aids,” Opt. Eng., (2017). Google Scholar

36.

P. Knabl and N. Peinecke, “Designing an obstacle display for helicopter operations in degraded visual environment,” Proc. SPIE, 8651 865111 (2013). http://dx.doi.org/10.1117/12.2004143 PSISDG 0277-786X Google Scholar

37.

N. Peinecke, “Is OpenSceneGraph an option for ESVS displays?,” Proc. SPIE, 9471 947109 (2015). http://dx.doi.org/10.1117/12.2179948 PSISDG 0277-786X Google Scholar

38.

C. Seidel et al., “Novel approaches to helicopter obstacle warning,” Proc. SPIE, 6214 621406 (2006). http://dx.doi.org/10.1117/12.664042 PSISDG 0277-786X Google Scholar

39.

T. Lueken, J. Ernst and H.-U. Doehler, “Virtual cockpit instrumentation using helmet mounted display technology,” in 41th European Rotorcraft Forum, (2015). Google Scholar

40.

S. Stassi et al., “Flexible tactile sensing based on piezoresistive composites: a review,” Sensors, 14 (3), 5296 –5332 (2014). http://dx.doi.org/10.3390/s140305296 SNSRES 0746-9462 Google Scholar

Biography

Niklas Peinecke received a diploma in mathematics in 2001 and a PhD in 2005 in computer science from Leibniz University of Hannover. He is coinventor of a technique for 3-D shape classification. Since 2007, he has worked at the German Aerospace Center in the areas of sensor simulation, displays, sensor fusion, and collision detection. His further research interests include computational geometry and computer graphics.

Sven Schmerwitz received his diploma in aeronautical engineering in 2004 from the Technical University of Braunschweig and a PhD in 2009 with the title “radar based pilot assistance system for visual guidance during poor visibility.” Since 2004, he is employed at the German Aerospace Center as project manager and researcher in the area of pilot assistance systems. He is focused on display content, sensor information, and human factor-related issues.

Hans-Ullrich Döhler received his diploma in electrical engineering from the Technical University of Braunschweig in 1983. He has worked for Siemens and the TU Braunschweig. In 1990 he received a PhD on “image segmentation on the basis of 2-D median filtering.” For DLR, he is responsible for imaging sensors, image analysis, and enhanced vision. In 1997, he was granted the innovation award for his development of an enhanced vision system based on an imaging radar.

Thomas Lüken received his diploma in computer science in 1999 from the Technical University of Braunschweig. In 1999, he was employed as a scientific assistant at the same university. Since 2002, he works for the German Aerospace Center DLR, where he is responsible for helicopter projects. In 2006, he passed the PhD with a work about “multilateration for collection and forecast of airplane movements within an airport’s terminal area.”

Citation Download Citation

Niklas Peinecke, Sven Schmerwitz, Hans-Ullrich Döhler, and Thomas Lüken "Review of conformal displays: more than a highway in the sky," Optical Engineering 56(5), 051406 (6 February 2017). https://doi.org/10.1117/1.OE.56.5.051406

Received: 10 October 2016; Accepted: 13 January 2017; Published: 6 February 2017

Access the abstract

JOURNAL ARTICLE
9 PAGES

DOWNLOAD PAPER SAVE TO MY LIBRARY

GET CITATION

CITATIONS

Cited by 12 scholarly publications.

Explore citations on Lens.org

RIGHTS & PERMISSIONS

Get copyright permission Get copyright permission on Copyright Marketplace

KEYWORDS

Head-mounted displays

Heads up displays

Glasses

Sensors

Safety

Virtual reality

Visualization

1.

Introduction

2.

History of Conformal Displays at DLR