RELATED CAPABILITIES:
Small UAS Detect and Avoid Requirements Necessary for Limited Beyond Visual Line of Sight (BVLOS) Operations (A18_A11L.UAS.22)
A team representing universities from the Alliance for System Safety of UAS through Research Excellence (ASSURE) conducted flight tests at the University of Alaska Fairbanks’s Poker Flat Research Range approximately 30 miles northeast of Fairbanks Alaska, during August 2019. These flight tests evaluated the IRIS CASIA Detect And Avoid (DAA) system as a part of research project: ‘A18_A11L.UAS.22: Small UAS Detect and Avoid Requirements Necessary for Limited Beyond Visual Line of Sight (BVLOS) Operations: Separation Requirements and Testing’. The goal of this research is to develop a repeatable and systematic test plan for DAA systems and use the results of real-world testing and evaluation of multiple current DAA technologies described here provided the information needed develop and refine the test plan. Three different DAA equipped aircraft were tested against four different intruder aircraft systems representatives of the types of small manned aircraft found in the National Airspace System. A total of 22 test flights executing 10 different mission profiles were flown over a span of four days. Data acquired during flight tests were post processed to calculate exactly when the intruder was within the field of view of the DAA system. These viable encounters, during which the intruder should have been detectable, were then compared to actual avoidance trigger events. The results of the IRIS CASIA system testing are as follows: 1) of the encounters with detections, the average time the intruder was within the FoV before triggering was 10.8 s. This value is unexpectedly high, and likely results from the data being skewed by predominantly head-on approaches where the plane is in the FoV but out of range for an extended period of time, 2) the effectiveness of the IRIS CASIA system decreased as range to the intruder increased, 3) the average detection distance was determined to be 656 m, and 4) IRIS’s stated detection rate for the tested system was 70% at 390 m when allowed 3s. The data acquired during this testing showed that IRIS achieved a 67% detection rate for separation distances less than 400 m. The data collected during these flights helped the A18 team refine the draft test plans and move towards the goal of creating a repeatable and systematic test plan for evaluating DAA systems. Discussions between FAA and team personnel about the test plan and the results of the test flights have led to a better understanding of how to maintain well-clear during testing and the value of using a geometric approach to conduct the flight testing of DAA systems.
TEAM: KSU MSU NMSU OSU UAF UND
FINAL REPORT | 
POC:
Mark Askelson
Exec. Director | Associate Dean of Research
Res. Inst. for Autonomous Systems (RIAS)
John D. Odegard School of Aerospace Sciences
University of North Dakota
Email: rias@UND.edu | askelson@aero.und.edu
Phone: 701.777.6880 | 701.777.6334
Discussion:
September 2020 UND Flight Test Report
The demand for Beyond Visual Line Of Sight (BVLOS) operations using small Unmanned Aircraft Systems (sUASs) is high. A major impediment to realization of these operations is the Detect And Avoid (DAA) function. Two critical DAA challenges are definition of sUAS DAA system performance requirements and development of test methods for those performance requirements. The ASTM (American Society for Testing and Materials) WK62668 Detect and Avoid Performance Requirements Task Group has developed proposed performance requirements for sUAS DAA. The ASTM WK62669 DAA Test Methods Task Group is currently developing test methods for evaluating compliance of sUAS DAA systems with performance requirements. This effort is informing the ASTM WK62669 DAA Test Methods Task Group.
A flight test method that is part of a broader testing approach that also involves simulation, lab testing, etc., is described herein. This test method leverages a geometric approach to gathering data, in which potential encounter geometries are varied. Justification for the subset chosen herein, including how it relates to the broader set of encounters, is provided.
Numerous metrics are developed and evaluated. These include sample risk ratio (from the DAA performance standard) and sample risk ratio uncertainty, which provides insight into whether the proposed approach (number of encounters) provides a viable basis for evaluation of risk ratio. In addition, encounter events (well clear/horizontal well clear/vertical well clear violations) are presented, overall encounter metrics such as Closest Point of Approach (CPA) are evaluated, and metrics for the major stages of DAA are proposed and presented. Maintenance of well clear status during testing, which is a goal during testing, is also evaluated.
For the encounter set that was evaluated, the sample risk ratio uncertainty, which was evaluated using three different methods, indicates that the set is viable for providing guidance regarding conformance with the performance standard. This performance must be placed in context with the broader set of encounters, the breadth of which can be evaluated through simulation. Moreover, the metrics that are developed herein for the major stages of DAA, which were developed through the need for information regarding how the system is performing and qualified by pragmatic considerations (e.g., timing challenges associated with data collection), provide useful information regarding these major stages (and are being considered by the ASTM WK62669 DAA Test Methods Task Group). Finally, methods developed for analyzing encounters, which include visualization techniques and summary metrics, enable understanding of encounter characteristics.
Read the full 2020 Flight Test Report Here
June 2021 UND Flight Test Report
A flight test method that is part of a broader testing approach that also involves simulation, lab testing, etc., is described herein. This test method leverages a geometric approach to gathering data, in which potential encounter geometries are varied. Justification for the subset chosen herein, including how it relates to the broader set of encounters, is provided. Relative to previous efforts, descend-into encounters were added to the set off encounter scenarios.
Analysis methods were modified, relative to previous efforts, to include a coast period that occurs at the end of the maneuver period. This is needed owing to the Uncrewed Aircraft (UA) reaching the visual range limit associated with Part 107 operations prior to the time when aircraft separation is minimized.
No horizontal well clear violations occurred during this test campaign, resulting in a loss of well clear risk ratio for this subset of encounter scenarios of 0.0. Uncertainty windows for this subset of encounters ranged from 0.0-0.01 to 0.0-0.02. The exceptional performance of the UA that was used in these tests—especially its cruising speed, significantly enabled maintenance of horizontal well clear.
These results support the ASTM WK62669 DAA Test Methods Task Group through identification of metrics that characterize DAA system performance. Moreover, metrics that can be used for comparison with simulation results, which comprise an overall strategy for demonstrating compliance with performance requirements, are suggested. This includes a new metric for characterization of aircraft separation/risk.
Participating Schools
