Investigation of Factorized Optical Flows
as Mid-Level Representations
IROS 2022

Abstract

In this paper, we introduce a new concept of incorporating factorized flow maps as mid-level representations, for bridging the perception and the control modules in modular learning based robotic frameworks. To investigate the advantages of factorized flow maps and examine their interplay with the other types of mid-level representations, we further develop a configurable framework, along with four different environments that contain both static and dynamic objects, for analyzing the impacts of factorized optical flow maps on the performance of deep reinforcement learning agents. Based on this framework, we report our experimental results on various scenarios, and offer a set of analyses to justify our hypothesis. Finally, we validate flow factorization in real world scenarios.

Framework

In order to validate our assumption that \( \mathcal{F}_{ego} \) and \( \mathcal{F}_{obj} \) are promising candidates for mid-level representations, we develop a configurable framework based on the Unity engine and the Unity ML-Agents Toolkit. The framework is designed with an aim to facilitate the incorporation and modification of environments, scenarios, mid-level representations, as well as DRL algorithms. Such a design philosophy allows us to fulfill our objective of investigating the impacts of \( \mathcal{F}_{ego} \) and \( \mathcal{F}_{obj} \) under different conditions, and examine the interplay of them with the other types of mid-level representations. In this framework, the inputs to the DRL agents (i.e., mid-level representations) are all high-dimensional. As a result, the agents are required to interpret the provided mid-level representations, and extract necessary information concealed in them to learn its policy \( \pi \). Please note that optical flow factorization is not a built-in feature of the Unity engine, and is completely developed by ourselves.

overview
An overview of our framework

Experimental Results

Mid-level Representations OOB Collision OOB-to-collision ratio Value
\(F_{ego} + F_{obj}\) 12.82% 87.18% 0.147
\(D_2\) 9.51% 90.49% 0.105
\(S_2\) 4.47% 95.53% 0.047
\(F_{ego} + F_{obj} + D_2\) 8.05% 91.95% 0.088
\(F_{ego} + F_{obj} + S_2\) 2.72% 97.28% 0.028
\(F_{ego} + F_{obj} + D_2 + S_2\) 3.20% 96.80% 0.033

Subsection Title

Subsection content placeholder...

@article{Yang2022InvestigationOF,
            title={Investigation of Factorized Optical Flows as Mid-Level Representations},
            author={Hsuan-Kung Yang and Tsu-Ching Hsiao and Tingbo Liao and Hsu-Shen Liu and Li-Yuan Tsao and Tzu-Wen Wang and Shan Yang and Yu-Wen Chen and Huang-Ru Liao and Chun-Yi Lee},
            journal={2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
            year={2022},
            pages={746-753}
          }
Copied!