Research on the development and challenges of dataflow processing technology

Rui She

doi:10.1117/12.2681619

23 May 2023 Research on the development and challenges of dataflow processing technology

Rui She

Proceedings Volume 12645, International Conference on Computer, Artificial Intelligence, and Control Engineering (CAICE 2023); 126450V (2023) https://doi.org/10.1117/12.2681619
Event: International Conference on Computer, Artificial Intelligence, and Control Engineering (CAICE 2023), 2023, Hangzhou, China

Abstract

Nowadays, unbounded and disordered large-scale data sets are becoming more and more common, and consumers' processing requirements for these data sets are also becoming more and more complex, such as time semantics, windows and processing delays. In order to meet the continuous development of data processing requirements on unbounded and disordered large-scale data sets, this paper analyzes the advantages of dataflow processing from the aspects of low latency, high parallelism, low synchronous memory access cost and simple on-chip logic. On the one hand, the dataflow diagram embodied in the data flow calculation model in big data processing is analyzed from the execution engine level. On the other hand, several typical dataflow processing architecture models are analyzed from the perspective of dataflow batch processing. On this basis, the problems faced in the development of current dataflow are analyzed, and the possible development direction in the future is proposed.

Citation Download Citation

Rui She "Research on the development and challenges of dataflow processing technology", Proc. SPIE 12645, International Conference on Computer, Artificial Intelligence, and Control Engineering (CAICE 2023), 126450V (23 May 2023); https://doi.org/10.1117/12.2681619

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available