Optimized parallel causal product for linear transformer

Siqi Liu

doi:10.1117/12.2684707

21 July 2023 Optimized parallel causal product for linear transformer

Siqi Liu

Proceedings Volume 12717, 3rd International Conference on Artificial Intelligence, Automation, and High-Performance Computing (AIAHPC 2023); 127172K (2023) https://doi.org/10.1117/12.2684707
Event: 3rd International Conference on Artificial Intelligence, Automation, and High-Performance Computing (AIAHPC 2023), 2023, Wuhan, China

Abstract

The transformer has revolutionized Natural Language Processing and its exceptional performance has extended its application to computer vision. However, as more complex tasks arise, the demand for models to process longer sequences has increased. To address this, efficient transformers with reduced time and memory complexities have been developed. Linear transformer is an outstanding one of them with linear time complexity with input length. Causal product is the core computation of causal linear attention in linear transformer, which are commonly used to perform autoregressive tasks. However, in order to preserve the linearity, the causal products of queries are computed sequentially in Linear Transformer. We propose an innovative CUDA kernel function which allows for parallel computation of the query attention instead of sequential. Furthermore, we optimized the GPU computations of linear attention in Linear Transformer by double pipeline that decreases the time of data transmission. Our experiments demonstrate that our method achieves a 9x speedup compared to the original causal product function when implemented on a CPU.

Citation Download Citation

Siqi Liu "Optimized parallel causal product for linear transformer", Proc. SPIE 12717, 3rd International Conference on Artificial Intelligence, Automation, and High-Performance Computing (AIAHPC 2023), 127172K (21 July 2023); https://doi.org/10.1117/12.2684707

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available