Enhancing cloud-based large language model processing with Elasticsearch and transformer models

Chunhe Ni; Jiang Wu; Hongbo Wang; Wenran Lu; Chenwei Zhang

doi:10.1117/12.3033606

13 June 2024 Enhancing cloud-based large language model processing with Elasticsearch and transformer models

Chunhe Ni, Jiang Wu, Hongbo Wang, Wenran Lu, Chenwei Zhang

Proceedings Volume 13180, International Conference on Image, Signal Processing, and Pattern Recognition (ISPP 2024); 131806M (2024) https://doi.org/10.1117/12.3033606
Event: International Conference on Image, Signal Processing, and Pattern Recognition (ISPP 2024), 2024, Guangzhou, China

Abstract

Large Language Models (LLMs) are a class of generative AI models built using the Transformer network, capable of leveraging vast datasets to identify, summarize, translate, predict, and generate language. LLMs promise to revolutionize society, yet training these foundational models poses immense challenges. Semantic/vector search within large language models is a potent technique that can significantly enhance search result accuracy and relevance. Unlike traditional keyword-based search methods, semantic search utilizes the meaning and context of words to grasp the intent behind queries and deliver more precise outcomes. Elasticsearch emerges as one of the most popular tools for implementing semantic search—an exceptionally scalable and robust search engine designed for indexing and searching extensive datasets. In this article, we delve into the fundamentals of semantic search and explore how to harness models to bolster large language model processing paradigms. We gain a comprehensive understanding of semantic search principles and acquire practical skills for implementing semantic search in real-world model application scenarios.

(2024) Published by SPIE. Downloading of the abstract is permitted for personal use only.

Citation Download Citation

Chunhe Ni, Jiang Wu, Hongbo Wang, Wenran Lu, and Chenwei Zhang "Enhancing cloud-based large language model processing with Elasticsearch and transformer models", Proc. SPIE 13180, International Conference on Image, Signal Processing, and Pattern Recognition (ISPP 2024), 131806M (13 June 2024); https://doi.org/10.1117/12.3033606

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available