主论坛演讲：从模型到市场：在云上部署开源模型的缺失环节是什么？| KC23

Keynote: From Models to Market: What's the Missing Link in Scaling Open Source Models on... Fog Dong

主论坛演讲：从模型到市场：在云上部署开源模型的缺失环节是什么？| Keynote: From Models to Market: What's the Missing Link in Scaling Open Source Models on Cloud? - Fog Dong, Senior Engineer, BentoML

开源模型在人工智能的领域里带来了前所未有的可能性。它们催化了创新的发展，并为潜在的突破性产品铺就了基础。然而，从利用这些模型到将其部署为云上的完整产品，这个过程充满了难题。对于像 LLM 这样的大型模型，如何以快速、高效且低成本的方式提供模型服务也是一大挑战。在这种情境下，BentoML 是一个可靠的解决方案。这个开源框架提供了一种封装模型为微服务的统一方法，有助于在理想的硬件类型上独立部署和扩展模型，从而简化了云部署和扩展的过程。它将从模型构建，到提供服务，再到在云上扩展的艰难过程，转变为一个流程化的步骤。在这次的演讲中，我们将以 OpenLLM 为例，深入剖析这个过程的细节。OpenLLM 是 BentoML 的一个生态项目，专为提供 LLM 服务而设计，它集成了各种开源 LLM 模型，是从模型转化为产品的理想典范。我们将探究从使用开源模型到在云上成功部署和扩展的全程，并分享在这个过程中获得的收货与经验。

Open-source models have ushered in a new era of possibilities in the world of AI. They serve as a catalyst for innovation and form the foundation of potentially groundbreaking products. However, the transition from using these models to deploying them as full-fledged products on the cloud is riddled with complexities. When it comes to large models like LLMs, challenges also include how to serve the models in a fast, efficient, and low-cost way. BentoML emerges as a beacon in this scenario. This open-source framework provides a unified approach to encapsulate models into microservices, which can help independently deploy and scale models on ideal hardware types, thereby simplifying cloud deployment and scaling. It turns the formidable journey from model building to serving, and ultimately, to scaling on the cloud, into a streamlined process. In this presentation, we'll delve into the intricacies of this process using OpenLLM as an example. OpenLLM, an ecosystem project in BentoML for efficient serving of LLMs, with its integration of various open-source LLM models, serves as an ideal exemplar of transforming from model to product. We'll explore the trajectory from using open-source models to their successful deployment and scaling in the cloud, pinpointing the gaps that exist along this pathway. Alongside this, we'll share the victories achieved and the lessons learned during this journey.

CNCF概况（幻灯片）

扫描二维码联系我们！

CNCF (Cloud Native Computing Foundation)成立于2015年12月，隶属于Linux  Foundation，是非营利性组织。 

CNCF（云原生计算基金会）致力于培育和维护一个厂商中立的开源生态系统，来推广云原生技术。我们通过将最前沿的模式民主化，让这些创新为大众所用。请关注CNCF微信公众号。