构建支持无限时间序列的Prometheus高性能远程存储方案 | KC23

Build a High Performance Remote Storage for Prometheus with Unlimited Time Series - Yang Xiang

构建支持无限时间序列的Prometheus高性能远程存储方案 | Build a High Performance Remote Storage for Prometheus with Unlimited Time Series - Yang Xiang, Yunshan Networks, Inc.

云原生开发人员希望将高维度、高基数的标签注入到指标中，但这对Prometheus来说是一个严峻的挑战。社区中有大量的TSDB提供远程存储解决方案，但所有这些解决方案都存在高基数问题。还有基于列式数据库的解决方案，但它们的性能无法与TSDB相媲美。我们发现Prometheus中的指标标签可以分为两种类型：由prometheus-server注入的目标标签和开发人员在应用代码中注入的应用标签。我们首先将所有目标标签编码到一列中，然后使用ClickHouse的列式存储能力将所有应用标签展平为独立的列。我们实现了与VictoriaMetrics相当的性能，甚至在查询大型时间序列时提升了近10倍。最重要的是，它支持无限数量的时间线 - 实际上不再有时间序列的概念。

Cloud-native developers hope to inject high-dimensional, high-cardinality tags into metrics, but this poses a serious challenge to Prometheus. A large number of TSDBs in the community provide remote-storage solutions, but all of these solutions suffer from high cardinality problems without exception. There are also solutions based on columnar databases, but they do not achieve comparable performance to TSDB. We found that the labels of metrics in Prometheus can be divided into two types: target-label injected by prometheus-server, and app-label injected by developers in application code. We first encode all target-labels into one column, and then use ClickHouse's columnar storage capability to flatten all app-labels into independent columns. We achieved performance comparable to VictoriaMetrics, and even had nearly 10x improvement when querying large time series. Most importantly, it supports an unlimited number of timelines - in fact there is no concept of time series anymore.

CNCF概况（幻灯片）

扫描二维码联系我们！

CNCF (Cloud Native Computing Foundation)成立于2015年12月，隶属于Linux  Foundation，是非营利性组织。 

CNCF（云原生计算基金会）致力于培育和维护一个厂商中立的开源生态系统，来推广云原生技术。我们通过将最前沿的模式民主化，让这些创新为大众所用。请关注CNCF微信公众号。