We use a few cookies

Essential cookies keep the library working. With your permission, we'd also like to use analytics to see what's useful and improve the experience. Read our Privacy Policy or Cookie Policy.

Skip to main content
Home/e-Resources/Data Science & Analytics/Enabling Efficient and General Subpopulation Analytics in Multidimensional Data Streams
PeriodicalsData Science & Analytics

Enabling Efficient and General Subpopulation Analytics in Multidimensional Data Streams

arXiv2022

Checking your access to this record.

Abstract & Overview

Today's large-scale services (e.g., video streaming platforms, data centers, sensor grids) need diverse real-time summary statistics across multiple subpopulations of multidimensional datasets. However, state-of-the-art frameworks do not offer general and accurate analytics in real time at reasonable costs. The root cause is the combinatorial explosion of data subpopulations and the diversity of summary statistics we need to monitor simultaneously. We present Hydra, an efficient framework for multidimensional analytics that presents a novel combination of using a ``sketch of sketches'' to avoid the overhead of monitoring exponentially-many subpopulations and universal sketching to ensure accurate estimates for multiple statistics. We build Hydra as an Apache Spark plugin and address practical system challenges to minimize overheads at scale. Across multiple real-world and synthetic multidimensional datasets, we show that Hydra can achieve robust error bounds and is an order of magnitude more efficient in terms of operational cost and memory footprint than existing frameworks (e.g., Spark, Druid) while ensuring interactive estimation times.

Cite this record

Antonis Manousis, Zhuo Cheng, Ran Ben Basat, Zaoxing Liu, Vyas Sekar (2022). Enabling Efficient and General Subpopulation Analytics in Multidimensional Data Streams. arXiv. https://library.shubhaarti.com/content/c2822055-cf58-479c-b116-b4be10284ce1

Copy in another style

Manousis, A., Cheng, Z., Basat, R. B., Liu, Z., & Sekar, V. (2022). Enabling Efficient and General Subpopulation Analytics in Multidimensional Data Streams. arXiv. https://library.shubhaarti.com/content/c2822055-cf58-479c-b116-b4be10284ce1
Author names are used as deposited, so check their order before you submit.

Metadata is harvested from open-access indexes and can be incomplete. Check a citation against the published version before relying on it — see our open access & attribution policy.