We use a few cookies

Essential cookies keep the library working. With your permission, we'd also like to use analytics to see what's useful and improve the experience. Read our Privacy Policy or Cookie Policy.

Skip to main content
Home/e-Resources/Data Science & Analytics/BLADE: Benchmarking Language Model Agents for Data-Driven Science
PeriodicalsData Science & Analytics

BLADE: Benchmarking Language Model Agents for Data-Driven Science

arXiv2024

Checking your access to this record.

Abstract & Overview

Data-driven scientific discovery requires the iterative integration of scientific domain knowledge, statistical expertise, and an understanding of data semantics to make nuanced analytical decisions, e.g., about which variables, transformations, and statistical models to consider. LM-based agents equipped with planning, memory, and code execution capabilities have the potential to support data-driven science. However, evaluating agents on such open-ended tasks is challenging due to multiple valid approaches, partially correct steps, and different ways to express the same decisions. To address these challenges, we present BLADE, a benchmark to automatically evaluate agents' multifaceted approaches to open-ended research questions. BLADE consists of 12 datasets and research questions drawn from existing scientific literature, with ground truth collected from independent analyses by expert data scientists and researchers. To automatically evaluate agent responses, we developed corresponding computational methods to match different representations of analyses to this ground truth. Though language models possess considerable world knowledge, our evaluation shows that they are often limited to basic analyses. However, agents capable of interacting with the underlying data demonstrate improved, but still non-optimal, diversity in their analytical decision making. Our work enables the evaluation of agents for data-driven science and provides researchers deeper insights into agents' analysis approaches.

Cite this record

Ken Gu, Ruoxi Shang, Ruien Jiang, Keying Kuang, Richard-John Lin, Donghe Lyu, et al. (2024). BLADE: Benchmarking Language Model Agents for Data-Driven Science. arXiv. https://library.shubhaarti.com/content/41f10642-ca3d-4bf3-b610-255335fbe0f7

Copy in another style

Gu, K., Shang, R., Jiang, R., Kuang, K., Lin, R., Lyu, D., Mao, Y., Pan, Y., Wu, T., Yu, J., Zhang, Y., & Zhang, T. M. (2024). BLADE: Benchmarking Language Model Agents for Data-Driven Science. arXiv. https://library.shubhaarti.com/content/41f10642-ca3d-4bf3-b610-255335fbe0f7
Author names are used as deposited, so check their order before you submit.

Metadata is harvested from open-access indexes and can be incomplete. Check a citation against the published version before relying on it — see our open access & attribution policy.