Scenario-distilling AI Benchmarking

Modern real-world application scenarios like Internet services not only consist of diversity of AI and non-AI modules with very long and complex execution paths, but also have huge code size, which raises serious benchmarking or evaluating challenges. Using AI components or micro benchmarks alone can lead to error-prone conclusions. This paper presents a scenario-distilling methodology to attack the above challenge. We formalize a real-world application scenario as a Directed Acyclic Graph-based model, and propose the rules to distill it into the permutation of essential AI and non-AI tasks as a high-level scenario benchmark specification. Together with seventeen industry partners, we extract nine typical application scenarios, and identify the primary components. We design and implement a highly extensible, configurable, and flexible benchmark framework, on the basis of which, we implement two Internet service AI scenario benchmarks as proxies to two real-world application scenarios. We claim scenario, component and micro benchmarks should be considered as three indispensable parts for evaluating. Our evaluation shows the advantage of our methodology against using component or micro AI benchmarks alone. The specifications, source code, testbed, and results are publicly available from \url{this https URL}.