Leveraging Code Embeddings to Identify and Address Blind Spots in Benchmark Creation

Authors

• Charles Moloney, University of Nebraska-LincolnFollow

First Advisor

Robert Dyer

Committee Members

Seth Polsley, Hamid Bagheri

Date of this Version

5-2026

Document Type

Thesis

Citation

A thesis presented to the faculty of the Graduate College at the University of Nebraska in partial fulfilment of requirements for the degree of Master of Science

Major: Computer Science

Under the supervision of Professor Robert Dyer

Lincoln, Nebraska, May 2026

Comments

Copyright 2026, Charles Moloney. Used by permission

Abstract

Formal software verification remains critical for early vulnerability detection, yet benchmarking these tools is costly and often reliant on centralized datasets such as SV-COMP. While such repositories enable standardized evaluation, they introduce risks of overfitting and bias, particularly due to first-party benchmark contributions. To address these limitations, we extend ARG-V, our tool for generating SV-COMP-compatible benchmarks from real-world Java code, with a novel approach of using code embedding techniques to selectively sample from mined code. By leveraging Nomic Embed Code and a cosine-based Minimum Hyperspherical Energy (MHE) objective, we systematically select and transform benchmarks from scraped GitHub code that maximize embedding diversity and minimize redundancy with existing corpora. From 5,100 candidate files, we identify 86 new benchmarks and evaluate them against the state-of-the-art in Java verification. We demonstrate that the proposed benchmarks significantly increase verification difficulty for all four leading Java verifiers at SV-COMP 2026, observing a notable decline in verifier accuracy for all verifiers across assertion safety and exception detection tasks. Qualitative analysis further demonstrates that the benchmarks selected with MHE address numerous ``blind spots'' for Java features in the SV-COMP corpus, including lambdas, streams, and GUI libraries. These findings highlight the potential of embedding-driven benchmark selection to improve the diversity, realism, and evaluation rigor within software verification.

Advisor: Robert Dyer