Robustness & Adversarial IR

Graph-based retrieval systems leverage structural and semantic rela- tionships among entities to enable context-aware search. However, their reliance on graph structure renders them vulnerable to adversarial perturbations of graph nodes and edges that distort embeddings and alter ranking outcomes. This paper introduces an approach for adversarial edge removal for targeting graph-based retrieval. We first establish that conventional structural heuristics, such as cen- trality degrees or PageRank, are non-deterministic predictors of rank degradation under edge perturbations, as embedding shifts depend on multi-hop spectral inter- actions rather than first-order topology. Building on this, we propose a learning- based estimator that models the mapping between local edge characteristics and their induced ranking distortion. The estimator, trained on observed perturba- tion–response pairs, enables efficient identification of high-impact edges within a constrained budget, operating in a black-box setting. Experiments on bench- mark graph retrieval datasets demonstrate that the proposed framework achieves stronger and more efficient rank demotion than state-of-the-art baselines

Recent research has shown that neural information retrieval techniques may be susceptible to adversarial attacks. Adversarial attacks seek to manipulate the ranking of documents, with the intention of exposing users to targeted content. In this paper, we introduce the Embedding Perturbation Rank Attack (EMPRA) method, a novel approach designed to perform adversarial attacks on black-box Neural Ranking Models (NRMs). EMPRA manipulates sentence-level embeddings, guiding them towards pertinent context related to the query while preserving semantic integrity. This process generates adversarial texts that seamlessly integrate with the original content and remain imperceptible to humans. Our extensive evaluation conducted on the widely-used MS MARCO V1 passage collection demonstrate the effectiveness of EMPRA against a wide range of state-of-the-art baselines in promoting a specific set of target documents within a given ranked results. Specifically, EMPRA successfully achieves a re-ranking of almost 96% of target documents originally ranked between 51-100 to rank within the top 10. Furthermore, EMPRA does not rely on surrogate models for generating adversarial documents, enhancing its robustness against various victim NRMs in realistic settings.

Modern information retrieval is increasingly implemented as a pipeline in which a retriever selects documents, an LLM synthesizes an answer grounded in the retrieved documents, and an LLM-based judge scores relevance or quality. In this setting, a poor outcome is hard to attribute because failure may originate in retrieval, generation, or judgment, and these failures are typically analyzed in isolation. This paper studies query failure across these three tasks through a unified operationalization of hard queries per task. Using four years of TREC Deep Learning benchmarks from 2019 to 2022, we define , , and query sets, and analyze their overlap, their robustness across retrievers, generators, and judging setups, and the query characteristics associated with each failure. We find that hard queries overlap only weakly across tasks, indicating that difficulty does not transfer reliably between retrieval, generation, and judgment. At the same time, while the specific hard queries vary by model, the overlap structure is stable across system choices, suggesting that difficulty is driven more by query characteristics interacting with task-specific constraints than by model-specific effects. We further induce a data-driven typology of roots of failure (textitdifficulty cues) and show that they can be used to improve system behavior. We further demonstrate that conditioning generation on task-relevant difficulty cues yields consistent gains in answer quality.

Graph-based retrieval systems rely heavily on structural dependencies, making them vulnerable to adversarial manipulation. We present a black-box graph poisoning attack that degrades a target node’s ranking using only edge deletions, without access to model parameters, gradients, or retraining. Prior heuristic methods treat edge influence as a static property and fail to capture how an edge’s impact varies with local neighborhood structure. We address this limitation by modeling edge influence as context-dependent. Our method samples multiple ego-networks around a target node, measures empirical utility changes from edge ablations, and trains a local scorer to predict context-sensitive edge effects. At inference, predictions are aggregated across sampled subgraphs to yield stable deletion decisions. Experiments on standard benchmarks show that this approach consistently outperforms existing black-box and white-box baselines across perturbation budgets while remaining model-agnostic.

Neural Ranking Models (NRMs) are central to modern information retrieval but remain highly vulnerable to adversarial manipulation. Existing attacks often rely on heuristics or surrogate models, limiting effectiveness and transferability. We propose CRAFT, a supervised framework for black-box adversarial rank attacks powered by large language models (LLMs). CRAFT operates in three stages: adversarial dataset generation via retrieval-augmented generation and self-refinement, supervised fine-tuning on curated adversarial examples, and preference-guided optimization to align generations with rank-promotion objectives. Extensive experiments on the MS~MARCO passage dataset, TREC Deep Learning 2019, and TREC Deep Learning 2020 benchmarks show that CRAFT significantly outperforms state-of-the-art baselines, achieving higher promotion rates and rank boosts while preserving fluency and semantic fidelity. Moreover, CRAFT transfers effectively across diverse ranking architectures, including cross-encoder, embedding-based, and LLM-based rankers, underscoring vulnerabilities in real-world retrieval systems. This work provides a principled framework for studying adversarial threats in NRMs, underscores the risks of generative AI in rank manipulation, and provides a foundation for developing more robust retrieval systems. To support reproducibility, we publicly release our source code, trained models, and prompt templates.

No abstract available for this paper.

We introduce a novel resource for adversarial rank attacks against neural rankers designed to support systematic research on the robustness of neural retrieval models. Existing adversarial methods for ranking are often unsupervised, rely on surrogate models, and lack ground-truth supervision. Our dataset addresses these limitations by leveraging Retrieval-Augmented Generation (RAG) with a Large Language Model (LLM) to construct high-quality adversarial examples that subtly manipulate document rankings while preserving linguistic coherence and indirect relevance. The dataset is generated through a self-refining LLM–Ranker feedback loop and released in two variants, Gold and Diamond, categorized by attack effectiveness. We provide comprehensive metadata, insertion points, ranking labels, and quality metrics (fluency, acceptability) for each instance. Accompanied by code and LLM prompts, our resource supports both training and evaluation of adversarial models and can serve as a benchmark for ranking robustness. This work offers a reproducible and extensible foundation for the development of robust retrieval systems and adversarial information retrieval methods.

Neural ranking models have become the backbone of modern information retrieval systems, yet they remain vulnerable to adversarial manipulation. This paper introduces Few-Shot Adversarial Prompting (FSAP), a novel framework that leverages large language models (LLMs) to generate harmful, high-ranking adversarial documents without access to model gradients or internal states. Unlike prior attacks that modify existing documents or rely on handcrafted templates, FSAP exploits in-context learning to synthesize realistic adversarial documents conditioned on a small support set of previously seen harmful examples. We propose two variants: FSAPIntraQ, which uses examples from the same query, and FSAPInterQ, which transfers adversarial patterns across unrelated topics. Through comprehensive evaluation on the TREC 2020 and TREC 2021 Health Misinformation Tracks and across four neural rankers, we show that FSAP achieves superior attack effectiveness, strong stance fidelity, and high undetectability. Our findings demonstrate that FSAP generalizes across different LLMs, posing a transferable and scalable threat model for neural retrieval systems.

Ad hoc retrieval, a cornerstone task in Information Retrieval (IR), aims to rank documents in response to a user’s query, often without prior knowledge of the user’s specific information need. While transformer-based neural rankers have achieved state-of-the-art performance in ad hoc retrieval, their effectiveness varies significantly across queries. Certain queries—commonly referred to as hard queries—remain particularly challenging, highlighting critical gaps in retrieval models. Identifying these hard queries is essential for improving retrieval systems, motivating the task of Query Performance Prediction (QPP), which aims to estimate the effectiveness of a query without requiring access to relevance judgments. In this paper, we propose Context-Aware Query Performance Prediction (CA–QPP), a novel post-retrieval QPP method, which builds on the foundations of perturbation-based QPP methods that hypothesize a relationship between query sensitivity to small perturbations and query retrieval effectiveness. Building on this foundation, our approach exposes the given query to perturbations by constructing two query variations: an effective variation emphasizing terms that enhance retrieval and an ineffective variation accentuating terms that hinder it. By contrasting the retrieval outcomes of these variations using a cross-encoder model, CA–QPP captures the interplay of term contributions and predicts the performance for the given query. We evaluate CA–QPP on the widely used MS MARCO datasets and their associated query sets, including TREC DL 2019, TREC DL 2020, DL-Hard, TREC DL 2021, and TREC DL 2022, which feature extensive human-labeled relevance judgments. Our experiments demonstrate that CA–QPP consistently outperforms traditional and neural-based QPP baselines across standard correlation metrics, including Pearson’s ρ, Kendall’s τ, and Spearman’s ρ. Through a detailed case study, we further illustrate the mechanics of CA–QPP and provide empirical evidence for its ability to model the contextual impact of individual query terms, making it a robust framework for query performance prediction.

Dense retrieval models provide representations in the form of embeddings in latent space and output a single deterministic score for a document based on the estimation of its relevance to the input query. While remarkable progress has been achieved in dense retrieval methods, they are limited by the fact that they consider queries and documents as deterministic points in latent space that encode the most likely features of the given query or document, and hence do not explicitly encode any degrees of noise, ambiguity or uncertainty. In this paper, we build on existing strong transformer-based dense retrievers by enabling them to capture uncertainty in latent space. In our proposed approach, embeddings in latent space are no longer a deterministic point, but rather a probabilistic distribution. With such probabilistic embeddings, the dense retrievers can be trained to achieve competitive performance on in-distribution queries and higher generalizability on out-of-distribution queries. Based on extensive experiments, we demonstrate that our proposed model consistently improves retrieval effectiveness in comparison to the state-of-the-art dense retrieval methods.

The varying performance of information retrieval (IR) methods, including state-of-the-art transformer-based neural retrievers, across diverse queries poses a significant challenge for achieving robust and reliable retrieval effectiveness. Query Performance Prediction (QPP) seeks to estimate the effectiveness of a retrieval method for individual queries, enabling adaptive strategies to improve retrieval outcomes, particularly for challenging queries. However, existing QPP approaches face fundamental challenges: pre-retrieval methods often rely on surface-level query features that fail to capture the nuanced relationship between queries and retrieval effectiveness, while post-retrieval methods depend heavily on the quality of retrieved documents, which can be unreliable for difficult queries. To this end, we propose the Query Space Distance-Based QPP (QSD-QPP) framework, which leverages the deterministic and consistent behavior of retrieval methods to estimate query performance by referencing historical queries with known effectiveness. The approach is motivated by the observation that semantically or syntactically similar queries often exhibit consistent retrieval performance, a property that can be exploited to make reliable predictions for unseen queries. QSD-QPP operates in two modes: (1) a lightweight pre-retrieval instantiation that dynamically constructs a query subspace based on embedding distances to interpolate the performance of proximate historical queries, and (2) an enriched post-retrieval instantiation that incorporates contextualized embeddings, document interactions, and historical query associations to enhance prediction accuracy. By utilizing large-scale contextualized embeddings derived from pre-trained language models, QSD-QPP efficiently identifies semantically similar queries and leverages their performance for robust predictions. By addressing the inherent limitations of prior approaches, QSD-QPP achieves a balanced trade-off between computational efficiency, prediction accuracy, and scalability. We evaluate QSD-QPP on four benchmark datasets, including MS MARCO Dev and TREC Deep Learning tracks (2019, 2020, and DL-Hard), demonstrating its superior accuracy and robustness compared to state-of-the-art baselines in both pre-retrieval and post-retrieval QPP tasks. To ensure reproducibility and encourage further research, we publicly release the implementation of our work.

The task of searching over large keyword graphs aims to identify a subgraph where the nodes collectively cover the input query keywords. Although finding an exact solution to this problem is NP-hard, we address it by proposing a novel graph neural network representation learning technique specifically tailored for graphs with missing information. We propose a novel keyword graph representation learning method that incorporates complementary aspects of graphs: global, local, adjusted, and feature semantics. Considering these multiple aspects, our approach remains robust and resilient to missing information. We adopt and fine- tune a transformer-based model to aggregate the various features of a graph to generate rich representations, recognizing the pivotal role of keywords in this task. We show through experiments on real-world data that our method outperforms the state-of-the-art approaches and is particularly robust in the face of missing values, underscoring its ability to effectively handle incomplete graphs.

No abstract available for this paper.

This paper introduces ADG-QPP (Adaptive Disturbance Generation), an unsupervised Query Performance Prediction (QPP) method designed specifically for dense neural retrievers. The underlying foundation of ADG-QPP is to measure query performance based on its degree of robustness towards perturbations. Traditional QPP methods rely on predefined lexical perturbations on the query, which only apply to sparse retrieval methods and fail to maintain consistent performance across different datasets. In our work, we address these limitations by perturbing the query by injecting disturbance leveraged by the focal network-based measurements including node-based, edge-based and cluster-based metrics, into its neural embedding representation. Rather than applying the same perturbation across all queries, our approach develops an instance-wise disturbance for each query that is then used for its perturbation. Through extensive experiments on three benchmark datasets, we demonstrate that ADG-QPP outperforms state-of-the-art baselines in terms of Kendall tau, Spearman rho, and Pearson’s rho correlations.

Query Performance Prediction (QPP), is concerned with assessing the retrieval quality of a ranking method for an input query. Most traditional unsupervised frequency-based models and many recent supervised neural methods have been designed specifically for predicting the performance of sparse retrievers such as BM25. In this paper we propose an unsupervised QPP method for dense neural retrievers which operates by redefining the well-known concept of query robustness i.e., a more robust query to perturbations is an easier query to handle. We propose to generate query perturbations for measuring query robustness by systematically injecting noise into the contextualized neural representation of each query. We then compare the retrieved list for the original query with that of the perturbed query as a way to measure query robustness. Our experiments on four different query sets including MS MARCO, TREC Deep Learning track 2019 and 2020 and TREC DL-Hard show consistently improved performance on linear and ranking correlation metrics over the state of the art.

No abstract available for this paper.

Bitcoin mining is the process of generating new blocks in the Bitcoin blockchain. This process is vulnerable to different types of attacks. One of the most famous attacks in this category is selfish mining. This attack is essentially a strategy that a sufficiently powerful mining pool can follow to obtain more revenue than its fair share. The reason that selfish mining is effective is the difficulty adjustment algorithm used in the Bitcoin network. In this paper, we analyze the profitability of selfish mining with respect to time and propose an alternative difficulty adjustment algorithm that discourages selfish mining while allowing the Bitcoin network to remain scalable. We analyze our proposed solution, present the results, and discuss its effectiveness. Based on our analysis, our proposed algorithm effectively increases the profitability waiting time for the attackers to almost double its original value. For example, for a miner with 40% of the network's hash power, the algorithm extends the waiting time from 4 weeks to more than 11 weeks. This will discourage attackers from performing their malicious activities. We also show that our proposed algorithm allows the network to scale while it increases the waiting time.