Neural Ranking & Retrieval Models

Dense neural retrievers have improved retrieval effectiveness but can also amplify social biases in ranked results. This paper investigates gender bias in retrieval systems and introduces a fairness-aware training approach that regularizes standard ranking losses with bias and fairness terms. The formulation applies penalty or reward signals at the document level within pairwise objectives, enabling a tunable trade-off between effectiveness and fairness. We evaluate the approach on MS MARCO-derived benchmarks using two encoders (BERT-mini and ELECTRA-small) and two query sets (gender-neutral and socially sensitive). Across ARaB, LIWC, and NFaiRR, our best configurations substantially reduce gender bias while preserving MRR@10 within small to moderate deltas, and in some cases improving effectiveness. We also compare against fairness-aware baselines such as adversarial and neutrality-regularized rankers and find competitive or superior bias reduction under comparable effectiveness. The findings are empirical and scoped to binary gender bias in English on the evaluated datasets and models, without claims of broader generality.

Large Language Models (LLMs) have recently been widely adopted for query reformulation and expansion in Information Retrieval, with numerous studies reporting substantial effectiveness gains. However, these results are typically obtained under heterogeneous experimental conditions, making it difficult to assess which findings are reproducible and which depend on specific implementation choices. In this work, we present a systematic reproducibility and comparative study of ten representative LLM-based query reformulation methods under a unified and strictly controlled experimental framework. We evaluate methods across two architectural LLM families at two parameter scales, three retrieval paradigms (lexical, learned sparse, and dense), and nine benchmark datasets spanning TREC Deep Learning and BEIR. All decoding parameters, preprocessing pipelines, indexing configurations, and evaluation protocols are held constant to isolate the effect of the reformulation strategy itself. Our results show that reformulation gains are strongly conditioned on the retrieval paradigm, that improvements observed under lexical retrieval do not consistently transfer to neural retrievers, and that larger LLMs do not uniformly yield better downstream performance. These findings clarify the stability and limits of reported gains in prior work and provide a controlled benchmark to support future evaluation and replication efforts. To support transparency and future extensibility, we release all prompts, evaluation scripts, and experimental artifacts.

Forming effective expert teams is central to domains where solving complex problems requires diverse, complementary skills. However, automating this task is highly challenging due to sparse co-occurrence data, long-tailed expert participation, and the combinatorial complexity of unseen skill configurations. Existing graph-based, probabilistic, and neural approaches often struggle with generalization, fairness, and robustness, leading to biased selections that favor historically popular experts over more suitable candidates. To address these challenges, we propose a generative framework for expert team formation based on denoising diffusion probabilistic models. We cast team formation as skill-conditioned imputation (i.e., inpainting), where skills are treated as observed context and the expert component is generated via conditional diffusion sampling. This design enables our method to preserve semantic skill-expert alignment, mitigate data sparsity, and generate diverse yet contextually coherent teams. Extensive experiments on DBLP and DOTA2 datasets show that our model consistently outperforms state-of-the-art baselines, achieving over 3× higher recall (16.4% vs. 5.0%) and MAP (9.7% vs. 2.2%) on DBLP, while delivering more than 5× improvement in MRR (13.3% vs. 2.5%) on DOTA2. Fairness analysis further demonstrates that our method reduces average overlap with the top-100 most popular experts to 2.6, compared to 86.7 for the strongest baseline, and achieves near-optimal diversity with NDKL ≈ 0.1 under high non-popular expert ratios. For reproducibility purposes, we made our code and model publicly available at https://anonymous.4open.science/r/DiffTF-7A20

Recent query expansion methods increasingly rely on Large Language Models (LLMs) to generate reformulations conditioned on documents retrieved from the target corpus. These retrieved documents are provided to the language model as context and constitute the information available during reformulation generation. While prior work has explored diverse reformulation strategies and iterative retrieval-generation pipelines, comparatively little attention has been paid to the selection of the documents used to condition generation. Yet the quality of a reformulation depends not only on the language model itself, but also on the documents it receives as input. In this paper, we investigate whether improvements in query expansion can be achieved through better document selection rather than more sophisticated reformulation strategies. We propose textscEviQE, a query expansion framework that aggregates documents retrieved by multiple reformulation methods, selects a compact set of informative passages, and provides them to the language model to guide a single grounded reformulation step. This formulation treats reformulators as complementary retrieval perspectives rather than alternative query generators and explicitly separates document selection from reformulation generation. Experiments on three TREC DL collections and five BEIR benchmarks show that different reformulation methods frequently retrieve complementary relevant documents, allowing pooled candidate sets to achieve substantially higher relevant-document coverage than any individual reformulator. Across all benchmarks, textscEviQE consistently outperforms direct reformulation, cold-start expansion, and single-source seeded expansion methods. We further show that once high-quality conditioning documents have been identified, additional retrieval-generation iterations provide limited benefit and often reduce effectiveness.

Modern information retrieval is increasingly implemented as a pipeline in which a retriever selects documents, an LLM synthesizes an answer grounded in the retrieved documents, and an LLM-based judge scores relevance or quality. In this setting, a poor outcome is hard to attribute because failure may originate in retrieval, generation, or judgment, and these failures are typically analyzed in isolation. This paper studies query failure across these three tasks through a unified operationalization of hard queries per task. Using four years of TREC Deep Learning benchmarks from 2019 to 2022, we define , , and query sets, and analyze their overlap, their robustness across retrievers, generators, and judging setups, and the query characteristics associated with each failure. We find that hard queries overlap only weakly across tasks, indicating that difficulty does not transfer reliably between retrieval, generation, and judgment. At the same time, while the specific hard queries vary by model, the overlap structure is stable across system choices, suggesting that difficulty is driven more by query characteristics interacting with task-specific constraints than by model-specific effects. We further induce a data-driven typology of roots of failure (textitdifficulty cues) and show that they can be used to improve system behavior. We further demonstrate that conditioning generation on task-relevant difficulty cues yields consistent gains in answer quality.

In information retrieval (IR), learning-to-rank (LTR) methods have traditionally limited themselves to discriminative machine learning approaches that model the probability of the document being relevant to the query given some feature representation of the query-document pair. In this work, we propose an alternative denoising diffusion-based deep generative approach to LTR that instead models the full joint distribution over feature vectors and relevance labels. While in the discriminative setting, an over-parameterized ranking model may find different ways to fit the training data, we hypothesize that candidate solutions that can explain the full data distribution under the generative setting are better equipped to estimate relevance. With this motivation, we propose DiffusionRank that extends TabDiff, an existing denoising diffusion-based generative model for tabular datasets, to create generative equivalents of classical discriminative pointwise and pairwise LTR objectives. We conduct thorough empirical evaluation on four standard LTR datasets to demonstrate improvements from DiffusionRank models over their discriminative counterparts. Our work points to a rich space for future research exploration on how we can leverage ongoing advancements in deep generative modeling approaches, such as diffusion, for learning-to-rank in IR.

Neural rankers are typically trained with surrogate objectives because target evaluation metrics such as Normalized Discounted Cumulative Gain () are non-differentiable. Existing approaches rely on hand-designed surrogate losses or fixed smooth relaxations, which can be misaligned with the target metric or impose rigid inductive biases. More recent efforts to learn the loss itself (e.g., NeuralLoss) require multi-stage pretraining on synthetic data and may not match real-world relevance distributions. We introduce LearnDCG, a differentiable and learnable approximation of that unifies ranker and loss optimization within a single end-to-end pipeline. LearnDCG parameterizes the gain, discount, and temperature components of and learns them jointly with the ranker, enabling adaptation to dataset-specific relevance patterns while preserving the theoretical guarantees of differentiable ranking formulations. Experiments on MQ2007, MQ2008, WEB10K, and YLTR show that LearnDCG consistently outperforms classical surrogate losses, differentiable relaxations, and learnable alternatives. Beyond improved effectiveness, LearnDCG eliminates synthetic pretraining and reduces training complexity, achieving state-of-the-art results with improved efficiency. Gains are statistically significant across diverse architectures, from MLPs to transformer-based rankers, demonstrating the generality of the approach.

The emergence of large language models and agentic search systems is transforming query expansion, elevating it from a standalone preprocessing step to a central reasoning component in modern retrieval pipelines. The shift from classical term-selection methods to LLM-based generation has opened new possibilities for expansion, while deep search agents, retrieval-augmented generation, and conversational retrieval have introduced fundamentally new settings in which query expansion must operate. This tutorial offers a comprehensive treatment of query expansion methods, tracing their progression from classical statistical techniques through neural and transformer-based approaches to LLM-based generation, while covering evaluation methodology, cross-paradigm deployment, failure analysis, and emerging research opportunities. The tutorial is accompanied by hands-on experimentation for implementing and comparing expansion methods across retrieval settings.

We present QueryGym, a lightweight, extensible Python toolkit that supports large language model (LLM)-based query reformulation. This is an important tool development since recent work on llm-based query reformulation has shown notable increase in retrieval effectiveness. However, while different authors have sporadically shared the implementation of their methods, there is no unified toolkit that provides a consistent implementation of such methods, which hinders fair comparison, rapid experimentation, consistent benchmarking and reliable deployment. QueryGym addresses this gap by providing a unified framework for implementing, executing, and comparing llm-based reformulation methods. The toolkit offers: (1) a Python API for applying diverse LLM-based methods, (2) a retrieval-agnostic interface supporting integration with backends such as Pyserini and PyTerrier, (3) a centralized prompt management system with versioning and metadata tracking, (4) built-in support for benchmarks like BEIR and MS MARCO, and (5) a completely open-source extensible implementation available to all researchers. QueryGym is publicly available at https://github.com/radinhamidi/QueryGym.

Pre-retrieval query performance prediction methods estimate the effectiveness of a query prior to retrieval and enable control of the search pipeline through tasks such as query routing, selective expansion, and retriever selection. However, because they operate solely on query features without access to retrieved documents, pre-retrieval predictors often struggle to provide accurate predictions. They typically treat query difficulty as a latent statistical property rather than one that can be reasoned about. We propose a new pre-retrieval framework that leverages the reasoning capabilities of large language models (LLMs) to uncover and encode interpretable reasons of query difficulty. By integrating these reasons with transformer-based neural regressors, our approach allows the predictor to be both aware of and conditioned on reasons that make a query challenging. Our experiments on MS MARCO Dev, TREC DL 2019, TREC DL 2020, and DL-Hard demonstrate that our method consistently achieves reliable and stable correlations across various TREC DL datasets.

Retrieval-Augmented Generation (RAG) systems are commonly evaluated using document relevance, despite growing evidence that relevance alone does not accurately capture a document’s contribution to generation quality. Recent utility-based evaluation approaches address this limitation by measuring the effect of retrieved documents on the final generated answer; however, these methods treat generation as a black box and rely solely on output-level signals, overlooking the reasoning process through which language models utilize retrieved evidence. At the same time, retrieved documents may be relevant yet unused during reasoning, while other documents may directly support intermediate reasoning steps that lead to successful generation. We propose a reasoning-aware formulation of document utility that leverages chain-of-thought (CoT) traces as evidence of how retrieved documents contribute to the model’s reasoning process. Rather than estimating utility exclusively from final answer correctness, our approach characterizes utility through document usage within intermediate reasoning steps, enabling a finer-grained assessment of retrieved evidence. Through an empirical analysis of retrieved documents, reasoning traces, and answer quality, we demonstrate that reasoning-based utility captures aspects of document contribution that are not reflected by relevance alone, providing a promising foundation for utility-aware retrieval evaluation in RAG systems.

We present ReFormeR, a pattern-guided approach for query reformulation. Instead of prompting a language model to generate refor- mulations of a query directly, ReFormeR first elicits short reformulation patterns from pairs of initial queries and empirically stronger reformula- tions, consolidates them into a compact library of transferable reformu- lation patterns, and then selects an appropriate reformulation pattern for a new query given its retrieval context. The selected pattern con- strains query reformulation to controlled operations such as sense dis- ambiguation, vocabulary grounding, or discriminative facet addition, to name a few. As such, our proposed approach makes the reformulation policy explicit through these reformulation patterns, guiding the LLM towards targeted and effective query reformulations. Our extensive ex- periments on TREC DL 2019, DL 2020, and DL Hard show consistent improvements over classical feedback methods and recent LLM-based query reformulation and expansion approaches.

LLM-based query reformulation can improve retrieval effectiveness, but recent work shows that no single reformulation strategy is consistently optimal across queries, domains, retrievers, or model backbones. This creates an inference-time decision problem: textit``Given an original query and a pool of candidate reformulations, which one should be issued to the retriever?'' Existing studies of this problem are difficult to compare because they use different reformulator pools, retrievers, relevance signals, training labels, and evaluation metrics. We introduce sys, a resource and benchmark for reproducible query reformulation selection. sys freezes the expensive artifacts needed for this task: original queries, LLM-generated variants, ranked lists under multiple retrievers, retrieval scores, and per-query oracle labels. We revisit query variant selection, a long-studied IR problem, in the setting of LLM-based reformulation, and provide a shared evaluation setup for measuring both retrieval effectiveness and selection behavior. The benchmark spans high-resource web search, heterogeneous zero-shot retrieval, and reasoning-intensive retrieval through TREC DL, BEIR, and BRIGHT-style settings. We instantiate representative baselines from supervised classification, similarity and matrix-factorization routing, QPP, and LLM-as-judge selection. Our empirical characterization shows that query reformulation selection has substantial oracle headroom, indicating significant room for improving over fixed reformulation strategies. At the same time, current selectors recover only part of this potential: selector behavior is strongly conditioned on domain and retriever choice, and mean effectiveness can hide substantially different routing behavior. sys enables future query-reformulation selectors to be evaluated without regenerating variants, rerunning retrieval, or rebuilding judge pipelines. We have released the codebase, dataset, implementation details, and evaluation scripts.~footnoteurlhttps://github.com/haisonle001/QueryRoute

LLM-based retrievers and rerankers have advanced passage ranking, yet both paradigms interact with the corpus in a single pass and commit to the resulting candidate set, leaving relevant documents permanently unrecoverable once missed. We introduce SEEK, Self-Evaluative Exploration for Knowledge Retrieval, a training-free framework that addresses this limitation through iterative corpus interaction at test time. At each round, an LLM generates pseudo-passages conditioned on accumulated relevance feedback, a retriever surfaces fresh candidates, and a dedicated assessor assigns graded relevance judgments that guide subsequent rounds. On TREC Deep Learning, SEEK matches trained rerankers in ranking quality while consistently improving Recall@100 over single-pass BM25. On the reasoning-intensive BRIGHT benchmark, SEEK with Qwen2.5-7B achieves an 82% relative gain over BM25, surpassing all trained baselines, and SEEK with GPT-4.1 reaches 37.4 average nDCG@10, exceeding the strongest baseline by 37%. We make our code, prompts, and data publicly available at https://anonymous.4open.science/r/SEEK-E212/.

Pre-retrieval query performance prediction (QPP) estimates a query’s effectiveness before retrieval, enabling tasks such as query rout- ing. Prior pre-retrieval methods built on hand-crafted indicators treat queries independently and often scale poorly across collections and met- rics; recent learning approaches often optimize relative difficulty with- out calibrated absolute estimates and still rely on single-query evidence, while neighborhood models assume a query behaves like the average of its nearest neighbors and cannot exploit multi-hop structure or han- dle heterogeneous neighborhoods. We propose a relational formulation that treats effectiveness as a smooth function over a query–query affinity graph structure, learning how information should propagate across re- lated queries to produce context-aware predictions and extending to un- seen queries by situating them within this graph structure. Our proposed method consistently outperforms classical and learning baselines, includ- ing higher rank correlations on MS MARCO Dev (Spearman 𝜌 = 0.409) and DL-Hard (Spearman 𝜌 = 0.440), and competitive results on TREC DL 2020 (higher Pearson and Spearman), indicating that our approach provides an effective basis for QPP.

We propose a novel approach, referred to as Contrastive Disentangled Representation for Query Performance Prediction (textttCoDiR-QPP), to estimate search query performance by disentangling query content semantics from query difficulty. Our proposed approach leverages neural disentanglement to isolate the information need expressed in search queries from the complexities that affect retrieval performance. Motivated by empirical observations that varying query formulations for the same information need can significantly impact retrieval outcomes, we hypothesize that separating content semantics from query difficulty can enhance query performance prediction. Utilizing contrastive learning, textttCoDiR-QPP distinguishes between well-performing and poorly performing query variants, facilitating the estimation of a given query's performance. Our extensive experiments on four standard benchmark datasets demonstrate that textttCoDiR-QPP outperforms state-of-the-art baselines in predicting query performance, offering improved semantic similarity computation and higher correlation metrics such as Kendall $tau$, Spearman $rho$, and scaled Mean Absolute Ranking Error (sMARE).

No abstract available for this paper.

Large language Models (LLMs) are highly sensitive to variations in prompt formulation, which can significantly impact their ability to generate accurate responses. In this paper, we introduce a new task, Prompt Sensitivity Prediction, and a dataset PromptSET designed to investigate the effects of slight prompt variations on LLM performance. Using TriviaQA and HotpotQA datasets as the foundation of our work, we generate prompt variations and evaluate their effectiveness across multiple LLMs. We benchmark the prompt sensitivity prediction task employing state-of-the-art methods from related tasks, including LLM-based self-evaluation, text classification, and query performance prediction techniques. Our findings reveal that existing methods struggle to effectively address prompt sensitivity prediction, underscoring the need to understand how information needs should be phrased for accurate LLM responses.

No abstract available for this paper.

We introduce a novel resource for adversarial rank attacks against neural rankers designed to support systematic research on the robustness of neural retrieval models. Existing adversarial methods for ranking are often unsupervised, rely on surrogate models, and lack ground-truth supervision. Our dataset addresses these limitations by leveraging Retrieval-Augmented Generation (RAG) with a Large Language Model (LLM) to construct high-quality adversarial examples that subtly manipulate document rankings while preserving linguistic coherence and indirect relevance. The dataset is generated through a self-refining LLM–Ranker feedback loop and released in two variants, Gold and Diamond, categorized by attack effectiveness. We provide comprehensive metadata, insertion points, ranking labels, and quality metrics (fluency, acceptability) for each instance. Accompanied by code and LLM prompts, our resource supports both training and evaluation of adversarial models and can serve as a benchmark for ranking robustness. This work offers a reproducible and extensible foundation for the development of robust retrieval systems and adversarial information retrieval methods.

No abstract available for this paper.

Neural ranking models have become the backbone of modern information retrieval systems, yet they remain vulnerable to adversarial manipulation. This paper introduces Few-Shot Adversarial Prompting (FSAP), a novel framework that leverages large language models (LLMs) to generate harmful, high-ranking adversarial documents without access to model gradients or internal states. Unlike prior attacks that modify existing documents or rely on handcrafted templates, FSAP exploits in-context learning to synthesize realistic adversarial documents conditioned on a small support set of previously seen harmful examples. We propose two variants: FSAPIntraQ, which uses examples from the same query, and FSAPInterQ, which transfers adversarial patterns across unrelated topics. Through comprehensive evaluation on the TREC 2020 and TREC 2021 Health Misinformation Tracks and across four neural rankers, we show that FSAP achieves superior attack effectiveness, strong stance fidelity, and high undetectability. Our findings demonstrate that FSAP generalizes across different LLMs, posing a transferable and scalable threat model for neural retrieval systems.

Ad hoc retrieval, a cornerstone task in Information Retrieval (IR), aims to rank documents in response to a user’s query, often without prior knowledge of the user’s specific information need. While transformer-based neural rankers have achieved state-of-the-art performance in ad hoc retrieval, their effectiveness varies significantly across queries. Certain queries—commonly referred to as hard queries—remain particularly challenging, highlighting critical gaps in retrieval models. Identifying these hard queries is essential for improving retrieval systems, motivating the task of Query Performance Prediction (QPP), which aims to estimate the effectiveness of a query without requiring access to relevance judgments. In this paper, we propose Context-Aware Query Performance Prediction (CA–QPP), a novel post-retrieval QPP method, which builds on the foundations of perturbation-based QPP methods that hypothesize a relationship between query sensitivity to small perturbations and query retrieval effectiveness. Building on this foundation, our approach exposes the given query to perturbations by constructing two query variations: an effective variation emphasizing terms that enhance retrieval and an ineffective variation accentuating terms that hinder it. By contrasting the retrieval outcomes of these variations using a cross-encoder model, CA–QPP captures the interplay of term contributions and predicts the performance for the given query. We evaluate CA–QPP on the widely used MS MARCO datasets and their associated query sets, including TREC DL 2019, TREC DL 2020, DL-Hard, TREC DL 2021, and TREC DL 2022, which feature extensive human-labeled relevance judgments. Our experiments demonstrate that CA–QPP consistently outperforms traditional and neural-based QPP baselines across standard correlation metrics, including Pearson’s ρ, Kendall’s τ, and Spearman’s ρ. Through a detailed case study, we further illustrate the mechanics of CA–QPP and provide empirical evidence for its ability to model the contextual impact of individual query terms, making it a robust framework for query performance prediction.

Dense retrieval models provide representations in the form of embeddings in latent space and output a single deterministic score for a document based on the estimation of its relevance to the input query. While remarkable progress has been achieved in dense retrieval methods, they are limited by the fact that they consider queries and documents as deterministic points in latent space that encode the most likely features of the given query or document, and hence do not explicitly encode any degrees of noise, ambiguity or uncertainty. In this paper, we build on existing strong transformer-based dense retrievers by enabling them to capture uncertainty in latent space. In our proposed approach, embeddings in latent space are no longer a deterministic point, but rather a probabilistic distribution. With such probabilistic embeddings, the dense retrievers can be trained to achieve competitive performance on in-distribution queries and higher generalizability on out-of-distribution queries. Based on extensive experiments, we demonstrate that our proposed model consistently improves retrieval effectiveness in comparison to the state-of-the-art dense retrieval methods.

Entity retrieval plays a critical role in information access systems, yet the development and evaluation of retrieval models remain constrained by the limited availability of high-quality supervision. While recent work has demonstrated the utility of large language models (LLMs) as relevance assessors in passage and document retrieval, their reliability in the context of entity retrieval—where targets are abstract, underspecified, and often semantically sparse—remains unexplored. In this work, we evaluate LLM-based judgments against two complementary supervision signals: human-annotated relevance labels from the DBpedia-Entity benchmark and implicit feedback from user clicks in the LaQuE dataset. We show that LLMs exhibit strong agreement with expert annotations and replicate user click patterns with over 91% agreement, suggesting alignment with behavioral judgments despite noisy input queries. We further identify and analyze systematic mismatches for user clicks on irrelevant entities. Our findings establish LLMs not only as effective annotators for entity relevance judgment—even when given only the entity title—but also as powerful tools for predicting click-through behavior and simulating explainable user intent. Our code, prompts, and data are publicly available at: https://anonymous.4open.science/r/ClickLLM-E812

The varying performance of information retrieval (IR) methods, including state-of-the-art transformer-based neural retrievers, across diverse queries poses a significant challenge for achieving robust and reliable retrieval effectiveness. Query Performance Prediction (QPP) seeks to estimate the effectiveness of a retrieval method for individual queries, enabling adaptive strategies to improve retrieval outcomes, particularly for challenging queries. However, existing QPP approaches face fundamental challenges: pre-retrieval methods often rely on surface-level query features that fail to capture the nuanced relationship between queries and retrieval effectiveness, while post-retrieval methods depend heavily on the quality of retrieved documents, which can be unreliable for difficult queries. To this end, we propose the Query Space Distance-Based QPP (QSD-QPP) framework, which leverages the deterministic and consistent behavior of retrieval methods to estimate query performance by referencing historical queries with known effectiveness. The approach is motivated by the observation that semantically or syntactically similar queries often exhibit consistent retrieval performance, a property that can be exploited to make reliable predictions for unseen queries. QSD-QPP operates in two modes: (1) a lightweight pre-retrieval instantiation that dynamically constructs a query subspace based on embedding distances to interpolate the performance of proximate historical queries, and (2) an enriched post-retrieval instantiation that incorporates contextualized embeddings, document interactions, and historical query associations to enhance prediction accuracy. By utilizing large-scale contextualized embeddings derived from pre-trained language models, QSD-QPP efficiently identifies semantically similar queries and leverages their performance for robust predictions. By addressing the inherent limitations of prior approaches, QSD-QPP achieves a balanced trade-off between computational efficiency, prediction accuracy, and scalability. We evaluate QSD-QPP on four benchmark datasets, including MS MARCO Dev and TREC Deep Learning tracks (2019, 2020, and DL-Hard), demonstrating its superior accuracy and robustness compared to state-of-the-art baselines in both pre-retrieval and post-retrieval QPP tasks. To ensure reproducibility and encourage further research, we publicly release the implementation of our work.

No abstract available for this paper.

No abstract available for this paper.

This study investigates the formation of expert teams that collectively possess a specified skill set. While traditional methods have employed graph search techniques to identify subgraphs that meet skill requirements or neural architectures to map skills to experts, we introduce a novel approach that emphasizes both cohesive team dynamics and comprehensive skill coverage. Our retrieval-augmented generation model is designed to optimize the probability of successful collaboration among team members. Extensive experiments demonstrate that our proposed method significantly outperforms existing state-of-the-art approaches, offering a more effective solution for expert team formation.

The task of searching over large keyword graphs aims to identify a subgraph where the nodes collectively cover the input query keywords. Although finding an exact solution to this problem is NP-hard, we address it by proposing a novel graph neural network representation learning technique specifically tailored for graphs with missing information. We propose a novel keyword graph representation learning method that incorporates complementary aspects of graphs: global, local, adjusted, and feature semantics. Considering these multiple aspects, our approach remains robust and resilient to missing information. We adopt and fine- tune a transformer-based model to aggregate the various features of a graph to generate rich representations, recognizing the pivotal role of keywords in this task. We show through experiments on real-world data that our method outperforms the state-of-the-art approaches and is particularly robust in the face of missing values, underscoring its ability to effectively handle incomplete graphs.

No abstract available for this paper.

The problem of assembling effective expert teams based on project needs is central to expert networks such as LinkedIn. However, current team formation methods typically depend on keyword-matching techniques that fail to capture the nuanced semantics of natural lan- guage project descriptions. This results in inadequate modeling of re- quired expertise and suboptimal team selection. Addressing this gap, we propose a contextual, prompt-driven framework for team formation that infers latent expertise from rich textual descriptions of project goals. Our approach fine-tunes a T5-Large sequence-to-sequence model to trans- late project prompts into expert team compositions by benefiting from enhanced expertise annotations. To facilitate this task, we curate, and publicly release, a dataset based on DBLP V14 collection, augmented with high-confidence expertise labels generated by large language mod- els. Experimental results across multiple evaluation metrics show that our proposed model outperforms existing state-of-the-art baselines, un- derscoring the importance of contextualized representations in expert discovery and team assembly.

The field of AI carries inherent risks such as algorithmic biases, security vulnerabilities, and ethical concerns related to privacy and data protection. Despite these risks, AI holds significant promise for social good, with applications ranging from improved healthcare diagnostics to enhanced education strategies. Teaching AI ethics in postsecondary settings has emerged as one of the strategies to mitigate AI-related harms. The objectives of this review are to (1) synthesize existing research related to teaching postsecondary students about the principles and practice of ethics and AI, and (2) identify how educators are evaluating changes in student knowledge, skills, attitudes, and behaviors. This scoping review will follow the first five steps articulated by Arksey and O’Malley. A structured search strategy developed by an academic librarian incorporates three primary concept groups related to education, AI, and ethics. Database search strategies emphasize sensitivity rather than precision, given that a supervised machine learning tool will be used to assist in the identification of relevant abstracts. Searches will be conducted in the following academic databases: PubMed, Embase, Scopus, ERIC, LISTA, IEEE Xplore, APA PsycInfo, and ProQuest Dissertations and Theses. Results will include an up-to-date synthesis of the current state of AI ethics education in postsecondary curricula, evaluated teaching strategies, and potential outcomes associated with AI ethics education. Search results will be reported according to the PRISMA-ScR checklist. Data charting will focus on AI ethics pedagogy. This review will inform future research, policy development, and teaching practices, offering valuable insights for educators, policymakers, and researchers working towards responsible AI integration. Findings will contribute to enhanced understandings of the complexities of AI ethics education and have the potential to shape the ways trainees in multiple disciplines learn about the ethical dimensions of AI in practice. Order of Authors:

Gender bias is a pervasive issue that continues to influence various aspects of society, including the outcomes of information retrieval (IR) systems. As these systems become increasingly integral to accessing and navigating the vast amounts of information available today, the need to understand and mitigate gender bias within them is paramount. This book provides a comprehensive examination of the origins, manifestations, and consequences of gender bias in IR systems, as well as the current methodologies employed to address these biases. Theoretical frameworks surrounding gender and its representation in artificial intelligence (AI) systems are explored, particularly focusing on how traditional gender binaries are perpetuated and reinforced through data and algorithmic processes. Metrics and methodologies used to identify and measure gender bias within IR systems are then analyzed, offering a detailed evaluation of existing approaches and their limitations. Subsequent chapters address the sources of gender bias, including biased input queries, retrieval methods, and gold standard datasets. Various data-driven and method-level debiasing strategies are presented, including techniques for debiasing neural embeddings and algorithmic approaches aimed at reducing bias in IR system outputs. The book concludes with a discussion of the challenges and limitations faced by current debiasing efforts and provides insights into future research directions that could lead to more equitable and inclusive IR systems. This book serves as a valuable resource for researchers, practitioners, and students in the fields of information retrieval, artificial intelligence, and data science, providing the knowledge and tools needed to address gender bias and contribute to the development of fair and unbiased information systems.

No abstract available for this paper.

Research has already found that many retrieval methods are sensitive to the choice and order of terms that appear in a query, which can significantly impact retrieval effectiveness. We capitalize on this finding in order to predict the performance of a query. More specifically, we propose to learn query term difficulty weights specifically within the context of each query, which could then be used as indicators of whether each query term has the likelihood of making the query more effective or not. We show how such difficulty weights can be learnt through the finetuning of a language model. In addition, we propose an approach to integrate the learnt weights into a cross-encoder architecture to predict query performance. We show that our proposed approach shows a consistently strong performance prediction on the MSMARCO collection and its associated widely used Trec Deep Learning tracks query sets.

Prior research has demonstrated that reformulation of queries can significantly enhance retrieval effectiveness. With advancements in neural architectures, the task of query reformulation has evolved into a supervised translation problem, aimed at rewriting queries into more effective alternatives. Despite notable successes, identifying optimal reformulations that cover the same information need while enhancing retrieval effectiveness is still challenging. This paper introduces a two-step query reformulation framework for generating and selecting optimal target query variants which not only achieve higher retrieval performance but also preserve the original query's information need. Our comprehensive evaluations on the MS MARCO dataset and TREC Deep Learning tracks demonstrate substantial improvements over original query's performance.

The state-of-the-art query performance prediction methods rely on the fine-tuning of contextual language models to estimate retrieval effectiveness on a per-query basis. Our work in this paper builds on this strong foundation and proposes to learn rich query representations by learning the interactions between the query and two important contextual information, namely (1) the set of documents retrieved by that query, and (2) the set of similar historical queries with known retrieval effectiveness. We propose that such contextualized query representations can be more accurate estimators of query performance as they embed the performance of past similar queries and the semantics of the documents retrieved by the query. We perform extensive experiments on the MSMARCO collection and its accompanying query sets including MSMARCO Dev set and TREC Deep Learning tracks of 2019, 2020, 2021, and DL-Hard. Our experiments reveal that our proposed method shows robust and effective performance compared to state-of-the-art baselines.

Traditional evaluation metrics, such as NDCG, have long been the conventional approach for evaluating the effectiveness of information retrieval methods. However, such metrics are focused on absolute measures of effectiveness. While they allow us to compare the absolute performance of one retrieval method to another, we do not know if systems with similar absolute performance achieve this performance by finding the same items, or by finding different items with similar relevance grades. To address this problem, several recent proposals have measured the relative performance of a retrieval method in the context of the results from one or more other methods. In this paper, we address theoretical limitations of these proposals and introduce a new metric called Normalized Residual Gain (NRG) that can be seen as an extension of the underlying absolute metric, rather than as an entirely new metric. Operating in the context of the results retrieved by one or more other methods, NRG adjusts gain values according to the browsing model of the absolute metric. Through testing over the MS MARCO dev small and TREC DL 2019 datasets, we find that higher absolute effectiveness does not necessarily correlate with a higher NRG score, which will vary depending on context. In particular, in the context of modern neural models, NRG suggests that a traditional BM25 ranker continues to find relevant items missed by even the best neural models.

Background Systematic reviews (SRs) are being published at an accelerated rate. Decision-makers may struggle with comparing and choosing between multiple SRs on the same topic. We aimed to understand how healthcare decision-makers (eg, practitioners, policymakers, researchers) use SRs to inform decision-making and to explore the potential role of a proposed artificial intelligence (AI) tool to assist in critical appraisal and choosing among SRs. Methods We developed a survey with 21 open and closed questions. We followed a knowledge translation plan to disseminate the survey through social media and professional networks. Results Our survey response rate was lower than expected (7.9% of distributed emails). Of the 684 respondents, 58.2% identified as researchers, 37.1% as practitioners, 19.2% as students and 13.5% as policymakers. Respondents frequently sought out SRs (97.1%) as a source of evidence to inform decision-making. They frequently (97.9%) found more than one SR on a given topic of interest to them. Just over half (50.8%) struggled to choose the most trustworthy SR among multiple. These difficulties related to lack of time (55.2%), or difficulties comparing due to varying methodological quality of SRs (54.2%), differences in results and conclusions (49.7%) or variation in the included studies (44.6%). Respondents compared SRs based on the relevance to their question of interest, methodological quality, and recency of the SR search. Most respondents (87.0%) were interested in an AI tool to help appraise and compare SRs. Conclusions Given the identified barriers of using SR evidence, an AI tool to facilitate comparison of the relevance of SRs, the search and methodological quality, could help users efficiently choose among SRs and make healthcare decisions.

Recent studies have demonstrated that while neural ranking methods excel in retrieval effectiveness, they also tend to amplify stereotypical biases, especially those related to gender. Current mitigation strategies often focus on adjusting training methods, like adversarial techniques or data balancing, but typically overlook explicit consideration of gender as an attribute. In this paper, we introduce a systematic approach that treats gender as a distinct component within neural ranker representations. Our neural disentanglement method separates content semantics from gender information, enabling the neural ranker to evaluate document relevance based on content alone, without the interference of gender-related information during retrieval. Our extensive experiments demonstrate that: (1) our disentanglement approach matches the effectiveness of baseline models and offers more consistent performance across queries of different gender affiliations; (2) isolating gender within the representations allows the neural ranker to produce an unbiased list of documents, not favoring any specific gender; and (3) the disentangled gender component effectively and concisely captures gender information independently from the semantic content.

The increasing complexity and multidisciplinary nature of queries on Community Question Answering (CQA) platforms have rendered the traditional model of individual expert response inadequate. This paper tackles the challenge of identifying a group of experts whose combined expertise can effectively address such complex inquiries collaboratively, leading to more accepted answers. Our approach jointly learns topological and textual information extracted from the CQA environment in an end-to-end fashion. Extensive experiments on several real-life datasets indicate that our approach improves the quality of expert ranks on average 4.6% and 7.1% in terms of NDCG and MAP, respectively, compared to the best baseline. The results also reveal that groups formed by our approach are more collaborative and on average 61.6% of members recommended by our approach are among the true answerers of questions which is around 6.1 times improvement compared to the baselines.

Entity search plays a crucial role in various information access domains, where users seek information about specific entities. Despite significant research efforts to improve entity search methods, the availability of large-scale resources and extensible frameworks has been limiting progress. In this work, we present LaQuE (Large-scale Queries for Entity search), a curated framework for entity search, which includes a reproducible and extensible code base as well as a large relevance judgment collection consisting of real-user queries based on the ORCAS collection. LaQuE is industry-scale and suitable for training complex neural models for entity search. We develop methods for curating and judging entity collections, as well as training entity search methods based on LaQuE. We additionally establish strong baselines within LaQuE based on various retrievers, including traditional bag-of-words-based methods and neural-based models. We show that training neural entity search models on LaQuE enhances retrieval effectiveness compared to the state-of-the-art. Additionally, we categorize the released queries in LaQuE based on their popularity and difficulty, encouraging research on more challenging queries for the entity search task. We publicly release LaQuE at urlhttps://anonymous.4open.science/r/LaQuE-0CDD/.

Recent empirical studies have shown that while neural rankers exhibit increasingly higher retrieval effectiveness on tasks such as ad hoc retrieval, these improved performances are not experienced uniformly across the range of all queries. There are typically a large subset of queries that are not satisfied by neural rankers. These queries are often referred to as difficult queries. Since neural rankers operate based on the similarity between the embedding representations of queries and their relevant documents, the poor performance of difficult queries can be due to their sub-optimal learnt representations. Our work in this paper aims to learn to rank documents and also transform query representations in tandem such that the representation of queries are transformed into one that shows higher resemblance to their relevant document. This way, our method will provide the opportunity to satisfy a large number of difficult queries that would otherwise not be addressed. To do so, we propose to integrate two forms of triplet loss functions into neural rankers such that they ensure that each query is moved along the embedding space, through the transformation of its embedding representation, in order to be placed close to its relevant document(s). We perform experiments based on the MS MARCO passage ranking task and show that our proposed method has been able to show noticeable performance improvement for queries that were extremely difficult for existing neural rankers. On average, our approach has been able to satisfy 277 queries with an MRR@10 of 0.21 for queries that had a reciprocal rank of zero on the initial neural ranker.

Query performance prediction (QPP) is a core task in information retrieval (IR) that aims at predicting the retrieval quality for a given query without relevance judgments. QPP has been investigated for decades and has witnessed a surge in research activity in recent years; QPP has been shown to benefit various aspects, e.g., improving retrieval effectiveness by selecting the most effective ranking function per query. Despite its importance, there is no recent tutorial to provide a comprehensive overview of QPP techniques in the era of pre-trained/large language models or in the scenario of emerging conversational search (CS); moreover, while research in QPP has yielded promising results, its practical implementation and integration into real-world search engines remain a challenge. In this tutorial, we have three main objectives. First, we aim to disseminate the latest advancements in QPP to the IR community. Second, we go beyond investigating QPP in ad-hoc search and cover QPP for CS. Third, the tutorial offers a unique opportunity to bridge the gap between theory and practice; we aim to equip participants with the essential skills and insights needed to navigate the evolving landscape of QPP, ultimately benefiting both researchers and practitioners in the field of IR and encouraging them to work around the future avenues on QPP.

Query performance prediction (QPP) is a key task in information retrieval (IR ), focusing on estimating the retrieval quality of a given query without relying on human-labeled relevance judgments. Over the decades, QPP has gained increasing significance, with a surge in research activity in recent years. It has proven to benefit various aspects of retrieval, such as optimizing retrieval effectiveness by selecting the most appropriate ranking function for each query. Despite its critical role, there were only a few tutorials that cover the QPP techniques. The topic is even playing a more important role in the new era of pre-trained and large language models (LLMs), and the emerging fields of multi-agent intelligent systems and conversational search (CS ). Moreover, while research in QPP has yielded promising outcomes, studies on its practical application and integration into real-world search engines remain limited. This tutorial has four main objectives. First, it aims to cover both the fundamentals and the latest advancements in QPP methods. Second, it broadens the scope of QPP beyond ad-hoc search to vari- ous search scenarios, e.g., CS and image search. Third, this tutorial provides a comprehensive review of QPP applications across vari- ous aspects of IR, providing insights on where and how to apply QPP in practice. Fourth, we equip participants with hands-on mate- rials, enabling them to apply QPP implementation in practice. This tutorial seeks to benefit both researchers and practitioners in IR, encouraging further exploration and innovation in QPP

The peer review process is a fundamental aspect of academic publishing, ensuring the quality and credibility of scholarly work. In this talk, we will explore the critical challenges associated specifically with the assignment of reviewers to submitted papers. We will introduce Reviewerly, our innovative solution designed to enhance the efficiency and effectiveness of reviewer assignments by leveraging data from diverse sources, including OpenAlex, PubMed, and DBLP. By modeling the reviewer assignment problem as an information retrieval task, we focus on retrieving a pool of relevant and diverse reviewers for each paper. We will highlight the challenges we faced and showcase the benefits of this approach in addressing the reviewer assignment problem.

This paper introduces ADG-QPP (Adaptive Disturbance Generation), an unsupervised Query Performance Prediction (QPP) method designed specifically for dense neural retrievers. The underlying foundation of ADG-QPP is to measure query performance based on its degree of robustness towards perturbations. Traditional QPP methods rely on predefined lexical perturbations on the query, which only apply to sparse retrieval methods and fail to maintain consistent performance across different datasets. In our work, we address these limitations by perturbing the query by injecting disturbance leveraged by the focal network-based measurements including node-based, edge-based and cluster-based metrics, into its neural embedding representation. Rather than applying the same perturbation across all queries, our approach develops an instance-wise disturbance for each query that is then used for its perturbation. Through extensive experiments on three benchmark datasets, we demonstrate that ADG-QPP outperforms state-of-the-art baselines in terms of Kendall tau, Spearman rho, and Pearson’s rho correlations.

Large neural-based Pre-trained Language Models (PLM) have recently gained much attention due to their noteworthy performance in many downstream Information Retrieval (IR) and Natural Language Processing (NLP) tasks. PLMs can be categorized as either general-purpose, which are trained on resources such as large-scale Web corpora, and domain-specific which are trained on in-domain or mixed-domain corpora. While domain-specific PLMs have shown promising performance on domain-specific tasks, they are significantly more computationally expensive compared to general-purpose PLMs as they have to be either retrained or trained from scratch. The objective of our work in this paper is to explore whether it would be possible to leverage general-purpose PLMs to show competitive performance to domain-specific PLMs without the need for expensive retraining of the PLMs for domain-specific tasks. By focusing specifically on the recent BioASQ Biomedical Question Answering task, we show how different general-purpose PLMs show synergistic behaviour in terms of performance, which can lead to overall notable performance improvement when used in tandem with each other. More concretely, given a set of general-purpose PLMs, we propose a self-supervised method for training a classifier that systematically selects the PLM that is most likely to answer the question correctly on a per-input basis. We show that through such a selection strategy, the performance of general-purpose PLMs can become competitive with domain-specific PLMs while remaining computationally light since there is no need to retrain the large language model itself. We run experiments on the BioASQ dataset, which is a large-scale biomedical question-answering benchmark. We show that utilizing our proposed selection strategy can show statistically significant performance improvements on general-purpose language models with an average of 16.7% when using only lighter models such as DistilBERT and DistilRoBERTa, as well as 14.2% improvement when using relatively larger models such as BERT and RoBERTa and so, their performance become competitive with domain-specific large language models such as PubMedBERT.

Team formation is concerned with the identification of a group of experts who have a high likelihood of effectively collaborating with each other in order to satisfy a collection of input skills. Solutions to this task have mainly adopted graph operations and at least have the following limitations: (1) they are computationally demanding as they require finding shortest paths on large collaboration networks; (2) they use various types of heuristics to reduce the exploration space over the collaboration network in order to become practically feasible; therefore, their results are not necessarily optimal; and, (3) they are not well-suited for collaboration network structures given the sparsity of these networks. Our work proposes a variational Bayesian neural network architecture that learns representations for teams whose members have collaborated with each other in the past. The learnt representations allow our proposed approach to mine teams that have a past collaborative history and collectively cover the requested desirable set of skills. Through our experiments, we demonstrate that our approach shows stronger performance compared to a range of strong team formation techniques from both quantitative and qualitative perspectives.

The objective of this paper is to show that it is possible to significantly reduce stereotypical gender biases in neural rankers without modifying the ranking loss function, which is the current approach in the literature. We systematically de-bias gold standard relevance judgement datasets with a set of balanced and well-matched query pairs. Such a de-biasing process will expose neural rankers to comparable queries from across gender identities that have associated relevant documents with compatible degrees of gender bias. Therefore, neural rankers will learn not to associate varying degrees of bias to queries from certain gender identities. Our experiments show that our approach is able to (1) systematically reduces gender biases associated with different gender identities, and (2) at the same time maintain the same level of retrieval effectiveness.

The Information Retrieval community has made strides in developing neural rankers, which have show strong retrieval effectiveness on large-scale gold standard datasets. The focus of existing neural rankers has primarily been on measuring the relevance of a document or passage to the user query. However, other considerations such as the convincingness of the content are not taken into account when retrieving content. We present a large gold standard dataset, referred to as CoRe, which focuses on enabling researchers to explore the integration of the concepts of convincingness and relevance to allow for the retrieval of relevant yet persuasive content. Through extensive experiments on this dataset, we report that there is a close association between convincingness and relevance that can have practical value in how convincing content are presented and retrieved in practice.

The team discovery task is concerned with finding a group of experts from a collaboration network who would collectively cover a desirable set of skills. Most prior work for team discovery either adopt graph-based or neural mapping approaches. Graph-based approaches are computationally intractable often leading to sub-optimal team selection. Neural mapping approaches have better performance, however, are still limited as they learn individual representations for skills and experts and are often prone to overfitting given the sparsity of collaboration networks. Thus, we define the team discovery task as one of learning subgraph representations from a heterogeneous collaboration network where the subgraphs represent teams which are then used to identify relevant teams for a given set of skills. As such, our approach captures local (node interactions with each team) and global (subgraph interactions between teams) characteristics of the representation network and allows us to easily map between any homogeneous and heterogeneous subgraphs in the network to effectively discover teams. Our experiments over two real-world datasets from different domains, namely DBLP bibliographic dataset with 10,647 papers and IMDB with 4,882 movies, illustrate that our approach outperforms the state-of-the-art baselines on a range of ranking and quality metrics. More specifically, in terms of ranking metrics, we are superior to the best baseline by approximately 15% on the DBLP dataset and by approximately 20% on the IMDB dataset. Further, our findings illustrate that our approach consistently shows a robust performance improvement over the baselines.

Recent studies have shown that significant performance improvements reported by neural rankers do not necessarily extend to a diverse range of queries. There is a large set of queries that cannot be effectively addressed by neural rankers primarily because relevant documents to these queries are not identified by first-stage retrievers. In this paper, we propose a novel document representation approach that represents documents within the query space, and hence increases the likelihood of recalling a higher number of relevant documents. Based on experiments on the MS MARCO dataset as well as the hardest subset of its queries, we find that the proposed approach shows synergistic behavior to existing neural rankers and is able to increase recall both on MS MARCO dev set queries as well as the hardest queries of MS MARCO.

The ad hoc retrieval task aims at ranking relevant documents to a user query such that the most relevant documents are ranked higher compared to less relevant ones. Given the performance of the ad hoc retrieval task can vary across a range of queries, researchers have extensively explored the interrelated task of query performance prediction, which aims at estimating the quality of the search results for a user query without having access to relevance judgments. Traditionally and to-date, the two tasks have been explored as separate tasks where ad hoc retrieval and query performance prediction have been performed in isolation. In this paper, we propose to learn joint tasks that would perform ad hoc retrieval and at the same time predict the quality of the produced rankings. More specifically, we propose a mutli-task learning approach, called Multi-task Query Performance Prediction Framework (M-QPPF), which learns document ranking and query performance prediction tasks simultaneously. In M-QPPF, we adopt a shared BERT layer, which is fine-tuned to learn representations for query- document pairs in the embedding space such that the representations effectively encode the cross-interaction between the query and documents. We perform comprehensive experiments against state-of-the-art methods over large scale datasets.

This paper presents the idea of systematically integrating relation triples derived from Open Information Extraction (OpenIE) with neural rankers in order to improve the performance of the ad-hoc retrieval task. This is motivated by two reasons: (1) to capture longer-range semantic associations between keywords in documents, which would not otherwise be immediately identifiable by neural rankers; and (2) identify closely mentioned yet semantically unrelated content in the document that could lead to a document being incorrectly considered to be relevant for the query. Through our extensive experiments on three widely used TREC collections, we show that our idea consistently leads to noticeable performance improvements for neural rankers on a range of metrics.

No abstract available for this paper.

No abstract available for this paper.

Query Performance Prediction (QPP), is concerned with assessing the retrieval quality of a ranking method for an input query. Most traditional unsupervised frequency-based models and many recent supervised neural methods have been designed specifically for predicting the performance of sparse retrievers such as BM25. In this paper we propose an unsupervised QPP method for dense neural retrievers which operates by redefining the well-known concept of query robustness i.e., a more robust query to perturbations is an easier query to handle. We propose to generate query perturbations for measuring query robustness by systematically injecting noise into the contextualized neural representation of each query. We then compare the retrieved list for the original query with that of the perturbed query as a way to measure query robustness. Our experiments on four different query sets including MS MARCO, TREC Deep Learning track 2019 and 2020 and TREC DL-Hard show consistently improved performance on linear and ranking correlation metrics over the state of the art.

The information retrieval community has observed significant performance improvements over various tasks due to the introduction of neural architectures. However, such improvements do not necessarily seem to have happened uniformly across a range of queries. As we will empirically show in this paper, the performance of neural rankers follow a long-tail distribution where there are many subsets of queries, which are not effectively satisfied by neural methods. Despite this observation, performance is often reported using standard retrieval metrics, such as MRR or nDCG, which capture average performance over all queries. As such, it is not clear whether reported improvements are due to incremental boost on a small subset of already well-performing queries or addressing queries that have been difficult to address by existing methods. In this paper, we propose the Task Subspace Coverage (TaSC /tAHsk/) metric, which systematically quantifies whether and to what extent improvements in retrieval effectiveness happen on similar or disparate query subspaces for different rankers. Our experiments show that the consideration of our proposed TaSC metric in conjunction with existing ranking metrics provides deeper insight into ranker performance and their contribution to overall advances on a given task.

No abstract available for this paper.

In light of recent studies that show neural retrieval methods may intensify gender biases during retrieval, the objective of this paper is to propose a simple yet effective sampling strategy for training neural rankers that would allow the rankers to maintain their retrieval effectiveness while reducing gender biases. Our work proposes to consider the degrees of gender bias when sampling documents to be used for training neural rankers. We report our findings on the MS MARCO collection and based on different query datasets released for this purpose in the literature. Our results show that the proposed light-weight strategy is able to show competitive (or even better) performance compared to the state of the art neural architectures specifically designed to reduce gender biases.

While neural rankers continue to show notable performance improvements over a wide variety of information retrieval tasks, there have been recent studies that show such rankers may intensify certain stereotypical biases. In this paper, we investigate whether neural rankers introduce retrieval effectiveness (performance) disparities over queries related to different genders. We specifically study whether there are significant performance differences between male and female queries when retrieved by neural rankers. Through our empirical study over the MS MARCO collection, we find that such performance disparities are notable and that the performance disparities may be due to the difference between how queries and their relevant judgements are collected and distributed for different gendered queries. More specifically, we observe that male queries are more closely associated with their relevant documents compared to female queries and hence neural rankers are able to more easily learn associations between male queries and their relevant documents. We show that it is possible to systematically balance relevance judgment collections in order to reduce performance disparity between different gendered queries without negatively compromising overall model performance.

Research has shown that neural rankers can pick up and intensify gender biases. The expression of stereotypical gender biases in retrieval systems can lead to their reinforcement in users' beliefs. As such, the objective of this paper is to propose a bias-aware fair ranker that explicitly incorporates a notion of gender bias and hence controls how bias is expressed in documents that are retrieved. The proposed approach is designed such that it learns the notion of relevance between the document and the query from the relevant sampled documents while incorporating the notion of gender bias by penalizing irrelevant biased sampled documents. We show that unlike the state of the art, our approach reduces bias while maintaining retrieval effectiveness over different query sets.

Finding a qualified individual who can independently answer a question on a community question answering platform is becoming more challenging due to the increasing multidisciplinary nature of posted questions. As such, finding a group of experts to collaboratively answer the questions is of paramount importance. To this end, our proposed method forms teams of experts who can collectively answer new questions. Our approach, called team2box, learns neural embedding representations based on the content of the posted questions, experts’ engagement with these questions, and past expert collaboration history in order to form a team to answer the posted question. It embeds experts and questions as points and existing teams as regions within the embedding space. Such an approach allows team2box to form a team whose members (1) collectively cover the knowledge required to answer a question, (2) have successful past experience in jointly answering similar questions, and (3) can work efficiently together to answer the question. Extensive experiments on real-life datasets from Stack Exchange show that team2box outperforms the state-of-the-art by discovering teams with on average 38.97% more covering the skills required to answer new questions and employing experts with collectively a high expertise level.

Community question answering (CQA) platforms are receiving increased attention and are becoming an indispensable source of information in different domains ranging from board games to physics. The success of these platforms dependent on how efficiently new questions are assigned to community experts, known as called question routing. In this paper, we address the problem of question routing by adopting a learning to rank approach over five CQA websites in the context of which we introduce 74 features and systematically classify them into content-based and social-based categories. Our extensive experiments on datasets from five real online question answering websites indicate that content-based features related to tags and topics as well as social features that are related to user characteristics and user temporality are effective for question routing. Our work shows the ability to improve performance compared to the state-of-the-art neural matchmaking methods that lack the interpretability offered by our work. The improvement can be as high as on average 2.74% and20 2.27% in terms of common ranking metrics, Normalized Discounted Cumulative Gain (NDCG) and Mean Average Precision (MAP) respectively, compared to our best baselines.

Recent studies have shown that it is possible for stereotypical gender biases to find their way into representational and algorithmic aspects of retrieval methods; hence, exhibit themselves in retrieval outcomes. In this tutorial, we inform the audience of various studies that have systematically reported the presence of stereotypical gender biases in Information Retrieval (IR) systems. We further classify existing work on gender biases in IR systems as being related to (1) relevance judgement datasets, (2) structure of retrieval methods, and (3) representations learnt for queries and documents. We present how each of these components can be impacted by or cause intensified biases during retrieval. Based on these identified issues, we then present a collection of approaches from the literature that have discussed how such biases can be measured, controlled, or mitigated. Additionally, we introduce publicly available datasets that are often used for investigating gender biases in IR systems as well as evaluation methodology adopted for determining the utility of gender bias mitigation strategies.

We study team formation whose goal is to form a team of experts who collectively cover a set of desirable skills. This problem has mainly been addressed either through graph search techniques, which look for subgraphs that satisfy a set of skill requirements, or through neural architectures that learn a mapping from the skill space to the expert space. An exact graph-based solution to this problem is intractable and its heuristic variants are only able to identify sub-optimal solutions. On the other hand, neural architecture-based solutions treat experts individually without concern for team dynamics. In this paper, we address the task of forming coherent teams and propose a neural approach that maximizes the likelihood of successful collaboration among team members while maximizing the coverage of the required skills by the team. Our extensive experiments show that the proposed approach outperforms the state-of-the-art methods in terms of both ranking and quality metrics.

With increased recent awareness on the possible impact of retrieval techniques on intensifying gender biases, researchers have embarked on defining quantifiable gender bias metrics that can provide the means to concretely measure such biases in practice. While successful in allowing for identifying possible sources of gender bias, there has been little work that systematically explores the characteristics of these metrics. This paper argues that effective future works on gender biases in information retrieval require a careful understanding of the bias metrics in terms of their consistency, robustness, sensitivity and also their relation with psychological characteristics and what they actually measure. Through our experiments, we show that more rigorous work on gender bias metrics need to be pursued as existing metrics may not necessarily be consistent and robust and often capture differing psychological characteristics.

The focus of our work is the ad hoc table retrieval task, which aims to rank a list of structured tabular objects in response to a user query. Given the importance of this task, various methods have already been proposed in the literature that focus on syntactic, semantic and neural representations of tables for determining table relevance. However, recent works have highlighted queries that are consistently difficult for baseline methods to satisfy, referred to as hard queries. For this reason, the objectives of this paper include: (1) effectively satisfying hard queries by proposing three classes of qualitative measures, namely coherence, interpretability and exactness, (2) offering a systematic approach to interpolate these three classes of measures with each other and with baseline table retrieval methods, and (3) performing extensive experiments using a range of baseline retrieval methods to show the feasibility of the proposed measures for hard queries. We demonstrate that the consideration of the proposed qualitative measures will lead to improved performance for hard queries on a range of state-of-the-art ad hoc table retrieval baselines. We further show that our proposed measures are synergistic and will lead to even higher performance improvements over the baselines when interpolated with each other. The improvements measure up to 22.94% on the Semantic Table Retrieval (STR) method with an NDCG@20 of 0.5, which is superior to the performance of any state-of-the-art baseline for hard queries in the ad hoc table retrieval task.

We present an open-source extensible python-based toolkit that provides access to a (1) range of built-in unsupervised query expansion methods, and (2) pipeline for generating gold standard datasets for building and evaluating supervised query refinement methods. While the information literature offers abundant work on query expansion techniques, there is yet to be a tool that provides unified access to a comprehensive set of query expansion techniques. The advantage of our proposed toolkit, known as ReQue (refining queries), is that it offers one-stop shop access to query expansion techniques to be used in external information retrieval applications. More importantly, we show how ReQue can be used for building gold standards datasets that can be used for training supervised deep learning-based query refinement techniques. These techniques require sizeable gold query refinement datasets, which are not available in the literature. Reque provides the means to systematically build such datasets.

No abstract available for this paper.

When searching, users are interested in accessing the most relevant and specific content related to their information need. Earlier research has shown that it is much easier to retrieve appropriate content for specific queries compared to generic ones as it is possible to discriminatively distinguish the content related to specific queries. The work in this paper builds on earlier findings on query and document specificity and provides a systematic account of ways through which document specificity can be measured. We present a comprehensive view of how various measures of document specificity can be defined and comparatively analyze the utility of various document specificity measures within the context of ad hoc retrieval based on three well-known TREC corpora, namely Robust04, ClueWeb09B, ClueWeb12B and their associated TREC topics. We report on our findings on the effectiveness of each type of document specificity measure.

Recent studies in information retrieval have shown that gender biases have found their way into representational and algorithmic aspects of computational models. In this paper, we focus specifically on gender biases in information retrieval gold standard datasets, often referred to as relevance judgements. While not explored in the past, we submit that it is important to understand and measure the extent to which gender biases may be present in information retrieval relevance judgements primarily because relevance judgements are not only the primary source for evaluating IR techniques but are also widely used for training end-to-end neural ranking methods. As such, the presence of bias in relevance judgements would immediately find its way into how retrieval methods operate in practice. Based on a fine-tuned BERT model, we show how queries can be labeled for gender at scale based on which we label MS MARCO queries. We then show how different psychological characteristics are exhibited within documents associated with gendered queries within the relevance judgement datasets. Our observations show that stereotypical biases are prevalent in relevance judgement documents.

Linking textual content to entities from the knowledge graph has received increasing attention in the context of which surface form representations of entities, e.g., terms or phrases, are disambiguated and linked to appropriate entities. This allows textual content, e.g., social user-generated content, to be interpreted and reasoned on at a higher semantic level. However, recent research has shown that at least 15% of social user-generated content do not have explicit surface form representation of entities that they discuss. In other words, the subject of the content is only implied. For such cases, existing entity linking methods, known as explicit entity linking, cannot perform linking because entity surface form is missing. In this paper, we investigate how implicit entities within social content can be identified and linked. The contributions of our work include (1) modeling the problem of implicit entity linking as a learn to rank problem where knowledge graph entities are ranked based on their relevance to the input tweet, (2) the introduction and systematic classification of appropriate features for identifying implicit entities, (3) extensive evaluation of the proposed approach in comparison with existing state of the art as well as performing feature analysis over proposed features, and (4) the qualitative assessment of the root causes for mislabeled instances in our experiments and careful discussion on how mislabeled entity links can be addressed as a part of future work. In our experiments, we show that our proposed features are able to improve the state of the art over the standard Precision at 1 (P@1) metric.

No abstract available for this paper.

Various researchers have recently explored the impact of different types of biases on information retrieval tasks such as ad hoc retrieval and question answering. While the impact of bias needs to be controlled in order to avoid increased prejudices, the literature has often viewed the relationship between increased retrieval utility (effectiveness) and reduced bias as a tradeoff where one can suffer from the other. In this paper, we empirically study this tradeoff and explore whether it would be possible to reduce bias while maintaining similar retrieval utility. We show this would be possible by revising the input query through a bias-aware pseudo-relevance feedback framework. We report our findings based on four widely used TREC corpora namely Robust04, Gov2, ClueWeb09 and ClueWeb12 and using two classes of bias metrics. The findings of this paper are significant as they are among the first to show that decrease in bias does not necessarily need to come at the cost of reduced utility.

No abstract available for this paper.

No abstract available for this paper.

No abstract available for this paper.

Given a set of required skills, the objective of the team formation problem is to form a team of experts that cover the required skills. Most existing approaches are based on graph methods, such as minimum-cost spanning trees. These approaches, due to their limited view of the network, fail to capture complex interactions among experts and are computationally intractable. More recent approaches adopt neural architectures to learn a mapping between the skills and experts space. While they are more effective, these techniques face two main limitations: (1) they consider a fixed representation for both skills and experts, and (2) they overlook the significant amount of past collaboration network information. We learn dense representations for skills and experts based on previous collaborations and bootstrap the training process through transfer learning. We also propose to fine-tune the representation of skills and experts while learning the mapping function. Our experiments over the DBLP dataset verify that our proposed architecture is able to outperform the state-of-the-art graph and neural methods over both ranking and quality metrics.

The ad hoc table retrieval task is concerned with satisfying a query with a ranked list of tables. While there are strong baselines in the literature that exploit learning to rank and semantic matching techniques, there are still a set of hard queries that are difficult for these baseline methods to address. We find that such hard queries are those whose constituting tokens (i.e., terms or entities) are not fully or partially observed in the relevant tables. We focus on proposing a latent factor model to address such hard queries. Our proposed model factorizes the token-table co-occurrence matrix into two low dimensional latent factor matrices that can be used for measuring table and query similarity even if no shared tokens exist between them. We find that the variation of our proposed model that considers keywords provides statistically significant improvement over three strong baselines in terms of NDCG and ERR.

The abundance of user generated content on social media provides the opportunity to build models that are able to accurately and effectively extract, mine and predict users' interests with the hopes of enabling more effective user engagement, better quality delivery of appropriate services and higher user satisfaction. While traditional methods for building user profiles relied on AI-based preference elicitation techniques that could have been considered to be intrusive and undesirable by the users, more recent advances are focused on a non-intrusive yet accurate way of determining users' interests and preferences. In this paper, we will cover five important subjects related to the mining of user interests from social media: (1) the foundations of social user interest modeling, such as information sources, various types of representation models and temporal features, (2) techniques that have been adopted or proposed for mining user interests, (3) different evaluation methodologies and benchmark datasets, (4) different applications that have been taking advantage of user interest mining from social media platforms, and (5) existing challenges, open research questions and exciting opportunities for further work.

No abstract available for this paper.

Implicit entity linking is the task of identifying an appropriate entity whose surface form is not explicitly mentioned in the text. Unlike explicit entity linking where an entity is linked to an observed phrase within the input text, implicit entity linking is concerned with determining specific yet implied entities. Existing work in the literature have already identified appropriate features that can be used for ranking relevant entities for explicit entity linking. In this paper, we (1) consider the applicability of such features for implicit entity linking, (2) introduce features that are suited for this task, (3) compare our work with the state of the art in implicit entity linking, and (4) and report on feature importance values and error analysis.

We focus on the composition of teams of experts that collectively cover a set of required skills based on their historical collaboration network and expertise. Prior works are primarily based on the shortest path between experts on the expert collaboration network, and suffer from three major shortcomings: (1) they are computationally expensive due to the complexity of finding paths on large network structures; (2) they use a small portion of the entire historical collaboration network to reduce the search space; hence, may form sub-optimal teams; and, (3) they fall short in sparse networks where the majority of the experts have only participated in a few teams in the past. Instead of forming a large network of experts, we propose to learn relationships among experts and skills through a variational Bayes neural architecture wherein: i) we consider all past team compositions as training instances to predict future teams; ii) we bring scalability for large networks of experts due to the neural architecture; and, iii) we address sparsity by incorporating uncertainty on the neural network's parameters which yields a richer representation and more accurate team composition. We empirically demonstrate how our proposed model outperforms the state-of-the-art approaches in terms of effectiveness and efficiency.

Query Performance Prediction (QPP) is concerned with estimating the effectiveness of a query within the context of a retrieval model. It allows for operations such as query routing and segmentation, leading to improved retrieval performance. Pre-retrieval QPP methods are oblivious to the performance of the retrieval model as they predict query difficulty prior to observing the set of documents retrieved for the query. Since neural embedding-based models are showing wider adoption in the information retrieval community, in this paper, we propose a set of pre-retrieval QPP metrics based on the properties of pre-trained neural embeddings and show that such metrics are more effective for performance prediction compared to the widely known QPP metrics such as SCQ, PMI and SCS. We report our findings based on Robust04, ClueWeb09 and Gov2 corpora and their associated TREC topics.

In information retrieval, the task of query performance prediction (QPP) is concerned with determining in advance the performance of a given query within the context of a retrieval model. QPP has an important role in ensuring proper handling of queries with varying levels of difficulty. Based on the extant literature, textitquery specificity is an important indicator of query performance and is typically estimated using corpus-specific frequency-based specificity metrics However, such metrics do not consider term semantics and inter-term associations. Our work presented in this paper distinguishes itself by proposing a host of corpus-independent specificity metrics that are based on pre-trained neural embeddings and leverage geometric relations between terms in the embedding space in order to capture the semantics of terms and their interdependencies. Specifically, we propose three classes of specificity metrics based on pre-trained neural embeddings: neighborhood-based, graph-based, and cluster-based metrics. Through two extensive and complementary sets of experiments, we show that the proposed specificity metrics (1) are suitable specificity indicators, based on the gold standards derived from knowledge hierarchies (Wikipedia category hierarchy and DMOZ taxonomy), and (2) have better or competitive performance compared to the state of the art QPP metrics, based on both TREC ad hoc collections namely Robust'04, Gov2 and ClueWeb'09 and ANTIQUE question answering collection. The proposed graph-based specificity metrics, especially those that capture a larger number of inter-term associations, proved to be the most effective in both query specificity estimation and QPP. We have also publicly released two test collections (i.e. specificity gold standards) that we built from the Wikipedia and DMOZ knowledge hierarchies.

In this paper, we implement and publicly share a configurable software workflow and a collection of gold standard datasets for training and evaluating supervised query refinement methods. Existing datasets such as AOL and MS MARCO, which have been extensively used in the literature for this purpose, are based on the weak assumption that users’ input queries improve gradually within a search session, i.e., the last query where the user ends her information seeking session is the best reconstructed version of her initial query. In practice, such an assumption is not necessarily accurate for a variety of reasons, e.g., topic drift. The objective of our work is to enable researchers to build gold standard query refinement datasets without having to rely on such weak assumptions. Our software workflow, which generates such gold standard query datasets, takes three inputs: (1) a dataset of queries along with their associated relevance judgements (e.g. TREC topics), (2) an information retrieval method (e.g., BM25), and (3) an evaluation metric (e.g., MAP), and outputs a gold standard dataset. The produced gold standard dataset includes a list of revised queries for each query in the input dataset, each of which effectively improves the performance of the specified retrieval method (e.g., BM25) in terms of the desirable evaluation metric (e.g., MAP). Since our workflow can be used to generate gold standard datasets for any input query set, in this paper, we have generated and publicly shared gold standard datasets for TREC queries associated with Robust04, Gov2, ClueWeb09, and ClueWeb12. The source code of our software workflow, the generated gold datasets, and benchmark results for three state-of-the-art supervised query refinement methods over these datasets are made publicly available for reproducibility purposes.

Predicting the performance of a retrieval method for a given query is a highly important and challenging problem in information retrieval. Accurate Query Performance Prediction (QPP) plays an important role in real time handling of queries with varying levels of difficulty. While there have been several successful query performance predictors, no predictors have yet been introduced within the context of the ad hoc table retrieval task, which is concerned with answering a query with a ranked list of tables. In this paper, we propose to perform query performance prediction based on neural embedding techniques for ad hoc table retrieval and introduce three neural features. The neural features are based on neural embedding techniques and leverage the distance between tokens in the embedding space in order to capture their semantic similarity. We evaluate our proposed work based on a gold standard test collection and compare it with the state-of-the-art post-retrieval query performance prediction methods. We find that our neural features (1) are effective for predicting the performance of content-based ranking functions; and not as effective for feature-based ranking functions, and (2) show a synergistic impact on existing QPP methods and hence are able to increase their performance in practice.

Product reviews written by the crowd on e-commerce shopping websites have become a critical information source for making purchasing decisions. Researchers have already extensively studied the impact of review sentiments and informativeness on product sales and success, finding a close relationship. An important challenge, however, is that the vast majority of products (e.g., >90% of products on amazon.com) do not receive enough attention and lack sufficient reviews by the users; hence, they constitute the so-called textitcold products. One solution to address cold products, which has already been studied in the literature, is to generate reviews for these products by sampling review sentences from closely related warm products. Our method proposed in this paper is specifically focused on such a solution. While a majority of the works in the literature rely on product specification similarity to identify relevant reviews that can be used for review sentence selection, our work differs in that it not only employs product specification similarity but also employs product-review, product-user, and user-review interactions when determining the suitability of a review sentence to be selected. More specifically, the contributions of our work can be enumerated as follows: (1) We propose that the selection of review sentences from other products should not only consider product-product similarity but also consider product-review, user-review, and user-user relationships. As such, we show how neural graph embeddings can be used to encode product, user, and review information into an attributed heterogeneous graph representation based on which similarities can be calculated. (2) We further propose how review textitrelevance and textitimportance can be considered using graph traversal to select appropriate review sentences for a given cold product. (3) Finally, we systematically compare the performance of our work with those of several state-of-the-art baselines on five datasets collected from CNET.com and rottentomatoes.com with different characteristics from both quantitative (e.g., the Recall-Oriented Understudy for Gisting Evaluation (ROUGE) metrics) and qualitative aspects and show how our proposed approach was able to provide statistically significantly improved performance over various strong baselines.

No abstract available for this paper.

No abstract available for this paper.

Specificity is the level of detail at which a given term is represented. Existing approaches to estimating term specificity are primarily dependent on corpus-level frequency statistics. In this work, we explore how neural embeddings can be used to define corpus-independent specificity metrics. Particularly, we propose to measure term specificity based on the distribution of terms in the neighborhood of the given term in the embedding space. The intuition is that a term that is surrounded by other terms in the embedding space is more likely to be specific while a term surrounded by less closely related terms is more likely to be generic. On this basis, we leverage geometric properties between embedded terms to define three groups of metrics: (1) neighborhood-based, (2) graph-based and (3) cluster-based metrics. Moreover, we employ learning-to-rank techniques to estimate term specificity in a supervised approach by employing the three proposed groups of metrics. We curate and publicly share a test collection of term specificity measurements defined based on Wikipedia's category hierarchy. We report on our experiments through metric performance comparison, ablation study and comparison against the state-of-the-art baselines.

Within the context of Twitter analytics, the notion of implicit entity linking has recently been introduced to refer to the identification of a named entity, which is central to the topic of the tweet, but whose surface form is not present in the tweet itself. Compared to traditional forms of entity linking where the linking process revolves around an identified surface form of a potential entity, implicit entity linking relies on contextual clues to determine whether an implicit entity is present within a given tweet and if so, which entity is being referenced. The objective of this paper, while introducing and publicly sharing a comprehensive gold standard dataset for implicit entity linking, is to perform the task of implicit entity linking. The dataset consists of 7,870 tweets, which are classified as either containing implicit entities, explicit entities, both, or neither. The implicit entities are then linked to three levels of entities on Wikipedia, namely coarse-grained level, e.g., Person, Fine-grained level, e.g., Comedian, and the actual entity, e.g., Seinfeld. The proposed model in this work formulates the problem of implicit entity linking as an ad-hoc document retrieval process where the input query is the tweet, which needs to be implicitly linked and the document space is the set of textual descriptions of entities in the knowledge base. The novel contributions of our work include: 1) designing and collecting a gold standard dataset for the task of implicit entity linking; 2) defining the implicit entity linking process as an ad-hoc document retrieval task; and 3) proposing a neural embedding-based feature function that is interpolated with prior term dependency and entity-based feature functions to enhance implicit entity linking. We systematically compare our work with a state of the art baseline and show that our method is able to provide statistically significant improvements.

No abstract available for this paper.

The focus of point-of-interest recommendation techniques is to suggest a venue to a given user that would match the users' interests and is likely to be adopted by the user. Given the multitude of venues and the sparsity of user check-ins, the problem of recommending venues has shown to be a difficult task. Existing literature has already explored various types of features such as geographical distribution, social structure and temporal behavioral patterns to make a recommendation. In this paper, we propose a new set of features derived based on the neural embeddings of venues and users. We show how the neural embeddings for users and venues can be jointly learnt based on the prior check-in sequence of users and then be used to define three types of features, namely user, venue, and user-venue interaction features. These features are integrated into a feature-based matrix factorization model. Our experiments show that the features defined over the user and venue embeddings are effective for venue recommendation.

No abstract available for this paper.

Identifying and extracting user communities is an important step towards understanding social network dynamics from a macro perspective. For this reason, the work in this paper explores various aspects related to the identification of user communities. To date, user community detection methods employ either explicit links between users (link analysis), or users' topics of interest in posted content (content analysis), or in tandem. Little work has considered temporal evolution when identifying user communities in a way to group together those users who share not only similar topical interests but also similar temporal behavior towards their topics of interest. In this paper, we identify user communities through textitmultimodal feature learning (embeddings). Our core contributions can be enumerated as (a) we propose a new method for learning neural embeddings for users based on their temporal content similarity; (b) we learn user embeddings based on their social network connections (links) through neural graph embeddings; (c) we systematically interpolate temporal content-based embeddings and social link-based embeddings to capture both social network connections and temporal content evolution for representing users, and (d) we systematically evaluate the quality of each embedding type in isolation and also when interpolated together and demonstrate their performance on a Twitter dataset under two different application scenarios, namely textitnews recommendation and textituser prediction. We find that (1) content-based methods produce higher quality communities compared to link-based methods; (2) methods that consider temporal evolution of content, our proposed method in particular, show better performance compared to their non-temporal counter-parts; (3) communities that are produced when time is explicitly incorporated in user vector representations have higher quality than the ones produced when time is incorporated into a generative process, and finally (4) while link-based methods are weaker than content-based methods, their interpolation with content-based methods leads to improved quality of the identified communities.

Neural embeddings have been effectively integrated into information retrieval tasks including ad hoc retrieval. One of the benefits of neural embeddings is they allow for the calculation of the similarity between queries and documents through vector similarity calculation methods. While such methods have been effective for document matching, they have an inherent bias towards documents that are sized relatively similarly. Therefore, the difference between the query and document lengths, referred to as the query-document size imbalance problem, becomes an issue when incorporating neural embeddings and their associated similarity calculation models into the ad hoc document retrieval process. In this paper, we propose that document representation methods need to be used to address the size imbalance problem and empirically show their impact on the performance of neural embedding-based ad hoc retrieval. In addition, we explore several types of document representation methods and investigate their impact on the retrieval process. We conduct our experiments on three widely used standard corpora, namely Clueweb09B, Clueweb12B and Robust04 and their associated topics. Summarily, we find that document representation methods are able to effectively address the query-document size imbalance problem and significantly improve the performance of neural ad hoc retrieval. In addition, we find that a document representation method based on a simple term-frequency shows significantly better performance compared to more sophisticated representation methods such as neural composition and aspect-based methods.

The systematic linking of explicitly-observed phrases within a document to entities of a knowledge base has already been explored in a process known as entity linking. The objective of this paper, however, is to identify and entity link those entities that are not mentioned but are implied within a document, more specifically within a tweet. This process is referred to as implicit entity linking. Unlike prior work that build a representation for each entity based on its related content in the knowledge base, we propose to perform implicit entity linking by determining how a tweet is related to user-generated content posted online and as such indirectly perform entity linking. We formulate this problem as an ad-hoc document retrieval process where the input query is the tweet, which needs to be implicitly linked and the document space is the set of user-generated content related to the entities of the knowledge base. We systematically compare our work with the state-of-the-art baseline and show that our method is able to provide statistically significant improvements.

Traditional information retrieval techniques that primarily rely on keyword-based linking of the query and document spaces face challenges such as the emphvocabulary mismatch problem where relevant documents to a given query might not be retrieved simply due to the use of different terminology for describing the same concepts. As such, semantic search techniques aim to address such limitations of keyword-based retrieval models by incorporating semantic information from standard knowledge bases such as Freebase and DBpedia. The literature has already shown that while the sole consideration of semantic information might not lead to improved retrieval performance over keyword-based search, their consideration enables the retrieval of a set of relevant documents that cannot be retrieved by keyword-based methods. As such, building indices that store and provide access to semantic information during the retrieval process is important. While the process for building and querying keyword-based indices is quite well understood, the incorporation of semantic information within search indices is still an open challenge. Existing work have proposed to build one unified index encompassing both textual and semantic information or to build separate yet integrated indices for each information type but they face limitations such as increased query process time. In this paper, we propose to use neural embeddings-based representations of term, semantic entity, semantic type and documents within the same embedding space to facilitate the development of a unified search index that would consist of these four information types. We perform experiments on standard and widely used document collections including Clueweb09-B and Robust04 to evaluate our proposed indexing strategy from both empheffectiveness and emphefficiency perspectives. Based on our experiments, we find that when neural embeddings are used to build inverted indices; hence relaxing the requirement to explicitly observe the posting list key in the indexed document: (a) textitretrieval efficiency will increase compared to a standard inverted index, hence reduces the index size and query processing time, and (b) while retrieval efficiency, which is the main objective of an efficient indexing mechanism improves using our proposed method, textitretrieval effectiveness also retains competitive performance compared to the baseline in terms of retrieving a reasonable number of relevant documents from the indexed corpus.

Learning low dimensional dense representations of the vocabularies of a corpus, known as neural embeddings, has gained much attention in the information retrieval community. While there have been several successful attempts at integrating embeddings within the ad hoc document retrieval task, yet, no systematic study has been reported that explores the various aspects of neural embeddings and how they impact retrieval performance. In this paper, we perform a methodical study on how neural embeddings influence the ad hoc document retrieval task. More specifically, we systematically explore the following research questions: i) do methods solely based on neural embeddings perform competitively with state of the art retrieval methods with and without interpolation? ii) are there any statistically significant difference between the performance of retrieval models when based on textitword embeddings compared to when knowledge graph entity embeddings are used? and iii) is there significant difference between using locally trained neural embeddings compared to when globally trained neural embeddings are used? We examine these three research questions across both hard and all queries.Our study finds that word embeddings do not show competitive performance to any of the baselines. In contrast, entity embeddings show competitive performance to the baselines and when interpolated, outperform the best baselines for both hard and soft queries.

No abstract available for this paper.

The removal of stopwords is an important preprocessing step in many natural language processing tasks, which can lead to enhanced performance and execution time. Many existing methods either rely on a predefined list of stopwords or compute word significance based on metrics such as tf-idf. The objective of our work in this paper is to identify stopwords, in an unsupervised way, for streaming textual corpora such as Twitter, which have a temporal nature. We propose to consider and model the dynamics of a word within the streaming corpus to identify the ones that are less likely to be informative or discriminative. Our work is based on the discrete wavelet transform (DWT) of word signals in order to extract two features, namely scale and energy. We show that our proposed approach is effective in identifying stopwords and improves the quality of topics in the task of topic detection.

The accurate identification of user interests on Twitter can lead to more efficient procurement of targeted content for the users. While the analysis of user content has engaged with on Twitter is a rich source for detecting the user’s interests, prior research have shown that it may not be sufficient. There have been work that attempt to identify a user’s implicit interests, i.e., those topics that could interest the user but the user has not engaged with them in the past. Prior work has shown that topic semantic relatedness is an important feature for determining users’ implicit interests. In this paper, we explore the possibility of identifying users’ implicit interests solely based on topic association through frequent pattern mining without regard for the semantics of the topics. We show in our experiments that topic association is a strong feature for determining users’ implicit interests.

The accurate prediction of users' future interests on social networks allows one to perform future planning by studying how users will react if certain topics emerge in the future. It can improve areas such as targeted advertising and the efficient delivery of services. Despite the importance of predicting user future interests on social networks, existing works mainly focus on identifying user current interests and little work has been done on the prediction of user potential interests in the future. There have been work that attempt to identify a user future interests, however they cannot predict user interests with regard to new topics since these topics have never received any feedback from users in the past. In this paper, we propose a framework that works on the basis of temporal evolution of user interests and utilizes semantic information from knowledge bases such as Wikipedia to predict user future interests and overcome the cold item problem. Through extensive experiments on a real-world Twitter dataset, we demonstrate the effectiveness of our approach in predicting future interests of users compared to state-of-the-art baselines. Moreover, we further show that the impact of our work is especially meaningful when considered in case of cold items.

This paper explores the possibility of using neural embedding features for enhancing the effectiveness of ad hoc document ranking based on learning to rank models. We have extensively introduced and investigated the effectiveness of features learnt based on word and document embeddings to represent both queries and documents. We employ several learning to rank methods for document ranking using embedding-based features, keyword-based features as well as the interpolation of the embedding-based features with keyword-based features. The results show that embedding features have a synergistic impact on keyword based features and are able to provide statistically significant improvement on harder queries.

No abstract available for this paper.

No abstract available for this paper.

No abstract available for this paper.

No abstract available for this paper.

In this paper, we address the problem of predicting future interests of users with regards to a set of unobserved topics in microblogging services which enables forward planning based on potential future interests. Existing works in the literature that operate based on a known interest space cannot be directly applied to solve this problem. Such methods require at least a minimum user interaction with the topic to perform prediction. To tackle this problem, we integrate the semantic information derived from the Wikipedia category structure and the temporal evolution of user’s interests into our prediction model. More specifically, to capture the temporal behaviour of the topics and user’s interests, we consider discrete intervals and build user’s topic profile in each time interval separately. Then, we generalize users’ interests that have been observed over several time intervals by transferring them over the Wikipedia category structure. Our approach not only allows us to generalize users’ interests but also enables us to transfer users’ interests across different time intervals that do not necessarily have the same set of topics. Our experiments illustrate the superiority of our model compared to the state of the art.

One of the major challenges in Web search pertains to the correct interpretation of users’ intent. Query Expansion is one of the well-known approaches for determining the intent of the user by addressing the vocabulary mismatch problem. A limitation of the current query expansion approaches is that the relations between the query terms and the expanded terms is limited. In this paper, we capture users’ intent through query expansion. We build on earlier work in the area by adopting a pseudo-relevance feedback approach; however, we advance the state of the art by proposing an approach for feature learning within the process of query expansion. In our work, we specifically consider the Wikipedia corpus as the feedback collection space and identify the best features within this context for term selection in two supervised and unsupervised models. We compare our work with state of the art query expansion techniques, the results of which show promising robustness and improved precision.

We propose a neural embedding approach to identify temporally like-minded user communities, i.e., those communities of users who have similar temporal alignment in their topics of interest. Like-minded user communities in social networks are usually identified by either considering explicit structural connections between users (link analysis), users' topics of interest expressed in their posted contents (content analysis), or in tandem. In such communities, however, the users' rich temporal behavior towards topics of interest is overlooked. Only few recent research efforts consider the time dimension and define like-minded user communities as groups of users who share not only similar topical interests but also similar temporal behavior. Temporal like-minded user communities find application in areas such as recommender systems where relevant items are recommended to the users at the right time. In this paper, we tackle the problem of identifying temporally like-minded user communities by leveraging unsupervised feature learning (embeddings). Specifically, we learn a mapping from the user space to a low-dimensional vector space of features that incorporate both topics of interest and their temporal nature. We demonstrate the efficacy of our proposed approach on a Twitter dataset in the context of three applications: news recommendation, user prediction and community selection, where our work is able to outperform the state-of-the-art on important information retrieval metrics.

The increasing performance and wider spread use of automated semantic annotation and entity linking platforms has empowered the possibility of using semantic information in information retrieval. While keyword-based information retrieval techniques have shown impressive performance, the addition of semantic information can increase retrieval performance by allowing for more accurate sense disambiguation, intent determination, and instance identification, just to name a few. Researchers have already delved into the possibility of integrating semantic information into practical search engines using a combination of techniques such as using graph databases, hybrid indices and adapted inverted indices, among others. One of the challenges with the efficient design of a search engine capable of considering semantic information is that it would need to be able to index information beyond the traditional information stored in inverted indices, including entity mentions and type relationships. The objective of our work in this paper is to investigate various ways in which different data structure types can be adopted to integrate three types of information including keywords, entities and types. We will systematically compare the performance of the different data structures for scenarios where i) the same data structure types are adopted for the three types of information, and ii) different data structure types are integrated for storing and retrieving the three different information types. We report our findings in terms of the performance of various query processing tasks such as Boolean and ranked intersection for the different indices and discuss which index type would be appropriate under different conditions for semantic search.

Inferring user interests from their activities in the social network space has been an emerging research topic in the recent years. While much work is done towards detecting explicit interests of the users from their social posts, less work is dedicated to identifying implicit interests, which are also very important for building an accurate user model. In this paper, a graph based link prediction schema is proposed to infer implicit interests of the users towards emerging topics on Twitter. The underlying graph of our proposed work uses three types of information: user’s followerships, user’s explicit interests towards the topics, and the relatedness of the topics. To investigate the impact of each type of information on the accuracy of inferring user implicit interests, different variants of the underlying representation model are investigated along with several link prediction strategies in order to infer implicit interests. Our experimental results demonstrate that using topics relatedness information, especially when determined through semantic similarity measures, has considerable impact on improving the accuracy of user implicit interest prediction, compared to when followership information is only used.

The vast array of information available on the Web makes it a challenge for readers to quickly browse through and decide about the importance and relevance of content. Interpreting large-volumes of data is particularly demanding for users with handheld devices in the social media and micro-blogging sphere. Various approaches address this challenge through text summarization, content ranking and personalized recommendation. We describe a family of techniques that help users understand text by automatically annotating text with pictures, referred to as text picturing . The objective is to find a set of pictures that cover the main concepts in a textual snippet. We provide an overview of text picturing, its constituent steps such as knowledge extraction, map ping, scene rendering, as well as application areas. We give a picturing-related literature overview, and list use-cases that offer IT professionals insight into how picturing techniques can be successfully incorporated into real world applications.

One of the major challenges in Web search pertains to the correct interpretation of users’ intent. Query Expansion is one of the well-known approaches for determining the intent of the user by addressing the vocabulary mismatch problem. A limitation of the current query expansion approaches is that the relations between the query terms and the expanded terms is limited. In this paper, we capture users’ intent through query expansion. We build on earlier work in the area by adopting a pseudo-relevance feedback approach; however, we advance the state of the art by proposing an approach for feature learning within the process of query expansion. In our work, we specifically consider the Wikipedia corpus as the feedback collection space and identify the best features within this context for term selection in two supervised and unsupervised models. We compare our work with state of the art query expansion techniques, the results of which show promising robustness and improved precision.

Product line-based software engineering is a paradigm that models the commonalities and variabilities of different applications of a given domain of interest within a unique framework and enhances rapid and low cost development of new applications based on reuse engineering principles. Despite the numerous advantages of software product lines, it is quite challenging to comprehensively test them. This is due to the fact that a product line can potentially represent many different applications; therefore, testing a single product line requires the test of its various applications. Theoretically, a product line with n software features can be a source for the development of 2^n application. This requires the test of 2^n applications if a brute-force comprehensive testing strategy is adopted. In this paper, we propose an evolutionary testing approach based on Genetic Algorithms to explore the configuration space of a software product line feature model in order to automatically generate test suites. We will show through the use of several publicly-available product line feature models that the proposed approach is able to generate test suites of O(n) size complexity as opposed to O(2^n) while at the same time form a suitable tradeoff balance between error coverage and feature coverage in its generated test suites.

Software product line engineering is a paradigm that advocates the reusability of software engi-neering assets and the rapid development of new applications for a target domain. These objectives are achieved by capturing the commonalities and variabilities between the applications of a target domain and through the development of comprehensive and variability-covering domain models. The domain models developed within the software product line development process need to cover all of the possible features and aspects of the target domain. In other words, the domain models often described using feature models should be elaborate representations of the feature space of that domain. In order to operationalize feature-based representations of a software application, appropriate implementation mechanisms need to be employed. In this paper, we propose a Seman-tic Web-oriented language, called Semantic Annotations for Feature Modeling Description Language (SAFMDL) that provides the means to semantically describe feature models. We will show that using SAFMDL along with Semantic Web Query techniques, we are able to bridge the gap between software product lines and SOA technology. Our proposed work allows software practitioners to use Semantic Web technology to quickly and rapidly develop new software prod-ucts based on SOA technology from software product lines.

No abstract available for this paper.

No abstract available for this paper.