The goal of my research is to develop self-learning search engines, that can learn online, i.e., directly from interactions with actual users. Such systems can continuously adapt to user preferences throughout their lifetime, leading to better search performance in settings where expensive manual tuning is infeasible. Challenges that are addressed in my work include the development of effective online learning to rank algorithms for IR, user aspects, and evaluation.