As artificial intelligence (AI) systems are increasingly deployed in critical domains such as healthcare, scientific discovery, and autonomous decision-making, ensuring that foundation AI models such as large language models (LLMs) align with human values and preferences has become essential for their safe and beneficial deployment. However, most existing approaches rely on large amounts of high-quality labeled preference data and assume clean, stable, well-controlled environments. These assumptions are often violated in real-world scenarios, where data may be limited, noisy, or subject to change over time. This project addresses the fundamental problem of aligning LLMs with human preferences under such real or open-world settings. The outcomes are expected to improve the data efficiency and reliability of LLMs in high-impact applications, including medical diagnosis and molecular discovery, while also contributing to education and workforce development through the integration of research and training activities. The project focuses on three key aspects of aligning LLMs with human preferences in open-world settings. First, it develops novel data-efficient preference alignment algorithms that enable LLMs to maintain effective alignment in open-world environments with limited human-annotated preference data. Specifically, when LLMs encounter new tasks or domains, the proposed algorithms can strategically minimize reliance on extensive human or AI annotation while maximizin