The next generation of service and assistive robots will need to operate under uncertainty, expected to complete tasks and perform well despite missing information about the state of the world or the needs of future agents. Many existing approaches turn to learning to overcome the challenges of planning under uncertainty, yet are often brittle and myopic, limiting their effectiveness. Our work introduces a model-based approach to long-horizon planning under uncertainty that augments (rather than replaces) planning with estimates from learning, allowing for both high-performance and reliability-by-design. In this talk, I will present a number of recent and ongoing projects that leverage our high-level planning abstraction to improve navigation and task planning in partially-mapped environments. I will additionally discuss how our high-level planning abstraction affords capabilities unique in this domain, including explanation generation, single-shot interventions (online behavior correction), and deployment-time reward estimator selection. Critically, I will present a unified perspective on these recent advances and show how they may be made compatible with STRIPS-like planning (e.g., via PDDL), and how our "learning-augmented PDDL" will allow for common-sense-like behaviors in both fully- and partially-revealed environments.