3D Data is often modeled with discrete symbolic structure to improve editability and interpretability. However, inferring this symbolic representation directly from the data is challenging - learning methods often suffer due to poor gradient estimation over stochastic computational graphs (especially with categorical distributions). An alternate strategy is to bootstrap a neurally guided search process by increasing the likelihood of “good” structures discovered during the search itself. In this talk, I will first present this approach, which we term “bootstrapped learning” and then present my recent work, which improves neurally guided inference of visual programs by code rewriting. Our work improves neurally guided visual program inference across different visual languages and matches the reconstruction accuracy of domain specific architectures while inferring concise and interpretable programs. Overall, the talk with present the challenges associated with inferring symbolic representations from 3D data, and how bootstrapped learning and code rewriting can aid with this task.