Rethinking The Foundations Of Visual-Inertial State Estimation