I first learned calculus in AP Physics with a similar approach, it was great! I did take a more "traditionally" taught sequence of calculus courses my first year of college. My college physics professors often prefaced math explanations with an apology to math majors in the room.