If you buy a house as your primary residence with a mortgage, and then you upgrade to a bigger house as a new primary residence and get another mortgage while keeping your first house as second home; do you have to tell Fannie Mae that your first house is no longer your primary residence? Doubt it.