Invasions from Hephaestus and Ares are controlled by Aphrodite. Athena and Demeter ones will require you to pray to Hades, when the goddesses are near his sanctuary, so he can fight them. If not, you can always place a granary near them and pray to Poseidon and he'll deal with them instead. Anyway, the city is
mythologically protected. It's important to do so, otherwise they'll cause some serious damage to the city.