It will not damage the filter. You could actualy run the aquaclear with no inserts at all, but the only thing that you do is circulate water. Each of the inserts has a role in the aquarium.
The foam keeps with dirty elements from going back into the water. The carbon/amonia clears out the ammonia produced by fish waste, the top is used for stuff like biomax wich can store good bacterias and others.
The bottom two are the most important. Good bacteria can also reside in the sponge, and the ammonia remover is very important. So I would not go without the ammonia remover, it would not help the fish at all not to have it present. Ammonia poisoning is not rare in fish death in aquarium.
Never change all three elements at once!