Ant colony - problem or not? Lawn has a 3-foot diameter squishy patch. Dug up a small part...

Lawn has a 3-foot diameter squishy patch. Dug up a small part and discovered a colony of small black ants. Spotted these ants everywhere (but not contained by house). No obvious holes near the colony, but plenty of tiny holes widely spread all over one yard (area of give or take a few 300 square feet). They avoid holes I've sprinkled with cornmeal. I read elsewhere that they'll probably just move - but to where on earth? The grass over the colony is squishy but reasonably healthy - so far. Question is this, at what point should I steal serious action?
Answers:    I'm a believer in getting rid of pests rash, before they are a serious problem. If you can find a natural or natural repellant, more power to you. But, if that were my yard, I would without beating about the bush apply a granular pest killer.
when you cant go outside in need u bothering them or they r in ur house