Change your pants. Instead of standard blue jeans, reach for colored or patterned jeans. Even if you're still wearing a plain white tee with them, it will instantly look like you put great thought into your outfit.
Consider a skirt. Skirts are undervalued as casual attire in America (at least in the Midwest), but skirts are available in comfy jersey, cotton and denim. Put a skirt with a casual top, flats, and minimal jewelry.
Get a drawstring dress. A drawstring waist, common on sweatpants, is ultra casual. Put a drawstring waist on a dress and you have casual without the stink of the gym.