The FHA article mentions explicitly several times that discrimination is allowed when it is necessary for health and safety of pool patrons. I believe banning infants from the pool could be argued as falling under both of these categories (it's unsafe for toddlers to be in adult pool; they should be in a safe shallow baby pool )