Families with children are a federally protected class. The restrictions on pool hours that you describe are prohibited as familial status discrimination. In HUD v. Paradise Gardens, a homeowners association was found liable for, among other things, limiting children to use of the pool only between 11:00 a.m. - 2:00 p.m. Such restrictions violate the Fair Housing Act unless the housing provider is exempt or unless the rules are necessary for health and safety.
In OP's case, housing for older persons is exempt from the prohibition on familial status discrimination. If the development meets the requirements at 42 USC 3607(b), the HOA may not be held liable.
Acceptable health and safety restrictions include requiring a parent to accompany a young child in the pool, or banning children who are not toilet-trained. But restrictions on pool hours for children are generally based on the convenience of adults, rather than health and safety.
