... Having public meetings there or community celebrations does not imply that the community endorses in any way the particular denomination that uses the structure.
Your belief echoes the belief of Justice Scalia and Justice Thomas.
The problem is that, if someone does not share the belief that holding public events in a church does not imply endorsement, you wind up with an Elmwood dispute - and potentially a decision like the one held by the 7th US Circuit Court of Appeals which barred the school district from using the church as a venue for the high school graduation ceremonies, calling it unconstitutional and "religiously coercive."
In the Elmwood School District, the church was chosen as a venue for the high school graduation for the same reason that many smaller communities will turn to a church for public meetings - a church is often the largest or most convenient venue for community members to gather. As FlyingRon noted, it is not uncommon to find polling places located in churches. And many churches rent out their facilities for events.
This does not eliminate the concern some have, or the discomfort some might feel, in being in a house of worship surrounded by the trappings of a religious belief that is contrary to their own.