Upon first entering Bavner, the building in the northeast corner is the Anama temple. You'll find Mother Loomis here. Ask her about her beliefs, and agree with what she says. She'll also give you a quest to visit Point Contemplation.
The building just west of the temple is the common room for Bavner. The last of the northern buildings, on the far west, is a barracks that has been transformed into an inn. It's run by Spurn, who stole the Nephil's necklace. Before demanding it back save your game, as you'll end up having to kill him if your leadership isn't high enough, and that will make the Anama angry with you. If that happens simply load your saved game and wait until you have a little bit more reputation to try again. I've never had to fight him to get it back, but others have said that they did.
The other building on the western side of town is two barracks.
There are two people who can sometimes be found wandering around in Bavner. The first is always there. Her name is Seela and she'll tell you about the GIFTS. She thinks they're "sweet". The second person can be found either here in Bavner or in Shayder. Her name is Judith, and she's a member of the Church of the Divine Lucre. She can heal you, and later on in the game you can find out about a certain artifact from her, for a price.
The building just to the southeast of the barracks is a Training center, where you can use your skill points to train your characters. It's run by Voorn.
The last building, in the southeast corner of town is a house. Walk close to the table to get a small pouch. This is a Health Charm. Now check the dresser to find some cockroaches in the process of getting intimate. They'll pop out of the drawer and attack you.
Now leave Bavner and head east to Kneece.