I always have an agenda for the board meeting. We are lucky to have a large board. (There are 12-15 positions) It gives everyone a chance to be heard, and it makes sure I don't forget to talk about something!
Yes, an agenda is important to stay on track and focused. Usually, we have only 2 board meetings per year and there is a lot of ground to cover. Not having an agenda... I know I would probably forget something and as Raf said, people would start digressing from the task at hand and most likely you would have a 4 hour meeting!
I did, but then again, I will probably prepare an agenda for my own funeral. It seems to keep board members from talking about stuff out of turn and lets them know how much more we have to cover before I'll let them go!