I do not have the parents sign any kind of release of liability form before we go.
That sure is scary. I suspect that children (assuming they are above the age of say 13) should have some degree of responsibility to stay with the chaperone, and thus I believe that a release of liability (placing behavior at least partly responsible on the child) would be a good idea. That doesn't mean you couldn't be negligent, but puts some degree of responsibility on the children (assuming you tried in good faith to prevent a bad situation).
You need to take at least some steps to protect you, other chaperones and the school system. That means things like having kids and their parents sign off on a code of conduct for the trip, limited liability (except as may be prohibited by law) and maybe even require some type of insurance (sure to increase costs though)
As to the actual legal ramifications, I go by the belief that a person can be sued by anyone, anywhere at any time. And there will probably be at least be some courts that would allow a suit to proceed (valid or not) and that means $$$, even if innocent.
But you bring up a good (and scary) point! I'll think twice before signing up as a chaperone again!