The elementary group I was involved with did all of the things your group does, but we also added some family events such as a picnic at open house and a family game night sometime during the winter. You're really not that far off from what the vast majority of parent groups do within their schools.
Perhaps the problem comes from not communicating what you do effectively. What kinds of school wide communications do you utilize? Our families got to see a complete layout of our budget for the year at a general meeting in the fall, so they knew ahead of any fundraisers and where those fundraising dollars would go. We also had a monthly newsletter and a display in the school with photos and financial info. Many groups now have websites too. I would suggest looking first at how your are communicating what you are doing.
If you are already doing lots of communications, then the next time someone asked me where the money was going I'd fit into my answer how that info had been communicated already. i.e. The money from the candy bar sale went to purchase 2 computers for the media center. Are you getting our monthly newsletter? We always put info like that in the newsletter.
Hello, I am the President of our PTO. But have gotten a couple of questions from the general student body parents (only 144 students) about what we do other than fundraise.
I explained that all of the money we raise goes into the school and listed lots of things they could go see & touch that came from us that enriched the school. Also we organize teacher helpers, homeroom parents, teacher appreciation, classroom parties.
I am asking you what are some roles your PTO/PTA has in your school. Maybe we need to do more, or maybe those parents (who are not involved in PTO by the way) are wrong and we do all we are needed.