When you file your application for exemption, they do ask when the organization was formed. So, it seems logical that they would question you about those earlier years, since they won't find any previous returns on file for you. You should really contact a tax professional (which I am not), or at a minimum, call the IRS and ask them anonymously. But, I doubt you'll have a big problem.
Assuming you're incorporated, you should have been filing Form 1120-A--there is no minimum revenue requirement. So, technically, even if you had no net income, you'd be liable for late filing fees, even if no tax.
But, the minimum revenue requirement for tax-exempt organizations is $25,000, which you fell far below. So, you could make a reasonable case that, even though the paperwork wasn't done, if you had been declared a tax-exempt organization, you would not only have had no tax due, but you would have had no filing due. (This all assumes that you had very little or no unrelated business income.) And then ask them to waive any associated penalties.
This isn't the kind of problem that the IRS spends tons of time and money on--there's just not enough benefit to justify the costs. I'd like to think they would work with you on cleaning this up for the future.
I didn't check your state requirements, but you may have an issue with your state tax office as well. But this probably isn't a big deal, either, especially if the IRS lets you off.