If you owe the IRS over $52,000 in back taxes, including penalties and interest, you could lose your passport if you're not involved in a good faith attempt to resolve the debt. This is considered a "seriously delinquent" tax debt under the Fixing America's Surface Transportation Act (FAST Act).
If you owe money to the IRS, you may understandably be quite worried. The IRS has a fearsome reputation. It's powerful, too: the agency has the power to collect when you do not pay, for example by garnishing your wages or levying your bank accounts.