Any negatively charged particle will do. You can even use an anti-proton. Lifetimes may be rather short, however.
Exotic atoms cast light on fundamental questions - CERN Courier

The Paul Scherrer Institut in Villigen has investigated pionic hydrogen (π^--p) and deuterium (π^--d), and DAFNE in Frascati has investigated their kaonic counterparts. Other no-less-important species include kaonic and antiprotonic helium, which have been studied at the Japanese High Energy Accelerator Research Organization (KEK) and CERN, and yet another exotic variety is formed by the non-baryonic π⁺- π^-(pionium) and πK atoms. Finally, the antihydrogen atom, pbar-e⁺, which CERN has copiously produced, is in a class of its own owing to its importance for testing the CPT theorem to extremely high precision.

I suppose Pauli exclusion is irrelevant since we're talking about Hydrogen. But, in addition, having light bosons filling the orbitals would mean that Pauli exclusion doesn't contribute an "exchange energy" to the interaction between various hydrogen atoms, and so you get the benefit not only of smaller atomic radius but also of the ability of the atoms to actually overlap significantly.

We have met the enemy, and he is us — Pogo