The main problem tends to be that things that resist Psi damage tend to do so quite heavily. But you should be fine. Just pick and choose your enemies and avoid ones with too many machines or psychics (Council/5th Column, Malta, high end Arachnos, and the Carnies, mostly; Freakshow count as human and Clockwork are actually vulnerable to Psi damage).
If you have to go against those groups solo, consider stealthing past spawns with a machine/psychic in them when you can (though try fighting through at least once in each level range - you might find they're doable, even with your lowered damage. And, if not, debt's temporary...).
Would I be setting myself up for fail? I honestly don't mind playing gimped power sets. I kind of enjoy it. I am not interested in /Mental or /Energy. The biggest problem I see is no other damage type besides Psychic outside of mines and my vet powers.