How do you make app remember access granted to USB?

I've been busy building my 1st app - time came to test connection to multiple devices - for serial over USB I use great serialOTG by @rkl099. The problem came as I connected to a device that exposes not only Serial but an audio card as well (my app does not use it). Android asked me for permission:

Ok, I'm fine with Android' idea on security and user consents.
The problem is how to make my app remember this grant so that next time this question does not re-appear again. Any suggestions ?
I've read multiple suggestions about editing manifest and putting there specific USB filters but it does not seem right for 2 reasons: 1. there are many radios on the market and each has different ID, 2. additionally changing and re-signing aab seems too complex for me.
Regards, Kuba