At the moment, the only other option I can think of, aside from not using google drive, is to use a google apps script to download the audio files as base64 then convert them back to binary in the app (or play them as datauris in a webviewer)
If you have your own/schools online server, you / others could upload your files with ftp or through a web interface and get the urls that way
Going back to @Anke's suggestion for downloading all files in a zip, here is an AI2 solution to generating a zip on google drive and downloading and unzipping the files (uses @Taifun's zip extension)