Problem Description:
How to allow an imported p5.js file (Tensorflow file from Teachable Machine) running as a 'html", to use the microphone and collect the speech samples ?
Is there a better way to use ML models from Teachable Machine ?
Development History:
Teachable Machine ML model created to recognize voices of 3 people. Final accuracy is between 70-90% (when run on the Teachable Machine platform with my Window laptop and a built in microphone).
Now I want to use the model in my MIT AppInventor app by extracting it from the Teachable Machine in a Tensorflow.js file version so it can be run in a browser as a html file.
My plan was (is) to use the WebViewer1 component as a framework and communicate with the imported ML model code (html file) by using::
- window.AppInventori.getWebString()
- window.AppInventori.setWebString()
I copy-pasted the ML model code snippet provided by the Teachable Machine (Tensorflow.js / p5.js) into a text editor and saved it with a "html" extension.
The first goal was to make the simplest, basic AppInventor app that can just start running the ML model. Here is the status of the testing so far:
a) My laptop (Windows 10, Google Chrome Version 109.0.5414.75 (Official Build), 64-bit),
Running the downloaded ML Model code ("person_voice_model_30epoch.html") in the browser by double clicking on it
The "model" starts, sets the "screen", and "prints" the text (label) "listening" while "asking" and waiting for permission to use the microphone (please see screenshot ILL 1).
After the permission is granted, the model works well
b) In my AppInventor app, on my smartphone (OnePlus 6T, Android version 11) I am experiencing challenges.
Uploaded the "ML model file" (html) to my app as a Media file (under the WebViewer1 component)
After "Screen1" initializes, the app activates the js script (the uploaded html model) but sits in the 1st step, wiht the text (label) "listening" on the screen (please seee screnshot Ill 2)
Most likely, the ML code is waiting on permission to use the microphone but that request popup window never shows up (unlike in the scenario "a" above")
Please see the ILL 3
What was tried:
Tried with "SoundRecorder" and "Speech Recognizer" Media components (activate them in a hope that it will "somehow enable" the microphone but obviously, no results.
Looked through Q&A in MIT AppInventor Forums and YouTube and saw some examples where a "Sound Sensor" was called and used (activated etc) in a context of handling sound recording / microphone, but I can not locate Sound Sensor in my App Inventor ..?
Questions:
- Any suggestion on how to allow that imported ML code (run as html in browser) to access the microphone in order to collect the voice samples ?
- Is there a better way to embed / use Teachable Machine ML models in MIT App Inventor ?
[I can comprehend java and html but am not a skillful java developer]
Thank you