Sign Language (ASL) from SpeechRecognizer Dictation; a Template

Sign Language Parrot Template is an App Inventor template aia.
hello

The example will help create an app that displays finger spelling sign language letter icons.

The app recognizes individual words captured by the SpeechRecognizer and displays a sign glyph for each letter in the word. The individual letters are captured from word text generated by a SpeechRecognizer This is one way to do to use sign language; there are many more potential ways to code a similar app (see resources below).

The app template knows only five signs. Add more glyphs using your own sign images. The example images are from the Internet. If a spoken word contains letters for which you have not provided a sign glyph, no image will display for the 'unknown' letters.

The decodeWordToLetters Procedure captures the SpeechRecognizer text. Label2 displays the individual letters of the text. The display rate of the signs is controlled by the Clock1.TimeInterval (currently set to 2000 ms [2 seconds] in Screen1.Initialize). Add Blocks to make this display rate selectable based on your preferences).

Try it out: With the glyphs provided, press the Speak button, then say 'hello help hello' and watch the ASL glyphs display in turn. After you do that, use the Manual button to 'walk through the characters' one at a time. Use the 'switch' to have the TextToSpeech 'say' the letters (if the device volume is set high, doing this can elicit a response from the SpeechRecognizer, so be aware.). The software works with single words or short phrases.

Individual glyphs are 'cut - out' from a larger image.
cutoutSign
I used the Microsoft Paint program on my PC. Each cut out image is saved as a 'letter' image to the Media; for example E.png, H.png, L.png . A glyph is required for each letter you want the app to be able to sign. Cut out glyphs or create your own images from photos of your signing.

The example is relatively simple code. You as a developer can expand its utility:

  • turn the app into a sign language trainer as a basic 'flash card' display . Make your app more complex; perhaps in conjunction of the Personal Image Classifier (PIC) extension (Create a training file with a collection of sign language images to compare to the signs you make with your hands). Personal Image Classifier: Part 1
  • use a complete example to communicate with a deaf friend (until you learn sign language).
  • use it with other images to create a game of word pictures of animals for children.
  • create an extension to do what the decodeWordToLetters Procedure does.

The signing can be automatic (as coded) or the app can display each glyph in turn using the Manual Button. Display the signs, one at a time by pressing the Manual Button after using the SpeechRecognizer to produce an automatic display.

The finger spelling example app works on an Android 8.1 and 11 using English. It is a template; not a finished app. Finish it by adding the rest of the alphabet glyphs, features you want/require, error control or adapt it to make it respond as you want it to work. The example does not require any extensions.

Additional Resources:

I hope someone will find this small bit of code useful.

When you improve it; consider posting your enhancements here for others to use.

srSignLanguage2.aia (46.5 KB)

Regards,
Steve

7 Likes

Resource: BSL (British Sign Language) alphabet images

BSL_Alphabet.zip.txt (456.0 KB)
(download and remove .txt to open zip.)