I am a big fan of App Inventor and regularly use it in coding classes with children.
I am currently exploring the ChatBot extension + the Provided ChatGPT Tutorial and would like to use it in a classroom setting. However, I have some concerns regarding content safety.
Since students can freely enter prompts (e.g., via the TextBox), I am worried that the AI might generate inappropriate or otherwise unsuitable content.
At the moment, I am considering implementing my own filtering mechanisms (hiding prompt specification with own blocks+blacklist), but I would prefer to understand what is already provided by App Inventor
I have read that prompt requests are sent through the MIT proxy server. Therefore, I would like to ask:
Is there any kind of content moderation or filtering applied to prompts or responses on the MIT side?
How do other educators handle this situation in practice?
My goal is to create a safe and age-appropriate learning environment, so any guidance or shared experiences would be greatly appreciated.
OpenAI offers a moderation API that accepts a prompt that it
evaluates for various kinds of harm. The MIT Chatbot proxy runs every
prompt through it, even prompts for other models. If the moderation
API flags a prompt as harmful, the proxy returns an error.
I cannot speak to how accurate the moderation API is, but it looked to
us as the best approach to provide some safety for large language
models. Many of the models themselves have “guardrails” in addition to
what we do by using the OpenAI moderation API.
Thank you very much for the fast response! That was really helpful!
I am still a bit confused- For using the experimental extension ChatBot- we are using the default settings (Api Key: blank, Model: default, Provider: ChatGPT,System blank)
We read that the last model used in the default settings is Sonnet-4.5 from Amazon-is this still uptodate?If the model is from amazon- how is OpenAI involved in this? And who is then responsible for the data processing of eg. the prompts- Amazon or OpenAI?
For me the used provider is crucial to know as parents of my class are highyl interested in the data protection regulation- which differs slightly from provider to provider.
The splash screen in the Designer indicates “we will now be using the “meta.llama4-maverick-17b-instruct-v1:0” model via Amazon’s “Bedrock” service.” .and imagine this is the model currently in use per @jis . “If you use the default, which sets the provider explicitly to “chatgpt” and do not select a model (leaving the model field blank, or set to “default”) we will use the llama4 model.”
We call OpenAI's content moderation API for every query, regardless of the API key, provider, or model. Claude Sonnet, served through AWS Bedrock, is the current default. So when someone calls the Converse function, we will first send their prompt to OpenAI to check it for moderation (this is free to us) and then if that passes, we send it to the Bedrock API naming Sonnet as the model to process the request. It is possible that the provider can still reject the request even if the moderation API allowed it. Google Gemini in particular seems to be more strict in terms of what it allows compared to what the moderation API allows.
Thank you very much- this really was helpful information!
Now I am still unsure which way is the best in regard to the data protection regulation. As now not only OpenAI but aslo Amazon is involved I am not sure what really is happening with the data.
Do you think it is better to only use OpenAI (own Api key)?
You can always use your own API key if you like. You can also choose an explicit model from the Model dropdown to prefer to use OpenAI's models. If you do not use your own API key, we provide up to 100k tokens of usage per day for free, which should be sufficient for most classroom usage.
It's also worth noting that for any sort of data protection regulation that all of the services in question are US based.