Hello, fellow inventors!
Have you ever wanted to use powerful AI like Google's Gemini in your apps, but found that the responses are often long, unpredictable paragraphs that are difficult to use?
This guide introduces a powerful technique to transform a general-purpose AI into a fleet of specialized, reliable, and predictable APIs that you can build and use directly within your App Inventor projects.
The secret is the "Instruction + Schema" pattern.
The Problem with Standard AI Prompts
When you send a standard prompt to an AI (e.g., "Is this image safe for work?"), you might get back answers like:
- "Yes, that image appears to be safe for a professional environment."
- "I have analyzed the image and it does not contain any NSFW content."
- "Based on my analysis, this image is SFW."
These are difficult to work with in your app's logic. You'd have to use text-matching blocks, which can easily break. We need a better way!
The Solution: The "Instruction + Schema" Pattern
The core idea is to force the AI to respond in a strict, structured format that is easy to parse. We do this by giving the AI two things:
The Instructions (System Prompt): A detailed set of rules that tells the AI its exact role, what task to perform, and the format of its response. This is like a "job description" for the AI.
The JSON Schema: A template that defines the exact structure of the response. This guarantees that you always get the data back in a clean, predictable way that App Inventor's JSON blocks can easily understand.
By combining these, you can create your own custom AI "micro-APIs" for virtually any task!
How It Works: A 5-Step Guide
- Define Your Goal: What specific task do you want the AI to perform? (e.g., classify an image, extract data from an invoice, analyze sentiment).
- Design Your JSON Schema: Decide what information you need back from the AI and create a simple JSON structure for it.
- Write the AI Instructions: Craft a detailed prompt that tells the AI its role, explains the task, and explicitly includes the JSON schema it must follow.
- Make the API Call: Use the Web component in App Inventor to send your instructions and user data (like an image or text) to the Gemini API.
- Parse the Response: When you get the response back, use the
JsonTextDecodeWithDictionariesblock to turn the AI's response into a dictionary. You can then easily get the values for each key!
Practical Examples
Here are two ready-to-use examples you can adapt for your own projects.
Example 1: The NSFW Image Classifier API 
- Goal: To check if an image is safe for work and get a structured, reliable response.
- JSON Schema:
"type": "object",
"properties": {
"is_nsfw": {
"type": "boolean",
"description": "A simple true/false flag if the content is considered NSFW."
},
"confidence_score": {
"type": "number",
"description": "The model's confidence in its classification, from 0.0 to 1.0."
},
"category": {
"type": "string",
"description": "The specific category of the content.",
"enum": ["SAFE", "SUGGESTIVE", "EXPLICIT", "VIOLENCE"]
}
},
"required": [
"is_nsfw",
"confidence_score",
"category"
]
}
- AI Instructions (System Prompt):
"You are an automated AI content safety moderator. Your only function is to analyze the provided image and return a JSON object with your classification. You MUST respond ONLY with a valid JSON object. Do not include any text, explanations, or markdown formatting before or after the JSON. The JSON object must conform to the following schema:
is_nsfw(boolean): Set totrueif the category is 'SUGGESTIVE', 'EXPLICIT', or 'VIOLENCE'. Otherwise, set tofalse.confidence_score(number): A value between 0.0 and 1.0 representing your confidence.category(string): Classify the image into one of these exact categories: 'SAFE', 'SUGGESTIVE', 'EXPLICIT', 'VIOLENCE'.
Analyze the image and provide the JSON response now."
- App Inventor Usage: After parsing the JSON, you can use a simple
ifblock to check the value of theis_nsfwkey. Iftrue, you can hide the image or show a warning.
Blocks example:
Results:
SFW :
NSFW:
Example 2: The Invoice Data Extractor API 
- Goal: To automatically extract key information from an image of an invoice.
- JSON Schema:
"type": "object",
"properties": {
"vendor_name": {
"type": "string",
"description": "The name of the company or vendor that sent the invoice."
},
"total_amount": {
"type": "number",
"description": "The final total amount due, as a number."
},
"due_date": {
"type": "string",
"description": "The payment due date, formatted as YYYY-MM-DD.",
"format": "date"
},
"invoice_number": {
"type": "string",
"description": "The unique invoice ID or number."
}
},
"required": [
"vendor_name",
"total_amount",
"due_date",
"invoice_number"
]
}
- AI Instructions (System Prompt):
"You are an expert data extraction AI. Your task is to analyze the provided image of an invoice and extract specific details into a JSON object. You MUST respond ONLY with a valid JSON object. The JSON object must contain these exact keys:
vendor_name(string): The name of the company that sent the invoice.
total_amount(number): The final total amount due.due_date(string): The payment due date in YYYY-MM-DD format.invoice_number(string): The invoice ID or number.
If you cannot find a specific piece of information, the value for that key should benull.
Analyze the invoice and provide the JSON response now."
- App Inventor Usage: You can use the extracted data to populate text fields, create a summary, or save the information to a database.
Why This is a Game-Changer
- Reliability: No more unpredictable text. You get the exact data structure you need, every single time.
- Simplicity: It makes handling complex AI responses incredibly simple using App Inventor's built-in JSON blocks.
- Power: It unlocks the ability to create advanced features that were previously too complex or required expensive, specialized APIs.
I encourage you to try this method and share the custom AI APIs you create!
Tools I used:
com.mrkoder.file.util.dev.fileutilities.aix (79.2 KB)
Happy Inventing!