I am a GSoC participant working this summer on “Visible Component Extension” (VCE). My mentor is @Susan_Lane, who is very helping, enthusiastic and encouraging.
My project is to upgrade the extension development by allowing visible components to be created as extensions. This will allow App Inventor users to use new and different kinds of UI elements to enrich their apps. Extension developers would no longer have to use hacks to create UI elements in the apps. The VCE will look almost identical to the built-in visible components.
I am requesting you all to provide your valuable feedback. There are some open questions (see the last page of the design doc) for us to brainstorm. If I miss something or you don’t like anything, please tell me. This is the right time we can work on designing this project better. More brains, more feedback, the better
Thank you for spending your valuable time in helping App Inventor develop further
I’m not an Extension developer but I am an Extension User and I can see this proposal has a lot of merit. I think it is going to be hard work to produce this enhancement but Susan Lane has great experience and is an ideal mentor.
Concerning the need for a Sandbox, I feel that provision should be a seperate project - ideally, the Companion would also be the Sandbox.
Hi Pavi! Thank you for sharing your design doc. I think it looks great
I’m not an extension developer either, but from my perspective it looks like you have a good plan! I think a delegation pattern is a good way to handle instantiating the MockComponents.
The sandboxing vs no sandboxing definitely sounds like a tricky problem. But after some contemplating, I think sandboxing is probably the better option.
The thing is that JavaScript opens up so many possibilities, it would be hard for extension devs to test things (you get into browser-specific issues). So in addition to helping with security, it seems like sandboxing would allow you to create a simplified “interface” for extension devs, which would make their code more testable/maintainable. I could be misunderstanding how that works though hehe.
As for how extension devs would create the visuals: maybe you could provide them with a canvas that the visuals must be drawn inside? That way they don’t need to deal with DOM objects (which can be a pain). And it would probably be easier to test on your end. I’m not sure if that would work though.
Thank you again for sharing your doc! I’m excited to see how this progresses
I would recommend using the DOM to load the preview as HTML is easy to learn and maintain, and the browser handles most of, if not all, draw and paint operations.
If you choose to go with the DOM, perhaps you could load it inside an iframe? That would provide some sandbox protections against accessing the parent (in this case, ode)
Hi @pavi2410
Thank you and best wishes for your project
I am an extension developer and honestly I was looking for a method to create visible extension components (Although I am weak in UI).
Security managers do not provide a secure environment for executing untrusted code and are unsupported on Android. Untrusted code cannot be safely isolated within a single VM on Android. Application developers can assume that there's no SecurityManager installed, i.e. System.getSecurityManager() will return null.
I doubt we can do much on Android side. Probably, the Kawa interpreter has some sandboxing ability which I don't know.
Probably, this is a good idea because it will give us more time to come up with a good solution.
I'm inclined towards using DOM elements as suggested by @vishwas. This is due to the fact that drawing on a canvas is not trivial as it requires the knowledge of geometry . On the other hand, the DOM is easier to manipulate and many UI elements can be created using plain HTML.
I am not sure if iframe would work best, but during implementation phase, I'll try out different methods.
Thank you everyone for your responses. Keep them coming!
I might be a bit outdated on this subject and recent work on this, but I'm kind of wondering why the focus is on pure HTML/DOM rather than the GWT abstractions, which is how the current visible components work in the Designer? (and also given that the Android side typically uses native Android components/external libraries rather than a WebView with HTML elements)
The first ideas that came to mind when working on a project such as this one was to allow external devs to make use of GWT's native functionality using widgets and the like, and potentially allow extra javascript additions using JSNI (this is also something that was done in the current Map component, unless that has been changed in recent time); As such, maybe there's something already present in GWT, or some external tools that could help in creating a safer environment for external users?
Generally in terms of security, I think there will more than likely be holes that cannot be patched; Arbitrary code execution has a lot of attack points on security, and achieving full security is most likely infeasible (even running Docker containers on certain applications is not 100% secure, so with our limits there will be trade-offs); Alternatively, we could take a look at many common security issues that arise, and how to circumvent them. For instance, perhaps the visible components developed by external users could be blocked the permission to download files via some tweaks on the AI backend/frontend, or maybe the Java's permission managing combined with GWT allows to restrict permissions to certain parts of App inventor in a way that blocks off quite a few security vulnerabilities (at least on the designer side, given that this is blocked off by Android).
This might be a possibility, since Kawa runs on the Java platform, which has measures for security management. It might be a good direction to look into in that sense, but I'd probably focus more on the Java side rather than the pure Kawa side, since you can call into Java from Kawa.
Hi @pavi2410, in the case of the Android side, we don’t enforce security of the extension code their for nonvisible extensions, so I don’t think it’s super critical to address that problem for visible extensions. The only way we could really potentially handle this is by running each extension in a separate Android process, but that would be way too heavy handed in the companion and not reflect the structure in the final app (where everything is compiled together).
Kawa doesn't really provide any support for this. Further, in many cases extensions will interact with Android native APIs, and so if they can talk to the rest of the Android system there's not much to stop them from including malicious code (this is also true of non-visible extensions today).
Good point, due to the interaction indeed Kawa shouldn’t provide any extra power in that regard.
I’d think that due to the fact that some security issues are unpreventable, we could directly outline the potential harm in using extensions (if not done so already?); For instance, upon importing an extension, a user could be presented with a warning saying that it comes from an untrusted source, and that they should use it at caution; In general, the rest probably boils down to the community & trust built around certain extensions. This is probably slightly outside of the scope of the VCE system however, and more of a future consideration.
It could also be worthwhile to analyze certain common Android vulnerabilities nowadays and especially in the older APIs. Maybe there’s something we could at least try to prevent/warn about (e.g. tampering with permissions? since it’s in control of upper-level components in AI)
Another (somewhat experimental approach) would be to introduce some form of static analysis if we want to go the extra step, e.g. per OWASP recommendations; This could be used in a notification outlining that there are some potential vulnerabilities; though that has it’s own downfalls, among which would be the general unreliability of these tools.
We could create an interface or an abstract class, which will then be extended by the extension developer (optional, as the extension dev may choose not to create the mock component, so we provide a default implementation which just shows the extension’s icon). Due to the limitations of GWT, we cannot use Java Reflection nor load JS files dynamically using eval() due to security concerns. After learning more about GWT, I found about “Deferred Binding”. But, according to the docs, “Deferred binding is a feature of the GWT compiler that works by generating many versions of code at compile time, only one of which needs to be loaded by a particular client during bootstrapping at runtime. …”, this may not work for us because we want to be able to load many different visible component extensions.
Moreover, there are two more points I'd add:
GWT compilation is too slow.
Using GWT will require extension devs to have a knowledge of GWT.
I'm not sure how we can dynamically link extension's Mock with the ODE. Maybe there's a way with the GWT compiler, but again, I'm not sure, given my little knowledge of GWT. I'm curious if there's a GWT way to do this. This might simplify things by using same MockComponent superclass and can use the type safety of Java.
What I have found is Deferred Binding which I have mentioned above why this wouldn't work.
The SimpleContainer Mock doesn’t accept any component inside it because it is just a MockComponent, and not a MockContainer. There needs to be a way to detect container components as the built-in ones are hardcoded.
While looking for a GWT based solution for creating Mock Compoents, I came across gwt-exporter library. This GWT library can export class, method, field. Using this library, we can export the MockVisibleComponent and MockContainer, so this will save a lot of time in matching their API manually to the <Component>.mock.js.
To save you a click, here’s what it does
package jschismes.client;
@Export
public class HelloClass implements Exportable {
String prefix = "Hello: ";
public String helloMethod(String name) {
return prefix + " " + name;
}
}
<script language='javascript'>
var hello = new jschismes.client.HelloClass();
hello.helloMethod("Manuel");
</script>
I would like to know your opinion before getting deep into this.
That seems like a nice abstraction that could definitely benefit in this scenario! I would like to point out though that the last commit was 5 years ago, so there can be issues with the library being unmaintained and so on; It’d be great to first experiment around with it to see if you don’t run into any walls while developing that require manually tinkering with the library
In addition to these two items, I imagine that there will be additional challenges insomuch as:
We compile App Inventor with optimizations turned on, which means that functions may be inlined, names minified, and entire chunks of code dropped based on static analysis, and this is done per browser/language combination.
There's no guarantee from version of App Inventor to version of App Inventor that the mapping will remain the same, which means that a future version of App Inventor could break a VCE. Use of GWT's JsInterop functionality may help us mitigate this issue. I think this would be preferable to gwt-exporter, since JsInterop is a part of GWT rather than a separate library.
If the VCE code is running in a sandbox, but makes use of code outside of the sandbox (App Inventor core GWT), that greatly increases the surface for potential security holes.
I am seeing a possibility for having a GWT based approach.
Extension devs create a Mock<Component>.java file. The Mock class can extend either MockVisibleComponent or MockContainer. If such file is not found for an extension, we will create a generic Mock instead.
The Mock<Component>.java file is compiled to JS code using GWT compiler. (I need to investigate further into this - Things to look for: setting up a compile classpath and a GWT module file).
The generated JS file will then be injected into the ODE using ScriptInjector.
The ODE will register the Mock<Component> class against the type name of the extension. This will allow us to instantiate the class when a VCE is found.
I need more help in evaluating the feasibility of this method. If this works, then this is making VCEs more simple and in line to the developement of built-in MockComponent.
I was biased here, so let me add a few points in favour of GWT :
Java code provides compile-type safety. So, fewer errors in runtime.
The MockComponent API doesn't need to be emulated.
The extension devs can write Mocks in Java and reuse the existing MockComponents.
I don't know how I missed JsInterop, but while I was searching for exporting Java types to JS, I came across J2CL (which has same JsInterop feature), and gwt-exporter from an old GWT group discussion.
It's been a month since I started working on this. I have been working on a solution to provide the ability to define custom Mocks. So far, I have something to share with you.
Here are the updates:
So, now the external Mocks would be written in JavaScript. This is because in my testing, GWT purposely compiles code statically and this is what limits the possibility of writing Mocks in GWT/Java (which could be a plus considering internal Mocks are written in GWT/Java).
(Please refer to the comments in the code)
// This class is instantiated by the GWT end
class MockSimpleLabel extends MockVisibleExtension {
static TYPE = "SimpleLabel" // the name of the Mock
constructor(editor) {
super(editor, MockSimpleLabel.TYPE)
this.label = document.createElement("p")
// initialise the Mock by using DOM elements
this.initComponent(this.label)
}
onCreateFromPalette() {
this.changeProperty("Text", this.getName());
}
// listen to changes in the property so as to make your Mocks reactive
onPropertyChange(propertyName, newValue) {
super.onPropertyChange(propertyName, newValue);
switch (propertyName) {
case "Text":
this.label.textContent = newValue
break
}
}
// static factory method used by the GWT end
static create(editor) {
return new MockSimpleLabel(editor)
}
}
// register the Mock class here
MockComponentRegistry.register(MockSimpleLabel.TYPE, MockSimpleLabel.create)
Preview of MockSimpleLabel in action
In the screenshot below, you can see that I was able to create a Mock from my custom extension (SimpleLabel; clone of the Label component) and was able to change its "Text" property successfully)
Note the above may change at a later point in time
There are more challenges, notably,
Auto-load Mock JS, handle multiple projects, upgrading/unloading of Mocks.
Securely load the Mock JS which could not tamper the App Inventor's client code running in the browser nor show annoying popups, ads, alerts, sent private data, misuse of being served from the same origin as appinventor.mit.edu (or even mit.edu)
If you have suggestions on how to tackle these challenging problems, I'd be more than happy
Another month has been passed and I'm here to share the update on the status of my project. This month, I tried to polish the implementation, fixed the Mock loading behavior (this was a lot trickier than I anticipated).
And for testing my project, I have a created a new extension "Cowsay"
You will be able to use this extension to make this cow say "Moo" or anything you want. But be careful
Find this extension's source here: GitHub - pavi2410/vce-samples at cowsay to get an early hands-on how a VCE is written and works. It is a rather simple extension but you should be able to create your own easily. If you find any trouble, ping me.
For the 3rd month, I am working on creating some more example extensions and write docs for writing a VCE.
in the apps. The VCE will look almost identical to the built-in visible components.
. On the other hand, the DOM is easier to manipulate and many UI elements can be created using plain HTML.
:
