User Feedback on "3D Components in App Inventor" [GSoC-22 Project]

Open Source Development Google Summer of Code
,
#1

3D Components in App Inventor

An approach to enhance the functionality of MIT App Inventor by adding support of 3D components using OpenGL and Google AR-Core.

Hi everyone,

I am Preet Vadaliya(@preetvadaliya) one of the Google Summer of Code 2022 contributors and this summer of code I will be working on the 3D Components in App Inventor project and my mentors are Evan Patton(@ewpatton) and Susan Lane(@Susan_Lane).

About the Project

As we all know that we have a very good system for rendering and manipulating 2D shapes in the App using the canvas component, but currently, there is no rendering and manipulating system for 3D shapes, this project will add a new rendering and manipulating system for 3D shapes using OpenGL ES.

Use Cases

There are lots of use cases for 3D visualization, some of them are mentioned below…

  1. 3D modeling & shape drawing
  2. 3D Game development
  3. It will open a new door of development in Augmented Reality (Using AR Core)

Deliverables

The project is a bit long, we will not only add the new 3D surface (in which the object will be rendered), but we will also add some basic 3D shapes that will help beginners.
Similar to 2D drawing (Drawing and Animation), it will add a new component category for 3D visualization.

  1. New component category 3D Drawings and Animation.
  2. 3D shapes holder Space component.
  3. AR SurfaceView
  4. Realtime plane detection
  5. Basic 3D shapes.
    1. Cube
    2. Pyramid
    3. Sprites
    4. Other custom 3D shapes using coordinates.

Key Notes

First of all, I would like to thank Evan Patton (@ewpatton) for guiding me to make this proposal. I discussed it with him almost every day. Here is the summary of the talk…

  1. We will use ARCore as much as possible to render 3D objects on a real plane.
  2. We will try to follow Nichole Clarke's thesis in which she applied it for iOS.
  3. Each 3D component will be a first-order component.
  4. First we will try to implement an example in Android Studio and then try to replicate it in App Inventor.
  5. Work done before GSOC proposal
    1. Simple shape rendering on GLSurfaceView (Android & App Inventor).
    2. Custom 3D object on GLSurfaceView (App Inventor).
    3. Plane detection with ARCore (Android Studio).

Thanks
Preet P. Vadaliya

6 Likes
#2

Good luck with your project. :+1::+1::+1:

1 Like
#3

@Peter Thank you.

1 Like
#4

I did this a while ago, but I get a 3d metaverse effect, thanks to the android gyroscope, the steps taken and placing each android in each hand and connecting everything with firebase... you get a realistic movement for your 3d avatar... It's not very difficult and thanks to image recognition, even without using the gyroscope, it's just synchronizing and doing good visual effects by combining tricks.

#5

Oh that's nice, I would like to take feedback from you if you are available for short video call?

2 Likes
#6

Best of Luck on your work. :+1:

There is an extension for this by the way, but built-in components are better options.

1 Like
#7

I translate to communicate! But I'll try to help as much as I can! You will see in the app store you can find creators of 3d scenarios such as houses and backgrounds to try Ikea furniture recreating your house or terrace... You just have to create all the scenarios and then take screenshots of the possible steps that a character would take! you have to think about taking the photo from all angles of the same point within all the images that mounted in sequence by sliding a sliding component will recreate the effect of walking through a 3d scene only by changing the frame according to the number on the slider ! with the character you have to have all his positions in all the angles that can be seen according to your tastes of perspective! if it were in first person you should only create the hands from the angle in which the stage is seen in perspective of the avatar's eyes!!!!
Then to make an augmented reality viewer you can easily do it with app inventor's image classifier component! where you would pass the image that you see in the web component to the canvas and once you see the camera screen on the canvas you can put any 3d sprite making your animations easily in augmented reality!!!!

#8

augmented reality demonstration with the image classification extension! on canvas

#9

I tried with an extension but it did not achieve the desired effect so I looked for the concepts of creating a game engine from scratch and how to create your own fake 3d scenarios it took me a while to learn how to do it but the effects are in the 3d style of the years 90! And I really like it because it shows the great capacity that App Inventor has to recreate everything that other programming engines can do. although for some things you need to have a connection or extensions... but for my part I chose that app inventor is my perfect work tool, you just have to learn to improve and hope they keep updating improvements.

Thank you

1 Like