How do I make an extension that retrieves the source code of a web page?

Meulencv · March 7, 2021, 3:42pm

Could you tell me where I can find it,
thank you

marco_tanzi · March 7, 2021, 4:05pm

If you have to retrive entire structure of web Page ,web component is useless.
You have to embed your Page in a function

Meulencv · March 7, 2021, 4:09pm

But I want to get the source code of the web to then do other things (like web scrapping), what I need is to get the code that I have to put in java to make the extension have the ability to get the source code of the web.

pavi2410 · March 7, 2021, 4:23pm

You can directly instantiate the Web component and call the Get function inside your extension code like this

Declare a field

private Web myWebComponent;

Instantiate the Web component inside the constructor

myWebComponent = new Web(...);

Register a GotText callback inside your extension class

@Override void GotText(...) {
    // Receive the text from here
}

Use the Get function from Web component like this

myWebComponent.Get(...);

Meulencv · March 7, 2021, 4:24pm

thanks @pavi2410

Kumaraswamy · March 7, 2021, 4:38pm

Did you search the community?

Meulencv · March 7, 2021, 4:55pm

thank you very much for the information

Meulencv · March 7, 2021, 5:03pm

Hello, Am I doing it right?

import com.google.appinventor.components.runtime.Web;

import android.content.Context;
import android.util.Log;
import com.google.appinventor.components.annotations.*;
import com.google.appinventor.components.runtime.*;
import com.google.appinventor.components.common.ComponentCategory;

@DesignerComponent(version = 1,  description = "This Extension was created with the AppyBuilder Code Editor.<br>" + 
                   "Create your own here:<br><a href='https://editor.appybuilder.com' target='_blank'>https://editor.appybuilder.com</a><br>",
        category = ComponentCategory.EXTENSION,
        nonVisible = true,   iconName = "http://appyBuilder.com/extensions/icons/extension.png")
@SimpleObject(external = true)
public class Get_source_web extends AndroidNonvisibleComponent {
    private ComponentContainer container;
  

  
    public Get_source_web(ComponentContainer container) {
        super(container.$form());
        this.container = container;
    }
  
 @SimpleFunction(
            description = "Website source-code")
    public void WebsiteContent(final String website) {
        AsynchUtil.runAsynchronously(new Runnable() { // Always do get request Asynchronously
            @Override
            public void run() {
                try {
                    BufferedReader readStream = new BufferedReader(
                            new InputStreamReader(
                                    new URL(website).openStream())); // Open stream and read the content from streamReader to BufferedReader
                    String readLine; // Variable which will have one line of data
                    StringBuilder data = new StringBuilder(); // The result data will be stored here
                    while ((readLine = readStream.readLine()) != null) data.append(readLine); // Read all the lines from the readStream
                    readStream.close(); // IMPORTANT close the stream

                    final String finalData = data.toString(); // Make the string final with the data variable
                    activity.runOnUiThread(new Runnable() { // Always raise events on UI thread
                        @Override
                        public void run() {
                            myEvent(finalData); // You're event with data
                        }
                    });
                } catch (IOException e) {
                    e.printStackTrace(); // Error occured
                }
            }
        });
  
  
}}

I get the following error:
Buildfile: /projects/goldv2/appinventor-sources/appinventor/build.xml

extensions:

clean:

init:

common_CommonUtils:

init:

CommonUtils:

common_CommonVersion:

init:

CommonVersion:
[exec] Result: 128
[exec] Result: 128

CopyToRunLibDir:

components_AndroidRuntime:

init:

CommonConstants:
[javac] Compiling 6 source files to /projects/goldv2/appinventor-sources/appinventor/components/build/classes/CommonConstants
[javac] warning: [options] bootstrap class path not set in conjunction with -source 1.7
[javac] 1 warning

HtmlEntities:
[javac] Compiling 1 source file to /projects/goldv2/appinventor-sources/appinventor/components/build/classes/HtmlEntities
[javac] warning: [options] bootstrap class path not set in conjunction with -source 1.7
[javac] 1 warning

common_CommonVersion:

init:

CommonVersion:
[exec] Result: 128
[exec] Result: 128

AndroidRuntime:
[javac] Compiling 332 source files to /projects/goldv2/appinventor-sources/appinventor/components/build/classes/AndroidRuntime
[javac] warning: [options] bootstrap class path not set in conjunction with -source 1.7
[javac] /projects/goldv2/appinventor-sources/appinventor/components/src/com/appybuilder/meulencv/Get_source_web/Get_source_web.java:57: error: reached end of file while parsing
[javac] }
[javac] ^
[javac] 1 error
[javac] 1 warning

BUILD FAILED
/projects/goldv2/appinventor-sources/appinventor/build.xml:35: The following error occurred while executing this line:
/projects/goldv2/appinventor-sources/appinventor/build-common.xml:372: The following error occurred while executing this line:
/projects/goldv2/appinventor-sources/appinventor/components/build.xml:141: The following error occurred while executing this line:
/projects/goldv2/appinventor-sources/appinventor/build-common.xml:118: Compile failed; see the compiler error output for details.

Total time: 2 seconds
mlEntities:
[mkdir] Created dir: /projects/goldv2/appinventor-sources/appinventor/components/build/classes/HtmlEntities
[javac] Compiling 1 source file to /projects/goldv2/appinventor-sources/appinventor/components/build/classes/HtmlEntities
[javac] warning: [options] bootstrap class path not set in conjunction with -source 1.7
[javac] 1 warning
[jar] Building jar: /projects/goldv2/appinventor-sources/appinventor/components/build/HtmlEntities.jar

common_CommonVersion:

init:

CommonVersion:
[exec] Result: 128
[exec] Result: 128

AndroidRuntime:
[mkdir] Created dir: /projects/goldv2/appinventor-sources/appinventor/components/build/classes/AndroidRuntime
[javac] Compiling 332 source files to /projects/goldv2/appinventor-sources/appinventor/components/build/classes/AndroidRuntime
[javac] warning: [options] bootstrap class path not set in conjunction with -source 1.7
[javac] /projects/goldv2/appinventor-sources/appinventor/components/src/com/appybuilder/meulencv/Get_source_web/Get_source_web.java:57: error: reached end of file while parsing
[javac] }
[javac] ^
[javac] 1 error
[javac] 1 warning

BUILD FAILED
/projects/goldv2/appinventor-sources/appinventor/build.xml:35: The following error occurred while executing this line:
/projects/goldv2/appinventor-sources/appinventor/build-common.xml:372: The following error occurred while executing this line:
/projects/goldv2/appinventor-sources/appinventor/components/build.xml:141: The following error occurred while executing this line:
/projects/goldv2/appinventor-sources/appinventor/build-common.xml:118: Compile failed; see the compiler error output for details.

Total time: 2 seconds

Kumaraswamy · March 7, 2021, 5:05pm

You are missing the closing-curly bracket(s) } somewhere.

Meulencv · March 7, 2021, 5:08pm

Ok, thanks, I will fix it....

Meulencv · March 7, 2021, 5:40pm

And now It gets this error:

Buildfile: /projects/goldv2/appinventor-sources/appinventor/build.xml

extensions:

clean:

init:

common_CommonUtils:

init:

CommonUtils:

common_CommonVersion:

init:

CommonVersion:
[exec] Result: 128
[exec] Result: 128

CopyToRunLibDir:

components_AndroidRuntime:

init:

CommonConstants:
[javac] Compiling 6 source files to /projects/goldv2/appinventor-sources/appinventor/components/build/classes/CommonConstants
[javac] warning: [options] bootstrap class path not set in conjunction with -source 1.7
[javac] 1 warning

HtmlEntities:
[javac] Compiling 1 source file to /projects/goldv2/appinventor-sources/appinventor/components/build/classes/HtmlEntities
[javac] warning: [options] bootstrap class path not set in conjunction with -source 1.7
[javac] 1 warning

common_CommonVersion:

init:

CommonVersion:
[exec] Result: 128
[exec] Result: 128

AndroidRuntime:
[mkdir] Created dir: /projects/goldv2/appinventor-sources/appinventor/components/build/classes/AndroidRuntime
[javac] Compiling 332 source files to /projects/goldv2/appinventor-sources/appinventor/components/build/classes/AndroidRuntime
[javac] warning: [options] bootstrap class path not set in conjunction with -source 1.7
[javac] /projects/goldv2/appinventor-sources/appinventor/components/src/com/appybuilder/meulencv/Get_source_web/Get_source_web.java:50: error: cannot find symbol
[javac] new URL(website).openStream())); // Open stream and read the content from streamReader to BufferedReader
[javac] ^
[javac] symbol: class URL
[javac] /projects/goldv2/appinventor-sources/appinventor/components/src/com/appybuilder/meulencv/Get_source_web/Get_source_web.java:60: error: cannot find symbol
[javac] myEvent(finalData); // You're event with data
[javac] ^
[javac] symbol: method myEvent(String)
[javac] /projects/goldv2/appinventor-sources/appinventor/components/src/com/appybuilder/meulencv/Get_source_web/Get_source_web.java:57: error: cannot find symbol
[javac] activity.runOnUiThread(new Runnable() { // Always raise events on UI thread
[javac] ^
[javac] symbol: variable activity
[javac] /projects/goldv2/appinventor-sources/appinventor/components/src/com/appybuilder/meulencv/Get_source_web/Get_source_web.java:44: error: cannot find symbol
[javac] AsynchUtil.runAsynchronously(new Runnable() { // Always do get request Asynchronously
[javac] ^
[javac] symbol: variable AsynchUtil
[javac] location: class Get_source_web
[javac] Note: Some input files use or override a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] Note: Some input files use unchecked or unsafe operations.
[javac] Note: Recompile with -Xlint:unchecked for details.
[javac] 4 errors
[javac] 1 warning

BUILD FAILED
/projects/goldv2/appinventor-sources/appinventor/build.xml:35: The following error occurred while executing this line:
/projects/goldv2/appinventor-sources/appinventor/build-common.xml:372: The following error occurred while executing this line:
/projects/goldv2/appinventor-sources/appinventor/components/build.xml:141: The following error occurred while executing this line:
/projects/goldv2/appinventor-sources/appinventor/build-common.xml:118: Compile failed; see the compiler error output for details.

Total time: 7 seconds
ncv/Get_source_web/Get_source_web.java:57: error: cannot find symbol
[javac] activity.runOnUiThread(new Runnable() { // Always raise events on UI thread
[javac] ^
[javac] symbol: variable activity
[javac] /projects/goldv2/appinventor-sources/appinventor/components/src/com/appybuilder/meulencv/Get_source_web/Get_source_web.java:44: error: cannot find symbol
[javac] AsynchUtil.runAsynchronously(new Runnable() { // Always do get request Asynchronously
[javac] ^
[javac] symbol: variable AsynchUtil
[javac] location: class Get_source_web
[javac] Note: Some input files use or override a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] Note: Some input files use unchecked or unsafe operations.
[javac] Note: Recompile with -Xlint:unchecked for details.
[javac] 4 errors
[javac] 1 warning

BUILD FAILED
/projects/goldv2/appinventor-sources/appinventor/build.xml:35: The following error occurred while executing this line:
/projects/goldv2/appinventor-sources/appinventor/build-common.xml:372: The following error occurred while executing this line:
/projects/goldv2/appinventor-sources/appinventor/components/build.xml:141: The following error occurred while executing this line:
/projects/goldv2/appinventor-sources/appinventor/build-common.xml:118: Compile failed; see the compiler error output for details.

Total time: 7 seconds

Kumaraswamy · March 7, 2021, 5:43pm

You need to import these:

And activity is the context.

Meulencv · March 7, 2021, 6:02pm

I have changed what you have told me and it is like this:

import com.google.appinventor.components.runtime.Web;

import java.io.BufferedReader
import java.io.IOException
android.app.Activity
import android.app.Activity
import java.io.InputStreamReader
import java.net.HttpURLConnection
import java.net.URL
import com.google.appinventor.components.runtime.util.AsynchUtil

import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;

import android.os.Build;
import android.os.Bundle;
import android.support.annotation.RequiresApi;
import android.support.v7.app.AppCompatActivity;
import android.view.View;
import android.widget.Button;
import android.widget.TextView;


import org.json.JSONArray;
import org.json.JSONException;
import org.json.JSONObject;

import android.app.Activity;
import android.content.Context;
import android.content.DialogInterface;
import android.preference.DialogPreference;
import android.util.AttributeSet;
import android.util.Log;
import android.view.LayoutInflater;
import android.view.View;
import android.widget.AdapterView;
import android.widget.ListView;

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.util.zip.GZIPInputStream;

import org.apache.http.Header;
import org.apache.http.HttpResponse;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.DefaultHttpClient;


import android.content.Context;
import android.util.Log;
import com.google.appinventor.components.annotations.*;
import com.google.appinventor.components.runtime.*;
import com.google.appinventor.components.common.ComponentCategory;

@DesignerComponent(version = 1,  description = "This Extension was created with the AppyBuilder Code Editor.<br>" + 
                   "Create your own here:<br><a href='https://editor.appybuilder.com' target='_blank'>https://editor.appybuilder.com</a><br>",
        category = ComponentCategory.EXTENSION,
        nonVisible = true,   iconName = "http://appyBuilder.com/extensions/icons/extension.png")
@SimpleObject(external = true)
public class Get_source_web extends AndroidNonvisibleComponent {
    private ComponentContainer container;
  
    public Get_source_web(ComponentContainer container) {
        super(container.$form());
        this.container = container;
    }
  
 @SimpleFunction(
            description = "Website source-code")
    public void WebsiteContent(final String website) {
        AsynchUtil.runAsynchronously(new Runnable() { // Always do get request Asynchronously
            @Override
            public void run() {
                try {
                    BufferedReader readStream = new BufferedReader(
                            new InputStreamReader(
                                    new URL(website).openStream())); // Open stream and read the content from streamReader to BufferedReader
                    String readLine; // Variable which will have one line of data
                    StringBuilder data = new StringBuilder(); // The result data will be stored here
                    while ((readLine = readStream.readLine()) != null) data.append(readLine); // Read all the lines from the readStream
                    readStream.close(); // IMPORTANT close the stream

                    final String finalData = data.toString(); // Make the string final with the data variable
                    activity.runOnUiThread(new Runnable() { // Always raise events on UI thread
                        @Override
                        public void run() {
                            myEvent(finalData); // You're event with data
                        }
                    });
                } catch (IOException e) {
                    e.printStackTrace(); // Error occured
                }
            }
        });
  
  
}}

It keep getting an error:

Buildfile: /projects/goldv2/appinventor-sources/appinventor/build.xml

extensions:

clean:

init:

common_CommonUtils:

init:

CommonUtils:

common_CommonVersion:

init:

CommonVersion:
[exec] Result: 128
[exec] Result: 128

CopyToRunLibDir:

components_AndroidRuntime:

init:
[mkdir] Created dir: /projects/goldv2/appinventor-sources/appinventor/build/components
[mkdir] Created dir: /projects/goldv2/appinventor-sources/appinventor/components/build
[mkdir] Created dir: /projects/goldv2/appinventor-sour
CopyToRunLibDir:

components_AndroidRuntime:

init:

CommonConstants:
[javac] Compiling 6 source files to /projects/goldv2/appinventor-sources/appinventor/components/build/classes/CommonConstants
[javac] warning: [options] bootstrap class path not set in conjunction with -source 1.7
[javac] 1 warning
[jar] Building jar: /projects/goldv2/appinventor-sources/appinventor/build/components/CommonConstants.jar
[jar] error while reading original manifest in file: /projects/goldv2/appinventor-sources/appinventor/build/components/CommonConstants-gwt.jar due to error in opening zip file
[jar] Building jar: /projects/goldv2/appinventor-sources/appinventor/build/components/CommonConstants-gwt.jar

HtmlEntities:
[javac] Compiling 1 source file to /projects/goldv2/appinventor-sources/appinventor/components/build/classes/HtmlEntities
[javac] warning: [options] bootstrap class path not set in conjunction with -source 1.7
[javac] 1 warning

common_CommonVersion:

init:

CommonVersion:
[exec] Result: 128
[exec] Result: 128

AndroidRuntime:
[javac] Compiling 332 source files to /projects/goldv2/appinventor-sources/appinventor/components/build/classes/AndroidRuntime
[javac] warning: [options] bootstrap class path not set in conjunction with -source 1.7
[javac] /projects/goldv2/appinventor-sources/appinventor/components/src/com/appybuilder/meulencv/Get_source_web/Get_source_web.java:7: error: ';' expected
[javac] import java.io.BufferedReader
[javac] ^
[javac] /projects/goldv2/appinventor-sources/appinventor/components/src/com/appybuilder/meulencv/Get_source_web/Get_source_web.java:8: error: ';' expected
[javac] import java.io.IOException
[javac] ^
[javac] /projects/goldv2/appinventor-sources/appinventor/components/src/com/appybuilder/meulencv/Get_source_web/Get_source_web.java:10: error: ';' expected
[javac] import android.app.Activity
[javac] ^
[javac] /projects/goldv2/appinventor-sources/appinventor/components/src/com/appybuilder/meulencv/Get_source_web/Get_source_web.java:11: error: ';' expected
[javac] import java.io.InputStreamReader
[javac] ^
[javac] /projects/goldv2/appinventor-sources/appinventor/components/src/com/appybuilder/meulencv/Get_source_web/Get_source_web.java:12: error: ';' expected
[javac] import java.net.HttpURLConnection
[javac] ^
[javac] /projects/goldv2/appinventor-sources/appinventor/components/src/com/appybuilder/meulencv/Get_source_web/Get_source_web.java:13: error: ';' expected
[javac] import java.net.URL
[javac] ^
[javac] /projects/goldv2/appinventor-sources/appinventor/components/src/com/appybuilder/meulencv/Get_source_web/Get_source_web.java:14: error: ';' expected
[javac] import com.google.appinventor.components.runtime.util.AsynchUtil
[javac] ^
[javac] 7 errors
[javac] 1 warning

BUILD FAILED
/projects/goldv2/appinventor-sources/appinventor/build.xml:35: The following error occurred while executing this line:
/projects/goldv2/appinventor-sources/appinventor/build-common.xml:372: The following error occurred while executing this line:
/projects/goldv2/appinventor-sources/appinventor/components/build.xml:141: The following error occurred while executing this line:
/projects/goldv2/appinventor-sources/appinventor/build-common.xml:118: Compile failed; see the compiler error output for details.

Total time: 3 seconds
m.google.appinventor.components.runtime.util.AsynchUtil
[javac] ^
[javac] 7 errors
[javac] 1 warning

BUILD FAILED
/projects/goldv2/appinventor-sources/appinventor/build.xml:35: The following error occurred while executing this line:
/projects/goldv2/appinventor-sources/appinventor/build-common.xml:372: The following error occurred while executing this line:
/projects/goldv2/appinventor-sources/appinventor/components/build.xml:141: The following error occurred while executing this line:
/projects/goldv2/appinventor-sources/appinventor/build-common.xml:118: Compile failed; see the compiler error output for details.

Total time: 2 seconds

Kumaraswamy · March 7, 2021, 6:12pm

It like you're missing the basics of programming. I recommend you to use a java code editor. Search for the errors in google and in the community once next time.

All the lines or end of the instructions in Java is with ;

For example:

import com.test.add

What you're doing , which correctly, should be:

import com.test.add;

Meulencv · March 7, 2021, 6:14pm

oh, that's right, it slipped my mind , thank you very much.

Meulencv · March 7, 2021, 6:25pm

Hello, I know I'm a bit heavy, but I can't understand the errors. I have fixed the ";" and I get the following error:

Buildfile: /projects/goldv2/appinventor-sources/appinventor/build.xml

extensions:

clean:

init:

common_CommonUtils:

init:

CommonUtils:

common_CommonVersion:

init:

CommonVersion:
[exec] Result: 128
[exec] Result: 128

CopyToRunLibDir:

components_AndroidRuntime:

init:

CommonConstants:
[javac] Compiling 6 source files to /projects/goldv2/appinventor-sources/appinventor/components/build/classes/CommonConstants
[javac] warning: [options] bootstrap class path not set in conjunction with -source 1.7
[javac] 1 warning

HtmlEntities:
[javac] Compiling 1 source file to /projects/goldv2/appinventor-sources/appinventor/components/build/classes/HtmlEntities
[javac] warning: [options] bootstrap class path not set in conjunction with -source 1.7
[javac] 1 warning

common_CommonVersion:

init:

CommonVersion:
[exec] Result: 128
[exec] Result: 128

AndroidRuntime:
[javac] Compiling 332 source files to /projects/goldv2/appinventor-sources/appinventor/components/build/classes/AndroidRuntime
[javac] warning: [options] bootstrap class path not set in conjunction with -source 1.7
[javac] /projects/goldv2/appinventor-sources/appinventor/components/src/com/appybuilder/meulencv/Get_source_web/Get_source_web.java:93: error: cannot find symbol
[javac] myEvent(finalData); // You're event with data
[javac] ^
[javac] symbol: method myEvent(String)
[javac] /projects/goldv2/appinventor-sources/appinventor/components/src/com/appybuilder/meulencv/Get_source_web/Get_source_web.java:90: error: cannot find symbol
[javac] activity.runOnUiThread(new Runnable() { // Always raise events on UI thread
[javac] ^
[javac] symbol: variable activity
[javac] Note: Some input files use or override a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] Note: Some input files use unchecked or unsafe operations.
[javac] Note: Recompile with -Xlint:unchecked for details.
[javac] 2 errors
[javac] 1 warning

BUILD FAILED
/projects/goldv2/appinventor-sources/appinventor/build.xml:35: The following error occurred while executing this line:
/projects/goldv2/appinventor-sources/appinventor/build-common.xml:372: The following error occurred while executing this line:
/projects/goldv2/appinventor-sources/appinventor/components/build.xml:141: The following error occurred while executing this line:
/projects/goldv2/appinventor-sources/appinventor/build-common.xml:118: Compile failed; see the compiler error output for details.

Total time: 7 seconds
cts/goldv2/appinventor-sources/appinventor/components/build/classes/AndroidRuntime
[javac] warning: [options] bootstrap class path not set in conjunction with -source 1.7
[javac] /projects/goldv2/appinventor-sources/appinventor/components/src/com/appybuilder/meulencv/Get_source_web/Get_source_web.java:93: error: cannot find symbol
[javac] myEvent(finalData); // You're event with data
[javac] ^
[javac] symbol: method myEvent(String)
[javac] /projects/goldv2/appinventor-sources/appinventor/components/src/com/appybuilder/meulencv/Get_source_web/Get_source_web.java:90: error: cannot find symbol
[javac] activity.runOnUiThread(new Runnable() { // Always raise events on UI thread
[javac] ^
[javac] symbol: variable activity
[javac] Note: Some input files use or override a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] Note: Some input files use unchecked or unsafe operations.
[javac] Note: Recompile with -Xlint:unchecked for details.
[javac] 2 errors
[javac] 1 warning

BUILD FAILED
/projects/goldv2/appinventor-sources/appinventor/build.xml:35: The following error occurred while executing this line:
/projects/goldv2/appinventor-sources/appinventor/build-common.xml:372: The following error occurred while executing this line:
/projects/goldv2/appinventor-sources/appinventor/components/build.xml:141: The following error occurred while executing this line:
/projects/goldv2/appinventor-sources/appinventor/build-common.xml:118: Compile failed; see the compiler error output for details.

Total time: 6 seconds

Meulencv · March 7, 2021, 6:36pm

I don't find the resolution of the errors on the internet

vknow360 · March 8, 2021, 1:09am

Your code have two errors:

you haven't declared and initialized activity
there is no myEvent method in your code

You should use a Java IDE.

Mayank_Kumar · March 8, 2021, 3:02am

This can be done with AI2 Blocks, just join view-source: before the http part?

Salman_Dev · March 8, 2021, 11:52am

this can do the same thing too, so I think we don't need an extension for that @Meulencv

blocks (3)