How do I make an extension that retrieves the source code of a web page?

Could you tell me where I can find it,
thank you

If you have to retrive entire structure of web Page ,web component is useless.
You have to embed your Page in a function

1 Like

But I want to get the source code of the web to then do other things (like web scrapping), what I need is to get the code that I have to put in java to make the extension have the ability to get the source code of the web.

You can directly instantiate the Web component and call the Get function inside your extension code like this

  1. Declare a field
private Web myWebComponent;
  1. Instantiate the Web component inside the constructor
myWebComponent = new Web(...);
  1. Register a GotText callback inside your extension class
@Override void GotText(...) {
    // Receive the text from here
}
  1. Use the Get function from Web component like this
myWebComponent.Get(...);
5 Likes

thanks @pavi2410

1 Like

Did you search the community?

1 Like

thank you very much for the information

1 Like

Hello, Am I doing it right?

import com.google.appinventor.components.runtime.Web;

import android.content.Context;
import android.util.Log;
import com.google.appinventor.components.annotations.*;
import com.google.appinventor.components.runtime.*;
import com.google.appinventor.components.common.ComponentCategory;

@DesignerComponent(version = 1,  description = "This Extension was created with the AppyBuilder Code Editor.<br>" + 
                   "Create your own here:<br><a href='https://editor.appybuilder.com' target='_blank'>https://editor.appybuilder.com</a><br>",
        category = ComponentCategory.EXTENSION,
        nonVisible = true,   iconName = "http://appyBuilder.com/extensions/icons/extension.png")
@SimpleObject(external = true)
public class Get_source_web extends AndroidNonvisibleComponent {
    private ComponentContainer container;
  

  
    public Get_source_web(ComponentContainer container) {
        super(container.$form());
        this.container = container;
    }
  
 @SimpleFunction(
            description = "Website source-code")
    public void WebsiteContent(final String website) {
        AsynchUtil.runAsynchronously(new Runnable() { // Always do get request Asynchronously
            @Override
            public void run() {
                try {
                    BufferedReader readStream = new BufferedReader(
                            new InputStreamReader(
                                    new URL(website).openStream())); // Open stream and read the content from streamReader to BufferedReader
                    String readLine; // Variable which will have one line of data
                    StringBuilder data = new StringBuilder(); // The result data will be stored here
                    while ((readLine = readStream.readLine()) != null) data.append(readLine); // Read all the lines from the readStream
                    readStream.close(); // IMPORTANT close the stream

                    final String finalData = data.toString(); // Make the string final with the data variable
                    activity.runOnUiThread(new Runnable() { // Always raise events on UI thread
                        @Override
                        public void run() {
                            myEvent(finalData); // You're event with data
                        }
                    });
                } catch (IOException e) {
                    e.printStackTrace(); // Error occured
                }
            }
        });
  
  
}}

I get the following error:
Buildfile: /projects/goldv2/appinventor-sources/appinventor/build.xml

extensions:

clean:

init:

common_CommonUtils:

init:

CommonUtils:

common_CommonVersion:

init:

CommonVersion:
[exec] Result: 128
[exec] Result: 128

CopyToRunLibDir:

components_AndroidRuntime:

init:

CommonConstants:
[javac] Compiling 6 source files to /projects/goldv2/appinventor-sources/appinventor/components/build/classes/CommonConstants
[javac] warning: [options] bootstrap class path not set in conjunction with -source 1.7
[javac] 1 warning

HtmlEntities:
[javac] Compiling 1 source file to /projects/goldv2/appinventor-sources/appinventor/components/build/classes/HtmlEntities
[javac] warning: [options] bootstrap class path not set in conjunction with -source 1.7
[javac] 1 warning

common_CommonVersion:

init:

CommonVersion:
[exec] Result: 128
[exec] Result: 128

AndroidRuntime:
[javac] Compiling 332 source files to /projects/goldv2/appinventor-sources/appinventor/components/build/classes/AndroidRuntime
[javac] warning: [options] bootstrap class path not set in conjunction with -source 1.7
[javac] /projects/goldv2/appinventor-sources/appinventor/components/src/com/appybuilder/meulencv/Get_source_web/Get_source_web.java:57: error: reached end of file while parsing
[javac] }
[javac] ^
[javac] 1 error
[javac] 1 warning

BUILD FAILED
/projects/goldv2/appinventor-sources/appinventor/build.xml:35: The following error occurred while executing this line:
/projects/goldv2/appinventor-sources/appinventor/build-common.xml:372: The following error occurred while executing this line:
/projects/goldv2/appinventor-sources/appinventor/components/build.xml:141: The following error occurred while executing this line:
/projects/goldv2/appinventor-sources/appinventor/build-common.xml:118: Compile failed; see the compiler error output for details.

Total time: 2 seconds
mlEntities:
[mkdir] Created dir: /projects/goldv2/appinventor-sources/appinventor/components/build/classes/HtmlEntities
[javac] Compiling 1 source file to /projects/goldv2/appinventor-sources/appinventor/components/build/classes/HtmlEntities
[javac] warning: [options] bootstrap class path not set in conjunction with -source 1.7
[javac] 1 warning
[jar] Building jar: /projects/goldv2/appinventor-sources/appinventor/components/build/HtmlEntities.jar

common_CommonVersion:

init:

CommonVersion:
[exec] Result: 128
[exec] Result: 128

AndroidRuntime:
[mkdir] Created dir: /projects/goldv2/appinventor-sources/appinventor/components/build/classes/AndroidRuntime
[javac] Compiling 332 source files to /projects/goldv2/appinventor-sources/appinventor/components/build/classes/AndroidRuntime
[javac] warning: [options] bootstrap class path not set in conjunction with -source 1.7
[javac] /projects/goldv2/appinventor-sources/appinventor/components/src/com/appybuilder/meulencv/Get_source_web/Get_source_web.java:57: error: reached end of file while parsing
[javac] }
[javac] ^
[javac] 1 error
[javac] 1 warning

BUILD FAILED
/projects/goldv2/appinventor-sources/appinventor/build.xml:35: The following error occurred while executing this line:
/projects/goldv2/appinventor-sources/appinventor/build-common.xml:372: The following error occurred while executing this line:
/projects/goldv2/appinventor-sources/appinventor/components/build.xml:141: The following error occurred while executing this line:
/projects/goldv2/appinventor-sources/appinventor/build-common.xml:118: Compile failed; see the compiler error output for details.

Total time: 2 seconds

You are missing the closing-curly bracket(s) } somewhere.

1 Like

Ok, thanks, I will fix it....

And now It gets this error:

Buildfile: /projects/goldv2/appinventor-sources/appinventor/build.xml

extensions:

clean:

init:

common_CommonUtils:

init:

CommonUtils:

common_CommonVersion:

init:

CommonVersion:
[exec] Result: 128
[exec] Result: 128

CopyToRunLibDir:

components_AndroidRuntime:

init:

CommonConstants:
[javac] Compiling 6 source files to /projects/goldv2/appinventor-sources/appinventor/components/build/classes/CommonConstants
[javac] warning: [options] bootstrap class path not set in conjunction with -source 1.7
[javac] 1 warning

HtmlEntities:
[javac] Compiling 1 source file to /projects/goldv2/appinventor-sources/appinventor/components/build/classes/HtmlEntities
[javac] warning: [options] bootstrap class path not set in conjunction with -source 1.7
[javac] 1 warning

common_CommonVersion:

init:

CommonVersion:
[exec] Result: 128
[exec] Result: 128

AndroidRuntime:
[mkdir] Created dir: /projects/goldv2/appinventor-sources/appinventor/components/build/classes/AndroidRuntime
[javac] Compiling 332 source files to /projects/goldv2/appinventor-sources/appinventor/components/build/classes/AndroidRuntime
[javac] warning: [options] bootstrap class path not set in conjunction with -source 1.7
[javac] /projects/goldv2/appinventor-sources/appinventor/components/src/com/appybuilder/meulencv/Get_source_web/Get_source_web.java:50: error: cannot find symbol
[javac] new URL(website).openStream())); // Open stream and read the content from streamReader to BufferedReader
[javac] ^
[javac] symbol: class URL
[javac] /projects/goldv2/appinventor-sources/appinventor/components/src/com/appybuilder/meulencv/Get_source_web/Get_source_web.java:60: error: cannot find symbol
[javac] myEvent(finalData); // You're event with data
[javac] ^
[javac] symbol: method myEvent(String)
[javac] /projects/goldv2/appinventor-sources/appinventor/components/src/com/appybuilder/meulencv/Get_source_web/Get_source_web.java:57: error: cannot find symbol
[javac] activity.runOnUiThread(new Runnable() { // Always raise events on UI thread
[javac] ^
[javac] symbol: variable activity
[javac] /projects/goldv2/appinventor-sources/appinventor/components/src/com/appybuilder/meulencv/Get_source_web/Get_source_web.java:44: error: cannot find symbol
[javac] AsynchUtil.runAsynchronously(new Runnable() { // Always do get request Asynchronously
[javac] ^
[javac] symbol: variable AsynchUtil
[javac] location: class Get_source_web
[javac] Note: Some input files use or override a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] Note: Some input files use unchecked or unsafe operations.
[javac] Note: Recompile with -Xlint:unchecked for details.
[javac] 4 errors
[javac] 1 warning

BUILD FAILED
/projects/goldv2/appinventor-sources/appinventor/build.xml:35: The following error occurred while executing this line:
/projects/goldv2/appinventor-sources/appinventor/build-common.xml:372: The following error occurred while executing this line:
/projects/goldv2/appinventor-sources/appinventor/components/build.xml:141: The following error occurred while executing this line:
/projects/goldv2/appinventor-sources/appinventor/build-common.xml:118: Compile failed; see the compiler error output for details.

Total time: 7 seconds
ncv/Get_source_web/Get_source_web.java:57: error: cannot find symbol
[javac] activity.runOnUiThread(new Runnable() { // Always raise events on UI thread
[javac] ^
[javac] symbol: variable activity
[javac] /projects/goldv2/appinventor-sources/appinventor/components/src/com/appybuilder/meulencv/Get_source_web/Get_source_web.java:44: error: cannot find symbol
[javac] AsynchUtil.runAsynchronously(new Runnable() { // Always do get request Asynchronously
[javac] ^
[javac] symbol: variable AsynchUtil
[javac] location: class Get_source_web
[javac] Note: Some input files use or override a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] Note: Some input files use unchecked or unsafe operations.
[javac] Note: Recompile with -Xlint:unchecked for details.
[javac] 4 errors
[javac] 1 warning

BUILD FAILED
/projects/goldv2/appinventor-sources/appinventor/build.xml:35: The following error occurred while executing this line:
/projects/goldv2/appinventor-sources/appinventor/build-common.xml:372: The following error occurred while executing this line:
/projects/goldv2/appinventor-sources/appinventor/components/build.xml:141: The following error occurred while executing this line:
/projects/goldv2/appinventor-sources/appinventor/build-common.xml:118: Compile failed; see the compiler error output for details.

Total time: 7 seconds

You need to import these:

And activity is the context.

2 Likes

I have changed what you have told me and it is like this:

import com.google.appinventor.components.runtime.Web;

import java.io.BufferedReader
import java.io.IOException
android.app.Activity
import android.app.Activity
import java.io.InputStreamReader
import java.net.HttpURLConnection
import java.net.URL
import com.google.appinventor.components.runtime.util.AsynchUtil

import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;

import android.os.Build;
import android.os.Bundle;
import android.support.annotation.RequiresApi;
import android.support.v7.app.AppCompatActivity;
import android.view.View;
import android.widget.Button;
import android.widget.TextView;


import org.json.JSONArray;
import org.json.JSONException;
import org.json.JSONObject;

import android.app.Activity;
import android.content.Context;
import android.content.DialogInterface;
import android.preference.DialogPreference;
import android.util.AttributeSet;
import android.util.Log;
import android.view.LayoutInflater;
import android.view.View;
import android.widget.AdapterView;
import android.widget.ListView;

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.util.zip.GZIPInputStream;

import org.apache.http.Header;
import org.apache.http.HttpResponse;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.DefaultHttpClient;


import android.content.Context;
import android.util.Log;
import com.google.appinventor.components.annotations.*;
import com.google.appinventor.components.runtime.*;
import com.google.appinventor.components.common.ComponentCategory;

@DesignerComponent(version = 1,  description = "This Extension was created with the AppyBuilder Code Editor.<br>" + 
                   "Create your own here:<br><a href='https://editor.appybuilder.com' target='_blank'>https://editor.appybuilder.com</a><br>",
        category = ComponentCategory.EXTENSION,
        nonVisible = true,   iconName = "http://appyBuilder.com/extensions/icons/extension.png")
@SimpleObject(external = true)
public class Get_source_web extends AndroidNonvisibleComponent {
    private ComponentContainer container;
  
    public Get_source_web(ComponentContainer container) {
        super(container.$form());
        this.container = container;
    }
  
 @SimpleFunction(
            description = "Website source-code")
    public void WebsiteContent(final String website) {
        AsynchUtil.runAsynchronously(new Runnable() { // Always do get request Asynchronously
            @Override
            public void run() {
                try {
                    BufferedReader readStream = new BufferedReader(
                            new InputStreamReader(
                                    new URL(website).openStream())); // Open stream and read the content from streamReader to BufferedReader
                    String readLine; // Variable which will have one line of data
                    StringBuilder data = new StringBuilder(); // The result data will be stored here
                    while ((readLine = readStream.readLine()) != null) data.append(readLine); // Read all the lines from the readStream
                    readStream.close(); // IMPORTANT close the stream

                    final String finalData = data.toString(); // Make the string final with the data variable
                    activity.runOnUiThread(new Runnable() { // Always raise events on UI thread
                        @Override
                        public void run() {
                            myEvent(finalData); // You're event with data
                        }
                    });
                } catch (IOException e) {
                    e.printStackTrace(); // Error occured
                }
            }
        });
  
  
}}

It keep getting an error:

Buildfile: /projects/goldv2/appinventor-sources/appinventor/build.xml

extensions:

clean:

init:

common_CommonUtils:

init:

CommonUtils:

common_CommonVersion:

init:

CommonVersion:
[exec] Result: 128
[exec] Result: 128

CopyToRunLibDir:

components_AndroidRuntime:

init:
[mkdir] Created dir: /projects/goldv2/appinventor-sources/appinventor/build/components
[mkdir] Created dir: /projects/goldv2/appinventor-sources/appinventor/components/build
[mkdir] Created dir: /projects/goldv2/appinventor-sour
CopyToRunLibDir:

components_AndroidRuntime:

init:

CommonConstants:
[javac] Compiling 6 source files to /projects/goldv2/appinventor-sources/appinventor/components/build/classes/CommonConstants
[javac] warning: [options] bootstrap class path not set in conjunction with -source 1.7
[javac] 1 warning
[jar] Building jar: /projects/goldv2/appinventor-sources/appinventor/build/components/CommonConstants.jar
[jar] error while reading original manifest in file: /projects/goldv2/appinventor-sources/appinventor/build/components/CommonConstants-gwt.jar due to error in opening zip file
[jar] Building jar: /projects/goldv2/appinventor-sources/appinventor/build/components/CommonConstants-gwt.jar

HtmlEntities:
[javac] Compiling 1 source file to /projects/goldv2/appinventor-sources/appinventor/components/build/classes/HtmlEntities
[javac] warning: [options] bootstrap class path not set in conjunction with -source 1.7
[javac] 1 warning

common_CommonVersion:

init:

CommonVersion:
[exec] Result: 128
[exec] Result: 128

AndroidRuntime:
[javac] Compiling 332 source files to /projects/goldv2/appinventor-sources/appinventor/components/build/classes/AndroidRuntime
[javac] warning: [options] bootstrap class path not set in conjunction with -source 1.7
[javac] /projects/goldv2/appinventor-sources/appinventor/components/src/com/appybuilder/meulencv/Get_source_web/Get_source_web.java:7: error: ';' expected
[javac] import java.io.BufferedReader
[javac] ^
[javac] /projects/goldv2/appinventor-sources/appinventor/components/src/com/appybuilder/meulencv/Get_source_web/Get_source_web.java:8: error: ';' expected
[javac] import java.io.IOException
[javac] ^
[javac] /projects/goldv2/appinventor-sources/appinventor/components/src/com/appybuilder/meulencv/Get_source_web/Get_source_web.java:10: error: ';' expected
[javac] import android.app.Activity
[javac] ^
[javac] /projects/goldv2/appinventor-sources/appinventor/components/src/com/appybuilder/meulencv/Get_source_web/Get_source_web.java:11: error: ';' expected
[javac] import java.io.InputStreamReader
[javac] ^
[javac] /projects/goldv2/appinventor-sources/appinventor/components/src/com/appybuilder/meulencv/Get_source_web/Get_source_web.java:12: error: ';' expected
[javac] import java.net.HttpURLConnection
[javac] ^
[javac] /projects/goldv2/appinventor-sources/appinventor/components/src/com/appybuilder/meulencv/Get_source_web/Get_source_web.java:13: error: ';' expected
[javac] import java.net.URL
[javac] ^
[javac] /projects/goldv2/appinventor-sources/appinventor/components/src/com/appybuilder/meulencv/Get_source_web/Get_source_web.java:14: error: ';' expected
[javac] import com.google.appinventor.components.runtime.util.AsynchUtil
[javac] ^
[javac] 7 errors
[javac] 1 warning

BUILD FAILED
/projects/goldv2/appinventor-sources/appinventor/build.xml:35: The following error occurred while executing this line:
/projects/goldv2/appinventor-sources/appinventor/build-common.xml:372: The following error occurred while executing this line:
/projects/goldv2/appinventor-sources/appinventor/components/build.xml:141: The following error occurred while executing this line:
/projects/goldv2/appinventor-sources/appinventor/build-common.xml:118: Compile failed; see the compiler error output for details.

Total time: 3 seconds
m.google.appinventor.components.runtime.util.AsynchUtil
[javac] ^
[javac] 7 errors
[javac] 1 warning

BUILD FAILED
/projects/goldv2/appinventor-sources/appinventor/build.xml:35: The following error occurred while executing this line:
/projects/goldv2/appinventor-sources/appinventor/build-common.xml:372: The following error occurred while executing this line:
/projects/goldv2/appinventor-sources/appinventor/components/build.xml:141: The following error occurred while executing this line:
/projects/goldv2/appinventor-sources/appinventor/build-common.xml:118: Compile failed; see the compiler error output for details.

Total time: 2 seconds

It like you're missing the basics of programming. I recommend you to use a java code editor. Search for the errors in google and in the community once next time.

All the lines or end of the instructions in Java is with ;

For example:

import com.test.add 

What you're doing :point_up:, which correctly, should be:

import com.test.add;
2 Likes

oh, that's right, it slipped my mind :no_mouth:, thank you very much.

1 Like

Hello, I know I'm a bit heavy, but I can't understand the errors. I have fixed the ";" and I get the following error:

Buildfile: /projects/goldv2/appinventor-sources/appinventor/build.xml

extensions:

clean:

init:

common_CommonUtils:

init:

CommonUtils:

common_CommonVersion:

init:

CommonVersion:
[exec] Result: 128
[exec] Result: 128

CopyToRunLibDir:

components_AndroidRuntime:

init:

CommonConstants:
[javac] Compiling 6 source files to /projects/goldv2/appinventor-sources/appinventor/components/build/classes/CommonConstants
[javac] warning: [options] bootstrap class path not set in conjunction with -source 1.7
[javac] 1 warning

HtmlEntities:
[javac] Compiling 1 source file to /projects/goldv2/appinventor-sources/appinventor/components/build/classes/HtmlEntities
[javac] warning: [options] bootstrap class path not set in conjunction with -source 1.7
[javac] 1 warning

common_CommonVersion:

init:

CommonVersion:
[exec] Result: 128
[exec] Result: 128

AndroidRuntime:
[javac] Compiling 332 source files to /projects/goldv2/appinventor-sources/appinventor/components/build/classes/AndroidRuntime
[javac] warning: [options] bootstrap class path not set in conjunction with -source 1.7
[javac] /projects/goldv2/appinventor-sources/appinventor/components/src/com/appybuilder/meulencv/Get_source_web/Get_source_web.java:93: error: cannot find symbol
[javac] myEvent(finalData); // You're event with data
[javac] ^
[javac] symbol: method myEvent(String)
[javac] /projects/goldv2/appinventor-sources/appinventor/components/src/com/appybuilder/meulencv/Get_source_web/Get_source_web.java:90: error: cannot find symbol
[javac] activity.runOnUiThread(new Runnable() { // Always raise events on UI thread
[javac] ^
[javac] symbol: variable activity
[javac] Note: Some input files use or override a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] Note: Some input files use unchecked or unsafe operations.
[javac] Note: Recompile with -Xlint:unchecked for details.
[javac] 2 errors
[javac] 1 warning

BUILD FAILED
/projects/goldv2/appinventor-sources/appinventor/build.xml:35: The following error occurred while executing this line:
/projects/goldv2/appinventor-sources/appinventor/build-common.xml:372: The following error occurred while executing this line:
/projects/goldv2/appinventor-sources/appinventor/components/build.xml:141: The following error occurred while executing this line:
/projects/goldv2/appinventor-sources/appinventor/build-common.xml:118: Compile failed; see the compiler error output for details.

Total time: 7 seconds
cts/goldv2/appinventor-sources/appinventor/components/build/classes/AndroidRuntime
[javac] warning: [options] bootstrap class path not set in conjunction with -source 1.7
[javac] /projects/goldv2/appinventor-sources/appinventor/components/src/com/appybuilder/meulencv/Get_source_web/Get_source_web.java:93: error: cannot find symbol
[javac] myEvent(finalData); // You're event with data
[javac] ^
[javac] symbol: method myEvent(String)
[javac] /projects/goldv2/appinventor-sources/appinventor/components/src/com/appybuilder/meulencv/Get_source_web/Get_source_web.java:90: error: cannot find symbol
[javac] activity.runOnUiThread(new Runnable() { // Always raise events on UI thread
[javac] ^
[javac] symbol: variable activity
[javac] Note: Some input files use or override a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] Note: Some input files use unchecked or unsafe operations.
[javac] Note: Recompile with -Xlint:unchecked for details.
[javac] 2 errors
[javac] 1 warning

BUILD FAILED
/projects/goldv2/appinventor-sources/appinventor/build.xml:35: The following error occurred while executing this line:
/projects/goldv2/appinventor-sources/appinventor/build-common.xml:372: The following error occurred while executing this line:
/projects/goldv2/appinventor-sources/appinventor/components/build.xml:141: The following error occurred while executing this line:
/projects/goldv2/appinventor-sources/appinventor/build-common.xml:118: Compile failed; see the compiler error output for details.

Total time: 6 seconds

I don't find the resolution of the errors on the internet

Your code have two errors:

  1. you haven't declared and initialized activity
  2. there is no myEvent method in your code

You should use a Java IDE.

3 Likes

This can be done with AI2 Blocks, just join view-source: before the http part?

3 Likes

this can do the same thing too, so I think we don't need an extension for that @Meulencv

blocks (3)

3 Likes