Capture webpage with web

Greeting. I am new to AI2 and would like to create a simple app to post-process the content of a webpage to extract some info. However when using “call web1.Get” to access the webpage, what the ‘responseContent’ receives is the page source of the webpage, rather than the content that’s presented through webviewer or any web browsers out there. I’d appreciate if anyone can help point me to the right direction. Thanks in advance.

That’s called web page shredding.
Here’s a sample:


P.S. Web Masters hate this.

I will look into that. Thanks a lot for the pointer.

Hi,
I'm looking at trying to do something like this also, but i'm not sure the shredding example answers the question or provides insight into how to scrape text from a page that uses Javascript to populate its content.

There is a web page I don't own or control. Its html code is that returns on a quite sparse and does not actually contain the test I need. i.e. a Web1.GotText "responseContent" does not contain the text I need.

It seems the content is populated using Javascripts, so when I inspect the element when I load the page in a Chrome browser window for example, the text I need is there.

Is there a way to retrieve this text using a similar webscraping process?

You can get webview text using JavaScript.

Try this on your web page in a webviewer:

The web page has to be rendered for this to work, so don't run the JS until the page has finished loading (there is a block for that :wink: )