Hi, guys.
I have problem in parsing html.
I want to get Source of page, and I know that I have to use ‘Web’ Component.
but If I use ‘Web’ Component, I would get wrong html code and It says
‘Please Check Your Browser is newest version’
I think solving this problem is pretty hard.
so I thought extracting Source(html) from ‘WebViewer’ is much better but I don’t know how to do.
Plz help me…OTL
Some web pages don’t want to be scraped.
Give us an example URL?
This should return the source for most web pages using the web component:
You can use view-source:<Your URL here> in the webviewer to see the source code
When this site is loaded in web component, it reponds strange html.
(This site is korean and you should change your user-agent to android)
My blocks above return this:
Do It Result: "<!doctype html><html lang="ko"><head><meta charset="utf-8"/><link rel="shortcut icon" href="/favicon.ico"/><meta name="viewport" content="width=device-width,initial-scale=1,minimum-scale=1,maximum-scale=1,user-scalable=no"/><meta http-equiv="X-UA-Compatible" content="ie=edge"/><meta http-equiv="Cache-Control" content="no-cache, no-store, must-revalidate"/><meta http-equiv="Pragma" content="no-cache"/><meta http-equiv="Expires" content="-1"/><meta property="og:type" content="website"/><meta property="og:site_name" content="조아라: 웹소설의 시작 "/><meta property="og:url" content="https://m.joara.com"/><meta property="og:title" content="조아라: 웹소설의 시작 "/><meta property="og:description" content="국내최대 웹소설 플랫폼, 당신의 스토리 본능을 깨우다"/><meta property="og:image" content="https://m.joara.com/assets/img/open_graph/og_800x400_1.jpg"/><meta property="og:image:width" content="800"/><meta property="og:image:height" content="400"/><meta property="og:image" content="https://m.joara.com/assets/img/open_graph/og_1200x630_1.jpg"/><meta property="og:image:width" content="1200"/><meta property="og:image:height" content="630"/><meta name="twitter:card" content="summary_large_image"/><meta name="twitter:site" content="https://m.joara.com"/><meta name="twitter:image" content="https://m.joara.com/assets/img/open_graph/og_twitter_1024x512_1.jpg"/><link rel="manifest" href="/manifest.json"/><title>조아라: 웹소설의 시작</title><link href="/static/css/2.78f1598c.chunk.css" rel="stylesheet"><link href="/static/css/main.65f8857c.chunk.css" rel="stylesheet"></head><body oncontextmenu="return!1" ondragstart="return!1" onselectstart="return!1"><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"><style>#background-loading{display:none}#background-loading.background-view{display:block}</style><div id="background-loading" style="text-align:left">조아라 사이트로 연결 중입니다.<br/>이 페이지에서 정상적으로 로딩이 완료되지 않고 멈추면,<br/>os 버전이 최신 버전인지 확인해주시고, 다음의 지원 브라우저로 이용해주시길 바랍니다.<br/>안드로이드: Chrome, Samsung Internet<br/>아이폰: Chrome, Safari<br/>PC: Chrome<br/>앞으로 더욱 나은 서비스를 제공하기 위해 노력하겠습니다.<br/><br/><a href="http://www.joara.com/main.html?from=mobile">PC 웹으로 가기</a></div><script>var ele=document.getElementById("background-loading");setTimeout((function(){ele.classList.add("background-view")}),3e3)</script></div><script>function getBrowser(e){var r,n=e.match(/(android)\s([0-9\.]*)/i)||e.match(/(opera|chrome|safari|firefox|msie|samsung|trident(?=\/))\/?\s*(\d+)/i)||[];return/android/i.test(n[1])?{name:"android",version:parseInt(n[2])}:/trident/i.test(n[1])?(r=/\brv[ :]+(\d+)/g.exec(e)||[],{name:"ie",version:parseInt(r[1])||""}):"Chrome"===n[1]&&null!==(r=e.match(/\b(OPR|Edge)\/(\d+)/))?{name:r[1].replace("OPR","opera").toLowerCase(),version:parseInt(r[2])}:(n=n[2]?[n[1],n[2]]:[navigator.appName,navigator.appVersion,"-?"],null!==(r=e.match(/version\/(\d+)/i))&&n.splice(1,1,r[1]),{name:n[0].toLowerCase(),version:parseInt(n[1])})}function isSupported(e){return"android"===e.name||"chrome"===e.name&&e.version>=58||"safari"===e.name&&e.version>=10||"firefox"===e.name&&e.version>=54||"edge"===e.name&&e.version>=14||"opera"===e.name&&e.version>=55||"samsung"===e.name&&e.version>=8||"netscape"===e.name&&e.version>=5}var browser=getBrowser(window.navigator.userAgent);isSupported(browser)||(sessionStorage.setItem("browser",JSON.stringify(browser)),setTimeout((function(){window.location.href="/error_browser.html"}),500))</script><script>!function(e){function r(r){for(var n,a,l=r[0],f=r[1],i=r[2],c=0,s=[];c<l.length;c++)a=l[c],Object.prototype.hasOwnProperty.call(o,a)&&o[a]&&s.push(o[a][0]),o[a]=0;for(n in f)Object.prototype.hasOwnProperty.call(f,n)&&(e[n]=f[n]);for(p&&p(r);s.length;)s.shift()();return u.push.apply(u,i||[]),t()}function t(){for(var e,r=0;r<u.length;r++){for(var t=u[r],n=!0,l=1;l<t.length;l++){var f=t[l];0!==o[f]&&(n=!1)}n&&(u.splice(r--,1),e=a(a.s=t[0]))}return e}var n={},o={1:0},u=[];function a(r){if(n[r])return n[r].exports;var t=n[r]={i:r,l:!1,exports:{}};return e[r].call(t.exports,t,t.exports,a),t.l=!0,t.exports}a.m=e,a.c=n,a.d=function(e,r,t){a.o(e,r)||Object.defineProperty(e,r,{enumerable:!0,get:t})},a.r=function(e){"undefined"!=typeof Symbol&&Symbol.toStringTag&&Object.defineProperty(e,Symbol.toStringTag,{value:"Module"}),Object.defineProperty(e,"__esModule",{value:!0})},a.t=function(e,r){if(1&r&&(e=a(e)),8&r)return e;if(4&r&&"object"==typeof e&&e&&e.__esModule)return e;var t=Object.create(null);if(a.r(t),Object.defineProperty(t,"default",{enumerable:!0,value:e}),2&r&&"string"!=typeof e)for(var n in e)a.d(t,n,function(r){return e[r]}.bind(null,n));return t},a.n=function(e){var r=e&&e.__esModule?function(){return e.default}:function(){return e};return a.d(r,"a",r),r},a.o=function(e,r){return Object.prototype.hasOwnProperty.call(e,r)},a.p="/";var l=this.webpackJsonpjoara=this.webpackJsonpjoara||[],f=l.push.bind(l);l.push=r,l=l.slice();for(var i=0;i<l.length;i++)r(l[i]);var p=f;t()}([])</script><script src="/static/js/2.17eb9c13.chunk.js"></script><script src="/static/js/main.fc654eec.chunk.js"></script></body></html>"
---
Didn’t have to change user agent
Hm…
but the html source that you and I received is not real website.
When you enter URL in your browser, there will be another html source. OTL