Web.Get returning 404 response code in Error


I thought I'd put this question out there for a little feedback before I post my code. Unfortunately, I'm on the train to work :frowning: , but reviewing the performance of my app so far :slight_smile: and I just can't post blocks right at the moment.

Part of my app creates a list and then downloads each of the files in that list sequentially. The community previously helped me with moderating the speed of those downloads in another forum question I raised. All good.

Now onto my current issue. The .Get function is mostly working well, bit for some reason the .Get function is returning a 404 error code for a number of files. Understandable - the files may actually be missing. However, I also create a .txt file of the http links that are in error. When I copy and paste each of these http links from that .txt file into the browser, the link is good and the files load.

I can reload my app and the same errors occur repeatedly.

I have tried both:

If response code = 404; and
If response code >< 200.

And both return the same list of supposedly erroneous links (which prove to be good links).

Looking for some clues during the day before I can get back to the PC and post some blocks.

1 Like

One thing I will try is changing the download sequencing. At the moment I use the whengotresponse block to initiate the next download.

I may try reverting to a clock based download process to set a minimum timeframe between each http.get call.

1 Like

Do you mean the when Web1.GotFile block ?

This may work, but it is better to 'wait' until the download process has finished, using the when Web1.GotFile block

You may need to test the urls for 'access' before you try to download, there must be some reason why the 'same' files do not download.

1 Like

Yes, sorry, I should have used the right block wording.

Yes, I was thinking of using a combination of both when Web1.GotFile and a clock timer - whichever is the longest. I'm not so confident that it's really the issue, but will try to be sure.

1 Like

Any spaces or special characters in the URLs that do not work? The web component may need the UriEncode block to get these.

1 Like

I don't believe so. The only things that are different between one URL and the next in the list are standard numbers (0-9) in the address. One URL will work, but the other not.

I'll try and review again tonight.

1 Like

I've done some further testing. I reviewed my code again and couldn't see why anything would be going wrong with a random number of files. None of the text in the URLs is unusual and when I copy/paste the text from the list (I write a txt file using the .AppendToFile block of the File extension), the links seem to work okay.... in some cases.

In doing that last step, I've actually found that the web page that I'm downloading the files from (not a web site under my control) does not make all the files available at once. In all, there are 65 files for each set of files. Here is an example. I don't know when exactly each batch of 65 files should become available, but at the following times, I found that:

7:00pm - 38 files were good (response code 200) and 27 were bad (response code 404)
7:15pm - 49 files were good (response code 200) and 16 were bad (response code 404)
7:20pm - 49 files were good (response code 200) and 16 were bad (response code 404)
8:00pm - 51 files were good (response code 200) and 14 were bad (response code 404)
and just now:
8:15pm - 58 files were good (response code 200) and 7 were bad (response code 404)

When I was doing my previous reviews of the problem code, I might have been checking near the end of a cycle and by the time I could verify (by copying pasting the URLs into my browser) that the files were bad, they had "come good" or otherwise made available on the web server.

I don't think the issue is with the code or anything to do with the app per se. Rather, it's the web server progressively making the files available. It's not a real good look for the server as any user will see all these broken links. I would have throught they would only make the batch of 65 files (or the index of the files) available once all of them were availble to be downloaded.

1 Like

Does sound like an issue on the web server....

Did you try using the UriEncode block ?


1 Like

I get a "The specified URL is not valid" error. I presume that's due to the : converting to %3A and the / converting to %2F. No other characters in the URL converted.

1 Like

Yes, messy. You would need to remove the https://, then uriEncode, then put the https:// back.

1 Like