Getting big file from SDcard; max row length 10,244 chars?


For my app I want to read a .csv file (12 x 2000 cells) into it.
I planned to use the blocks .ReadFrom and .GotText and get the data in a usable format with list-from-csv-table.

I experienced 2 problems:

  • A single row longer then 10,244 chars is broken up by list-from-csv-table in two pieces;
    more exact I found that the number of characters plus - because list-from-csv-table is adding a comma between all cells - the number of cells in a row row minus 1, may not be larger than 10,244.

  • Very strange behaviour of my app; after a long time struggling I understood that in this case my project has probably gone beyond the maximum size of 10MB.
    I found this rule on this very helpful site

My solution was to break up the original file in three pieces, two of 12x550 cells and one with 12x900. All rows met the max-length criterium.

In my app I used a do-loop with append to concatenate the rows and add to place them in a matrix as intended.

Can somebody comment on this?

My questions are:

  • Should, in general, rows to be processed by list-from-csv-table, be smaller than 10,244 chars?
  • Is there a way to estimate the size of your project, so you can check you are safe away from 10 MB? Together my files count in Windows only 200 kB.
  • Is there a better way to get such a file into an app than the road I took?

Thanks for all comments!

Can somebody comment on this?
big_file_append_and_add_SDcard_lean.aia (7.4 KB)

Can you share the original csv file? If you do not want it on the forum, private message a Power User....

OK, had a look, and using list from csv table is not the way to go.

Try with the split block. First split the text returned from the file by "\n" in order to get each row in a list. Then split an item in the list again by ",". You should see that your very long "cell" is kept all together in the list. (Note, this is dependent on their being no other commas that might break it up). If you do have commas in your long cell, you could try a find and replace using a character that is not a comma.

Example blocks

1 Like

Here is a method that handles commas in the very long text. There is an assumption, that the very long text is the only item in each csv row that is surrounded by double quotes. I used text manipulation to handle most things.

Say the data in the csv file looks like this (I have shortened the very long cell for demo purposes) - note invisible "\n" in the file

1,Lorem,"Very, long, ....text 1"
2,Ipsum,"Very, long, ....text 2"

These blocks then use the split function to break up the csv, convert the commas, then put them back, and eventually returning an AI2 list with the very long cell intact.


SCREEN (after import)

1 Like

I usually export the aia project to my computer, check the file size, and add 3.5mb (size of empty Ai2 compiled apk)

The main server now handles projects up to 30mb, if you use the code server then up to 50mb. You can go bigger if you compile, decompile, add files then recompile, up to 100mb (Play Store limit)

1 Like

Thank you TIMAI2 for your comments!
I'm not wired allday so I will not react fast in every case, sorry for that.
The double split suggestion looks good as a simple way to get rid of from-csv-table.
I expect no comma's in my data, so thats ok. When I'm back home I start trying this.
My data consists of bird names in different languages en keywords to look for them. Some of those are key to so many birds that the cell with all the necessary index numbers gets a long string to hold.

I'll send you the original file via DM anyway.

That will be good. there may be a better way of organising you base data - sounds like a database....

!!! Solved !

I'm very glad, days and days of trying is fixed in 30 mins!
The double split route works fine. I only had to change the split text to a semicolon ; to get a nice row.
In a do-loop adding over 2079 rows gave me the matrix I would like to have.
Now I have to change other parts of my app because transposing the original Excel file is not necessary with this method.

I'll keep this in mind for when I increase my files with more languages/birdnames.

Thank you very much TIMAI2 for your clear comments.
Your last advice I will store for later use in cases where I got stuck again.

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.