KML File storage

[Fossil]

I realise there has been quite a lot written about files and the associated difficulty of accessing storage post Android 10 in particular - however I've not seen anything on KML files. I am developing an app that reads a Google sheet holding hills data which includes Lat / Log coordinates. For each line of the spreadsheet it produces KML script so that a pin and its associated 'balloon' can be displayed on Google Earth. I can get it to work on Android 6, but not Android 11, so I think re-coding is required. I have two questions: (1) Where is the best place in storage to save a KMl file? I have a couple of thoughts (a) should it be regarded as 'media' and stored under /storage/emulated/0/Documents or (b) an App specific location say /storage/emulated/o/Android/data/. I would like the KML file to persist if the App is uninstalled. (2) Given all the permissions issues (whicj I struggle to understand) is there any way to code this so that works for both Android 6 and Android 11?

BTW: After much reading, I've drawn the diagram below which is my understanging for Androd file storage - but if that is wrong then I'm not going to get very far. Comments welcome!

[SteveJG]

App Inventor can work with xml or json files; a kml file is a specially formatted xml. Technically, it is a basic text file.

See Overlay kmz data on maps

You can convert kml to json or json to kml. How? Here is a beginning https://groups.google.com/forum/#!category-topic/mitappinventortest/app-inventor-extensions/DqEeorciOC0 These links provide some other potential ideas.

https://www.google.com/search?q=kml+to+json&oq=kml+to+json&aqs=chrome..69i57.4175j0j7&sourceid=chrome&ie=UTF-8

Is this possible? What you do depends on why you need the coordinate information in kml format. https://developers.google.com/kml/documentation/kml_tut and why you need to work with Google Earth. KML traks can be displayed (with some transformation) on the Map component as a LineString.

A location in kml format looks like

Where to store on the device so that when the app is removed, the kml file persists.
On Android 11+, this is certainly a problem. Since you can store offline and your original data is on a Google Sheet; I would store on your Google Drive.

The state of storing files since the recent version of App Inventor 2 nb187 and the total revision of how the File component works makes writing to the internalSD card or an external card almost impossible. You could consider storing on a FlashDrive stick using OTG.

You show an interesting diagraml. Be aware the File component changes and ramifications of nb187 are being discussed Release nb187: Copy a text file / image from assets to shared storage (→ Android 10+) . MIT is working to fix bugs and solve issues related as to what can read/write and where as you read this.

[Anke]

@Fossil

I've drawn the diagram below which is my understanging for Androd file storage - but if that is wrong then I'm not going to get very far. Comments welcome!

See also here:

Some basics on Android storage system

From an Android point of view there are an → Internal Storage and an → External Storage . 1. Internal Storage The Internal Storage can only be accessed with a rooted device. 1.1 The app package is saved in /data/data/<packageName>/ 1.2. The Private directory is /data/user/0/<packageName>/files/ This private directory can be used with the File component to save / read text (setting the path without a slash). It can only be accessed by your app and is automatically removed when the app is u…

[Fossil]

Hi Steve,
Thanks for your feedback on KML and KMZ and various links. In my case I am ‘simply’ using my ‘PinZ’ app to generate the KML file. I’m quite happy to use a file manager to manually zip the KML file with the display pins (16 png images) to create the KMZ. That KMZ can be opened in Google Earth and it shows all the classified hills in the UK and Ireland and which ones I’ve climbed (green pins) along with associated data. See screenshot below - which was previously generated from VBA, but I’ve given up on Microsoft.

Reasons for this app - it is just a planning tool and convenient when discussing where we might go next. The primary record used to be Excel, but is now a Google Sheet in Drive. I don’t suppose it is vital that the KML is persistent if the app is uninstalled, as by that time I could manually move it to Google Drive and in any case it is periodically updated. Given the total mess that Google seem to have got into with file storage on Android devices I guess I could write the KML back directly to Drive under my PinZ app control. (Mind you, never done that – so that might be a different headache.)

For information I find the easiest approach to generating the KML is to have template statements say ##10 where the ## is replaced with the actual data. These can be stored as variables or lists. In my case I need about 40 templates to cover all the usual KML header stuff plus folders, normal and rollover styles, place, baloons, coordinates etc. It’s not difficult, just tedious and code clutter can be much reduced with the use of two or three efficient and robust procedures.

Thanks again.

[Fossil]

PS I've just noticed the kml template statement example got chewed up - but I'm sure you know what I mean.

[Fossil]

Hi Anke

Thanks for responding with your diagram of file storage paradigm. I like it and it is certainly a lot more detailed than my efforts. I must consider it’s implications in more detail, but meantime I have some comments – but I know you will be busy, so don’t feel obliged to reply.

(1) I’m probably not alone in finding the terminology ‘Internal’ and ‘External’ confusing. Surely in physical terms this storage is all internal within the device as opposed to say external being in the cloud or a removable card. Therefore, are these really conceptually ‘Internal’ and ‘External’ in relation to a particular App’s point of view – ie ‘Internal’ to the App or ‘External’ to the App. Thus, what might be ‘Internal’ to one App, is ‘External’ to another. If that is the case maybe some annotation can be added to the diagram. My diagram was trying to convey differences between the physical and conceptual storage and their relationship.

(2) That said the fact that the app-specific-directory (ASD) sits under ‘External’ but crosses over and points back to Private Storage does my head in! What is the point of having designations ‘Internal’ and ‘External’ when their meaning seems so obtuse? The diagram might as well start at the row below with the five categories of storage.

(3) Also, the diagram suggests that ‘Storage’ can go direct to ‘Private Storage’ and ‘Shared Storage’ without being any of the five categories in the row above. How can that be?

(4) In AI2 the code can designate a file related action with a FileScope such as App, Private or Shared etc. Although there is a category termed scoped / shared storage I don’t see how those designations fit into the storage hierarchy in the diagram. What do the designations actually do?

I’m sorry if I seem rather negative and of course nothing to do with you. It is not as if I’m new to computing as I first wrote code decades ago, but I have never ever encountered such a confusing file structure under any operating system. The ability to read and write to file is fundamental and as far as I’m concerned if this situation is due to Google then they have got it very badly wrong. I guess MIT AI just have to implement it the best they can.
Thanks again.

[Anke]

@Fossil

(3) Also, the diagram suggests that ‘Storage’ can go direct to ‘Private Storage’ and ‘Shared Storage’ without being any of the five categories in the row above. How can that be?

I only tried to integrate the missing central terms → Internal Storage and → External Storage into the diagram. And I think I managed to do that quite well.

As well as naming special folders that work as shared storage.

[Anke]

Theoretical explanations are always nice, but you better tell us what your problem is.
Then I would like to come back to my overview.

[Anke]

@Fossil

but meantime I have some comments

Here are a few more comments on your points:

(1) Internal storage means, among other things, that the user can only access it with a rooted device.
The path to the appPackage and the private data dir is:
/data/data/...
/data/user/0/...

The path to the external storage is:
/storage/emulated/0/...
where /storage/emulated/0/ is the root directory of the external storage.

The right understanding of internal and external storage is fundamental for any work with Android.

(2) The private storage is in the internal storage and, as far as the ASD is concerned, in the external storage. What's hard to understand about that? Both will be deleted when the app is uninstalled.

(3) No, I just haven't bothered to remove those lines from the original. But the above part should complement and explain the logic of the whole storage system.

(4) That is described precisely. There are some shared folders that act as shared storage and are reserved for certain file types. What is unclear about that? Basically, that is what Scope Storage Stuff is all about.

But I like to admit that this scoped storage stuff is confusing, see also here.
That's why I created this overview.

[Fossil]

Well, the actual problem is I want to write a text file into a 'standard' Android folder like Documents or Download or even one of my own, say 'FossilDir', that I've manually set up using filemanager. It should be simple. No? But it seems impossible.
I've written a test app that allows me to select all the FileScopes and a few candidate directories - but everything I've tried goes to somewhere like /Android/data/files///MyFile.txt
Surely it must be possible to write to a 'standard' folder?

[Anke]

@Fossil

I want to write a text file into a 'standard' Android folder like Documents or Download

I have already shown several times that it is basically possible:

Release nb187: Copy a text file / image from assets to shared storage (→ Android 10+)

By the way, I've already mentioned that several times before:

Testing the next release of MIT App Inventor (nb187)

Here is a test APK that only works with the File component (no extension is used): Tested on Android 11 (Pixel 2XL). Bugs: File.Exists from assets does not work with the APK (Companion works). [grafik] WRITE permission must be declared in the Manifest so that it works after the automatic request of storage permission. If the permission is denied: "Error 908: The permission READ_/WRITE_EXTERNAL_STORAGE permission has been denied. Please ..." depending on which action is carried out. …

Testing the next release of MIT App Inventor (nb187)

But there are still some bugs with the File component and other components that can save something (Canvas, ...).

[SteveJG]

If the app you build is for your personal use; place the text file into Assets when you create the app. I don't have an Android 11 so I can't test but expect you will be able to read the text file using the File component. The File scope probably will have to be Asset or Legacy. When you modify the text file externally, reload it to Assets and rebuild the app.

The File component is currently in a state of flux. The File system is broken and a solution needs to be found to store data consistently internally in various folders. This will take time. Google has made it plain, it will no longer be possible to write from an app to certain locations with Android 11+.

Work arounds for storing and updating (writing to) files are suggested that use various extensions and gimmicks that probably work. Go ahead and try them. While MIT experiments to find a panacea, those work arounds should become obsolete when it finds a path forward. Sooner or later, MIT will release a new update to the File component.

I'm not worried, I have an Android 8.1 and expect a File solution before I have to replace my Android. If not, I also program using several professional compilers like Android Studio.

@Fossil

Surely it must be possible to write to a 'standard' folder?

Certainly, but only some folders; see Anke's advice.

[Fossil]

Thanks folks for your comments and suggestions. The app is for my own personal use and I've got it to write and read text files to a directory. Not the directory I'd like it in, as it is burried deep inside \Android\data\stuff\stuff\ - but with a half decent file manager I can find it and manually move it to Google Drive or wherever. So that will have to do me meantime - but it is clear that the whole file system is as you say Steve 'in a state of flux'. I sure hope that Google / MIT sort out the 'flux' soon otherwise a whole category of apps that need to w/r to convenient locations will become effectively impossible and the concept of AppInventor as an attractive educational first step very much compromised. PS Just discovered that you can't have 3x's in a row here hence '\stuff' above - how perverse is that!

[SteveJG]

@Fossil

but with a half decent file manager I can find it and manually move it to Google Drive or wherever.

If you now go to all that trouble, consider the two ways to use modified data mentioned in this tutorial to store certain types of data HOW TO: Set Runtime Map Marker Characteristics using a CSV file

These methods work with Android 8.1 and might work with Android 11. One involves writing your file outside the app then storing it in Media and read all you want using the new File control. The other involves storing the data in a Google Sheet and downloading to the app whenever you need it.

I realize you want to store kml files. The tutorial doesn't address a kml. You might be able to convert your spreadsheet data to a csv ( use the Web component to convert it to a csv) and convert the downloaded csv to a kml. You already know KML uses a tag-based structure with nested elements and attributes and is based on the XML standard and App Inventor can handle XML.

None the less there are options other than what you described. Good luck with your Project.

The discussion in progress indicate Google has closed the door to writing to the sdCard directories but will allow reading. App Inventor developers will have to adapt. Either save/write within the allowed app directory or save and read from the cloud and retrieve what you need using a Web Get storing everything offline.