Help parsing XML data

it is like the whether... I read the installation instrction. Found out that I use Companion version 2.77. tried the test page...

Connected it in AI2 via USB and... it works. The screen ist displayed with all components (e.g. buttons). The same project 30min before... the same setup, the same connection (tablet was not disconnected from pc) and the screen keeps white without any button or other components shown.

I also avoid aistarter.
It's like putting on your socks, shoes, and pants on in one operation.

I just start the Companion on my emulator,
do a Connect->Companion from my Project,
and type in the 6 letter code to the Companion app.

1 Like

Here's a test run of my on-the-fly filter approach:



shred_epg_de.aia (5.1 KB)

1 Like

Very cool. Thank you very much! It looks great. Let me test it tomorrow with the device itself if it will run into low memory issues. If not that would be really a great thing.

Maybe I have some questions for details of work principe. I I won't hesitate to ask you :slight_smile: but I will try to understand it by thinking about myself first!

hi @ABG, i tested it with my real device via usb connection. the good news: it works :slight_smile: I compared the result with the "real" epg app. applying the time offset for CET currently 1 hour fits exactly the tv program :slight_smile:

But I run into problems also faced with in the past and which were topic of many of my threads in the past. Despite that it takes a while to load data (that would be no problem because it will do it one time a day) the programm could not be loaded at first approach... I got the memory error.

image

I ignored that and pushed the button again for a channel and it works. Programm ist displayed in the list.

The second problem is that the popup for no reaction comes again and again. I had this problem often in the past when I deal with self created apps for arduino bluetooth control. But not always... the same app without changes one day no problem another day problems. Also like the weather. I could not reproduce it. I hate such phenomenons. I tried a lot of things... .heartbeat to arduino... and so on.

So all in all... it is exactly what I was trying to build. But I am not sure if it is possible to reach a reasonable level of stability. Here in that context auf AI2 I don't have the experience to judge that.

Your aia with your blocks helps me understand better the way of design and thinking for realization of ai2 projects. Thanks again!

Have you tried building the app into an apk file and downloading that to the tablet for installation as an Android app?

That should speed it up and reduce memory requirements.

I also did not yet read the docs for options to reduce time ranges on the web request.

I also skipped adding a downstream time filter for the display.

image

You have introduced an infinite loop in your blocks, somewhere.

@ABG I checked it on the phone via apk-file. The "no reaction" popup occurs while parsing the list. It takes around 2 - 5 minutes. After parsing (where channels are identified and programs are prepared / isolated for filter logic) it works without crash and the "no reaction" popup does not come up.

Rearding you bonus with the favorite list I have to check. When pushing different channels always the same programm is shown.

I will have in the following weeks a further look at it. During the week unfortunately have no time. And this thread will be closed after 7 days inactivity.

I have ideas for fixing those, will post here when done.
Bookmarking thread.

1 Like

I have tamed time filtering and progress display for program loading under heavy volume, but have not integrated multiple filtering (channel and time range), because that's a matter of personal taste.

Here's my Designer view:

I added a Status Label to show where the app is in the progress of receiving and splitting the program listing xml data.

I also added a Horizontal Arrangement with enough in it to set a time range for time based filtering of the program list, to reduce its volume.
image

To avoid memory overflows and to avoid swamping the Companion connection, I display only summaries while processing unfiltered data.

(I should add another Label to summarize post filtered data, or use it as a .Text of its next filter control.)

To free up the main UI, I added Clock Timers for splitting and filtering.

I use global variables as stepping stones, to capture intermediate results.
image

Screen1.Initialize:

First I ask for the full XML file. (The site offers no upstream filtering, unfortunately.)
I also prime the time filter components with current time, for later when they are needed.
I also disable the Clock Timer that is not yet ready to Run, because it has no data yet.

When the Web Requested XML arrives ...

I announce the entry into the next phase of the data load, and save the response Content for the next phase. (It's also handy for debugging).
The next phase is being handled in a Clock Timer, to give the UI a chance to update.

The Clock Timer is meant for single shot usage, so I immediately disable it.

I abandoned my shred procedure, in favor of a simpler and faster text split at the front markup of each program. The breaks the XML of the fragments, but they're simple to parse individually using the Taifun parse procedure.
image

The first item of the split has the channel catalog, which is discarded from the program list. Lastly I announce the list length to mollify the user for waiting so long.

Next post: Filtering the program list.

Program filtering is centered around these two global variables, setting the start and stop range that the app should show:
image

They are set in this Arrangement:
image

The setrange procedure loads the two global range limit variables from the Date and Time Pickers and the Hours span value.

The global rangeStart variable gets a 12 digit yyMMddHHmm value from the Date and Time Picker.

The global rangeStop variable needed to be calculated and formatted in steps, based on the rangeStart variable and the hours duration, using Clock Instant blocks.

(This could probably have been done in one step, but I have little capacity for complexity at the moment.)

Next step: the actual filtering of the program list, based on time range.

These two value procedures can extract the start and stop timestamps from a program XML fragment:

blocks (8)

blocks (9)

The end of each timestamp is a blank:
image
I have no idea what that +0000 is for, and it breaks math on the timestamp, so I clip it off in these functions.

Given these functions, we are now ready to write a value procedure to decide if a program overlaps our start stop range:


I use text comparison here, because I am not forcing the timestamps to the same length. The DateTime format yyyyMMddHHmm pays off here, because text comparison is the same as chronological comparison.
It would be easier to understand the overlap logic if you were to draw two parallel line segments on graph paper, with their own start and stop points.

...........start----------stop.............
...start----------stop................

That makes the Filter button Click event simpler:


I load a new global variable from the filter result.

Then I rashly sort it by start time and load it into the Elements list of the Programs ListView, without the chance to filter it further by channel, because that is a matter of taste I leave to you.


shred_epg_de (1).aia (10.0 KB)

Hi @ABG, I received your posts via notification on my smartphone and don' t know what I should say or how should I thank you. At least it triggered me to turn computer on while I was on the way to bed.

I will have a detailed look at it on weekend. I'm looking forward to it. Regarding the date / time I am very sure to know what it is. I deal with that in business programming. I guess it is a timestamp +0000 is an offset. A timestamp always is UTC. So the offset is 0000.

In Germany (CET) the timestamp would be +01:00 hour (normal time) +02:00 hours (summer time). I found e.g. that link It seams to be related to XML and an Oracle DB.

Best Regards
Thomas

1 Like

I had some free time to show an example. Compared the xml with the data shown in the app in local (CET) time. You can see, that currently 1 hour offsetz (we are currently in normal time not in summer time) must applied for display.

So... there are two options... convert the local time to utc and search the list. Then convert the UTC timestamp from search result to local time for display in the app. Or convert the UTC timestamp in XML to local time. But this is only unique when using the offset. Because last sunday in october some times are doubled e.g. 2:15 pm.

In business live the 1 approach turned out best. Convert local time to UTC, search, converte the search result back to local time for display. One aspect is: do the more complex conversion from local to UTC only one time (UI) then get the search result. Then convert the fount fewer records from utc to local.

Lets say: I want to see the whole program list for a sender from 00:00 to 23:59:59 for today. Then search sould be done for: 20260218230000 +0000 to 20260219225959 +0000 while a compareable timestamp in local would be: 20260219000000 +0100 to 20260219235959 +0100

Add Info: i found other links with epg data. But they turned out to be slower on the other hand have api' s (was not able to use them) or filter possibilities via URL parameters (I guess). The last other alternatives I found just now... Link 1 Here it would be possible to load channels separately. This version I tried also but it is very slow. XML larger.

We have been down this rabbit hole (wormhole?) before.

1 Like

Yes. I will have a look at it on weekend. The timestamp topic was only an information for you. This is the less important topic in my opinion.

image

There will need to be some conversion across time zones to make the lexigraphic start/stop comparisons work.

This requires further thought, which is in short supply on my end at the moment.

1 Like

I am absolutly fine with your support. Your input will help me if I will continue follow up this topic. I have some Arduino projects first. What I tried (after a nightly flash of inspiration) is to import that in Excel. Knowing that the newer versions internal deal with XML (.xlsx) suggest that it supports handlng capapilities. In fact there is a Power Query Editor. It handles the file of about 17 MB within seconds on PC. Of course not compareable to a tablet.

I'm still unsure whether AI2 can process this, or whether it needs to be programmed natively (Android Developer Studio).

Though AI2 has no Excel support, there is a Google Sheets Spreadsheet component and some history of using that with Google Scripting.

I found
https://www.google.com/search?q=google+sheets+import+xml+from+URI
to be of possible use if you want to filter and sort your xml data upstream for smaller downloads, in a hybrid approach.

I have no personal experience with Google Script yet, though.

hi @ABG, I made one last attempt to test the feasibility. I won't be pursuing this topic further for now. Using the following approach, I was able to achieve results that could be expanded upon (hybrid approach). But that's obviously not what I want.

Preparing the data in Excel. Problem: I only have a current version that supports Power Query on my work computer. I've now programmed this in VBA. Quick and "the word ist not allowed". This is also a good opportunity to address the UTC to CET issue. If you're already working on it, you don't have to do it later in the app.

I exported the data. CSV is impractical because the name "TV program" can contain a comma. The same applies to semicolons. I tried tab stops. I could not parse them. \n\r and '\t' and all the other tricks you can find online didn't work.

There are two versions: a short list and a long (complete) one. You can switch between them. Displaying a long list in the list viewer takes 30 seconds. I'm not going to test whether it can still filter the data (given the amount of data).

image

You'll find all the files attached if you're interested and want to try it out. Please don't be confused by the name of the AIA file. It's no longer a CSV file, but a text file with its own separator.

EPG_CSV_X1.aia (550.4 KB)

Excel File I can not post because of maybe containing personal data. I post some parts how conversion was done... converting the data ist very fast...

All marked data (column A rows 1 to n) will be analyzed... I trigger it directly in VBA but via Macros is also possible

This results in Tabelle1...

and is overworked with a formula. then copied directly from cells to a notepad document.

So... thats it so far from my side for that topic at the moment. Will continue build my mini retro Arduino game unit :slight_smile:

PS (VBA snippet)

Sub LoopThroughSelection()
    Dim cell As Range
    Dim parts() As String
    Dim curRow As Long
    Dim curCol As Long
    Dim count As Long
    
    Set regex = CreateObject("VBScript.RegExp")
    With regex
        .Global = False
        .IgnoreCase = True
        ' Muster: Zwischen diesen Tags
        .Pattern = ">(.*?)</title>"
    End With
    
    ' kein screen update
    Application.ScreenUpdating = False
    
    ' Sicherstellen, dass nur Zellen (keine Grafiken) durchlaufen werden
    If TypeName(Selection) = "Range" Then
        For Each cell In Selection
            If InStr(1, cell.Value, "<programme") > 0 Then
                ' Teilt den Text anhand des " Zeichens
                parts = Split(cell.Value, """")
                
                If UBound(parts) >= 2 Then
                    curRow = curRow + 1
                    ' channel
                    Worksheets("Tabelle1").Cells(curRow, 1).Value = parts(5)
                    ' start
                    ' Dieses Zeitstempel-Format (yyyymmddhhmmss +hhmm) ist eine Variante
                    ' der ISO 8601 Norm, speziell das sogenannte Basisformat (Basic Format)
                    ' für Datum und Uhrzeit. +hhmm (oder -hhmm): Zeitzonen-Offset (Stunden
                    ' und Minuten Abweichung von UTC
                    Worksheets("Tabelle1").Cells(curRow, 2).Value = parts(1)
                    ' stopp
                    Worksheets("Tabelle1").Cells(curRow, 3).Value = parts(3)
                End If
                
                ' jetzt in den nächsten zeilen nach dem titel suchen; aktuell
                ' werden maximal 3 folgezeilen ausgewertet
                count = 0
                Do
                    count = count + 1
                    
                    Set matches = regex.Execute(cell.Offset(count, 0).Value)
                    
                    If matches.count > 0 Then
                        Worksheets("Tabelle1").Cells(curRow, 4).Value = matches(0).SubMatches(0)
                        Exit Do
                    End If
                Loop Until count = 3
            End If
            
            ' Beispiel: Debug.Print cell.Value ' Wert im Direktfenster anzeigen
        Next cell
    End If
    
    ' screen update an
    Application.ScreenUpdating = True
End Sub