Slowdown in Retrieving tags from firebase in Real-time

latest timestamp with the set of readings

So that's only a couple of dozen bytes, right?

Is the tag 1684067825, and the value
{"bodytemp":79,"gas":1680,...} ?

That's easy to decode with the Web JSONDecodeWithDictionaries block into a dictionary, with the parts you want.

(But I don't see "bpm" or "spo" in your data tree?)

image
I changed the variables overtime as I was testing, also a new data set is being recieved with each new timestamp

In the Data Changed Event?

By a new dataset you mean which?

  • All the data so far, or
  • the latest timestamp and its fields?

If the second, can you display the value for us in a Label or capture it in a global variable for a Do It?

the latest timestamp and its fields

A setup like this should return the last reading/timestamp/data

in your firebase rules:
image

in your blocks, use a web component to query the data:
image

1 Like

Here is how to show the newly arrived data for a single timestamp, assuming the tag is the timestamp and the value is a dictionary with your keys and values:


This dictionary ties text keys to Label components, simplifying extraction later.



firebase_data_changed_timestamped.aia (2.7 KB)
P.S. The firebase component I used was set to the default MIT server, so you would have to adapt it to your server.

@ABG

how do you know the timestamp value to call back the tag?

According to the OP, it arrives all by itself in the Data Changed event, without having to ask for it, timestamp by timestamp, in real time.

I am taking it on faith that the pump has been primed.

I believe all the data arrives from "readings", which is causing the slow down.

See my solution , getting firebase to do all the work....

I conducted a little experiment on the MIT Firebase server to see if timestamped entries could be caught one by one in a Data Changed event.

It worked for me, using BlueStacks multi-emulators ...
Animation

firebase_data_changed_timestamped.aia (4.8 KB)

Note: My List Picker Elements lag behind my Tag Lists because in the interest of speed I only refresh my List Picker Elements when I use the List Pickers, not when new data arrives.

(added to FAQ)

But the OP indicated they wanted the latest / last timestamp ?

In my example, they get the latest added while they are monitoring.

Am I moving the target to fit where the arrow landed?

In my example, I get Firebase to do all the grunt work, and just return the latest/last timestamp value of the tag (that may or may not be the same as the timestamp value - in an earlier screenshot of the data the tag and the timestamp value were different)

Your arrows are rarely, if ever, wide of the mark though :wink:

I will leave it up to the OP which solution fits him.

I have not heard back from him if he has control over the indexing in the database, or if he is stuck at the receiving end with no control.

I would check into Firebase docs to see what they think of indexing on a steadily ascending value like a milliseconds timestamp. Those tend to skew B-tree indexes, unless the technology has improved since I last saw one.

If he goes with an index, it would be interesting to see if the extra index slows down his fast arriving inserts.

Your query definitely gets just the last value, so if he is bouncing among different data collections, that's the way to go.

Sorry bout replying late, I was busy with the other aspects of the project, the thing is, I'm trying to build a real time monitoring system so that the user can see their reading on the MIT application, in real time as they are using the sensors, so if any change in data occurs, they can see it on the app, from firebase, the data is continuously generating when the ESP32 is connected to the internet. so I need the data on the app to be transmitting to the app continuously in real time.

Let us take a few steps back.

Do you want all the readings for a user returned to the app, or just the latest/last reading available for a user, at any given time ?

How often is a new reading generated for a user ?

The readings are being updated on firebase every 6 seconds, I require the latest timestamp, whenever it is updated on firebase to be displayed on the app.

image

Setup your indexOn as I showed above:

image