CloudDB Remove Last item from a List Value results in an empty Dictionary (Bug Report)

I encountered what surely must be a bug in the CloudDB retrieval of emptied CloudDB lists.

Here is my experiment sequence ...

P.S. These blocks can be dragged directly into your Blocks Editor workspace.

Initialize a CloudDB list ...

Add an item (a pair) to the list

Set aside a global variable to receive the list on GotData
global result
when CloudDb1 GotValue

Verify the contents of the tag/value pre-delete

one row
(This looks reasonable, a list of one sublist)

Remove the item from the CloudDB tag/value

Get the new value

The new result:
new result

The switchover from list to dictionary in the result variable makes problems for code that needs to sense depletion of the list after a CloudDb GotData event or CloudDB dataChanged event.

Oops. I think I broke it. I was able to reproduce this bug (not hard, thanks Abraham!). I'm working on a fix which hopefully will be deployed today!


Hmm. I may have spoken to quickly, I'm not sure if I really broke it, but it is broken. I'm still digging...


My testing did not test which end was broken:

  • the receipt of the value from the database
  • the form of the storage in the database.

I should have tested adding the same pair to the list twice, to see if
the database was treating it like

  • a list of lists (2 entries making it a 2 by 2 table) or
  • a dictionary (only one entry, a single pair)


We broke (slightly) CloudDB when we introduced dictionaries.

Read on for the nitty gritty details.


We use Redis software to implement CloudDB storage. The CloudDB server
itself provides a front end that enforces confidentiality (aka
encryption in transport) as well as isolation between users (so you
can only mess with your variables).

Redis is a key value store with a simple Publication/Subscription (aka pub/sub)
event system built in. CloudDB variables are keys in the Redis store.

However, the pub/sub system isn’t sufficient for our needs in
CloudDB. In particular when a variable changes, it emits an event, but
the event itself doesn’t provide the new value. So, you have to then do
a GET to get the value. However, in apps that are making many updates
(like our Sketch and Guess Tutorial) you can lose updates in the time
between the event is triggered, and you manage to do a GET.

Fortunately, Redis implements scripting using the LUA language. The LUA
implementation permits you do to “atomic” operations within Redis. So
when CloudDB does a StoreValue, we do not simply turn that into a
Redis SET. Instead, we invoke a LUA script which does the set and emits
an event with the new value. It’s kind of cool actually.

Now we do limit which scripts can be run to avoid a security hole, but
that isn’t at issue here.

The “Append” and “Remove First” functionality is also implemented as
LUA scripts for the same reason. The low-level implementation stores
the list as a Redis “table” object. We use JSON to encode the table
when we return it to CloudDB. For some reason that I don’t understand
(yet), Redis internally JSON encodes a table with contents as a list
(good, this works for us) but an empty table is returned as an empty
JSON “object” (aka {}) instead of an empty list (aka []).

Prior to our support for dictionaries, we silently turned this empty
object {} into an empty list [] because we didn’t support
dictionaries, which are represented as JSON objects.

Since we added support for dictionaries, we no longer do this

Welcome to the land of unintended consequences. This is a classic
example. We were not aware of this Redis “feature” and prior to our
support for dictionaries, it was completely benign for us.

How to fix this

The correct fix is for us to update the LUA script used in the
POP_FIRST LUA Script to test for the empty object and turn it into an
empty list. However, this is a component change, which means a new
Companion and all that entails.

I’m seeing if I can do a kludge in the CloudDB server itself and
detect when a list has been emptied and explicitly set it to an empty
list instead of an empty object. My current thinking is that I can
write a new LUA script, which would eventually get put into the
CloudDB Component, but also do a hack on the CloudDB server that
recognizes the older script and replaces it with the newer one… I
sill have to think about this though…


Thanks for the explanation.

Here is a somewhat raw multiplayer Rock Paper Scissors test bed for this feature ...

RPS_CloudDB_using_lists.aia (12.2 KB)
(It's not yet ready for Prime Time.)

So, I have a fix which I'm testing. There is a fix for the CloudDB component, which will be in an upcoming component release. There is also a work-around which I can deploy to the CloudDB server in the meantime. I expect to deploy the work around this weekend.


So, the fix is now in review for the next component release.

The work-around is also installed on the CloudDB server, so you should see this work as expected now, even with the existing Companion and existing compiled/packaged apps.


Thanks, but I am having trouble testing with the Companion, getting stuck at 20%.

There seems to be a bunch of this going around.

Switching from Brave to Firefox let me finish my Companion connect.

Testing was able to proceed, confirming the fix.

I'm looking at the 20% problem. I'll write more about it tomorrow.


P.S. What version of Chrome is having problems?

In my case, 89.0.4389.82

Brave is up to date

Version 1.21.74 Chromium: 89.0.4389.72 (Official Build) (64-bit)

The problem was a change made in Chrom(ium) 89 broke our WebRTC implementation. I have deployed a mitigation and it should work again.