TL;DR
We broke (slightly) CloudDB when we introduced dictionaries.
Read on for the nitty gritty details.
Background
We use Redis software to implement CloudDB storage. The CloudDB server
itself provides a front end that enforces confidentiality (aka
encryption in transport) as well as isolation between users (so you
can only mess with your variables).
Redis is a key value store with a simple Publication/Subscription (aka pub/sub)
event system built in. CloudDB variables are keys in the Redis store.
However, the pub/sub system isn’t sufficient for our needs in
CloudDB. In particular when a variable changes, it emits an event, but
the event itself doesn’t provide the new value. So, you have to then do
a GET to get the value. However, in apps that are making many updates
(like our Sketch and Guess Tutorial) you can lose updates in the time
between the event is triggered, and you manage to do a GET.
Fortunately, Redis implements scripting using the LUA language. The LUA
implementation permits you do to “atomic” operations within Redis. So
when CloudDB does a StoreValue, we do not simply turn that into a
Redis SET. Instead, we invoke a LUA script which does the set and emits
an event with the new value. It’s kind of cool actually.
Now we do limit which scripts can be run to avoid a security hole, but
that isn’t at issue here.
The “Append” and “Remove First” functionality is also implemented as
LUA scripts for the same reason. The low-level implementation stores
the list as a Redis “table” object. We use JSON to encode the table
when we return it to CloudDB. For some reason that I don’t understand
(yet), Redis internally JSON encodes a table with contents as a list
(good, this works for us) but an empty table is returned as an empty
JSON “object” (aka {}) instead of an empty list (aka []).
Prior to our support for dictionaries, we silently turned this empty
object {} into an empty list [] because we didn’t support
dictionaries, which are represented as JSON objects.
Since we added support for dictionaries, we no longer do this
conversion.
Welcome to the land of unintended consequences. This is a classic
example. We were not aware of this Redis “feature” and prior to our
support for dictionaries, it was completely benign for us.
How to fix this
The correct fix is for us to update the LUA script used in the
POP_FIRST LUA Script to test for the empty object and turn it into an
empty list. However, this is a component change, which means a new
Companion and all that entails.
I’m seeing if I can do a kludge in the CloudDB server itself and
detect when a list has been emptied and explicitly set it to an empty
list instead of an empty object. My current thinking is that I can
write a new LUA script, which would eventually get put into the
CloudDB Component, but also do a hack on the CloudDB server that
recognizes the older script and replaces it with the newer one… I
sill have to think about this though…
-Jeff
