DataChanged for php tinywebDB

For those of you using or thinking of using a php based tinywebdb, I have, with some help and encouragement from @Juan_Antonio, come up with a data changed event solution.

You see this event in use with CloudDB and FirebaseDB components, and believe what I have devised follows a similar approach of keeping a connection open to the data, watching for changes. In php terms is is called "long polling". The original concept comes from here:

Try as I might, I could not get the caching to switch off on the webviewer, in order to use the javascript (as above) and the webviewstring to return data to the app, so decided to use a web component instead to replace the client side coding. Thus, all that is needed is the php file on the server, with any other php files and the database file/s, and a few blocks in the app.

You need:

The php file:

<?php

set_time_limit(0);
$data_source_file = 'database.json';

while (true) {
    $last_ajax_call = isset($_GET['timestamp']) ? (int)$_GET['timestamp'] : null;
    clearstatcache();
    $last_change_in_data_file = filemtime($data_source_file);

    if ($last_ajax_call == null || $last_change_in_data_file > $last_ajax_call) {

        $data = file_get_contents($data_source_file);
        $result = array(
            'data_from_file' => $data,
            'timestamp' => $last_change_in_data_file
       );
        $json = json_encode($result);
        echo $json;
        break;

    } else {
        sleep( 1 );
        continue;
    }
}

?>

You will have to replace the name of the database file with your own database file if different. In the example above it is called database.json. You could feed this as a parameter to the php file if you have more than one database running...

For the blocks:

A global variable for the timestamp

blocks (7)

your server URL and the php file name

blocks (17)
blocks (18)

a call to the php url from the web component (note: my "server.php" is called "dataChanged.php"), with the timestamp parameter

blocks (19)

and then the web gotText block, which handles the returned json array to output the changed data and set the new timestamp, with a looping call to the web component again (which will sit and wait for a response / change in data). I found in testing that I needed to set the responseContent to a variable/local variable for the dictionary processing to work. For some reason this wouldn't work directly.

That is all that is required. The web component will keep the connection open until a change is made to the database, then report back with new data, and then open up the connection again, wait for a change....

Probably some work here for @ABG to do a dictionary comparison to find exactly what changed :wink:

2 Likes

I couldn't add to this.
It is already a work of art.

1 Like

OK, php provides a nice differencing function for arrays, in this case array_diff_assoc, I have used this along with some if/else statements to return the data changed in all circumstances, whether it is done in the app you are using, the same app used by others, or any changes to the database made either directly of by other means. It may also pick up any changes made while the app is closed/asleep....but I have not specifically handled this - my approach copes with one change at a time.

I also added the setting of the database as a parameter into the blocks and the php.

blocks (20)

The php uses a text file to grab the "previous" set in the database, then brings this back for differencing. You will see that if/else statements are used to check for the type of change: CREATED/UPDATED/DELETED - and NO CHANGE.

<?php

set_time_limit(0);
$data_source_file = $_GET['db'];


while (true) {
    $last_ajax_call = isset($_GET['timestamp']) ? (int)$_GET['timestamp'] : null;
    clearstatcache();
    $last_change_in_data_file = filemtime($data_source_file);

    if ($last_ajax_call == null || $last_change_in_data_file > $last_ajax_call) {

        $data = file_get_contents($data_source_file);
        
	$new = json_decode($data,true);
    $getprev = file_get_contents('prev.txt');
    $prev = json_decode($getprev,true);
    
	if ( empty($prev)) {
		$prev = $new;
	}

	$n = count($new);
	$p = count($prev);

	if ( $n < $p ) {
		$result = array_diff_assoc($prev, $new);
		$output = array("DATA_DELETED",$result);
	} elseif ( $n > $p ) {
		$result = array_diff_assoc($new, $prev);
		$output = array("DATA_CREATED",$result);
	} elseif ( $n == $p ) {
		$result = array_diff_assoc($new, $prev);
		$c = count($result);
		if ( $c == 0 ) {
			$output = array("DATA_NOT_CHANGED",$result);
		} else {
			$output = array("DATA_UPDATED",$result);
		}
	}
	

	$res = array(
            //'data_from_file' => $data,
            'timestamp' => $last_change_in_data_file,
	        'change' => $output
        );
        
    	file_put_contents("prev.txt",$data);
        $json = json_encode($res);
        echo $json;
        break;

    } else {
        sleep( 1 );
        continue;
    }
}

?>

For the Web GotText, I needed to add a check in order to return and display the change information, otherwise the process runs again and returns an NO CHANGE output (and then sits and waits for a change). The output is returned as a json array, so if the developer wants to use this as a list, they must perform the same check to capture the change output.

1 Like

The long hold on a connection is great for the client (not having to poll), but it must have some cost for the server.

How does this affect the server's ability to service client requests from other clients?

Is there a maximum number of open connections per server?

Do server hosts charge by the connection minute?

How does this affect the server's ability to service client requests from other clients?

The php file has a 1 second wait in it that blocks the PHP/Apache process for that time

Is there a maximum number of open connections per server?

Not tested :wink:

Do server hosts charge by the connection minute?

Not mine :wink:

@ABG - update:

I had a look at my hosting providers dashboard. Whilst I was tooling around with setting up and testing my php file (e.g. open connection) I was using @ 4% of my CPU and 11mb of memory, fairly constant over a 3 hour period.

Author's comments:

@Juan_Antonio

links to the same thing on github as above....

In my testing with two devices, the data changed event only fired in one of the devices (didn't seem to matter if it was the device that made the change, but probably the device that last called the php file...?)

It may be that for this to work with multiple devices, each one will need its own dataChanged php file? This is possibly doable using includes, but it might clog up the server with all the connections open ?

Hi,

I was using this method as described (and it was working), however, I then received a notice from Awardspace.com that I had violated their Terms of Service, because " Too many hits (9946) counted for the past hour. Occurred 2 times for the past 3h".

Any recommendations for changes that could be made to reduce the number of hits to the server?

Thanks

This is an issue on your web host, not the script/blocks.

There is no real point in trying to change the existing script/blocks, because that would negate the whole point of getting a dataChanged event.