Relational databases are not adequate for the real time analysis needs of the IoT.
While preparing for my presentation and panel discussion “Semantic Learning in Message Bus” at the 2013 WI-IAT conference in Atlanta GA on November 20, 2013 I came across this quote that shows how visions and needs change with time. “Digital Equipment Corp. founder Ken Olsen’s 1977 quip, “There is no reason for any individual to have a computer in his home.” (Olsen did actually say that, but he said later that the quote was taken out of context, and that he was referring not to PCs but to computers set up to control houses.)”[i] This is a good reminder to us all to NEVER say never.
As I get deeper into the structure of the IoT and what the true needs to create efficiencies in the IoT are, I started to understand this statement and the focus of industry to continue on the path of rebranding legacy technologies and in the IoT case, keeping all the data that can potentially be captured. Continuing on this path will only create over blown databases that will keep growing, producing enormous revenue streams for the database, hardware and data center companies whilst creating money pits for their customers and most importantly impede the efficiencies that we are trying to create.
In my presentation at the WI-IAT conference I submitted that IoT will only reach its full potential when the capability to request, detect and direct only data pertaining to out of normal operation events of devices in real-time is needed and normal operational state data is only necessary in time stamped periodic updates. This structure will make the relational database insignificant in real time IoT, it will create a new type of technology which allows the data to be read while in transit, decisions made in real time allowing more expeditious delivery of message to application or human that can act in real time on the message and send controls back to the device for adjustment and efficient operation.
Relational databases can be optimized for fast inserts or fast queries but it is difficult to do both. In a traditional system where the database is trying to acquire all the data and then queries are run to find things of interest, queries tend to take a long time as they are running against an unnecessarily large amount of data and the optimization is likely for fast data insertion, not for fast queries.
IoT systems will want to transform the original data, producing a secondary stream containing only data relevant for IoT, and some of which may not have existed directly in the original data (hence the transform). This volume will be significantly less than the original (which may still go to the original destination) and can be more effectively utilized for the more real time IoT operations.
In the Semantic IoT structure, any potential unique events that could threaten equipment health or human life depends on the ability to identify events in message bus, send only those events to an analytic engine which can make decision on proper routing of message based on understanding of either exact or similar data points to define a potential endpoint (person or application) to make the control decision and adjustments to the device.
The lack of capability to forward a message to the proper control endpoint, due to the lack of a proper real-time interaction between message/service bus and database to delegate and deliver the message to proper endpoint will cause IoT to fail. An IoT that realizes all that we talk about can only be achieved by allowing the analytic engine to consume the relevant messages, not all messages, eliminating “clogging” the decision making engine. The historical data is important to the manufacturer of the equipment for the purpose of design changes in the specific device model, for the end user of the equipment which is working with many devices from the same or different manufacturer needs to be concerned with how the devices interact as a whole as well as individually.
I back this design overview with statements Dr. Ahmed El Adl, Ph.D[ii]. during a conversation on LinkedIn with myself and Dr. Steven Gustafson[iii] and a Futurescape: Robot Revolution on Science channel[iv] in which Dr. Michio Kaku[v] discussed what was needed for a computer to act as a brain.
Dr. El Adl LinkedIn comment
#IoT shouldn’t be just tool to collect #BigData. It is all about taking the right decision on the right time. BigData aren’t always required
Michael Holdmann Could not agree more, just posted blog in this subject, “Which Real time is truly important to IoT”. Getting big data to a db to queue until it can be analyzed which could be minutes, hours, days or weeks? or getting message to application or human that can act in real time to the change in value or state of device.
Steven Gustafson So, you are suggesting that we overemphasize the Sense part of IoT and need to focus more on the Act? What are key enablers for Act in your opinion?
Michael Holdmann The automated conducting of data to the correct ends points, via auto-attendants (bots/shepherds) of virtual rosters (flocks/dynamic rosters) of devices and data source to be auto-discovered, auto-configured, auto-managed, and auto-controlled through a secure channel that can be configured on the fly. Auto-attendants can be written in any language and reside inside or outside of the platform (assuming in process extension, out of process extensions, or client extensions).
Dr. Ahmed El Adl, Ph.D. Steven, Michael: great points. In my post Doc work in biologically inspired mobile robots, I used the traditional techniques to implement a mobile robot navigation system based on 3D cameras did the same using math model inspired by Cat’s vision system. As a research team, we achieved huge increase in smoothness of movements & reduction in the amount of data needed. Today, you see such techniques applied in most of robot systems. We’ll do better if we would learn from nature while we’re designing the overall IoT ecosystem; h/w, s/w and processes. The success of the IoT will not depend on how BIG is the data, but what data and how smart people, machines and processes are using it.
Dr. Michio Kaku’s presentation, he states when a human walks into a room the brain notices every item and event that is in the space, but only retains only the data that is necessary. Dr Kaku goes on that the brain is not able to retain all data as it would consume too much of the conscious state for the brain to operate efficiently so it only disseminates data and allows us to consciously be aware of the events that are relevant.
The comparison of IoT to robotics by Dr. El Adl could not be truer and the observation by Dr. Kaku is specific to how the IoT technologies need to be designed. In order for the projected efficiencies to be realized we must have an uninterrupted flow of information. Relational databases and other old technologies are not capable of making decisions and hence changes to critical events which produce machine failure or loss of life, if we are waiting for the data to be analyzed for seconds, minutes, hours due to the use of ancient messaging and analysis systems we will not succeed.