These early CEP products already incorporated a number of performance-enhancing features such as parallelization and data caching. In the era of Big Data, however, data volume is measured in petabytes instead of terabytes. A Boeing 787 aircraft, for example, generates 40TB per hour of flight. Massive challenges arise in managing this data and making it “useful.”
With the sheer growth in data volume follows the inherent complexity in the data. Data is ingested with or without schema in textual, audio, video, imagery, and binary forms, sometimes multi-lingual and often encrypted, but almost always with real-time velocity.
The initial technology challenge in harnessing CEP is an infrastructural upgrade to address the data storage, integration, and analytic requirements. The end goal is to generate meaningful business insights from the ocean of data that can translate to strategic business advantages.
A new generation of architecture and engineering disciplines must be introduced into the practice – among them parallel streaming, linearly scalable databases designed for time series data, and in-memory computing. Fortunately, there are great open source applications and frameworks such as Spark and Hadoop that have emerged to address these challenges. Similarly, advances in Deep Neural Networks and CEP technologies help drive evermore sophisticated analysis processes.