New markets are unleashed when key platforms emerge, allowing different products from different companies to work together seamlessly. But every market has multiple platforms contending to be number one, so we typically go through years of uncertainty until a de facto choice emerges.
With the announcement below from IBM, we may now know the winner in the race to be the preferred platform for Big Data analytics.
Spark is an open-source analytics tool that is part of the Berkeley Data Analytics Stack -- a platform brought forward by the University of California, Berkeley. It competes with another open-source technology known as Hadoop, and with dozens of proprietary approaches from vendors.
Spark already allows cities to process Big Data more quickly and more effectively, with one exec calling it "universal access to data." With the IBM enhancements, Spark will now get even more powerful. IBM has committed to improve Spark by linking it to some of IBM's most powerful technologies. And by pledging that more than 3,500 IBM researchers will contribute to its further evolution.
Big Data is simultaneously the most important and the most confusing aspect of a smart city. The path forward may just have gotten simpler. Now we need to learn if the sector's other heavyweights will also jump on the Spark bandwagon, or whether they will promote a rival instead. -- Jesse Berst
IBM believes Apache®Spark™ is potentially the most important new open source project in a decade that is being defined by data. Council Lead Partner IBM plans to embed Spark into its Analytics and Commerce platforms, and to offer Spark as a service on IBM Cloud.
IBM also intends to:
- Put more than 3,500 IBM researchers and developers to work on Spark-related projects at more than a dozen labs worldwide
- Donate its breakthrough IBM SystemMLmachine learning technology to the Spark open source ecosystem
- Educate more than one million data scientists and data engineers on Spark
Simplifies intelligent app development
As data and analytics are embedded into the fabric of business and society – from popular apps to the Internet of Things (IoT) – IBM says Spark brings essential advances to large-scale data processing. First, it dramatically improves the performance of data dependent apps. Second, it radically simplifies the process of developing intelligent apps, which are fueled by data.
Spark, IBM suggests, brings essential advances to large-scale data processing. First, it dramatically improves the performance of data dependent apps. Second, it radically simplifies the process of developing intelligent apps, which are fueled by data.
Open source innovator
"IBM has been a decades-long leader in open source innovation," said Beth Smith, General Manager, Analytics Platform, IBM Analytics. "We believe strongly in the power of open source as the basis to build value for clients, and are fully committed to Spark as a foundational technology platform for accelerating innovation and driving analytics across every business in a fundamental way."
Here is a real-world example of Spark in action: Real-time transportation planning software from Optibus is changing the way public transport is organized. "Spark, together with IBM, provides a highly scalable platform for Optibus, making it easy for us to expand our software as a service offering into new markets, and helps us simplify deployment, maintenance and application development for transportation companies worldwide," said Amos Haggiag, Optibus CTO and Co-Founder.
More on Big Data…
Alstom video: Big Data and Security in the Smart Grid
Microsoft case study: Big Data Solution Transforms Healthcare with Faster Access to Information
Free download: Smart Cities Open Data Guide