ODPi Advances Hadoop Standards with Open Source Runtime Specification

The ODPi, a Linux Foundation collaborative project building, has moved one step closer to building a better integrated big data ecosystem based on open source technology. This week, it announced that several major Hadoop distributors have adopted its runtime standard.

The ODPi’s goal is to solve a challenge that has become a major issue not only for big data companies, but for virtually all organizations that resell or integrate open source platforms like Apache Hadoop. That challenge is ensuring that different solutions remain inter-operable even though they come from competing vendors and diverge from the core open source product.

In the case of big data, the ODPi aims to solve this problem through an open source runtime specification that defines how Hadoop-based applications should work. The ODPi launched in September 2015 with backing from companies like IBM, Toshiba and VMware, as well as a variety of big data analytics and business intelligence vendors.

Now, the Linux Foundation has announced that several commercial Hadoop distributions are compliant with the ODPi runtime specification. They include the distributions from Altiscale, ArenaData, Hortonworks, IBM and Infosys.

The ODPi and the Linux Foundation are celebrating the announcement as proof that the big data space can be standardized in ways that benefit all stakeholders, from vendors and consumers to ISVs. “Our members represent all key roles in the big data ecosystem — Apache Hadoop distros, ISVs, system integrators, solution providers and end users — and have brought their real-world experience in helping to align enterprise demands with the developer community,” said John Mertic, director of program management for ODPi. “Having Altiscale, ArenaData, Hortonworks, IBM, and Infosys declare compliance with the ODPi Runtime Specification is a strong step toward simplifying and standardizing the big data ecosystem to accelerate the delivery of business outcomes.”

The ODPi’s next step is to finalize an operations specification, which will simplify Hadoop installation and management, according to the organization. That will debut later this year, the group says.

Gloom-and-doomers might note that, in the best of all possible worlds, Hadoop standards would have existed before the ecosystem grew so large. They didn’t, and the ODPi is consequently attempting to retrofit standards onto products that have already been in the wild for a long time.

But that’s certainly better than having no standards at all. And the fact that some vendors are now signing on to the ODPi runtime specification shows that the retrofitting can be done.