Proposal for Principles of Foundation of Open Cloud Federation

This post has been prepared as contribution to the Boston Open Research Cloud Workshop that MIT is hosting on May 11-12, 2017. The aim of the workshop is to discuss the issues, reach consensus on how to move forward, and begin actively working towards the establishment of a federated scientific research cloud.

The two main cloud computing deployment models are public and private clouds. Public clouds are multi-organization, shared environments that are hosted, managed and operated by a third-party provider. Private clouds are single-organization, dedicated environments that can be hosted on-site at the organization data center or off-site at a service provider’s data center. Both deployment models provide agility, improve experience, flexibility and instant provisioning, but while private clouds offer greater control of security, customization and performance, public clouds offer higher scalability, simplicity, low upfront investment costs and pay per use.

Both public and private clouds can be multi-site infrastructures consisting of multiple geographically distributed, but tightly coupled data centers. These datacenters are typically replicated with a few instances on a continent wide scale in semi-autonomous zones. For example, at the time of this writing, Amazon maintains 38 availability zones within 14 geographic regions around the world, Microsoft operates 30 regions, Softlayer operates 31 data center facilities and Google operates 6 cloud regions.

Although cloud computing has grown, developed and evolved very rapidly over the last half decade, cloud federation continues being an open issue. Vendor lock-in, trust, security and incompatibility issues are plaguing current cloud offerings and hampering innovation.

A federated cloud is not a large-scale cloud based on a multi-site distributed data center infrastructure but a cloud composed of resources from different providers. In the federated cloud model, clouds managed by different organizations federate to allow users to utilize any of the connected clouds.

Hybrid clouds combine the existing on-premise private cloud infrastructure with outsourced resources from one or more external public clouds, enabling to transform the local datacenter into a highly scalable IT environment. Hybrid cloud computing is the next step beyond private clouds in the evolution of cloud computing, and it is emerging as the mainstream of IT as more and more organizations are embracing or planning to embrace hybrid cloud as part of their IT strategy.

With the quickly increasing number of scientific institutions building private clouds to support their research projects, a growing pressure has been placed to establish conventions supporting the federation of research and scientific cloud resources, and their hybrid combination with public cloud resources. There are several cloud federation initiatives around the world building distributed infrastructures made of academic and science private clouds and driven by requirements of the scientific community. Some fundamental principles should be adopted in order to define an open framework for cloud federation that integrates existing standards and best practices. This  would allow any research institution, hosting provider, telecom or public cloud provider to join this open interoperable ecosystem.  

I suggest the following principles as the foundation of Open Research Cloud Federation:

To read this article in full or to leave a comment, please click here

9 Years of Open-source Driving Innovation in Cloud Computing

Nine years ago, in February 2008, OpenNebula announced the first open-source management software for deploying private and hybrid clouds, and released one of the most transformative platforms for innovation in the cloud. OpenNebula was the first of many open-source software components that now constitute an active and quickly growing open ecosystem in a private cloud market estimated in $7 Billion.

“Open source” software refers to source code that is freely available for use, modification and, depending on the terms of its license, redistribution of modified and extended versions of the software. It is not only that the code is open and free, collaboration between technology providers and users is a huge advantage in the open source arena. One of the beauties of open source is that it is a meeting point between users and innovators, both from industry and academia, where they can freely share resources and knowledge. Open source software is also playing a vital role in the timely dissemination and transfer of important research and technology advances in cloud computing.

Open-source has been a growing phenomenon for more than two decades, but it has been with cloud computing when open-source has become the leader for innovation and the dominating form of technology, disrupting the traditional, commercial closed-source software business. During these years, we have witnessed how open source has played a key role in driving and supporting the transition to cloud computing, accelerating the pace of innovation, and facilitating new products and services.

To read this article in full or to leave a comment, please click here

OpenStack Cloud Bubble Has Started to Burst

It was a couple of years ago in march 2014 when we wrote about the small user base and penetration in the market of OpenStack, mostly compared with the investment made by vendors and VCs. At that time we estimated that the investment in OpenStack was approximately $300 million per year. This was quite conservative, there were other estimates above $0.5 billion per year, some reaching to $1 billion per year. These estimates mainly include development cost and marketing of upstream OpenStack community. Additionally, there are other direct costs from many companies also allocating resources to develop their proprietary OpenStack distribution, training or documentation, a big overhead in indirect costs, and of course opportunity costs.

Some months after this post, in September 2014, 451 Research made public a market analysis forecasting that OpenStack market was growing from $883 million in revenue in 2014 to $1.7 billion in 2016, with service providers with clouds enabled by OpenStack representing 73% of that revenue total. This revenue is relatively very small in comparison with the investment being made and the revenue of the market leaders such as VMware ($6 billion in 2014 ), AWS (now close to $7 billion a year) and Red Hat Linux (now close to $2 billion a year). Cumulative investment was disproportionate to any realistic return. It just does not seem feasible that there is room for several vendors to profit from OpenStack. There were sufficient indicators, for those willing to open their eyes, showing that OpenStack was in a bubble.

To read this article in full or to leave a comment, please click here

Which Are the Top Cited Publications and Authors in Cloud Computing?

The IEEE Transactions on Cloud Computing Journal is publishing a paper entitled “A Scientometric Analysis of Cloud Computing Literature” by Leonard Heilig and Stefan Voß (University of Hamburg) that presents an analysis of publication patterns, research impact and research productivity in the field of Cloud Computing. Two interesting results are the top cited publications and top cited authors in Cloud Computing.

 The results on the top publications in cloud computing show that the following paper is the top cited publication.

• “Armbrust M., Fox A., Griffith R., Joseph A.D., Katz R., Konwinski A., Lee G., Patterson D., Rabkin A., Stoica I., Zaharia M. (2010) A view of cloud computing. Communications of the ACM 53(4):50-58.”  

I am proud to be a co-author of two publications listed number 7 and 17 respectively.

• “Sotomayor B., Montero R.S., Llorente I.M., Foster I. (2009) Virtual infrastructure management in private and hybrid clouds. IEEE Internet Computing 13(5):14-22”, which describes the original OpenNebula design and architetcure, and 

• “Rochwerger B., Breitgand D., Levy E., Galis A., Nagin K., Llorente I.M., Montero R., Wolfsthal Y., Elmroth E., Caceres J., Ben-Yehuda M., Emmerich W., Galan F. (2009) The Reservoir model and architecture for open federated cloud computing. IBM Journal of Research and Development 53(4):1-11”, which describes the cloud federation research and technology developed in the Reservoir Project. 

Regarding top authors, Rajkumar Buyya from University of Melbourne is the top cited author, while Ignacio M. Llorente and Rubén S. Montero are in the 26th position of the list. Only three European authors are part of the top cited authors list.

Bringing Private Cloud Computing to HPC and Science

HPC-optimized clouds provide access to flexible and elastic scientific and technical computing to solve complex problems and drive innovation. You can find bellow the slides of my keynote at EGI-TF 2013 where I described the most demanded features for building HPC and science clouds, and illustrated using real-life case studies from leading research and industry organizations how OpenNebula effectively addresses these challenges of cloud usage, scheduling, security, networking and storage. The keynote ended with a view of private cloud's future in HPC and science, and grid as the foundation of cloud federation.

 

OpenNebula: The European Management Platform to Build IaaS Clouds

Last Thursday, the blog of the European Commission Digital Agenda for Europe published an article entitled "OpenNebula: the only European open-source management platform to build IaaS clouds! A success story in exploitation of FP7 research results". It briefly shows how OpenNebula is playing an important role in driving and supporting the transition to cloud computing and thus accelerating the pace of innovation in Europe.

We are proud of being part of a project that is shaping the future of cloud computing!.

Key Challenges in Cloud Computing: Enabling the Future Internet of Services

"Key Challenges in Cloud Computing: Enabling the Future Internet of Services" is the title of our last article published in the August 2013 issue of the IEEE Internet Computing Magazine that describes how Cloud computing will play a major role in the future Internet of Services, enabling on-demand provisioning of applications, platforms, and computing infrastructures. The article describes the several technology challenges that must be addressed to turn this vision into reality. Specific issues relate to deploying future infrastructure-as-a-service clouds and include efficiently managing such clouds to deliver scalable and elastic service platforms on demand, developing cloud aggregation architectures and technologies that let cloud providers collaborate and interoperate, and improving cloud infrastructures' security, reliability, and energy efficiency.

Enjoy the article!

About the Openness of Eucalyptus, CloudStack, OpenStack and OpenNebula

We have just published a post at the OpenNebula blog with our view about the different measures that should be considered to evaluate the openness of the main open-source Cloud Management Platforms (CMPs). Which is the most important measure of openness in cloud computing? Do the cloud users really care about this? Users mainly want a solution that meets their functional needs, and are interested in open-source as a way to enhance flexibility, lower costs and avoid lock-in. However, as discussed in the post, most of these benefits are only available when an open-source software can be used in production environments without the addition of proprietary components. 

 It is worth reading!.

 

A Roadmap for Advanced Cloud Technologies under H2020

This is the title of the last report prepared by the EC Cloud Expert Group that proposes a concrete roadmap to help Europe overcome existing technological challenges and reach a key position in the global Cloud market. The previous reports on Advances in Clouds - Research in Future Cloud Computing evaluated the state-of-the-art and developed future research directions in cloud computing. This report is the main result of our last meeting in October 2012 to analyze the expected market development in which Europe can participate and which technological challenges will have to be faced in order to reach these scenarios.

So, enjoy the report. I think it is a must-read for people defining Cloud Computing strategies, developing innovative research lines, or exploring emerging market opportunities beyond today’s Clouds.

About the Complementarity of Eucalyptus, CloudStack, OpenStack and OpenNebula

We have just published at the OpenNebula blog our experience about the different types of cloud models, from datacenter virtualization to infrastructure provision, and about how the main open-source Cloud Management Platforms (CMPs) are targeting their needs. The post introduces the CMP Quadrant, a tool aimed at aiding corporations to better understand the present and future landscape of the cloud market. 

I think it would be best for everyone from these open source projects to figure out how we can work together to better serve our users.

It is worth reading!.

IaaS Cloud Architecture: From Virtualized Datacenters to Federated Cloud Infrastructures

"IaaS Cloud Architecture: From Virtualized Datacenters to Federated Cloud Infrastructures"  is the title of our last article published in the December 2012 issue of the IEEE Computer Magazine that presents an architecture reference model for IaaS clouds. The article introduces the cloud OS (Operating System) platform as the core component of this architecture that is responsible for managing the physical and virtual infrastructure, orchestrating and commanding service provisioning and deployment, and providing federation capabilities for accessing and deploying virtual resources in remote cloud infrastructures. The article ends with a description of different architectures for cloud federation. OpenNebula is also mentioned as an open-source reference implementation of the cloud OS platform.

Enjoy the article!

3 Ways the EU Is Supporting Open Source

Neelie Kroes, European Commissioner for Digital Agenda, presented on Friday at the Open Source Conference 2012 the three ways the European Union (EU) is supporting Open ICT systems, namely open-source, open-procurement, and open-data. Mrs. Kroes outlined the huge benefits of open-source and open-standards, the several hundred-million-euro benefits per year of Open ICT systems for the public sector alone, and how the EU is using open source solutions itself.

 

Mrs. Kroes made reference to OpenNebula as flagship of European open-source cloud innovation supported by EU investments that is laying the basis for interoperable data centers.

Since 2005, OpenNebula has helped many organizations develop value by building innovative cloud services and solutions to meet their user and customer needs in new ways or to meet new market needs. OpenNebula is playing an important role in driving and supporting the transition to cloud computing, and in accelerating the pace of innovation on the datacenter side.

We are really proud of this!

OpenNebula: An Open-source Cloud Management Platform Focused on User Needs

Here is the recording of the OpenNebula session at GigaOM Structure Europe. The interview covers the OpenNebula's focus on user needs and its recipe for success described in the post David Versus Goliath in Open-source Cloud Computing: The OpenNebula Recipe for Success.

David Versus Goliath in Open-source Cloud Computing: The OpenNebula Recipe for Success

Now that OpenNebula has been listed by GigaOM on the list of main reasons why Europe really matters to cloud computing, I am receiving several emails asking about how OpenNebula is able, what’s its "secret recipe", to compete with other open-source projects, like OpenStack, CloudStack or Eucalyptus, backed by strong IT vendors and with wide media presence. Here are some of its ingredients, mix them into a single product:

• A vibrant and engaged community. Many people and organizations contribute in different ways to the project, from the expertise and dedication of our core committers and hundreds of contributors to the valuable feedback of our thousands of users. 
• The power of user-driven development. OpenNebula's roadmap is completely driven by users needs with features that meet real demands, and not features that result from an agreement between IT vendors planning to create their own proprietary cloud solution. 
• Volunteer development by the users. Most of contributors are users of the software that are willing to contribute new innovative features from their production environments, they are not developers hired by vendors to contribute to the project. 
• Delivery as a production-proven, packaged product. OpenNebula comprises all key functionalities for cloud computing with a single installing, patching and updating process. Other open-source alternatives require proprietary components to be ready for the enterprise. 
• Cloud-API agnostic. OpenNebula provides cloud consumers with choice of interfaces, from open cloud by main standards bodies to de-facto standards. OpenNebula does not try to reinvent the wheel and create a new cloud API. 
• Richer functionality and wider integration capabilities. OpenNebula does not only bring an open-source implementation of the most common public cloud interfaces, but also the latest innovations in the management of virtualized data centers for the deployment of cutting edge enterprise clouds.  
• Marketing-free technology. OpenNebula mainly invests its resources in developing technology and serving its users, being really vendor agnostic and free of marketing. 

In summary, a vibrant and engaged community, along with our focus on solving real user needs in innovative ways and the involvement of the users in a fully vendor-agnostic project, constitute the OpenNebula’s recipe to compete with the rest of open-source cloud management platforms.

Thanks to all the people and organizations that have contributed to OpenNebula since its foundation in 2005!

European Union Unveils its Cloud Computing Strategy

During the last two years I have been reporting about the different consultation events undertaken during the development of the cloud computing strategy. Now, finally, the European Union has just announced a strategy to boost the use of cloud computing among its members in the private and public sectors and an implementation plan based on three cloud-specific actions:

• Cutting through the Jungle of Standards 
• Safe and Fair Contract Terms and Conditions 
• Establishing a European Cloud Partnership to drive innovation and growth from the public sector

The document entitled Unleashing the Potential of Cloud Computing in Europe sets out these actions and serves as a call on all stakeholders to participate in their implementation.
This is very good news, but we have to remember that creating a good strategy is only the start. The real challenge is in execution.

What Role Does Open-source Play in Cloud Computing Innovation?

What Role Does Open-source Play in Cloud Computing Innovation? is a guest post that I have written on GigaOM to show how open-source is playing an important role in driving and supporting the transition to cloud computing. According to our experience, the availability of open-source cloud management tools like OpenNebula is accelerating the pace of innovation on the datacenter side.

Since we started the OpenNebula project in 2005, we have helped many organizations develop value by building innovative cloud services and solutions to meet their user and customer needs in new ways or to meet new market needs. The guest post looks at cloud innovation from different perspectives, including some specific examples.

Why is Europe Behind the US on Cloud Computing?

I have decided to write this post after receiving several emails from different people asking me about the reasons for the slower adoption of cloud computing in Europe. In comparison to the US, it is clear that EU is some years behind in size and maturity of cloud infrastructure provision and adoption.

In my view, this delay is mostly due to different factors:

A. From the supply side

• Most of the main providers of cloud services are US, and although they are building data centers in Europe, they initially focused on the US market. Only Amazon reaches almost a 60% market share of the IaaS Market according to the 451 Group 
• Europe does not have large suppliers who may compete in public provision with the major US suppliers. Only the European Telecoms are big and powerful enough to compete, but they still have to become big players in this space
• European ICT industry is mostly formed by innovative SMEs. More than 99% of all European businesses are SMEs
• Main skills of European ICT industry are in provision of software services, so working at a level of abstraction higher than the provision of infrastructure as a service. A trait of the European software industry is its focus on applications as opposed to development tools, middleware, and system infrastructure software

B. From the demand side, and according to the study Quantitative Estimates of the Demand for Cloud Computing in Europe and the Likely Barriers to Take-Up by IDC

• EU organizations are more conservative, being slower in the adoption of new services and technologies
• Small organizations are usually the most reluctant adopters 

C. Last, but not least, market fragmentation

• Europe does not have a single digital market, and the main concerns about cloud services are security, data location and legal jurisdiction over data. So there is a clear need for a legal framework for international data protection and privacy to overcome fragmentation and allow to trade across borders

In my view the US has currently a better developed cloud infrastructure and Europe runs a high risk of becoming dependent on non-European providers, and this is critical if we consider cloud as the next utility. However, Europe has many advantages and strengths that can lead to opportunities in a cloud market which is about to rapidly accelerate:

• A strong Telecom industry with operators that are starting to leverage their unique advantages
• Many hosting and small cloud providers developing innovative cloud offerings 
• EU-funded research projects and infrastructures, and open source technologies that are bringing innovation to the market and helping create an open interoperable cloud ecosystem 

The Commission is preparing the European Cloud Computing Strategy considering demand and supply side concerns in order to make Europe not only cloud-friendly but cloud active. This Strategy seeks to

• Harmonize the legal framework for cloud
• Raise awareness & promote uptake of Cloud computing, mostly among SMEs and public authorities
• Promote interoperability, portability and open-source to create an open interoperable cloud ecosystem 
• Support research on critical issues to create better cloud 

Work has started with the publication of a report that a select industry group has prepared with key recommendations to the European Commission on the orientation of a Cloud computing strategy for Europe, and the report on Advances in Clouds - Research in Future Cloud Computing prepared by the EC Group of Experts in Cloud Computing that includes a state-of-the-art view on cloud computing technologies, its position in and its relevance for Europe.

European Digital Agenda: From Cloud-friendly to Cloud-active

Yesterday we participated in the second edition of the cloud computing workshop within the Digital Agenda Assembly 2012 aimed at defining the European Cloud Computing Strategy. Last year, the aim of the workshop was to identify the main elements of a European cloud strategy and the possible need for public-policy intervention, considering demand and supply side concerns. This edition has mainly focused on policy issues to coordinate cloud computing in Europe, international aspects of cloud, and key areas in contract terms, such as data preservation after termination of the contract, data integrity and disclosure. 

The agenda, presentations and recommendations are available at the Digital Agenda web site.

OpenNebula Cloud for HPC at NASA Ames

"Cloud Computing Architecture with Open Nebula – HPC Cloud Use Cases" is the title of the invited talk that I gave yesterday at the NASA Advanced Supercomputing Division – NASA Ames. I provided an overview of cloud computing architecture with OpenNebula, with special focus on cloud deployments for High Performance Computing environments. You can find bellow the slides and the abstract of the presentation. It was a great opportunity to interact with Piyush Mehrotra's group in the Computational Technologies Branch to discuss how OpenNebula can address their needs, and to talk with some of the people involved in the Nebula Project. Thanks for the invitation!.

Cloud Computing Architecture with Open Nebula - HPC Cloud Use Cases - NASA Ames 2012

OpenNebula is a fully open-source cloud management platform, with excellent performance and scalability to manage tens of thousands of virtual machines, and with the most advanced functionality for building virtualized enterprise data centers and private cloud infrastructures. OpenNebula is the result of many years of research and development in efficient and scalable management of virtual machines on large-scale distributed infrastructures. Its innovative features have been developed to address the requirements of business use cases from leading companies in the context of flagship international projects in cloud computing. OpenNebula is being used by many supercomputing and leading research centers to build HPC and science clouds for hosting virtualized computational environments, such as batch farms and computing clusters, or for providing users with new “HPC as a service” resource provisioning models. The talk describes how to design a cloud architecture with OpenNebula and its innovative features to enable the execution of flexible and elastic cluster and high performance computing services on demand while reducing the associated cost of building the datacenter infrastructure.

Advances in Clouds - Research in Future Cloud Computing

It has been more than two years since the European Commission published in January 2010 its pioneering report about the Future of Cloud Computing. A group of experts was established with the aim to evaluate the state-of-the-art and develop future research directions in cloud computing. Since then, there has been considerable advances in the field, developments have closed some gaps that were identified in this report, but more challenges have emerged.

We were re-convened by the European Commission in 2011 in order to capture these changes and maintain a state-of-the-art view on cloud computing technologies, its position in and its relevance for Europe. The experts, led by Keith Jeffery, Lutz Schubert and Maria Tsakali, have produced a final version of this report entitled Advances in Clouds - Research in Future Cloud Computing. The report brings valuable information for people defining Cloud Computing strategies, developing innovative research lines, or exploring emerging market opportunities beyond today’s Clouds. It is a must-read.