Proposal for Principles of Foundation of Open Cloud Federation

This post has been prepared as contribution to the Boston Open Research Cloud Workshop that MIT is hosting on May 11-12, 2017. The aim of the workshop is to discuss the issues, reach consensus on how to move forward, and begin actively working towards the establishment of a federated scientific research cloud.

The two main cloud computing deployment models are public and private clouds. Public clouds are multi-organization, shared environments that are hosted, managed and operated by a third-party provider. Private clouds are single-organization, dedicated environments that can be hosted on-site at the organization data center or off-site at a service provider’s data center. Both deployment models provide agility, improve experience, flexibility and instant provisioning, but while private clouds offer greater control of security, customization and performance, public clouds offer higher scalability, simplicity, low upfront investment costs and pay per use.

Both public and private clouds can be multi-site infrastructures consisting of multiple geographically distributed, but tightly coupled data centers. These datacenters are typically replicated with a few instances on a continent wide scale in semi-autonomous zones. For example, at the time of this writing, Amazon maintains 38 availability zones within 14 geographic regions around the world, Microsoft operates 30 regions, Softlayer operates 31 data center facilities and Google operates 6 cloud regions.

Although cloud computing has grown, developed and evolved very rapidly over the last half decade, cloud federation continues being an open issue. Vendor lock-in, trust, security and incompatibility issues are plaguing current cloud offerings and hampering innovation.

A federated cloud is not a large-scale cloud based on a multi-site distributed data center infrastructure but a cloud composed of resources from different providers. In the federated cloud model, clouds managed by different organizations federate to allow users to utilize any of the connected clouds.

Hybrid clouds combine the existing on-premise private cloud infrastructure with outsourced resources from one or more external public clouds, enabling to transform the local datacenter into a highly scalable IT environment. Hybrid cloud computing is the next step beyond private clouds in the evolution of cloud computing, and it is emerging as the mainstream of IT as more and more organizations are embracing or planning to embrace hybrid cloud as part of their IT strategy.

With the quickly increasing number of scientific institutions building private clouds to support their research projects, a growing pressure has been placed to establish conventions supporting the federation of research and scientific cloud resources, and their hybrid combination with public cloud resources. There are several cloud federation initiatives around the world building distributed infrastructures made of academic and science private clouds and driven by requirements of the scientific community. Some fundamental principles should be adopted in order to define an open framework for cloud federation that integrates existing standards and best practices. This  would allow any research institution, hosting provider, telecom or public cloud provider to join this open interoperable ecosystem.  

I suggest the following principles as the foundation of Open Research Cloud Federation:

To read this article in full or to leave a comment, please click here

Which Are the Top Cited Publications and Authors in Cloud Computing?

The IEEE Transactions on Cloud Computing Journal is publishing a paper entitled “A Scientometric Analysis of Cloud Computing Literature” by Leonard Heilig and Stefan Voß (University of Hamburg) that presents an analysis of publication patterns, research impact and research productivity in the field of Cloud Computing. Two interesting results are the top cited publications and top cited authors in Cloud Computing.

 The results on the top publications in cloud computing show that the following paper is the top cited publication.

• “Armbrust M., Fox A., Griffith R., Joseph A.D., Katz R., Konwinski A., Lee G., Patterson D., Rabkin A., Stoica I., Zaharia M. (2010) A view of cloud computing. Communications of the ACM 53(4):50-58.”  

I am proud to be a co-author of two publications listed number 7 and 17 respectively.

• “Sotomayor B., Montero R.S., Llorente I.M., Foster I. (2009) Virtual infrastructure management in private and hybrid clouds. IEEE Internet Computing 13(5):14-22”, which describes the original OpenNebula design and architetcure, and 

• “Rochwerger B., Breitgand D., Levy E., Galis A., Nagin K., Llorente I.M., Montero R., Wolfsthal Y., Elmroth E., Caceres J., Ben-Yehuda M., Emmerich W., Galan F. (2009) The Reservoir model and architecture for open federated cloud computing. IBM Journal of Research and Development 53(4):1-11”, which describes the cloud federation research and technology developed in the Reservoir Project. 

Regarding top authors, Rajkumar Buyya from University of Melbourne is the top cited author, while Ignacio M. Llorente and Rubén S. Montero are in the 26th position of the list. Only three European authors are part of the top cited authors list.

Bringing Private Cloud Computing to HPC and Science

HPC-optimized clouds provide access to flexible and elastic scientific and technical computing to solve complex problems and drive innovation. You can find bellow the slides of my keynote at EGI-TF 2013 where I described the most demanded features for building HPC and science clouds, and illustrated using real-life case studies from leading research and industry organizations how OpenNebula effectively addresses these challenges of cloud usage, scheduling, security, networking and storage. The keynote ended with a view of private cloud's future in HPC and science, and grid as the foundation of cloud federation.

 

OpenNebula: The European Management Platform to Build IaaS Clouds

Last Thursday, the blog of the European Commission Digital Agenda for Europe published an article entitled "OpenNebula: the only European open-source management platform to build IaaS clouds! A success story in exploitation of FP7 research results". It briefly shows how OpenNebula is playing an important role in driving and supporting the transition to cloud computing and thus accelerating the pace of innovation in Europe.

We are proud of being part of a project that is shaping the future of cloud computing!.

Key Challenges in Cloud Computing: Enabling the Future Internet of Services

"Key Challenges in Cloud Computing: Enabling the Future Internet of Services" is the title of our last article published in the August 2013 issue of the IEEE Internet Computing Magazine that describes how Cloud computing will play a major role in the future Internet of Services, enabling on-demand provisioning of applications, platforms, and computing infrastructures. The article describes the several technology challenges that must be addressed to turn this vision into reality. Specific issues relate to deploying future infrastructure-as-a-service clouds and include efficiently managing such clouds to deliver scalable and elastic service platforms on demand, developing cloud aggregation architectures and technologies that let cloud providers collaborate and interoperate, and improving cloud infrastructures' security, reliability, and energy efficiency.

Enjoy the article!

A Roadmap for Advanced Cloud Technologies under H2020

This is the title of the last report prepared by the EC Cloud Expert Group that proposes a concrete roadmap to help Europe overcome existing technological challenges and reach a key position in the global Cloud market. The previous reports on Advances in Clouds - Research in Future Cloud Computing evaluated the state-of-the-art and developed future research directions in cloud computing. This report is the main result of our last meeting in October 2012 to analyze the expected market development in which Europe can participate and which technological challenges will have to be faced in order to reach these scenarios.

So, enjoy the report. I think it is a must-read for people defining Cloud Computing strategies, developing innovative research lines, or exploring emerging market opportunities beyond today’s Clouds.

About the Complementarity of Eucalyptus, CloudStack, OpenStack and OpenNebula

We have just published at the OpenNebula blog our experience about the different types of cloud models, from datacenter virtualization to infrastructure provision, and about how the main open-source Cloud Management Platforms (CMPs) are targeting their needs. The post introduces the CMP Quadrant, a tool aimed at aiding corporations to better understand the present and future landscape of the cloud market. 

I think it would be best for everyone from these open source projects to figure out how we can work together to better serve our users.

It is worth reading!.

IaaS Cloud Architecture: From Virtualized Datacenters to Federated Cloud Infrastructures

"IaaS Cloud Architecture: From Virtualized Datacenters to Federated Cloud Infrastructures"  is the title of our last article published in the December 2012 issue of the IEEE Computer Magazine that presents an architecture reference model for IaaS clouds. The article introduces the cloud OS (Operating System) platform as the core component of this architecture that is responsible for managing the physical and virtual infrastructure, orchestrating and commanding service provisioning and deployment, and providing federation capabilities for accessing and deploying virtual resources in remote cloud infrastructures. The article ends with a description of different architectures for cloud federation. OpenNebula is also mentioned as an open-source reference implementation of the cloud OS platform.

Enjoy the article!

3 Ways the EU Is Supporting Open Source

Neelie Kroes, European Commissioner for Digital Agenda, presented on Friday at the Open Source Conference 2012 the three ways the European Union (EU) is supporting Open ICT systems, namely open-source, open-procurement, and open-data. Mrs. Kroes outlined the huge benefits of open-source and open-standards, the several hundred-million-euro benefits per year of Open ICT systems for the public sector alone, and how the EU is using open source solutions itself.

 

Mrs. Kroes made reference to OpenNebula as flagship of European open-source cloud innovation supported by EU investments that is laying the basis for interoperable data centers.

Since 2005, OpenNebula has helped many organizations develop value by building innovative cloud services and solutions to meet their user and customer needs in new ways or to meet new market needs. OpenNebula is playing an important role in driving and supporting the transition to cloud computing, and in accelerating the pace of innovation on the datacenter side.

We are really proud of this!

OpenNebula: An Open-source Cloud Management Platform Focused on User Needs

Here is the recording of the OpenNebula session at GigaOM Structure Europe. The interview covers the OpenNebula's focus on user needs and its recipe for success described in the post David Versus Goliath in Open-source Cloud Computing: The OpenNebula Recipe for Success.

European Union Unveils its Cloud Computing Strategy

During the last two years I have been reporting about the different consultation events undertaken during the development of the cloud computing strategy. Now, finally, the European Union has just announced a strategy to boost the use of cloud computing among its members in the private and public sectors and an implementation plan based on three cloud-specific actions:

• Cutting through the Jungle of Standards 
• Safe and Fair Contract Terms and Conditions 
• Establishing a European Cloud Partnership to drive innovation and growth from the public sector

The document entitled Unleashing the Potential of Cloud Computing in Europe sets out these actions and serves as a call on all stakeholders to participate in their implementation.
This is very good news, but we have to remember that creating a good strategy is only the start. The real challenge is in execution.

Why is Europe Behind the US on Cloud Computing?

I have decided to write this post after receiving several emails from different people asking me about the reasons for the slower adoption of cloud computing in Europe. In comparison to the US, it is clear that EU is some years behind in size and maturity of cloud infrastructure provision and adoption.

In my view, this delay is mostly due to different factors:

A. From the supply side

• Most of the main providers of cloud services are US, and although they are building data centers in Europe, they initially focused on the US market. Only Amazon reaches almost a 60% market share of the IaaS Market according to the 451 Group 
• Europe does not have large suppliers who may compete in public provision with the major US suppliers. Only the European Telecoms are big and powerful enough to compete, but they still have to become big players in this space
• European ICT industry is mostly formed by innovative SMEs. More than 99% of all European businesses are SMEs
• Main skills of European ICT industry are in provision of software services, so working at a level of abstraction higher than the provision of infrastructure as a service. A trait of the European software industry is its focus on applications as opposed to development tools, middleware, and system infrastructure software

B. From the demand side, and according to the study Quantitative Estimates of the Demand for Cloud Computing in Europe and the Likely Barriers to Take-Up by IDC

• EU organizations are more conservative, being slower in the adoption of new services and technologies
• Small organizations are usually the most reluctant adopters 

C. Last, but not least, market fragmentation

• Europe does not have a single digital market, and the main concerns about cloud services are security, data location and legal jurisdiction over data. So there is a clear need for a legal framework for international data protection and privacy to overcome fragmentation and allow to trade across borders

In my view the US has currently a better developed cloud infrastructure and Europe runs a high risk of becoming dependent on non-European providers, and this is critical if we consider cloud as the next utility. However, Europe has many advantages and strengths that can lead to opportunities in a cloud market which is about to rapidly accelerate:

• A strong Telecom industry with operators that are starting to leverage their unique advantages
• Many hosting and small cloud providers developing innovative cloud offerings 
• EU-funded research projects and infrastructures, and open source technologies that are bringing innovation to the market and helping create an open interoperable cloud ecosystem 

The Commission is preparing the European Cloud Computing Strategy considering demand and supply side concerns in order to make Europe not only cloud-friendly but cloud active. This Strategy seeks to

• Harmonize the legal framework for cloud
• Raise awareness & promote uptake of Cloud computing, mostly among SMEs and public authorities
• Promote interoperability, portability and open-source to create an open interoperable cloud ecosystem 
• Support research on critical issues to create better cloud 

Work has started with the publication of a report that a select industry group has prepared with key recommendations to the European Commission on the orientation of a Cloud computing strategy for Europe, and the report on Advances in Clouds - Research in Future Cloud Computing prepared by the EC Group of Experts in Cloud Computing that includes a state-of-the-art view on cloud computing technologies, its position in and its relevance for Europe.

OpenNebula Cloud for HPC at NASA Ames

"Cloud Computing Architecture with Open Nebula – HPC Cloud Use Cases" is the title of the invited talk that I gave yesterday at the NASA Advanced Supercomputing Division – NASA Ames. I provided an overview of cloud computing architecture with OpenNebula, with special focus on cloud deployments for High Performance Computing environments. You can find bellow the slides and the abstract of the presentation. It was a great opportunity to interact with Piyush Mehrotra's group in the Computational Technologies Branch to discuss how OpenNebula can address their needs, and to talk with some of the people involved in the Nebula Project. Thanks for the invitation!.

Cloud Computing Architecture with Open Nebula - HPC Cloud Use Cases - NASA Ames 2012

OpenNebula is a fully open-source cloud management platform, with excellent performance and scalability to manage tens of thousands of virtual machines, and with the most advanced functionality for building virtualized enterprise data centers and private cloud infrastructures. OpenNebula is the result of many years of research and development in efficient and scalable management of virtual machines on large-scale distributed infrastructures. Its innovative features have been developed to address the requirements of business use cases from leading companies in the context of flagship international projects in cloud computing. OpenNebula is being used by many supercomputing and leading research centers to build HPC and science clouds for hosting virtualized computational environments, such as batch farms and computing clusters, or for providing users with new “HPC as a service” resource provisioning models. The talk describes how to design a cloud architecture with OpenNebula and its innovative features to enable the execution of flexible and elastic cluster and high performance computing services on demand while reducing the associated cost of building the datacenter infrastructure.

Advances in Clouds - Research in Future Cloud Computing

It has been more than two years since the European Commission published in January 2010 its pioneering report about the Future of Cloud Computing. A group of experts was established with the aim to evaluate the state-of-the-art and develop future research directions in cloud computing. Since then, there has been considerable advances in the field, developments have closed some gaps that were identified in this report, but more challenges have emerged.

We were re-convened by the European Commission in 2011 in order to capture these changes and maintain a state-of-the-art view on cloud computing technologies, its position in and its relevance for Europe. The experts, led by Keith Jeffery, Lutz Schubert and Maria Tsakali, have produced a final version of this report entitled Advances in Clouds - Research in Future Cloud Computing. The report brings valuable information for people defining Cloud Computing strategies, developing innovative research lines, or exploring emerging market opportunities beyond today’s Clouds. It is a must-read.

Roadmap on Infrastructures for e-Science in Europe

The last version of the Roadmap on Distributed Computing Infrastructure for e-Science and Beyond in Europe was released last week by the SIENA initiative. The main open-source cloud computing projects, like OpenNebula, and standards bodies, like DMTF, OASIS, OGF, ETSI, and SNIA, have contributed to this roadmap that assesses the situation, identifies issues, and makes recommendations regarding the adoption and evolution of open standards-based interoperable grid and cloud computing infrastructure (e-infrastructure) or to support research in Europe.

As member of its Editorial Board, I recommend to use this roadmap as a reference of the work in cloud standards being developed by the large number of standards bodies and other collaborative groups. I hope this roadmap is a first step to achieve a closer collaboration between them to avoid the existing situation where different working groups are covering the same functionality and needs. As it was pointed out in the Workshop Towards a Cloud Computing Strategy for Europe: Matching Supply and Demand organized at the 1st Digital Agenda Assembly, one of the main barriers to cloud computing adoption is interoperability and portability across cloud providers and products. This is needed to avoid vendor lock-in and create a healthy competitive cloud computing market in Europe.

The roadmap was presented at Cloudscape IV Advances in Interoperability and Cloud Computing Standards, the 23rd of February in Brussels, including presentations from key stakeholders. One very interesting conclusion of the workshop is how open-source is driving forward development and adoption of standards in cloud computing. Most of the initiatives in research infrastructures that were presented during the event are using OpenNebula as vendor-agnostic open platform for building and managing their cloud, and its interfaces are evolving into the standard in this area.

Key Challenges in Cloud Computing to Enable Future Internet of Things

"Key Challenges in Cloud Computing to Enable Future Internet of Things" is the title of the talk about future Internet of Things that I gave today January 19th at The 4th EU-Japan Symposium on New Generation Networks and Future Internet. The talk provides an architectural view of Internet of Things from the perspective of cloud computing, defines its main requirements for the underlying processing infrastructure, and describes the challenges in cloud computing that should be addressed to meet these requirements. The talk concludes with the description of the instruments that should be used to maximize the value of research and to support the collaboration between Japan and the European Union, namely openness, standards, coordination with running initiatives, and code re-use.

Industry Recommendations and Consultation Process to the European Cloud Strategy

The European Commission opened a consultation process in May 2011 to collect the opinions of individuals, businesses and public bodies across the continent, prior to the release of a European Cloud Computing Strategy in 2012. The online consultation closed in September, and the Commission has recently made public a report with its main results. Regarding innovation, respondents broadly agreed on the need for future research to improve current cloud computing commercial offerings.

A select industry group has prepared a report with key recommendations to the European Commission on the orientation of a Cloud computing strategy for Europe, proposing some actions for the European Commission and industry. The report presents 10 key recommendations and proposes some actions. They cover legal framework, market and technology related recommendations and actions. Main research and innovation recommendations are to build on the past and foster collaborative research in cloud computing, and to foster interoperability and data portability in the Cloud.

More information about the development of the EU-wide strategy on Cloud Computing is available at the European Commission site.

Critical Components of a Government Strategy to Accelerate Adoption of Cloud Computing

The adoption of cloud computing goes beyond cutting costs or transforming fixed costs into variable costs. Cloud is a powerful innovation and business transformation platform to increase productivity and competitiveness by supporting improved services and new business and service models.

However, for its complete adoption by industry, there are three challenges that, according to my experience, should be addressed by a Government Strategic Plan for Cloud Computing. Here we very briefly describe these main challenges and the actions that should be taken to help bridge these gaps.

• Technology Challenge. Funding agencies and big companies should support research on critical issues to create better clouds. Such research activity should generate innovative outcomes that can be quickly and easily transferred to the industry and the market. In this context, open-source is very important to create an open interoperable ecosystem, avoiding vendor lock-in, mostly at this initial stage of the market.
• Usability Challenge. Existing open standards for interoperability and portability should be promoted and used. The different governments should support the creation of a legal framework for international data protection and privacy. And existing cloud service and middleware providers should enhance their levels of trust and security.
  
• Cultural Challenge. User education, pilot projects and experimental testbeds are very important to help spread the benefits, expertise and knowledge of cloud computing. Public procurement should support the adoption of cloud, open standards and open source.
  

Cloud Innovation at Cloud Day 2011

Last Wednesday I had the pleasure to give a keynote about Challenges in Hybrid and Federated Cloud Computing at the first annual Cloud Day in Kista. The annual Cloud Days in Kista featured leading international and Swedish experts from industry and academia, who presented the cutting edge of cloud computing technologies. My presentation focused on how federated and hybrid clouds will play a significant role in IT strategies and e-infrastructures in the coming years, the different hybrid cloud computing scenarios, and the existing challenges for interoperability.

Organized by the Swedish Institute of Computer Science, the PDC Center for High Performance Computing, and the European Institute of Innovation and Technology under the enthusiastic leadership of Ake Edlund and Seif Haridi, the successful event also included keynotes by Michael Franklin (UC Berkeley) and Peter S Magnusson (Google). All the presentations are available for download.

Results of the Workshop Towards a Cloud Computing Strategy for Europe

The European Commission has published the results of the Workshop Towards a Cloud Computing Strategy for Europe: Matching Supply and Demand organized at the 1st Digital Agenda Assembly. The aim of the workshop was to identify the main elements of a European cloud strategy and the possible need for public-policy intervention, considering demand and supply side concerns. The results include two reports and one video. One of the reports is the below slide with the summary of the workshop presented in the plenary session. The second report is a more complete description of its purpose, context, discussions, actions and future steps.


The video is a very short interview with me, as speaker at the workshop, describing the challenges identified and the actions to address them.

The following is a transcript of that interview.

Cloud Computing will play a major role in tomorrow’s economy, first as innovation platform to increase productivity and competitiveness, and secondly as a new service industry that will provide many opportunities for European ICT companies. The workshop has helped identify the main areas of a European Cloud Strategy both to boost adoption of cloud computing and to lead cloud computing research and innovation. The workshop consisted of three different panels.

• The first panel has identified the different aspects that should be addressed to ensure that Europe can play a leading and active role both in using and in enabling cloud computing
• The second panel has explored the different opportunities for innovation and research to help create better clouds and to accelerate its adoption
• The third panel has discussed the needs of standards for interoperability, including a legal framework for international data protection and privacy