AWS re:Invent 2018 Recap – The Freedom to Build

Team YugaByte was at AWS re:Invent in Las Vegas last week. While AWS was announcing a flurry of new product releases and existing product updates, we had some excellent deep dive conversations at our booth on the future of transactional databases and how YugabyteDB is playing its part in shaping that future. This post summarizes our key learnings from the conference, which continues to set the record every year as world’s largest gathering of cloud platform engineers and executives. The big theme this year was “The Freedom to Build”.

We start with the database news and follow it up with other key announcements.

Aurora Remains the Fastest Growing AWS Service Ever

Amazon Aurora has come a long way since its GA in July 2015 and is now being used by tens of thousands of customers. AWS is clearly making the most of the sweet spot it has found in the database market where customers are willing to migrate away from their existing expensive Oracle DB and Microsoft SQL Server installations running on-premises.

Fear of Lock-in: Is Aurora the new Oracle?

At the same time, the fact that Aurora itself is another proprietary database with “Oracle” type of lock-in is starting to dawn on some of the smarter enterprises. Additionally, as transactional data volumes grow beyond what can be ingested and processed by a single monolithic Aurora node, customers are realizing that they are more future proof with a completely distributed SQL database. At YugaByte, we hear these lock-in and write scalability concerns often from AWS customers. Enter YSQL, YugabyteDB’s PostgreSQL-compatible Distributed SQL offering, that solves both these concerns at the root. YugabyteDB and its API implementations are fully open source and YSQL has built-in write scalability without compromising on the SQL needs of JOINs, integrity constraints and complex relationships.

DynamoDB Now Marketed for Business-Critical Apps

After 10+ years of evangelizing the benefits of eventual consistency and the overkill of ACID transactions in distributed databases, Amazon DynamoDB finally announced support for ACID transactions. As one can expect for an initial release, the offering is severely restrictive.

• Each transaction can include up to 10 unique items or up to 4 MB of data, including conditions.
• Available on only single-region DynamoDB tables.
• Transactions are disabled on global tables by default. You can choose to enable transactions on global tables by request, but replication across regions is asynchronous and eventually consistent. You may observe partially completed transactions during replication to other regions. Additionally, simultaneous writes to the same item in different regions are not guaranteed to be serially isolated.
• Using transactions will deplete your provisioned throughput faster and end up costing you more. DynamoDB performs two underlying reads or writes of every item in the transaction, one to prepare the transaction and one to commit the transaction.

Contrast the above design limitations with YugabyteDB’s Google Spanner-inspired globally consistent ACID transactions architecture. Kannan, our Co-Founder & CEO, summed it up well in a tweet.

In YugabyteDB, there are no artificial limits on how many items can be touched in a transaction. Transactions can be multi-region if you so desire — the only cost is wide-area network latency to commit the writes across the regions. For some workloads, such as user identity for a global SaaS service, this sort of global consistency is a must-have since writes are infrequent and immediate consistency is a must-have for high user satisfaction. For workloads that do not need such cross-region consistency, YugabyteDB also supports remote region read-replicas powered by asynchronous replication.

Managed Kafka & Envoy Offerings

How can there be a re:Invent without AWS announcing managed services for popular open source infrastructure? No surprises that AWS announced managed offerings for Apache Kafka, the de-facto distributed messaging framework, and a control plane for Envoy, the de-facto service mesh used to tie together microservices. Apache Kafka was originally developed at LinkedIn but Confluent is the primary commercial company behind it today. Envoy was originally developed at Lyft but does not have any commercial company behind it yet. Naturally, some of the original developers behind these open source technologies were not too happy with AWS simply monetizing their work without making any commensurate contributions back to the open source project. This debate is a replay of the one that we had a couple years back when AWS Elasticsearch service was launched. As we have previously highlighted in “Building a High Growth Business by Monetizing Open Source Software,” the only way to beat AWS or any competition in open source is to build a better product that customers cannot live without.

Serverless Frameworks or Containerized Microservices: Why Not Both?

AWS wants every compute-based workload on the planet and hence provides a full spectrum of options starting from EC2 VMs to serverless Lambda functions. In between, there’s ability to run containerized microservices and data services on EKS, their managed Kubernetes offering. If you are up for letting the containers get managed using AWS’s own proprietary (read non-Kubernetes) way then you have ECS and Fargate as two more options.

The most exciting news for the Serverless world was the ability to bring your own runtime to AWS Lambda. With this selection, the function must include (in its code or in a layer) an executable file (called bootstrap), responsible for the communication between your code (that can use any programming language) and the Lambda environment. Another important addition was the ability to reuse code across multiple functions using AWS Lambda Layers.

On the containerized microservices front, AWS took a big step forward in letting you run Lambda functions in your own containerized or VM environment with the open-sourcing of Firecracker, the underlying serverless compute that powers AWS Lambda & AWS Fargate.

Firecracker is a virtual machine manager that is responsible for launching, managing, and terminating tons of micro-VMs on a server. It’s well suited for serverless because it brings together the goodness of virtual machines (security, isolation) with the goodness of small and agile functions (speed, resource efficiency). Your Kubernetes cluster can now use Firecracker to start these micro-VMs and thereby bring you the benefit of serverless functions and containerized microservices on top of a common infrastructure and that too outside of AWS if you so desire.

Multi-Cloud and Hybrid Cloud Are Here to Stay

Microsoft Azure & Google Cloud Are Also Growing

While it may be tempting to conclude that AWS is the only game in town for public cloud services, that is certainly not the case. Microsoft Azure, Alibaba Cloud (active primarily in the Chinese market segment) and Google Cloud are chipping away slowly and steadily.

Enterprise IT is clearly hesitant to get locked into a single cloud provider in the long run. Having the flexibility to pick and choose a cloud provider on individual application characteristics makes more sense. After all, that’s the promise of the cloud — programmable infrastructure available in minutes! This approach also ensures that applications can be moved over to new cloud platforms when the feature-cost-performance tradeoff calculation changes in favor of the new platform.

Acknowledging Hybrid Cloud – Introducing AWS Outposts

It’s great to see AWS finally acknowledge that it can no longer ignore hybrid cloud architectures that make applications run across both public clouds and private on-premises datacenters with similar ease. AWS announced AWS Outposts which are fully managed and configurable compute and storage racks built with AWS-designed hardware that one can use to run AWS compute, storage and networking services in any private datacenter. A wide selection of current generation EC2 instances (C5, M5, R5) and storage options (EBS volumes, local instance storage, local disk options) are available for configuration.

To get started, you order your AWS Outposts from the AWS Management Console. AWS ships Outposts directly to the desired datacenter, where you can simply plug them into your local power and network. Once connected, you can view the newly added Outposts capacity in the AWS Management Console, and launch instances in these servers as part of your Amazon Virtual Private Cloud (VPC). There is another option to run the outposts on the VMWare Cloud using the VMWare control panel.

AWS Business is a Rocketship

The re:Invent conference presents an annual opportunity for AWS to highlight its market dominance. Today it has millions of active customers and upon closer inspection of the slide below, we can see that much of that growth has come in since 2015.

AWS became the preferred platform for startups when it launched its very first service (AWS S3) in 2006. Since 2015, the tides shifted to include even largest of the large enterprises. Today, we would be hard pressed to find a public company that doesn’t have a public cloud and AWS strategy.

Coming to actual revenue numbers, the above-mentioned enterprise adoption is at the heart of AWS’s extremely impressive $27B annual run rate and that too at a spectacular 46% YoY growth. If AWS were an independent technology company, then it would be clearly one of the most valuable on the market.

Summary

It’s exciting to see enterprises and startups alike participate in the cloud native revolution that is now fully underway. AWS has been a pioneer for a long time but it should not rest on its laurels alone. “Freedom to Build” does not mean build it on AWS using its proprietary services such as Aurora, DynamoDB, Lambda. Time and again we heard from AWS re:Invent attendees that they want to architect, develop and deploy modern distributed applications in a cloud neutral way. They want the flexibility to run the dev/test environments and CI/CD pipelines on say Kubernetes running on their internal VM infrastructure (which has a fixed cost that is mostly paid off). Thereafter, they want to be able to choose a public cloud platform at will depending on the cost-performance characteristics of the application rather than say that AWS is the only such platform. This approach of decoupling the actual infrastructure from modern distributed applications is at the heart of the cloud native revolution. It is the natural next step forward from the first decade of public cloud adoption which was simply about lifting and shifting existing monolithic applications to the rented hardware of the cloud.

As an open source, transactional, scale-out database with unparalleled data modeling freedom (key-value, flex-schema, relational), YugabyteDB is uniquely positioned to accelerate the cloud native revolution. If such a mission sounds exciting to you, join us and build the future with your own hands!

What’s Next?

• Compare YugabyteDB to databases like Amazon DynamoDB, Cassandra, MongoDB and Azure Cosmos DB.
• Get started with YugabyteDB on macOS, Linux, Docker and Kubernetes.
• Contact us to learn more about licensing, pricing or to schedule a technical overview.