Building a Data-Centric Business Ready for Any Future—DSS Fireside Chat with Wells Fargo

Part of the Fireside Chat Series

October 6, 2022

At our recent Distributed SQL Summit, Nathaniel Drehmel, Principal Engineer, Database Strategy and Innovation at Wells Fargo, shared his experience in mapping and executing Wells Fargo’s database and innovation strategy plan with Yugabyte Co-Founder and CTO Karthik Ranganathan.

This wide-ranging discussion covered the financial services firm’s need for a resilient, secure, scalable, and high-performance data platform and how distributed SQL aligned to those needs and their overall innovation strategy.

Below is an excerpt of the key points covered during the conversation, or you can watch the full fireside chat below.

Wells Fargo Chat Highlights

Q: Let’s start with the question everyone asks, “Why distributed SQL?” What pain points or needs led you to distributed SQL?

It was an interesting start to the journey. I became aware of distributed SQL while we were charting Wells Fargo’s strategy. The concept was a bit nascent, but it was something we knew that we needed to start thinking about strategically.

Then came an inflection point that dramatically accelerated our strategy. We had a pretty significant resiliency issue in one of our data centers that caused a loss in some of our technology portfolios.

What manifested in the data layer was a need to conduct—at scale— failover-type events in our database estate. And when I say at scale, it was at pretty significant scale.

So, to step back a bit, at Wells Fargo we run a number of different database products, but certainly the prevalence is still towards a legacy, relational database estate. We were doing a lot of failovers, which required a database administrator or engineer to physically sit down and do them. We needed to get enough hands on keyboards quickly to get everything done.

But, something else became clear. In these types of disaster recovery scenarios, somebody has to make decisions to move needed workloads from Point A to Point B. So, even though we spent time doing it (in addition to rallying people and getting them to the right places), we spent more time trying to determine if we would do something. And this was due to the way that the legacy and relational database platforms were designed.

We also had (and still have) a portfolio of non-relational database compute platforms active in multiple locations. They survived this particular event very well. An active-active platform already deployed to multiple data centers using a singular topology proved very resilient.

What we took away from this was that we needed this type of setup, not just for our non-relational space but for our relational space. We thought there must be something in the industry—or something being built—that would give us that level of resiliency so that everything stays up in an event like this.

We needed a centrally managed topology that was simple to deploy and could give us a relational platform with ACID-compliant transaction support.

So what really accelerated our interest in distributed SQL? It came down to two things—resiliency and scalability. It was about ensuring a resilient, active-active solution that could support a relational data model and ACID-compliant transactions, along with the ability to achieve near linear horizontal scalability. That’s what we needed to bring to market.

Q: What were some of the things that became important as you looked at distributed SQL?

First, let me point out one more thing that was very important as we moved into the distributed SQL space. That was the idea of having a horizontally scalable system.

What we are all really in search of is the ability to scale in a near linear manner, so we can get more (or less) horsepower for our workloads as we need it. Within our estate, and I would imagine this holds true for most, there are some limits on what is readily available from commodity hardware in terms of a vertical limit.

Those limits constrict your database; even if you’re using single-node database solutions or things that can scale, you still have to do a lot to actually scale. So, we want to use distributed SQL to increase scalability, deliver a relational data model, and support ACID-compliant transactions.

It all comes down to resiliency and scalability, encompassed within an aspect of manageability, because neither will do us any good if everything is too complex to manage.

We also want to ensure that we are co-developing a solution or a roadmap because distributed SQL is relatively new. So, what should go into that? What should be prioritized? We are very clear on that. We need to manage risk in a very tight manner, and we need to ensure resiliency. So, being able to co-develop with an organization throughout our journey to bring features and functionality into our estate is key.

But, at the end of the day, what an organization like Wells Fargo is going to need is a platform that we can use, not for a niche use case, but for a significant portion of our estate. We’re looking for something that can be a relational database platform for us—the primary relational database platform, which is pretty significant. It can’t just work for two workloads. It’s got to work for 2000 workloads.

The litmus test is whether, for the majority of use cases, we can lift workloads from our legacy platform and bring them to this new platform, achieve the needed benefits, and do it all in a way that has relatively low friction. If it requires a complete rewrite of the application and the data layer, that is not low friction. We need to be able to move to a platform that can support the majority of our workloads.

So, why are we going into distributed SQL?

It’s not just for the newness of distributed SQL. We need an ACID-compliant, relational platform that will bring enhanced scalability and resiliency to market. It needs to support the majority of our workloads.

Other aspects we are looking at include whether or not it is open source, if it can follow our roadmap, and if it is well understood in the industry. Finally, it needs to have an interface that developers and our administrators are familiar with and can be productive with quickly.

Q: What are the things you’re looking forward to in the evolution of databases and database strategy?What are the things that you think about as you look to the future?

Let’s talk about two things briefly.

First, technology is not technology without people. When you talk about adopting a new technology within an enterprise like Wells Fargo, it is just as important to have the communities of people you’re asking to either design, develop on, or use that platform to be excited about doing so. The key word here is “community.” It’s about building a groundswell of support, which, I think, is often overlooked. We go into these situations and believe that we’re going to build a platform and everybody will just use it. Build it, and they will come, as they say.

A much more successful strategy, or one that I’ve seen executed successfully, is to go into it with a fundamental understanding of why we would use that platform. Then we need to talk about that and demonstrate its use throughout the organization hands-on. Everything comes down to building technology awareness, not just building out the technology. That is one thing that we look forward to doing a lot more of at Wells Fargo.

Second, (and I alluded to this before), is being able to use our distributed SQL platform as a general-purpose, ubiquitous platform within Wells Fargo.

Distributed SQL should just be how we do relational computing, and its success will be all about the database that rides on top of the underlying distributive SQL system. I think when we put those things together we start to look at a platform that is not only solid in terms of bringing distributed SQL to market, but has a commitment behind it to evolve as we go into the next three, six, twelve years, maturing it and growing its use.