Prioritizing Security, Stability, and Client Satisfaction: Insight from Fiserv’s VP and Fellow Architect
March 14, 2023
Dimitri Farafonov, VP and Fellow Architect at Fiserv, recently sat down with Karthik Rangatharan, CTO and Co-Founder of Yugabyte, to discuss his role at this top global fintech and payments company.
During their conversation, they discussed the priorities that guide Fiserv’s database selection process, along with the complexities involved in working to achieve low latency, data consistency, and massive bi-directional scalability, which are critical to Fiserv’s operations.
Below is an excerpt of the key points covered during the conversation. Or watch the full interview below.
Q: Can you provide more information about the role Fiserv plays in the financial industry? How does that impact how you approach your role and responsibilities?
We help banks, big and small, to efficiently operate their credit and debit businesses. To achieve this, our solution is multi-tenant, allowing us to process transactions for multiple clients simultaneously.
This adds another level of complexity. The banking industry is also heavily regulated, so we must comply with internal and external regulations, adding more complexity. We must follow numerous processes and protocols, which are constantly reviewed to ensure we meet the highest standards.
The sheer volume of transactions we handle is enormous, with our online issuing business alone processing up to 15,000 financial transactions per second. In addition, there are over a billion account cards on file. It’s just very large, with challenges that must be tackled on a massive scale.
As a result, our focus is on delivering solutions that are scalable, highly available, and equipped with disaster recovery and data replication capabilities. These are the challenges we primarily deal with on a daily basis.
Q: What are Fiserv’s priorities in terms of how you look at your market and how you work with vendors?
Our priorities guide our vendor selection process, with security being our top concern.
Stability follows closely behind, with a focus on ensuring our software solutions are reliable and robust. Client satisfaction is also a key factor, which actually deals with how fast and we can react to changes. So when we talk about vendor selection and how we work with Yugabyte, we always start with security.
For us, security and stability rank higher than innovation. We actually cut many “cutting edge” solutions from our selection process because they were not ready for primetime financial services applications. We only select the most secure and the most stable software to implement. For us, innovation usually entails taking very stable solutions and creating the right mix. We do a lot of integration work, and most of the movement we are making into real-time processing has everything to do with how we can react to changes quickly.
When it comes to databases, we distinguish between two categories: data capture and data aggregation. Data capture involves copying live transactions for audit purposes or to save for future processing, while data aggregation requires transactionality, such as updating account balances after a purchase So, when we make a database determination, again, we focus on our three priorities—security, production stability, and client satisfaction. Then we consider all the new demands of data capturing and data aggregation.
Q: You’ve mentioned that security is crucial, but would you take us through your evaluation process when considering different features of a database? Also, could you explain what brought you to YugabyteDB and which of the criteria it met during the selection process?
We work with multiple vendors. In fact, if you name a database, we probably have it. We ran open source Cassandra for years and have vendor-managed Cassandra instances as well. But let me take a step back. As you know, in traditional, on-prem development, infrastructure capacity is allocated upfront based on predictions for how many clients will ultimately need to be on-boarded for that product. You usually go with the best-case scenario because you want everyone to love the product. So, you allocate for maximum capacity, with enough physical servers to manage it, up front. This often results in unused databases sitting idle for years until enough clients are onboarded to justify their existence.
As we were moving to the cloud, our conversations with YugabyteDB and other vendors focused on starting small and growing the database cluster as needed, without upfront infrastructure capacity allocation. We didn’t want to buy 25 servers upfront. Our ask was to only pay for what we needed at that moment in time in terms of infrastructure, licensing, and compute allocations.
To meet both our internal high availability commitments and our customer SLAs, we mandate replication within the same region or data center. We also require cross-regional and cross-data center replication. So, we run in two regions, rather than three, but we replicate in each region. We triple replicate.
Our first YugabyteDB implementation was a reporting database. We used the CQL language with local quorum within a region and another copy in a different region, with eventual consistency for reporting. However, in our second implementation, we switched to SQL and the PostgreSQL client to ensure SQL transactionality and maintain records in a consistent state. We guarantee transactionality within one region with eventual consistency across the other region, and have set up preferred routes for disaster recovery events.
This approach, which we call “active, active prefer local,” allows for low latency transactionality in one region while maintaining a disaster recovery strategy in the other region. In this setup, one region is constantly processing, while the other is used for disaster recovery. Each region acts as primary and secondary for different clients. This enables us to achieve transactionality with low latency in one region while also having a disaster recovery strategy in the other.
Q: So you’re using both the Cassandra interface (CQL)and PostgreSQL interface to the Yugabyte database? And this is based on the applications’ needs?
Yes, that’s correct. However, we maintain two separate database clusters to cater to the specific needs of each application. One application focuses on data capture and stores it for end-of-day reporting. The other is a real-time API for a tracking system that involves lookups and insertions throughout the day.
Our approach entails utilizing SQL or CQL inserts for data capture and report generation while using transactional locking of SQL for standard PROD operations in the other application.
Q: What are some of your security requirements? How do they impact your migration to the cloud and your relationships with the vendors you work with?
We demand a lot when it comes to security, making it difficult for many vendors to work with us. We don’t compromise on security requirements.
It’s not solely about data access. Yes, there does have to be access levels, but we have very specific auditing requirements for those access levels. For example, we categorize every interaction within the database from both an application and administrative perspective. We audit every SQL command run. But the difficulty is not with the audit. The difficulty comes in terms of integrating it with an existing system, especially if that system is in a different region.
Because a different group manages the security integration, we must submit our audit data to them for evaluation. So, it’s a process.
When selecting vendors, we have a checklist of security requirements, including access levels, specific auditing requirements, encryption at rest, and various other security protocols that must be implemented. We do not compromise on security protocols, and we mandate vendors to implement specific ciphers and protocols. Encryption at rest is mandatory, even if we don’t store PCI data in the cloud.
We intentionally make it challenging to work with us because if we fail to post your check to your account, you will definitely notice. We take our impact on people’s lives seriously, and that’s why we’ve purposefully designed all of our processes. Security and stability are prioritized over cutting edge technology
–Dimitri Farafonov, VP and Fellow Architect at Fiserv
But once we select the vendor we’re not done. We continually submit enhancements to vendors, such as token authentication and user management integration. Choosing a database-as-a-service provider is challenging due to our long checklist of security requirements. Because of that, we have a goal to achieve platform independence by building a private cloud based on Kubernetes and bringing YugabyteDB on-premises while applying the same protocols and requirements as the public cloud.
Q: So tell me a little bit more about that.
Our goal is to achieve true platform independence, and we’re starting by constructing our own private cloud. We develop our applications to standards, utilizing a Java persistence API to interact with YugabyteDB without any reference to Yugabyte code.
Let me explain why.
Previously, we encountered challenges when developing applications for specific databases. As we expanded, we had to rewrite the code for different regions that didn’t support those databases. However, by coding to a standard, we can identify the most suitable PostgreSQL vendor for each region. That’s why we write applications for PostgreSQL and employ Yugabyte as the vendor that supports PostgreSQL for us. This approach enables us to replace YugabyteDB with other PostgreSQL vendors as required, thereby achieving the ultimate platform independence.
Q: So how has the relationship/partnership been between Fiserv and Yugabyte from a support standpoint? What is your confidence in deploying YugabyteDB into your mission-critical apps?
At Fiserv, we have a centralized procurement process for vendor onboarding. Once vendors meet security protocols and other requirements, we establish an enterprise relationship with them. We then implement the vendor’s services, making them available for anyone within Fiserv to use. We have an enterprise relationship with YugabyteDB, and it’s deployed across multiple lines of businesses here.
Now let’s chat for a moment about what it is like to work with us operationally, after the vendor has passed through the evaluation process. As we work with our vendors, we need a way to manage multiple clusters consistently, rather than managing them individually. This is because scaling and efficiency rely on sharing actual solutions.
Creating consistent operational models and databases that fit those models is very important because our distributed application DBAs manage multiple databases, and they work with the vendors specifically for their operational requirements.
This is why I highlighted the importance of having an SQL client that can interact with multiple SQL databases. YugabyteDB should be no different from any other SQL database that our teams work with.
Day 2 operations can be difficult enough, so having consistent operational models and databases that fit within those models are essential for scalability and efficiency. This is why Fiserv pushes all vendors towards it, furthering our reputation as being hard to work with. But for Fiserv to scale, we need consistency.
Security mandates drive consistency as well. We coordinate patching, both at the operating system and software levels, with all vendors following the same schedule. This can be challenging to manage operationally, especially when implementing solutions in different regions and clouds internationally.
Fiserv is trying to push for consistency and uses repeatable processes to solve problems once instead of solving them multiple times. It’s essential to code and implement to standards; otherwise, you’re faced with a nightmare to manage.
