Actual-time information streaming and occasion processing are essential parts of contemporary distributed methods architectures. Apache Kafka has emerged as a number one platform for constructing real-time information pipelines and enabling asynchronous communication between microservices and functions. Nonetheless, working and managing Kafka clusters at scale might be difficult, requiring specialised experience and vital operational overhead.
Amazon Managed Streaming for Apache Kafka (Amazon MSK) is a completely managed service that permits you to construct and run manufacturing Kafka functions. With Amazon MSK, you’ll be able to depend on AWS to deal with the heavy lifting of provisioning and managing Kafka clusters, when you give attention to constructing modern functions and real-time information processing pipelines.
On this put up, we discover how Fitch Group, one of many high credit standing firms, used Amazon MSK and Amazon MSK Replicator to attain multi-Area resiliency for his or her mission-critical Kafka infrastructure.
About Fitch Group and their want for multi-region resiliency
As a number one international monetary data providers supplier, Fitch Group delivers very important credit score and threat insights, strong information, and dynamic instruments to champion extra environment friendly, clear monetary markets. With workers in over 30 nations, Fitch Group’s tradition of credibility, independence, and transparency is embedded all through its construction, which incorporates Fitch Scores, one of many world’s high three credit score scores businesses, and Fitch Options, a number one supplier of insights, information, and analytics.
To remain aggressive and environment friendly within the fast-paced monetary business, Fitch Group strategically adopted an event-driven microservices structure. On the coronary heart of this ecosystem lies Kafka, particularly Amazon MSK, which serves because the spine for his or her information integration methods.
Fitch Group makes use of Kafka to allow functions to ship ratings-related enterprise occasions, facilitating automation inside their scores workflow methods and offering real-time or close to real-time processing. This architectural alternative has considerably lowered the time to marketplace for end-user-facing methods like Fitch Scores Professional and Fitch Group Scores web sites. Furthermore, Kafka’s strong capabilities enable for seamless aggregation and distribution of knowledge from many disparate methods by way of their information platform, enhancing information consistency, reliability, and accessibility throughout the group.
Given the essential position that Kafka performs in Fitch Group structure, offering strong catastrophe restoration (DR) mechanisms grew to become paramount. Any disruption to their Kafka infrastructure may have vital repercussions on their scores workflow automation, real-time processing, and end-user-facing methods, doubtlessly exposing Fitch Group to regulatory, monetary, and reputational dangers.
To attain the specified ranges of resiliency, Fitch Group had the next key necessities:
- Multi-Area deployment – Deploy MSK clusters throughout a number of AWS Areas to supply enterprise continuity and preserve service availability throughout Regional or service occasions
- Automated replication – Replicate Kafka information throughout Areas in close to actual time with minimal latency and information loss
- Constant subject namespaces – Keep the identical Kafka subject names and constructions throughout supply and vacation spot clusters to reduce software modifications
- Fast restoration – Within the occasion of a failover, allow functions to seamlessly begin consuming from the replicated cluster with minimal Restoration Time Goal (RTO) and Restoration Level Goal (RPO)
Answer overview
Fitch Group selected to implement their multi-Area Kafka deployment utilizing Amazon MSK and MSK Replicator. MSK Replicator is a completely managed replication service that permits steady, automated information replication between MSK clusters inside the identical Area or throughout completely different Areas. It helps replicating information between clusters with completely different configurations, together with various dealer counts, storage volumes, and Kafka variations. Right here’s how Fitch Group used MSK Replicator to attain their multi-Area resiliency objectives:
- Deployed MSK clusters in two separate Areas, with the first cluster in the principle Area and the secondary cluster in a distinct Area for catastrophe restoration
- Configured MSK Replicator to repeatedly replicate information from the first cluster to the secondary cluster, sustaining the identical subject names and constructions throughout each clusters
- Carried out software failover logic to robotically swap to consuming from the secondary cluster in case of a major cluster unavailability, with minimal restoration time and information loss
The next diagram illustrates this structure
Advantages achieved
By implementing Amazon MSK and MSK Replicator, Fitch Group realized a number of key advantages:
- Enhanced catastrophe restoration – The multi-Area deployment gives enterprise continuity even within the face of Regional or service occasions.
- Simplified operations – The managed functionality of MSK Replicator offloads the operational complexity of self-managing customized replication options, decreasing the burden on Fitch Group’s IT group
- Scalability – The answer can scale to deal with various information hundreds, ensuring that DR capabilities develop alongside enterprise wants
- Minimal software modifications – MSK Replicator helps replicating subjects with the identical identify, which eliminates the necessity for shopper software modifications, decreasing growth effort and potential errors
- Seamless failover and failback – Bidirectional replication capabilities allow fast switching of operations to the standby Area with minimal disruption, and simple reversion after the first Area is restored
- Improved testing capabilities – The setup facilitates common DR workout routines with out impacting manufacturing methods, permitting Fitch Group to validate their DR plans persistently
Conclusion
By utilizing Amazon MSK and MSK Replicator, Fitch Group has efficiently applied a extremely resilient and scalable Kafka infrastructure that meets their stringent enterprise continuity and catastrophe restoration necessities. This multi-Area deployment permits them to course of mission-critical monetary information at scale whereas offering minimal downtime and information loss within the occasion of service occasions or disasters. As Fitch Group continues to innovate and develop, their strong Kafka infrastructure gives a strong basis for future growth and the event of recent data-driven providers, finally enhancing their skill to ship well timed and correct monetary insights to their shoppers.
In regards to the authors
Kalyan Janaki is Senior Large Knowledge & Analytics Specialist with Amazon Internet Companies. He helps clients architect and construct extremely scalable, performant, and safe cloud-based options on AWS.
Venu Nemallikanti is the Enterprise Architect and Lead for Occasion Streaming at Fitch Group, a globally acknowledged monetary data providers supplier working in over 30 nations. His major duties embody overseeing the structure and implementation of occasion streaming options, making certain the seamless integration and efficiency of methods that ship credit score scores, analysis, information, and analytics to a worldwide clientele.
Chaitanya Shah is a Principal Technical Account Supervisor with AWS, primarily based out of New York. He likes to code and actively contributes to the AWS options labs to assist clients clear up complicated issues. He gives steerage to AWS clients on finest practices for his or her Cloud migrations. He’s additionally specialised in AWS information switch and the info and analytics area.
Oleg Chugaev is a Principal Options Architect and Serverless evangelist with 20+ years in IT, holding a number of AWS certifications. At AWS, he drives clients by way of their cloud transformation journeys by changing complicated challenges into actionable roadmaps for each technical and enterprise audiences.
