Samsung Account is a worldwide account service that brings the Samsung universe collectively, from all Samsung companies to on-line and offline shops. It handles large-scale site visitors with safety and reliability. As a core Samsung service, all duties on Samsung Account, from common service deployments to cloud infrastructure upgrades, have to be carried out with out interruption to the service. This weblog introduces the structure designed for an Elastic Kubernetes Service improve and shares our expertise with upgrading the cloud infrastructure with out interruptions to the high-traffic Samsung Account service.
What’s Samsung Account?
Samsung Account is an account service that brings collectively greater than 60 companies and purposes in 256 international locations with over 1.7 billion consumer accounts. It’s used for Samsung Electronics companies together with Samsung Pay, SmartThings, and Samsung Well being, in addition to for authentication on numerous units reminiscent of cellular, wearable, TV, PC, and many others. Samsung Account helps ship a safe and dependable buyer expertise with one account on a wide range of contact factors from on-line shops (reminiscent of samsung.com) and offline shops to our buyer companies.
Evolution of Present Samsung Account Structure
Because the variety of consumer accounts and linked companies has grown, the infrastructure and repair of Samsung Account has additionally developed. It switched to the AWS-based cloud for service stability and effectivity in 2019, and is at the moment servicing 4 areas: 3 world areas (EU, US, AP) and China.
Presently, Samsung Account consists of greater than 70 microservices. In 2022, Samsung Account switched to the Kubernetes base so as to reliably help Microservices Structure (MSA). Kubernetes is an open-source orchestration platform that helps the straightforward deployment, scaling, and administration of containerized purposes. In 2023, Samsung Account bolstered catastrophe restoration (DR) to have the ability to present failover throughout world areas, and expanded the AP area to enhance consumer expertise.
In different phrases, Samsung Account has repeatedly developed its infrastructure and companies, and is at the moment operating stably with site visitors over 2.7 million requests per second (RPS) and over 200K DB transactions per second (TPS).
Every AWS-based Samsung Account area, with its personal digital non-public cloud, (VPC) is accessible via consumer units, server-to-server, or the net. Particularly, the net entry supplies a wide range of options reminiscent of samsung.com and TV QR login on AWS CloudFront, a Content material Supply Community (CDN).
Samsung Account microservices are being serviced on containers inside Elastic Kubernetes Service (EKS) clusters, and inner communication between areas makes use of VPC peering.
Samsung Account is utilizing a number of managed companies from AWS to ship numerous options. It’s utilizing Aurora, DynamoDB, and Managed Streaming for Apache Kafka (MSK) as storage to construct information sync between areas, and it supplies account companies primarily based on totally different managed companies together with ElastiCache, Pinpoint, and Easy Queue Service (SQS).
Let’s elaborate on the AWS Managed Providers that Samsung Account makes use of. The primary is EKS, which is a Kubernetes service for operating over 70 microservices on MSA. Subsequent, Aurora is used to avoid wasting and question information as an RDB and DynamoDB does the identical however as a NoSQL database. Together with them, ElastiCache (Redis OSS) is used to handle cache and periods and MSK handles delivering occasions from built-in companies and information sync. In the event you’re constructing an AWS-based service your self, you’d in all probability use these managed companies as properly.
Irritating Upgrades Contrasting the Comfort of Managed Providers
There’s a main problem to contemplate whenever you use these managed companies, although. Finish of help comes, on common, after 1.5 years for EKS and a pair of years for Aurora. Numerous different companies like ElastiCache and MSK face the identical drawback. Such service help termination is pure for AWS, however upgrading these companies when help ceases is usually a painful activity for these operating them. As a result of operation sources are sometimes diminished upon switching to the cloud, large-scale upgrades that come round each 1 or 2 years should be carried out with out sufficient sources for emergency response.
These managed service upgrades put a serious burden on Samsung Account. Greater than 60 built-in companies should be upgraded with out inflicting interruptions, and the upgrades have to be rolled out throughout a complete of 4 areas. On high of that, Samsung Account is creating and operating greater than 70 microservices, so a big quantity of help and cooperation from improvement groups is required. Probably the most difficult of all is that the upgrades should be carried out whereas coping with site visitors of over 2.7M RPS and DB site visitors of 200K TPS.
EKS Improve Sequence and Restrictions
You would possibly assume upgrading EKS on AWS is simple. Basically, when upgrading EKS, you begin with the management airplane together with etcd and the APIs that handle EKS. Afterwards, you progress to the info airplane the place the precise service pods are on, and at last to EKS add-ons. In principle, it’s doable to improve EKS following this sequence with none influence to the service operation.
Nevertheless, there are restrictions to common EKS upgrades. If an improve fails in any of the three steps above attributable to lacking EKS API specs or incompatibility points, a rollback will not be obtainable in any respect. As well as, it’s troublesome to do a compatibility verify for the companies and add-ons upfront.
Multi-cluster Structure for Non-disruptive EKS Upgrades
After a lot thought, Samsung Account determined to go along with a easy however dependable choice to carry out EKS upgrades. It is doable that many different companies are utilizing an analogous technique to improve EKS or run precise companies.
Samsung Account selected to improve EKS primarily based on a multi-cluster structure with 2 EKS clusters. The structure is constructed to allow an present EKS model to proceed offering the service, whereas a brand new EKS model on a separate cluster performs a compatibility validation with numerous microservices and add-ons earlier than receiving site visitors.
The benefit of this technique is which you can implement a rollback plan the place the previous EKS model takes over the site visitors if any points happen when switching to the brand new EKS model. A lesson we have now realized from offering the Samsung Account service below excessive site visitors is that there shall be points whenever you truly begin processing site visitors, regardless of how completely you have constructed your infrastructure or service. For these causes, it’s important to have a rollback plan in place everytime you deploy a service or improve your infrastructure.
If you carry out a multi-cluster improve, site visitors have to be switched between the previous and new EKS clusters. Merely put, there are 2 important approaches. One method is to change site visitors by inserting a proxy server between the two clusters. The opposite method is to change the goal IP utilizing DNS. Evidently, there could also be a wide range of different methods to perform this.
Within the first choice, utilizing a proxy server, you could encounter overload points when dealing with high-volume site visitors, reminiscent of with Samsung Account. Moreover, there are too many Software Load Balancers (ALBs) used for roughly 70 microservices, making it impractical to create a proxy server for every ALB.
Within the second choice, utilizing DNS, the precise consumer, shopper, and server exchange the service IP of the previous EKS with that of the brand new EKS throughout a DNS lookup, redirecting requests to a distinct goal on the consumer stage. The DNS choice doesn’t require a proxy server, and switching site visitors is simple by merely modifying the DNS document. Nevertheless, there’s a threat that the site visitors swap won’t occur instantly attributable to propagation-related delays with DNS.
The DNS-based site visitors swap structure was utilized to realize a non-disruptive EKS improve for Samsung Account.
Allow us to describe the DNS layers of Samsung Account with a hypothetical instance. The highest area is account.samsung.com, and there are 3 world area domains below it, categorised primarily based on latency or geolocation. For us.account.samsung.com, the layers are cut up into service.us-old-eks.a.s.com and repair.us-new-eks.a.s.com, representing the previous and new domains. This can be a easy, hypothetical instance. In actuality, Samsung Account makes use of extra DNS layers. Throughout the latest EKS improve, we switched site visitors between the inner domains of the two EKS clusters primarily based on weighted information whereas adjusting the ratio, slightly than switching . As an illustration, when a consumer sends a request to account.samsung.com, it goes via us.account.samsung.com, and the precise EKS service IP is utilized on the finish primarily based on the required weight.
Retrospective of the Non-disruptive EKS Improve
In abstract, I might say “it is a profitable improve if the linked companies have not seen.” With this EKS improve, we deployed and switched site visitors for a complete of three areas, 6 EKS clusters, and greater than 210 microservices over the course of 1 month. The site visitors swap was performed with ratios set primarily based on every service’s load and traits, and no points with linked companies have been reported throughout this one month EKS improve.
After all, as they are saying, “it is not over till it is over.” We did have a minor incident the place there have been inadequate inner IPs within the inner subnet attributable to many EKS nodes and repair pods turning into energetic concurrently, which scared us for a second. We secured the IP sources by lowering the variety of pods for Kubelet and add-ons by a couple of thousand and shortly scaling up the EKS nodes. One factor we realized whereas switching site visitors with DNS is that 99.9% of all the site visitors may be switched inside 5 minutes when the DNS weight is adjusted.
Closing Word
Richard Branson, co-founder of Virgin Group, as soon as stated, “You do not be taught to stroll by following guidelines. You be taught by doing, and by falling over.” Samsung Account has been rising and evolving, addressing many bumps alongside the way in which. We proceed to resolve numerous challenges with the soundness of our service because the precedence, conserving this “studying whereas falling over” spirit in thoughts. Thanks.
