ECS + Fargate
ECS (Elastic Container Service)
Amazon ECS allows running Docker containers in a standardized, AWS-optimized environment. The containers can contain any code or application module written in any language.
Rather than being handled by AWS, scaling and server management is set up by the user, either manually or using AWS tools, such as auto-scaling groups. The containers themselves run on standard Amazon EC2 instances that are configured with a special Amazon ECS agent running on them. These underlying Amazon EC2 instances within an individual cluster of servers can be of any size or quantity, depending on your application’s scaling needs. Via the Amazon ECS software, configuration and management of the underlying cluster is used to determine where, how many, and how each container is to execute on the given cluster. The Amazon EC2 instances in the cluster must be sized and scaled by the user to handle the quantity and execution demands of the containers.
Fargate
AWS Fargate is a technology that provides on-demand, right-sized compute capacity for containers. With AWS Fargate, you no longer have to provision, configure, or scale groups of virtual machines to run containers. This removes the need to choose server types, decide when to scale your node groups, or optimize cluster packing.
Comparison between Fargate and Lambda
Fargate is a good choice for consistent workloads or applications that want to use docker generally.
Good use cases for Lambda include unpredictable or inconsistent workloads and applications that can be easily expressed as a single function with predictable resource usage on each invocation.
Both Fargate and Lambda are serverless tools, but are inherently different in functionality and solutions they provide. For this project, it seems that Fargate might be a better fit, because of the consistent workload required by the data platform and the delays that might be caused by Lambda’s cold functions.
Pros
More flexibility than Lambda - we can size and scale the fleet of EC2 instances ourselves, depending on the requirements of the data platform
Slightly easier to monitor than Lambda, as Lambda does not have any underlying server to run monitoring agents on.
Better performance, as it runs on more dedicated resources
Cons
Requires significantly more setup - storing, caching, etc need to be managed separately.
Lambda is much easier to use for spinning up more architectures
External reference: https://www.bluematador.com/blog/serverless-in-aws-lambda-vs-fargate
Created from: Project overview 202007010930
uid: 202007011310 tags: #amazon