Warning:
This blogpost has been posted over two years ago. That is a long time in development-world! The story here may
not be relevant, complete or secure. Code might not be complete or obsoleted, and even my current vision might
have (completely) changed on the subject. So please do read further, but use it with caution.
Posted on 20 May 2020
Tagged with:
[ aws ]
[ amazon ]
More often than not, I’m using Amazon Web Services (AWS) as my “cloud”. Not only for my own projects, but almost all customers I’m working for use Amazon for hosting their applications. So over time you build up a lot of experience on AWS service: you know how to (correctly) setup VPC’s, know when to you ECS, EC2 or lambda to host code and even services like S3, SNS and SQS pose no challenges anymore.
But there are a lot of AWS services available. And I do mean: a LOT. Currently, there are 163 (!) different services that are available from the Amazon Dashboard, each with their own way of working, difficulties, catches and best practises.
Discovering AWS
You might realise that it’s probably near impossible to dive into each service and completely understand how they work and most likely, you don’t really need to know the exact ins and outs. But, having a basic understanding on each service can be a major benefit as a developer, architect or administrator. It makes it easier to see if there is an already existing solution for your
problem at hand.
So, I dove into each and every service to figure out what it exactly was for and how it works in the basics. I tried to experiment with as many components as possible (time and money permitting, I didn’t want to spend 15.000$ on AWS Data Exchange). I tried to capture what the service does in a single one-liner to give you a global overview.
I think most of them are correct enough but if you have any suggestions or corrections, please tell me!
| Compute |
|
| EC2 |
Virtual Private Servers |
| Lightsail |
Amazon’s hosting provider (vps, dns, storage) |
| Lambda |
Functions you can run, written in Python, NodeJS, Go etc. Can run many in parallel. |
| Batch |
Run software jobs in Docker containers on EC2 machines |
| Elastic Beanstalk |
Run software on managed virtual machines |
| Serverless Application Repository |
Repository of serverless applications that you can deploy (on lambda) |
| AWS Outposts |
Run Amazon services in your own data center |
| EC2 Image Builder |
Create EC2 (ami?) images automatically |
| Storage |
|
| S3 |
File / object storage. Not primarily used for mounting as filesystem, but you can directly download files through HTTP. |
| EFS |
NFS. Mount network disks to your machines. |
| FSx |
Windows / Lustre filesystems you can connect to your ec2 machines |
| S3 Glacier |
Low cost storage system for backups and archives and such |
| Storage Gateway |
iSCSI so you can connect s3 to your own (remote) machine. |
| AWS Backup |
Automatically create backups of different AWS service (ec2, rds etc) |
| Database |
|
| RDS |
Managed mysql, postgres databases etc. |
| DynamoDB |
Large & scalable non-relational database |
| ElastiCache |
Managed memcache and redis machines |
| Neptune |
Graph database |
| Amazon Redshift |
Warehousing. Store lots of data that can be processed through streams. |
| Amazon QLDB |
Database for immutable and cryptographically verifiable data (money transactions etc) |
| Amazon DocumentDB |
MongoDB clone (but not really compatible anymore) |
| Amazon Keyspaces |
Managed Apache Cassandra clone |
| Migration & Transfer |
|
| AS Migration Hub |
Migrate things from your DC to AWS |
| Application Discovery Service |
Discover services in your datacenter |
| Database Migration Service |
Migrate databases to RDS while staying online (can convert structures as well) |
| Server Migration Service |
Migrate virtual machines to amazon. |
| AWS Transfer Family |
(s)FTP service with S3 backend. Upload to FTP, directly store on S3 bucket. |
| Snowball |
Get a machine from AWS, plug in your DC, transfer data fast to AWS, return machine |
| DataSync |
Sync data between your datacenter and AWS |
| Networking & Content Delivery |
|
| VPC |
Create your own virtual private network within AWS. |
| CloudFront |
Content Delivery Network. |
| Route 53 |
Manage domain names and records. |
| API Gateway |
Create HTTP APIs and let them connect to different backends. |
| Direct Connect |
Create a (physical) connection between you (or DC) to AWS. |
| AWS App Mesh |
Automatically run Envoy as a sidecar for your containers (ECS or EKS). |
| AWS Cloud Map |
Service discovery for your containers. |
| Global Accelerator |
Run your app on edge locations so they are closer to your customers (CDN for apps). |
| Developer Tools |
|
| CodeStar |
Quickly develop applications by using template code and codecommit, codebuild etc |
| CodeCommit |
Amazon source repositories (git repo’s etc) |
| CodeBuild |
CI service |
| CodeDeploy |
Deployment service |
| CodePipeline |
Code delivery with workflows |
| Cloud9 |
Online IDE |
| X-Ray |
Allows tracing in your applications, supports Python, NodeJs, Go etc. |
| Robotics |
|
| AWS RoboMaker |
Cloud solution for robotic developers to simulate, test and securely deploy robotic applications |
| Customer Enablement |
|
| AWS IQ |
Job board: Hire AWS experts for whatever you need. |
| Support |
AWS support center |
| Managed Services |
Let AWS handle your AWS services for you. |
| Blockchain |
|
| Amazon Managed Blockchain |
Block chains |
| Satellite |
|
| Ground Station |
Timeshare radios and large antennas pointed at space |
| Quantum Technologies |
|
| Amazon Braket |
Some quantum thing. It’s in preview so I have no idea what it is. |
| Management & Governance |
|
| AWS Organizations |
Configure (sub)organisations and accounts |
| CloudWatch |
Logging from various AWS components |
| AWS Auto Scaling |
Scale resources based on your custom inputs and rules |
| CloudFormation |
Templates to create and configure AWS components (think terraform/sls) |
| CloudTrail |
Figure out who did what in your AWS services |
| Config |
Audit the configurations of your AWS resources |
| OpsWorks |
Use Ansible to automate stuff |
| Service Catalog |
Manage list of items/codes etc you have in the cloud |
| Systems Manager |
View data from your resources grouped in ways you like (like application specific etc) |
| AWS AppConfig |
Store and publish application configuration data |
| Trusted Advisor |
Checks your account for issues (costs, performance, security etc) |
| Control Tower |
Manage multi-accounts |
| AWS License Manager |
Manage licenses |
| AWS Well-Architected Tool |
Generate questionnaires about your architecture to see if you follow best practices |
| Personal Health Dashboard |
StatusPage for AWS |
| AWS Chatbot |
Connect AWS to slack |
| Launch Wizard |
Deploy MSSQL or SAP |
| AWS Compute Optimizer |
Finds your resources and advices on how to save costs |
| Media Services |
|
| Elastic Transcoder |
Encode files from S3 into different other formats and store back at S3 |
| Kinesis Video Streams |
Capture media streams |
| MediaConnect |
? |
| MediaConvert |
Convert media into different formats |
| MediaLive |
Share live video with many others |
| MediaPackage |
? |
| MediaStore |
? |
| MediaTailor |
Insert advertisements into your broadcasts |
| Elemental Appliances & Software |
create videos on-premise. Basically a mix of all of the above services. Seems expensive. Probably is. |
| Machine Learning |
|
| Amazon SageMaker |
Machine learning tools |
| Amazon CodeGuru |
Profile java code with machine learning |
| Amazon Comprehend |
Understand and classify data like emails, tweets etc |
| Amazon Forecast |
Create forecasts from data |
| Amazon Fraud Detector |
in preview so no idea. |
| Amazon Kendra |
Search service where you can ask questions |
| Amazon Lex |
Create voice and chatbots |
| Amazon Machine Learning |
Deprecated. Use SageMaker instead. |
| Amazon Personalize |
Create personalized recommendations based on data (mahout??) |
| Amazon Polly |
Convert text to speech in different languages |
| Amazon Rekognition |
Recognize objects and people in images |
| Amazon Textract |
Convert text found in images to text (OCR) |
| Amazon Transcribe |
Convert audio to text |
| Amazon Translate |
Translates text from one language to another |
| AWS DeepLens |
A video camera that does machine learning |
| AWS DeepRacer |
Some kind of game where you program a racecar to race against others. |
| Amazon Augmented AI |
Let humans in the loop to make AI learn things better |
| AWS DeepComposer |
Computer generated music. It’s as horrible as it sounds. |
| Analytics |
|
| Athena |
Query data stored in s3 buckets. |
| EMR |
Elastic Map/Reduce |
| CloudSearch |
AWS version of managed document search system (like elasticsearch) |
| Elasticsearch Service |
Elasticsearch as a service |
| Kinesis |
Collect massive amount of data so you can do analytics (like ELK?) |
| QuickSight |
Business Intelligence service |
| Data Pipeline |
Move and transform data to dynamodb, rds, s3 etc. |
| AWS Data Exchange |
Find APIs which data you can consume, which can be very expensive |
| AWS Glue |
ETL service. Enrich, validate data. |
| AWS Lake Formation |
Create data lakes |
| MSK |
Kafka as a service |
| Security, Identity, & Compliance |
|
| IAM |
AWS’s permission system that can control users and AWS services. |
| Resource Access Manager |
Share certain AWS resources like Route53, licenses, ec2 with other accounts. |
| Cognito |
User and password management system. Useful for managing users for your applications. |
| Secrets Manager |
Secrets key/value store. Can automatically rotate secrets. |
| GuardDuty |
Automatically scan your cloudtrail/vpc logs for threats. |
| Inspector |
Automatically find (security) issues in your network and machines. |
| Amazon Macie |
Analyzes data in your S3 buckets and check for PII data. |
| AWS Single Sign-On |
Allow single-sign on to your applications. |
| Certificate Manager |
Manage and even create (free) SSL certificates. |
| Key Management Service |
Manage secret keys |
| CloudHSM |
Hardware security modules. Allows you to generate and operate on cryptographic keys. |
| Directory Service |
Active directory as a service |
| WAF & Shield |
Web Application Firewall (for loadbalancers, cloudfront, api gateway). Can setup your own rules or use predefined ones |
| AWS Firewall Manager |
Firewall manager for different accounts in your organisation |
| Artifact |
Documents for cloud compliance (things like 27001 certification etc) |
| Security Hub |
Overall security checker that uses guardduty, inspector, macie etc |
| Detective |
Log security issues found (from security hub etc) |
| Mobile |
|
| AWS Amplify |
Let AWS automatically generate frontend & backend apps and deploy them automatically. |
| Mobile Hub |
Part of AWS Amplify now. |
| AWS AppSync |
Create API backends that you can connect to. Can be created through AWS Amplify as well. |
| Device Farm |
AWS BrowserStack. Automatically test apps on many different mobile devices and browsers. |
| AR & VR |
|
| Amazon Sumerian |
3D Editor and Engine for rapidly prototyping AR/VR and 3D experiences |
| Application Integration |
|
| Step Functions |
State machines written in amazon’s own language |
| Amazon AppFlow |
Automatically connects apps together (zapier?). For instance: slack to s3 buckets. |
| Amazon EventBridge |
Some kind of eventbus system |
| Amazon MQ |
ActiveMQ |
| Simple Notification Service |
Notification system that can notify through email, api endpoints, sms etc. |
| Simple Queue Service |
Message queue system |
| SWF |
Create workflows. |
| AWS Cost Management |
|
| AWS Cost Explorer |
Gives an overview and projection of your budgets |
| AWS Budgets |
Create budgets for your AWS components |
| AWS Marketplace Subscriptions |
Find (and buy) AMI’s with software installed |
| Customer Engagement |
|
| Amazon Connect |
AWS call center platform |
| Pinpoint |
Create transactional emails, SMS or voice calls based on templates. |
| Simple Email Service |
Send out emails. Email provider. |
| Business Applications |
|
| Alexa for Business |
Connect Alexa to your business needs. |
| Amazon Chime |
AWS version of Zoom. |
| WorkMail |
AWS version of Gmail / Calendar. |
| End User Computing |
|
| WorkSpaces |
Virtual desktops from Windows or Linux. |
| AppStream 2.0 |
Stream applications running native onto your browser |
| WorkDocs |
Store your documents and manage them online. |
| WorkLink |
Connect mobile users to your intranet. |
| Internet Of Things |
|
| IoT Core |
Manage fleets of IOT devices through MQTT broker |
| FreeRTOS |
RTOS operating system for microcontrollers to automatically connect to IOT-Core or greengrass. |
| IoT 1-Click |
Manage 1-click buttons that can be connected to other systems like Lambda |
| IoT Analytics |
Clean up and save messages from topics into a data-store for analytics |
| IoT Device Defender |
Detect unwanted issues on your devices and take actions |
| IoT Device Management |
Organize IoT devices into groups, schedule jobs on the devices and configure remote access |
| IoT Events |
Monitor telemetry from devices and then trigger other AWS services or jobs on the devices themselves |
| IoT Greengrass |
A message broker can buffer messages for groups of up to 200 devices which can communicate and process data locally if connectivity to IoT Core is intermittent. |
| IoT SiteWise |
Collect, organize, analyze and visualize data from industrial equipment at scale |
| IoT Things Graph |
Cloudformation-like designer for graphing how devices should communicate with other AWS services |
| Game Development |
|
| Amazon GameLift |
Deploy game servers with low latency on AWS |
| Containers |
|
| Elastic Container Registry |
Store docker images like on DockerHub |
| Elastic Container Service |
Run containers, either on your own EC2 machines, or on managed machines called Fargate. |
| Elastic Kubernetes Service |
Kubernetes as a service |
Thanks to Brian Thomas Smith for filling in the blanks on IoT.
Thanks to all the others from #HN who suggested changes and updates on the different services.
Conclusion
With over 150 services running on Amazon AWS, it’s near impossible to be an expert on all of them. And that’s ok: when you deal with large EKS or ECS clusters, changes are you never will touch the IOT services for instances. I found that most services are pretty well explained and easy to start with.
One of the biggest issues are the IOT services: since I have no experience with smart devices, MQTT or IOT in general, i finally managed to get a simple GO application to connect to IOT-core, and connect different rules, pipelines, analytics etc to it. But even still it’s very unclear what the different services actually do. Hopefully there will be AWS IOT experts out there that can
enlighten me.
Posted on 20 May 2020