This projects demonstrates usage of AWS API for collecting various types of usage data.
For deployment and additional information reference to the documentation.
- Amazon EventBridge rule invokes AWS Step Functions for every deployed data collection module based on schedule.
- The Step Function launches a AWS Lambda function Account Collector that assumes Read Role in the Management accounts and retrieves linked accounts list via AWS Organizations API.
- Step Functions launches Data Collection Lambda function for each collected Account.
- Each data collection module Lambda function assumes an IAM role in linked accounts and retrieves respective optimization data via AWS SDK for Python (Boto3). Retrieved data is aggregated in an Amazon S3 bucket.
- Once data is stored in the S3 bucket, Step Functions trigger an AWS Glue crawler which creates or updates the table in the AWS Glue Data Catalog.
- Collected data is visualized with the Cloud Intelligence Dashboards using Amazon QuickSight to get optimization recommendations and insights.
List of modules and objects collected:
| Module Name | AWS Services | Collected In | Details |
|---|---|---|---|
organization |
AWS Organizations | Management Accounts | |
budgets |
AWS Budgets | Linked Accounts | |
compute-optimizer |
AWS Compute Optimizer | Management Accounts | Requires Enablement of Compute Optimizer |
trusted-advisor |
AWS Trusted Advisor | Linked Accounts | Requires Business, Enterprise or On-Ramp Support Level |
support-cases |
AWS Support | Linked Accounts | Requires Business, Enterprise On-Ramp, or Enterprise Support plan |
cost-explorer-cost-anomaly |
AWS Cost Anomaly Detection | Management Accounts | |
cost-explorer-rightsizing |
AWS Cost Explorer | Management Accounts | DEPRECATED. Please use Data Exports for Cost Optimization Hub |
inventory |
Various services | Linked Accounts | Collects Amazon OpenSearch Domains, Amazon ElastiCache Clusters, RDS DB Instances, EBS Volumes, AMI, EC2 Instances, EBS Snapshot, RDS Snapshot, Lambda, RDS DB Clusters, EKS Clusters |
pricing |
Various services | Data Collection Account | Collects pricing for Amazon RDS, Amazon EC2, Amazon ElastiCache, AWS Lambda, Amazon OpenSearch, AWS Compute Savings Plan |
rds-usage |
Amazon RDS | Linked Accounts | Collects CloudWatch metrics for chargeback |
transit-gateway |
AWS Transit Gateway | Linked Accounts | Collects CloudWatch metrics for chargeback |
ecs-chargeback |
Amazon ECS | Linked Accounts | |
backup |
AWS Backup | Management Accounts | Collects Backup Restore and Copy Jobs. Requires activation of cross-account |
health-events |
AWS Health | Management Accounts | Collect AWS Health notifications via AWS Organizational view |
licence-manager |
AWS License Manager | Management Accounts | Collect Licenses and Grants |
aws-feeds |
N/A | Data Collection Account | Collects Blog posts and News Feeds |
quicksight |
Amazon QuickSight | Data Collection Account | Collects QuickSight User and Group information in the Data Collection Account only |
resilience-hub |
AWS Resilince Hub | Linked Accounts | |
marketplace |
AWS Marketplace | Linked Accounts | Collects AWS Marketplace data and terms |
reference |
Various services | Data Collection Account | Collects reference data for other modules and dashboard to function |
-
Deploy the Advanced Data Collection Permissions CloudFormation stack to Management (Payer) AWS Account. The Permissions CloudFormation stack in the Management (Payer) Account also deploys Permissions stacks to each of Linked accounts via StackSets.
-
Deploy the Data Collection Stack to the Data Collection AWS Account
For deployment and further information please reference to this documentation.
See CONTRIBUTING.md