XC3

Xgrid Cloud Cost Control is a cloud agnostic and risk free package offering powered by Cloud Custodian that provides security enforcement, tagging, unused or invalid resources cleanup, account maintenance, cost control, and backups. It supports managing AWS public cloud environments and provides a visualization of usage of resources in account with support of managing resource utilization on a click. It spins up automation scripts and triggers lambdas to control cost of running resources in aws accounts and maintain state of each resource on which action performed having real-time visibility into who made what changes from where, enables us to detect misconfigurations and non-compliance. It supports rollback plans to prevent risks from materializing. Cloud Cost Control supports conditional policy execution. It generates reports, region vise and maintains state as well.

Check the below video for a quick demo of XC3.

Features

• One platform to track all your cloud resources be it cloud, multi-cloud, or hybrid infrastructure. It can track GCP, Azure, and AWS resources on a single UI.

• Enforces Tagging compliance that plays a vital role in determining the resources cost and many other aspects as well

• Provides Scheduled monitoring and alerting workflow that helps to track resource utilization and take action immediately.

• Provides cost optimization recommendation workflow without exposing your private information

XC3 System Architecture Visual Overview

XC3 has two architecture diagrams, representing its 'Dev' and 'Prod' environments.

XC3 Dev Architecture

This diagram illustrates the architecture of the "dev" environment for XC3. Below are the key components:

• EC2 Instance (Public Subnet): Acts as the entry point for the "dev" environment.
• Lambda Functions (Private Subnet): Executes serverless tasks within a secure private subnet.
• SQS (Simple Queue Service): Provides queuing capability for asynchronous tasks.
• SES (Simple Email Service): Handles email communications.
• Cost Explorer: Assists in analyzing and managing costs.
• Scheduled CloudWatch Events: Enables automated event triggering.
• S3 (Simple Storage Service): Used for storing state files and other data.
• Additional services (Push Gateway, Grafana, Prometheus, Cloud Custodian) run on the EC2 instance to monitor and manage the environment.

Access to the "dev" environment is primarily through the EC2 instance's IP address. SSH is available for administrative purposes.

XC3 Prod Architecture Diagram

This diagram illustrates the architecture of the "prod" environment for XC3. It includes the following components:

• Cognito: Manages user authentication and authorization.
• Route 53: Provides DNS routing services for efficient access.
• Elastic Load Balancer: Distributes incoming traffic to ensure high availability.
• EC2 Instance and Lambda Functions (Private Subnet): Similar to the "dev" environment but with additional security and scalability measures.
• SQS (Simple Queue Service): Handles queuing tasks.
• SES (Simple Email Service): Manages email communications.
• Cost Explorer: Assists in analyzing and managing costs.
• Scheduled CloudWatch Events: Enables automated event triggering.
• S3 (Simple Storage Service): Used for storing state files and other data.
• EIC Endpoint (Endpoint Isolation and Control): Enhances security and isolation within the "prod" environment.

Access to the "prod" environment is facilitated through a DNS URL, thanks to Route 53. This architecture prioritizes security, scalability, and high availability to support the production environment for XC3.

To start using XC3

Requirements

• Terraform 1.0+
• Python 3.9
• AWScli
• Cloud Custodian
• Prometheus/Grafana/Pushgateway
• checkov 2.0.574 or later
• shellcheck 0.7.1 or later

Pre-requisites

1. Clone GitHub repo git clone https://github.com/XgridInc/xc3.git

2. An AWS user with specific permission set user access.

   Refer the IAM Permission Set created in pre_requirement folder to setup XC3.

3. VPC needs to be present in the master account where you want to set up XC3

4. To store terraform state and to maintain lock, S3 bucket and dynamodb should be available in master account.

5. ACM certificate should be available. It will be associated with loadbalancer and domain.

6. The user has to enable CostExplorer by following the below link.

   https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/ce-enable.html

   Note: After enabling CE, it may take up to 24hours for AWS to start capturing your AWS account cost data, hence XC3 may not show the data until CE data is available in AWS account
   

Deployment

1. Clone the GitHub repository in your local computer to setup XC3 infrastructure.

   git clone https://github.com/XgridInc/xc3.git
   

2. Go to the directory xc3/ and configure the input.sh file and run the below command

   cd xc3/
   
       Note :
           - Configure the input.sh file in directory xc3/
   
              namespace="example"
              project="example"
              region="eu-west-1"
              allow_traffic="0.0.0.0/0"
              domain="" #  [Optional] - If you want to use your own domain then set this variable.
              account_id="123456789012"
              hosted_zone_id="Z053166920YP1STI0EK5X"
              owner_email="[email protected]"
              creator_email="[email protected]"
              ses_email_address="[email protected]"
              bucket_name="terraform-state-example"
   
           - Before running the below mentioned command:
   
   bash init.sh
   

3. Wait for few minutes before proceeding further for the application to come online. Verify the readiness of the metrics system. Load the Grafana URL in a browser. Live Grafana UI ensures the system is ready to accept and visualize metrics.

   Verify the readiness of metrics system by accessing Grafana UI: https://xc3.xxx.com/login

   Verify the readiness of metrics system by accessing Grafana UI: loadbalancer-dns. If Hosted zone ID is not provided in input.tfvars.

4. Now setup is complete. If domain is provided in the input.sh then users needs to be added in Cognito pool with requested role (admin/editor/viewer) in respective cognito group. User get random username/password from cognito then you can set password on domain by sign in using random credentials.

5. SSH into the private instance using EIC Endpoint to check if everything is working fine. Here replace [instance-id] needs to be replaced with ID

   ssh ubuntu@[instance-id] -i keypair.pem -o ProxyCommand='aws ec2-instance-connect open-tunnel --instance-id %h'

6. Go to AWS Systems Manager, select Parameter Store, and create a new parameter named "/{namespace}/region_names". Set the value as a dictionary with region IDs as keys and region names as values.

7. SSH into the private instance using EIC Endpoint to check if everything is working fine. Here replace [instance-id] needs to be replaced with ID

   ssh ubuntu@[instance-id] -i keypair.pem -o ProxyCommand='aws ec2-instance-connect open-tunnel --instance-id %h'

8. Now XC3 will run at 05:00AM UTC every day to generate data and populate Grafana. Few lambdas (Total Account Cost and Project spend) will run twice in a month.

   Note :
       1. If data is not available in Grafana UI then follow the troubleshooting guide at the last section of this page.
   

Troubleshooting Guide

case 1: If data is not showing into Grafana UI, there could be several reasons as shown below.

1. If AWS account was created freshly within last 24 hours then, you need to enable CostExplorer by following below link

   https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/ce-enable.html

2. If the AWS account was created freshly within the last 24 hours then, it may take up to 24 hours for the AWS team to generate cost information in your account. you may see below error in lambda logs in Cloudwatch

   [ERROR] DataUnavailableException: An error occurred (DataUnavailableException) when calling the GetCostAndUsage operation: Data is not available. Please try to adjust the time period. If just enabled Cost Explorer, data might not be ingested yet

3. XC3 Budget Detail/IAM Role/User Workflow lambda may have failed to execute , please check Cloudwatch logs to address the issue.

4. Check if XC3's most expensive services data is missing, and if so, verify the existence of the corresponding SSM parameter in AWS Systems Manager. To address this issue, ensure you follow step 5 of the deployment instructions.

case 2: user not able to change/update/modify default dashboards in Grafana UI

1. You can't change/update default dashboards.
2. If you need to make changes, please request for access for Editor/Admin role on

Contributor Guide

XC3 is a community-driven project; we welcome your contribution! For code contributions, please read our contribution guide.

• File a GitHub issue to report a bug or request a feature.
• Join our Slack for live conversations and quick questions.

RoadMap

We welcome feedback and suggestions from our community! Please feel free to create an issue or join our discussion forum to share your thoughts. For project updates, please read our roadmap guide.

License

XC3 is licensed under Apache License, Version 2.0.