Years back when I started my Cloud journey, my mentors Sachin Nayyar and Amit Saha, gave me my first task of setting up a working AWS environment for the organization. Me and my development team were fascinated by the ease with which we were able to spin up workloads, set up VPCs, created subnets and did our deployments. Fast forward that time to the next 3 months and I had my first set of learnings (ofcourse the hard way)
It is “easy” to spin workloads, get code deployed and stand-up a working environment in Cloud.
It is “easier” to create an “insecure environment” and “lose control” with “higher costs” in Cloud.
Of all my learnings and challenges, one of the biggest and most critical one was the use of privileged access (users having administrator or elevated access on AWS Objects) and monitoring the same in our AWS environments.
This blog post aims to share the challenges for privileged access management and monitoring in an AWS Ecosystem, our unique set of learnings and also the solutions which were designed to solve them
1. AWS Policies Inherent Design and Monitoring
Let’s face it, AWS policy design is inherently complex and it is a lot easier for anyone to choose a star (*) based action in a policy rather than defining a fine-grained actions based policy.
With *’s an AWS IAM Policy transforms to a high privileged policy thereby when attached to an IAM User or an IAM role making them a High Privileged AWS Entity.
Learning – Monitoring for privileged access in AWS requires an effective and robust mechanism to monitor the High Privileged IAM Policies across AWS Accounts
Resolution – Saviynt was in its early phase of providing security for IAAS providers like AWS. From the outset, we realized that in order to gain the trust of customers, we would focus on securing our own environment first. With this in mind, we started Saviynt’s deep integration with the AWS and AWS IAM framework. The platform parsed the IAM JSON based policies and started providing the visibility of high privileged policies across our AWS accounts, users and roles associated with such policies and also the set of users who have been creating such IAM policies. This helped us in launching a cleaning process of such policies and also educate the users creating such policies. Further, we designed a periodic attestation and certification of high privileged policies which ensured such policies do not get “accumulated” in our environment.
2. Privileged Access Assignment and Removal Process
Access to AWS objects is primarily provided by IAM Policies and Roles and the Access Assignment is long term with no lifecycle or approvals associated to it .
Access Assignments removal require intervention (human or via scripts) based on events and often lead to long term privileged access and residual privileged access (due to partial policy clean-ups).
Learning – Duration based access provisioning is critical for privileged access assignment
Resolution – We built an emergency access request module within Saviynt which has pre-defined Roles (These roles get mapped to AWS Roles at the time of deployment). Users can request for these roles for a limited duration(configurable) enabling them to get high privileged access in AWS for a limited duration only. The emergency request modules are also integrated with approval workflows and allowed the definition for creating custom high privileged roles. Post the expiration of allowed duration of the role, users assignment to the roles is automatically removed, without the need for any human intervention.
3. Monitoring Privilege Access on AWS Console using AWS Cloudtrail Logs is complex
Sifting and monitoring AWS Cloudtrail logs is never easy because of the sheer volume of logs. Don’t get me wrong, Cloudtrail Logs are great, but parsing them and deducing meaningful information out of them requires a solid feat of engineering. Adding further complexities would be aggregating activities across Multiple AWS Accounts. Above all filtering the high privileged events across AWS accounts and correlating the same adds more complexity.
Learning – Ingestion of Cloudtrail Logs to traditional data stores and searching them based on traditional search technologies does not scale well
Resolution – Saviynt provides a Cloudtrail Ingestion module based on pipeline architecture, making it scalable. The platform ingests Cloudtrail Logs from multiple AWS Accounts and indexes them with technologies like Elasticsearch. Further the searching and rendering of the logs via interactive dashboards is done using Kibana. The platform also supports Apache Spark and Scala are as alternative options to Elasticsearch and Kibana, which is a great option for organizations already invested on big data platforms.
4. Monitoring Privilege Access for AssumeRole operations (via Command Line) is extremely complex
As mentioned in Challenge No. 3, parsing CloudTrail logs is complex. However, with AssumeRole operations (Cross Account Access) monitoring privilege access becomes extremely complex. Let’s look at an example below where a user is trying to assume a privileged role from command line
AssumeRole operation via a command line would look something like
aws sts assume-role –role-arn “arn:aws:iam::xxxxxxxxxxxx:role/Saviynt-SaviyntSuperAdministrator-6UNP1H2JPILG” –role-session-name “test123”
This command would provide the user with the temporary access keys. Post this the user tries to perform a “Create User” operation. In Cloudtrail this event would look as shown below –
The role session name is now being logged as …/test123 in the logs making it difficult to correlate and tie it back to the original user who assumed this role.
Further the temporary access keys get rotated every 1 hour making the correlation extremely difficult.
Learning – Ingestion of Cloudtrail Logs for AssumedRole operations require “Stacked Corelation”, tying the changing access keys and role session names across multiple events across AWS accounts.
Resolution – Saviynt with its “Stacked Correlation” methodology can effectively monitor the AssumedRole AWS API operations via Command Line. The methodology does not rely on the user’s free form input of rolesessionname thereby making the correlation accurate and effective, which by merely parsing CloudTrail logs is not feasible.
As we continue to learn and evolve , our challenges, learnings and solutions evolve too. I look forward to share the same and learn from you.