Share This Page

Author: Abhideep Chakravarty |07/12/18

AWS Big Data Automation with DevOps


To see how easily we can do a small big data (sample) flow including DevOps

Tech Stack Used:

  1. Cloud Formation template
  2. Jenkins
  3. Spark Jobs written in Scala
  4. EMR
  5. Apache Kafka
  6. S3

Disclaimer: This architecture is minimalistic and just to show the ease of work with AWS, not the efficiency of the architecture

Solution Architecture:

AWS Big data DevOps


  1. With the help of CFT (Cloud Formation Template) it is super easy to setup the whole infrastructure in few minutes. This can be replicated as many as times we need.
  2. Once the infra is setup, running the EMR to process a job can be done from Jenkins.
  3. This is where the AWS IAM roles comes handy and makes the whole process secure.
  4. Fine grained IAM role given to the Jenkins can be tuned to access the EMR and S3 as per the needed.
  5. By configuring right profiles with AWS CLI using the IAM roles, we can ensure that Jenkins is able to put the JAR/Other files in S3 and also able to trigger the EMR command.

Cherry on the top:

AWS Bigdata DevOps Automation

AWS gives a wonderful tool – Data Pipelines. Using data pipelines, we can get rid of unused EMR costing. This is suitable for the batch processing. Like earlier, we can use the Jenkins, with correct profile and IAM set in, to send commands to the data pipeline and it ensure that the required EMR cluster is created, job is processed and then the EMR is decommissioned, to save cost.


Tooling: With sufficient tooling, it’s very easy to create and manage AWS environments using CFT. There are enough supports around AWS. Plugins are available for majorly used platforms like Jenkins, Eclipse, etc.

Security: With fine grained access control of AWS IAM, it makes the remote deployment possible and yet secure.

Cost: If our applications are written well to be fault tolerant, we can use the data pipelines and pre-emptible resources and get the work done around 80% cheaper cost.


Let's Talk About Your Needs

Thank you for your submission. We'll be in touch.