SERVERLESS-DATA-PIPELINE
Using AWS Cloud Services Lambda, S3, Glue and Athena we are going to build a data pipeline written in python and deploy it using the Serverless Framework.
You can read the article here:
https://medium.com/@vincentclaes_43752/build-a-serverless-data-pipeline-on-aws-7c7d498d9707
Deploy and run the data pipeline
make sure you have correct user and roles defined:
Create a user to access AWS
create an admin user using the AWS console and set the credentials under a [serverless] section in the credentials file located in
~/.aws/credentials
a step by step guide can be found here: https://medium.com/@vincentclaes_43752/create-a-user-for-the-serverless-framework-8e5c336d47c7
Create the necessary roles
for our project we need two roles;
- one for lambda
- one for glue.
create a role with administrator access for both these roles and keep the ARN somewhere accessible. A step by step guide to creating a role can be found here: https://medium.com/@vincentclaes_43752/create-a-role-on-aws-for-the-serverless-framework-for-any-resource-c49712a5eee0
Replace the ARN of the lambda role and the ARN of the Glue role with the ones defined in the serverless.yml file. There is a comment above both roles that indicates that you should replace the ARN.
configure your local environment
git clone https://github.com/vincentclaes/serverless_data_pipeline_example.git
cd serverless_data_pipeline_example
under the root of this project execute:
export AWS_PROFILE="serverless"
pip install awscli
sudo npm install -g serverless
npm install serverless-s3-remover
to deploy the data pipeline execute:
# replace the `unique-identifier` with something unique
sls deploy --stage unique-identifier
to remove the data pipeline execute:
# replace the `unique-identifier` with something unique
sls remove --stage unique-identifier
to get more context please refer to the article: https://medium.com/@vincentclaes_43752/build-a-serverless-data-pipeline-on-aws-7c7d498d9707