Intro to AWS and Boto3 IN TRODUCTION TO AW S BOTO IN P YTH ON - - PowerPoint PPT Presentation

▶
intro to aws and boto3
SMART_READER_LITE
LIVE PREVIEW

Intro to AWS and Boto3 IN TRODUCTION TO AW S BOTO IN P YTH ON - - PowerPoint PPT Presentation

Feb 28, 2023 436 likes •990 views

Intro to AWS and Boto3 IN TRODUCTION TO AW S BOTO IN P YTH ON Maksim Pecherskiy Data Engineer What is Amazon Web Services? Our Home Our Data Project INTRODUCTION TO AWS BOTO IN PYTHON What is Boto3? import boto3 s3 = boto3.client('s3',

slide-1
SLIDE 1

Intro to AWS and Boto3

IN TRODUCTION TO AW S BOTO IN P YTH ON

Maksim Pecherskiy

Data Engineer

slide-2
SLIDE 2

INTRODUCTION TO AWS BOTO IN PYTHON

What is Amazon Web Services?

Our Home Our Data Project

slide-3
SLIDE 3

INTRODUCTION TO AWS BOTO IN PYTHON

What is Boto3?

import boto3 s3 = boto3.client('s3', region_name='us-east-1', aws_access_key_id=AWS_KEY_ID, aws_secret_access_key=AWS_SECRET) response = s3.list_buckets()

slide-4
SLIDE 4

INTRODUCTION TO AWS BOTO IN PYTHON

AWS console

slide-5
SLIDE 5

INTRODUCTION TO AWS BOTO IN PYTHON

Creating keys with IAM.

slide-6
SLIDE 6

INTRODUCTION TO AWS BOTO IN PYTHON

Creating keys with IAM

slide-7
SLIDE 7

INTRODUCTION TO AWS BOTO IN PYTHON

slide-8
SLIDE 8

INTRODUCTION TO AWS BOTO IN PYTHON

slide-9
SLIDE 9

INTRODUCTION TO AWS BOTO IN PYTHON

slide-10
SLIDE 10

INTRODUCTION TO AWS BOTO IN PYTHON

slide-11
SLIDE 11

INTRODUCTION TO AWS BOTO IN PYTHON

slide-12
SLIDE 12

INTRODUCTION TO AWS BOTO IN PYTHON

slide-13
SLIDE 13

INTRODUCTION TO AWS BOTO IN PYTHON

slide-14
SLIDE 14

INTRODUCTION TO AWS BOTO IN PYTHON

AWS services

slide-15
SLIDE 15

INTRODUCTION TO AWS BOTO IN PYTHON

AWS services

slide-16
SLIDE 16

INTRODUCTION TO AWS BOTO IN PYTHON

AWS services

slide-17
SLIDE 17

INTRODUCTION TO AWS BOTO IN PYTHON

AWS services

slide-18
SLIDE 18

INTRODUCTION TO AWS BOTO IN PYTHON

AWS services

slide-19
SLIDE 19

INTRODUCTION TO AWS BOTO IN PYTHON

Sam

slide-20
SLIDE 20

INTRODUCTION TO AWS BOTO IN PYTHON

GetItDone

https://data.sandiego.gov/datasets/get it done 311/

1 2 3 4

slide-21
SLIDE 21

INTRODUCTION TO AWS BOTO IN PYTHON

Summary

AWS Services = Home Utilities IAM, S3, SNS, Comprehend and Rekognition AWS Key / Secret Connecting to S3 Using Boto

import boto3 s3 = boto3.client('s3', region_name='us-east-1', aws_access_key_id=AWS_KEY_ID, aws_secret_access_key=AWS_SECRET) response = s3.list_buckets()

slide-22
SLIDE 22

Let's harness the cloud!

IN TRODUCTION TO AW S BOTO IN P YTH ON

slide-23
SLIDE 23

Diving into buckets

IN TRODUCTION TO AW S BOTO IN P YTH ON

Maksim Pecherskiy

Data Engineer

slide-24
SLIDE 24

INTRODUCTION TO AWS BOTO IN PYTHON

S3 Components - Buckets

Desktop folders Own permission policy Website storage Generate logs

slide-25
SLIDE 25

INTRODUCTION TO AWS BOTO IN PYTHON

S3 Components - Objects

slide-26
SLIDE 26

INTRODUCTION TO AWS BOTO IN PYTHON

What can we do with buckets?

Create Bucket List Buckets Delete Bucket

slide-27
SLIDE 27

INTRODUCTION TO AWS BOTO IN PYTHON

Creating a Bucket

Create boto3 client

import boto3 s3 = boto3.client('s3', region_name='us-east-1', aws_access_key_id=AWS_KEY_ID, aws_secret_access_key=AWS_SECRET)

Create bucket!

bucket = s3.create_bucket(Bucket='gid-requests')

slide-28
SLIDE 28

INTRODUCTION TO AWS BOTO IN PYTHON

Bang!

slide-29
SLIDE 29

INTRODUCTION TO AWS BOTO IN PYTHON

Our bucket in the console

slide-30
SLIDE 30

INTRODUCTION TO AWS BOTO IN PYTHON

Listing buckets

Create boto3 client

import boto3 s3 = boto3.client('s3', region_name='us-east-1', aws_access_key_id=AWS_KEY_ID, aws_secret_access_key=AWS_SECRET)

List Buckets

bucket_response = s3.list_buckets()

slide-31
SLIDE 31

INTRODUCTION TO AWS BOTO IN PYTHON

Listing Buckets

Get Buckets Dictionary

buckets = bucket_response['Buckets'] print(buckets)

slide-32
SLIDE 32

INTRODUCTION TO AWS BOTO IN PYTHON

Listing Buckets

slide-33
SLIDE 33

INTRODUCTION TO AWS BOTO IN PYTHON

Deleting buckets

Create boto3 client

import boto3 s3 = boto3.client('s3', region_name='us-east-1', aws_access_key_id=AWS_KEY_ID, aws_secret_access_key=AWS_SECRET)

Delete Bucket

response = s3.delete_bucket('gid-requests')

slide-34
SLIDE 34

INTRODUCTION TO AWS BOTO IN PYTHON

Bye Bye Bucket

slide-35
SLIDE 35

INTRODUCTION TO AWS BOTO IN PYTHON

Bye Bye Bucket

slide-36
SLIDE 36

INTRODUCTION TO AWS BOTO IN PYTHON

Other operations

slide-37
SLIDE 37

INTRODUCTION TO AWS BOTO IN PYTHON

Summary

s3.create_bucket(Bucket='buck') s3.list_buckets() s3.delete_bucket(Bucket='buck')

slide-38
SLIDE 38

Let's practice!

IN TRODUCTION TO AW S BOTO IN P YTH ON

slide-39
SLIDE 39

Uploading and retrieving les

IN TRODUCTION TO AW S BOTO IN P YTH ON

Maksim Pecherskiy

Data engineer

slide-40
SLIDE 40

INTRODUCTION TO AWS BOTO IN PYTHON

Buckets and objects

slide-41
SLIDE 41

INTRODUCTION TO AWS BOTO IN PYTHON

A Bucket

A bucket has a name Name is a string Unique name in all of S3. Contains many objects

An Object

An object has a key Name is full path from bucket root Unique key in the bucket Can only be in one parent bucket

slide-42
SLIDE 42

INTRODUCTION TO AWS BOTO IN PYTHON

Creating the client

s3 = boto3.client( 's3', region_name='us-east-1', aws_access_key_id=AWS_KEY_ID, aws_secret_access_key=AWS_SECRET )

slide-43
SLIDE 43

INTRODUCTION TO AWS BOTO IN PYTHON

Uploading les

s3.upload_file( Filename='gid_requests_2019_01_01.csv', Bucket='gid-requests', Key='gid_requests_2019_01_01.csv')

slide-44
SLIDE 44

INTRODUCTION TO AWS BOTO IN PYTHON

Uploading les

slide-45
SLIDE 45

INTRODUCTION TO AWS BOTO IN PYTHON

Uploading more objects

slide-46
SLIDE 46

INTRODUCTION TO AWS BOTO IN PYTHON

Listing objects in a bucket

response = s3.list_objects( Bucket='gid-requests', MaxKeys=2, Prefix='gid_requests_2019_') print(response)

slide-47
SLIDE 47

INTRODUCTION TO AWS BOTO IN PYTHON

Listing objects in a bucket

slide-48
SLIDE 48

INTRODUCTION TO AWS BOTO IN PYTHON

Listing objects in a bucket

slide-49
SLIDE 49

INTRODUCTION TO AWS BOTO IN PYTHON

Listing objects in a bucket

slide-50
SLIDE 50

INTRODUCTION TO AWS BOTO IN PYTHON

Getting object metadata

response = s3.head_object( Bucket='gid-requests', Key='gid_requests_2018_12_30.csv') print(response)

slide-51
SLIDE 51

INTRODUCTION TO AWS BOTO IN PYTHON

Getting object metadata

slide-52
SLIDE 52

INTRODUCTION TO AWS BOTO IN PYTHON

Downloading les

s3.download_file( Filename='gid_requests_downed.csv', Bucket='gid-requests', Key='gid_requests_2018_12_30.csv')

slide-53
SLIDE 53

INTRODUCTION TO AWS BOTO IN PYTHON

Deleting objects

s3.delete_object( Bucket='gid-requests', Key='gid_requests_2018_12_30.csv')

slide-54
SLIDE 54

INTRODUCTION TO AWS BOTO IN PYTHON

Summary

Buckets are like folders Objects are like les

boto3.client() s3.upload_file() s3.list_objects() s3.head_object() s3.download_file() s3.delete_object()

slide-55
SLIDE 55

Let's make some

  • bjects!

IN TRODUCTION TO AW S BOTO IN P YTH ON