Labs #2 WebDev Web ebDe Dev v Lab #1 On your local Ubuntu VM, - - PowerPoint PPT Presentation

Labs #2

WebDev

Web ebDe Dev v Lab #1

 On your local Ubuntu VM, install the Python and C development

tools, then run the app locally

sudo apt update sudo apt install python3-dev build-essential git clone https://bitbucket.org/wuchangfeng/cs430-src cd cs430-src/WebDev_Guestbook_v3_nginx_uwsgi virtualenv -p python3 env source env/bin/activate pip install -r requirements.txt python app.py

 Bring up a browser on the VM and point the browser to the web app

(localhost:5000)

 Add several entries that include your OdinID and show the resulting

page

Portland State University CS 430P/530 Internet, Web & Cloud Systems

installs Flask, uwsgi packages

 On GCP (Compute Engine), create instance

 Ubuntu 18.04 LTS on an f1-micro instance within the us-west1-b zone  Also allow both HTTP and HTTPS traffic to the instance

Portland State University CS 430P/530 Internet, Web & Cloud Systems

 Record the external IP address of your Compute Engine instance

 Important: keep this instance running for rest of lab.

 Register a DNS name cs430-<OdinID> on ipq.co, fdns.uk,

• r noip.com that points to your instance

 https://ipq.co/

 If you get a rate-limit error on Let's Encrypt, use https://www.noip.com/ or

https://fdns.uk

Portland State University CS 430P/530 Internet, Web & Cloud Systems

 https://fdns.uk  https://www.noip.com/

 Note: no dash is allowed

Portland State University CS 430P/530 Internet, Web & Cloud Systems

 https://www.freenom.com/  Or use xip.io and its rip-offs

 http://nip.io/  https://sslip.io/  http://xip.io/

Portland State University CS 430P/530 Internet, Web & Cloud Systems

 ssh into instance and clone course repository

git clone https://bitbucket.org/wuchangfeng/cs430-src

 Navigate to WebDev_Guestbook_v3_nginx_uwsgi for uwsgi setup

 Examine app.ini

 Configures entry-point, number of processes, communication channel

 Examine wsgi.py

 Called upon initialization of uwsgi  Import app from app.py and run it

Portland State University CS 430P/530 Internet, Web & Cloud Systems

 Examine etc/systemd.template

 systemd system and service manager for Ubuntu 18  Configures startup of uwsgi daemon and its environment  Permissions set to Unix account of nginx (www-data)  Working directory set to PROJECT_DIR (replaced on installation with your

directory)

 PATH environment variable set to location of binaries for pre-installed Python

environment (assumes virtualenv's env is in PROJECT_DIR)

 Specify command to start uwsgi daemon using config file from previous slide

Portland State University CS 430P/530 Internet, Web & Cloud Systems

 Examine etc/nginx.template

 Configure nginx web server using your DNS name and point reverse proxy to

uwsgi socket

 server_name being served by configuration (replaced on installation with

IPQuick name, e.g. cs430-wuchang.ipq.co)

 root file system location to serve web application files from  Specify / path served by uwsgi and the python app

 uwsgi_pass to specify location of unix domain socket for communication

 Specify /static path to be served directly by nginx from specified directory

Portland State University CS 430P/530 Internet, Web & Cloud Systems

 Examine install.sh

 Will be invoked with your DNS name

 (e.g. ./install.sh cs430-wuchang.ipq.co)

 Script pulls out first part of the DNS-name (cs430-wuchang) (e.g. eliminates first

period and all subsequent characters)

 Used to name systemd service

 Installs python, virtualenv, nginx, and certbot. Create virtualenv

environment for web application and install packages into it from requirements.txt

Portland State University CS 430P/530 Internet, Web & Cloud Systems

 Examine install.sh (cont.)

 Sets up systemd startup file for uwsgi. Name it using the first part of site

name ($SITE from previous slide)

 Substitute PROJECT_DIR and PROJECT_USER based on current

directory ($PWD) and current user ($SUDO_USER)

 Note sed use of + instead of / (since / is used in PROJECT_DIR)

Portland State University CS 430P/530 Internet, Web & Cloud Systems

 Examine install.sh (cont.)

 Sets up nginx startup to add site (done via adding config file to

/etc/nginx/sites-available and a link in /etc/nginx/sites-enabled to it)

Portland State University CS 430P/530 Internet, Web & Cloud Systems

 Examine install.sh (cont.)

 Changes ownership to allow nginx (via www-data account) to create domain

socket

 Starts systemd service for site via systemctl , enables it by default (on

startup), and restarts nginx for changes to take hold

 Obtains the TLS certificate via certbot in non-interactive mode (see TLS slides on

Let's Encrypt)

Portland State University CS 430P/530 Internet, Web & Cloud Systems

Web ebDe Dev Lab #1

 Run the install script, show site

 sudo ./install.sh cs430-wuchang.ipq.co

Portland State University CS 430P/530 Internet, Web & Cloud Systems

Databases

Da Databases tabases Lab #1

Do this quiz

 http://www.w3schools.com/sql/sql_quiz.asp  Screenshot of "Check your answers"  Does not have to be perfect (mine wasn't), but check the ones you got

incorrect

Portland State University CS 430P/530 Internet, Web & Cloud Systems

Da Databases tabases Lab #2 (Cloud ud SQL QL)

 Create a Managed MySQL database with Cloud SQL

 Set-up and query using a Cloud MySQL instance

Portland State University CS 430P/530 Internet, Web & Cloud Systems

Setup etup

 From Google Cloud Shell, download data  Examine schema and data in CSV (comma separated values) format  Stage data definition file and values in a storage bucket so it can be

imported by Cloud SQL service

 Create a storage bucket  Verify files are in Cloud Storage bucket via web console

Portland State University CS 430P/530 Internet, Web & Cloud Systems

git clone https://github.com/GoogleCloudPlatform/training-data-analyst cd training-data-analyst/CPB100/lab3a less cloudsql/table_creation.sql head cloudsql/*.csv gsutil mb -c regional -l us-west1 gs://<BUCKET-NAME> gsutil cp cloudsql/* gs://<BUCKET-NAME>/sql/

Create a new Cloud SQL instance

 In the Google Cloud Console, scroll down and select SQL in the

Storage subsection

Portland State University CS 430P/530 Internet, Web & Cloud Systems

Create a new Cloud SQL instance

 Create an instance  Choose MySQL (Second Generation)

 Name the Instance ID: rentals  Set and remember the root password for the database  Place in us-west1-b

Portland State University CS 430P/530 Internet, Web & Cloud Systems

Conf nfig igure ure access cess to i instance stance from

• m Cloud

ud Shell ell

 In Cloud Shell, find the external IP address of the shell  Within Cloud SQL, click on the instance, then on "Edit", then on

"Connectivity" under Configuration Options, then on "Add Network"

 We will be adding the Cloud Shell IP address to authorized addresses that can

connect to the instance

Portland State University CS 430P/530 Internet, Web & Cloud Systems

curl http://ipecho.net/plain; echo

Conf nfig igure ure access cess to i instance stance from

• m Cloud

ud Shell ell

 Enter the IP address found previously with ipecho as a CIDR route

 e.g. 131.252.220.66/32

 Click on "Done", then "Close"  Ensure it appears in "Authorized Networks", then "Save"

Portland State University CS 430P/530 Internet, Web & Cloud Systems

Fill in your Cloud Shell's IP address here as a CIDR prefix

Import table definitions and tables (from Cloud Storage bucket)

 Click on the hyperlink named rentals (i.e. your Cloud SQL

instance name)

 Click on Import (on the top menu bar)  Click on the Browse button and browse to your storage bucket

containing the SQL lab files

 Navigate to table_creation.sql  Select and click Import.

Portland State University CS 430P/530 Internet, Web & Cloud Systems

Populate tables

 Import the two CSV files from Cloud Storage

 Within Cloud SQL instance, click on Import (top menu)  Click Browse, then find and select accommodation.csv  Fill out the rest of the dialog

 Database = recommendation_spark  Table = Accommodation

 Repeat for rating.csv

 Table = Rating

Portland State University CS 430P/530 Internet, Web & Cloud Systems

Use se Cloud ud SQL QL

 Connect to instance  Set database for MySQL session  View tables created  Verify data  Run queries for accommodations at various price levels and types

Portland State University CS 430P/530 Internet, Web & Cloud Systems

mysql --host=<MySQLIP> --user=root --password use recommendation_spark; show tables; select * from Rating; select * from Accommodation where…

Cleanup eanup

 Shutdown the Cloud SQL instance to avoid…

Portland State University CS 430P/530 Internet, Web & Cloud Systems

Da Databases tabases Lab #2 (Cloud ud SQL QL)

 https://codelabs.developers.google.com/codelabs/cpb100-cloud-sql (22

min)

Portland State University CS 430P/530 Internet, Web & Cloud Systems

Da Databases tabases Lab #3 (AWS RDS DS)

 AWS RDS 10-minute tutorial  Sign-in to AWS Educate and go to your "AWS Starter Account"  Then click "AWS Console" to get to AWS Management console

 Note, ensure the browser is not blocking the pop-up window

Portland State University CS 430P/530 Internet, Web & Cloud Systems

 Example console

Portland State University CS 430P/530 Internet, Web & Cloud Systems

 Select RDS under Database  Ensure region is us-east-1 (N. Virginia)  Click on Instances, then "Create database" and choose MySQL  Then, choose "Dev/Test"

Portland State University CS 430P/530 Internet, Web & Cloud Systems

 Choose MySQL and RDS Free Usage Tier

Portland State University CS 430P/530 Internet, Web & Cloud Systems

 Set instance identifier, and database credentials  Follow instructions to configure database, then create and view

details

Portland State University CS 430P/530 Internet, Web & Cloud Systems

 Create New VPC and allow public access

(do not do this in practice)

 Create new VPC security group  Set database name  Then, create and view details

Portland State University CS 430P/530 Internet, Web & Cloud Systems

 Show your running instance

Portland State University CS 430P/530 Internet, Web & Cloud Systems

 Connecting

 By default, the IP address that was used to set up the database is

allowed to connect to the database port 3306

 This means you must install the MySQL client libraries locally to do

the rest of the lab

 If you install the libraries on your local Ubuntu VM, the command is



apt install mysql-client

 You may use the MySQL client on linux.cs.pdx.edu

(131.252.208.103) by ssh'ing into it

Portland State University CS 430P/530 Internet, Web & Cloud Systems

 Enabling access to the database server

 The IP address of your client must be added to the security group for the

database server

 The screenshot below shows two addresses with access to the RDS instance

 131.252.208.103 is linux.cs.pdx.edu  131.252.220.66 is my machine

 NAT makes it difficult to find your public IP address  To get around this run the command below to discover it, then add it to the

group

 curl https://ipinfo.io/ip

Portland State University CS 430P/530 Internet, Web & Cloud Systems

 Connect to database via Linux command line (or optionally via

Windows MySQL Workbench client as described in tutorial)

 mysql -h <instance_endpoint> -P 3306 -u mymasteruser -p

Portland State University CS 430P/530 Internet, Web & Cloud Systems

 Delete the instance (do not create final snapshot)

Portland State University CS 430P/530 Internet, Web & Cloud Systems

Da Databases tabases Lab #3 (AWS RDS DS)

 https://aws.amazon.com/getting-started/tutorials/create-mysql-

db/

Portland State University CS 430P/530 Internet, Web & Cloud Systems

Homework #2

Previe iew

 Homework #2-4  Incremental construction of a scalable bubble tea web site

 HW #2: Simple Python Flask MVC/MVP web app with sqlite3

database backend, read/create (delete as a stretch goal)

 HW #3: Containerize web app, deploy on Compute Engine  HW #4: Deploy HW #2 on App Engine with Cloud Datastore

backend

Portland State University CS 430P/530 Internet, Web & Cloud Systems

Home mewor

• rk

k #2

 Create a toy Python Flask web application for viewing (and

eventually submitting) bubble tea store locations

 Development and submission

 Within the BitBucket repository from HW #1, create a directory

"hw2" for your application

 Develop using git and Bitbucket

 Ensure changes are committed frequently

 When complete, submit a zip file for a timestamp onto D2L dropbox

Portland State University CS 430P/530 Internet, Web & Cloud Systems

 Application should …

 Follow an MVC/MVP pattern and support the following routes/views

 Default landing page  Page for reading all bubble tea stores  Page for creating/inserting a new bubble tea store via an HTML form

 Listen on port 8000 when called directly from python  Contain an abstract data model class that serves as the base class for

specific model instantiations (sqlite3, Cloud Datastore, etc)

 Abstract model should be documented via Docstrings including parameters and

return values with their types

 Abstract model should support individual fields that a bubble tea store would

typically have (name, street address, city, state, zip code, store hours, phone number, rating, review, drink to order, etc.)  Contain a derived data model class (e.g. model_sqlite3) that supports

creation and reading of stores via a sqlite3 database

 You may derive your application out of the supplied code in the

course repository at https://bitbucket.org/wuchangfeng/cs430- src

 Rubric posted on course web page

Portland State University CS 430P/530 Internet, Web & Cloud Systems

Containers

Container tainer Lab #1

 On a Ubuntu 18.04 VM

 Clone the repository if you haven't already, then goto the Container Lab

 git clone https://bitbucket.org/wuchangfeng/cs430-src  cd cs430-src/Container_Guestbook

 Examine the Python/Flask application in app.py and Dockerfile.ubuntu

to see what they do. Change the MAINTAINER to yourself

Portland State University CS 430P/530 Internet, Web & Cloud Systems

# Use Ubuntu 18.04 as the base image FROM ubuntu:18.04 # Specify your e-mail address as the maintainer of the container image MAINTAINER Your Name "yourname@pdx.edu" # Execute apt-get update and install to get Python's package manager # installed (pip) RUN apt-get update -y RUN apt-get install -y python-pip # Copy the contents of the current directory into the container directory /app COPY . /app # Set the working directory of the container to /app WORKDIR /app # Install the Python packages specified by requirements.txt into the container RUN pip install -r requirements.txt # Set the program that is invoked upon container instantiation ENTRYPOINT ["python"] # Set the parameters to the program CMD ["app.py"]

Container tainer Lab #1

 On a Ubuntu 18.04 VM

 Ensure you are added to the docker group

 sudo usermod -a -G docker $(whoami)  Then, logout and log back in

Portland State University CS 430P/530 Internet, Web & Cloud Systems

 Build a container for the app called helloubuntu with Ubuntu 18.04 as a base

 docker build -f Dockerfile.ubuntu -t helloubuntu .

 Show the image generated and list the size of the image created

 docker images

 Run the container in detached mode, mapping the host's port 8000 to the container's

port 5000

 docker run -p 8000:5000 --name hellou -di helloubuntu

 Test the container by retrieving http://127.0.0.1:8000/ using a browser, wget or

curl

 Show the container in the docker listing to find its name hellou under the

"NAMES" column.

 docker ps -a

Portland State University CS 430P/530 Internet, Web & Cloud Systems

 Stop the running container via its name, for a container called

"hellou", the command is

 docker stop hellou

 See that it is no longer running via

 docker ps –a

 Start the container via its name

 docker start hellou  Note that this only starts the container, but doesn't give you a session on it.

 Now, execute an interactive shell on the container

 docker exec -it hellou /bin/bash  Do an ls and a ps –ef to show the filesystem and processes running in the

container

 Exit out of the container

Portland State University CS 430P/530 Internet, Web & Cloud Systems

 Publish the container to your Docker Hub account  Login to Docker Hub using account from Homework #1

 docker login

 Tag image to your Docker Hub repo

 docker tag helloubuntu <dockerhub_id>/helloubuntu

 Push image to your Docker Hub repo

 docker push <dockerhub_id>/helloubuntu

Portland State University CS 430P/530 Internet, Web & Cloud Systems

 Now, stop and remove the local container

 docker stop hellou  docker rm hellou  Note that while this removes the container, it does not remove the container image

it was derived from (i.e. helloubuntu).  Now, remove both container images from your local machine

 docker rmi helloubuntu <dockerhub_id>/helloubuntu

 Run the image directly from Docker Hub

 docker run -di -p 8000:5000 --name hellou

<dockerhub_id>/helloubuntu  Show the output of the command  Validate that the container works by retrieving http://127.0.0.1:8000/

using a browser, wget or curl

 Show the container image on Docker Hub  Show the container image metadata on MicroBadger

(https://microbadger.com)

Portland State University CS 430P/530 Internet, Web & Cloud Systems

Container tainer Lab #2

 Build a container for the application with Python Alpine as a base

 Repeat the same steps used in the Ubuntu container helloubuntu,

but with Dockerfile.alpine to create a local container image helloalpine, a local container helloa, and a Docker Hub container <dockerhub_id>/helloalpine

 For the the docker exec command, what is its output?

 What happens when you replace /bin/bash with /bin/sh in the

docker exec command?  Repeat all other steps

Portland State University CS 430P/530 Internet, Web & Cloud Systems

Container tainer Lab #3

 Running containers in Google Cloud (UbuntuVM)

 Create VM on Google Cloud's Compute Engine

 Create instance  Place in zone us-west1-b  Select Ubuntu 18.04 LTS as a boot disk  Click on Allow HTTP Traffic  Click on Create

 Install docker on the VM and add your username to its group

 sudo apt update  sudo apt install docker.io  sudo usermod -a -G docker $(whoami)  Then, logout and log back in

 Run the helloalpine container image, but map the host's port 80 to the

container's port 5000

 docker login  docker run -di -p 80:5000 --name helloa

<dockerhub_id>/helloalpine  Go to a web browser and point it to the External IP address of the VM

 Can be done by clicking on the IP address from the Compute Engine console  Show the output

Portland State University CS 430P/530 Internet, Web & Cloud Systems

Container tainer Lab #4

 Running containers directly in Google Cloud (ContainerOSVM)

 We will run our container in the previous labs and deploy it directly on a

Compute Engine instance

 Unlike in previous labs, Compute Engine automatically maps container

ports to host ports when an instance is started with a specified container image

 As a result, we need to create a VPC firewall rule to allow the port

• ur container runs on by default (5000)

Portland State University CS 430P/530 Internet, Web & Cloud Systems

 Name the rule and set its tag to http-5000

 Rules applied based on tag name

 Specify allow on match, ingress filtering, all source IP addresses, TCP

port 5000

Portland State University CS 430P/530 Internet, Web & Cloud Systems

 Create a VM to deploy your container on

 Specify container image name (e.g. your Docker Hub image name for

your web app)

 Boot disk will be set to Google's Container OS  Also, enable HTTP access

Portland State University CS 430P/530 Internet, Web & Cloud Systems

 Apply the tag you created

 Create the VM and wait for it to come up

 Then, ssh into the instance

Portland State University CS 430P/530 Internet, Web & Cloud Systems

 Perform a local request to the web app (e.g. wget

http://localhost:5000) to ensure the container is running

Portland State University CS 430P/530 Internet, Web & Cloud Systems

 Visit the site via the external IP address on port 5000 to show the

site running

Portland State University CS 430P/530 Internet, Web & Cloud Systems

Homework #3

Home mewor

• rk

k #3

 Containerize Homework #2 using Ubuntu 18.04 as a base Linux image

 Upload image to your Docker Hub account under the name

"<dockerhub_id>/hw3large"

 Repeat using a different base image (Alpine, Busybox, Minimal Ubuntu) in

• rder to make it as *small* as possible

 Upload it to your Docker Hub account under the name

"<dockerhub_id>/hw3small"

 Show (via screenshot) the size of both containers on Docker Hub by

clicking on Tags

Portland State University CS 430P/530 Internet, Web & Cloud Systems

 Remove all containers and container images on your system, then

use docker run to instantiate each container directly from one

• f your images on Docker Hub

 Show the console output displaying container image layers being pulled  Connect to your web app locally and show it works (wget or browser)

 Submission

 Containers via Docker Hub (hw3large, hw3small)  Bitbucket submission of source files (including the two Dockerfiles used

to build the containers), along with a single document with your screenshots in directory <your_cs430_repo>/hw3

 D2L zipfile submission of hw3 for timestamp

 Prize for the smallest 3 working containers submitted

 https://blog.codeship.com/reduce-docker-image-size/  https://blog.realkinetic.com/building-minimal-docker-containers-for-

python-applications-37d0272c52f3

 https://tortuemat.gitlab.io/blog/2018/02/05/weigthless-docker-

images/

Portland State University CS 430P/530 Internet, Web & Cloud Systems