USING OPENSTACK TO INTEGRATE NON-OPENSTACK SERVICE JUNHO YOON, - - PowerPoint PPT Presentation

USING OPENSTACK TO INTEGRATE NON-OPENSTACK SERVICE

JUNHO YOON, ANDREW LIU, JACK NING

AGENDA

• INTRODUCTION
• MOTIVATIONS
• INTEGRATE AUTHN/AUTHZ
• INTEGRATE PLATFORM UI
• INTEGRATE PLATFORM COMMUNICATION
• CONTINOUS DEPLOYMENT WITH CUSTOMIZATION

Introduction

JUNHO YOON

Senior developer of NAVER

ANDREW LIU

Senior developer of NAVER China

JACK NING

Senior developer of NAVER China

Introduction

• Established in 1999, South Korea
• Handle more than half of internet search market in Korea
• Have more than 8000 employees
• Some apps have more than 100m users

Introduction

• Have a own IDC and a public cloud service https://www.ncloud.com
• However NOT OpenStack based

PASTA - IN-HOUSE PAAS

1000+ projects / 800+ daily user 10+ integrated platforms so far

PASTA – Architecture

PASTA-web keystone horizon cinder platforms ceph Nova Shipdock

(In-house docker cluster)

Company SSO platforms platforms Experimental

Today’s Topic

Users

ADENDA

• INTRODUCTION
• MOTIVATIONS
• INTEGRATE AUTHN/AUTHZ
• INTEGRATE PLATFORM UI
• INTEGRATE PLATFORM COMMUNICATION
• CONTINOUS DEPLOYMENT WITH CUSTOMIZATION

• About 40 platforms – It’s impossible even to remember URL
• No single entrance/catalog
• No resource utilization
• No common user experience
• Reinvent wheel

Motivations – Too many platforms

Motivations - authz / authn

• Each platforms had its own authz/authn
• Takes too much time for first access
• Has different permission set
• Requires even different user id/password sometimes

PlatformA PlatformB Common problems in big company

AWS comes to our sight

• What does AWS provide
• Integrated UI/UX - consistency
• Organized services catalog
• Separated PaaS UI with the main UI
• Centralized user management - AWS IAM
• We decide to make our platforms as a PaaS like AWS

Component which enables PaaS

PAAS INTEGRATED CONSOLE DYNAMIC RESOURCE PROVISIONING

• Consistent UX
• Integrated Authz/Authn
• Seamless integration b/w

platforms

• Resource Provisioning on

demand

• Docker Cluster?

https://www.slideshare.net/d eview/221-docker-

• rchestration

• Make new from scratch?
• Start from opensouce or

commercial system?

• Or OpenStack …

Composable Infrastructure

Decide to adopt openstack

ADENDA

• INTRODUCTION
• MOTIVATIONS
• INTEGRATE AUTHN/AUTHZ
• INTEGRATE PLATFORM UI
• MAKE PLATFORMS INTEROPERATE
• PACKAGE/DEPLOY WITH CUSTOMIZATION

Keystone

• Authn/Authz in OpenStack
• Feature
• Configurable auth/identity backend
• Easy to extend by Adding plugin for Authz/Authn
• Abundant API interface

Token Backend Catalog Backend Assignments Backend Identity Backend OpenStack Services Keystone API Policy Backend Credentials Backend

Keystone

Problem we are facing:

• Need to integrate into our existing SSO
• Need to identify not logged-in user as well
• Want to avoid to save user’s ID/PW in our DB

HORIZON KEYSTONE OPENSTACK COMPONENT

1) ID/PW

PROJECT

2) issue X-AUTH-TOKEN 3) connect to ENDPOINT

with X-AUTH-TOKEN

4) ask the X-AUTH-TOKEN info

(PROJECT ID + ROLE + USER)

extended Keystone v1

KEYSTONE PLATFORMS PASTA-WEB (IN-HOUSE CONSOLE) OAUTH2 PROVIDER

1.1) verify OAUTH-TOKEN 0) OAUTH Auth 1) ID/OAUTH-TOKEN

• r

ID/PASSWORD

IN-HOUSE LDAP

2) GET IDENTITY 1.2) verify PASSWORD

LDAP IDENTITY PLUGIN

4) ACCESS WITH X-AUTH-TOKEN 5) verify X-AUTH-TOKEN 3) ISSUE X-AUTH-TOKEN

AUTH PLUGIN

USING COMPANY’S SSO USING COMPANY’S LDAP FOR IDENTITY

Auth Plugin

SSO HTTP API /api/Auth/tokenInfo to verify token default identity auth PASSWORD AUTH AUTH PLUGIN LOGIN LOGIN FAILED LOGIN DEFAULT AUTH SSO AUTH Success Success Fail Fail

Auth Plugin

• Keypoint is…

üTreat SSO token as password üTry default auth method first. If failed, use auth using SSO next üExtends auth handler Keystone.auth.plugins.password.Password

• Keystone Configuration

Extended Keystone v1

KEYSTONE PLATFORMS PASTA-WEB (IN-HOUSE CONSOLE) OAUTH2 PROVIDER

1.1) verify OAUTH-TOKEN 0) OAUTH Auth 1) ID/OAUTH-TOKEN

• r

ID/PASSWORD

IN-HOUSE LDAP

2) GET IDENTITY 1.2) verify PASSWORD

LDAP PLUGIN

4) ACCESS WITH X-AUTH-TOKEN 5) verify X-AUTH-TOKEN 3) ISSUE X-AUTH-TOKEN

AUTH PLUGIN

Problem

• Do not have right to save OpenStack system

users in LDAP

• Deadly slow when retrieving all users.

Extended Keystone v2

KEYSTONE PLATFORMS PASTA-WEB (IN-HOUSE CONSOLE) OAUTH2 PROVIDER

1.1) verify OAUTH-TOKEN 0) OAUTH Auth 1) ID/OAUTH-TOKEN

• r

ID/PASSWORD 2) GET IDENTITY 1.2) verify PASSWORD

HYBRID IDENTITY PLUGIN (AUTH + IDENTITY)

4) ACCESS WITH X-AUTH-TOKEN 5) verify X-AUTH-TOKEN 3) ISSUE X-AUTH-TOKEN

PASTA AUTH HANDLER IN-HOUSE LDAP SQL

INTRODUCE HYBRID INDENTITY PLUGIN

• Save new user in SQL
• Read from only SQL when querying all users

Hybrid Backend Plugin

• About the auth part
• Based on keystone-hybrid-backend

ü Implement LDAP Indentity ü extending SQL Indentity

IDENTITY AUTH LOGIN LOGIN FAILED SQL Auth LOGIN LDAP Auth Success Success Failed Failed

Hybrid Backend Plugin

• Identity

ü For API like get/update user just like the auth flow

• Why customzied for list large users

ü 2000+ LDAP USER ü List all user take 10~60s in horizon ü No domain concept when adopting legacy platforms

• Configuration

LIST_USERS SQL + LDAP USERS SQL USERS Filter by Name Yes No

ADENDA

• INTRODUCTION
• MOTIVATIONS
• INTEGRATE PLATFORM AUTHN/AUTHZ
• INTEGRATE PLATFORM UI
• MAKE PLATFORMS INTEROPERATE
• PACKAGE/DEPLOY WITH CUSTOMIZATION

Previously our platforms….

• Have each own web based management console
• No consistent user experience
• Implemented using various tech set
• Backend : Spring/Node.js/Golang (No python…)
• Framework: Backbone.js/Angular/Vue.js/React/Jquery

Openstack - Horizon

Horizon

Nova Neutron

Nova UI

Cinder

Manilla Plugin BlarBlar Plugin

Keystone

Manilla

Neutron UI Cinder UI

Blar Blar

Openstack - Horizon

• Not fit for NAVER
• Is not working very well with large user set
• Seems “”little bit”” UGLY for us
• Implemented with Python + Django
• Need to restart and test whenever some platform’s UI upgraded
• How to evenly distribute the UI development job

to each platform’s developer guaranteeing consistency?

• Make UI independently developed

without forcing to use specific tech set

Micro Service Architecture

UI /

Monolithic App Microservice

Logic + Database

Microservice Microservice Microservice Microservice Microservice

Horizon

Micro Service Architecture - modified

UI

Integrator Microservice+UI

Logic + Database

Microservice+UI

</> </> </> </> </> </>

PASTA

Microservice+UI Microservice+UI Microservice+UI Microservice+UI

Micro Service Architecture - modified

UI integrator handles this part Each platform handles this part

Spring Cloud – Netflix ZUUL

HTTP Request “pre” filters “routing” filters “post” filters “custom” filters “error” filters BACKEND SERVER

Realized Runtime Flow

Pasta WEB PlatformA PlatformB PlatformC

• 2. Access

service-id.pasta.navercorp.com/platform-id/a.txt

• 6. https//{{platform-host}}/platform-id/*

OAUTH-PROVIDER

• 3. OAUTH

KEYSTONE

(OPENSTACK)

• 4. Service Permission Check &

Issue X-AUTH-TOKEN

ZUUL

• 7. User permission check

using X-AUTH-TOKEN X-AUTH-TOKEN

• 8. Final HTML Rendering
• 5. Decide where to route based on context path
• 8. Render platform page
• 1. Prepare routing table

Platform Info Extension

• Be able to keep each platform endpoint info

in keystone’s Service catalog and endpoints

• Service(Openstack Term) = Platform (PASTA Term)

Use the platform name as a context path Pick internal interface URL for routing

Platform Info Extension

• Need extra room to store extra routing info
• Ex) Platform Icons / Display order …
• Need separate DB to store these?
• Use description section with JSON

Service Info Extension

• Should store the project’s extra info into keystone
• Project(Openstack Term) = Service (PASTA Term)
• https://blueprints.launchpad.net/horizon/+spec/support-extra-prop-

for-project-and-user

• OpenStack4J

ADENDA

• INTRODUCTION
• MOTIVATIONS
• INTEGRATE AUTHN/AUTHZ
• INTEGRATE PLATFORM UI
• MAKE PLATFORMS INTEROPERATE
• PACKAGE/DEPLOY WITH CUSTOMIZATION

UI Level Interoperability

• Already be able to call the other platform’s REST API
• Because all platform UI share same authn/z in a user session

PLATFORM-A PLATFORM-B PLATFORM-C

$.get(“/platform-a/api/functionA”) $.get(“/platform-b/api/functionB”) $.get(“/platform-c/api/functionC”) http://sample.pasta-host/kaleido/

Backend Interoperability

• Need special way to communicate each other
• ex) Batch / Event Handler which run outside of the user session
• ex) Run user’s platform interoperation code when event is triggered
• Introduce Serverless Framework (openwisk)
• Not cover detail here

OPENWISK PlatformA PlatformB

ADENDA

• INTRODUCTION
• MOTIVATIONS
• INTEGRATE AUTHN/AUTHZ
• INTEGRATE PLATFORM UI
• MAKE PLATFORMS INTEROPERATE
• PACKAGE/DEPLOY WITH CUSTOMIZATION

PASTA- WEB PASTA OpenStack Pasta web package and delivered in traditional way How we package and deliver OpenStack?

Packaging/Deployment

• Requirement
• Should guarantee 7*24 service
• No downtime allowed
• Should be one-click deployment
• Adopt existing infrastructure (L4, MySQL) into OpenStack

deployment

• Minimize our own customization – to reduce rebase burden
• Kolla

ü Easy to customize by adding ansible plays ü Easy to scale out ü Highly configurable

Kolla – minimize customization

• Kolla plugin

ü Inspired by Kolla plugin for neutron ü Contributed for keystone https://review.openstack.org/#/c/401145/ ü Documents: kolla plugin functionality https://docs.openstack.org/kolla/ocata/image-

building.html#plugin-functionality

ü Format: [<image>-plugin-<plugin-name>] ü Install the plugin as a python module

kolla-build.conf

Kolla – plugin

• Benefits

ü Minimize Customization: only 6 commit ü Separated our customization from KOLLA

• Make your own plugin

ü Refer to plugin functionality of KOLLA image building ü Add similar template in the dockerfile.j2 ü Change the block name based on your images ü Add plugin source in the kolla-build.conf file

Kolla – integrate into legacy

• Integrate into company’s existing Database and LoadBalancer

üDisable DB and LB in Kolla globals.yml :

enable_mariadb: "no" enable_haproxy: "no"

üAdd New configurations for integration

Kolla – no downtime

• Requires no down time when upgrading keystone
• kolla-ansible option: --limit <host>

ü Deploy keystone on limit node

• Upgrading one by one

ü Ansible/site.yml: Serial attribute ü Loadbalancer: support health check by a specific URL and switch traffic

automatically

ü Customized ansible plays

Start to deploy one keystone node Remove health check url LB switch traffic to other nodes Stop and deploy keystone Wait for keystone back Added health check url Waiting for LB switch traffic back

Q&A THANKS