EDU 2.0 Migrating to CloudSearch Graham Glass, Founder Background - - PowerPoint PPT Presentation

EDU 2.0

Migrating to CloudSearch

Graham Glass, Founder

Background

• Cloud-hosted E-learning Platform (LMS)
• www.edu20.org for academia
• www.edu20.com for businesses
• 1,000,000+ users
• 15,000 new users a week
• Customers include Disney, Large

Universities, California School Districts, small Kindergartens, etc.

Search

• Originally using Sphinx/ThinkingSphinx
• All data came from RDS/MySQL
• One Sphinx per app server
• Full indexing once a day
• Noticeable slowdown during indexing
• Sphinx daemons would sometimes fail

CloudSearch

• Decided to move to CloudSearch
• Simple, scaleable
• No Sphinx servers to manage
• Reliable, fast, delta-indexing
• Easy to index anything, including DynamoDB

Migration

• 14 different types were being indexed
• Decided to index just one item for testing
• Use script to upload initial contents
• Then index everything except for high

volume items (messages, postings), which were migrated last of all.

• Finally, index messages and postings

Configuration

• Two search domains, one for each site
• 20 index fields (only 2 text)
• Truncate messages/postings to 1000 bytes

Rails Integration

• Used aws_cloud_search gem
• Added hooks into object model to add

search update records to database

• Separate workers update search every 15

minutes with records from database

• Had issues with XML characters

Example Hooks

def after_create

super update_search end def update_search if ((type = material_class).searchable? && (scope != 'None')) SearchUpdate.add(type, material_id) end end

Example DB update

def self.add(type, ids)

begin search_update = SearchUpdate.new(:class_name => type.name, :ids => (ids.kind_of?(Array) ? ids.join(',') : ids), :operation => 'Update') search_update.save! rescue Exception => exception puts "SearchUpdate.add exception: #{exception.message}" end end

End Result

• Migrated all of search in about two weeks,

spending 1-2 hours a day.

• edu20.org: 7,000,000 documents, m2.xlarge
• edu20.com: 500,000 documents, m1.small
• Simplified architecture, positioned for

scalability and DynamoDB

• Only downside: $500/month for search