EDU 2.0 Migrating to CloudSearch Graham Glass, Founder Background - PowerPoint PPT Presentation

EDU 2.0 Migrating to CloudSearch Graham Glass, Founder

Background ● Cloud-hosted E-learning Platform (LMS) ● www.edu20.org for academia ● www.edu20.com for businesses ● 1,000,000+ users ● 15,000 new users a week ● Customers include Disney, Large Universities, California School Districts, small Kindergartens, etc.

Search ● Originally using Sphinx/ThinkingSphinx ● All data came from RDS/MySQL ● One Sphinx per app server ● Full indexing once a day ● Noticeable slowdown during indexing ● Sphinx daemons would sometimes fail

CloudSearch ● Decided to move to CloudSearch ● Simple, scaleable ● No Sphinx servers to manage ● Reliable, fast, delta-indexing ● Easy to index anything, including DynamoDB

Migration ● 14 different types were being indexed ● Decided to index just one item for testing ● Use script to upload initial contents ● Then index everything except for high volume items (messages, postings), which were migrated last of all. ● Finally, index messages and postings

Configuration ● Two search domains, one for each site ● 20 index fields (only 2 text) ● Truncate messages/postings to 1000 bytes

Rails Integration ● Used aws_cloud_search gem ● Added hooks into object model to add search update records to database ● Separate workers update search every 15 minutes with records from database ● Had issues with XML characters

Example Hooks def after_create super update_search end def update_search if ((type = material_class).searchable? && (scope != 'None')) SearchUpdate.add(type, material_id) end end

Example DB update def self.add(type, ids) begin search_update = SearchUpdate.new(:class_name => type.name, :ids => (ids.kind_of?(Array) ? ids.join(',') : ids), :operation => 'Update') search_update.save! rescue Exception => exception puts "SearchUpdate.add exception: #{exception.message}" end end

End Result ● Migrated all of search in about two weeks, spending 1-2 hours a day. ● edu20.org: 7,000,000 documents, m2.xlarge ● edu20.com: 500,000 documents, m1.small ● Simplified architecture, positioned for scalability and DynamoDB ● Only downside: $500/month for search