MongoDB @ SourceForge Mark Ramm We had a problem six weeks the - - PowerPoint PPT Presentation

▶
mongodb sourceforge
SMART_READER_LITE
LIVE PREVIEW

MongoDB @ SourceForge Mark Ramm We had a problem six weeks the - - PowerPoint PPT Presentation

Apr 07, 2023 331 likes •765 views

MongoDB @ SourceForge Mark Ramm We had a problem six weeks the other sourceforge over 90% of traffic Design goals Improve Usability (more data, more dynamic pages) Improve Performance Improve Reliability Big Green Button

slide-1
SLIDE 1

MongoDB @ SourceForge

Mark Ramm

slide-2
SLIDE 2

We had a problem

slide-3
SLIDE 3

six weeks

slide-4
SLIDE 4

the other sourceforge

slide-5
SLIDE 5
  • ver 90% of traffic
slide-6
SLIDE 6
  • Improve Usability

(more data, more dynamic pages)

  • Improve Performance
  • Improve Reliability

Design goals

slide-7
SLIDE 7

Big Green Button

slide-8
SLIDE 8

Tools Matter

SQL ACID well known DSL a good mix simple robust scalable flexible NoSQL CAP consistant available partition tolerant focused base basically available soft state eventual consistency scalable simple fast flexible Know Your Tools Screws and Nails deck siding mongodb
slide-9
SLIDE 9

ACID

http://www.flickr.com/photos/di4b0liko/2292648884/

slide-10
SLIDE 10

Why I NEED Relational

  • I have to have ACID because...
  • It’s financial data

(need consistency)

  • My data is relational

B U L L S H I T

slide-11
SLIDE 11

But...

slide-12
SLIDE 12

Tools Matter

SQL ACID well known DSL a good mix simple robust scalable flexible

slide-13
SLIDE 13

NoSQL

slide-14
SLIDE 14
slide-15
SLIDE 15

Tools Matter

NoSQL CAP consistant available partition tolerant focused scalable simple fast flexible Topic
slide-16
SLIDE 16

CAP

http://www.cs.berkeley.edu/~brewer/cs262b-2004/PODC-keynote.pdf

slide-17
SLIDE 17

http://www.flickr.com/photos/beigephotos/900974545/

slide-18
SLIDE 18

blah, blah, blah, blah, blah

slide-19
SLIDE 19

typology of NoSQL

  • key/value store
  • distributed key/value stores
  • column oriented stores
  • map-reduce store/system
  • document oriented store
  • graph oriented stores
slide-20
SLIDE 20

Enough theory

slide-21
SLIDE 21

We had documents

{ 'source': 'sf.net', 'shortname': 'azureus', 'related': [ 'shortname': 'foo', 'description':'bar', 'screenshots':[...], 'project_url': 'http://asdf', 'name'; 'Azureus',] 'sf_id': 5383, 'sf_piwik_siteid': '2', 'name': 'Azureus', 'doap': 'http://sourceforge.net/api/project/name/azureus/doap', 'created': datetime.datetime(2003, 6, 24, 0, 0), 'homepage': 'http://azureus.sourceforge.net', 'project_url': 'http://sourceforge.net/projects/combined-for-all-data', 'resources': { 'news': [{'feed': 'http://sourceforge.net/api/news/index/..., 'name': 'News', 'url': 'http://sourceforge.net/news/?group_id=84122'}], 'forums': [{'feed': 'http://sourceforge.net/api/post/index/.../rss', 'name': 'Help', 'url': 'http://sourceforge.net/forum/forum.php...', 'item_count': 1,}, {'feed': 'http://sourceforge.net/api/post/index.../rss', 'name': 'Discussion', 'url': 'http://sourceforge.net/forum/forum.php/...', 'item_count': 28216,}],
slide-22
SLIDE 22

we did get some “lucky breaks”

  • consistency not critical
  • scale reads, not writes
slide-23
SLIDE 23

We wanted replication

slide-24
SLIDE 24 Mongo Master directory Mongo Slave directory Mongo Slave directory Mongo Slave sf.gobble fetcher fetcher feed api's sf.net freshmeat.net hosted apps etc
slide-25
SLIDE 25

We didn’t have a lot of time

slide-26
SLIDE 26

MongoDB has a query language

select * from document where x=3 and y="foo" db.things.find( { x : 3, y : "foo" } );

slide-27
SLIDE 27

partial updates

  • $inc
  • $set
  • $unset
  • $push
  • $pushAll
  • $addToSet
  • $pop
  • $pull
  • $pullAll

{ $inc : { field : value } }

slide-28
SLIDE 28

Conditional Updates

db.people.update( { name:"Joe" }, { $inc: { x:1, y:1 } }, true

slide-29
SLIDE 29 { 'source': 'sf.net', 'shortname': 'azureus', 'related': [ 'shortname': 'foo', 'description':'bar', 'screenshots':[...], 'project_url': 'http://asdf', 'name'; 'Azureus',] 'sf_id': 5383, 'sf_piwik_siteid': '2', 'name': 'Azureus', 'doap': 'http://sourceforge.net/api/project/name/azureus/doap', 'created': datetime.datetime(2003, 6, 24, 0, 0), 'homepage': 'http://azureus.sourceforge.net', 'project_url': 'http://sourceforge.net/projects/combined-for-all-data', 'resources': { 'news': [{'feed': 'http://sourceforge.net/api/news/index/..., 'name': 'News', 'url': 'http://sourceforge.net/news/?group_id=84122'}], 'forums': [{'feed': 'http://sourceforge.net/api/post/index/.../rss', 'name': 'Help', 'url': 'http://sourceforge.net/forum/forum.php...', 'item_count': 1,}, {'feed': 'http://sourceforge.net/api/post/index.../rss', 'name': 'Discussion', 'url': 'http://sourceforge.net/forum/forum.php/...', 'item_count': 28216,}], "url": "http://freshmeat.net/
slide-30
SLIDE 30 ], }, 'screenshot_page': 'http://sourceforge.net/project/screenshots.php?group_id=93438', 'screenshots': [{'url': 'http://sourceforge.net/dbimage.php?id=208967', 'thumb': 'http://sourceforge.net/dbimage.php?id=208966', "name" : "Table structure view"}, {'url': 'http://sourceforge.net/dbimage.php?id=99723', 'thumb': 'http://sourceforge.net/dbimage.php?id=99722'}], # ID,shortname,description only present for SF projects. 'categories': {'Development Status': [{'description': '4 - Beta', 'id': 10, 'name': '4 - Beta', 'shortname': 'beta'}], 'Intended Audience': [{'description': 'Developers', 'id': 3, 'name': 'Developers', 'shortname': 'developers'}, {'description': 'End Users/Desktop', 'id': 2, 'name': 'End Users/Desktop', 'shortname': 'endusers'}, {'description': 'System Administrators', 'id': 4, 'name': 'System Administrators', 'shortname': 'sysadmins'}], 'License': [{'description': 'Apache License V2.0', 'id': 401, 'name': 'Apache License V2.0', 'shortname': 'apache2'}, {'description': 'GNU Library or Lesser General Public License (LGPL)', 'id': 16, 'name': 'GNU Library or Lesser General Public License (LGPL)', 'shortname': 'lgpl'}], 'Operating System': [{'description': 'All POSIX (Linux/BSD/UNIX-like OSes)', 'id': 200, 'name': 'All POSIX (Linux/BSD/UNIX-like OSes)', 'shortname': 'posix'}, {'description': 'OS Independent (Written in an interpreted language)', 'id': 235, 'name': 'OS Independent (Written in an interpreted language)', 'shortname': 'independent'}], 'Programming Language': [{'description': 'Python', 'id': 178, 'name': 'Python', 'shortname': 'python'}],
slide-31
SLIDE 31

Tools Matter

Know Your Tools Screws and Nails deck siding

mongodb

slide-32
SLIDE 32

AKA Learning by doing

slide-33
SLIDE 33

AKA Horror stories

slide-34
SLIDE 34
slide-35
SLIDE 35
slide-36
SLIDE 36

Tools Matter

Know Your Tools Screws and Nails deck siding

mongodb

slide-37
SLIDE 37
slide-38
SLIDE 38

merciless.sourceforge.net

slide-39
SLIDE 39
  • F igure out what

YOUR app needs

  • Don’t obsess about

SCALE you’ll never achieve

  • Use the right tool for

the job

slide-40
SLIDE 40

Lessons learned

  • a tool is only right when you know

how to use it

  • DomainModel style setup is critical if

you use more than one persistance type

slide-41
SLIDE 41

Mongo Lessons learned

  • you will have to repeat yourself
  • autosharding (still) not ready
  • local mongo on the web server is

*really* fast

  • be carefull if the index does not fit in

memory