HAZELCAST DISTRIBUTED DATA STRUCTURES FOR JAVA WHO AM I Fuad - - PowerPoint PPT Presentation

HAZELCAST

DISTRIBUTED DATA STRUCTURES FOR JAVA

WHO AM I

Fuad Malikov @fuadm Hazelcast Co-founder Java Developer since 2005 Leading Hazelcast Technical Operations

COMPANY: HAZELCAST

Founded in 2008 Open Source Business Model Head Quartered in Palo Alto, CA Distributed Team with R&D mainly in Europe $14M+ Total Investment

PRODUCT: HAZELCAST

In-Memory DataGrid Open Source - Apache License 2.0 In-Mempory Distributed Data Store Distributed Messaging Distributed Computation NoSQL

WHO USES HAZELCAST?

and many more ...

USECASES

ALTERNATIVES?

Oracle Coherence IBM eXtreme Scale VMware Gemfire Gridgain Redhat Infinispan Terracotta Gigaspaces

WHY HAZELCAST?

Scale-out Computing Fast Application Performance Resilience Programming Model

FEATURES

Java Collection API Map, Queue, Set, List, +MultiMap, +ReplicatedMap Topic (PubSub) Ring Buffer Java Concurrency API Lock, ExecutorService, AtomicLong, Semaphore, CountDownLatch, Entry Processor, Query, Map Reduce, Aggregations Transactions Custom Serialization Native Memory support (off-heap) Native client: C#, C++, Java, REST, memcached Hibernate 2nd Level Cache Web Session Replication

CACHE AS A SERVICE

EASY API

// Creating a new Hazelcast node HazelcastInstance hz = Hazelcast.newHazelcastInstance(); // Getting a Map, Queue, Topic, ... Map map = hz.getMap("my-map"); Queue queue = hz.getQueue("my-queue"); ITopic topic = hz.getTopic("my-topic"); //Creating a Hazelcast Client HazelcastInstance client = Hazelcast.newHazelcastClient(); // Shutting down the node hz.shutdown();

HOW DOES IT WORK?

DATA PARTITIONING (1/2)

Multiple partitions per node Consistent Hashing: Controlled partitioning: "key@partitionkey", c.h.c.PartitionAware Possibility to find key owner for every key Support for Near-Caching and executions on key owner Automatic Fault-Tolerance Synchronous and Asynchronous backups Define sync / async backup counts

partition_id = hash(serialize(key)) % partitionCount ;

With 4 cluster nodes every server holds 1/4 real data and 1/4 of backups

DATA PARTITIONING (2/2)

A HAZELCAST NETWORK

HAZELCAST IN NUMBERS

Default partition count 271 Biggest cluster 600+ nodes 1 TB in-memory data Handles 1M+/sec topic messages Native Memory (Off-Heap) support for low GC overhead

COMMUNITY VS. ENTERPRISE

Feature Community Enterprise Java Collection API X X Java Concurrency API X X JCache API X X Map Reduce / Aggregations X X HD Memory Store (off-heap) X Advanced Security X Native Clients (C++ / C#) X Management Center X

HEAP VS HD MEMORY STORE

CODE SAMPLES

EASY TO UNITTEST

public class SomeTestCase { private HazelcastInstance[] instances; @Before public void before() throws Exception { // Multiple instances on the same JVM instances = new HazelcastInstance[2]; instances[0] = Hazelcast.newHazelcastInstance(); instances[1] = Hazelcast.newHazelcastInstance(); } @After public void after() throws Exception { Hazelcast.shutdownAll(); } }

SERIALIZATION

// java.io.Serializable public class User implements Serializable {} // or java.io.Externalizable public class User implements Externalizable {} // or (com.hazelcast.nio.serialization).DataSerializable public class User implements DataSerializable {} // or (com.hazelcast.nio.serialization).DataSerializable public class User implements IdentifiedDataSerializable {} // or new in Hazelcast 3 (multi version support) Portable public class User implements Portable {} //Support for Pluggable Custom Serializer

MAP

interface com.hazelcast.core.IMap<K, V> extends java.util.Map, java.util.ConcurrentMap HazelcastInstance hz = getHazelcastInstance(); //java.util.concurrent.ConcurrentMap implementation IMap<String, User> hzMap = hz.getMap("users"); hzMap.put("Peter", new User("Peter", "Veentjer")); hzMap.putIfAbsent("Peter", new User("Peter", "Veentjer")); //Distributed Lock hzMap.lock("Peter"); User peter = map.get("Peter");

LIST

interface com.hazelcast.core.IList<E> extends java.util.List HazelcastInstance hz = getHazelcastInstance(); //java.util.List implementation L ist<User> l ist = hz.getList("users"); l ist.add(new User("Peter", "Veentjer")); User peter = list.get(0);

QUEUE

interface com.hazelcast.core.IQueue<E> extends java.util.concurrent.BlockingQueue HazelcastInstance hz = getHazelcastInstance(); //java.util.concurrent.BlockingQueue implementation IQueue<Task> queue = hz.getQueue("tasks"); queue.offer(newTask()); queue.offer(newTask(), 500, TimeUnit.MILLISECONDS); Task task = queue.poll(); Task task = queue.poll(100, TimeUnit.MILLISECONDS); Task task = queue.take();

LOCK (1/3)

interface com.hazelcast.core.ILock extends java.util.concurrent.locks.Lock

LOCK (2/3)

HazelcastInstance hz = getHazelcastInstance(); // Distributed Reentrant L

• ck l
• ck = hz.getLock("myLock");

l

• ck.lock();

try { // Do something } finally { l

• ck.unlock();

}

LOCK (3/3)

HazelcastInstance hz = getHazelcastInstance(); // Map (Row-)locks IMap<String, User> map = hz.getMap("users"); map.lock("Peter"); try { // Do something with Peter } finally { map.unlock("Peter"); }

TOPIC / PUBSUB

public class Example implements MessageListener<String> { public void sendMessage { HazelcastInstance hz = getHazelcastInstance(); ITopic<String> topic = hz.getTopic("topic"); topic.addMessageListener(this); topic.publish("Hello World"); } @Override public void onMessage(Message<String> message) { System.out.println("Got message: " + message.getMessageObject()); } }

EXECUTORSERVICE

public interface com.hazelcast.core.IExecutorService extends java.util.concurrent.ExecutorService HazelcastInstance hz = getHazelcastInstance(); //java.util.concurrent.ExecutorService implementation IExecutorService es = hz.getExecutorService("name"); es.executeOnAllMembers(buildRunnable()); es.executeOnKeyOwner(buildRunnable(), "Peter"); es.execute(buildRunnable()); Map<..> futures = es.submitToAllMembers(buildCallable()); Future<..> future = es.submitToKeyOwner(buildCallable(), "Peter"); es.submitToAllMembers(buildCallable(), buildCallback()); es.submitToKeyOwner(buildCallable(), "Peter", buildCallback());

MORE ADVANCED FEATURES

ADVANCED FEATURES

Secondary Indexes and SQL-Like Query, Write-Behind / Write-Through persistence Read-Through (if key not loaded use MapLoader) Local & XA Transactions EntryListeners / EntryProcessors Map Reduce / Aggregators Hibernate 2nd Level Cache HTTP Session Clustering Automatic eviction Controlled partitioning and many more ...

CODE SAMPLES

INDEXING

IMap<String, User> map = Hazelcast.getMap("users"); map.addIndex("age", true); // ordered map.addIndex("active", false); // not ordered <map name="employees"> ... <indexes> <index ordered="true">age</index> <index ordered="false">name</index> </indexes> </map>

DISTRIBUTED SQL-LIKE QUERIES

HazelcastInstance hz = getHazelcastInstance(); IMap<String, User> map = hz.getMap("users"); Predicate predicate = new SqlPredicate("active AND age <= 30"); Set<User> users = map.values(predicate); Set<Entry<String, User>> entries = map.entrySet(predicate);

PERSISTENCY: MAPLOADER / MAPSTORE

public class MapStorage implements MapStore<String, User>, MapLoader<String, User> { // Some methods missing ... @Override public User load(String key) { return loadValueDB(key); } @Override public Set<String> loadAllKeys() { return loadKeysDB(); } @Override public void delete(String key) { deleteDB(key); } @Override public void store(String key, User value) { storeToDatabase(key, value); } } <map name="users"> <map-store enabled="true"> <class-name>com.hazelcast.example.MapStorage</class-name> <write-delay-seconds>0</write-delay-seconds> </map-store> </map>

TRANSACTIONS

HazelcastInstance hz = getHazelcastInstance(); TransactionContext context = hz.newTransactionContext(); context.beginTransaction(); TransactionalMap map = context.getMap("default"); TransactionalQueue queue = context.getQueue("default"); try { Tweet tweet = (Tweet) queue.poll(); processTweet(tweet); map.put(buildKey(tweet), tweet); context.commitTransaction(); } catch (Exception e) { context.rollbackTransaction(); }

Force location of corresponding data in the same partition by providing a special partition key

CONTROLLED PARTITIONING

HazelcastInstance hz = getHazelcastInstance(); Map users = hz.getMap("users"); users.put("Peter@Peter", new User("Peter", "Veentjer")); Map friends = hz.getMap("friends"); friends.put("Chris@Peter", new User("Christoph", "Engelbert")); friends.put("Nils@Peter", new User("Fuad", "Malikov")); //All three will be stored on the same partition

REPLICATED MAP

HazelcastInstance hazelcastInstance = Hazelcast.newHazelcastInstance(); Map<String, Customer> customers = hazelcastInstance .getReplicatedMap("customers"); customers.put( "1", new Customer( "Joe", "Smith" ) ); customers.put( "2", new Customer( "Ali", "Selam" ) ); customers.put( "3", new Customer( "Avi", "Noyan" ) ); Collection<Customer> colCustomers = customers.values(); for ( Customer customer : colCustomers ) { // process customer }

JCACHE

JCACHE (1/2)

<dependency> <groupid>javax.cache</groupid> <artifactid>cache-api</artifactid> <version>1.0.0</version> </dependency> // Retrieve the CachingProvider which is automatically baced by // the chosen Hazelcast server or client provider CachingProvider cachingProvider = Caching.getCachingProvider(); // Create a CacheManager CacheManager cacheManager = cachingProvider.getCacheManager(); // Cache<String, String> cache = cacheManager // .getCache( name, String.class, String.class ); // Create a simple but typesafe configuration for the cache CompleteConfiguration<String, String> config = new MutableConfiguration<String, String>() .setTypes( String.class, String.class );

JCACHE (2/2)

// Create and get the cache Cache<String, String> cache = cacheManager .createCache( "example", config ); // Alternatively to request an already existing cache // Cache<String, String> cache = cacheManager // .getCache( name, String.class, String.class ); // Put a value into the cache cache.put( "world", "Hello World" ); // Retrieve the value again from the cache String value = cache.get( "world" ); // Print the value 'Hello World' System.out.println( value );

MAP REDUCE

HazelcastInstance hz = getHazelcastInstance(); Map users = hz.getMap("users"); JobTracker tracker = hz.getJobTracker("default"); KeyValueSource source = KeyValueSource.fromMap(users); Job job = tracker.newJob(source); ICompleteFuture future = job.mapper(new MyMapper()) .reducer(new MyReducer()) .submit(); Map result = future.get();

AGGREGATIONS

HazelcastInstance hz = getHazelcastInstance(); Map users = hz.getMap("users"); int sum = users.aggregate( Supplier.all((user) -> user.getSalary()), Aggregations.longSum() );

NEW IN HAZELCAST 3.5

NEW IN HAZELCAST

Ring Buffers Client Protocol Cluster Quorum

RING BUFFERS

@fuadm, @hazelcast hazelcast@googlegroups.com http://www.hazelcast.com http://github.com/hazelcast/hazelcast

THANK YOU!

ANY QUESTIONS?

We are hiring > http://hazelcast.com/company/careers/

Images: www.clipartist.info, Gnome Nebula Theme, KDE theme