high dimensional computing the upside of the curse of
play

High dimensional computing - the upside of the curse of - PowerPoint PPT Presentation

Mar 14, 2023 •155 likes •1.26k views

High dimensional computing - the upside of the curse of dimensionality Peer Neubert Stefan Schubert Kenny Schlegel Peer Neubert, TU Chemnitz Topic: (Symbolic) Computation with large vectors Roughly synonyms: High dimensional Computing

  1. Teaser application 2: Place recognition in changing environments Deep This is where the Neural hyperdimensional Network magic happens d + ⊗ Y i = (X i−k P −k ) k=0 Peer Neubert, TU Chemnitz

  2. Outline: Introduction to high dimensional computing 1) Historical note 2) High dimensional vector spaces and where they are used 3) Mathematical properties of high dimensional vector spaces 4) Vector Symbolic Architectures or “How to do symbolic computations using vectors spaces” including “What is the Dollar of Mexico?” Peer Neubert, TU Chemnitz

  3. Historical note See: “Geometry and Meaning” by Dominic Widdows 2004, CSLI Publications, Stanford, ISBN 9781575864488 Ancient Greeks: Roots of geometry ● Plato: geometric theory of creation and elements – Journey and work of Aristotle – Euclid: “Elements of geometry” – Modern scientific progress: Geometry and vectors ● 1637 Descartes “Analytic Geometry” – 1844 Graßmann and 1853 Hamilton introduce vectors – 1936 Birkhoff and von Neumann introduce quantum logic – More recently: Hyperdimensional Computing ● Kanerva: Sparse Distributed Memory, Computing with large random vectors – Smolensky, Plate, Gaylor: Vector Symbolic Architectures – Fields: Vector models for NLP, Quantum cognition, ... – Peer Neubert, TU Chemnitz

  4. Historical note See: “Geometry and Meaning” by Dominic Widdows 2004, CSLI Publications, Stanford, ISBN 9781575864488 Ancient Greeks: Roots of geometry ● Plato: geometric theory of creation and elements – Journey and work of Aristotle – Euclid: “Elements of geometry” – Modern scientific progress: Geometry and vectors ● 1637 Descartes “Analytic Geometry” – 1844 Graßmann and 1853 Hamilton introduce vectors – 1936 Birkhoff and von Neumann introduce quantum logic – More recently: Hyperdimensional Computing ● Kanerva: Sparse Distributed Memory, Computing with large random vectors – Smolensky, Plate, Gaylor: Vector Symbolic Architectures – Fields: Vector models for NLP, Quantum cognition, ... – Peer Neubert, TU Chemnitz

  5. Historical note See: “Geometry and Meaning” by Dominic Widdows 2004, CSLI Publications, Stanford, ISBN 9781575864488 Ancient Greeks: Roots of geometry ● Plato: geometric theory of creation and elements – Journey and work of Aristotle – Euclid: “Elements of geometry” – Modern scientific progress: Geometry and vectors ● 1637 Descartes “Analytic Geometry” – 1844 Graßmann and 1853 Hamilton introduce vectors – 1936 Birkhoff and von Neumann introduce quantum logic – More recently: Hyperdimensional Computing ● Kanerva: Sparse Distributed Memory, Computing with large random vectors – Smolensky, Plate, Gaylor: Vector Symbolic Architectures – Fields: Vector models for NLP, Quantum cognition, ... – Peer Neubert, TU Chemnitz

  6. Vector space e.g., n-dimensional real valued vectors ● Intuitive meaning in 1 to 3 dimensional Euclidean spaces ● e.g., position of a book in a rack Peer Neubert, TU Chemnitz

  7. Vector space e.g., n-dimensional real valued vectors ● Intuitive meaning in 1 to 3 dimensional Euclidean spaces ● e.g., position of a book in a rack or bookshelf – Peer Neubert, TU Chemnitz

  8. Vector space e.g., n-dimensional real valued vectors ● Intuitive meaning in 1 to 3 dimensional Euclidean spaces ● e.g., position of a book in a rack, bookshelf or library – Image: Ralf Roletschek / Roletschek.at. Science library of Upper Lusatia in Görlitz, Germany. Peer Neubert, TU Chemnitz

  9. Vector space e.g., n-dimensional real valued vectors ● Intuitive meaning in 1 to 3 dimensional Euclidean spaces ● e.g., position of a book in a rack, bookshelf or library – ? Peer Neubert, TU Chemnitz

  10. Where are such vectors used? Feature vectors, e.g., in computer vision or information retrieval ● (Intermediate) representations in deep ANN ● Vector models for natural language processing ● Memory and storage models, e.g., Pentti Kanerva’s Sparse Distributed ● Memory or Deepmind’s long-short term memory Computational brain models, e.g. Jeff Hawkins’ HTM or Chris Eliasmith’s ● SPAUN Quantum cognition approaches ● ... ● Peer Neubert, TU Chemnitz

  11. Hierarchical Temporal Memory Jeff Hawkins Image source: https://numenta.com/ Peer Neubert, TU Chemnitz

  12. Hierarchical Temporal Memory: Neuron model Peer Neubert, TU Chemnitz

  13. Hierarchical Temporal Memory: Neuron model Peer Neubert, TU Chemnitz

  14. Hierarchical Temporal Memory: Neuron model Sparse Distributed Representations Peer Neubert, TU Chemnitz

  15. Quantum Cognition Not quantum mind (=“the brain works by micro-physical quantum mechanics”) ● A theory that models cognition by the same math that is used to describe ● quantum mechanics Important tool: representation using vector spaces and vector operators (e.g. ● sums and projections) Motivation: Some paradox or irrational judgements of humans can’t be ● explained using classical probability theory and logic, e.g. conjunction and disjunction errors or order effects Busemeyer, J., & Bruza, P. (2012). Quantum Models of Cognition and Decision. Cambridge: Cambridge University Press. doi:10.1017/CBO9780511997716 Peer Neubert, TU Chemnitz

  16. Quantum Cognition Not quantum mind (=“the brain works by micro-physical quantum mechanics”) ● A theory that models cognition by the same math that is used to describe ● quantum mechanics Important tool: representation using vector spaces and operators (e.g. sums ● and projections) Motivation: Some paradox or irrational judgements of humans can’t be ● explained using classical probability theory and logic, e.g. conjunction and disjunction errors or order effects Busemeyer, J., & Bruza, P. (2012). Quantum Models of Cognition and Decision. Cambridge: Cambridge University Press. doi:10.1017/CBO9780511997716 Peer Neubert, TU Chemnitz

  17. Outline: Introduction to high dimensional computing 1) Historical overview 2) High dimensional vector spaces and where they are used 3) Mathematical properties of high dimensional vector spaces 4) Vector Symbolic Architectures or “How to do symbolic computations using vectors spaces” including “What is the Dollar of Mexico?” Peer Neubert, TU Chemnitz

  18. Four properties of high-dimensional vector spaces “The good, the bad, and the ugly” The Trivial The Bad The Surprising The Good Peer Neubert, TU Chemnitz

  19. Properties 1/4: High-dimensional vector spaces have huge capacity The Trivial The Bad The Surprising Capacity grows exponentially ● The Good Here: “high-dimensional” means thousands of dimensions ● This property also holds for other vector spaces than ● Binary, e.g. {0, 1} n , {-1, 1} n – Ternary, e.g. {-1, 0, 1} n – Real, e.g. [-1, 1] n – Sparse or Dense – Peer Neubert, TU Chemnitz

  20. Properties 1/4: High-dimensional vector spaces have huge capacity The Trivial The Bad The Surprising Capacity grows exponentially ● The Good Here: “high-dimensional” means thousands of dimensions ● This property also holds for other vector spaces than ● Binary, e.g. {0, 1} n , {-1, 1} n – Ternary, e.g. {-1, 0, 1} n – Real, e.g. [-1, 1] n – Sparse or Dense – Peer Neubert, TU Chemnitz

  21. Properties 1/4: High-dimensional vector spaces have huge capacity The Trivial The Bad The Surprising Capacity grows exponentially ● The Good Here: “high-dimensional” means thousands of dimensions ● This property also holds for other vector spaces than ● Binary, e.g. {0, 1} n , {-1, 1} n – Ternary, e.g. {-1, 0, 1} n – Real, e.g. [-1, 1] n – Sparse or Dense – Peer Neubert, TU Chemnitz

  22. Properties 1/4: High capacity approx. # atoms in the universe Binary vectors 10 80 Sparsity d is ratio of “1s” 10 60 d=0.01 capacity d=0.03 10 40 d=0.05 d=0.10 dense (d=1) 10 20 10 0 0 500 1000 1500 2000 # dimensions Peer Neubert, TU Chemnitz

  23. Properties 1/4: High capacity approx. # atoms in the universe Binary vectors 10 80 Sparsity d is ratio of “1s” 10 60 d=0.01 capacity d=0.03 10 40 d=0.05 d=0.10 dense (d=1) 10 20 10 0 0 500 1000 1500 2000 # dimensions Peer Neubert, TU Chemnitz

  24. Properties 1/4: High capacity approx. # atoms in the universe Binary vectors 10 80 Sparsity d is ratio of “1s” 10 60 d=0.01 capacity d=0.03 10 40 d=0.05 d=0.10 dense (d=1) 10 20 10 0 0 500 1000 1500 2000 # dimensions Peer Neubert, TU Chemnitz

  25. Properties 1/4: High capacity approx. # atoms in the universe Binary vectors 10 80 Sparsity d is ratio of “1s” 10 60 d=0.01 capacity d=0.03 10 40 d=0.05 d=0.10 dense (d=1) 10 20 10 0 0 500 1000 1500 2000 # dimensions Peer Neubert, TU Chemnitz

  26. Properties 2/4: Nearest neighbor becomes unstable or meaningless The Trivial The Bad The Surprising Downside of so much space: The Good Bellman, 1961: “ Curse of dimensionality ” “Algorithms that work in low dimensional space fail in higher dimensional – spaces” We require exponential amounts of samples to represent space with – statistical significance (e.g., Hastie et al. 2009) Bellman, R. E. (1961) Adaptive Control Processes: A Guided Tour. MIT Press, Cambridge Hastie, Tibshirani and Friedman (2009). The Elements of Statistical Learning (2nd edition)Springer- Verlag Peer Neubert, TU Chemnitz

  27. Properties 2/4: Nearest neighbor becomes unstable or meaningless The Trivial The Bad The Surprising Downside of so much space: The Good Bellman, 1961: “ Curse of dimensionality ” “Algorithms that work in low dimensional space fail in higher dimensional – spaces” We require exponential amounts of samples to represent space with – statistical significance (e.g., Hastie et al. 2009) Bellman, R. E. (1961) Adaptive Control Processes: A Guided Tour. MIT Press, Cambridge Hastie, Tibshirani and Friedman (2009). The Elements of Statistical Learning (2nd edition)Springer- Verlag Peer Neubert, TU Chemnitz

  28. Example: Sorted library The Trivial The Bad The Surprising The Good ● Library contains books about 4 topics ● We can’t infer the topic from the pose directly, only by nearby samples. Peer Neubert, TU Chemnitz

  29. Example: Sorted library The Trivial The Bad The Surprising The Good History Novels Geometry Sports ● Library contains books about 4 topics ● We can’t infer the topic from the pose directly, only by nearby samples. Peer Neubert, TU Chemnitz

  30. Example: Sorted library The Trivial The Bad Query The Surprising The Good History Novels Geometry Sports A single sample per topic ● Library contains books about 4 topics ● We can’t infer the topic from the pose directly, only by nearby samples. Peer Neubert, TU Chemnitz

  31. Example: Sorted library The Trivial The Bad The Surprising Novels Novels Novels The Good Geometry History History History Geometry Geometry Sports Sports Sports The more dimensions, the more samples are required to represent the shape of the clusters. Peer Neubert, TU Chemnitz

  32. Example: Sorted library The Trivial The Bad The Surprising Novels Novels Novels The Good Geometry History History History Geometry Geometry Sports Sports Sports The more dimensions, the more samples are required to represent the shape of the clusters. Peer Neubert, TU Chemnitz

  33. Example: Sorted library The Trivial The Bad The Surprising Novels Novels Novels The Good Geometry History History History Geometry Geometry Sports Sports Sports Peer Neubert, TU Chemnitz

  34. Example: Sorted library Novels Novels Novels Geometry History History History Geometry Geometry Sports Sports Sports The more dimensions, the more samples are required to represent the shape of the clusters. Exponential growth! Peer Neubert, TU Chemnitz

  35. Properties 2/4: Nearest neighbor becomes unstable or meaningless Beyer K, Goldstein J, Ramakrishnan R, Shaft U (1999) When Is nearest neighbor ● meaningful? In: Database theory—ICDT’99. Springer, Berlin, Heidelberg, pp 217–235 The Trivial The Bad The Surprising The Good “under a broad set of conditions (much broader than independent and identically distributed dimensions)” Increasing #dimensions ● Aggarwal CC, Hinneburg A, Keim DA (2001) On the surprising behavior of distance metrics in high dimensional space. In: Database theory—ICDT 2001. Springer, Berlin Heidelberg, pp 420–434 Peer Neubert, TU Chemnitz

  36. Properties 2/4: Nearest neighbor becomes unstable or meaningless Beyer K, Goldstein J, Ramakrishnan R, Shaft U (1999) When Is nearest neighbor ● meaningful? In: Database theory—ICDT’99. Springer, Berlin, Heidelberg, pp 217–235 The Trivial The Bad The Surprising The Good “under a broad set of conditions (much broader than independent and identically distributed dimensions)” Increasing #dimensions ● Aggarwal CC, Hinneburg A, Keim DA (2001) On the surprising behavior of distance metrics in high dimensional space. In: Database theory—ICDT 2001. Springer, Berlin Heidelberg, pp 420–434 Peer Neubert, TU Chemnitz

  37. Properties 2/4: Nearest neighbor becomes unstable or meaningless Beyer K, Goldstein J, Ramakrishnan R, Shaft U (1999) When Is nearest neighbor ● meaningful? In: Database theory—ICDT’99. Springer, Berlin, Heidelberg, pp 217–235 The Trivial The Bad The Surprising The Good “under a broad set of conditions (much broader than independent and identically distributed dimensions)” Increasing #dimensions ● Aggarwal CC, Hinneburg A, Keim DA (2001) On the surprising behavior of distance metrics in high dimensional space. In: Database theory—ICDT 2001. Springer, Berlin Heidelberg, pp 420–434 Peer Neubert, TU Chemnitz

  38. Properties 2/4: Nearest neighbor becomes unstable or meaningless Beyer K, Goldstein J, Ramakrishnan R, Shaft U (1999) When Is nearest neighbor ● meaningful? In: Database theory—ICDT’99. Springer, Berlin, Heidelberg, pp 217–235 The Trivial The Bad The Surprising The Good “under a broad set of conditions (much broader than independent and identically distributed dimensions)” Increasing #dimensions ● Aggarwal CC, Hinneburg A, Keim DA (2001) On the surprising behavior of distance metrics in high dimensional space. In: Database theory—ICDT 2001. Springer, Berlin Heidelberg, pp 420–434 Peer Neubert, TU Chemnitz

  39. The Trivial Properties 3/4: Time to gamble! The Bad The Surprising The Good Peer Neubert, TU Chemnitz

  40. The Trivial Properties 3/4: Experiment The Bad The Surprising The Good Random vectors: ● uniformly distributed angles – obtained by sampling each – dimension iid. ~N(0,1) I bet: ● Given a random vector A, we – can sample a second random vector B and it will be almost orthogonal (+/- 5°) ... … if we are in a 4,000 – dimensional vector space. Peer Neubert, TU Chemnitz

  41. The Trivial Properties 3/4: Experiment The Bad The Surprising The Good Random vectors: ● uniformly distributed angles – obtained by sampling each – dimension iid. ~N(0,1) I bet: ● Given a random vector A, we – can sample a second random vector B and it will be almost orthogonal (+/- 5°) ... … if we are in a 4,000 – dimensional vector space. Peer Neubert, TU Chemnitz

  42. The Trivial Properties 3/4: Experiment The Bad The Surprising The Good Random vectors: ● uniformly distributed angles – obtained by sampling each – A dimension iid. ~N(0,1) I bet: ● Given a random vector A, we can – independently sample a second random vector B and it will be almost orthogonal (+/- 5°) ... … if we are in a 4,000 – dimensional vector space. Peer Neubert, TU Chemnitz

  43. The Trivial Properties 3/4: Experiment The Bad The Surprising The Good Random vectors: ● uniformly distributed angles – obtained by sampling each B – A dimension iid. ~N(0,1) I bet: ● Given a random vector A, we can – independently sample a second random vector B and it will be almost orthogonal (+/- 5°) ... … if we are in a 4,000 – dimensional vector space. Peer Neubert, TU Chemnitz

  44. The Trivial Properties 3/4: Experiment The Bad The Surprising The Good Random vectors: ● uniformly distributed angles – obtained by sampling each B – A dimension iid. ~N(0,1) I bet: ● Given a random vector A, we can – independently sample a second random vector B and it will be almost orthogonal (+/- 5°) ... … if we are in a 4,000 – dimensional vector space. Peer Neubert, TU Chemnitz

  45. The Trivial Properties 3/4: Experiment The Bad The Surprising +/- 5° The Good Random vectors: ● uniformly distributed angles – obtained by sampling each B – A dimension iid. ~N(0,1) I bet: ● Given a random vector A, we can – independently sample a second random vector B and it will be almost orthogonal (+/- 5°) ... … if we are in a 4,000 – dimensional vector space. +/- 5° Peer Neubert, TU Chemnitz

  46. The Trivial Properties 3/4: Experiment The Bad The Surprising +/- 5° The Good Random vectors: ● uniformly distributed angles – obtained by sampling each B – A dimension iid. ~N(0,1) I bet: ● Given a random vector A, we can – independently sample a second random vector B and it will be almost orthogonal (+/- 5°) ... … if we are in a 4,000 – dimensional vector space. +/- 5° Peer Neubert, TU Chemnitz

  47. Properties 3/4: Random vectors are very likely almost orthogonal The Trivial The Bad Random vectors: iid, uniform The Surprising ● The Good Peer Neubert, TU Chemnitz

  48. Properties 3/4: Random vectors are very likely almost orthogonal The Trivial The Bad Random vectors: iid, uniform The Surprising ● The Good Peer Neubert, TU Chemnitz

  49. Properties 3/4: Random vectors are very likely almost orthogonal The Trivial The Bad Random vectors: iid, uniform The Surprising ● The Good Surface areas 7 area (varying unit) Similar 6 Almost orthogonal 5 4 3 2 1 0 0 5 10 15 20 25 # dimensions Peer Neubert, TU Chemnitz

  50. Properties 3/4: Random vectors are very likely almost orthogonal The Trivial The Bad Random vectors: iid, uniform The Surprising ● The Good Surface areas Surface area of unit n-sphere 7 35 area (varying unit) area (varying unit) Similar 6 30 Almost orthogonal 5 25 4 20 3 15 2 10 1 5 0 0 0 5 10 15 20 25 0 5 10 15 20 25 # dimensions # dimensions Peer Neubert, TU Chemnitz

  51. Properties 3/4: Random vectors are very likely almost orthogonal Random vectors: iid, uniform ● Ratio of surface areas 10 25 Surface areas 7 Similar area (varying unit) Similar 10 20 6 Almost orthogonal /A 5 A Almost orthogonal 10 15 4 10 10 3 2 10 5 1 10 0 0 0 5 10 15 20 25 0 5 10 15 20 25 # dimensions # dimensions Peer Neubert, TU Chemnitz

  52. Properties 3/4: Random vectors are very likely almost orthogonal Random vectors: iid, uniform ● Random sampling probability Random sampling probability (extended) 0.4 1 0.8 probability 0.3 probability 0.6 0.2 0.4 0.1 0.2 Similar Similar Almost orthogonal Almost orthogonal 0 0 0 200 400 600 800 1000 0 5 10 15 20 25 # dimensions # dimensions Peer Neubert, TU Chemnitz

  53. Properties 4/4: Noise has low influence on nearest neighbor queries with random vectors The Trivial The Bad The Surprising Example 1: ● 1. One million random The Good feature vectors [0,1] d 2. query: noisy measurements of feature vectors Database 1 Mio vectors What is the probability to get the wrong query answer? Peer Neubert, TU Chemnitz

  54. Properties 4/4: Noise has low influence on nearest neighbor queries with random vectors The Trivial The Bad The Surprising Example 1: ● 1. One million random The Good feature vectors [0,1] d 2. query: noisy measurements of feature vectors Database 1 Mio vectors What is the probability to get the wrong query answer? Peer Neubert, TU Chemnitz

  55. Properties 4/4: Noise has low influence on nearest neighbor queries with random vectors The Trivial The Bad The Surprising Example 1: ● 1. One million random The Good feature vectors [0,1] d 2. query: noisy measurements of feature vectors Database 1 Mio vectors What is the probability to get the wrong query answer? Peer Neubert, TU Chemnitz

  56. Properties 4/4: Noise has low influence on nearest neighbor queries with random vectors Example 1: ● Probability of wrong query answer 1 =0.10 noise =0.25 0.8 noise =0.50 noise 0.6 0.4 0.2 0 0 50 100 150 200 # dimensions Peer Neubert, TU Chemnitz

  57. Properties 4/4: Noise has low influence on nearest neighbor queries with random vectors Example 1: ● Probability of wrong query answer 1 2 =0.10 noise =0.25 1.5 0.8 noise =0.50 noise 1 0.6 value 0.5 0.4 0 0.2 -0.5 -1 0 0 50 100 150 200 0 50 100 150 200 dimension index # dimensions Peer Neubert, TU Chemnitz

  58. Properties 4/4: Noise has low influence on nearest neighbor queries with random vectors Example 2: ● 1. One million random feature vectors [0,1] d 2. query: noisy measurements of feature vectors Database 1 Mio vectors What if the noise-vector is again a database vector? Peer Neubert, TU Chemnitz

  59. Properties 4/4: Noise has low influence on nearest neighbor queries with random vectors Example 2: ● 1. One million random feature vectors [0,1] d 2. query: sum of feature vectors Database 1 Mio vectors How many database vectors can we add (=bundle) and still get exactly all the added vectors as answer? Peer Neubert, TU Chemnitz

  60. Properties 4/4: Noise has low influence on nearest neighbor queries with random vectors Example 2: ● How many database vectors can we add (=bundle) and still get exactly – all the added vectors as answer? Probability of wrong query answer [0,1] d 1 k=2 k=3 0.8 k=4 k=5 0.6 0.4 0.2 0 0 200 400 600 800 1000 Peer Neubert, TU Chemnitz # dimensions

  61. Properties 4/4: Noise has low influence on nearest neighbor queries with random vectors Example 2: ● How many database vectors can we add (=bundle) and still get exactly – all the added vectors as answer? Probability of wrong query answer Probability of wrong query answer [0,1] d [-1,1] d 1 1 k=2 k=2 k=3 k=3 0.8 0.8 k=4 k=4 k=5 k=5 k=6 0.6 0.6 k=7 k=8 k=9 0.4 0.4 k=10 0.2 0.2 0 0 0 200 400 600 800 1000 0 200 400 600 800 1000 # dimensions Peer Neubert, TU Chemnitz # dimensions

  62. Example application: Object recognition query Database Peer Neubert, TU Chemnitz

  63. Example application: Object recognition query Database Peer Neubert, TU Chemnitz

  64. Example application: Object recognition + query Database Peer Neubert, TU Chemnitz

  65. Example application: Object recognition + 1 0.9 0.8 Individual Accuracy Bundle 0.7 query Static B4 Database Static B8 0.6 0.5 0.4 0 45 90 135 180 Angular distance of query to known vectors Peer Neubert, TU Chemnitz

  66. Credits: How to store structured data? Pentti Kanerva Given are 2 records: Mexico Name : United States Name : USA Mexico of America Capital City : Washington DC Capital City : Mexico City Currency : Dollar Currency : Peso Question : What is the Dollar of Mexico? fillers roles Peer Neubert, TU Chemnitz

  67. Credits: How to store structured data? Pentti Kanerva Given are 2 records: Mexico Name : United States Name : USA Mexico of America Capital City : Washington DC Capital City : Mexico City Currency : Dollar Currency : Peso Question : What is the Dollar of Mexico? fillers roles Hyperdimensional computing approach: 1. Assign a random high-dimensional vector to each entity ”Name” is a random vector NAM ”USA” is a random vector USA ”Capital city” is a random vector CAP … 2. Calculate a single high-dimensional vector that contains all information F = (NAM*USA+CAP*WDC+CUR*DOL)*(NAM*MEX+CAP*MCX+CUR*PES) 3. Calculate the query answer: F*DOL ~ PES Peer Neubert, TU Chemnitz

  68. Vector Symbolic Architectures (VSA) VSA = high dimensional vector space + operations ● Operations in a VSA: ● Bind() Pentti Kanerva. 2009. Hyperdimensional – Computing: An Introduction to Computing in Distributed Representation with High-Dimensional Bundle() – Random Vectors . Cognitive Computation 1, 2 (2009), 139–159. https://doi.org/10.1007/s12559- Permute()/Protect() – 009-9009-8 Term: Gayler RW (2003) Vector symbolic architectures Additionally ● answer Jackendoff’s challenges for cognitive neuroscience. In: Proc. of ICCS/ASCS Int. Conf. on Encoding/decoding – cognitive science, pp 133–138. Sydney, Australia Clean-up memory – Peer Neubert, TU Chemnitz

  69. Vector Symbolic Architectures (VSA) VSA = high dimensional vector space + operations ● Operations in a VSA: ● Bind() Pentti Kanerva. 2009. Hyperdimensional – Computing: An Introduction to Computing in Distributed Representation with High-Dimensional Bundle() – Random Vectors . Cognitive Computation 1, 2 (2009), 139–159. https://doi.org/10.1007/s12559- Permute()/Protect() – 009-9009-8 Term: Gayler RW (2003) Vector symbolic architectures Additionally ● answer Jackendoff’s challenges for cognitive neuroscience. In: Proc. of ICCS/ASCS Int. Conf. on Encoding/decoding – cognitive science, pp 133–138. Sydney, Australia Clean-up memory – Peer Neubert, TU Chemnitz

  70. Vector Symbolic Architectures (VSA) VSA = high dimensional vector space + operations ● Operations in a VSA: ● Bind() Pentti Kanerva. 2009. Hyperdimensional – Computing: An Introduction to Computing in Distributed Representation with High-Dimensional Bundle() – Random Vectors . Cognitive Computation 1, 2 (2009), 139–159. https://doi.org/10.1007/s12559- Permute()/Protect() – 009-9009-8 Term: Gayler RW (2003) Vector symbolic architectures Additionally ● answer Jackendoff’s challenges for cognitive neuroscience. In: Proc. of ICCS/ASCS Int. Conf. on Encoding/decoding – cognitive science, pp 133–138. Sydney, Australia Clean-up memory – Database Peer Neubert, TU Chemnitz

  71. Vector Symbolic Architectures (VSA) VSA = high dimensional vector space + operations ● Operations in a VSA: ● Bind() Pentti Kanerva. 2009. Hyperdimensional – Computing: An Introduction to Computing in Distributed Representation with High-Dimensional Bundle() – Random Vectors . Cognitive Computation 1, 2 (2009), 139–159. https://doi.org/10.1007/s12559- Permute()/Protect() – 009-9009-8 Term: Gayler RW (2003) Vector symbolic architectures Additionally ● answer Jackendoff’s challenges for cognitive neuroscience. In: Proc. of ICCS/ASCS Int. Conf. on Encoding/decoding – cognitive science, pp 133–138. Sydney, Australia Clean-up memory – Peer Neubert, TU Chemnitz

  72. VSA operations Bundling + ● Goal: combine two vectors, such that – the result is similar to both vectors ● Application: superpose information – Binding ⊗ ● Goal: combine two vectors, such that – the result is nonsimilar to both vectors ● one can be recreated from the result using the other ● Application: bind value “a” to variable “x” (or a “filler” to a “role” or ...) – Peer Neubert, TU Chemnitz

  73. VSA operations Bundling + ● Goal: combine two vectors, such that – the result is similar to both vectors ● Application: superpose information – Binding ⊗ ● Goal: combine two vectors, such that – the result is nonsimilar to both vectors ● one can be recreated from the result using the other ● Application: bind value “a” to variable “x” (or a “filler” to a “role” or ...) – Peer Neubert, TU Chemnitz

  74. VSA operations ● Binding ⊗ – Properties ● Associative, commutative ● Self-inverse: X ⊗ X=I (or additional unbind operator) ● Result nonsimilar to input – Example: ● Hypervector space {-1,1} D ● bind can be elementwise multiplication – Application: bind value “a” to variable “x” (or a “filler” to a “role” or ...) ● Bind: – x = a → H = X ⊗ A ● Unbind: – x = ? ⊗ → X H = X ⊗ (X ⊗ A) = A Peer Neubert, TU Chemnitz

  75. VSA operations ● Binding ⊗ – Properties ● Associative, commutative ● Self-inverse: X ⊗ X=I (or additional unbind operator) ● Result nonsimilar to input – Example: ● Hypervector space {-1,1} D ● binding can be elementwise multiplication – Application: bind value “a” to variable “x” (or a “filler” to a “role” or ...) ● Bind: – x = a → H = X ⊗ A ● Unbind: – x = ? ⊗ → X H = X ⊗ (X ⊗ A) = A Peer Neubert, TU Chemnitz

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Motivation High Dimensional Issues Subspace Clustering Full Dimensional Clustering Issues

Motivation High Dimensional Issues Subspace Clustering Full Dimensional Clustering Issues

Motivation High Dimensional Issues Subspace Clustering Full Dimensional Clustering Issues Accuracy Issues Andrew Foss PhD Candidate Database Lab, Dept. of Computing Science University of Alberta For CMPUT 695 March 2007 Curse

457 views • 12 slides
. . . 1 / 5 The curse of dimensionality . many applications require high dimensional data .

. . . 1 / 5 The curse of dimensionality . many applications require high dimensional data .

The curse of dimensionality . . . 1 / 5 The curse of dimensionality . many applications require high dimensional data . . 1 / 5 The curse of dimensionality . many applications require high dimensional data many algorithms become

447 views • 12 slides
COMPUTING with HIGH-DIMENSIONAL VECTORS Pentti Kanerva UC Berkeley, Redwood Center for

COMPUTING with HIGH-DIMENSIONAL VECTORS Pentti Kanerva UC Berkeley, Redwood Center for

COMPUTING with HIGH-DIMENSIONAL VECTORS Pentti Kanerva UC Berkeley, Redwood Center for Theoretical Neuroscience Stanford, CSLI pkanerva@csli.stanfrod.edu . Motivation and Background . What is HD Computing? . Example from Language . HD

532 views • 49 slides
What can we say about high- dimensional objects from a low-dimensional representation? 2

What can we say about high- dimensional objects from a low-dimensional representation? 2

Dimension Reduction with Certainty Rasmus Pagh IT University of Copenhagen S CALABLE S IMILARITY ECML-PKDD S EARCH September 21, 2016 Slides: goo.gl/hZoRWo 1 What can we say about high- dimensional objects from a low-dimensional

940 views • 75 slides
The curse of dimensionality Julie Delon Laboratoire MAP5, UMR CNRS 8145 Universit Paris

The curse of dimensionality Julie Delon Laboratoire MAP5, UMR CNRS 8145 Universit Paris

The curse of dimensionality Julie Delon Laboratoire MAP5, UMR CNRS 8145 Universit Paris Descartes up5.fr/delon 1 Introduction Modern data are often high dimensional. computational biology: DNA, few observations and huge number of

775 views • 51 slides
High Dimensional Data, Covariance Matrices High Dimensional Data Examples and Application to

High Dimensional Data, Covariance Matrices High Dimensional Data Examples and Application to

Outline Motivation High Dimensional Data, Covariance Matrices High Dimensional Data Examples and Application to Genetics Theoretical Underpinnings Random Matrices Shrinkage Estimation Decision Theory Samprit Banerjee, Ph.D Bayesian

1.34k views • 80 slides
Statistics for High-Dimensional Data: Selected Topics Peter B uhlmann Seminar f ur

Statistics for High-Dimensional Data: Selected Topics Peter B uhlmann Seminar f ur

Statistics for High-Dimensional Data: Selected Topics Peter B uhlmann Seminar f ur Statistik, ETH Z urich June 2014 High-dimensional linear model: basic concepts in methodology, theory and computation High-dimensional data

998 views • 48 slides
High Dimensional Data Alark Joshi High dimensional data Data with multiple dimensions,

High Dimensional Data Alark Joshi High dimensional data Data with multiple dimensions,

High Dimensional Data Alark Joshi High dimensional data Data with multiple dimensions, multiple variables or multiple attributes Cars dataset Economy Cylinders Displacement Power Weight Mph Year Scatterplots

1.12k views • 65 slides
CONNECTIVITY RESEARCH INTERN, BANKS LAB FROM HIGH- DEPARTMENT OF DIMENSIONAL ANESTHESIOLOGY

CONNECTIVITY RESEARCH INTERN, BANKS LAB FROM HIGH- DEPARTMENT OF DIMENSIONAL ANESTHESIOLOGY

INFERRING EFFECTIVE CHRIS ENDEMANN CONNECTIVITY RESEARCH INTERN, BANKS LAB FROM HIGH- DEPARTMENT OF DIMENSIONAL ANESTHESIOLOGY ECOG UW MADISON, SMPH RECORDINGS THE BRAIN AS A NETWORK OF SPECIALIZED COMPUTING COMPARTMENTS

370 views • 18 slides
High-Dimensional Nearest Neighbor Search High-Dimensional Nearest Neighbor Search Who?

High-Dimensional Nearest Neighbor Search High-Dimensional Nearest Neighbor Search Who?

High-Dimensional Nearest Neighbor Search High-Dimensional Nearest Neighbor Search Who? About Cliqz and me What? Problem statement Why? Applications How? Exact solutions in low dimensions Approximate

690 views • 19 slides
High-dimensional and infinite-dimensional hyperbolic crosses and their applications in

High-dimensional and infinite-dimensional hyperbolic crosses and their applications in

High-dimensional and infinite-dimensional hyperbolic crosses and their applications in approximation and uncertainty quantification Dinh D ung Vietnam National University, Hanoi, Vietnam Workshop on Information-Based Complexity and

996 views • 42 slides
A review of dimensionality reduction in high-dimensional data using multi-core and many-core

A review of dimensionality reduction in high-dimensional data using multi-core and many-core

Second Workshop on Software Challenges to Exascale Computing (13 th , 14 th December 2018, New Delhi) A presentation on A review of dimensionality reduction in high-dimensional data using multi-core and many-core architecture by Mr.

604 views • 39 slides
Three-dimensional Radial Visualization of High-dimensional Continuous or Discrete Datasets Fan

Three-dimensional Radial Visualization of High-dimensional Continuous or Discrete Datasets Fan

Three-dimensional Radial Visualization of High-dimensional Continuous or Discrete Datasets Fan Dai, Yifan Zhu and Ranjan Maitra Department of Statistics Iowa State University {fd43,yifanzhu,maitra} @ iastate.edu Motivation Multivariate

640 views • 34 slides
High-Dimensional Function Approximation for Knowledge-Free Reinforcement Learning: a Case Study

High-Dimensional Function Approximation for Knowledge-Free Reinforcement Learning: a Case Study

High-Dimensional Function Approximation for Knowledge-Free Reinforcement Learning: a Case Study in SZ-Tetris Wojciech Jakowski Marcin Szubert Pawel Liskowski Krysztof Krawiec Institute of Computing Science July 14, 2015 Introduction RL

405 views • 22 slides
Oracle-based algorithms for high-dimensional polytopes Vissarion Fisikopoulos Joint work with

Oracle-based algorithms for high-dimensional polytopes Vissarion Fisikopoulos Joint work with

Oracle-based algorithms for high-dimensional polytopes Vissarion Fisikopoulos Joint work with I.Z. Emiris (UoA), B. G artner (ETHZ) Dept. of Informatics & Telecommunications, University of Athens Workshop on Geometric Computing,

433 views • 28 slides
Deep Neural Network Mathematical Mysteries for High Dimensional Learning Stphane Mallat

Deep Neural Network Mathematical Mysteries for High Dimensional Learning Stphane Mallat

Deep Neural Network Mathematical Mysteries for High Dimensional Learning Stphane Mallat cole Normale Suprieure www.di.ens.fr/data High Dimensional Learning High-dimensional x = ( x (1) , ..., x ( d )) R d : Classification:

1.34k views • 87 slides
5-Axis Machining Some Best Practices Longxiang Yang FANUC America IMTS 2018 Conference

5-Axis Machining Some Best Practices Longxiang Yang FANUC America IMTS 2018 Conference

5-Axis Machining Some Best Practices Longxiang Yang FANUC America IMTS 2018 Conference September 11, 2018 Full Utilization of 5-axis Machines 5-axis machines were developed and used in production more than 50 years Because of

868 views • 37 slides
Vectors Slide 2 / 36 Scalar versus Vector A scalar has only a physical quantity such as mass,

Vectors Slide 2 / 36 Scalar versus Vector A scalar has only a physical quantity such as mass,

Slide 1 / 36 Vectors Slide 2 / 36 Scalar versus Vector A scalar has only a physical quantity such as mass, speed, and time. A vector has both a magnitude and a direction associated with it, such as velocity and acceleration. A vector is

733 views • 36 slides
Support Vector Manifold Learning for Solving Regression Problems via Clustering Marcin Orchel

Support Vector Manifold Learning for Solving Regression Problems via Clustering Marcin Orchel

Support Vector Manifold Learning for Solving Regression Problems via Clustering Marcin Orchel AGH University of Science and Technology in Poland 1 / 29 2 / 29 1.0 y -1.0 1.0 x (a) 3 / 29 new kernel c )) T ( ( ( ( x i ) +

878 views • 29 slides
GloVe: Global Vectors for Word Representation Fengyang Zhang, Yutong Wang Presentation Overview

GloVe: Global Vectors for Word Representation Fengyang Zhang, Yutong Wang Presentation Overview

GloVe: Global Vectors for Word Representation Fengyang Zhang, Yutong Wang Presentation Overview 2. GloVe 1. What Model 3. Demo 4. Result is GloVe? Inference What is GloVe? Word Embedding: Word embeddings are in fact a class of

695 views • 14 slides
An Empirical Comparison of Text Categorization Methods Ana Cardoso-Cachopo and Arlindo L.

An Empirical Comparison of Text Categorization Methods Ana Cardoso-Cachopo and Arlindo L.

An Empirical Comparison of Text Categorization Methods Ana Cardoso-Cachopo and Arlindo L. Oliveira acardoso@gia.ist.utl.pt and aml@inesc-id.pt Instituto Superior T ecnico / ALGOS-INESC-ID An Empirical Comparison of Text Categorization

300 views • 16 slides
Introduction to R Nishant Gopalakrishnan, Martin Morgan Fred Hutchinson Cancer Research Center

Introduction to R Nishant Gopalakrishnan, Martin Morgan Fred Hutchinson Cancer Research Center

Introduction to R Nishant Gopalakrishnan, Martin Morgan Fred Hutchinson Cancer Research Center 19-21 January, 2011 Getting Started Atomic Data structures Creating vectors Subsetting vectors Factors Matrices and arrarys Lists, data frames,

1.07k views • 48 slides
2017 Presentation Descriptions Thursday November 16, 2017 Risk, Resilience, and Other Related

2017 Presentation Descriptions Thursday November 16, 2017 Risk, Resilience, and Other Related

2017 Presentation Descriptions Thursday November 16, 2017 Risk, Resilience, and Other Related Variables: What Does Research Tell Us About the Relationships? Alan Ewert, PhD., Indiana University This presentation will seek to provide a research

267 views • 3 slides
The Principles of Regulation for Vector Control Products Vector Control Team Contents 1. The

The Principles of Regulation for Vector Control Products Vector Control Team Contents 1. The

The Principles of Regulation for Vector Control Products Vector Control Team Contents 1. The global challenge to Human Health and benefits of Vector Control technology 2. Regulation as the key to addressing the challenge 3. The Principles

414 views • 22 slides

More recommend