Efficient Bayesian inference with Hamiltonian Monte Carlo -- Michael Betancourt (Part 1)

Not an expert in this field, but I'm guessing this is related to unintuitive nature of geometry in high-dimensional spaces. One rough example I can think of is the fact that the number of ways to move away from the origin increases exponentially (or even faster?) as the dimensionality goes up. There is _way_ more volume away from the origin than near it, I've seen this explained as something like "most of the volume of a high-dimensional orange is in the peel". One result of this is the fact that samples of a standard gaussian end up forming a "shell" as opposed to a "ball" that you would expect (this phenomenon is called the concentration of measure in general). Also, very roughly, high-dimensional objects have lots of corners, and these corners are also very sharp. I would guess that gradient descent would get stuck in these corners and have a hard time getting out. Some links related to this: - Spikey Spheres: [link] - Thinking outside the 10-dimensional box - 3blue1brown: [link] - This is a fairly long talk about HMC, but it does talk about some problems that come up when sampling high-dimensional distributions: [this video]