Two things stand our for me: Tammet's quote from Einstein, "make things as simple as possible, but no simpler" and his observation that, in its manifold dimensions, mathematics works as a kind of literature.
Many mathematicians have said that a hallmark of a particularly fruitful set of equations is beauty. Tammet, a synesthete, sees the numbers as shapes and colors, which, as he points out in the video here, only heightens the raw experience for him.
"Simple" can also come in effect, which poets have been exploited since language first emerged from our lips. So when Robert Duncan says the sky is "diseased with stars," he's saying a particularly kind thing, that the sky has the majesty - and the finitude - of a living thing. Forged in those furnaces, so do we. "Simple" to a sculpture or painter might mean removing the excess from a theme or object or the world. Done with skill the distillation produces strong drink. And pushed too far, of course, over-simplfiying produces a caricature, or worse, banalities. What's left ceases to describe so much as mock.
While I'll never be able to take in the topologies that describe theoretical physics at its extremes, I've always wondered what it must feel like to look at a set of equations and see a universe, to see a particular beauty that would lead one to conclude that this set of encoded logic would be more likely than another to find its expression in the crushing depths of a black hole, or in the vanishingly small warp and woof of quantum dimension. It must be endlessly satisfying.
The video is only a few minutes long. I'd encourage you to watch it. You might also be interested in this video from the World Science Festival, in which a panel, including Brian Greene and Robert Krulwich, talk about the creative nature of mathematics.