(iterate think thoughts): Painting with Gaussians
Summary
Dmitri builds a digital painter that uses 2D Gaussian splats guided by structure tensors and Haar wavelets to emulate brush strokes instead of using slow gradient descent.
Gaussians model brush strokes.
A 2D Gaussian's mean, covariance, color, and opacity map to stroke position, orientation, elongation, and translucency, allowing natural over-compositing.
The covariance can be used to encode the brush direction and its elongation
Structure tensors provide orientation.
Eigenvectors from a Di Zenzo color tensor determine stroke direction and coherence, ensuring strokes follow contours and handle isoluminant color edges.
The minor eigenvector points along the edge, giving the direction of the brush stroke.
Wavelets determine stroke density.
A multi-scale 2D Haar decomposition creates a luma-relative detail map, placing more small strokes in textured regions and fewer in flat areas.
the wavelet identifies the density map needed to drive adaptive placement.
Wang hashes break regularity.
Replacing stratified grids with Wang avalanche hashes provides white-noise coordinates, preventing the diagonal stripes and periodic artifacts caused by linear hashes.
This method gives true white-noise coordinates with no periodicity.
Perlin noise adds organic flow.
Blending the structure tensor's angle with a Perlin vector field introduces hand-like wavering, especially in low-coherence flat regions.
each stroke's orientation can be derived by blending the structure tensor's edge angle with the Perlin flow angle
Highlights
It turns out that a 2D Gaussian splat maps onto this idea surprisingly well. It has a mean which is where the stroke lands, a
The major eigenvector points across the contour, providing the direction of the strongest gradient. The minor eigenvector points along the edge, giving the direction of
My idea was to emulate a physical painting process where large splats can be used to define general color regions, and then to layer progressively
What we need here, again, is a luma-relative detail map where each cell's energy is divided by its local mean brightness plus a fraction of
Read the original ↗Save to your Sponge
Summarized with Sponge. Quotes are the author’s own words.