← @bgSponge

(iterate think thoughts): Painting with Gaussians

Summary

Dmitri builds a digital painter that uses 2D Gaussian splats guided by structure tensors and Haar wavelets to emulate brush strokes instead of using slow gradient descent.

  1. Gaussians model brush strokes.

    A 2D Gaussian's mean, covariance, color, and opacity map to stroke position, orientation, elongation, and translucency, allowing natural over-compositing.

    The covariance can be used to encode the brush direction and its elongation

  2. Structure tensors provide orientation.

    Eigenvectors from a Di Zenzo color tensor determine stroke direction and coherence, ensuring strokes follow contours and handle isoluminant color edges.

    The minor eigenvector points along the edge, giving the direction of the brush stroke.

  3. Wavelets determine stroke density.

    A multi-scale 2D Haar decomposition creates a luma-relative detail map, placing more small strokes in textured regions and fewer in flat areas.

    the wavelet identifies the density map needed to drive adaptive placement.

  4. Wang hashes break regularity.

    Replacing stratified grids with Wang avalanche hashes provides white-noise coordinates, preventing the diagonal stripes and periodic artifacts caused by linear hashes.

    This method gives true white-noise coordinates with no periodicity.

  5. Perlin noise adds organic flow.

    Blending the structure tensor's angle with a Perlin vector field introduces hand-like wavering, especially in low-coherence flat regions.

    each stroke's orientation can be derived by blending the structure tensor's edge angle with the Perlin flow angle

Highlights

  • It turns out that a 2D Gaussian splat maps onto this idea surprisingly well. It has a mean which is where the stroke lands, a

  • The major eigenvector points across the contour, providing the direction of the strongest gradient. The minor eigenvector points along the edge, giving the direction of

  • My idea was to emulate a physical painting process where large splats can be used to define general color regions, and then to layer progressively

  • What we need here, again, is a luma-relative detail map where each cell's energy is divided by its local mean brightness plus a fraction of

Read the original ↗Save to your Sponge

Summarized with Sponge. Quotes are the author’s own words.

More from @bg

All readings →