[PR12] Generative Models as Distributions of Functions

Generative Models as Distributions
of Functions
PR12와 함께 이해하는
Jaejun Yoo
(current) Postdoc. @EPFL
(from July) Assistant Prof., @UNIST
PR-312, 11th April, 2021

Today’s contents
“For all datasets, we use an MLP with 3
hidden layers of size 128 … and an MLP
with 2 hidden layers of size 256 and 512”
“We performed all training on a single
2080Ti GPU with 11GB of RAM.”

Motivation and Main Problem
“Conventional signal representations are usually discrete.”
However, Mother Nature is continuous!
(well… up to planck constant…?)
2D Images Audio 3D Shapes

Of course, these functions are usually not analytically tractable. it is impossible to "write down"
the function that parameterizes a natural image as a mathematical formula.
Continuous representation?
Why hard?

Why important?
• independent of spatial resolution (infinite resolution)
• Geometric transformation of images: zoom, rotation, super-resolution.
• Derivatives are well-defined.

Why important?
Piecewise Constant Bilinear Cubic Spline

Why important?
Piecewise constant Bilinear Cubic Spline

Why important?

Spline: Variational properties
Explicit derivative!

Spline: Variational properties

Continuous representation?
• DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation (Park et al. 2019)
• Occupancy Networks: Learning 3D Reconstruction in Function Space (Mescheder et al. 2019)
• IM-Net: Learning Implicit Fields for Generative Shape Modeling (Chen et al. 2018)
• … NeRF (PR-302)…
“Implicit Neural Representations approximate this function via a neural network!”
Implicit Neural Representation!

Implicit Neural Representation
- Remarkably, the representation !" is independent of
the number of pixels. The representation !" therefore,
unlike most image representations, does not depend
on the resolution of the image.
- The core property of these representations is that
they scale with signal complexity and not with
signal resolution.

Learning Distributions of Functions
1. Parameterizing a distribution over neural
networks with a hypernetwork (Ha et al., 2017)
Overall Scheme
“Sample the weights of a neural network”
to obtain a function.
: Learning a distribution over functions !" is equivalent to
learning a distribution over weights #(%).
: Then, #(%), where % = () * , is refer to as a neural
function distribution (NFD).

Overall Scheme
“Sample the weights of a neural network”
to obtain a function.
: Learning a distribution over functions !" is equivalent to
learning a distribution over weights #(%).
: Then, #(%), where % = () * , is refer to as a neural
function distribution (NFD).
However! How do we get access to the
ground truth functions to train the network?

Overall Scheme
“We do have access to input/output
pairs of these functions through the
coordinates and features, allowing us to
learn function distributions without
operating directly on the functions!”

Overall Scheme
2. Training this distribution with an adversarial
approach (Goodfellow et al., 2014).

Overall Scheme
2. Training this distribution with an adversarial
approach (Goodfellow et al., 2014).
* is a kind of position encoding (Fourier feature).

Overall Scheme
NFD
Now we know how to design a network to learn continuous functions!

Overall Scheme
Discriminator
But, the data we consider may not necessarily lie on a grid…

Overall Scheme
Discriminator
… in which case it is not possible to use convolutional discriminators.

Overall Scheme
Discriminator
Our discriminator should be able to distinguish between
real and fake sets of coordinate and feature pairs.

Point Cloud Discriminator
Point Convolution
In contrast to regular convolutions,
where the convolution kernels are only
defined at certain grid locations, the
convolution filters in PointConv are
parameterized by an MLP mapping
coordinates to kernel values:

Experiments
“For all datasets, we use an MLP with 3
hidden layers of size 128 … and an MLP
with 2 hidden layers of size 256 and 512”
“We performed all training on a single
2080Ti GPU with 11GB of RAM.”
“Remarkably, such a simple architecture
is sufficient for learning rich distributions
of images and 3D shapes.“
“Use the exact same model for both
images and 3D shapes except for the
input and output dimensions of the
function representation.”
Implementation Setups

Results
2D Image generation
• Samples from our model trained on CelebAHQ.
• 64×64 (top) and 128×128 (bottom)
• Each image corresponds to a function which
was sampled from our model and then
evaluated on the grid.
• To produce this figure we sampled 5 batches
and chose the best batch by visual inspection.

Results
“To the infinity and beyond!”
- Buzz Lightyear, Toy Story
Super-resolution
NFD
64×64
NFD
256×256
Bicubic
256×256
NFD
28×28
NFD
256×256
Bicubic
256×256

Results
3D shapes
Voxel grids from Choy et al. (2016) representing the chairs category from the ShapeNet (Chang et al.,
2015) dataset. The dataset contains 6778 chairs each of dimension 32#
. For each 3D model, uniformly
subsample K = 4096 points among 32# = 32,768 points and use them for training.

• A step towards making implicit neural representation methods genuinely useful
for modeling datasets rather than individual data points.
• The first framework to model data of this complexity in an entirely continuous
fashion.
• The ability of being independent to resolution and operating outside of a grid.
• A unique way of using point cloud discriminators.
Conclusion
Summary of Contributions (I think)

Things to discuss about…
• What kinds of study would be derived from this?
• Architectural developments (better quality)?
• Then How? Or what would be helpful?
• Other applications?
• Again, compute-driven AI vs human-knowledge based?
• Big model vs inductive bias?
• Etc.?

[PR12] Generative Models as Distributions of Functions

Recommended

More Related Content

What's hot (20)

Similar to [PR12] Generative Models as Distributions of Functions (20)

More from JaeJun Yoo (12)

Recently uploaded (20)

[PR12] Generative Models as Distributions of Functions