lucent.optvis package

The lucent.optvis package is where all the actual feature visualization magic happens.

Here we contain objective functions, image parameterizations and util methods.

If you want to write your own objective, you can do so by using the wrap_objective decorator and copy the behavior of the existing objective functions. For a more detailed explanation, see Custom Objectives.

optvis.param

lucent.optvis.param package

optvis.objectives module

class lucent.optvis.objectives.Objective(objective_func: Callable[[Module], Any], name: Optional[str] = '', description: Optional[str] = '')

Bases: object

Base class for objectives. Implements basic arithmetic on objectives.

static sum(objs: Iterable[Objective]) → Objective

Alternative to sum(objs) which would return a nested description Sum(d1 + Sum(d2 + Sum(…))) for descriptions d_i which is unreadable. Using this method will produce description Sum(d1 + d2 + …) instead. To call this, do Objective.sum(objs).

Parameters: objs (Iterable[Objective]) – The objectives that should be summed.
Returns: New Objective instance with the sum of the objective functions as objective function.
Return type: Objective

lucent.optvis.objectives.alignment(layer: str, decay_ratio: Optional[float] = 2) → Objective

Encourage neighboring images to be similar. When visualizing the interpolation between two objectives, it’s often desirable to encourage analogous objects to be drawn in the same position, to make them more comparable. This term penalizes L2 distance between neighboring images, as evaluated at layer. In general, we find this most effective if used with a parameterization that shares across the batch. (In fact, that works quite well by itself, so this function may just be obsolete.)

Parameters

layer (str) – layer to penalize at.
decay_ratio (Optional[float], optional) – how much to decay penalty as images move apart in batch., defaults to 2

Returns

Objective to optimize input towards alignment across batch dimension

Return type

Objective

lucent.optvis.objectives.as_objective(obj: Union[Objective, Callable, str]) → Union[Callable, Objective]

Strings of the form “layer:n” become the Objective channel(layer, n). Objectives and Callables are returned unchanged.

Parameters: obj (Union[Objective, Callable, str]) – Objective, Callable or layer:channel string
Returns: Objective instance or Callable representing the objective function
Return type: Union[Callable, Objective]

lucent.optvis.objectives.blur_input_each_step() → Objective

Minimizing this objective is equivelant to blurring input each step. Optimizing (-k)*blur_input_each_step() is equivalant to: input <- (1-k)*input + k*blur(input) An operation that was used in early feature visualization work. See Nguyen, et al., 2015.

Returns: Objective which optimizes input to be blurred.
Return type: Objective

lucent.optvis.objectives.channel(layer: str, n_channel: int, batch: Optional[int] = None) → Objective

Visualize a single channel

Parameters

layer (str) – Name of the layer
n_channel (int) – Channel number
batch (Optional[int], optional) – which position at the batch dim of the image tensor this objective is applied to, defaults to None

Returns

Objective to optimize input for a single channel

Return type

Objective

lucent.optvis.objectives.channel_interpolate(layer1: str, n_channel1: int, layer2: str, n_channel2: int) → Objective

Interpolate between layer1, n_channel1 and layer2, n_channel2. Optimize for a convex combination of layer1, n_channel1 and layer2, n_channel2, transitioning across the batch.

Parameters

layer1 (str) – layer to optimize 100% at batch=0.
n_channel1 (int) – neuron index to optimize 100% at batch=0.
layer2 (str) – layer to optimize 100% at batch=N.
n_channel2 (int) – neuron index to optimize 100% at batch=N.

Returns

Objective to optimizes input towards the channel interpolation between the given channels

Return type

Objective

lucent.optvis.objectives.channel_weight(layer: str, weight: Tensor, batch: Optional[int] = None) → Objective

Linearly weighted channel activation as objective

Parameters

layer (str) – Name of the layer
weight (torch.Tensor) – A torch.Tensor of same length as the number of channels
batch (Optional[int], optional) – which position at the batch dim of the image tensor this objective is applied to, defaults to None

Returns

Objective to optimize input for a linearly weighted channel activation

Return type

Objective

lucent.optvis.objectives.direction(layer: str, direction: Tensor, batch: Optional[int] = None) → Objective

Visualize a direction in activation space.

Parameters

layer (str) – Name of the layer
direction (torch.Tensor) – torch.Tensor of shape (num_channels, ) giving the direction
batch (Optional[int], optional) – which position at the batch dimension of the image tensor this objective is applied to, defaults to None

Returns

Objective to optimize input for a particular direction in activation space.

Return type

Objective

lucent.optvis.objectives.direction_neuron(layer: str, direction: Tensor, x: Optional[int] = None, y: Optional[int] = None, batch: Optional[int] = None) → Objective

Visualize a single (x, y) position along the given direction

Similar to the neuron objective, defaults to the center neuron.

Parameters

layer (str) – Name of layer
direction (torch.Tensor) – torch.Tensor of shape (num_channels, ) that gives the direction optimize
x (Optional[int], optional) – x-position, defaults to None
y (Optional[int], optional) – y-position, defaults to None
batch (Optional[int], optional) – which position at the batch dimension of the image tensor this objective is applied to, defaults to None

Returns

Objective to optimize input for a particular direction in activation space at a single position

Return type

Objective

lucent.optvis.objectives.diversity(layer: str) → Objective

Encourage diversity between each batch element.

A neural net feature often responds to multiple things, but naive feature visualization often only shows us one. If you optimize a batch of images, this objective will encourage them all to be different.

In particular, it calculates the correlation matrix of activations at layer for each image, and then penalizes cosine similarity between them. This is very similar to ideas in style transfer, except we’re penalizing style similarity instead of encouraging it.

Parameters: layer (str) – layer to evaluate activation correlations on.
Returns: Objective that encourages input towards diversity
Return type: Objective

lucent.optvis.objectives.handle_batch(batch=None)

lucent.optvis.objectives.localgroup_weight(layer: str, weight: Optional[Tensor] = None, x: Optional[int] = None, y: Optional[int] = None, wx: Optional[int] = 1, wy: Optional[int] = 1, batch: Optional[int] = None) → Objective

Linearly weighted channel activation around some spot as objective

Parameters

layer (str) – Name of the layer
weight (Optional[torch.Tensor], optional) – A torch.Tensor of same length as the number of channels, defaults to None
x (Optional[int], optional) – x-position, defaults to None
y (Optional[int], optional) – y-position, defaults to None
wx (Optional[int], optional) – window size in x-direction, defaults to 1
wy (Optional[int], optional) – window size in y-direction, defaults to 1
batch (Optional[int], optional) – which position at the batch dimension of the image tensor this objective is applied to, defaults to None

Returns

Objective to optimize linearly weighted channel activation around some spot

Return type

Objective

lucent.optvis.objectives.neuron(layer: str, n_channel: int, x: Optional[int] = None, y: Optional[int] = None, batch: Optional[int] = None) → Objective

Visualize a single neuron of a single channel.

Defaults to the center neuron. When width and height are even numbers, we choose the neuron in the bottom right of the center 2x2 neurons.

Odd width & height: Even width & height:

+—+—+—+ +—+—+—+—+ | | | | | | | | | +—+—+—+ +—+—+—+—+ | | X | | | | | | | +—+—+—+ +—+—+—+—+ | | | | | | | X | | +—+—+—+ +—+—+—+—+

| | | |

Parameters

layer (str) – Name of the layer
n_channel (int) – Channel/neuron number
x (Optional[int], optional) – x-position, defaults to None
y (Optional[int], optional) – y-position, defaults to None
batch (Optional[int], optional) – which position at the batch dimension of the image tensor this objective is applied to, defaults to None

Returns

Objective to optimize input for a single position of a single channel

Return type

Objective

lucent.optvis.objectives.neuron_weight(layer: str, weight: Tensor, x: Optional[int] = None, y: Optional[int] = None, batch: Optional[int] = None) → Objective

Linearly weighted channel activation at one location as objective

Parameters

layer (str) – Name of the layer
weight (torch.Tensor) – A torch.Tensor of same length as the number of channels
x (Optional[int], optional) – x-position, defaults to None
y (Optional[int], optional) – y-position, defaults to None
batch (Optional[int], optional) – which position at the batch dimension of the image tensor this objective is applied to, defaults to None

Returns

Objective to optimize input for a linearly weighted channel activation at one location

Return type

Objective

lucent.optvis.objectives.wrap_objective() → Callable[[Callable], Objective]

Decorator to construct objectives from function definitions

Returns: Objective that is constructed from the decorated function
Return type: Callable[[Callable], Objective]

optvis.objectives_util module

Utility functions for Objectives.

lucent.optvis.objectives_util._T_handle_batch(T, batch=None)

lucent.optvis.objectives_util._extract_act_pos(acts: Tensor, x: Optional[int] = None, y: Optional[int] = None) → Tensor

Given a tensor of activations, returns the part of the activations that correspond to the given (x,y) position.

Parameters

acts (torch.Tensor) – Activation tensor
x (Optional[int], optional) – x-coordinate, defaults to None
y (Optional[int], optional) – y-coordinate, defaults to None

Returns

activation at coordinate (x, y)

Return type

torch.Tensor

lucent.optvis.objectives_util._make_arg_str(arg: Any) → str

Helper function to convert arg to str.

Parameters: arg (Any) – argument
Returns: arg converted to str
Return type: str

optvis.render module

lucent.optvis.render.export(tensor: Tensor, image_name: Optional[str] = 'image.jpg') → None

Saves tensor as image to disk under image_name.

Parameters

tensor (torch.Tensor) – tensor to be saved
image_name (Optional[str], optional) – absolute path to image, defaults to “image.jpg”

Raises

ValueError – image has invalid shape

lucent.optvis.render.render_vis(model: Module, objective_f: Union[str, Objective, Callable], param_f: Optional[Callable] = None, optimizer: Optional[Callable[[Any], Optimizer]] = None, transforms: Optional[Iterable[Callable[[Tensor], Tensor]]] = None, thresholds: Iterable[int] = (512,), verbose: Optional[bool] = False, preprocess: Optional[bool] = True, preprocess_f: Optional[Callable[[Tensor], Tensor]] = None, progress: Optional[bool] = True, show_image: Optional[bool] = True, save_image: Optional[bool] = False, image_name: Optional[str] = None, show_inline: Optional[bool] = False, fixed_image_size: Optional[int] = None) → List[ndarray]

Main function to perform feature visualization. It takes a model, an objective function and an image paremeterization to compute the optimal image for the given objective.

Parameters

model (torch.nn.Module) – Model for which the objective is computed.
objective_f (Union[str, Objective, Callable]) – Objective that is being optimized
param_f (Optional[Callable], optional) – Image parameterization function, defaults to None
optimizer (Optional[Callable[[Any], torch.optim.Optimizer]], optional) – Optimizer constructor function (maps parameters to optimizer instance), defaults to None #TODO just pass an optimizer directly?
transforms (Optional[Iterable[Callable[[torch.Tensor], torch.Tensor]]], optional) – Iterable of transforms that should be applied on image before forward pass, defaults to None
thresholds (Iterable[int], optional) – Iterable of steps after which to show/save images. max(thresholds) is equal to the total amount of optimization steps, defaults to (512,)
verbose (Optional[bool], optional) – whether to be verbose, defaults to False
preprocess (Optional[bool], optional) – Whether to apply preprocessing (default or custom), defaults to True
preprocess_f (Optional[Callable[[torch.Tensor], torch.Tensor]], optional) – Preprocessing function. If None, at least torchvision.models normalization is applied, defaults to None
progress (Optional[bool], optional) – Whether to display progress bar, defaults to True
show_image (Optional[bool], optional) – Whether to show images in a pop-up, defaults to True
save_image (Optional[bool], optional) – Whether to save images, defaults to False
image_name (Optional[str], optional) – Overwrite default image names, defaults to None
show_inline (Optional[bool], optional) – Whether to show images inline (in ipynb), defaults to False
fixed_image_size (Optional[int], optional) – Set image size if necessary for your model, defaults to None #TODO change name of this kwarg and allow non-square images

Raises

KeyboardInterrupt –

Returns

List of generated images as np.float32 arrays of shape (H, W, 3) with values between 0 and 1.

Return type

List[np.ndarray]

lucent.optvis.render.tensor_to_img_array(tensor: Tensor) → ndarray

Converts tensor to image with channel-last ordering

Parameters: tensor (torch.Tensor) – tensor that should be converted
Returns: image with channel-last ordering
Return type: np.ndarray

lucent.optvis.render.view(tensor: Tensor) → None

Displays tensor on screen by converting it to an image first

Parameters: tensor (torch.Tensor) – tensor to be displayed
Raises: ValueError – image has invalid shape

optvis.transform module

lucent.optvis.transform.compose(transforms: Iterable[Callable]) → Callable

Helper function to compose transformations

Parameters: transforms (Iterable[Callable]) – Iterable of transformation functions.
Returns: Transform function
Return type: Callable

lucent.optvis.transform.jitter(d: int) → Callable

Returns a transform that randomly translates an image up to d pixels in either direction.

Parameters: d (int) – Maximum absolute shift in x/y direction
Raises: ValueError – d should be > 1
Returns: Transform function
Return type: Callable

lucent.optvis.transform.normalize() → Callable

Returns transform that performs the ImageNet normalization for torchvision models

See https://pytorch.org/vision/stable/models.html

Returns: Transform function
Return type: Callable

lucent.optvis.transform.pad(w: int, mode: Optional[str] = 'reflect', constant_value: Optional[float] = 0.5) → Callable

Returns a transform that pads the input tensor.

Parameters

w (int) – How many rows/columns to pad on each side
mode (Optional[str], optional) – padding mode - one of ‘reflect’, constant, ‘replicate’, or ‘circular’, defaults to “reflect”
constant_value (Optional[float], optional) – value that the image is padded with, defaults to 0.5

Returns

Transform function

Return type

Callable

lucent.optvis.transform.preprocess_inceptionv1() → Callable

Returns preprocessing function to prepare input for original Tensorflow’s InceptionV1 model.

InceptionV1 takes in values from [-117, 138] so the preprocessing function takes in values from 0-1 and maps them to [-117, 138]

See https://github.com/tensorflow/lucid/blob/master/lucid/modelzoo/other_models/InceptionV1.py#L56 for details. Thanks to ProGamerGov for this!

Returns: Preprocessing function
Return type: Callable

lucent.optvis.transform.random_rotate(angles: Union[List, Tuple, ndarray], units: Optional[str] = 'degrees') → Callable

Returns a transform that randomly rotates an input image.

Parameters

angles (Union[List, Tuple, np.ndarray]) – list, tuple, or np.ndarray of allowed angles.
units (Optional[str], optional) – “degrees” or “radians”, defaults to “degrees”

Returns

Transform function

Return type

Callable

lucent.optvis.transform.random_scale(scales: Union[List, Tuple, ndarray]) → Callable

Returns a transform that randomly scales the input tensor.

Parameters: scales (Union[List, Tuple, np.ndarray]) – list, tuple, or ndarray of allowed scales
Returns: Transform function
Return type: Callable

lucent.optvis package

optvis.param

optvis.objectives module

optvis.objectives_util module

optvis.render module

optvis.transform module

Module contents