Degrees of algorithmic equivalence between the brain and its DNN models

, and niche, researchers should train DNNs on stimulus samples andbehaviorsthatmatchtheparticipant'sniche( cf Newell's ﬁ rstandsecondinjunctionsabove).Thismattersbecausewe aim to model the competence of a categorizing brain which bathes inside the skull of a given individual. We should therefore levelour modeling benchmark to thiscategorizer'sspeci ﬁ c knowledgeand visual inputsrather than to theuniversalist assumption of sampling statistics and the generic ground-truth responses of the modeling community re ﬂ ected in the published convenience databases.

Deep neural networks (DNNs) have become powerful and increasingly ubiquitous tools to model human cognition, and often produce similar behaviors.For example, with their hierarchical, brain-inspired organization of computations, DNNs apparently categorize real-world images in the same way as humans do.Does this imply that their categorization algorithms are also similar?We have framed the question with three embedded degrees that progressively constrain algorithmic similarity evaluations: equivalence of (i) behavioral/brain responses, which is current practice, (ii) the stimulus features that are processed to produce these outcomes, which is more constraining, and (iii) the algorithms that process these shared features, the ultimate goal.To improve DNNs as models of cognition, we develop for each degree an increasingly constrained benchmark that specifies the epistemological conditions for the considered equivalence.

DNN models of human cognition
As is often said, models in science should be used as simple, but not simpler, mechanisms to formulate less-accessible systemsby being glass boxes that model black boxes.In cognitive science, models offer explicit hypotheses for the computations (see Glossary) that realize a variety of cognitive abilities [1][2][3][4].Developing from connectionism, deep neural networks (DNNs) have become an increasingly popular class of such algorithmic models [5,6] either on their own or as front-ends to hybrid models.When using DNNs as models of human visual categorization [7][8][9], their brain-like layered architectures (feedforward or recurrent) perform a series of nonlinear computations that take in 2D images as inputs (e.g., of real-world faces, objects, and scenes) and deliver outputs such as their category memberships (e.g., 'Mary', 'cars and buildings', or 'New York City') or their mappings into low-dimensional semantics [10].Following training, the output of a DNN can therefore be directly compared to human behavioral responses.This generally frames DNN categorizations as an end-to-end algorithmic function A that returns responses R to stimulus S (Equation 1).

R ¼ A S ð Þ ½1
Such end-to-end, 2D image-to-category labeling is a breakthrough in modeling history that naturally invites addressing again the enduring question [3,6,[11][12][13]: is the algorithmic function of categorization A(S) 'the same' in the brain and its DNN models?
The question of 'the same' algorithm is notoriously difficult to frame and evaluate for the brain because we cannot directly access its unknown categorization algorithm A Brain , as we can in principle access that of the DNN, A DNN .To clarify algorithmic similarity, consider that A DNN (S) is the classic feedforward composition of L embedded stages of consecutive computations [14], where each stage corresponds to one of L network layers, resulting in L stages of embedded nonlinear functions a, i.e. a L (…(a 2 (a 1 (S))).The question of algorithmic similarity then becomes

Highlights
Deep Neural Network (DNN) models categorize complex visual stimuli 'much like' the human brain.That is, their stimulusresponse relationships are similar.
However, stimulus-response relationships are notoriously underconstrained because the brain and its DNN models could process different features from the same stimuli to produce the same responses.
How can we better constrain DNNs such that they better model human cognition?
whether, at some level of abstraction useful for a cognitive theory [3,6,12,15], the L embedded computations of A DNN approximate those of the brain algorithm A Brain .
Having simply restated the enduring question (leaving aside many caveats to which we return later), research practice addresses algorithmic similarity by comparing how A Brain and A DNN perform the same cognitive task.We develop here a new epistemology to develop this comparison, that models the stimuli as much as it models the responses to the stimuli, to understand how A Brain and A DNN process features of the same stimulus space, thereby introducing three embedded degrees of increasingly constrained algorithmic similarity (Figure 1, Key figure).
At this juncture it is important to clarify the scope of our contribution.Models in science are generally used in three complementary ways: to predict new data, to explain their causes, and, from this better understanding, to explore new models [16].DNN models of cognition can predict human behavior with high accuracy, although we still need to explain how A DNN causes behavior, possibly with the same computations as A Brain .We lay out the knowledge and the conditions (i.e., the epistemology) that are pivotal to developing new DNN models of cognition, from predictions of behavior to algorithmic explanations of their causes.The exposition is therefore conceptual rather than technical, although we illustrate key points with concrete examples.

Three increasingly constrained degrees of algorithmic equivalence
Current practices primarily target a first degree of equivalent behavioral (or layer) outcomes between brain A Brain and A DNNs [17] in other words, the extent to which their responses (R) to the same stimuli are similar (R DNN ≅ R Human ).A more constrained second degree requires that A Brain and A DNNs process the same stimulus features (F) to produce these same responses [18].Feature equivalence is pivotal to our epistemology, and allows meaningful comparisons at the third degree of the 'how' question, which requires that A Brain and A DNN compute this same F in the same way, with the same algorithmin other words, with the same sequence of L layered computations (cf Figure 1, where DNN feature F 1 DNN ≅ brain feature F 1 Brain , F 2 DNN ≅ F 2 Brain , …, and F L DNN ≅ F L Brain ).Comparisons of several DNNs at these three degrees could reveal that, of those tested, A DNN is most algorithmically similar to A Brain for the cognitive task considered, and A DNN ≅ A Brain .Of course, third-degree algorithmic equivalence applies to any DNN architecture that models human cognition (e.g., feedforward and recurrent).For simplicity of exposition, we illustrate our points with the simple feedforward architecture.
Before developing the three degrees, it is useful to define an algorithm as a method formulated by a sequence of operations (basic and control structures) to resolve a task (e.g., labeling an input image) in finite time.Equivalence of algorithms between humans and their models has been the enduring goal of cognitive science [3,12], as Newell's famous '20 questions with nature' [19] highlighted half a century ago.In context, the advent of 'end-to-end' models recently enabled comparisons of humans and DNN algorithms that perform the same real-world categorizations.However, Newell's injunctions for the study of human algorithms [19] listed below, with our highlightsremain very much timely for psychology, neuroimaging, and DNN sciences alike.We will refer to these injunctions as we develop the three degrees.

Glossary
Algorithm: a guided sequence of computations (i.e., a method) that resolves a task (here, cognitive) from its inputs in finite time by producing an output (e.g., producing the category label of an input image in n steps of computations).Brain feature: a measurable property of brain responses to stimuli.Computation: the transformation of a representation.Deep neural network (DNN): a hierarchically organized connectionist model loosely inspired by neurobiology, and whose nonlinear layers (usually many) comprise parameters (weights) that can be adjusted ('trained') to minimize a defined cost-function.DNN feature: a measurable property of DNN responses to stimuli.End-to-end: a model that learns to map raw inputs (e.g., pixel images) to a target format (e.g., category labels).Generalization gradient: the relationship between different values of the generative features of stimuli and the responses (behavioral, brain, DNN layer) that these values elicit.Generative adversarial network (GAN): a type of generative DNN that synthesizes artificial (visual) stimuli with complex statistical structures.Generative model: a model that defines the causal components and computations to produce a defined class of outputs (e.g., images for the output class of visual stimuli).Hybrid model: a model that combines different artificial intelligence (AI) techniques, such as a DNN front-end, to label an image used for reasoning within a system of symbolic knowledge structures.ImageNet: an influential image dataset with >14 million images from the 20 000 categories that were famously used to train end-to-end categorization DNNs.Layer: a collection of artificial neurons that share the same hierarchical level within a DNN.These neurons perform computations on their inputs as specified by their weights and nonlinearities (e.g., convolution, non-linear scaling) Multiple realizability: the notion that 'something' (e.g., flying, adding) can be realized in multiple ways to achieve the same outcomes (e.g., wings and feathers vs. man-made articulated wings; an abacus vs. a digital computer).

First-degree equivalence of categorization behaviors
Suppose that a DNN model trained on a large image dataset (e.g., ImageNet [20]) matches trialby-trial human categorization responses (R DNN ≅ R Human ) using the 〈S, R〉 stimulus-response benchmark.Does this qualify as algorithmic equivalence A DNN ≅ A Brain ?At the level of behavioral outputs, yes, but only in this task [21,22] and with the stimulus distribution implicitly captured in the benchmark.Box 1 discusses how the contents of 〈S, R〉 should capture the specific competence of the human categorizer in their niche, and the ensuing risks for modeling categorization when 〈S, R〉 contents assume universality (as is the norm) and do not capture the niche.

Second-degree equivalence of the features that predict behavior
Suppose that an improved 〈S, R〉 benchmark addresses the risks discussed in Box 1 and delivers a first-degree equivalence of categorization behaviors between A Brain and A DNN .Have we now established A Brain ≅ A DNN algorithmic equivalence?No, because we still face the classic argument of the multiple realizability [38] of responses.To illustrate with a behavioral example, Mary can categorize a face image as 'happy' using its mouth, whereas Peter categorizes the same face as 'happy' using the wrinkles around the eyes [30,31,33].Mary and Peter similarly respond 'happy', although their individual algorithms are de facto different because each primarily uses a different feature from the same stimulus (i.e., F mouth vs. F eyes ).An identical argument applies to brain activations (e.g., measured in the right fusiform gyrus).Suppose that A Brain responds to images of houses, cars, and faces by processing their chimney, windshield, and eye features, whereas Representation: a correspondence between a stimulus feature and a brain or DNN feature that has a specific functional role within the cognitive theory of a task.
A DNN processes their roof, bumper, and mouth features.Measuring the similarity of the activation patterns across A Brain and A DNN could suggest similar internal representations of the categories on the layers [39,40], when in fact A Brain and A DNN process different stimulus features.As the example illustrates, establishing stimulus feature equivalence across A Brain and A DNN is indeed necessary to define the respects for (the) similarity (measure) [41,42] in other words, the stimulus feature(s) that properly constrain the similarity responses between the brain and its DNN models.
At the second degree, to address the argument of multiple realizability that affects the first degree, we add the constraint that the same responses must arise from processing the same explicit stimulus features (F).That is, Mary's algorithm A Brain and its model A DNN must process the mouth (not the wrinkles) to both respond 'happy'.Relatedly, A Brain and its model A DNN must both process the chimney to ensure that their similar activations refer to algorithms that process the same stimulus features.Thus, the more constrained second-degree benchmark 〈S, R〉 F adds the similarity of causal categorization features F to the first-degree similarity of response (Figure 1).
Importantly, note that the second degree still does not commit to how these causal features are processed (i.e., with which algorithmic method), although of course constraining any algorithm to process specific causal features is a strong constraint on what this algorithm must do [43].Note also that F can be idiosyncratic to the individual (Box 1), and we should not average across human Box 1. Caveats to the first degree of behavioral equivalence Modern DNNs often achieve near-or even super-human performance on categorization tasks.We could be tempted to conclude that these DNNs are behaviorally equivalent to humans.However, their equivalence only holds if the data on which the DNNs were trained are similar to the visual information that the modeled humans sampled.Unfortunately, the similarity of sampling distributions is rarely considered.
Stimulus sampling distributions can differ in two main ways.First, by under-sampling, where DNN benchmark stimuli under-sample the sampling diversity of the human categorizernamely conspecifics (3D faces, bodies) in their surroundings (objects, scenes), at multiple retinal projections (owing to natural distributions of viewing distances, illuminations, translations, rotations, and so forth).Second, by over-sampling, where benchmark stimuli over-represent the distributional diversity of the categorizer.For example, consider ImageNet, which comprises samples of 120 different dog breeds, a distributional diversity more akin to that of a dog expert [23].Recent acknowledgment of such sampling problems has led to the development of datasets with an improved balance of visual stimulus sampling [24].
Another risk to behavioral equivalence is that most training datasets contain 'ground-truth' labels as responses, and not the labels used by human participants.Ground-truth labels are more representative of the intuitions of the DNN community (which can be inaccurate [25]) than are typical human responses.Thus, the behavioral equivalence tested with typical DNN benchmarks might not straightforwardly generalize human participant categorizations (e.g., [26]).
Relatedly, datasets can neglect well-known human categorization idiosyncrasies.Categorizers apply varying knowledge to the same stimuli, depending both on broad cultural factors [27] and on idiosyncratic factors (e.g., of hobbies such as gothic architecture, renaissance paintings, Fender electric guitars, and so forth [23]).Together, these factors shape the categorization niche of an individual, and this incurs differences in categorization behavior.Studies of the representations that support the categorizations of individual participants have often refuted the universalist assumption, suggesting instead that humans actively engage with their categorization niche [21] rather than passively observing the same image statistics [18,[28][29][30][31][32][33][34][35].Notable examples of these differences include cultural influences on categorization of emotional facial expressions [31] and the general role of expertise on the development of knowledge taxonomies in object recognition [23,36,37].
To circumvent the risks of undersampling, oversampling, and niche, researchers should train DNNs on stimulus samples and behaviors that match the participant's niche (cf Newell's first and second injunctions above).This matters because we aim to model the competence of a categorizing brain which bathes inside the skull of a given individual.We should therefore level our modeling benchmark to this categorizer's specific knowledge and visual inputs rather than to the universalist assumption of sampling statistics and the generic ground-truth responses of the modeling community reflected in the published convenience databases.
algorithms, as is often the norm in neuroimaging studies (cf Newell's third injunction [19,44]).Finally, it is important to emphasize that the second degree is an overarching systems constraint that is pivotal to our epistemology.When the second degree is established, we know that the brain and its DNN models use the same F for categorization, and we can meaningfully open the black boxes to compare how they do so.The F then become the needles whose algorithmic processing we seek to equate across the haystacks of neural and DNN model activity.How can we characterize F?

Characterizing categorization features in humans and their DNN models
The second-degree equivalence requires that we characterize what stimulus features cause responses.This is rarely achieved because the stimulus itself is not modeled.When we use 2D image databases, we stimulate DNN algorithms with pixels without controlling the process that generates these pixelated images.Naturalistic images are observational datasets without a known causal structure [45].To understand the causal stimulus features (F S ) that influence categorization responses (i.e., F S = F Human ), we need to break through this 'wall of pixels' and generate (i.e., control and vary) the 2D images with actual 3D generative models of face, body, object, and scene categories ([33]; also [46][47][48][49]) whose 2D projections produce the pixelated images of the database (Box 2).
Such generative modeling is the hallmark of psychophysics, where for example classic sinewave gratings are generated with four dimensions of spatial frequency, amplitude, phase, and orientation, to ascertain how each causally contributes to responses.It is time to develop an analogous generative psychophysics for higher-level vision by similarly controlling and varying the generative dimensions of dynamic 3D models of conspecifics and their surroundings [48,50].For example, a recent study [18] used a generative model that synthesized, varied, and controlled the 3D features of faces and projected them as 2D stimuli (Box 2), which each participant rated according to their resemblance to the faces of familiar work colleagues [33].With the same dataset, the authors independently modeled the responses of each individual participant with competing DNNs.
Applying the more stringent second-order benchmark 〈S, R〉 F for model selection, they selected the DNN that similarly identified familiar faces because its algorithm processed similar 3D faceshape features as those provided by the generative stimulus model (i.e., F S = F Human = F DNN ).
A second-order equivalence of the modeled stimulus features addresses the multiple equivalence argument raised at the first degreenamely that multiple different features can realize the same behaviors in the human algorithm and DNN models.The study [18] could therefore conclude that A Brain and A DNN were most algorithmically similar according to the second-degree 〈S, R〉 F benchmark.

Generalization gradients
To identify the categorization features at the second degree (i.e., F), we must circumscribe the generative feature subset that causes similar responses.Using this subset, we can then rigorously test and compare the generalization gradients of human and DNN responses.Such generalization testing can address the notorious adversarial examples of DNNs by not only revealing catastrophic failures [63,64] but also the specific parametric range under which they occur, which is a prerequisite for developing targeted remediations.Establishing equivalent F and generalization gradients at the second degree, therefore, develops more comprehensive causal explanations of human and DNN behaviors [18] with matched boundary conditions.We believe that generative psychophysics would benefit visual cognition, neuroscience, and their DNN modeling alike [48].
However, widespread adoption of generative psychophysics requires that we resolve a thorny methodological challenge.How can we parameterize the high-dimensional features of real-Box 2. Assessing the second degree of feature equivalence 〈S, R〉 F In a reverse correlation experiment [30,33], a human participant sees 2D images of 3D faces and categorizes them (e.g., according to their similarity to a familiar face; black arrow in Figure I).Different DNNs predict these human categorizations from the same 2D images (colored arrows).To establish the second degree of feature equivalence between humans and their DNN models (blue arrows), we can use classic techniques that quantify the similarities of image pixel features (i.e., heat-mapping techniques [55,56]).For example, (2) in Figure I illustrates locally linear receptive fields applied at the pixel level [53,54].This technique rewrites the computations of a full non-linear DNN as a single linear weight-map per image, such that multiplying this linear weight-map with the image produces the response as the full non-linear DNN.This is analogous to psychophysical reverse-correlation methods that quantify task-relevant pixels/decision templates in individual human participants [51,[57][58][59].
A high correlation of such pixel maps between humans and their DNN models establishes a second degree of feature equivalence at the level of pixels [60].
However, real-world 3D faces (and bodies, objects, and scenes) hide behind the wall of their 2D pixel projections (as illustrated in Figure I).Our cognitive explanations should break through this wall of pixels to explain human and DNN categorizations in terms of the causes of the pixels.For example, we can compute 3D shape receptive fields to represent the 3D F that explain the common causes of human (Figure I-3) and DNN responses (Figure I-4).With extensive psychophysical testing of generalization of responses (e.g., to changes of 3D shape, and also to F of illumination, scale, rotation, gender, identity, expression, and so forth), we can better test the gradient of generalization to establish the equivalence of the causal 3D F.
Then, to evaluate DNN modeling results, we consider the first degree of equivalence of human and DNN responses (Figure I-5, y axis).We expand to the second degree by quantifying the similarity of the 3D F underlying human and DNN responses (e.g., by correlating the 3D receptive fields [18] or using information-theoretic redundancy [61,62]).The relationship between the two axes (Figure I-5) shows that the tested DNNs improve their first degree of behavioral predictions (y axis) when they infer the 3D F of the generative model from the 2D stimuli (x axis).Importantly, these results can then be subjected to extensive generalization testing [18].generates variations in the causal 3D F, which project onto the wall of pixels as 2D images.(5) At the first degree, we compare how competing DNN models predict human behavior (y axis).At the second degree, we quantify the overlap between predictions from the DNNs and categorizations from the 3D F (with informationtheoretic redundancy, x axis).The scatter shows that the first-degree similarity of human and DNN responses can be explained with the same causal 3D F of the generative model (second degree).Abbreviations: DNN, deep neural network; MI, mutual information; PCA, principal components analysis; RGB, red green blue.

Trends Trends in in Cognitive
world 3D face, object, and scene categories for realistic stimulus synthesis?From the outset we should be aware that no currently existing model might correspond to the ground truth features processed in the brain and DNN models, perhaps because the mathematics and/or visualizations necessary to explore these high-dimensional spaces are not yet sufficiently developed.To explain the stimulus causes of responses, all models might be wrong, but some could be useful.At the second degree, we should therefore compare different generative models of the stimulus in the same way as we compare different DNN models of human responses at the first degree.Two generative approaches are currently promising: direct engineering of computer graphics and indirect engineering of generative DNNs.Computer graphics can render increasingly realistic images [46,50,65].Importantly, the parameters of 3D modeling and rendering can serve as models of the causal structure (i.e., the 3D F) of the generated 2D image categories that serve as stimuli [18,66,67].However, such direct engineering of the generative features restricts the models of each stimulus category to the imagination of human engineers.A promising alternative uses generative adversarial networks (GANs) [68] which can produce arguably more realistic images.However, although such indirect engineering could expand generative parameters beyond human imagination, GANs are more difficult to control and therefore so far cannot deliver controlled 3D models of stimulus categories as computer graphics does (but note progress on structuring latent spaces [69][70][71]).Although still very much in its infancy, generative modeling of the stimuli should become as important as modeling the behavioral and brain responses to the stimuli because we cannot analyze what we cannot synthesize.
Transition: behaviorist (first degree) to pivotal cognitivist explanation (second and third degrees) At this juncture, cognitivists would remark that the first-degree, 〈S, R〉 equivalence is akin to stimulus-response behaviorism.The cognitive revolution expanded this restriction to the intervening internal states (here, of processing F) that enable causal explanations via successive stages of computations that relate stimulus to behavior (i.e., A Brain and A DNN similarly process F).
Our second-degree benchmark 〈S, R〉 F therefore subscribes to cognitivism by constraining the stages of computations, whatever they may be, to process the same F. Consequently, multiple realizability strikes again, this time with the potentially infinite number of algorithms [3,72,73] that can process the same stimulus features to produce the same behaviors.This segues into the third degree of equivalence, where we constrain the hierarchical computations of A DNN = a L (…(a 2 (a 1 (S))) to equate to those of A Brainin other words, similar hierarchical computations between generative stimulus features ( and brain computations (F 1 Brain , F 2 Brain , …, F L Brain ).Strictly speaking, the sequential order of DNN and brain computations should invert that of the generative modelin other words F 1 Brain = F L Stim .The benchmark to evaluate the third degree of algorithmic equivalence becomes 〈S, R〉 A(F) .Note that embedding the pivotal second degree of stimulus feature similarity into the third degree of algorithmic similarity clarifies the point made earlierthat we can only meaningfully compare brain and DNN computations (i.e., F i Stim = F i Brain = F i DNN for stage i = 1 … L) when we know that they categorize using the same stimulus features F. Otherwise, we could literally compare the processing of apples and oranges across systems (that would similarly respond 'fruit basket' to the same stimuli).Thus, the argument of multiple realizability and the solutions developed (by using generative psychophysics) seamlessly apply from behavior to any other brain/DNN response measured at any level of granularity.

Third-degree equivalence of behavior because equivalent algorithms process equivalent F
At the third degree, we seek equivalence of the algorithms that process the same F from the same S to produce the same R.We evaluate algorithmic similarity with the 〈S, R〉 A(F) benchmark and seek to equate the outputs of L embedded computational stages (F 1 , F 2 , …, F L ) between the feedforward A DNN and A Brain .Such algorithmic equivalence is like reaching for Neptuneit is beyond the current development of science.We realistically aim for an 'algorithmic lift-off' with judicious use of generative psychophysics, as illustrated in Figure 2.
Suppose a 1 (S) nonlinearly filters its input with banks of multi-scale, oriented Gabor kernels covering the input image.Subsequently, a 2 (a 1 (S)) max-pools the outputs of a 1 (S) to reduce their dimensionality and implement their local invariance to translation.Following learning, a 2 (a 1 (S)) in DNNs would be similar to a 2 (a 1 (S)) models of visual cortex (V1/V2) cell responses in the primate brain [76].However, instead of generically representing the entire image, a constrained a L (… (a 1 (S))) could now deliver a compact representation of the task-relevant stimulus features F. We can address the challenge of understanding how A Brain and A DNNs ultimately yield the F that categorize the stimuli while addressing the well-documented human higher-level invariances to other stimulus features.For example, we know that A Brain identifies faces across parametric variations of many generated features (e.g., in Figure 2, background, illumination, translation, rotation, and the categorical feature of identity, as well as other features such as size and occlusion and categorical features of expression, aging, and so forth, that are not illustrated in Figure 2 [77]).Conversely, generalization performance decreases when we move away from the typical illumination from above, as well as for upside-down faces [78] and specific occlusions [79].Relatedly, visual illusions (e.g., a 3D concave face moving from left to right produces a systematic illusory motion in the opposite direction [80]) can reveal priors (e.g., for convex surfaces) and logical dependencies of computations (e.g., A Brain reconstructs face surfaces before identifying them).
To 'lift-off' an algorithmic understanding we could apply generative psychophysics to feedforward A DNNs , as Figure 2 illustrates, and locate the sufficient sub-nested computationsa L (… … (S)) F of A DNNsthat generalize each stimulus feature in the same way that A Brain does.This approach could map out the algorithmic functions that different sub-nested computations achieve, their relative complexity (e.g., four layers to generalize the face over backgrounds in Figure 2, but 4 + 3 to identify it), and their sequential dependencies (e.g., face extraction at layer 4, invariances to translation at 4 + 2, and finally identification at 4 + 2 + 1).We could then ask whether a similar sequence of algorithmic functions organizes A Brain across the hierarchically activated regions of the ventral and dorsal pathways.
Current evidence falls short of such algorithmic lift-off because of reliance on under-constrained first-degree comparisons of brain and DNN representations of the same 2D images.That is, when claims are made of equivalent brain and DNN layer responses [40,81,82], we rarely know the causal origin of the F being processed and represented in the brain (a second-degree question with more constrained benchmark 〈S, R〉 F ; cf [83]), let alone the specific computations that cause them [a third-degree question, with even more constrained 〈S, R〉 A(F) ].Thus, first-degree equivalence between brain and DNN activations exchanges one black box (the DNN) for another (the brain) [2,84]), teaching us little about either.
Figure 2. Third-degree algorithmic lift-off using the 〈S, R〉 A(F) benchmark.With our generative psychophysics we can locate the DNN layer that responds invariantly to different generative features of the stimulus [75].Following training, a feedforward architecture (ResNet 7) responds invariantly to the background stimulus feature in layer L 4 .Before L 4 , layers L 1-3 explicitly represent the background features.Invariance to face background suggests that the embedded algorithmic functions a 3 [… (S)] F of A ResNet8 realize ExtractFace from the input images.A similar argument applies to the translation feature at L 6 and to the rotation feature at L 7 .A ResNet8 can be written in pseudo-language as IdentifyFace(image), itself defined as the sequence of embedded functions revealed by the generative psychophysics.A ResNet8 therefore predicts 〈S, R〉, by processing the F in a constrained sequence of algorithmic computations in different layers.In turn, we can turn to the hierarchically organized brain and ask whether A Brain computes these same F across its layers in the same algorithmic order as A ResNet8 .Abbreviations: a, embedded algorithmic function; A, algorithmic function; DNN, deep neural network; F, stimulus features; ID, identity; L, layer; R, response; ResNet, residual network, S, stimulus/stimuli.
To reveal dynamic computations in the brain, a methodology was recently proposed based on generative psychophysics (albeit with simple features) [85].This approach incorporates a welldefined algorithmic constraint into the task [86] to document the corresponding four main algorithmic stages of dynamic linear and nonlinear computations in different layers of brain networks that together transform stimuli into behavior.This approach in principle extends to the generative features of face, object, and scene categories.
The third degree will also need to address the classic argument that algorithms depend in part on the hardware that implements them [6,87], and how this hardware develops together with its software [88].DNN architectures could embed well-documented properties of neuroanatomy and neurophysiologyfor example, two retinas project in specific ways in two occipital cortices and then onwards into the ventral and dorsal categorization pathways, and their developmental trajectories (e.g., the initial processing of low-frequency visual information [89]).A step change from the typical feedforward architectures would enable recurrence to propagate top-down predictions that reverse the bottom-up flow [90].In turn, predictions could guide active sampling of the visual input in the task [91] through sequences of 'retinal' fixations on a generated 3D world to dynamically extract categorization features [34,35].Lastly, additional constraints such as parsimony could be used to narrow down the search for appropriate algorithms [92].Although third-degree algorithmic equivalence remains a tall order, DNN modeling more strongly constrained with a well-established methodology and evidence from vision and neuroscience would likely Box 3. Is the second degree really necessary?Several arguments can be developed against the need for our second-degree 〈S, R〉 F to develop A DNNs as better models of A Brain .We argue that all imply 〈S, R〉 F , at least implicitly.
(i) The 'suitably rich' S. A first-degree DNN modeler of 〈S, R〉 could object that the second-degree 〈S, R〉 F remains unnecessary because, as the argument would go, all boils down to the composition of a stimulus set S. A 'suitably rich' S could dissociate responses R across A DNNs and A Brain without requiring a priori hypotheses about the features F that generated S.However, this argument rests on composing the 'suitably rich' S that enables such response dissociations.Without an explicit formalism of what F might be, this composition would likely be guided by the modeler's hypotheses about the images that contain (or not) the dissociating Fin other words a second-degree endeavor, where F remains latent in the mind of the first-degree modeler.(ii) The asymptotic S. One could address the above shortcoming using brute force, piling millions of images into S until the targeted responses are eventually dissociated.Even though this approach might asymptotically work (although self-driving cars and adversarial testing illustrate practical difficulties), the severely restricted experimental time of a neuroimaged brain makes brute force impractical.With neuroimaging, gathering a 'suitably rich' set (with low cardinality) that dissociates the responses of the systems at the first degree requires explicit F hypotheses at the second degree.(iii) A continuum of first-degree 〈S, R〉 differences between A Brain and A DNN .Because different generative models of the same stimuli make as many different predictions about the F of natural computations (e.g., from simple Gabors to multivariate 3D-shaped F), first-degree 〈S, R〉 performance comparisons between A Brain and its A DNN models would likely reveal a continuum of differences.Such a first-degree continuum, the argument would go, is all we need.However, the second degree introduces the qualitative difference of causal explanations, whereby 〈S, R〉 F adds the cognitivist link 'because processing of F' to the first-degree 〈S, R〉 predictions.
A system of axes clarifies causal second-degree explanations (illustrated in Figure 1 in main text).Different sets of hypothesized generative features (e.g., Gabors vs. 3D multivariate shapes) each provide a different system of axes with explicitly defined dimensions (ideally, as few as possible) and units for each image that comprises S. Notions such as 'suitable richness' of S and asymptotic S all become defined within a generative space, with defined sampling density and coverage [74].Performance overlap (or lack thereof) between A Brain and A DNN on any stimulus set S becomes referenced to the explored regions of the space.Consequently, we now navigate along a defined system of axes to test generalization gradients, and we can target specific regions of uncertain performance overlap and increase sampling density to resolve the uncertainty.
To conclude, second-degree 〈S, R〉 F is necessary to transform first-degree 〈S, R〉 from an odyssey in a randomly accessible territory into a plannable exploration that delivers systematically sampled explanatory maps of performance overlap with respect to (ideally understandable) generative features.
deliver improved algorithms that reflect human visual competence better than the prevalent bruteforce empiricism (e.g., of brain score [17]).

Concluding remarks
We have addressed the algorithmic equivalence between the categorizing brain and its DNN models in the context of categorizations of conspecifics and their surroundings.We discussed three embedded degrees that progressively increase the constraints that bear on the evaluation of algorithmic similarity: the first-degree equivalence of their categorization responses R to the same stimuli (R DNN ≅ R Human ), the second-degree equivalence of the stimulus features F that produce these same behaviors (F DNN ≅ F Human ), and the third degree of the L stages of algorithmic computations, reflected as layered brain features (F 1 , F 2 , …, F L ) and DNN features between A Brain and A DNN (F 1 DNN ≅ F 1 Brain , F 2 DNN ≅ F 2 Brain , …, F L DNN ≅ F L Brain ).We have discussed how the first degree requires a modeling benchmark 〈S, R〉 that better incorporates the niche of the individual categorizer rather than the generic ground-truth responses of the DNN modeling community and the published convenience databases.At the second degree, we have emphasized the importance of developing a genuine psychophysics of human recognition and its DNN modeling that 'breaks through the wall of pixels' by developing 3D generative models of face, object, and scene categories.However, it is possible that the second degree may not be essential (Box 3).Nevertheless, such generative models will now be necessary to develop the type of algorithmic understanding that has been the hallmark of cognitive explanations and that is relevant for many cognitive questions and modeling approaches [74,93] (see Outstanding questions).Finally, we showed how generative psychophysics enables algorithmic lift-off by comparing the layered order of computations across regions of the categorizing brain and the layers of DNN models.

Outstanding questions
How do published image datasets compare with naturalistic human visual sampling in a categorization niche?
How do DNNs predict the diversity of human categorization behaviors?Can we train DNNs to indirectly engineer a generative (causal) model of 3D visual categories (e.g., conspecifics and their surroundings) from 2D image databases, or will we be restricted to directly engineering 3D graphics models that represent the real-world variations of one category at a time (e.g., face, or car, or city)?
Can DNN generalization gradients approximate human generalization gradients (e.g., to changes of illumination, scale, orientation, and categorizationrelevant F)?
Can we develop different models of a given stimulus category to compare the processing of their generative features in the brain and DNN models?Can DNN models account for classic visual illusions which reveal crucial differences between veridically versus humanly represented physical reality?
How deeply can DNNs model human visual cognition into scene semantics (e.g., from the features of 'human at position x, y, z' to those of 'student hiker rescued from a Scottish hill after 2 days'), and how can we construct generative models that incorporate such semantic constraints?What algorithmic computations and representations can we extract from different modalities and granularities of brain measures?How can we relate algorithmic computations and representations across modalities and granularities of brain measures?

'
Fundamental fact.[Human] behavior is programmable [via an information-processing algorithm].Injunction 1. Know the method [information-processing algorithm] that your subject is using to perform the experimental task [categorize the visual input].Injunction 2. Never average over methods [information-processing algorithms].'Trends in Cognitive Sciences OPEN ACCESS Trends in Cognitive Sciences, December 2022, Vol. 26, No. 12 1091

Figure 1 .
Figure 1.First degree: typical equivalence (white intersection) tested between human responses (green) and 2D images of real-world 3D faces, bodies, objects, and scenes, and deep neural network (DNN) predictions (red).Such images typically form an experimental 'black box' because we cannot control and therefore explicitly test how real-world 3D features (projected on these 2D images) cause similar responses between humans and DNN models (white intersection).Second degree: a stimulus model generates 2D images from variations of its generative features F (blue), and provides a coordinate system to chart and systematically explore the match between human and models.Such generative psychophysics controls the images, thereby enabling decomposition of human and DNN responses into: (A) the white triple intersection of similar human and DNN responses to the same controlled 3D F, (B) the cyan remainder of human responses to stimulus features F that the DNN model cannot predict, (C) the magenta remainder of DNN predictions from F that are dissimilar to human responses, and (D) the crucial yellow remainder flagging that the F of this particular stimulus model fails to account for the total similarity of human and DNN behavior (suggesting testing of other F from other generative stimulus models).Third degree: equivalence of the brain and DNN algorithms (across schematized brain and DNN layers) whose similar computations process the same stimulus features F to produce the same responses.
Figure I. Second equivalence 〈S, R〉 F .(1) Reverse correlation techniques such as Bubbles[48,51] can quantify the 2D pixel features F used by each human categorizer[52].(2) Relatedly, visualization techniques[53,54] can quantify the 2D pixel features that cause DNN model responses.(3,4) A 3D face model[50] generates variations in the causal 3D F, which project onto the wall of pixels as 2D images.(5) At the first degree, we compare how competing DNN models predict human behavior (y axis).At the second degree, we quantify the overlap between predictions from the DNNs and categorizations from the 3D F (with informationtheoretic redundancy, x axis).The scatter shows that the first-degree similarity of human and DNN responses can be explained with the same causal 3D F of the generative model (second degree).Abbreviations: DNN, deep neural network; MI, mutual information; PCA, principal components analysis; RGB, red green blue.

Trends
Trends in in Cognitive Cognitive Sciences Sciences (See figure legend at the bottom of the next page.)