ONE MODEL, BEFORE AND AFTER BIOLOGY TRAINING

From CLIP to BioCLIP 2

BioCLIP 2 began as a general image–text model and was trained further on 214 million organism photos. It now names species twice as well. Both models have the same layout of 428 million weights, so each weight in one has a partner in the other and the two can be compared directly. Three questions follow: what did BioCLIP 2 get worse at, where in the network does it get worse, and can one model keep the naming without that loss?

45.5%

Starting model: species named correctly, given five example photos of each of 223 species

90.6%

The same test after BioCLIP 2’s training

THE TWO MODELS

The two models are two files of 427,616,513 numbers, laid out the same way

BioCLIP 2’s authors did not start from random weights. They loaded the LAION-2B CLIP ViT-L/14 weights and kept training them on organism photos. Nothing was added or removed, so each number in one file has a partner in the other.

Starting model

LAION-2B CLIP ViT-L/14

Trained on about 2.3 billion image–caption pairs from the web.

BioCLIP 2

The same network, trained further

214 million organism images with their taxonomic names, plus 26 million general web photos and captions mixed back in, so it would not forget everything else.

A mix

Any mix of the two

For every number, mix the two values: α parts BioCLIP 2 and 1 − α parts the starting model. α = 0 is the starting model and α = 1 is BioCLIP 2. Each mix is a working model.

Sources: LAION-2B CLIP model card; Gu et al. 2025, BioCLIP 2.

HOW THE MODEL READS A PHOTO AND A NAME

A photo and a name each become 768 numbers

The model scores a photo against a name by the cosine of the angle between their two lists of numbers (cosine similarity). A small angle means a match.

Photo, 224 × 224 pixels

Cut into 256 patches of 14 × 14 pixels.

257 tokens, 1,024 wide

One per patch, plus one summary token (often called the class token) for the whole photo.

24 image blocks

Each block lets tokens read each other (attention), then reworks each one separately (a small feed-forward network, MLP).

768 numbers

The summary token after a final reshaping to 768 numbers (the projection).

Name, up to 77 tokens

Looked up in the word table of 49,408 token vectors.

Tokens, 768 wide

One per word piece, with its position added.

12 text blocks

The same two steps as the image blocks.

768 numbers

The end-of-text token after the same kind of final reshaping.

Image side: 303,966,208 numbers. Text side: 123,650,304. One more number sets the scale of the scores. Together: 427,616,513.

THE FIRST-LAYER FILTERS, DRAWN AS PICTURES

Training on organism photos changed the first-layer filters by 20% and the text side’s word table by 88%

Grid of 1,024 colour filters from the starting model, most showing stripes, blobs and checker patterns

Starting model

The same grid from BioCLIP 2; the patterns look almost identical to the starting model

BioCLIP 2

The difference between the two grids, shown four times brighter; mostly fine-grained speckle

Difference, shown 4× brighter

Each tile is one 14 × 14 pixel filter that reads a patch of the photo. Tiles are ordered by how much they changed. Training changed these filters by 20% of their original size.

ONE PHOTO SCORED AGAINST SPECIES NAMES

BioCLIP 2 scores a junco photo’s right name 0.717, up from 0.291, and its four highest wrong names higher too

A dark-eyed junco, a small grey and white bird, lying in an open hand

Dark-eyed Junco (Junco hyemalis), as the model sees it: 224 × 224 pixels. Photo (c) jessicaturtle, CC BY-NC, cropped.

Starting model

0.291

Junco hyemalis

0.248

Zootoca vivipara

0.232

Phoca vitulina

0.229

Grallina cyanoleuca

0.218

Coccyzus americanus

The right name scores highest, by a small margin. A lizard and a seal come next.

Halfway mix

0.499

Junco hyemalis

0.322

Grallina cyanoleuca

0.261

Zootoca vivipara

0.247

Coccyzus americanus

0.232

Phoca vitulina

The right name’s lead widens. A bird moves into second place.

BioCLIP 2

0.717

Junco hyemalis

0.592

Coccyzus americanus

0.562

Branta canadensis

0.560

Grallina cyanoleuca

0.558

Nycticorax nycticorax

All five top names are now birds.

Scores are the cosine of the angle between photo and name (1 is a perfect match). The halfway mix averages every weight of the two models. The nine other names were drawn at random from the 223 species.

WHAT TRAINING IMPROVED

BioCLIP 2 names 90.6% of photos of 223 species correctly, up from the starting model’s 45.5%

40%

60%

80%

100%

0

0.25

0.5

0.75

1

Start

BioCLIP 2

+45 points

Give the model five example photos of each species, then ask it to name new photos. BioCLIP 2 gets 9 in 10 right.

Each model expects pixel values adjusted in its own way (input scaling). The 45.5% and 90.6% give each model its own. The curve uses BioCLIP 2’s for every mix, so the starting model scores 43.2% on it.

Horizontal axis: BioCLIP 2’s share of every weight, from 0, the starting model (Start), to 1, BioCLIP 2. iNaturalist photos, 223 species. Each mix names more photos correctly than the one before it.

QUESTION 1

What did BioCLIP 2 get worse at?

A natural guess: only subjects far from biology. Training on organism photos made BioCLIP 2 better at naming species, and it should be no worse at any other question about animals.

Check 1: a question about animals that BioCLIP 2 was not trained on

Is the animal in the photo alive or dead? BioCLIP 2 was trained to match photos to taxonomic names, so its training did not target this.

Check 2: a subject outside biology

Ten kinds of vehicle, from bicycle to rocket, named from words and from example photos.

Check 3: which photos each model treats as alike

Pick one photo and ask each model which other photos it rates as most similar.

QUESTION 1 · ALIVE OR DEAD

BioCLIP 2 scores 85.7% at telling alive animals from dead ones; the starting model scores 92.1%

80%

85%

90%

95%

100%

0

0.25

0.5

0.75

1

Species the classifier was trained on

Species held out from the classifier

92.1% → 85.7%

Balanced accuracy (the average of accuracy on alive photos and on dead ones), worked out for each of 115 held-out species and then averaged: starting model, then BioCLIP 2.

A simple classifier reads “alive” or “dead” from the model’s 768 numbers. For species it was trained on, the score stays near 91%. The score falls only for species it was not trained on.

Horizontal axis: BioCLIP 2’s share of every weight, from 0, the starting model, to 1, BioCLIP 2. 95% interval on the change: −9.5 to −3.4 points. Every mix is given BioCLIP 2’s adjustment of pixel values, so the starting model scores 91.4% here.

QUESTION 1 · PHOTO BY PHOTO

159 photos are labelled alive or dead correctly by the starting model alone, and 59 by BioCLIP 2 alone

All 1,323 test photos, by which model’s classifier labels them correctly

1,063 both right

159

59

42

656 photos recorded as alive

539 both right

78

20

19

667 photos recorded as dead

524 both right

81

39

23

Both classifiers right

Only the starting model right

Only BioCLIP 2 right

Both wrong

The 159 photos only the starting model gets right split evenly: 78 alive animals and 81 dead ones. Of the 59 only BioCLIP 2 gets right, 39 are dead animals and 20 alive. Counting every photo equally, the starting model is right on 92.4% and BioCLIP 2 on 84.8%; counting each of the 115 species equally gives 92.1% and 85.7%.

Right means the classifier agrees with the alive-or-dead label recorded by the iNaturalist contributor. 115 species, each tested by a classifier trained without that species.

QUESTION 1 · PHOTOS BIOCLIP 2 GETS WRONG

Eight photos the starting model labels correctly as alive or dead and BioCLIP 2 labels wrongly

A dead European robin lying on its back on tarmac

Dead

European Robin

A dead brown hare lying on its side on tarmac

Dead

Brown Hare

A dead American red squirrel lying on a mown lawn

Dead

American Red Squirrel

A dead young European robin lying on paving tiles

Dead

European Robin

A live Eurasian red squirrel climbing a tree trunk among ivy

Alive

Eurasian Red Squirrel

A live ovenbird perched on a branch among green leaves

Alive

Ovenbird

A live ovenbird close up among leaves

Alive

Ovenbird

A live hedgehog on grass at night

Alive

Common Hedgehog

Top row. A dead robin on its back on tarmac, a dead hare on tarmac, a dead red squirrel on a lawn and a dead young robin on paving tiles. BioCLIP 2’s classifier calls all four alive.

Bottom row. A live red squirrel on a tree trunk, two live ovenbirds among leaves and a live hedgehog on grass at night. It calls all four dead.

The starting model’s classifier gets all eight right.

Eight of the 159 photos only the starting model gets right: the largest disagreements between the two classifiers among photos with a display licence. The word under each photo is the status its contributor recorded. iNaturalist photos, cropped; credits on the last slide.

QUESTION 1 · PHOTOS THE STARTING MODEL GETS WRONG

Eight photos BioCLIP 2 labels correctly as alive or dead and the starting model labels wrongly

A dead Moorish gecko on a tree trunk in a street

Dead

Moorish Gecko

A dead hedgehog at the edge of long grass beside a road

Dead

Common Hedgehog

A dead hedgehog on gravel beside grass

Dead

Common Hedgehog

A dead dark-eyed junco, very small in the frame, on a ledge outside a window

Dead

Dark-eyed Junco

A dead great horned owl in a cardboard box

Dead

Great Horned Owl

A dead striped skunk, small and distant, on a road seen from a car

Dead

Striped Skunk

Live wapiti, small and distant, on a grassy clifftop above the sea

Alive

Wapiti

A live white dog asleep on its side on bare ground

Alive

Domestic Dog

Top row. A dead gecko on a tree trunk, a dead hedgehog beside long grass, a dead hedgehog on gravel and a dead junco on a window ledge.

Bottom row. A dead owl in a cardboard box, a dead skunk on a road, live wapiti on a clifftop and a live dog asleep on the ground.

The starting model’s classifier calls the six dead animals alive and the two live ones dead. BioCLIP 2’s classifier gets all eight right.

Eight of the 59 photos only BioCLIP 2 gets right: the largest disagreements between the two classifiers among photos with a display licence. The word under each photo is the status its contributor recorded. iNaturalist photos, cropped; credits on the last slide.

QUESTION 1 · NAMING VEHICLES

Given ten vehicle names to choose from, BioCLIP 2 calls 83% of motorcycle photos “rocket”

Starting model

BioCLIP 2

its most common wrong name

Rocket

99%

100%

Streetcar

68%

96%

rocket 3%

Tank

98%

83%

rocket 14%

Bicycle

96%

71%

rocket 22%

Tractor

95%

29%

tank 48%

Bus

88%

24%

streetcar 50%

Pickup truck

97%

21%

streetcar 32%

Lawn mower

98%

14%

rocket 52%

Train

92%

12%

streetcar 38%

Motorcycle

99%

8%

rocket 83%

93.0% → 45.8%

Vehicle photos named correctly from words, starting model then BioCLIP 2. The model compares each photo with ten phrases such as “a photo of a bus” and picks the closest.

91.5% → 86.7%

The same photos named with no words involved: the model is given five example photos of each vehicle. The score falls 4.8 points.

Violet and teal bars: the share of each vehicle’s 100 test photos that each model names correctly from words. Copper bars: the share BioCLIP 2 gives its most common wrong name. Ten vehicle classes from a standard set of small labelled photos (CIFAR-100), enlarged to 224 pixels.

QUESTION 1 · THE VEHICLE NAMES COMPARED WITH EACH OTHER

In BioCLIP 2 the ten vehicle names are less similar to each other: 0.662 on average, down from 0.774

Starting model · average 0.774

BioCLIP 2 · average 0.662

Bicycle

Bus

Lawn mower

Motorcycle

Pickup truck

Rocket

Streetcar

Tank

Tractor

Train

Outlined: “bus” with “streetcar”, 0.863

The same pair, 0.741

Each cell compares two vehicle names by cosine similarity, where 1 means the model treats the two names as identical. Darker is more alike.

A natural guess: the text side forgot these words, so it now gives “bus” and “streetcar” almost the same numbers.

The grids show the opposite. In BioCLIP 2 almost every cell is lighter. “Bus” and “streetcar” are still each other’s closest name, at 0.741, down from 0.863.

Columns follow the same order as the rows. The grey diagonal is each name with itself. Colour runs from white at a similarity of 0.5 to full copper at 0.95. Names were given as “a photo of a bus” and so on.

QUESTION 1 · BUS PHOTOS SCORED AGAINST FOUR NAMES

On bus photos BioCLIP 2 scores the name “streetcar” 0.566, above “bus” at 0.555

0

0.2

0.4

0.6

Starting model

“bus”

0.250

“streetcar”

0.187

“rocket”

0.145

“tank”

0.131

BioCLIP 2

“bus”

0.555

“streetcar”

0.566

“rocket”

0.540

“tank”

0.546

In the starting model a bus photo scores 0.250 against “bus” and 0.187 against “streetcar”, so it answers “bus”.

In BioCLIP 2 each of the four names measured scores higher on every one of the 1,000 vehicle photos. The right name rose least.

On a bus photo all four now score between 0.540 and 0.566, with “streetcar” highest.

The junco photo showed the same overall rise: its right name went from 0.291 to 0.717 and its highest wrong names rose too.

Bars: average cosine similarity between the 100 bus photos and each name. The model answers with the highest-scoring name.

QUESTION 1 · WHAT EACH MODEL TREATS AS ALIKE

Of the 8 photos each model rates most like a robin photo, BioCLIP 2 picks 7 robins and the starting model 1

A European robin, small in the frame, on a branch among green leaves

The photo we ask about: a European Robin, small and far away, which the starting model usually misnames.

Starting model · 1 of 8 is a robin

Red cardinal on a branchSquirrel in a treeBird at the top of a leafy treeBrown bird among leavesRobin on a mossy trunkDark red bird among leavesBird looking out of a hole in a tree trunkRed cardinal in a leafy tree

Weights mixed, 0.75 BioCLIP 2 · 6 of 8 are robins

Robin on a signRobin on the ground with wings spreadRobin on dark soilRobin close upRobin on a mossy trunkSpeckled young robin on leaf litterRed cardinal on a branchRed cardinal in a leafy tree

BioCLIP 2 · 7 of 8 are robins

Robin on a mossy trunkRobin close upRobin on dark soilRobin on a signRobin on the ground with wings spreadSmall bird hidden among bare twigsSpeckled young robin on leaf litterSmall bird on a seed feeder

Most similar first, from other photographers only. iNaturalist photos, CC BY and CC BY-NC, cropped; credits on the last slide.

QUESTION 1 · WHAT EACH MODEL TREATS AS ALIKE

The 8 photos the starting model rates most like a dead moose are all dead animals; BioCLIP 2’s are mostly kangaroos

A dead moose lying on leaf litter in open woodland

The photo we ask about: a Moose, dead.

Starting model · 8 of 8 are dead animals, none is a moose

Dead animal lying on grassDead animal lying on bare groundDead animal lying on grassDead animal lying among leavesDead animal lying on dry groundDead fox lying on dry grassDead animal curled on bare soilDead animal lying on grass

Weights mixed, 0.75 BioCLIP 2 · 4 of 8 are dead, 1 is a moose

Kangaroo lying beside a roadKangaroos standing in a fieldDead animal lying on grassDead young deer with a person’s hand beside itDead roe deer lying on grassMoose standing in dim woodlandDeer standing in tall grassDeer standing among trees

BioCLIP 2 · 5 kangaroos, 2 moose and a wombat; 3 of the 8 are dead

Kangaroo lying beside a roadKangaroos standing in a fieldTwo kangaroos standing on grassDead animal lying on grassMoose standing in dim woodlandMoose in a green fieldKangaroo with a joeyDead animal lying on grass

Most similar first, from other photographers only. iNaturalist photos, CC BY and CC BY-NC, cropped; credits on the last slide.

QUESTION 1 · EACH PHOTO’S MOST SIMILAR PHOTOS

In BioCLIP 2, 59% of a photo’s most similar photos share its alive-or-dead label, down from 81%

0%

25%

50%

75%

100%

0

0.25

0.5

0.75

1

Share with the same alive-or-dead label

Share of the same species

Chance, 50%

81% → 59%

For each photo, take the eight photos a model rates most similar. In the starting model, 81% share its alive-or-dead label. In BioCLIP 2, 59% do, nine points above chance.

The share of the same species rises from 24% to 79%. A classifier still tells alive from dead from BioCLIP 2’s 768 numbers (85.7%), but photos with the same label are now only slightly more likely to be rated similar.

Horizontal axis: BioCLIP 2’s share of every weight (0 is the starting model). All 1,323 alive-or-dead test photos, similar photos by other photographers only.

QUESTION 1 · MIXING THE TWO MODELS’ WEIGHTS

Averaging the two models’ weights gives 72.7% on species naming and 92.2% at telling alive from dead

40%

60%

80%

100%

0

0.25

0.5

0.75

1

Alive or dead, unseen species

Species naming

92.2%

72.7%

90.6%

85.7%

43.2%

91.4%

72.7%

Species naming at a mix of 0.5, where every weight is the average of the two models’ values. That is 62% of the way from 43.2% to 90.6%.

92.2%

Telling alive from dead at the same mix; the starting model scores 91.4%. Past 0.5 it falls: 90.1% at 0.75, 86.3% at 0.9.

Naming vehicles from words is 92.8% correct at this mix.

Horizontal axis: BioCLIP 2’s share of every weight (0 is the starting model). 223 species; 1,323 photos of 115 held-out species.

QUESTION 1 · ANSWER

BioCLIP 2 got worse at telling alive from dead and at naming vehicles from words, which fell from 93.0% to 45.8%

Species naming rose from 45.5% to 90.6%. With every weight averaged between the two models, naming is 72.7% and telling alive from dead is no worse than in the starting model.

Worse at one question about animals

Accuracy at telling alive animals from dead ones fell from 92.1% to 85.7% in species the classifier had not seen. On 159 photos only the starting model is right, among them a robin on its back on tarmac.

Worse at matching vehicle photos to words

Vehicle photos named correctly from words fell from 93.0% to 45.8%. Named from example photos they fall 4.8 points, to 86.7%. On bus photos BioCLIP 2 scores “streetcar” above “bus”.

Similar photos are now mostly the same species

A photo’s most similar photos are the same species 79% of the time, up from 24%. They share its alive-or-dead label 59% of the time, down from 81%.

QUESTION 2

Where in the network does BioCLIP 2 get worse at telling alive from dead?

A natural guess: a little in every block. Training changed the weights of all 24 image blocks, so BioCLIP 2 should fall a little further behind the starting model at each block, from the first to the last.

Read each block

Stop the model after a block, take the tokens it holds there, and test them the same way as the final output.

Watch the attention

See which patches of the photo the summary token, the one token that becomes the output, draws on in the last blocks.

Look at the weights

Subtract the starting model’s weights from BioCLIP 2’s, block by block, and see whether any block’s change stands out.

QUESTION 2 · WHAT A BLOCK HOLDS

Every block holds 257 tokens: 256 patch tokens and 1 summary token

256 patch tokens, one per patch

1 summary token

reads from the patch tokens in every block

final steps

768 numbers

not passed on after block 24

A token is a list of 1,024 numbers. Each patch token describes one 14 × 14 pixel patch of the photo. The summary token describes no patch of its own. In every block it reads from the patch tokens through attention.

After block 24 only the summary token goes on, through two final steps: a rescaling step and a final projection. They turn it into the model’s 768 output numbers.

So any block can be read two ways: from its summary token, or from the average of its 256 patch tokens.

The image side cuts a 224 × 224 pixel photo into a 16 by 16 grid of patches, 256 in all. The summary token is often called the class token.

QUESTION 2 · SPECIES NAMING, BLOCK BY BLOCK

Up to block 16, neither model can name 1 species photo in 10 from its summary token

0%

25%

50%

75%

100%

2

6

10

14

18

22

24

BioCLIP 2

Start

14% → 91%

BioCLIP 2’s species naming read from its summary token at block 18 and at block 24. The starting model goes from 5% to 43% over the same blocks.

Up to block 16 neither model names 1 photo in 10 from its summary token.

Horizontal axis: image block, 2 to 24. Each point is read afresh from that block’s summary token, the token that becomes the output. Start is the starting model. These scores come from a separate run of the naming test, so the block-24 values differ from other slides by under a point.

QUESTION 2 · ALIVE OR DEAD, BLOCK BY BLOCK

Through block 22, BioCLIP 2’s summary token scores higher than the starting model’s at telling alive from dead

84%

86%

88%

90%

92%

94%

18

19

20

21

22

23

24

norm

output

BioCLIP 2, summary token

Starting model, summary token

91.9%

90.5%

91.9%

Accuracy at telling alive from dead when a classifier reads BioCLIP 2’s summary token at block 22, against 90.5% for the starting model. BioCLIP 2 is ahead at every block measured, from block 2 to block 22.

If the loss were gradual, BioCLIP 2 would fall further behind at each block. After 22 blocks it is ahead. Its final output scores 85.7%, so all of the fall happens after block 22.

Horizontal axis: image block, then the two final steps, rescaling (norm) and projection (output). Each point: a fresh alive-or-dead classifier trained on that block’s summary token and tested on 1,323 photos of 115 species held out from it.

QUESTION 2 · BLOCK 22 TO BLOCK 23

Between blocks 22 and 23, BioCLIP 2’s summary token becomes 4.2 points worse at telling alive from dead

84%

88%

92%

0%

50%

100%

18

19

20

21

22

23

24

norm

output

Alive or dead, from the summary token

Species naming, from the same token

91.9%

87.7%

71.9%

87.3%

Starting model

BioCLIP 2

Starting model

BioCLIP 2

−4.2 points

BioCLIP 2’s summary token at telling alive from dead: 91.85% at block 22, 87.65% at block 23 (95% interval −6.38 to −2.33). That is 4.2 of the 6.2 points lost between block 22 and the output. Another 1.7 is lost at the final projection.

71.9% → 87.3%

Species naming from the same token at the same step. It rose more one block earlier, from 42.5% to 71.9%, while the alive-or-dead score moved from 93.0% to 91.9%.

Horizontal axis: image block 18 to 24, then the two final steps, rescaling (norm) and projection (output). Shaded: the step from block 22 to block 23. Over that step the starting model changes by +0.15 points at telling alive from dead and its species naming rises 15 points.

QUESTION 2 · THE PATCH TOKENS

BioCLIP 2’s patch tokens still tell alive from dead at block 24: 91.0%, against 87.0% from its summary token

84%

86%

88%

90%

92%

94%

18

19

20

21

22

23

24

norm

output

BioCLIP 2, summary token

BioCLIP 2, patch tokens

Starting model, summary token

Starting model, patch tokens

Patch tokens: 91.2% at block 23, 91.0% at block 24

Summary token: 87.7%, 87.0%

91.0%

Accuracy at telling alive from dead read from the average of BioCLIP 2’s 256 patch tokens at block 24.

After block 23 BioCLIP 2 still holds what is needed to tell alive from dead. The summary token, the one token that becomes the output, carries less of it.

Solid lines: summary token. Dotted lines: average of the 256 patch tokens, one per patch of the photo. The same 1,323 photos and kind of classifier for all four lines. After block 24 come the final rescaling (norm) and projection (output).

QUESTION 2 · WHICH PATCHES THE SUMMARY TOKEN DRAWS ON

On this junco photo, BioCLIP 2’s summary token at block 24 takes 89.6% of its attention from a quarter of the patches

A dark-eyed junco, a small grey and white bird, lying in an open hand above tarmac

Dark-eyed Junco. Photo (c) jessicaturtle, CC BY-NC, cropped.

Start, block 22: 16 by 16 grid of the summary token’s attention over the junco photoBioCLIP 2, block 22: 16 by 16 grid of the summary token’s attention over the junco photoStart, block 23: 16 by 16 grid of the summary token’s attention over the junco photoBioCLIP 2, block 23: 16 by 16 grid of the summary token’s attention over the junco photoStart, block 24: 16 by 16 grid of the summary token’s attention over the junco photoBioCLIP 2, block 24: 16 by 16 grid of the summary token’s attention over the junco photo

Block 22

Block 23

Block 24

73.4%

89.6%

Starting model

BioCLIP 2

Each tile shows how much the summary token draws on each of the 256 patches at that block. Palest is none, copper is most.

The percentage under block 24 is the share of attention on the 64 patches that model attends to most; the starting model’s takes 73.4%. Only 28 of those 64 are the same in both models, and BioCLIP 2 puts 46.6% of its attention on the starting model’s 64.

Both models draw most on the bird’s head and shoulders. On all eight photos checked, BioCLIP 2’s top 64 patches take a larger share than the starting model’s top 64.

Each tile averages the block’s parallel attention steps (heads) and leaves out the attention the summary token pays to itself. All six tiles share one colour scale.

QUESTION 2 · THE WEIGHT CHANGE AS A PICTURE

57.7% of block 1’s weight change falls where the starting model’s strongest patterns meet; in block 24, 9.04%

Block 1: the change is dark in a band down the left quarter and along the top edge, strongest in the top-left corner, and close to white elsewhere
Block 24: an even pale copper wash across the whole square with a thin diagonal line

Block 1

Block 24

57.7% of the change is inside the outlined corner

9.04% is inside the same corner

Each picture is one block’s attention query weights in BioCLIP 2 minus the same weights in the starting model. These weights set what each token looks for. White is no change.

Any table of weights can be split into ranked pairs of an input pattern and an output pattern, called directions. Columns are the starting model’s input patterns, rows its output patterns, strongest first.

The outlined corner, where the strongest patterns meet, is a sixteenth of the picture, so a change spread evenly would put 6.25% there. Block 1’s change puts 57.7% there. Block 24’s puts 9.04% there and shows a faint diagonal: a slightly larger change where an input pattern meets its own output pattern.

Each picture is 256 × 256 cells; a cell averages the size of the change over 4 × 4 of the starting model’s directions. Both pictures share one colour scale. Percentages are shares of the squared change.

QUESTION 2 · WEIGHT CHANGE, ALL 24 BLOCKS

Block 23’s weight change needs 514 directions to hold 90% of it, close to the 523 of a random change

0

200

400

600

1

7

12

18

24

Random change: 523

Teal bars: blocks 1 to 7

141 of 1,024

Directions (pairs of an input and an output pattern) needed to hold 90% of the change to block 1’s attention query weights. A small, targeted edit needs few. A random change of the same size needs 523.

From block 8 on the count is 481 to 515, close to a random change. Blocks 23 and 24 need 514 and 506, like every other late block.

Horizontal axis: image block, 1 to 24. Bars: directions needed for 90% of the squared change to the attention query weights, which set what each token looks for. On the text side every block needs 376 to 387 directions, against 391 or 392 for a random change.

QUESTION 2 · ANSWER

Two-thirds of BioCLIP 2’s fall at telling alive from dead happens at block 23, in its summary token

On all eight photos checked, BioCLIP 2’s summary token at block 24 takes a larger share of its attention from its 64 most-attended patches: 89.6% against 73.4% on the junco photo.

Late in the network, and mostly in one step

The loss does not build up block by block. Through block 22, BioCLIP 2 tells alive from dead better than the starting model at every block measured. Between blocks 22 and 23 its summary token falls 4.2 points, of the 6.2 it loses between block 22 and the output.

Only in the summary token

At block 24 the average of the patch tokens scores 91.0% and the summary token 87.0%. BioCLIP 2 still holds the information. The token that becomes the output carries less of it.

Block 23’s weights changed like any late block’s

Nothing in the weights marks block 23 out. The fall appears only when each block’s tokens are tested.

QUESTION 3

Can one model name species like BioCLIP 2 and tell alive from dead like the starting model?

A natural guess: only by training BioCLIP 2 again with telling alive from dead added as a goal, because the same weights produce both scores.

Swap blocks

Both models have the same 24 image blocks in the same layout. Copy some blocks from one model into the other and test the result.

Mix weights

Average every weight of the two models and test the result.

Read a different token

Leave BioCLIP 2 untouched and test its patch tokens in place of its summary token.

QUESTION 3 · SWAPPING BLOCKS

The starting model with BioCLIP 2’s last 12 blocks copied in names 76.6% of species photos, up from 43%

40%

60%

80%

100%

0

last 6

last 12

last 18

all 24

Alive or dead, species the classifier never saw

Species naming

43% → 77%

Species naming when the starting model keeps its first 12 image blocks and takes BioCLIP 2’s last 12. No one trained this combination. That is 33 of the 47 points between the two models.

Accuracy at telling alive from dead falls as naming rises. With BioCLIP 2’s last 6 blocks the starting model loses 1.8 points. With the last 18 it scores 84.2%, below BioCLIP 2’s 85.7%.

Horizontal axis: image blocks taken from BioCLIP 2, counted from the output end. 223 species; 1,323 photos of 115 species.

QUESTION 3 · BIOCLIP 2 WITH STARTING-MODEL BLOCKS

BioCLIP 2 with the starting model’s last 12 blocks copied in scores 92.9% at telling alive from dead

84%

86%

88%

90%

92%

94%

Alive or dead, unseen species

91.4%

Start

85.7%

BioCLIP 2

92.9%

Hybrid

0%

25%

50%

75%

100%

Species naming

43.2%

Start

90.6%

BioCLIP 2

52.3%

Hybrid

The hybrid, which no one trained: BioCLIP 2’s image blocks 1 to 12 in teal, then the starting model’s blocks 13 to 24 in violet.

It scores 92.9% at telling alive from dead, where the starting model (Start) scores 91.4% and BioCLIP 2 85.7%.

It names 52.3% of species, against the starting model’s 43.2% and BioCLIP 2’s 90.6%. Both scores are closer to those of the model that supplies blocks 13 to 24.

The hybrid keeps BioCLIP 2’s patch filters, text side and the steps after block 24. 223 species; 1,323 photos of 115 species.

QUESTION 3 · SWAPPING BLOCKS 23 AND 24

BioCLIP 2 tells alive from dead 4.1 points better with the starting model’s blocks 23 and 24 swapped in

Image blocks 1 to 24, then the final steps

Naming

Alive or dead

BioCLIP 2

90.63%

85.68%

with the starting model’s blocks 23 and 24

87.42% −3.21

89.74% +4.06

with the starting model’s block 24 only

89.45% −1.18

88.56% +2.88

with the starting model’s final steps only

91.04%

86.03%

with the starting model’s blocks 23, 24, final steps

87.22%

90.51%

Starting model with BioCLIP 2’s blocks 23 and 24

40.60%

89.79% −1.62

Teal cells are BioCLIP 2’s, violet cells the starting model’s. With blocks 23 and 24 replaced, the alive-or-dead score rises to 89.74% (95% interval on the change +1.66 to +6.38) and naming falls 3.21 points. In the last row the starting model loses 1.6 points, less than the 4.2 BioCLIP 2 loses at its own block 23.

Final steps: rescaling and projection. Naming: 223 species. Alive or dead: 1,323 photos of 115 held-out species.

QUESTION 3 · READING THE PATCH TOKENS

Read from its patch tokens, an unchanged BioCLIP 2 tells alive from dead with 89.6% accuracy, up from 85.7%

1 summary token

256 patch tokens, averaged

final steps

final steps

85.7%

89.6%

Alive or dead, unseen species

Both routes use the same photos, weights and final steps and differ only in the token read.

89.6%

Accuracy at telling alive from dead when a classifier reads the average of BioCLIP 2’s block-24 patch tokens, passed through BioCLIP 2’s own final steps. That is 3.9 points above the summary token (95% interval 1.6 to 6.2).

The summary token still names 90.6% of species photos. One run of an unchanged BioCLIP 2 produces both results.

Final steps: rescaling and projection. Read straight after block 24, before the final steps, the same patch tokens score 91.0%.

QUESTION 3 · ALL THE VARIANTS COMPARED

Reading the patch tokens makes BioCLIP 2 better at telling alive from dead without lowering its naming

84%

86%

88%

90%

92%

94%

40%

60%

80%

100%

Alive or dead ↑

Species naming →

Start

BioCLIP 2 with Start’s blocks 13 to 24

Mix 0.5

Mix 0.75

blocks 23, 24, final steps

patch tokens read, no weight changed

blocks 23 and 24

block 24

final steps

BioCLIP 2

Start with BioCLIP 2’s blocks 13 to 24

Right is better naming, up is better at telling alive from dead. The grey line is the weight mix from the starting model (Start) to BioCLIP 2. Dark points are block swaps between the two models; the hollow point swaps only the final steps.

Every weight change that raises the alive-or-dead score by more than a point lowers naming. Swapping blocks 23 and 24 gives 87.4% naming and 89.7% alive or dead; the 0.75 mix, 86.6% and 90.1%.

The teal diamond changes no weight. Patch tokens give 89.6% on alive or dead; the summary token still names 90.6%.

Final steps: rescaling and projection. 223 species; 1,323 photos of 115 held-out species. Unlabelled grey dots are mixes of 0.1, 0.25 and 0.9. The diamond pairs naming from the summary token with the alive-or-dead score from the patch tokens of the same unchanged model.

LOOKING BACK · THE WHOLE MODEL

Six measurements placed on the model, from block 1 to where photos meet names

Block 1. The weight change is concentrated: 141 directions (pairs of an input and an output pattern) of 1,024 hold 90% of it.

Block 23. In this one step BioCLIP 2’s summary token becomes 4.2 points worse at telling alive animals from dead ones. The patch tokens still score 91%.

Photo

→

256 patch tokens and 1 summary token

→

→

Rescaling and projection

→

768 numbers

→

Cosine score between the two lists of 768 numbers

24 image blocks

Blocks 8 to 24. About 500 directions hold 90% of the weight change, close to the 523 for a random change.

Output. Species named correctly from five example photos: 45.5% in the starting model, 90.6% in BioCLIP 2.

Name

→

Word table

→

12 text blocks

→

768 numbers

→

Word table. Training changed it by 88%.

Where photos meet names. Vehicle photos named correctly from words: 93.0% in the starting model, 45.8% in BioCLIP 2.

Top row: the image side. Bottom row: the text side. The fewer directions hold a weight change, the more ordered the change is. Telling alive animals from dead ones was tested on 1,323 photos of 115 species that the test’s classifier never saw.

LOOKING BACK · FORGETTING

The word “forgetting” covers three different results in BioCLIP 2

A model trained further on new data usually gets worse at things it could do before; researchers call this catastrophic forgetting. In BioCLIP 2 the scores fell in three different ways.

93.0% → 45.8%

Named from words, most vehicle photos now get a wrong name

The ten vehicle names became less alike: average similarity fell from 0.774 to 0.662. Every name measured scores higher on every vehicle photo than before. The right name rose least, so “rocket”, “streetcar” or “tank” often scores highest.

91.5% → 86.7%

Named from example photos, the same vehicle photos score 4.8 points lower

Each photo is compared with five example photos of each vehicle, with no words involved. The image side still separates the ten vehicles.

85.7%

summary token

89.6%

patch tokens

BioCLIP 2’s patch tokens still tell alive animals from dead ones

The starting model scores 91.4%. BioCLIP 2’s summary token, the one token that becomes the output, scores 85.7%. Its patch tokens score 89.6% passed through the same last steps of the model, and 91.0% read straight after block 24.

In two of the three, the model still held most of what the lower score suggested was lost.

LOOKING BACK · KEEPING BOTH SKILLS

Three methods beat the starting model at naming and BioCLIP 2 at telling alive from dead, with no retraining

All three use only the two finished models

Naming the species (scale 0 to 100%)

Telling alive animals from dead ones (scale 84% to 94%)

Starting model

43.2%

91.4%

BioCLIP 2

90.6%

85.7%

1. Average the weights

Each weight is the average of the two models’ values.

72.7%

92.2%

2. Read two tokens

BioCLIP 2 unchanged: the summary token names the species and the patch tokens tell alive from dead.

90.6%

89.6%

3. Exchange two blocks

BioCLIP 2 with the starting model’s image blocks 23 and 24.

87.4%

89.7%

Averaging the weights of a general model and a model trained from it is a known technique (Wortsman et al., 2022). Here the averaged model also keeps the starting model’s score at telling alive animals from dead ones, a task BioCLIP 2’s training never targeted.

Naming: 223 species, five example photos of each. Alive or dead: 1,323 photos of 115 held-out species.

LOOKING AHEAD

Could several specialist models share the same first 12 blocks?

What the measurements showed

A hybrid’s two scores follow the model that supplies image blocks 13 to 24. With the starting model’s last 12 blocks, BioCLIP 2 scores 92.9% on telling alive animals from dead ones and names 52.3% of species photos. With BioCLIP 2’s last 12, the starting model scores 87.7% and names 76.6%. Neither combination was trained.

The summary token can name species only in the late blocks. Neither model names 1 photo in 10 from any block up to block 16.

BioCLIP 2’s patch tokens can still tell alive animals from dead ones. They score 91.0% at block 24.

birds

plants

satellite images

Shared: blocks 1 to 12

Specialists: blocks 13 to 24

Open questions, not yet measured

1. Could one shared lower half carry several specialist upper halves, for birds, plants or satellite images, with one chosen for each photo?

2. Could training be changed so the summary token keeps the information the patch tokens still carry?

3. Does the summary token’s score also fall at one block in other models trained from CLIP?

4. How specialised can the trained model be before averaging the weights stops giving both skills?

TO TAKE AWAY

Further training on organism photos raised the share of species photos named correctly from 45.5% to 90.6%.

Much of what looked forgotten was still inside BioCLIP 2: it still tells vehicles apart from example photos (86.7%), and its patch tokens still tell alive animals from dead ones (89.6%).

Three methods combined the two skills with no retraining: average the two models’ weights, read BioCLIP 2’s patch tokens, or exchange two blocks.

If you have a general model and a model trained from it, try all three on your own pair, and test a skill the training never targeted.

PHOTO CREDITS

Photos are from iNaturalist, cropped to 224 × 224 pixels

CC BY-NC 4.0

jessicaturtle · Chloé de Mazancourt · Алексей Эбель · Alex Heffernan · Ezri · Isaac Krauss · Andrew Thompson · Gabrielle · errror44 · Andrea Angiari · Susie Prescott · sabsi31 · Bluecross · Дарья Месюсина · katharper · Ramiro Pereira Garbero · lincolnwallerr · gshugart · רעות בן אלישר · Jeff McIntyre · anf-flaxweiler · gaye_white · wyliecath · Dean Fisher · jakob_mair · Jarkko Peltoniemi · Dorota Walkiewicz · Kasey P · cajis · rigbynature · tanjapikaia · zebra1193 · dsunday · albertoaries · jeanhampson · gmurray24 · anschamo · Susanne Vogel · sara_biana · grahamwoodhouse · annabry · worthj · Татьяна Химера · jollytswift · David Bell

CC BY 4.0

Lindsey G. · Andrew Murray · Sophie Hale · Nathan May

CC0, no rights reserved

The American Red Squirrel photo among the photos only one model labels correctly

Each CC BY and CC BY-NC photo is (c) its photographer, some rights reserved. The only change is the crop the models apply to their input.

Licences: creativecommons.org/licenses/by-nc/4.0, creativecommons.org/licenses/by/4.0 and creativecommons.org/publicdomain/zero/1.0.

Photo sources