Appearance
Contents
Model Interpretability
This section refers to the most accurate multimodal model trained, CNN 3 (see CNN Multimodal – CNN Model 3).
Interpretability of the Image Branch with Grad-Cam
Let's first look at the model interpretability of the image branch.
Grad-Cam Script Overview
The script first loads the validation data and merges it with the corresponding product labels. It then constructs the full file paths for each image based on its product and image ID. A random sample of images is selected from the validation set to provide a representative overview of model behavior. For each sampled image, the script loads the image and passes it through the trained model.
Grad-CAM is applied to the last convolutional layer to generate a heatmap that highlights the regions most relevant to the model's prediction. The resulting heatmap is overlaid on the original image and both are saved for visual inspection.
Grad-Cam Results (Sample)
Image interpretability is important because it allows us to verify that the model is using relevant visual features, such as contours and main objects, rather than background or empty spaces.
The Grad-CAM results below show that the model generally highlights the main features of the images.
Images 742113432, 1242235356, and 1308731891 are particularly good examples, where the highlighted regions closely match the main object in the original image.
Left: Original image 742113432 of product 1217640.
Right: Grad-CAM image of image 742113432 of product 1217640.
Left: Original image 1242235356 of product 3716120648.
Right: Grad-CAM image of image 1242235356 of product 3716120648.
Left: Original image 1308731891 of product 4177821854.
Right: Grad-CAM image of image 1308731891 of product 4177821854.
Interpretatbility of the Text Branch with SHAP
Text 1: revue encyclopedique ° 43 15/09 1892 academies societes savantes trawinski tassilly maison zaborowski mellion bertrand petit pellissier grand-carteret 45 gravures archeologie orient grece rome revue encyclopedique ° 43 15/09 1892 academies societes savantes trawinski tassilly maison zaborowski mellion bertrand petit pellissier grand-carteret 45 gravures archeologie orient grece rome
| Text | 10 | 40 | 50 | 60 | 1140 | 1160 | 1180 | 1280 | 1281 | 1300 | 1301 | 1302 | 1320 | 1560 | 1920 | 1940 | 2060 | 2220 | 2280 | 2403 | 2462 | 2522 | 2582 | 2583 | 2585 | 2705 | 2905 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| revue encyclopedique ° 43 15/09 1892 academies societes savantes trawinski tassilly maison zaborowski mellion bertrand petit pellissier grand-carteret 45 gravures archeologie orient grece rome revue encyclopedique ° 43 15/09 1892 academies societes savantes trawinski tassilly maison zaborowski mellion bertrand petit pellissier grand-carteret 45 gravures archeologie orient grece rome | -0.0331 | -0.0012 | -0.0004 | -0.0001 | -0.0038 | -0.0024 | -0.0054 | -0.0532 | -0.0039 | -0.0516 | -0.0008 | -0.0014 | -0.0324 | -0.1491 | -0.0330 | -0.0005 | -0.0932 | -0.0004 | 0.7022 | 0.0992 | -0.0025 | -0.0692 | -0.0225 | -0.1981 | -0.0436 | 0.0004 | -0.0000 |
Text 2: saison chez lévy saison chez lévy
| Text | 10 | 40 | 50 | 60 | 1140 | 1160 | 1180 | 1280 | 1281 | 1300 | 1301 | 1302 | 1320 | 1560 | 1920 | 1940 | 2060 | 2220 | 2280 | 2403 | 2462 | 2522 | 2582 | 2583 | 2585 | 2705 | 2905 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| saison chez lévy saison chez lévy | 0.3435 | 0.0223 | 0.0074 | 0.0009 | 0.0012 | -0.0001 | 0.0025 | -0.0496 | -0.0012 | -0.0499 | -0.0001 | -0.0011 | 0.0106 | -0.1480 | -0.0230 | 0.0027 | -0.0919 | -0.0001 | -0.0281 | 0.2834 | 0.0356 | -0.0653 | -0.0221 | -0.1991 | -0.0433 | 0.0129 | 0.0001 |
Text 3: noces feeriques disney noces feeriques disney
| Text | 10 | 40 | 50 | 60 | 1140 | 1160 | 1180 | 1280 | 1281 | 1300 | 1301 | 1302 | 1320 | 1560 | 1920 | 1940 | 2060 | 2220 | 2280 | 2403 | 2462 | 2522 | 2582 | 2583 | 2585 | 2705 | 2905 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| noces feeriques disney noces feeriques disney | 0.2284 | 0.0292 | 0.0129 | 0.0015 | 0.0640 | 0.0093 | 0.1068 | 0.0582 | 0.0404 | -0.0339 | 0.0038 | 0.0107 | 0.0970 | -0.1460 | -0.0232 | 0.0026 | -0.0909 | 0.0004 | -0.0453 | -0.0449 | 0.0425 | -0.0618 | -0.0215 | -0.1986 | -0.0436 | 0.0018 | 0.0004 |
Text 4: plongeoir dynamic flexible 1.61 bleu marine plongeoir couleur blanche polyester armé fibre verre recouvert gelcoat isoftalmique plus résistance livré ancrages dimensions measurement_value
| Text | 10 | 40 | 50 | 60 | 1140 | 1160 | 1180 | 1280 | 1281 | 1300 | 1301 | 1302 | 1320 | 1560 | 1920 | 1940 | 2060 | 2220 | 2280 | 2403 | 2462 | 2522 | 2582 | 2583 | 2585 | 2705 | 2905 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| plongeoir dynamic flexible 1.61 bleu marine plongeoir couleur blanche polyester armé fibre verre recouvert gelcoat isoftalmique plus résistance livré ancrages dimensions measurement_value | 0.0031 | -0.0007 | 0.0003 | 0.0001 | 0.0637 | -0.0008 | 0.0105 | 0.0114 | -0.0000 | -0.0378 | 0.0102 | 0.0687 | 0.0950 | -0.1138 | 0.0114 | 0.0127 | -0.0301 | 0.0020 | -0.0441 | -0.1026 | -0.0004 | 0.0558 | 0.0290 | -0.0395 | -0.0040 | -0.0009 | 0.0008 |
Text 5: cadre peinture acrylique measurement_value l'huile toile apprêtée mélange lin cadre peinture acrylique measurement_value l'huile toile apprêtée mélange lin type peinture toile utilisation peinture numéro modèle bcf 12210 age > 3 ans taille measurement_value nom marque conda matériel lin matière lin type cadre cadre grande taille utilise épaisseur élargie caractéristiques 1 technologie tissu strerch professionnel tissu base lin texture claire image peinte 2 peut peindre correctement applicable convient peintures huile pigments acryliques peintures alkydes applicable convient peintures huile pigments acryliques peintures alkydes poids paquet paquet poids measurement_value quantité carton 21 poids carton measurement_value taille carton measurement_value * measurement_value * measurement_value
| Text | 10 | 40 | 50 | 60 | 1140 | 1160 | 1180 | 1280 | 1281 | 1300 | 1301 | 1302 | 1320 | 1560 | 1920 | 1940 | 2060 | 2220 | 2280 | 2403 | 2462 | 2522 | 2582 | 2583 | 2585 | 2705 | 2905 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| cadre peinture acrylique measurement_value l'huile toile apprêtée mélange lin cadre peinture acrylique measurement_value l'huile toile apprêtée mélange lin type peinture toile utilisation peinture numéro modèle bcf 12210 age > 3 ans taille measurement_value nom marque conda matériel lin matière lin type cadre cadre grande taille utilise épaisseur élargie caractéristiques 1 technologie tissu strerch professionnel tissu base lin texture claire image peinte 2 peut peindre correctement applicable convient peintures huile pigments acryliques peintures alkydes applicable convient peintures huile pigments acryliques peintures alkydes poids paquet paquet poids measurement_value quantité carton 21 poids carton measurement_value taille carton measurement_value * measurement_value * measurement_value | -0.0630 | -0.0012 | -0.0004 | -0.0001 | -0.0048 | -0.0026 | -0.0054 | -0.0529 | -0.0039 | -0.0521 | -0.0007 | -0.0006 | -0.0089 | 0.1119 | 0.2273 | -0.0002 | 0.2789 | 0.0003 | -0.0481 | -0.1106 | -0.0024 | -0.0648 | 0.0418 | -0.1974 | -0.0390 | -0.0011 | -0.0000 |
Limitations of SHAP
With the current model architecture, SHAP cannot provide word-level attributions for text inputs.
This is because the model expects raw text strings as input and performs tokenization internally using a TextVectorization layer.
As a result, SHAP treats the entire text as a single feature, rather than a sequence of words or tokens.
Consequently, SHAP can only assign importance to the whole text input, not to individual words.
For word-level attributions, the model must accept pre-tokenized input (such as integer sequences), enabling SHAP to perturb or mask individual tokens and measure their effect on the prediction.
Interpretatbility of the Text Branch with LIME
LIME (Local Interpretable Model-agnostic Explanations) is an interpretability tool designed to explain individual model predictions.
For text data, LIME generates multiple variations of the input by removing or masking specific words and observes the resulting changes in the model’s output. This approach identifies which words have the greatest influence on the prediction for each class.
The output is a set of importance scores or highlights for each word, indicating their contribution to the model’s decision. Positive scores indicate words that push the prediction toward the class, while negative scores indicate words that push it away.
LIME is model-agnostic and can be applied to any model that accepts raw text as input.
LIME for Text Samples
Each time the script is run, it randomly selects a single text sample from the validation set. This random sampling ensures that different runs can provide explanations for different examples, offering a broader view of model behavior across the dataset.
Example Output
LIME explanation for predicted class 2583
| Word | Importance |
|---|---|
| piscine | 0.7515 |
| kerlis | 0.0599 |
| aquatique | 0.0518 |
| gonflable | 0.0343 |
| fauteuils | 0.0140 |
| deux | -0.0558 |
| fluo | -0.0382 |
| fauteuil | -0.0366 |
| stables | -0.0283 |
| pool | -0.0116 |
Text: fauteuil gonflable piscine fluo pool pod kerlis très stables deux fauteuils transforment piscine salon aquatique assortiments deux couleurs
Global LIME across all classes
The global LIME analysis provides an overview of which words are most influential for each class across the entire dataset, rather than for a single prediction.
100 text samples were randomly selected from the validation set. For each sample, LIME generates explanations for every class by perturbing the text and observing how the model’s predictions change. The script then aggregates the absolute importance scores for each word across all samples and all explanations for a given class. This aggregation highlights the words that, on average, have the greatest impact on the model’s predictions for each class. The result is a ranked list of the top influential words for every class, offering insight into the global decision patterns of the model.
Top 5 Influential words globally by class (sample of 100 texts)
| Class | Word | Aggregate Importance |
|---|---|---|
| 10 | measurement_value | 0.9361 |
| lot | 0.6779 | |
| xbox | 0.6311 | |
| maquette | 0.6229 | |
| console | 0.6079 | |
| 40 | jeu | 0.1452 |
| nintendo | 0.1258 | |
| version | 0.1193 | |
| gamecube | 0.1186 | |
| windows | 0.0839 | |
| 50 | téléchargement | 0.4375 |
| jeu | 0.2791 | |
| version | 0.2778 | |
| gamecube | 0.2238 | |
| wii | 0.1452 | |
| 60 | console | 0.1848 |
| jeu | 0.0735 | |
| téléchargement | 0.0612 | |
| rebelle | 0.0528 | |
| system | 0.0395 | |
| 1140 | kamen | 0.7320 |
| figurine | 0.6798 | |
| magnet | 0.6508 | |
| colorful | 0.2691 | |
| japan | 0.2170 | |
| 1160 | mtg | 0.8536 |
| yu | 0.8417 | |
| attax | 0.7854 | |
| commune | 0.7552 | |
| magic | 0.6898 | |
| 1180 | téléchargement | 0.1565 |
| mtg | 0.1154 | |
| jeu | 0.1129 | |
| magic | 0.1098 | |
| console | 0.0876 | |
| 1280 | générique | 1.4210 |
| toy | 0.5272 | |
| téléchargement | 0.4761 | |
| magnet | 0.4311 | |
| électronique | 0.3650 | |
| 1281 | téléchargement | 0.4394 |
| jeu | 0.3956 | |
| supplies | 0.1277 | |
| toy | 0.0908 | |
| solaire | 0.0804 | |
| 1300 | générique | 1.7283 |
| maquette | 0.9872 | |
| téléchargement | 0.5475 | |
| livre | 0.5420 | |
| jeu | 0.5054 | |
| 1301 | piscine | 0.0748 |
| promo | 0.0590 | |
| measurement_value | 0.0452 | |
| hors | 0.0258 | |
| attax | 0.0224 | |
| 1302 | stream | 2.7236 |
| jeu | 0.7602 | |
| nerf | 0.5095 | |
| gonflable | 0.2590 | |
| chien | 0.2225 | |
| 1320 | draeger | 0.4185 |
| bébé | 0.3414 | |
| measurement_value | 0.2486 | |
| éducatif | 0.2167 | |
| électronique | 0.1978 | |
| 1560 | stream | 0.5916 |
| germenly | 0.3400 | |
| measurement_value | 0.2822 | |
| chêne | 0.2031 | |
| cellule | 0.2002 | |
| 1920 | coussin | 0.8545 |
| drap | 0.6854 | |
| coton | 0.5496 | |
| stream | 0.4901 | |
| oreiller | 0.3936 | |
| 1940 | paquet | 0.1186 |
| citron | 0.0998 | |
| measurement_value | 0.0716 | |
| piscine | 0.0510 | |
| d | 0.0288 | |
| 2060 | stream | 0.8333 |
| coussin | 0.4806 | |
| d | 0.4262 | |
| germenly | 0.3988 | |
| measurement_value | 0.2949 | |
| 2220 | chien | 0.8458 |
| idéal | 0.0551 | |
| chat | 0.0541 | |
| stream | 0.0499 | |
| permanence | 0.0496 | |
| 2280 | dossier | 0.2815 |
| sauvage | 0.2462 | |
| vo | 0.2295 | |
| presse | 0.1748 | |
| 1960 | 0.1739 | |
| 2403 | lot | 1.2401 |
| volumes | 0.6616 | |
| livre | 0.4284 | |
| travail | 0.4117 | |
| livres | 0.3891 | |
| 2462 | console | 1.3615 |
| xbox | 0.8702 | |
| jeu | 0.2467 | |
| attax | 0.2258 | |
| version | 0.2246 | |
| 2522 | draeger | 0.8600 |
| classement | 0.7412 | |
| stylo | 0.6201 | |
| chemises | 0.5750 | |
| piscine | 0.4809 | |
| 2582 | measurement_value | 0.5715 |
| piscine | 0.3209 | |
| dîner | 0.3145 | |
| teck | 0.2737 | |
| stream | 0.2693 | |
| 2583 | piscine | 1.9446 |
| cellule | 0.9374 | |
| bayrol | 0.9266 | |
| condensateur | 0.7608 | |
| jeu | 0.6521 | |
| 2585 | measurement_value | 0.4861 |
| tronçonneuse | 0.4615 | |
| lien | 0.4209 | |
| silencieux | 0.2497 | |
| spa | 0.2490 | |
| 2705 | livre | 0.2872 |
| où | 0.2024 | |
| l | 0.1829 | |
| mauvaise | 0.1418 | |
| etats | 0.1369 | |
| 2905 | téléchargement | 2.3772 |
| jeu | 0.2222 | |
| utiliser | 0.1147 | |
| note | 0.0917 | |
| dlc | 0.0865 |
Top 5 Influential words globally by class (sample of 600 texts)
| Class | Word | Aggregate Importance |
|---|---|---|
| 2583 (2) | piscine | 16.1992 |
| measurement_value | 2.1359 | |
| m3 | 1.8849 | |
| chlore | 1.8514 | |
| douche | 1.6516 | |
| 1302 | stream | 10.3993 |
| jeu | 1.6673 | |
| trampoline | 1.1178 | |
| measurement_value | 0.6299 | |
| coussin | 0.5864 | |
| 1300 | générique | 8.4748 |
| maquette | 2.7140 | |
| ho | 1.1105 | |
| téléchargement | 1.0943 | |
| drone | 1.0915 | |
| 1140 | figurine | 6.4371 |
| pop | 1.1628 | |
| kamen | 0.7337 | |
| logo | 0.6567 | |
| measurement_value | 0.6488 | |
| 1280 | générique | 5.6198 |
| measurement_value | 1.6851 | |
| doudou | 1.6472 | |
| toy | 1.4146 | |
| jouets | 1.3362 | |
| 1920 | coussin | 5.7599 |
| taie | 4.2629 | |
| oreiller | 2.3914 | |
| piscine | 1.2464 | |
| stream | 1.1891 | |
| 2905 (1) | téléchargement | 5.3811 |
| jeu | 0.4039 | |
| utiliser | 0.2055 | |
| note | 0.2050 | |
| propos | 0.1506 | |
| 10 | measurement_value | 4.5938 |
| lot | 2.9278 | |
| figurine | 2.2096 | |
| livres | 1.6262 | |
| vf | 1.4783 | |
| 2403 | lot | 4.0343 |
| livres | 2.6407 | |
| volumes | 1.8383 | |
| measurement_value | 1.7270 | |
| figurine | 1.3451 | |
| 2060 | lampe | 2.8920 |
| stream | 2.8815 | |
| taie | 2.5881 | |
| coussin | 2.4300 | |
| piscine | 2.3741 | |
| 1320 | bébé | 2.9919 |
| measurement_value | 1.9311 | |
| poussette | 1.4738 | |
| d | 0.9074 | |
| doudou | 0.6889 | |
| 2585 | piscine | 3.2275 |
| measurement_value | 2.1295 | |
| stream | 1.2226 | |
| outil | 0.9295 | |
| tondeuse | 0.9266 | |
| 2582 | piscine | 3.0865 |
| measurement_value | 2.6044 | |
| stream | 2.1300 | |
| jardin | 1.6949 | |
| d | 1.0211 | |
| 2462 | xbox | 2.1025 |
| console | 2.0692 | |
| jeux | 1.7040 | |
| psp | 1.5831 | |
| lot | 1.0744 | |
| 1160 | vf | 2.5338 |
| carte | 2.1621 | |
| pokemon | 2.1582 | |
| mtg | 1.8446 | |
| commune | 1.4598 | |
| 1560 | supplies | 1.5151 |
| matelas | 1.3829 | |
| measurement_value | 1.3307 | |
| stream | 1.2944 | |
| coussin | 1.1387 | |
| 1281 | jeu | 1.6474 |
| game | 1.2123 | |
| téléchargement | 1.1717 | |
| warhammer | 0.6821 | |
| jeux | 0.4971 | |
| 2522 (3) | stylo | 1.6847 |
| measurement_value | 1.6431 | |
| draeger | 1.5950 | |
| lindner | 1.3435 | |
| agenda | 0.9237 | |
| 50 | téléchargement | 0.9691 |
| psp | 0.9349 | |
| wii | 0.6580 | |
| jeu | 0.5847 | |
| joystick | 0.5637 | |
| 2220 | chien | 1.7897 |
| chat | 0.4242 | |
| stream | 0.2425 | |
| measurement_value | 0.2190 | |
| piscine | 0.1983 | |
| 40 | version | 0.9174 |
| import | 0.4959 | |
| jeu | 0.3922 | |
| japan | 0.3556 | |
| edition | 0.3031 | |
| 1940 | bio | 0.7374 |
| sucre | 0.4538 | |
| measurement_value | 0.4194 | |
| piscine | 0.1721 | |
| dosettes | 0.1464 | |
| 2280 | sauvage | 0.6107 |
| paris | 0.4537 | |
| parisien | 0.3684 | |
| magazine | 0.3645 | |
| rail | 0.3538 | |
| 2705 | où | 0.3506 |
| tome | 0.2975 | |
| comme | 0.2934 | |
| livre | 0.2898 | |
| entre | 0.2761 | |
| 1180 | warhammer | 1.1357 |
| téléchargement | 0.3407 | |
| jeu | 0.3322 | |
| edition | 0.2699 | |
| mtg | 0.2695 | |
| 1301 | piscine | 0.1058 |
| measurement_value | 0.0978 | |
| warhammer | 0.0865 | |
| carte | 0.0765 | |
| 1 | 0.0753 | |
| 60 | console | 0.3345 |
| téléchargement | 0.1392 | |
| jeu | 0.1298 | |
| game | 0.1076 | |
| ecran | 0.0560 |
(1) – Class with highest F1 score in predictions (2) – Class with second highest F1 score in predictions (3) – Class with third highest F1 score in predictions