This repository is deprecated and preserved for archival purposes only.
For the current WOMBAT repository, please use:
https://github.com/gmum/wombat
This repository contains the code used to conduct the experiments mentioned in the paper WOMBAT: Whitebox Oracle for Molecular Benchmarking and Attribution Testing.
A link to the paper will be coming soon.
The dataset used for validating the whiteboxes (sourced from PubChem, as described in the paper) can be found here: https://huggingface.co/datasets/dmtsh/wombat-smiles
artifacts/explainability_method_tester/20260501_115658contains raw data from testing explainability methods on readout r_1.artifacts/explainability_method_tester/20260429_221014contains raw data from testing explainability methods on readout r_2.
These artifacts are used in various notebooks in the repository for the purpose of performing qualitative and quantitative analysis.
-
create_dataset_local.pyconvertsttl.gzfiles downloaded from PubChem's FTP server into a CSV. -
filter_dataset_local.pyfilters the aforementioned CSV file (as described in the paper) and generates ECFP4 fingerprints for all molecules. -
dataset/pubchem_processed_smiles_dataset.pycontains the code used to sample from all the filtered PubChem molecules to create whitebox validation datasets (as described in the paper). -
xai_testers/explainability_method_tester.pythen downsamples the whitebox validation datasets to evaluate XAI methods, as described in the paper.
-
The MPNN architecture (as described in the paper) is implemented in
mpnn/mpnn_arch.py; the input encoding is located inmpnn/molecule_converter.py. -
The whitebox validation code is located in the files
test.pyandmpnn/validate_whiteboxes.py. -
Whiteboxes for specific patterns (as described in the paper) are localised in the following files:
- Pattern 1:
mpnn/molecule_c_detector.py - Pattern 2:
mpnn/molecule_e_detector.py - Pattern 3:
mpnn/molecule_f_detector.py - Pattern 4:
mpnn/molecule_g_detector.py - Pattern 5:
mpnn/molecule_h_detector.py - Pattern 6:
mpnn/molecule_i_detector.py - Pattern 7:
mpnn/molecule_j_detector.py - Pattern 8:
mpnn/molecule_k_detector.py - Pattern 9:
mpnn/molecule_l_detector.py - Pattern 10:
mpnn/molecule_m_detector.py - Pattern 11:
mpnn/molecule_n_detector.py - Pattern 12:
mpnn/molecule_o_detector.py - Pattern 13:
mpnn/molecule_p_detector.py - Pattern 14:
mpnn/molecule_q_detector.py
- Pattern 1:
-
The file
mpnn/visualize_activations.pycontains functions that can be used to visualise internal activations in MPNNs. It highlights which elements of the latent vector had values > 0.5, essentially meaning they were activated.- It uses a specific notation, where the activation of the n-th element of the latent vector in the i-th layer is denoted as
{N-th letter of the alphabet}.{i}. For example,F.2would be the sixth element of the activation vector after the second MPNN layer.
- It uses a specific notation, where the activation of the n-th element of the latent vector in the i-th layer is denoted as
-
Models, along with their respective SMARTS, can be found in the file
krfp_models.py. -
krfp_smarts.jsoncontains all KRFP SMARTS;krfp_vis.pyis a helper script to visualise them (we used it to pick acyclical motifs that seemed tractable).
-
highlight_smarts.pyis used to derive the ground truth from SMARTS strings. -
The directory
xai_methodscontains classes implementing theAttributionMethodinterface, returning the attributions of a given model according to a specific XAI method. -
The directory
xai_testerscontainsexplainability_method_tester.py, which performs tests of all explainability methods (as described in the paper) and saves them (with a timestamp) to theartifacts/explainability_method_testerdirectory. It also containsgnn_explainer_grid.pyandpgexplainer_grid.py, used to test more specific hyperparameters of these explainers.
create_figures.ipynbis used to show the distributions of Tanimoto and Tversky distances towards specific patterns and to generate tables for the main results.gnn_explainer_grid.ipynbandpge_grid_results.ipynbare used to analyse the results of GNN Explainer and PGE Explainer.show_explanations.ipynbis used to generate an example explanation for visualisation purposes in the paper. It relies on thedatavis.pyfile.- For further qualitative analysis, the notebooks
why_ig_fails.ipynbandwhy_shap_fails.ipynbare also provided.