Mollusk

Mollusk is a lightweight test harness for Solana programs. It provides a simple interface for testing Solana program executions in a minified Solana Virtual Machine (SVM) environment.

It does not create any semblance of a validator runtime, but instead provisions a program execution pipeline directly from lower-level SVM components.

In summary, the main processor - process_instruction - creates minified instances of Agave's program cache, transaction context, and invoke context. It uses these components to directly execute the provided program's ELF using the BPF Loader.

Because it does not use AccountsDB, Bank, or any other large Agave components, the harness is exceptionally fast. However, it does require the user to provide an explicit list of accounts to use, since it has nowhere to load them from.

The test environment can be further configured by adjusting the compute budget, feature set, or sysvars. These configurations are stored directly on the test harness (the Mollusk struct), but can be manipulated through a handful of helpers.

Single Instructions
Instruction Chains
Account Setup Helpers
Stateful Testing with MolluskContext
Benchmarking Compute Units
Fixtures
- Generating Fixtures from Mollusk Tests
- Loading and Executing Fixtures
Inner Instructions Tracking
Register tracing
Interactive Debugging

Single Instructions

Both process_instruction and process_and_validate_instruction deal with single instructions. The former simply processes the instruction and returns the result, while the latter processes the instruction and then performs a series of checks on the result. In both cases, the result is also returned.

use {
    mollusk_svm::Mollusk,
    solana_sdk::{account::Account, instruction::{AccountMeta, Instruction}, pubkey::Pubkey},
};

let program_id = Pubkey::new_unique();
let key1 = Pubkey::new_unique();
let key2 = Pubkey::new_unique();

let instruction = Instruction::new_with_bytes(
    program_id,
    &[],
    vec![
        AccountMeta::new(key1, false),
        AccountMeta::new_readonly(key2, false),
    ],
);

let accounts = vec![
    (key1, Account::default()),
    (key2, Account::default()),
];

let mollusk = Mollusk::new(&program_id, "my_program");

// Execute the instruction and get the result.
let result = mollusk.process_instruction(&instruction, &accounts);

To apply checks via process_and_validate_instruction, developers can use the Check enum, which provides a set of common checks.

use {
    mollusk_svm::{Mollusk, result::Check},
    solana_sdk::{
        account::Account,
        instruction::{AccountMeta, Instruction},
        pubkey::Pubkey
        system_instruction,
        system_program,
    },
};

let sender = Pubkey::new_unique();
let recipient = Pubkey::new_unique();

let base_lamports = 100_000_000u64;
let transfer_amount = 42_000u64;

let instruction = system_instruction::transfer(&sender, &recipient, transfer_amount);
let accounts = [
    (
        sender,
        Account::new(base_lamports, 0, &system_program::id()),
    ),
    (
        recipient,
        Account::new(base_lamports, 0, &system_program::id()),
    ),
];
let checks = vec![
    Check::success(),
    Check::compute_units(system_processor::DEFAULT_COMPUTE_UNITS),
    Check::account(&sender)
        .lamports(base_lamports - transfer_amount)
        .build(),
    Check::account(&recipient)
        .lamports(base_lamports + transfer_amount)
        .build(),
];

Mollusk::default().process_and_validate_instruction(
    &instruction,
    &accounts,
    &checks,
);

Note: Mollusk::default() will create a new Mollusk instance without adding any provided BPF programs. It will still contain a subset of the default builtin programs. For more builtin programs, you can add them yourself or use the all-builtins feature.

Instruction Chains

Both process_instruction_chain and process_and_validate_instruction_chain deal with chains of instructions. The former processes each instruction in the chain and returns the final result, while the latter processes each instruction in the chain and then performs a series of checks on each result. In both cases, the final result is also returned.

use {
    mollusk_svm::Mollusk,
    solana_sdk::{account::Account, pubkey::Pubkey, system_instruction},
};

let mollusk = Mollusk::default();

let alice = Pubkey::new_unique();
let bob = Pubkey::new_unique();
let carol = Pubkey::new_unique();
let dave = Pubkey::new_unique();

let starting_lamports = 500_000_000;

let alice_to_bob = 100_000_000;
let bob_to_carol = 50_000_000;
let bob_to_dave = 50_000_000;

mollusk.process_instruction_chain(
    &[
        system_instruction::transfer(&alice, &bob, alice_to_bob),
        system_instruction::transfer(&bob, &carol, bob_to_carol),
        system_instruction::transfer(&bob, &dave, bob_to_dave),
    ],
    &[
        (alice, system_account_with_lamports(starting_lamports)),
        (bob, system_account_with_lamports(starting_lamports)),
        (carol, system_account_with_lamports(starting_lamports)),
        (dave, system_account_with_lamports(starting_lamports)),
    ],
);

Just like with process_and_validate_instruction, developers can use the Check enum to apply checks via process_and_validate_instruction_chain. Notice that process_and_validate_instruction_chain takes a slice of tuples, where each tuple contains an instruction and a slice of checks. This allows the developer to apply specific checks to each instruction in the chain. The result returned by the method is the final result of the last instruction in the chain.

use {
    mollusk_svm::{Mollusk, result::Check},
    solana_sdk::{account::Account, pubkey::Pubkey, system_instruction},
};

let mollusk = Mollusk::default();

let alice = Pubkey::new_unique();
let bob = Pubkey::new_unique();
let carol = Pubkey::new_unique();
let dave = Pubkey::new_unique();

let starting_lamports = 500_000_000;

let alice_to_bob = 100_000_000;
let bob_to_carol = 50_000_000;
let bob_to_dave = 50_000_000;

mollusk.process_and_validate_instruction_chain(
    &[
        (
            // 0: Alice to Bob
            &system_instruction::transfer(&alice, &bob, alice_to_bob),
            &[
                Check::success(),
                Check::account(&alice)
                    .lamports(starting_lamports - alice_to_bob) // Alice pays
                    .build(),
                Check::account(&bob)
                    .lamports(starting_lamports + alice_to_bob) // Bob receives
                    .build(),
                Check::account(&carol)
                    .lamports(starting_lamports) // Unchanged
                    .build(),
                Check::account(&dave)
                    .lamports(starting_lamports) // Unchanged
                    .build(),
            ],
        ),
        (
            // 1: Bob to Carol
            &system_instruction::transfer(&bob, &carol, bob_to_carol),
            &[
                Check::success(),
                Check::account(&alice)
                    .lamports(starting_lamports - alice_to_bob) // Unchanged
                    .build(),
                Check::account(&bob)
                    .lamports(starting_lamports + alice_to_bob - bob_to_carol) // Bob pays
                    .build(),
                Check::account(&carol)
                    .lamports(starting_lamports + bob_to_carol) // Carol receives
                    .build(),
                Check::account(&dave)
                    .lamports(starting_lamports) // Unchanged
                    .build(),
            ],
        ),
        (
            // 2: Bob to Dave
            &system_instruction::transfer(&bob, &dave, bob_to_dave),
            &[
                Check::success(),
                Check::account(&alice)
                    .lamports(starting_lamports - alice_to_bob) // Unchanged
                    .build(),
                Check::account(&bob)
                    .lamports(starting_lamports + alice_to_bob - bob_to_carol - bob_to_dave) // Bob pays
                    .build(),
                Check::account(&carol)
                    .lamports(starting_lamports + bob_to_carol) // Unchanged
                    .build(),
                Check::account(&dave)
                    .lamports(starting_lamports + bob_to_dave) // Dave receives
                    .build(),
            ],
        ),
    ],
    &[
        (alice, system_account_with_lamports(starting_lamports)),
        (bob, system_account_with_lamports(starting_lamports)),
        (carol, system_account_with_lamports(starting_lamports)),
        (dave, system_account_with_lamports(starting_lamports)),
    ],
);

It's important to understand that instruction chains should not be considered equivalent to Solana transactions. Mollusk does not impose constraints on instruction chains, such as loaded account keys or size. Developers should recognize that instruction chains are primarily used for testing program execution.

Account Setup Helpers

Because the harness has nowhere to load accounts from, every test has to hand it an explicit list. Spelling those out as raw (Pubkey, Account) tuples means packing state by hand and looking up rent-exempt minimums.

mollusk-svm-account offers a builder for each of the common account types, which converts into the pair the harness expects:

use {
    mollusk_svm_account::{Mint, Stake, System, TokenAccount},
    solana_account::Account,
    solana_pubkey::Pubkey,
};

let accounts: Vec<(Pubkey, Account)> = vec![
    // A wallet holding 10 SOL, and an empty one.
    System::new(alice).lamports(10_000_000_000).into(),
    System::new(bob).into(),
    // A mint with 9 decimals and a billion base units in supply.
    Mint::initialized(mint)
        .decimals(9)
        .supply(1_000_000_000)
        .mint_authority(alice)
        .into(),
    // Alice's token account for that mint.
    TokenAccount::initialized(alice_token_account)
        .mint(mint)
        .owner(alice)
        .balance(1_000)
        .into(),
    // ...and 5 SOL of active stake, delegated to a vote account.
    Stake::initialized(alice_stake_account)
        .staker(alice)
        .withdrawer(alice)
        .delegated_to(vote_account)
        .stake(5_000_000_000)
        .into(),
];

mollusk.process_and_validate_instruction(&instruction, &accounts, &[Check::success()]);

Balances default to the rent-exempt minimum, resolved against the account's final data length at conversion time. lamports overrides with an exact value.

Each helper also takes the SDK's own state type through new (Mint, StakeStateV2, and so on), for anything the builder methods don't reach.

Your Own Program State

For state the crate knows nothing about, implement AccountState on it and hand it to account, which sizes the data, funds it to the rent-exempt minimum, and sets the owner:

use mollusk_svm_account::{account, AccountState};

#[derive(wincode::SchemaWrite)]
struct Counter {
    count: u64,
}

impl AccountState for Counter {
    fn encode(&self) -> Vec<u8> {
        wincode::serialize(self).unwrap()
    }

    fn owner() -> Pubkey {
        MY_PROGRAM_ID
    }
}

let account = account(&Counter { count: 42 });

The built-in builders convert the other way as well.

let result = mollusk.process_and_validate_instruction(
    &instruction,
    &accounts,
    &[Check::success()],
);

let token_account = TokenAccount::try_from((
    alice_token_account,
    result.get_account(&alice_token_account).unwrap().clone(),
))
.unwrap();

mollusk.process_and_validate_instruction(
    &next_instruction,
    &[token_account.balance(500).into()],
    &[Check::success()],
);

Mollusk Config

Mollusk::config controls how results are validated. It has three fields.

panic (default true) decides what a failed check does. When true, the check panics, which is what you want from a test. Set it to false and checks report instead, returning normally so you can inspect the result yourself.

verbose (default false) prints the details of failed checks when panic is false. It has no effect while checks are panicking, since the panic already carries the message.

rent_exempt_checks (default true) asserts that no account was left in a rent state the runtime would reject.

let mut mollusk = Mollusk::default();
mollusk.config.rent_exempt_checks = false;

Stateful Testing with MolluskContext

For complex testing scenarios involving multiple instructions or persistent state between calls, MolluskContext provides a stateful wrapper around Mollusk. It automatically manages an account store and provides the same API methods without requiring explicit account management.

MolluskContext is ideal for:

Testing instruction chains where account state persists between calls
Complex program interactions that require maintaining account state
Scenarios where manually managing accounts becomes cumbersome

To use MolluskContext, you need to provide an implementation of the AccountStore trait:

use {
    mollusk_svm::{Mollusk, account_store::AccountStore},
    solana_account::Account,
    solana_instruction::Instruction,
    solana_pubkey::Pubkey,
    solana_system_interface::instruction as system_instruction,
    std::collections::HashMap,
};

// Simple in-memory account store implementation
#[derive(Default)]
struct InMemoryAccountStore {
    accounts: HashMap<Pubkey, Account>,
}

impl AccountStore for InMemoryAccountStore {
    fn get_account(&self, pubkey: &Pubkey) -> Option<Account> {
        self.accounts.get(pubkey).cloned()
    }

    fn store_account(&mut self, pubkey: Pubkey, account: Account) {
        self.accounts.insert(pubkey, account);
    }
}

let mollusk = Mollusk::default();
let context = mollusk.with_context(InMemoryAccountStore::default());

let alice = Pubkey::new_unique();
let bob = Pubkey::new_unique();

// Execute instructions without managing accounts manually
let instruction1 = system_instruction::transfer(&alice, &bob, 1_000_000);
let result1 = context.process_instruction(&instruction1);

let instruction2 = system_instruction::transfer(&bob, &alice, 500_000);
let result2 = context.process_instruction(&instruction2);

// Account state is automatically preserved between calls

The MolluskContext API provides the same core methods as Mollusk:

process_instruction: Process an instruction with automatic account management
process_instruction_chain: Process a chain of instructions
process_and_validate_instruction: Process and validate an instruction
process_and_validate_instruction_chain: Process and validate an instruction chain

All methods return InstructionResult, just like the base Mollusk methods. The resulting_accounts field will reflect the final state after execution, and these accounts are automatically stored in the context's account store.

Note that HashMap<Pubkey, Account> implements AccountStore directly, so you can use it as a simple in-memory account store without needing to implement your own.

Benchmarking Compute Units

The Mollusk Compute Unit Bencher can be used to benchmark the compute unit usage of Solana programs. It provides a simple API for developers to write benchmarks for their programs, or compare multiple implementations of their programs in a matrix, which can be checked while making changes to the program.

A markdown file is generated, which captures all of the compute unit benchmarks. In the case of single program if a benchmark has a previous value, the delta is also recorded. This can be useful for developers to check the implications of changes to the program on compute unit usage.

use {
    mollusk_svm_bencher::MolluskComputeUnitBencher,
    mollusk_svm::Mollusk,
    /* ... */
};

// Optionally disable logging.
solana_logger::setup_with("");

/* Instruction & accounts setup ... */

let mollusk = Mollusk::new(&program_id, "my_program");

MolluskComputeUnitBencher::new(mollusk)
    .bench(("bench0", &instruction0, &accounts0))
    .bench(("bench1", &instruction1, &accounts1))
    .bench(("bench2", &instruction2, &accounts2))
    .bench(("bench3", &instruction3, &accounts3))
    .must_pass(true)
    .out_dir("../target/benches")
    .execute();

The must_pass argument can be provided to trigger a panic if any defined benchmark tests do not pass. out_dir specifies the directory where the markdown file will be written.

Developers can invoke this benchmark test with cargo bench. They may need to add a bench to the project's Cargo.toml.

[[bench]]
name = "compute_units"
harness = false

The markdown file will contain entries according to the defined benchmarks.

| Name   | CUs   | Delta  |
| ------ | ----- | ------ |
| bench0 | 450   | --     |
| bench1 | 579   | -129   |
| bench2 | 1,204 | +754   |
| bench3 | 2,811 | +2,361 |

With the cpi-compute-units feature enabled, cpi_breakdown(true) adds two columns. CPIs is everything consumed inside the CPI syscalls the benched program made: the syscall's own cost (the invoke_units base cost and any per-byte charges) plus the invoked programs' execution, including their nested CPIs. Program is the rest. Deltas are computed on the total.

| Name   | CUs   | Delta  | Program | CPIs  |
| ------ | ----- | ------ | ------- | ----- |
| bench0 | 450   | --     | 450     | 0     |
| bench1 | 2,811 | +2,361 | 1,204   | 1,607 |

Matrix Benchmarking

If you want to compare multiple program implementations (e.g., comparing an optimized version against a baseline), use MolluskComputeUnitMatrixBencher. This generates a table where each program is a column.

use {
    mollusk_svm_bencher::MolluskComputeUnitMatrixBencher,
    mollusk_svm::Mollusk,
    /* ... */
};

/* Instruction & accounts setup ... */

let mut mollusk = Mollusk::new(&program_id, "program_v1");

MolluskComputeUnitMatrixBencher::new(&mut mollusk)
    .programs(&["program_v1", "program_v2", "program_v3"])
    .bench(("bench0", &instruction0, &accounts0))
    .bench(("bench1", &instruction1, &<
FFA8
/span>accounts1))
    .must_pass(true)
    .out_dir("../target/benches")
    .execute();

Fixtures

Mollusk also supports working with multiple kinds of fixtures, which can help expand testing capabilities. Note this is all gated behind either the fuzz or fuzz-fd feature flags.

A fixture is a structured representation of a test case, containing the input data, the expected output data, and any additional context required to run the test. One fixture maps to one instruction.

A classic use case for such fixtures is the act of testing two versions of a program against each other, to ensure the new version behaves as expected. The original version's test suite can be used to generate a set of fixtures, which can then be used as inputs to test the new version. Although you could also simply replace the program ELF file in the test suite to achieve a similar result, fixtures provide exhaustive coverage.

Generating Fixtures from Mollusk Tests

Mollusk is capable of generating fixtures from any defined test case. If the EJECT_FUZZ_FIXTURES environment variable is set during a test run, Mollusk will serialize every invocation of process_instruction into a fixture, using the provided inputs, current Mollusk configurations, and result returned. EJECT_FUZZ_FIXTURES_JSON can also be set to write the fixtures in JSON format.

EJECT_FUZZ_FIXTURES="./fuzz-fixtures" cargo test-sbf ...

Note that Mollusk currently supports two types of fixtures: Mollusk's own fixture layout and the fixture layout used by the Firedancer team. Both of these layouts stem from Protobuf definitions.

These layouts live in separate crates, but a snippet of the Mollusk input data for a fixture can be found below:

/// Instruction context fixture.
pub struct Context {
    /// The compute budget to use for the simulation.
    pub compute_budget: ComputeBudget,
    /// The feature set to use for the simulation.
    pub feature_set: FeatureSet,
    /// The runtime sysvars to use for the simulation.
    pub sysvars: Sysvars,
    /// The program ID of the program being invoked.
    pub program_id: Pubkey,
    /// Accounts to pass to the instruction.
    pub instruction_accounts: Vec<AccountMeta>,
    /// The instruction data.
    pub instruction_data: Vec<u8>,
    /// Input accounts with state.
    pub accounts: Vec<(Pubkey, Account)>,
}

Loading and Executing Fixtures

Mollusk can also execute fixtures, just like it can with instructions. The process_fixture method will process a fixture and return the result, while process_and_validate_fixture will process a fixture and compare the result against the fixture's effects.

An additional method, process_and_partially_validate_fixture, allows developers to compare the result against the fixture's effects using a specific subset of checks, rather than the entire set of effects. This may be useful if you wish to ignore certain effects, such as compute units consumed.

use {
    mollusk_svm::{Mollusk, result::Compare},
    solana_sdk::{account::Account, pubkey::Pubkey, system_instruction},
    std::{fs, path::Path},
};

let mollusk = Mollusk::default();

for file in fs::read_dir(Path::new("fixtures-dir"))? {
    let fixture = Fixture::load_from_blob_file(&entry?.file_name());

    // Execute the fixture and apply partial checks.
    mollusk.process_and_partially_validate_fixture(
       &fixture,
       &[
           Compare::ProgramResult,
           Compare::ReturnData,
           Compare::all_resulting_accounts(),
        ],
    );
}

Fixtures can be loaded from files or decoded from raw blobs. These capabilities are provided by the respective fixture crates.

Inner Instructions Tracking

Mollusk can track inner instructions (cross-program invocations) executed during program execution when the inner-instructions feature is enabled. This feature captures all CPIs made by a program, including the invoked instruction details and the stack height at which each CPI was called.

To enable inner instructions tracking, add the feature flag to your Cargo.toml:

[dependencies]
mollusk-svm = { version = "0.8.0", features = ["inner-instructions"] }

Once enabled, every InstructionResult will include an inner_instructions field containing a vector of all CPIs that occurred during execution. You can validate the number of CPIs using the Check::inner_instruction_count check:

use {
    mollusk_svm::{Mollusk, result::Check},
    solana_sdk::{
        account::Account,
        instruction::{AccountMeta, Instruction},
        pubkey::Pubkey,
    },
};

let program_id = Pubkey::new_unique();
let cpi_target_id = Pubkey::new_unique();

let mut mollusk = Mollusk::new(&program_id, "my_program");
mollusk.add_program(&cpi_target_id, "cpi_target_program");

/* Setup instruction that performs a CPI ... */

mollusk.process_and_validate_instruction(
    &instruction,
    &accounts,
    &[
        Check::success(),
        Check::inner_instruction_count(1), // Verify exactly 1 CPI occurred
    ],
);

The tracked inner instructions include:

The program ID being invoked
The instruction data passed to the CPI
The accounts passed to the CPI
The stack height (nesting level) of the invocation

This feature is particularly useful for:

Testing programs that make CPIs to verify correct invocation behavior
Debugging complex program interactions
Validating that the expected number of CPIs occur during execution
Inspecting the exact parameters passed to cross-program invocations

Register tracing

Mollusk can be instantiated with the capability to provide register tracing data from processed instructions. This functionality is gated behind the register-tracing feature flag, which in turn relies on the invocation-inspect-callback flag. To enable it, users can either construct Mollusk with the Mollusk::new_debuggable initializer - allowing register tracing to be configured directly - or simply set the SBF_TRACE_DIR environment variable, which Mollusk interprets as a signal to turn tracing on upon instantiation. The latter allows users to take advantage of the functionality without actually doing any changes to their code.

A default post-instruction callback is provided for storing the register tracing data in files. It persists the register sets, the SBPF instructions, and a SHA-256 hash identifying the executable that was used to generate the tracing data. If the SBF_TRACE_DISASSEMBLE environment variable is set, a disassembled register trace will also be produced for each collected register trace. The motivation behind providing the SHA-256 identifier is that files may grow in number, and consumers need a deterministic way to evaluate which shared object should be used when analyzing the tracing data.

Register tracing also powers Mollusk's interactive debugger. See Interactive Debugging.

Once enabled register tracing can't be changed afterwards because in nature it's baked into the program executables at load time. Yet a user may want a more fine-grained control over when register tracing data should be collected - for example, only for a specific instruction. Such control could be achieved by resetting the invocation callback to EmptyInvocationInspectCallback and later by restoring it to DefaultRegisterTracingCallback.

use mollusk_svm::{register_tracing, EmptyInvocationInspectCallback, Mollusk};

assert!(std::env::var("SBF_TRACE_DIR").is_ok());

/* Mollusk setup .. */
/* Load programs .. */

/* .. */
/* Have tracing here. */
/* .. */

/* Disable trace collection here. */
mollusk.invocation_inspect_callback = Box::new(EmptyInvocationInspectCallback {});
/* .. */

/* Re-enable trace collection. */
mollusk.invocation_inspect_callback =
    Box::new(register_tracing::DefaultRegisterTracingCallback {
        sbf_trace_dir: std::env::var("SBF_TRACE_DIR").unwrap(),
        sbf_trace_disassemble: std::env::var("SBF_TRACE_DISASSEMBLE").is_ok(),
        sbf_debug_port: std::env::var("SBF_DEBUG_PORT")
            .map(|port| port.parse::<u16>().ok())
            .unwrap_or_default(),
    });

Interactive Debugging

Mollusk can pause a program mid-execution and hand it to a debugger, so you can step through your program's source, set breakpoints, and inspect registers, memory, and local variables - all from an ordinary Mollusk test, with no validator involved.

Every program invocation gets its own debug session, including each CPI. A session reports which program is running, how deep it sits in the CPI stack, and who called it, so you can follow an instruction across program boundaries.

Try It

The repository ships a walkthrough built on two of its test programs: test_program_primary CPIs into test_program_cpi_target, which writes some data to an account. You need cargo build-sbf, which downloads the platform tools that ship solana-lldb, and Python 3.10's shared library (libpython3.10), which solana-lldb links against.

In one terminal, build the programs with debug info and run the walkthrough:

make debugger-demo

Execution stops before the first instruction of test_program_primary:

Debugging executable with metadata (program_id=239vxAL9Q7e3uLoinJpJ873r3bvT9sPFxH7yekwPppNF;cpi_level=0;caller=none)
Waiting for a Debugger connection on "127.0.0.1:9001"...

In a second terminal, attach solana-lldb with the program's symbols loaded:

make debugger-attach

Stop at the program's entrypoint and look around:

(lldb) process plugin packet monitor metadata
program_id=239vxAL9Q7e3uLoinJpJ873r3bvT9sPFxH7yekwPppNF;cpi_level=0;caller=none
(lldb) breakpoint set --name process_instruction
(lldb) continue
(lldb) frame variable input
(lldb) register read pc icount_remain

icount_remain is the compute budget left. Next, break on the CPI call in test-programs/primary/src/lib.rs, inspect the data it's about to send, and let the CPI happen:

(lldb) breakpoint set --file lib.rs --line 98
(lldb) continue
(lldb) frame variable data
(lldb) breakpoint disable
(lldb) continue

The caller now waits inside the CPI while a second session opens on the same port for the callee. The first terminal prints its metadata:

Debugging executable with metadata (program_id=MD24T7azhc2q9ZXaeskbLpmVA41k7StzTGgcfvGcpHj;cpi_level=1;caller=239vxAL9Q7e3uLoinJpJ873r3bvT9sPFxH7yekwPppNF)
Waiting for a Debugger connection on "127.0.0.1:9001"...

Attach to it from a third terminal, loading the callee's symbols:

make debugger-attach DEBUG_PROGRAM=test_program_cpi_target

(lldb) breakpoint set --name process_instruction
(lldb) continue
(lldb) frame variable input
(lldb) continue

Once the callee exits, the caller's continue returns too, and the walkthrough finishes with Instruction succeeded. Use SBF_DEBUG_PORT=<port> with both make targets to pick a different port.

Debugging Your Own Program

Enable the sbpf-debugger feature, which also enables register-tracing:

[dev-dependencies]
mollusk-svm = { version = "0.17", features = ["sbpf-debugger"] }

Build the program with debug info, targeting sBPF v2:
```
cargo build-sbf --debug --arch v2
```
This writes target/deploy/debug/<program>.so, a stripped object for the VM to run, and target/deploy/debug/<program>.so.debug, which holds the symbols for the debugger.

Breakpoints don't hit in sBPF v3 programs (the cargo build-sbf default) until Mollusk picks up sbpf 0.26 with Agave 4.5, which fixes the debugger's v3 program counters (anza-xyz/sbpf#224).
Run a single test with register tracing on and a debug port set. Point SBF_OUT_DIR at the debug build so Mollusk loads the object that matches the symbols:
```
SBF_OUT_DIR=target/deploy/debug SBF_TRACE_DIR=target/sbf/trace SBF_DEBUG_PORT=9001 \
    cargo test my_test -- --nocapture --test-threads=1
```
Setting SBF_TRACE_DIR turns register tracing on with no code changes (trace files are written there as well). Alternatively, construct Mollusk with Mollusk::new_debuggable(&program_id, "my_program", true).

Attach from another terminal. solana-lldb ships with the platform tools; point PYTHONPATH at its bundled Python module:

LLVM=~/.cache/solana/<version>/platform-tools/llvm
PYTHONPATH=$LLVM/lib/python3.10/dist-packages $LLVM/bin/solana-lldb
(lldb) target create target/deploy/debug/my_program.so.debug
(lldb) gdb-remote 127.0.0.1:9001

How It Works

When the debug port is set, each sBPF invocation runs in the interpreter (the JIT is bypassed) and opens a GDB Remote Serial Protocol stub on 127.0.0.1:<port>. The invocation blocks until a client connects, then runs under the client's control. When a program makes a CPI, the callee opens its own stub on the same port while the caller waits inside the CPI, so the callee must run to completion before the caller can continue.

Each stub answers the metadata monitor command with the program ID, the CPI level, and the caller's program ID (none at the top level). In lldb, that's process plugin packet monitor metadata.

Mollusk also writes program_ids.map to the trace directory, mapping each program ID in the instruction (including CPI targets) to the SHA-256 hash of the ELF it loaded. Tooling can use this to find the right .so.debug for a session automatically.

The mollusk_svm::debugger module exposes the protocol helpers Mollusk's own tests use, for building scripted clients. See test_debugger in harness/tests/register_tracing.rs for an example that reads registers, memory, and metadata across a CPI.

Things to Know

Each invocation blocks until a debugger attaches. Run one test at a time (--test-threads=1). Parallel tests contend for the same port.
Use --nocapture. Otherwise the test harness hides the "Waiting for a Debugger connection" message.
Continue to completion before detaching. Disconnecting stops execution where it stands, so the instruction result won't reflect a full run.
Step with step, si, and breakpoints. next (step over) currently runs the program to completion, and bt only shows the current frame.

Name		Name	Last commit message	Last commit date
.github		.github
account		account
bencher		bencher
cli		cli
error		error
fuzz		fuzz
harness		harness
programs		programs
result		result
scripts		scripts
test-programs		test-programs
.gitignore		.gitignore
Cargo.lock		Cargo.lock
Cargo.toml		Cargo.toml
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
rust-toolchain.toml		rust-toolchain.toml
rustfmt.toml		rustfmt.toml

Navigation Menu

Repository navigation

About

Repository files navigation

Mollusk

Single Instructions

Instruction Chains

Account Setup Helpers

Your Own Program State

Mollusk Config

Stateful Testing with MolluskContext

Benchmarking Compute Units

Matrix Benchmarking

Fixtures

Generating Fixtures from Mollusk Tests

Loading and Executing Fixtures

Inner Instructions Tracking

Register tracing

Interactive Debugging

Try It

Debugging Your Own Program

How It Works

Things to Know

About

Releases

Packages

Used by

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

Mollusk

Single Instructions

Instruction Chains

Account Setup Helpers

Your Own Program State

Mollusk Config

Stateful Testing with MolluskContext

Benchmarking Compute Units

Matrix Benchmarking

Fixtures

Generating Fixtures from Mollusk Tests

Loading and Executing Fixtures

Inner Instructions Tracking

Register tracing

Interactive Debugging

Try It

Debugging Your Own Program

How It Works

Things to Know

About

Resources

Stars

Watchers

Forks

Releases

Packages

Used by

Contributors

Languages