From 125c3f63e6d7c03a3bf5f2779466e17c8a6b9b33 Mon Sep 17 00:00:00 2001 From: Nic Volanschi Date: Tue, 5 Dec 2023 12:38:26 +0100 Subject: [PATCH 01/10] doc: split DAL page in several pieces --- docs/shell/dal.rst | 167 ++---------------------------------- docs/shell/dal_overview.rst | 67 +++++++++++++++ docs/shell/dal_p2p.rst | 34 ++++++++ docs/shell/dal_slots.rst | 57 ++++++++++++ 4 files changed, 167 insertions(+), 158 deletions(-) create mode 100644 docs/shell/dal_overview.rst create mode 100644 docs/shell/dal_p2p.rst create mode 100644 docs/shell/dal_slots.rst diff --git a/docs/shell/dal.rst b/docs/shell/dal.rst index e5957f9ab4c1..ae31c393b692 100644 --- a/docs/shell/dal.rst +++ b/docs/shell/dal.rst @@ -1,163 +1,14 @@ -Data-Availability Layer +Data Availability Layer ======================= -Overview -^^^^^^^^ +The Data-Availability Layer (or DAL) is a software component of Octez which allows users to publish data outside of the Tezos Layer 1 blocks, with no compromises on decentralization principles. +Its primary use case today is to scale the data bandwidth of :doc:`smart rollups <../active/smart_rollups>`, but other applications could be envisioned in the future. -The *Data-Availability Layer* (DAL) enables users to publish blobs -of data outside of the Tezos Layer 1 (L1) blocks. A blob (for “binary -large object”) is a piece of data in binary form. While the primary -use case for these blobs is to store Layer 2 (L2) operations for Tezos -smart rollups, it is important to note that the DAL is more generic -and could be used for other use cases in the future. +The following pages provide an overview of the DAL and cover its different aspects and features. -In practice, the DAL employs an additional P2P protocol to publish -blobs, enabling a better bandwidth sharing between the peers (further -elaborated in the P2P section below). The DAL aims to support a -bandwidth of 10 MiB/s, a stark contrast to the current situation on -L1, which has an approximate bandwidth of 32 KiB/s. This highlights -the significant boost in bandwidth provided by the DAL. +.. toctree:: + :maxdepth: 2 -Thanks to the DAL, the data bandwidth constraints imposed by the L1 -block size are put back by several orders of magnitudes. This -translates into reduced fees for users when posting L2 operations -without compromising the fundamental principle of decentralization. - -The amount of data that can be transmitted over the DAL network is, -however, still controlled by the economic protocol. The economic -protocol also plays a crucial role in determining the availability of -those data through the participation of bakers. - -Similarly to the Tezos L1, the DAL is permissionless, enabling any -user to effectively contribute data to it, and allowing any smart -rollup kernel or smart rollup operator to access this data. - -.. |DAL overview| image:: dal_overview.png - -|DAL overview| - -The figure illustrates the standard process for interacting with the -DAL as follows: - -The diagram depicts a scenario where a user intends to upload data for -a dedicated rollup. - -Anyone engaging with the DAL must utilize a tool known as the *DAL -node* (named ``octez-dal-node``). When a user decides to provide a new -blob to the DAL (depicted as step 1 in the diagram), the user -transmits the data to the DAL node to calculate a commitment to the -data. This commitment is then communicated to L1 via a dedicated -operation (indicated as step 2). Following L1’s approval of this -operation (step 3), the DAL expands this data for redundancy, -partitions the expanded data into segments called shards, and -disseminates the shards across the DAL network (shown in step 4). -Shards are assigned to bakers proportionally to their stake by the -economic protocol. Hence, the bakers, also connected to the DAL -network, retrieve these assigned shards (step 5). The bakers must -download and verify these shards within a specific timeframe, -precisely defined by the economic protocol as the ``attestation lag`` -period. At the end of the attestation lag period, bakers declare using -another dedicated operation, whether they were able to download the -shards effectively (illustrated in step 6). The economic protocol -collates these attestations and, if a sufficient number of bakers have -successfully obtained the shards, the data is declared as available -(step 7). Only when the data is labeled as available can the rollup -utilize it (represented as step 8). - -The rationale for having the attestation lag parameter is to give -bakers sufficient time to download their assigned shards and to -guarantee that the latency stays within acceptable limits (around one -minute). - -Slots -^^^^^ - -There is a limited number of blobs, called *DAL slots* (or slots for -short), that can be published at a L1 level. A slot is identified by -the L1 level at which it is published, an index, and its -commitment. Depending on the context, a “slot” may refer to the data -blob or to the placeholder (the slot index) of that blob. Each slot is -treated independently of other slots. Presently, all slots are of the -same size, though this may change with future updates. To publish a -slot, it is first expanded to create redundancy through erasure -coding, increasing its size by a factor known as the *redundancy -factor*. This expanded slot is then divided into a set of smaller data -pieces of equal size, called *shards*. Finally, shards are distributed -across the DAL P2P network. - -The advantage of using erasure coding is that one only needs a subset -of the shards, specifically, “number of shards / redundancy factor” to -recover the full original slot data. A key feature for the shards is -that each shard can be verified to ensure it corresponds to a -particular commitment, which safeguards against spam in the DAL P2P -network. - -For each slot index, multiple commitments can be included in a block, -but only one will be acknowledged, the first one appearing in the -block ordering. Because bakers typically prioritize operations that -offer higher fees, the commitment with the higher fee is usually -chosen for inclusion, although this outcome is not guaranteed if a -baker uses a different selection criterion. - -For smart rollups, it is necessary to divide the slot into smaller -segments called *pages* (see -:ref:`populating_the_reveal_channel`). The economic protocol specifies -the size of each page as a constant, with 4KiB being a practical -size. While it might seem feasible to choose shards fitting this page -limit and directly feed shards to the rollup, accessing the original -data from shards involves complex cryptographic computations. We -prefer to avoid performing such computations within the smart -rollups. Therefore, even though the DAL network uses shards -internally, the DAL node can serve individual pages as requested by -the smart-rollup node. These pages can then be imported individually -by the smart rollup node to the kernel on demand. To reconstruct the -entire slot, one simply needs to arrange these pages in the correct -sequence. For developers working with smart-rollup kernels, the -technicalities of reconstructing the entire slot out of pages should -ideally be handled by the SDK they use, simplifying the process even -more. - -.. |DAL slot| image:: dal_slot.png - -|DAL slot| - - -When the slot producer (user publishing a slot) posts the commitment -onto the L1, it also posts a proof that the slot does not exceed the -slot size. This prevents malicious users from producing and posting -data larger than the expected size. - -The DAL's P2P protocol -^^^^^^^^^^^^^^^^^^^^^^ - -The peer-to-peer (P2P) protocol for the DAL is made out of two components: - -- A gossipsub algorithm, instantiated as detailed below. -- A transport layer for handling connections with peers. - We reuse for that the P2P protocol used by the Octez node (see :doc:`./p2p`). - -The gossip algorithm used for the DAL is an in-house version of the gossipsub v1.1 P2P protocol defined by the lib-p2p project. A detailed overview of this protocol is available `here `__ and an informal English specification can be found `here `__. This gossip algorithm allows to partition the network into virtual subnetworks, each identified by a **topic**. The topic also determines the valid data that can be exchanged over the corresponding virtual subnetwork, as any exchanged message has exactly one associated topic. Each peer subscribes to topics of interest to him. This protocol enhances the network's scalability compared to traditional gossip algorithms. - -For each message, there is a **message id** that uniquely identifies this message. When a message is pushed, it comes with its message id. When the message is pulled, it is done via the message id. -For every topic a node subscribes to, it maintains a virtual subnetwork, or **mesh**, of peers also subscribed to that topic. When a node has a new message to share (originating from the application layer) or needs to relay a received message, it does so to all peers in the corresponding topic's mesh. Moreover, the node broadcasts the ids of the last received messages to a random selection of peers outside the mesh. Peers receiving these teasers can request the full message if they are interested in it. -For the DAL instantiation of gossipsub, a message is defined as a 3-tuple: a shard, the shard’s index, and the shard’s proof proving that the shard corresponds to the commitment given by the message id. The associated message id consists of the shard index and the associated slot index, (published) level, slot commitment, and attestor’s public key hash. - -A topic is defined as a pair ``(slot_index, public_key_hash)``. The first component identifies the slot associated to any shard published under this topic, while the second component identifies the baker assigned to this shard. -Such a set of topics ensures that the bandwidth of bakers and slot producers is bounded (for valid messages) over a cycle. - -A slot producer should subscribe to all relevant topics associated with their slot index. This includes every topic where a baker is assigned at least one shard for that slot index. -On the other hand, a baker should subscribe to all topics that feature their public key address. - -Gossipsub also defines a notion of score which is used to only connect to peers with a good score. - -Regarding peer discovery, the current implementation of the DAL relies on gossipsub v1.1 peer exchanges. In particular, DAL nodes can be configured in bootstrap mode to facilitate peer discovery. - -.. note:: - - The current topic structure in the DAL for Tezos may be revised in a future update. Presently, topics include the bakers’ address (public key hash), which leads to a potentially unbounded number of topics over time. Another approach under consideration involves using a ``(slot_index, shard_index)`` pair, offering a more scalable solution in the long run, when the number of attesters surpasses the number of slots. - -.. warning:: - - Attention must be paid to the security implications for bakers in the DAL network. Since a baker's bandwidth is proportional to their stake, it can become relatively straightforward to identify the IP address of their DAL node, particularly for those with substantial stakes. To mitigate this risk, bakers are advised to operate their DAL node using an IP address different from their L1 node. This separation helps in preventing the unintentional exposure of the L1 node's IP address. - - Plans are underway to address these concerns. One proposed solution is to enable bakers to divide their bandwidth across multiple DAL nodes, enhancing both security and operational flexibility. + dal_overview + dal_slots + dal_p2p diff --git a/docs/shell/dal_overview.rst b/docs/shell/dal_overview.rst new file mode 100644 index 000000000000..67396ebdaf48 --- /dev/null +++ b/docs/shell/dal_overview.rst @@ -0,0 +1,67 @@ +DAL overview +============ + +The *Data-Availability Layer* (DAL) enables users to publish blobs +of data outside of the Tezos Layer 1 (L1) blocks. A blob (for “binary +large object”) is a piece of data in binary form. While the primary +use case for these blobs is to store Layer 2 (L2) operations for Tezos +smart rollups, it is important to note that the DAL is more generic +and could be used for other use cases in the future. + +In practice, the DAL employs an additional P2P protocol to publish +blobs, enabling a better bandwidth sharing between the peers (further +elaborated in the P2P section below). The DAL aims to support a +bandwidth of 10 MiB/s, a stark contrast to the current situation on +L1, which has an approximate bandwidth of 32 KiB/s. This highlights +the significant boost in bandwidth provided by the DAL. + +Thanks to the DAL, the data bandwidth constraints imposed by the L1 +block size are put back by several orders of magnitudes. This +translates into reduced fees for users when posting L2 operations +without compromising the fundamental principle of decentralization. + +The amount of data that can be transmitted over the DAL network is, +however, still controlled by the economic protocol. The economic +protocol also plays a crucial role in determining the availability of +those data through the participation of bakers. + +Similarly to the Tezos L1, the DAL is permissionless, enabling any +user to effectively contribute data to it, and allowing any smart +rollup kernel or smart rollup operator to access this data. + +.. |DAL overview| image:: dal_overview.png + +|DAL overview| + +The figure illustrates the standard process for interacting with the +DAL as follows: + +The diagram depicts a scenario where a user intends to upload data for +a dedicated rollup. + +Anyone engaging with the DAL must utilize a tool known as the *DAL +node* (named ``octez-dal-node``). When a user decides to provide a new +blob to the DAL (depicted as step 1 in the diagram), the user +transmits the data to the DAL node to calculate a commitment to the +data. This commitment is then communicated to L1 via a dedicated +operation (indicated as step 2). Following L1’s approval of this +operation (step 3), the DAL expands this data for redundancy, +partitions the expanded data into segments called shards, and +disseminates the shards across the DAL network (shown in step 4). +Shards are assigned to bakers proportionally to their stake by the +economic protocol. Hence, the bakers, also connected to the DAL +network, retrieve these assigned shards (step 5). The bakers must +download and verify these shards within a specific timeframe, +precisely defined by the economic protocol as the ``attestation lag`` +period. At the end of the attestation lag period, bakers declare using +another dedicated operation, whether they were able to download the +shards effectively (illustrated in step 6). The economic protocol +collates these attestations and, if a sufficient number of bakers have +successfully obtained the shards, the data is declared as available +(step 7). Only when the data is labeled as available can the rollup +utilize it (represented as step 8). + +The rationale for having the attestation lag parameter is to give +bakers sufficient time to download their assigned shards and to +guarantee that the latency stays within acceptable limits (around one +minute). diff --git a/docs/shell/dal_p2p.rst b/docs/shell/dal_p2p.rst new file mode 100644 index 000000000000..39eb5fbe195a --- /dev/null +++ b/docs/shell/dal_p2p.rst @@ -0,0 +1,34 @@ +The DAL's P2P protocol +====================== + +The peer-to-peer (P2P) protocol for the DAL is made out of two components: + +- A gossipsub algorithm, instantiated as detailed below. +- A transport layer for handling connections with peers. + We reuse for that the P2P protocol used by the Octez node (see :doc:`./p2p`). + +The gossip algorithm used for the DAL is an in-house version of the gossipsub v1.1 P2P protocol defined by the lib-p2p project. A detailed overview of this protocol is available `here `__ and an informal English specification can be found `here `__. This gossip algorithm allows to partition the network into virtual subnetworks, each identified by a **topic**. The topic also determines the valid data that can be exchanged over the corresponding virtual subnetwork, as any exchanged message has exactly one associated topic. Each peer subscribes to topics of interest to him. This protocol enhances the network's scalability compared to traditional gossip algorithms. + +For each message, there is a **message id** that uniquely identifies this message. When a message is pushed, it comes with its message id. When the message is pulled, it is done via the message id. +For every topic a node subscribes to, it maintains a virtual subnetwork, or **mesh**, of peers also subscribed to that topic. When a node has a new message to share (originating from the application layer) or needs to relay a received message, it does so to all peers in the corresponding topic's mesh. Moreover, the node broadcasts the ids of the last received messages to a random selection of peers outside the mesh. Peers receiving these teasers can request the full message if they are interested in it. +For the DAL instantiation of gossipsub, a message is defined as a 3-tuple: a shard, the shard’s index, and the shard’s proof proving that the shard corresponds to the commitment given by the message id. The associated message id consists of the shard index and the associated slot index, (published) level, slot commitment, and attestor’s public key hash. + +A topic is defined as a pair ``(slot_index, public_key_hash)``. The first component identifies the slot associated to any shard published under this topic, while the second component identifies the baker assigned to this shard. +Such a set of topics ensures that the bandwidth of bakers and slot producers is bounded (for valid messages) over a cycle. + +A slot producer should subscribe to all relevant topics associated with their slot index. This includes every topic where a baker is assigned at least one shard for that slot index. +On the other hand, a baker should subscribe to all topics that feature their public key address. + +Gossipsub also defines a notion of score which is used to only connect to peers with a good score. + +Regarding peer discovery, the current implementation of the DAL relies on gossipsub v1.1 peer exchanges. In particular, DAL nodes can be configured in bootstrap mode to facilitate peer discovery. + +.. note:: + + The current topic structure in the DAL for Tezos may be revised in a future update. Presently, topics include the bakers’ address (public key hash), which leads to a potentially unbounded number of topics over time. Another approach under consideration involves using a ``(slot_index, shard_index)`` pair, offering a more scalable solution in the long run, when the number of attesters surpasses the number of slots. + +.. warning:: + + Attention must be paid to the security implications for bakers in the DAL network. Since a baker's bandwidth is proportional to their stake, it can become relatively straightforward to identify the IP address of their DAL node, particularly for those with substantial stakes. To mitigate this risk, bakers are advised to operate their DAL node using an IP address different from their L1 node. This separation helps in preventing the unintentional exposure of the L1 node's IP address. + + Plans are underway to address these concerns. One proposed solution is to enable bakers to divide their bandwidth across multiple DAL nodes, enhancing both security and operational flexibility. diff --git a/docs/shell/dal_slots.rst b/docs/shell/dal_slots.rst new file mode 100644 index 000000000000..7806f655fd84 --- /dev/null +++ b/docs/shell/dal_slots.rst @@ -0,0 +1,57 @@ +DAL Slots +========= + +There is a limited number of blobs, called *DAL slots* (or slots for +short), that can be published at a L1 level. A slot is identified by +the L1 level at which it is published, an index, and its +commitment. Depending on the context, a “slot” may refer to the data +blob or to the placeholder (the slot index) of that blob. Each slot is +treated independently of other slots. Presently, all slots are of the +same size, though this may change with future updates. To publish a +slot, it is first expanded to create redundancy through erasure +coding, increasing its size by a factor known as the *redundancy +factor*. This expanded slot is then divided into a set of smaller data +pieces of equal size, called *shards*. Finally, shards are distributed +across the DAL P2P network. + +The advantage of using erasure coding is that one only needs a subset +of the shards, specifically, “number of shards / redundancy factor” to +recover the full original slot data. A key feature for the shards is +that each shard can be verified to ensure it corresponds to a +particular commitment, which safeguards against spam in the DAL P2P +network. + +For each slot index, multiple commitments can be included in a block, +but only one will be acknowledged, the first one appearing in the +block ordering. Because bakers typically prioritize operations that +offer higher fees, the commitment with the higher fee is usually +chosen for inclusion, although this outcome is not guaranteed if a +baker uses a different selection criterion. + +For smart rollups, it is necessary to divide the slot into smaller +segments called *pages* (see +:ref:`populating_the_reveal_channel`). The economic protocol specifies +the size of each page as a constant, with 4KiB being a practical +size. While it might seem feasible to choose shards fitting this page +limit and directly feed shards to the rollup, accessing the original +data from shards involves complex cryptographic computations. We +prefer to avoid performing such computations within the smart +rollups. Therefore, even though the DAL network uses shards +internally, the DAL node can serve individual pages as requested by +the smart-rollup node. These pages can then be imported individually +by the smart rollup node to the kernel on demand. To reconstruct the +entire slot, one simply needs to arrange these pages in the correct +sequence. For developers working with smart-rollup kernels, the +technicalities of reconstructing the entire slot out of pages should +ideally be handled by the SDK they use, simplifying the process even +more. + +.. |DAL slot| image:: dal_slot.png + +|DAL slot| + + +When the slot producer (user publishing a slot) posts the commitment +onto the L1, it also posts a proof that the slot does not exceed the +slot size. This prevents malicious users from producing and posting +data larger than the expected size. -- GitLab From 8cf3812f90e24a7cd28c5b3accc57ae3c5bd2c52 Mon Sep 17 00:00:00 2001 From: Nic Volanschi Date: Wed, 6 Dec 2023 11:07:51 +0100 Subject: [PATCH 02/10] doc: add page for DAL node --- docs/shell/dal.rst | 1 + docs/shell/dal_node.rst | 116 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 117 insertions(+) create mode 100644 docs/shell/dal_node.rst diff --git a/docs/shell/dal.rst b/docs/shell/dal.rst index ae31c393b692..4c244367190c 100644 --- a/docs/shell/dal.rst +++ b/docs/shell/dal.rst @@ -12,3 +12,4 @@ The following pages provide an overview of the DAL and cover its different aspec dal_overview dal_slots dal_p2p + dal_node diff --git a/docs/shell/dal_node.rst b/docs/shell/dal_node.rst new file mode 100644 index 000000000000..f98e37f08881 --- /dev/null +++ b/docs/shell/dal_node.rst @@ -0,0 +1,116 @@ +The DAL node +============ + +A DAL node is an executable (called ``octez-dal-node``) whose main roles are to publish, store, and exchange data for the :doc:`DAL layer <./dal_overview>`. Interacting with a DAL node is done through the command-line interface (CLI) and through RPCs. In what follows, we describe the main components and features of a DAL node, and then present some operational aspects on configuring and running a DAL node. + +Concepts and features +--------------------- + +DAL P2P network +^^^^^^^^^^^^^^^ + +The ``octez-dal-node`` executable runs a node in the DAL’s P2P network. Recall that :doc:`the DAL's P2P protocol <./dal_p2p>` is based on a gossip algorithm for distributing shards, running on top of a networking layer using the same p2p library as L1 nodes. + +Actors with various roles may need to run the DAL node, for instance bootstrap DAL node operators, slot producers, bakers. + +Bootstrap DAL nodes represent DAL network entry points. Their only role is to provide a way for other DAL nodes to become part of the DAL P2P network. A bootstrap node remains connected to a large number of peers, and is subscribed to all topics. It can thus provide another node with a list of peers to connect to for each of the topics that node is interested in. + +Non-bootstrap DAL nodes distinguish themselves only in the topics they subscribe to, for instance: + +- Slot producers subscribe to all topics containing some specified slot indexes. +- Bakers subscribe to all topics containing the attester identities they run for (for all possible slot indexes). + +Profiles +~~~~~~~~ + +These differences in the behavior of a DAL node are specified using **profiles**. There are therefore three profiles: bootstrap, producer, and attester profiles. These can be given in the node’s configuration file, or as CLI arguments to the node’s commands, or via RPCs. +The attester and producer profiles can be combined, the node will then subscribe to the union of the corresponding topics, while the bootstrap profile is incompatible with the other profiles. + +Storage +^^^^^^^ + +The DAL node essentially stores slots and shards. Slots are injected into the node through an RPC (see details at :ref:`slots_lifetime`), at which moment the corresponding commitment is computed and stored. Shards and their proofs are computed and stored via another RPC. It is important to also compute the shards’ proofs, because shards can be exchanged over the P2P network only if they are accompanied by their proof. Shards received over the P2P network are also stored. The node also tracks and stores the status of commitments by monitoring the L1 chain, connecting to this end to an L1 node specified at startup. + +The size of the node’s storage depends on its profile. A bootstrap node uses negligible storage. A DAL node with a producer profile stores in the order of tens MiB per slot, per level. Note that this translates to at least 100GiB per day per slot. A DAL node with an attester profile with the attester having a stake fraction of 1% stores an order of magnitude less data per level. + +L1 monitoring +^^^^^^^^^^^^^ + +The DAL node tracks the L1 chain, in particular it monitors the heads of the L1 chain via an RPC. At each final block, it retrieves and stores the newly published commitments, and if its storage contains shards for these commitments, it publishes them on the P2P network. The DAL node also retrieves the slot attestation status from the block’s metadata. Finally, based on this information it updates the status of the stored commitments, where the status of a commitment can be one of the following: + +- *waiting for attestation*: The commitment was included and applied in a finalized L1 block but the corresponding slot remains to be attested. +- *attested*: The commitment was included in an L1 block and the corresponding slot is attested. +- *unattested*: The commitment was included in an L1 block but the corresponding slot was not timely attested. +- *not selected*: The commitment was included in an L1 block but was not selected for that slot index. This can happen if there is another commitment in the same block for the same slot index that was placed before the commitment in question (in operation order) by the baker. +- *unseen or not finalized*: The commitment was not seen in a final L1 block. For instance, this could happen if the RPC ``PATCH /commitments/`` was called but the corresponding commitment was never included into a block; or the commitment was included in a non-final block. This means that the publish operation was not sent (yet) to L1, or sent but not included (yet) in a block, or included in a not (yet) final block. + +RPC server +^^^^^^^^^^ + +The DAL node incorporates an RPC server which answers to RPC queries to update or retrieve information about the node’s state. For instance, one can post slots, ask the node to compute and store shards, to update profiles, or to connect or disconnect to peers. For the full list of the DAL node’s RPC see :doc:`./dal_rpc`. + +Operational aspects +------------------- + +Running a DAL node +^^^^^^^^^^^^^^^^^^ + +The DAL node has two commands ``config init`` and ``run``. + +The command ``init config`` creates a new configuration file in the specified data directory or in the default location (ie ``~/.tezos-dal-node``) with the parameters provided on the command-line by the corresponding arguments, in case no configuration file exists already. If such a file already exists, it overrides it with the provided parameters (old parameters are lost). + +The command ``run`` runs the DAL node. The CLI arguments take precedence over the configuration file arguments, except for the list of bootstrap peers and of profiles, which are considered in addition to the ones from the configuration file. The configuration file is however not overridden with the new values of the node’s parameters. However, at the end of the execution, the node’s profiles, which may have been given as arguments or set via RPCs, are written to the configuration file. + +DAL node manual +~~~~~~~~~~~~~~~ + +The command line of the DAL node is documented in Unix man format, which can be obtained by running ``octez-dal-node --help``. +The arguments of each DAL node command can be listed with ``octez-node --help``. + +DAL configuration of the L1 node +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +All DAL nodes should use the same initialization parameters of the cryptographic primitives used by the DAL. These parameters are provided by running the script :src:`scripts/install_dal_trusted_setup.sh`, which downloads and installs them, and which should be run once by any DAL node operator. However, for simplicity, on some test networks the initialization parameters are mocked-up and built-in. + +Also, in order for the nodes to be able to join the P2P network, a set of bootstrap nodes can be provided using the ``network.dal_config.bootstrap_peers`` configuration parameter of the L1 node (thus using the same mechanism as for L1 nodes, see :doc:`../user/multinetwork` and :ref:`configure_p2p`). + +Monitoring +^^^^^^^^^^ + +As for L1 nodes, to monitor the DAL node, one can inspect the node’s logs, query the node through RPCs, or use the incorporated Octez Metrics server. The usage of metrics is analogous to that for the L1 nodes, and we therefore refer to the :doc:`L1 node’s documentation <../user/node-monitoring>` for details. + +The bootstrap mechanism +^^^^^^^^^^^^^^^^^^^^^^^ + +The gossipsub algorithm establishes upper bounds on the size of a topic mesh. Whenever a peer B would like to join the mesh of a peer A, but the peer A cannot add B to its mesh due to the upper bound, peer A responds with a random list of peers it knows are subscribed to the same topic. Peer B will then try to connect to these advertised peers. This peer exchange mechanism is used to bootstrap the P2P network: for bootstrap DAL nodes, the upper bound is set to 0, and thus, while they do not participate in exchanging shards, they keep advertising known peers so that new peers can connect to them and thus enter the P2P network. + + +If a node is behind a NAT, it is important to make the distinction between its "public address" and its "network address". It is the public address that is advertised, and this should be the node’s external address, while the network address should be the node’s internal address. In other words, the public address behaves as the ``--advertised-net-port`` option of the L1 node, see :ref:`private-mode`. + + +.. _slots_lifetime: + +Lifetime of slots and shards +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The life cycle of slots and shards is described by the following steps: + +#. The slot producer posts the slot data to some DAL node of its choice. The node computes the corresponding commitment and adds the association commitment - slot to the store. + This is done via the RPC ``POST /commitments/``, which returns the corresponding commitment. +#. The slot producer instructs the DAL node to compute and save the shards of the slot associated with the given commitment. + It is important to set the query flag ``with_proof`` to true to be able to publish the shards on the P2P network. + This is done via the RPC ``PUT /commitments//shards``. +#. The slot producer instructs the DAL node to compute the proof associated with the commitment. + This is done via the RPC ``GET /commitments//proof``, which returns the corresponding commitment proof. +#. The slot producer selects a slot index for its slot, and posts the commitment to L1, via the ``publish_slot_header`` operation. + This can be done via RPCs for injecting an operation into L1, or using the Octez client, via the following command:: + + octez-client publish dal commitment from for slot with proof + +#. Once the operation is included in a final block (that is, there is another block on top of the one including the operation), and the slot is considered published (see :doc:`./dal_overview`), all DAL nodes exchange the slot’s shards they have in their store on the P2P network, depending on their profile (see :doc:`./dal_p2p`), and they store previously unknown shards. +#. Attesters check, for all published slots, the availability of the shards they are assigned by interrogating their DAL node, via the RPC ``GET /profiles//attested_levels//attestable_slots``, where level is the level at which the slot was published plus ``attestation_lag``, and ``pkh`` is the attester’s public key hash. (See also :doc:`dal_bakers`) +#. Attesters inject via their baker binary a DAL attestation operation containing the information received at step 6. (See also :doc:`dal_bakers`) +#. The protocol aggregates the received DAL attestations, and declares each published slot as available or unavailable, depending on whether some threshold is reached, via the blocks metadata. +#. Rollups and other users can request stored pages or shards for an attested slot from any DAL node via the RPCs ``GET /slot/pages/`` or ``GET /shards/`` respectively. Only nodes that store enough shards to reconstruct the slot can provide the requested pages. + +Step 2 can be done in parallel with steps 3-4, but before step 5. -- GitLab From f0a6d2ea4ce7910351bf629c18f205920978d359 Mon Sep 17 00:00:00 2001 From: Nic Volanschi Date: Wed, 6 Dec 2023 11:08:18 +0100 Subject: [PATCH 03/10] doc: add page for DAL bakers --- docs/shell/dal.rst | 1 + docs/shell/dal_bakers.rst | 27 +++++++++++++++++++++++++++ 2 files changed, 28 insertions(+) create mode 100644 docs/shell/dal_bakers.rst diff --git a/docs/shell/dal.rst b/docs/shell/dal.rst index 4c244367190c..108db6b4ee29 100644 --- a/docs/shell/dal.rst +++ b/docs/shell/dal.rst @@ -13,3 +13,4 @@ The following pages provide an overview of the DAL and cover its different aspec dal_slots dal_p2p dal_node + dal_bakers diff --git a/docs/shell/dal_bakers.rst b/docs/shell/dal_bakers.rst new file mode 100644 index 000000000000..a9b3b2f1baaa --- /dev/null +++ b/docs/shell/dal_bakers.rst @@ -0,0 +1,27 @@ +Bakers & the DAL +================ + +This page documents some aspects that are most useful to know for bakers when participating to the DAL network (baking the DAL is a serious matter!). +We assume the reader is familiar with the DAL basics and terminology introduced in :doc:`./dal_overview` and :doc:`./dal_slots`. + +Bakers play a crucial role in Tezos through the Tenderbake protocol, by validating transactions and securing the network. This participation involves producing new blocks when assigned the rights by the protocol, attesting proposed valid blocks to make the chain progress, and (indirectly) slashing misbehavior that would put the chain at risk. Due to their strategic role in Tezos, entrusting them with verifying and attesting data availability using the DAL is natural. + +From a baker's point of view, DAL attestation is carried out in three main steps. + +First, the baker is attributed DAL attestation rights (automatically, based on its consensus attestation rights). Similar to Tenderbake's baking and attestation rights, the L1 protocol assigns a range of DAL shards to bakers. For a given level, the list of shards attributed to a baker is the same for every DAL slot. + +Second, Tezos bakers should launch DAL nodes and join the DAL P2P network subscribing to the appropriate topics. The topics of interest for a baker are those of the form ``(tz1TheBakerSPublicKeyHash, slot_index)`` where the first component is the baker's public key hash and the slot index ranges from ``0`` to ``number of DAL slots - 1``. Note that the baker binary automatically subscribes to those topics when launched with ``--dal-node ``. Once launched and connected to the baker's binary, the DAL node is responsible for downloading the shards assigned to the corresponding public key hash(es) for every slot via the DAL P2P network, assuming the slot commitment operation has been accepted by L1 and the slot's shards are published on the DAL/P2P (see steps 3 and 4 in :doc:`dal_overview`). + +Finally, just like for Tenderbake pre-attestations or attestations, bakers inject DAL attestations after every L1 block to inform the protocol, for each slot, whether they managed to download all assigned shards or not. Concretely, this new consensus operation carries a bitset, where the bit at position ``n`` is set to ``1`` if and only if the DAL node connected to the baker managed to download all the shards assigned to that baker for the slot number ``n``. + +Based on the DAL attestations included in a block , the protocol decides which slots are available and announces them in that block's metadata. + +It's important for bakers to note the following: + +- No DAL attestation is injected if there is no slot to attest (i.e. if the bitset is zero); +- If a slot header is included in an L1 block at some level PL (a.k.a ``published_level``), the DAL consensus operations that possibly attest its shards are injected on top of the block(s) at level ``PL + attestation_lag - 1`` and target inclusion at level ``PL + attestation_lag``; +- Theoretically, if a baker is assigned more than ``number_of_shards / redundancy_factor`` shards, it could declare the shards as available if it succeeds in downloading ```number_of_shards / redundancy_factor` shards at least. The missing shards could be calculated by reconstructing the whole slot (but this is not implemented yet); +- It is planned to merge Tenderbake and DAL attestations in the same operation in the future, as they share a few fields. This would avoid an extra signature verification per baker; +- As the number of shards is quite small compared to the number of Tenderbake slots, DAL and Tenderbake committees might not always coincide in practice: there might be bakers in the Tenderbake's committee that are not part of the DAL's for some levels. However, any baker in the DAL committee is part of the Tenderbake committee; +- Like Tenderbake attestations, bakers receive a proportionate number of shards to attest based on their staking balance, directly impacting the required bandwidth to download shards' content at each level. A baker with 1% of the stake will need a bandwidth between 5-10MiB/s. This is why we plan to give the opportunity to bakers to run multiple DAL nodes so that this bandwidth can be spread between those DAL nodes. +- With the DAL, it becomes easier to associate the baker identity with an IP address. Besides, the more stake a baker has, the easier it is. Fortunately, bakers could run their DAL nodes on distinct machines to mitigate possible DDoS attacks on their bakery infrastructure. -- GitLab From e9911258b1dfa3ecbe05a707853a85857b45011a Mon Sep 17 00:00:00 2001 From: Nic Volanschi Date: Wed, 6 Dec 2023 12:26:07 +0100 Subject: [PATCH 04/10] doc: add page for DAL support in protocol --- docs/alpha/dal_support.rst | 84 ++++++++++++++++++++++++++++++++ docs/alpha/protocol.rst | 5 ++ docs/alpha/protocol_overview.rst | 1 + docs/shell/dal.rst | 4 ++ 4 files changed, 94 insertions(+) create mode 100644 docs/alpha/dal_support.rst diff --git a/docs/alpha/dal_support.rst b/docs/alpha/dal_support.rst new file mode 100644 index 000000000000..a4b3b4cf5b74 --- /dev/null +++ b/docs/alpha/dal_support.rst @@ -0,0 +1,84 @@ +=========== +DAL support +=========== + +The support for the :doc:`DAL <../shell/dal>` within the economic protocol relies on two operations: + +#. ``DAL_publish_slot_header``: Allowing anyone to publish a DAL commitment +#. ``DAL_attestation``: Allowing bakers to attest the data seen onto the DAL P2P network + +DAL publish slot header +======================= + +``DAL_publish_slot_header`` is a manager operation that can be issued by a user wishing to publish data onto the DAL. The payload of this operation consists in the following fields: + +- Slot index: Identifies the specific slot for which the data is being published. It is an integer between ``0`` and ``number_of_slots - 1``. +- Commitment: The `KZG commitment `__ over the data. +- Commitment proof: A proof that the commitment commits over data that does not exceed the size ``slot_size``. + +Users can create and manage these commitments and proofs through the :doc:`DAL node ` using these RPCs: + +- To create a commitment: ``POST /commitment`` +- To retrieve a commitment’s proof: ``GET /commitments//proof`` + +Concurrent operations +--------------------- + +If a block contains a valid ``DAL_publish_slot_header`` operation, any subsequent operations of the same kind within the same block (and the same slot index) will be recognized as valid but will fail during execution. They still incur transaction fees. More details about this can be found in the :doc:`validator documentation <../active/validation>`. + +Economics +--------- + +Currently, the fees are estimated based on the execution cost of this operation alone. There are no additional charges related to the bandwidth required for bakers to download data from the DAL for this commitment. However, this might be subject to changes in the future. + +DAL attestation +=============== + +``DAL_attestation`` is a new :ref:`consensus operation ` performed by attesters. It serves to verify whether they were able to successfully download the shards assigned to them. The payload of this operation consists of the following fields: + +- Level: This refers to the level of the block that is being attested, specifically the level preceding the block that contains the ``DAL_attestation`` operation. +- Tenderbake slot: This element identifies the attester, in particular, it determines the public key required for signature verification and the assigned DAL shards. +- Attestation: This is a bitset reflecting the status of each slot. The size of the bitset corresponds to the total number of slots. A value of 1 indicates successful retrieval of all assigned shards by the baker for that slot, while 0 indicates an unsuccessful attempt. + The lowest field of the bitset corresponds to the lowest slot index. + +Attestation timing +------------------ + +When a commitment is published at a certain level, say level ``PL``, the corresponding DAL attestations are expected to be included in the block at level ``PL + attestation_lag``. + +Block metadata +-------------- + +In the block’s metadata, there is a specific field for the DAL. This field reflects the availability of slots based on the DAL attestations received. It is a bitfield with one bit per slot (its format is the same as the attestation part of the ``DAL_attestation`` operation). The bit is set to 1 if the slot is declared available. The lowest slot index corresponds to the lowest bit. To consider a slot as available, there must be a minimum number of shards, as defined by the ``availability_threshold`` parameter, marked as available by the attesters for that slot (e.g. if the number of shards is 2048 and the availability threshold is 50%, then 1024 shards are required). + +Therefore, for data committed (published) at level ``L``, the slot's availability is determined by the metadata of the block at level ``L + attestation_lag``. Consequently, a smart rollup can only utilize this data from level ``L + attestation_lag + 1`` onward. + +Smart rollups integration +========================= + +The DAL is integrated with :doc:`smart rollups <../active/smart_rollups>` so that kernels can request pages from the DAL via the :ref:`reveal data channel `. A smart rollup can fetch any page from the DAL node if the commitment respects some conditions: + +- The commitment should have been published after the rollup origination (this constraint will be leveraged so that the kernel can request any commitment in the past) +- The commitment should not have been published in a level in the future after the level of the next commitment of the state (at most 30 levels in the future). + This constraint might be leveraged thanks to the internal messages ``Start of Level`` and ``End of Level`` (abbreviated as SOL and EOL), which allow the kernel to track the L1 level, so that the kernel cannot request a page for a commitment published after the current L1 level. + +If the kernel requests a page that does not satisfy the mentioned conditions, then the rollup node must answer with an empty page indicating there is no data. Similarly, if the kernel requests for a commitment that was not attested, the rollup node must also provide an empty page. As a consequence, if a kernel requests a page for a commitment for which its availability is still unknown, the rollup node cannot answer and will have to wait for the availability status to be confirmed by the L1. + +Moreover, the rollup kernel has access to the protocol constants so that the same kernel code can be used on different test networks. + +.. _dal_constants_alpha: + +DAL-related protocol constants +============================== + +This section describes the protocol constants specific to the DAL as well as their default values on mainnet (see :ref:`protocol_constants` on how to find the values for tests networks): + +- ``feature_enable`` (false): Whether the DAL is available +- ``number_of_slots`` (256): How many slots are available per block +- ``attestation_lag`` (4): The timeframe for bakers to download shards between the published level of a commitment and the time they must attest the availability of those shards +- ``attestation_threshold`` (50): The percentage of shards attested for a given slot to declare the slot available +- ``blocks_per_epoch`` (1): Unused. Could be removed in the future +- ``page_size`` (4KiB): The size of a page (see :doc:`../shell/dal_slots`) +- ``slot_size`` (1MiB): The size of a slot (see :doc:`../shell/dal_slots`) +- ``redundancy_factor`` (16): The erasure-code factor (see :doc:`../shell/dal_slots`) +- ``number_of_shards`` (2048): The number of shards per slot (see :doc:`../shell/dal_slots`) diff --git a/docs/alpha/protocol.rst b/docs/alpha/protocol.rst index d17b27ac09c6..909c0f84f86c 100644 --- a/docs/alpha/protocol.rst +++ b/docs/alpha/protocol.rst @@ -115,3 +115,8 @@ Sapling, etc), and some details about its implementation. :maxdepth: 2 adaptive_issuance + +.. toctree:: + :maxdepth: 2 + + dal_support diff --git a/docs/alpha/protocol_overview.rst b/docs/alpha/protocol_overview.rst index 09e0a774ad1f..8b6c2d6252f7 100644 --- a/docs/alpha/protocol_overview.rst +++ b/docs/alpha/protocol_overview.rst @@ -139,6 +139,7 @@ See, for example: - :ref:`proof-of-stake parameters ` - :ref:`consensus-related parameters ` - :ref:`randomness generation parameters ` +- :ref:`DAL parameters ` See also ~~~~~~~~ diff --git a/docs/shell/dal.rst b/docs/shell/dal.rst index 108db6b4ee29..41a7491b49d6 100644 --- a/docs/shell/dal.rst +++ b/docs/shell/dal.rst @@ -14,3 +14,7 @@ The following pages provide an overview of the DAL and cover its different aspec dal_p2p dal_node dal_bakers + +See also: + +- :doc:`Protocol support for the DAL <../alpha/dal_support>`. -- GitLab From d3e5c243f9fae95c976fede56e5c71eb8cb7ab45 Mon Sep 17 00:00:00 2001 From: Nic Volanschi Date: Wed, 6 Dec 2023 17:15:15 +0100 Subject: [PATCH 05/10] doc: add DAL node manual --- docs/Makefile | 2 ++ docs/shell/cli-commands.rst | 11 +++++++++++ docs/shell/dal_node.rst | 6 +----- 3 files changed, 14 insertions(+), 5 deletions(-) diff --git a/docs/Makefile b/docs/Makefile index 5497c2f127f9..5b01aa807e59 100644 --- a/docs/Makefile +++ b/docs/Makefile @@ -73,6 +73,8 @@ manuals: \ ./scripts/man2txt.py > api/octez-proxy-server.txt @../octez-smart-rollup-node man -verbosity 3 -format html | \ sed "s#${HOME}#\$$HOME#g" > api/octez-smart-rollup-node.html + @../octez-dal-node --help | grep -v '^Octez-dal-node' | grep -v 'OCTEZ-DAL-NODE(1)' | \ + ./scripts/man2txt.py > api/octez-dal-node.txt .PHONY: odoc odoc: diff --git a/docs/shell/cli-commands.rst b/docs/shell/cli-commands.rst index 67936e60d6e8..996d8e51417d 100644 --- a/docs/shell/cli-commands.rst +++ b/docs/shell/cli-commands.rst @@ -86,6 +86,17 @@ DAC client manual (Experimental) .. raw:: html :file: ../api/octez-dac-client.html +.. _dal_node_manual: + +DAL node manual (Experimental) +============================== + +The command line of the Octez DAL node is documented in Unix ``man`` format, as shown below. You can also obtain this manual by running ``octez-dal-node --help``. + +The manual briefly shows the available DAL node commands. +Each command accepts its own set of options and arguments, that you can discover by running ``octez-dal-node --help``. + +.. include:: ../api/octez-dal-node.txt .. _smart_rollup_node_manual: diff --git a/docs/shell/dal_node.rst b/docs/shell/dal_node.rst index f98e37f08881..25d02df16e78 100644 --- a/docs/shell/dal_node.rst +++ b/docs/shell/dal_node.rst @@ -61,11 +61,7 @@ The command ``init config`` creates a new configuration file in the specified da The command ``run`` runs the DAL node. The CLI arguments take precedence over the configuration file arguments, except for the list of bootstrap peers and of profiles, which are considered in addition to the ones from the configuration file. The configuration file is however not overridden with the new values of the node’s parameters. However, at the end of the execution, the node’s profiles, which may have been given as arguments or set via RPCs, are written to the configuration file. -DAL node manual -~~~~~~~~~~~~~~~ - -The command line of the DAL node is documented in Unix man format, which can be obtained by running ``octez-dal-node --help``. -The arguments of each DAL node command can be listed with ``octez-node --help``. +See the :ref:`DAL node manual ` for more details. DAL configuration of the L1 node ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -- GitLab From b98922c2eaac5dcef0e73afc557eaf4538848b39 Mon Sep 17 00:00:00 2001 From: Nic Volanschi Date: Thu, 7 Dec 2023 15:11:51 +0100 Subject: [PATCH 06/10] doc: add option --RST to script man2txt.py --- docs/Makefile | 6 +++--- docs/scripts/man2txt.py | 27 ++++++++++++++++++++++----- 2 files changed, 25 insertions(+), 8 deletions(-) diff --git a/docs/Makefile b/docs/Makefile index 5b01aa807e59..6830c26b5e45 100644 --- a/docs/Makefile +++ b/docs/Makefile @@ -68,13 +68,13 @@ manuals: \ @ ../octez-dac-node man -verbosity 3 -format html | sed "s#${HOME}#\$$HOME#g" > api/octez-dac-node.html @ ../octez-dac-client man -verbosity 3 -format html | sed "s#${HOME}#\$$HOME#g" > api/octez-dac-client.html @../octez-node --help | grep -v '^Octez-node' | grep -v 'OCTEZ-NODE(1)' | \ - ./scripts/man2txt.py > api/octez-node.txt + ./scripts/man2txt.py -r > api/octez-node.txt @../octez-proxy-server --help | grep -v '^Octez-proxy-server' | grep -v 'OCTEZ-PROXY-SERVER(1)' | \ - ./scripts/man2txt.py > api/octez-proxy-server.txt + ./scripts/man2txt.py -r > api/octez-proxy-server.txt @../octez-smart-rollup-node man -verbosity 3 -format html | \ sed "s#${HOME}#\$$HOME#g" > api/octez-smart-rollup-node.html @../octez-dal-node --help | grep -v '^Octez-dal-node' | grep -v 'OCTEZ-DAL-NODE(1)' | \ - ./scripts/man2txt.py > api/octez-dal-node.txt + ./scripts/man2txt.py -r > api/octez-dal-node.txt .PHONY: odoc odoc: diff --git a/docs/scripts/man2txt.py b/docs/scripts/man2txt.py index 23b3668557d6..72ed4817dc81 100755 --- a/docs/scripts/man2txt.py +++ b/docs/scripts/man2txt.py @@ -1,15 +1,32 @@ #!/usr/bin/env python3 # Convert man pages to simple text by removing formatting directives - -# Currently, only bold characters are replaced: x^Hx -> x +# (e.g., bold characters are replaced: x^Hx -> x) +# With option -r, further make output compatible with RST syntax import re import fileinput +import argparse + +parser = argparse.ArgumentParser() +parser.add_argument( + "-r", + "--RST", + action="store_true", + help="make output compatible with ReS", +) +parser.add_argument( + 'files', + metavar='FILE', + nargs='*', + help='files to read, if empty, stdin is used', +) +args = parser.parse_args() -for line in fileinput.input(): +for line in fileinput.input(files=args.files): line = line.rstrip('\n') # get rid of the overwritten letters line = re.sub(r'.[\b]', '', line) - # protect double dashes from being interpreted as a long dash - line = re.sub(r'--', r'-\-', line) + if args.RST: + # protect double dashes from being interpreted as a long dash + line = re.sub(r'--', r'-\-', line) print(line) -- GitLab From b721ef2c2d8e917aa3606e9a03a3cd294ad4c86a Mon Sep 17 00:00:00 2001 From: Nic Volanschi Date: Thu, 7 Dec 2023 15:14:53 +0100 Subject: [PATCH 07/10] doc: include generated options for dal_node commands --- docs/.gitignore | 3 +-- docs/Makefile | 2 ++ docs/shell/dal_node.rst | 8 +++++++- 3 files changed, 10 insertions(+), 3 deletions(-) diff --git a/docs/.gitignore b/docs/.gitignore index c8661cd2e7dd..3fe782fc0526 100644 --- a/docs/.gitignore +++ b/docs/.gitignore @@ -10,8 +10,7 @@ octez-dac-node.html /api/octez-codec.html /api/octez-admin-client.html /api/octez-signer.html -/api/octez-node.txt -/api/octez-proxy-server.txt +/api/octez-*.txt /alpha/rpc.rst /shell/rpc.rst diff --git a/docs/Makefile b/docs/Makefile index 6830c26b5e45..022ca4ad515c 100644 --- a/docs/Makefile +++ b/docs/Makefile @@ -75,6 +75,8 @@ manuals: \ sed "s#${HOME}#\$$HOME#g" > api/octez-smart-rollup-node.html @../octez-dal-node --help | grep -v '^Octez-dal-node' | grep -v 'OCTEZ-DAL-NODE(1)' | \ ./scripts/man2txt.py -r > api/octez-dal-node.txt + @../octez-dal-node config init --help | grep -v '^Octez-dal-node' | grep -v 'OCTEZ-DAL-NODE(1)' | \ + ./scripts/man2txt.py > api/octez-dal-node-config-init.txt .PHONY: odoc odoc: diff --git a/docs/shell/dal_node.rst b/docs/shell/dal_node.rst index 25d02df16e78..9ce81717bf21 100644 --- a/docs/shell/dal_node.rst +++ b/docs/shell/dal_node.rst @@ -47,7 +47,7 @@ The DAL node tracks the L1 chain, in particular it monitors the heads of the L1 RPC server ^^^^^^^^^^ -The DAL node incorporates an RPC server which answers to RPC queries to update or retrieve information about the node’s state. For instance, one can post slots, ask the node to compute and store shards, to update profiles, or to connect or disconnect to peers. For the full list of the DAL node’s RPC see :doc:`./dal_rpc`. +The DAL node incorporates an RPC server which answers to RPC queries to update or retrieve information about the node’s state. For instance, one can post slots, ask the node to compute and store shards, to update profiles, or to connect or disconnect to peers. Operational aspects ------------------- @@ -61,6 +61,12 @@ The command ``init config`` creates a new configuration file in the specified da The command ``run`` runs the DAL node. The CLI arguments take precedence over the configuration file arguments, except for the list of bootstrap peers and of profiles, which are considered in addition to the ones from the configuration file. The configuration file is however not overridden with the new values of the node’s parameters. However, at the end of the execution, the node’s profiles, which may have been given as arguments or set via RPCs, are written to the configuration file. +Both commands have the same arguments, which can be seen by executing, e.g., ``octez-dal-node config init --help``: + +.. literalinclude:: ../api/octez-dal-node-config-init.txt + :start-at: OPTIONS + :end-before: COMMON OPTIONS + See the :ref:`DAL node manual ` for more details. DAL configuration of the L1 node -- GitLab From 21898749daf97e1525e99f28d55e438b341779a0 Mon Sep 17 00:00:00 2001 From: Nic Volanschi Date: Thu, 7 Dec 2023 16:20:51 +0100 Subject: [PATCH 08/10] doc: refactor DAL P2P & Slots into DAL overview --- docs/alpha/dal_support.rst | 8 +-- docs/shell/dal.rst | 2 - docs/shell/dal_bakers.rst | 2 +- docs/shell/dal_node.rst | 4 +- docs/shell/dal_overview.rst | 99 +++++++++++++++++++++++++++++++++++++ docs/shell/dal_p2p.rst | 34 ------------- docs/shell/dal_slots.rst | 57 --------------------- 7 files changed, 106 insertions(+), 100 deletions(-) delete mode 100644 docs/shell/dal_p2p.rst delete mode 100644 docs/shell/dal_slots.rst diff --git a/docs/alpha/dal_support.rst b/docs/alpha/dal_support.rst index a4b3b4cf5b74..afcc613f527b 100644 --- a/docs/alpha/dal_support.rst +++ b/docs/alpha/dal_support.rst @@ -78,7 +78,7 @@ This section describes the protocol constants specific to the DAL as well as the - ``attestation_lag`` (4): The timeframe for bakers to download shards between the published level of a commitment and the time they must attest the availability of those shards - ``attestation_threshold`` (50): The percentage of shards attested for a given slot to declare the slot available - ``blocks_per_epoch`` (1): Unused. Could be removed in the future -- ``page_size`` (4KiB): The size of a page (see :doc:`../shell/dal_slots`) -- ``slot_size`` (1MiB): The size of a slot (see :doc:`../shell/dal_slots`) -- ``redundancy_factor`` (16): The erasure-code factor (see :doc:`../shell/dal_slots`) -- ``number_of_shards`` (2048): The number of shards per slot (see :doc:`../shell/dal_slots`) +- ``page_size`` (4KiB): The size of a page (see :ref:`dal_slots`) +- ``slot_size`` (1MiB): The size of a slot (see :ref:`dal_slots`) +- ``redundancy_factor`` (16): The erasure-code factor (see :ref:`dal_slots`) +- ``number_of_shards`` (2048): The number of shards per slot (see :ref:`dal_slots`) diff --git a/docs/shell/dal.rst b/docs/shell/dal.rst index 41a7491b49d6..0bce2f602968 100644 --- a/docs/shell/dal.rst +++ b/docs/shell/dal.rst @@ -10,8 +10,6 @@ The following pages provide an overview of the DAL and cover its different aspec :maxdepth: 2 dal_overview - dal_slots - dal_p2p dal_node dal_bakers diff --git a/docs/shell/dal_bakers.rst b/docs/shell/dal_bakers.rst index a9b3b2f1baaa..ce9021f9ac81 100644 --- a/docs/shell/dal_bakers.rst +++ b/docs/shell/dal_bakers.rst @@ -2,7 +2,7 @@ Bakers & the DAL ================ This page documents some aspects that are most useful to know for bakers when participating to the DAL network (baking the DAL is a serious matter!). -We assume the reader is familiar with the DAL basics and terminology introduced in :doc:`./dal_overview` and :doc:`./dal_slots`. +We assume the reader is familiar with the DAL basics and terminology introduced in :doc:`./dal_overview` and :ref:`dal_slots`. Bakers play a crucial role in Tezos through the Tenderbake protocol, by validating transactions and securing the network. This participation involves producing new blocks when assigned the rights by the protocol, attesting proposed valid blocks to make the chain progress, and (indirectly) slashing misbehavior that would put the chain at risk. Due to their strategic role in Tezos, entrusting them with verifying and attesting data availability using the DAL is natural. diff --git a/docs/shell/dal_node.rst b/docs/shell/dal_node.rst index 9ce81717bf21..27ea07394312 100644 --- a/docs/shell/dal_node.rst +++ b/docs/shell/dal_node.rst @@ -9,7 +9,7 @@ Concepts and features DAL P2P network ^^^^^^^^^^^^^^^ -The ``octez-dal-node`` executable runs a node in the DAL’s P2P network. Recall that :doc:`the DAL's P2P protocol <./dal_p2p>` is based on a gossip algorithm for distributing shards, running on top of a networking layer using the same p2p library as L1 nodes. +The ``octez-dal-node`` executable runs a node in the DAL’s P2P network. Recall that :ref:`the DAL's P2P protocol ` is based on a gossip algorithm for distributing shards, running on top of a networking layer using the same p2p library as L1 nodes. Actors with various roles may need to run the DAL node, for instance bootstrap DAL node operators, slot producers, bakers. @@ -109,7 +109,7 @@ The life cycle of slots and shards is described by the following steps: octez-client publish dal commitment from for slot with proof -#. Once the operation is included in a final block (that is, there is another block on top of the one including the operation), and the slot is considered published (see :doc:`./dal_overview`), all DAL nodes exchange the slot’s shards they have in their store on the P2P network, depending on their profile (see :doc:`./dal_p2p`), and they store previously unknown shards. +#. Once the operation is included in a final block (that is, there is another block on top of the one including the operation), and the slot is considered published (see :doc:`./dal_overview`), all DAL nodes exchange the slot’s shards they have in their store on the P2P network, depending on their profile (see :ref:`dal_p2p`), and they store previously unknown shards. #. Attesters check, for all published slots, the availability of the shards they are assigned by interrogating their DAL node, via the RPC ``GET /profiles//attested_levels//attestable_slots``, where level is the level at which the slot was published plus ``attestation_lag``, and ``pkh`` is the attester’s public key hash. (See also :doc:`dal_bakers`) #. Attesters inject via their baker binary a DAL attestation operation containing the information received at step 6. (See also :doc:`dal_bakers`) #. The protocol aggregates the received DAL attestations, and declares each published slot as available or unavailable, depending on whether some threshold is reached, via the blocks metadata. diff --git a/docs/shell/dal_overview.rst b/docs/shell/dal_overview.rst index 67396ebdaf48..5caef108f9d6 100644 --- a/docs/shell/dal_overview.rst +++ b/docs/shell/dal_overview.rst @@ -1,3 +1,4 @@ +============ DAL overview ============ @@ -65,3 +66,101 @@ The rationale for having the attestation lag parameter is to give bakers sufficient time to download their assigned shards and to guarantee that the latency stays within acceptable limits (around one minute). + + +.. _dal_slots: + +DAL slots +========= + +There is a limited number of blobs, called *DAL slots* (or slots for +short), that can be published at a L1 level. A slot is identified by +the L1 level at which it is published, an index, and its +commitment. Depending on the context, a “slot” may refer to the data +blob or to the placeholder (the slot index) of that blob. Each slot is +treated independently of other slots. Presently, all slots are of the +same size, though this may change with future updates. To publish a +slot, it is first expanded to create redundancy through erasure +coding, increasing its size by a factor known as the *redundancy +factor*. This expanded slot is then divided into a set of smaller data +pieces of equal size, called *shards*. Finally, shards are distributed +across the DAL P2P network. + +The advantage of using erasure coding is that one only needs a subset +of the shards, specifically, “number of shards / redundancy factor” to +recover the full original slot data. A key feature for the shards is +that each shard can be verified to ensure it corresponds to a +particular commitment, which safeguards against spam in the DAL P2P +network. + +For each slot index, multiple commitments can be included in a block, +but only one will be acknowledged, the first one appearing in the +block ordering. Because bakers typically prioritize operations that +offer higher fees, the commitment with the higher fee is usually +chosen for inclusion, although this outcome is not guaranteed if a +baker uses a different selection criterion. + +For smart rollups, it is necessary to divide the slot into smaller +segments called *pages* (see +:ref:`populating_the_reveal_channel`). The economic protocol specifies +the size of each page as a constant, with 4KiB being a practical +size. While it might seem feasible to choose shards fitting this page +limit and directly feed shards to the rollup, accessing the original +data from shards involves complex cryptographic computations. We +prefer to avoid performing such computations within the smart +rollups. Therefore, even though the DAL network uses shards +internally, the DAL node can serve individual pages as requested by +the smart-rollup node. These pages can then be imported individually +by the smart rollup node to the kernel on demand. To reconstruct the +entire slot, one simply needs to arrange these pages in the correct +sequence. For developers working with smart-rollup kernels, the +technicalities of reconstructing the entire slot out of pages should +ideally be handled by the SDK they use, simplifying the process even +more. + +.. |DAL slot| image:: dal_slot.png + +|DAL slot| + + +When the slot producer (user publishing a slot) posts the commitment +onto the L1, it also posts a proof that the slot does not exceed the +slot size. This prevents malicious users from producing and posting +data larger than the expected size. + +.. _dal_p2p: + +The DAL's P2P protocol +====================== + +The peer-to-peer (P2P) protocol for the DAL is made out of two components: + +- A gossipsub algorithm, instantiated as detailed below. +- A transport layer for handling connections with peers. + We reuse for that the P2P protocol used by the Octez node (see :ref:`dal_p2p`). + +The gossip algorithm used for the DAL is an in-house version of the gossipsub v1.1 P2P protocol defined by the lib-p2p project. A detailed overview of this protocol is available `here `__ and an informal English specification can be found `here `__. This gossip algorithm allows to partition the network into virtual subnetworks, each identified by a **topic**. The topic also determines the valid data that can be exchanged over the corresponding virtual subnetwork, as any exchanged message has exactly one associated topic. Each peer subscribes to topics of interest to him. This protocol enhances the network's scalability compared to traditional gossip algorithms. + +For each message, there is a **message id** that uniquely identifies this message. When a message is pushed, it comes with its message id. When the message is pulled, it is done via the message id. +For every topic a node subscribes to, it maintains a virtual subnetwork, or **mesh**, of peers also subscribed to that topic. When a node has a new message to share (originating from the application layer) or needs to relay a received message, it does so to all peers in the corresponding topic's mesh. Moreover, the node broadcasts the ids of the last received messages to a random selection of peers outside the mesh. Peers receiving these teasers can request the full message if they are interested in it. +For the DAL instantiation of gossipsub, a message is defined as a 3-tuple: a shard, the shard’s index, and the shard’s proof proving that the shard corresponds to the commitment given by the message id. The associated message id consists of the shard index and the associated slot index, (published) level, slot commitment, and attestor’s public key hash. + +A topic is defined as a pair ``(slot_index, public_key_hash)``. The first component identifies the slot associated to any shard published under this topic, while the second component identifies the baker assigned to this shard. +Such a set of topics ensures that the bandwidth of bakers and slot producers is bounded (for valid messages) over a cycle. + +A slot producer should subscribe to all relevant topics associated with their slot index. This includes every topic where a baker is assigned at least one shard for that slot index. +On the other hand, a baker should subscribe to all topics that feature their public key address. + +Gossipsub also defines a notion of score which is used to only connect to peers with a good score. + +Regarding peer discovery, the current implementation of the DAL relies on gossipsub v1.1 peer exchanges. In particular, DAL nodes can be configured in bootstrap mode to facilitate peer discovery. + +.. note:: + + The current topic structure in the DAL for Tezos may be revised in a future update. Presently, topics include the bakers’ address (public key hash), which leads to a potentially unbounded number of topics over time. Another approach under consideration involves using a ``(slot_index, shard_index)`` pair, offering a more scalable solution in the long run, when the number of attesters surpasses the number of slots. + +.. warning:: + + Attention must be paid to the security implications for bakers in the DAL network. Since a baker's bandwidth is proportional to their stake, it can become relatively straightforward to identify the IP address of their DAL node, particularly for those with substantial stakes. To mitigate this risk, bakers are advised to operate their DAL node using an IP address different from their L1 node. This separation helps in preventing the unintentional exposure of the L1 node's IP address. + + Plans are underway to address these concerns. One proposed solution is to enable bakers to divide their bandwidth across multiple DAL nodes, enhancing both security and operational flexibility. diff --git a/docs/shell/dal_p2p.rst b/docs/shell/dal_p2p.rst deleted file mode 100644 index 39eb5fbe195a..000000000000 --- a/docs/shell/dal_p2p.rst +++ /dev/null @@ -1,34 +0,0 @@ -The DAL's P2P protocol -====================== - -The peer-to-peer (P2P) protocol for the DAL is made out of two components: - -- A gossipsub algorithm, instantiated as detailed below. -- A transport layer for handling connections with peers. - We reuse for that the P2P protocol used by the Octez node (see :doc:`./p2p`). - -The gossip algorithm used for the DAL is an in-house version of the gossipsub v1.1 P2P protocol defined by the lib-p2p project. A detailed overview of this protocol is available `here `__ and an informal English specification can be found `here `__. This gossip algorithm allows to partition the network into virtual subnetworks, each identified by a **topic**. The topic also determines the valid data that can be exchanged over the corresponding virtual subnetwork, as any exchanged message has exactly one associated topic. Each peer subscribes to topics of interest to him. This protocol enhances the network's scalability compared to traditional gossip algorithms. - -For each message, there is a **message id** that uniquely identifies this message. When a message is pushed, it comes with its message id. When the message is pulled, it is done via the message id. -For every topic a node subscribes to, it maintains a virtual subnetwork, or **mesh**, of peers also subscribed to that topic. When a node has a new message to share (originating from the application layer) or needs to relay a received message, it does so to all peers in the corresponding topic's mesh. Moreover, the node broadcasts the ids of the last received messages to a random selection of peers outside the mesh. Peers receiving these teasers can request the full message if they are interested in it. -For the DAL instantiation of gossipsub, a message is defined as a 3-tuple: a shard, the shard’s index, and the shard’s proof proving that the shard corresponds to the commitment given by the message id. The associated message id consists of the shard index and the associated slot index, (published) level, slot commitment, and attestor’s public key hash. - -A topic is defined as a pair ``(slot_index, public_key_hash)``. The first component identifies the slot associated to any shard published under this topic, while the second component identifies the baker assigned to this shard. -Such a set of topics ensures that the bandwidth of bakers and slot producers is bounded (for valid messages) over a cycle. - -A slot producer should subscribe to all relevant topics associated with their slot index. This includes every topic where a baker is assigned at least one shard for that slot index. -On the other hand, a baker should subscribe to all topics that feature their public key address. - -Gossipsub also defines a notion of score which is used to only connect to peers with a good score. - -Regarding peer discovery, the current implementation of the DAL relies on gossipsub v1.1 peer exchanges. In particular, DAL nodes can be configured in bootstrap mode to facilitate peer discovery. - -.. note:: - - The current topic structure in the DAL for Tezos may be revised in a future update. Presently, topics include the bakers’ address (public key hash), which leads to a potentially unbounded number of topics over time. Another approach under consideration involves using a ``(slot_index, shard_index)`` pair, offering a more scalable solution in the long run, when the number of attesters surpasses the number of slots. - -.. warning:: - - Attention must be paid to the security implications for bakers in the DAL network. Since a baker's bandwidth is proportional to their stake, it can become relatively straightforward to identify the IP address of their DAL node, particularly for those with substantial stakes. To mitigate this risk, bakers are advised to operate their DAL node using an IP address different from their L1 node. This separation helps in preventing the unintentional exposure of the L1 node's IP address. - - Plans are underway to address these concerns. One proposed solution is to enable bakers to divide their bandwidth across multiple DAL nodes, enhancing both security and operational flexibility. diff --git a/docs/shell/dal_slots.rst b/docs/shell/dal_slots.rst deleted file mode 100644 index 7806f655fd84..000000000000 --- a/docs/shell/dal_slots.rst +++ /dev/null @@ -1,57 +0,0 @@ -DAL Slots -========= - -There is a limited number of blobs, called *DAL slots* (or slots for -short), that can be published at a L1 level. A slot is identified by -the L1 level at which it is published, an index, and its -commitment. Depending on the context, a “slot” may refer to the data -blob or to the placeholder (the slot index) of that blob. Each slot is -treated independently of other slots. Presently, all slots are of the -same size, though this may change with future updates. To publish a -slot, it is first expanded to create redundancy through erasure -coding, increasing its size by a factor known as the *redundancy -factor*. This expanded slot is then divided into a set of smaller data -pieces of equal size, called *shards*. Finally, shards are distributed -across the DAL P2P network. - -The advantage of using erasure coding is that one only needs a subset -of the shards, specifically, “number of shards / redundancy factor” to -recover the full original slot data. A key feature for the shards is -that each shard can be verified to ensure it corresponds to a -particular commitment, which safeguards against spam in the DAL P2P -network. - -For each slot index, multiple commitments can be included in a block, -but only one will be acknowledged, the first one appearing in the -block ordering. Because bakers typically prioritize operations that -offer higher fees, the commitment with the higher fee is usually -chosen for inclusion, although this outcome is not guaranteed if a -baker uses a different selection criterion. - -For smart rollups, it is necessary to divide the slot into smaller -segments called *pages* (see -:ref:`populating_the_reveal_channel`). The economic protocol specifies -the size of each page as a constant, with 4KiB being a practical -size. While it might seem feasible to choose shards fitting this page -limit and directly feed shards to the rollup, accessing the original -data from shards involves complex cryptographic computations. We -prefer to avoid performing such computations within the smart -rollups. Therefore, even though the DAL network uses shards -internally, the DAL node can serve individual pages as requested by -the smart-rollup node. These pages can then be imported individually -by the smart rollup node to the kernel on demand. To reconstruct the -entire slot, one simply needs to arrange these pages in the correct -sequence. For developers working with smart-rollup kernels, the -technicalities of reconstructing the entire slot out of pages should -ideally be handled by the SDK they use, simplifying the process even -more. - -.. |DAL slot| image:: dal_slot.png - -|DAL slot| - - -When the slot producer (user publishing a slot) posts the commitment -onto the L1, it also posts a proof that the slot does not exceed the -slot size. This prevents malicious users from producing and posting -data larger than the expected size. -- GitLab From db3cf0ef0d0e3df22444b139615019a863594da6 Mon Sep 17 00:00:00 2001 From: Eugen Zalinescu Date: Thu, 7 Dec 2023 08:29:19 +0000 Subject: [PATCH 09/10] doc: final touches in dal node page --- docs/shell/dal_node.rst | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/docs/shell/dal_node.rst b/docs/shell/dal_node.rst index 27ea07394312..41fc0f5d8f2b 100644 --- a/docs/shell/dal_node.rst +++ b/docs/shell/dal_node.rst @@ -36,7 +36,7 @@ The size of the node’s storage depends on its profile. A bootstrap node uses n L1 monitoring ^^^^^^^^^^^^^ -The DAL node tracks the L1 chain, in particular it monitors the heads of the L1 chain via an RPC. At each final block, it retrieves and stores the newly published commitments, and if its storage contains shards for these commitments, it publishes them on the P2P network. The DAL node also retrieves the slot attestation status from the block’s metadata. Finally, based on this information it updates the status of the stored commitments, where the status of a commitment can be one of the following: +The DAL node tracks the L1 chain, in particular it monitors the heads of the L1 chain via an RPC. Each time a block becomes final, it retrieves and stores the commitments published in that block, and if its storage contains shards for these commitments, it publishes them on the P2P network. The DAL node also retrieves the slot attestation status from the block’s metadata. Finally, based on this information it updates the status of the stored commitments, where the status of a commitment can be one of the following: - *waiting for attestation*: The commitment was included and applied in a finalized L1 block but the corresponding slot remains to be attested. - *attested*: The commitment was included in an L1 block and the corresponding slot is attested. @@ -47,7 +47,12 @@ The DAL node tracks the L1 chain, in particular it monitors the heads of the L1 RPC server ^^^^^^^^^^ -The DAL node incorporates an RPC server which answers to RPC queries to update or retrieve information about the node’s state. For instance, one can post slots, ask the node to compute and store shards, to update profiles, or to connect or disconnect to peers. +The DAL node incorporates an RPC server which answers to RPC queries to update or retrieve information about the node’s state. +For instance, one can post slots, ask the node to compute and store shards, to update profiles, or to connect to or disconnect from peers. + +The default listening port for RPCs is 10732, but can be changed using option ``--rpc-addr``. +The RPC server is started by default, even if this option is not given. + Operational aspects ------------------- -- GitLab From 6736b8e33b54de5a3ab1071b2a9db291fb930a28 Mon Sep 17 00:00:00 2001 From: Nic Volanschi Date: Thu, 7 Dec 2023 17:14:27 +0100 Subject: [PATCH 10/10] doc: final touches on dal baker page --- docs/alpha/dal_support.rst | 2 +- docs/shell/dal.rst | 2 +- docs/shell/dal_bakers.rst | 14 +++++++------- 3 files changed, 9 insertions(+), 9 deletions(-) diff --git a/docs/alpha/dal_support.rst b/docs/alpha/dal_support.rst index afcc613f527b..abb805f8ad1b 100644 --- a/docs/alpha/dal_support.rst +++ b/docs/alpha/dal_support.rst @@ -16,7 +16,7 @@ DAL publish slot header - Commitment: The `KZG commitment `__ over the data. - Commitment proof: A proof that the commitment commits over data that does not exceed the size ``slot_size``. -Users can create and manage these commitments and proofs through the :doc:`DAL node ` using these RPCs: +Users can create and manage these commitments and proofs through the :doc:`DAL node <../shell/dal_node>` using these RPCs: - To create a commitment: ``POST /commitment`` - To retrieve a commitment’s proof: ``GET /commitments//proof`` diff --git a/docs/shell/dal.rst b/docs/shell/dal.rst index 0bce2f602968..e0152ef1b474 100644 --- a/docs/shell/dal.rst +++ b/docs/shell/dal.rst @@ -15,4 +15,4 @@ The following pages provide an overview of the DAL and cover its different aspec See also: -- :doc:`Protocol support for the DAL <../alpha/dal_support>`. +- :doc:`Protocol support for the DAL <../alpha/dal_support>` diff --git a/docs/shell/dal_bakers.rst b/docs/shell/dal_bakers.rst index ce9021f9ac81..d98ff4b512a6 100644 --- a/docs/shell/dal_bakers.rst +++ b/docs/shell/dal_bakers.rst @@ -2,7 +2,7 @@ Bakers & the DAL ================ This page documents some aspects that are most useful to know for bakers when participating to the DAL network (baking the DAL is a serious matter!). -We assume the reader is familiar with the DAL basics and terminology introduced in :doc:`./dal_overview` and :ref:`dal_slots`. +We assume the reader is familiar with the DAL basics and terminology introduced in :doc:`./dal_overview` and :doc:`../alpha/dal_support`. Bakers play a crucial role in Tezos through the Tenderbake protocol, by validating transactions and securing the network. This participation involves producing new blocks when assigned the rights by the protocol, attesting proposed valid blocks to make the chain progress, and (indirectly) slashing misbehavior that would put the chain at risk. Due to their strategic role in Tezos, entrusting them with verifying and attesting data availability using the DAL is natural. @@ -10,7 +10,7 @@ From a baker's point of view, DAL attestation is carried out in three main steps First, the baker is attributed DAL attestation rights (automatically, based on its consensus attestation rights). Similar to Tenderbake's baking and attestation rights, the L1 protocol assigns a range of DAL shards to bakers. For a given level, the list of shards attributed to a baker is the same for every DAL slot. -Second, Tezos bakers should launch DAL nodes and join the DAL P2P network subscribing to the appropriate topics. The topics of interest for a baker are those of the form ``(tz1TheBakerSPublicKeyHash, slot_index)`` where the first component is the baker's public key hash and the slot index ranges from ``0`` to ``number of DAL slots - 1``. Note that the baker binary automatically subscribes to those topics when launched with ``--dal-node ``. Once launched and connected to the baker's binary, the DAL node is responsible for downloading the shards assigned to the corresponding public key hash(es) for every slot via the DAL P2P network, assuming the slot commitment operation has been accepted by L1 and the slot's shards are published on the DAL/P2P (see steps 3 and 4 in :doc:`dal_overview`). +Second, Tezos bakers should launch :doc:`DAL nodes <./dal_node>` and join the :ref:`DAL P2P network ` subscribing to the appropriate topics. The topics of interest for a baker are those of the form ``(tz1TheBakerSPublicKeyHash, slot_index)`` where the first component is the baker's public key hash and the slot index ranges from ``0`` to ``number of DAL slots - 1``. Note that the baker binary automatically subscribes to those topics when launched with ``--dal-node ``. Once launched and connected to the baker's binary, the DAL node is responsible for downloading the shards assigned to the corresponding public key hash(es) for every slot via the DAL P2P network, assuming the slot commitment operation has been accepted by L1 and the slot's shards are published on the DAL network (see steps 3 and 4 in :doc:`dal_overview`). Finally, just like for Tenderbake pre-attestations or attestations, bakers inject DAL attestations after every L1 block to inform the protocol, for each slot, whether they managed to download all assigned shards or not. Concretely, this new consensus operation carries a bitset, where the bit at position ``n`` is set to ``1`` if and only if the DAL node connected to the baker managed to download all the shards assigned to that baker for the slot number ``n``. @@ -18,10 +18,10 @@ Based on the DAL attestations included in a block , the protocol decides which s It's important for bakers to note the following: -- No DAL attestation is injected if there is no slot to attest (i.e. if the bitset is zero); -- If a slot header is included in an L1 block at some level PL (a.k.a ``published_level``), the DAL consensus operations that possibly attest its shards are injected on top of the block(s) at level ``PL + attestation_lag - 1`` and target inclusion at level ``PL + attestation_lag``; -- Theoretically, if a baker is assigned more than ``number_of_shards / redundancy_factor`` shards, it could declare the shards as available if it succeeds in downloading ```number_of_shards / redundancy_factor` shards at least. The missing shards could be calculated by reconstructing the whole slot (but this is not implemented yet); -- It is planned to merge Tenderbake and DAL attestations in the same operation in the future, as they share a few fields. This would avoid an extra signature verification per baker; -- As the number of shards is quite small compared to the number of Tenderbake slots, DAL and Tenderbake committees might not always coincide in practice: there might be bakers in the Tenderbake's committee that are not part of the DAL's for some levels. However, any baker in the DAL committee is part of the Tenderbake committee; +- No DAL attestation is injected if there is no slot to attest (i.e. if the bitset is zero). +- If a slot header is included in an L1 block at some level PL (a.k.a ``published_level``), the DAL consensus operations that possibly attest its shards are injected on top of the block(s) at level ``PL + attestation_lag - 1`` and target inclusion at level ``PL + attestation_lag``. +- Theoretically, if a baker is assigned more than ``number_of_shards / redundancy_factor`` shards, it could declare the shards as available if it succeeds in downloading ``number_of_shards / redundancy_factor`` shards at least. The missing shards could be calculated by reconstructing the whole slot (but this is not implemented yet). +- It is planned to merge Tenderbake and DAL attestations in the same operation in the future, as they share a few fields. This would avoid an extra signature verification per baker. +- As the number of shards is quite small compared to the number of Tenderbake slots, DAL and Tenderbake committees might not always coincide in practice: there might be bakers in the Tenderbake's committee that are not part of the DAL's for some levels. However, any baker in the DAL committee is part of the Tenderbake committee. - Like Tenderbake attestations, bakers receive a proportionate number of shards to attest based on their staking balance, directly impacting the required bandwidth to download shards' content at each level. A baker with 1% of the stake will need a bandwidth between 5-10MiB/s. This is why we plan to give the opportunity to bakers to run multiple DAL nodes so that this bandwidth can be spread between those DAL nodes. - With the DAL, it becomes easier to associate the baker identity with an IP address. Besides, the more stake a baker has, the easier it is. Fortunately, bakers could run their DAL nodes on distinct machines to mitigate possible DDoS attacks on their bakery infrastructure. -- GitLab