Introduction to Waku Simulator

The waku-simulator tool allows simulating a waku network with a set of interconnected nwaku nodes with the following features:

• Configurable amount of nodes. Limits depend on the machine and are upper bounded at around 200.
• Runs in a single machine, using docker-compose to orchestrate the containers.
• It uses discv5 for peer discovery, using a common bootstrap node.
• It runs a custom ad hoc network, isolated from the existing waku networks.
• It uses a freshly deployed private blockchain, with full control over it and minimum state to track.
• It deploys an RLN contract in the said private blockchain and configures it to be used by all nodes.
• It registers an RLN membership for each node in the network, configuring it in the node to publish valid messages.
• It exposes each node’s API, so that it can be used to inject traffic into the network.
• Simple to run. Everything is automated. Requires two commands to run.

The main goals of waku-simulator include but are not limited to:

• Test new features in an end to end setup with multiple nodes.
• Use as a long-lived running network on latest master, to anticipate breaking changes.
• Explore waku's limits by using different loads and configurations.
• Offer a tool to debug problems in a controlled and easy to replicate environment.

Deploy a waku network

The network can be deployed in a few commands, and requires docker and docker-compose. Some of the configuration is exposed via env flags, but if you are missing some, PRs are accepted.

Some of the most important parameters are:

• NWAKU_IMAGE Docker image of nwaku that all nodes will run
• NUM_NWAKU_NODES Amount of nwaku nodes
• RLN_RELAY_EPOCH_SEC and RLN_RELAY_MSG_LIMIT configure the RLNv2 parameter, specifying the amount of messages that are allowed per unit of time.
• TRAFFIC_DELAY_SECONDS and MSG_SIZE_KBYTES are used to inject traffic via the rest API into the network.

export NWAKU_IMAGE=quay.io/wakuorg/nwaku-pr:2759-rln-v2
export NUM_NWAKU_NODES=5
export RLN_RELAY_EPOCH_SEC=1
export RLN_RELAY_MSG_LIMIT=1

export TRAFFIC_DELAY_SECONDS=15
export MSG_SIZE_KBYTES=10

export PRIVATE_KEY=0xac0974bec39a17e36ba4a6b4d238ff944bacb478cbed5efcae784d7bf4f2ff80
export ETH_FROM=0xf39fd6e51aad88f6f4ce6ab8827279cfffb92266

Once configured, start all containers:

docker-compose --compatibility up -d

After a couple of minutes, everything should be running at:

• http://localhost:3000 Block explorer
• http://localhost:3001 Grafana metrics

For greater observability, one can access each node logs as follows:

docker logs waku-simulator-nwaku-1
docker logs waku-simulator-nwaku-2

Or if you want to follow the logs

docker logs waku-simulator-nwaku-1 --follow

Once the network of nwaku nodes is up and running we can use it to perform different tests, connecting other nodes that we fully control with some specific characteristics. This ranges from connecting spammer nodes, light clients, and in the future unsynced nodes, etc.

Now that we have the network deployed we can use it. Hereunder we describe how to use the network deployed by waku-simulator to perform end-to-end tests of any desired feature. We focus on the following ones:

• Inject traffic:
• Connect external full node:
• Connect external spammer node:
• Connect external light node:
• Register memberships:

⚠️ For every use case, ensure that your node is configured in the same way as the rest of the nodes, otherwise messages may be lost. Note that it can be also an intended test, seeing how the network reacts to other nodes connecting to it.

Inject traffic

In order to inject traffic into the network, we can use the REST API of each nwaku node. We have a simple dockerized script in rest-traffic, that can perform this task. In the following command we run a docker container, connected to the waku-simulator network. This script will inject a message every delay-seconds with a size of msg-size-kbytes into a given pubsub-topic. Note that in multiple-nodes you can configure the nodes that will publish messages, where [1..5] will publish to node 1, 2, 3, 4, 5. You can publish to a single node (e.g. node 1) by using [1..1].

docker run -it --network waku-simulator_simulation alrevuelta/rest-traffic:d936446 \
--delay-seconds=10 \
--msg-size-kbytes=5 \
--pubsub-topic=/waku/2/rs/66/0 \
--multiple-nodes="http://waku-simulator-nwaku-[1..5]:8645"

Note that the REST API doesn’t allow to publish messages exceeding the rate limit, so this tool can’t be used to test beyond the rate limits.

🎯Goals:

• Test message publishing via the REST API and that they are valid across the network.
• Test the network under different message load: rates and sizes.

👀Observability:

• Check the logs of each nwaku node, verifying the messages are correctly received via the API and forwarded to other nodes.
• Check grafana dashboard localhost:3001 ”RLN Valid Messages Total” increases as expected.

Connect external full node

If you want to connect to the existing waku network a node with some custom configuration. Perhaps a different image or some other configuration, you can do it as follows. Bear in mind that if this node has other configuration (eg rln-relay-epoch-sec or rln-relay-user-message-limit) then it won’t behave properly.

• ⚠️set your own staticnode

docker run -it --network waku-simulator_simulation quay.io/wakuorg/nwaku-pr:2759-rln-v2 \
      --relay=true \
      --rln-relay=true \
      --rln-relay-dynamic=true \
      --rln-relay-eth-client-address=http://foundry:8545 \
      --rln-relay-eth-contract-address=0xCf7Ed3AccA5a467e9e704C703E8D87F634fB0Fc9 \
      --rln-relay-epoch-sec=1 \
      --rln-relay-user-message-limit=1 \
      --log-level=DEBUG \
      --staticnode=/ip4/10.2.0.16/tcp/60000/p2p/16Uiu2HAmAA99YfoLitSXgY1bHaqjaTKhyrU4M4y3D1rVj1bmcgL8 \
      --pubsub-topic=/waku/2/rs/66/0 \
      --cluster-id=66

You can for example try to connect a node running in a different cluster-id or other weird scenarios.

You can also try to connect multiple nodes with a loop. Note the &. Remember to kill the new nodes once you are done.

for i in {1..5}; do
    docker run -it --network waku-simulator_simulation quay.io/wakuorg/nwaku-pr:2759-rln-v2 \
      --relay=true \
      --rln-relay=true \
      --rln-relay-dynamic=true \
      --rln-relay-eth-client-address=http://foundry:8545 \
      --rln-relay-eth-contract-address=0xCf7Ed3AccA5a467e9e704C703E8D87F634fB0Fc9 \
      --rln-relay-epoch-sec=1 \
      --rln-relay-user-message-limit=1 \
      --log-level=DEBUG \
      --staticnode=/ip4/10.2.0.16/tcp/60000/p2p/16Uiu2HAmAA99YfoLitSXgY1bHaqjaTKhyrU4M4y3D1rVj1bmcgL8 \
      --pubsub-topic=/waku/2/rs/66/0 \
      --cluster-id=66 &
done

🎯Goals:

• Connect a different node(s) to the network for some ad hoc test.
• See how the network reacts to a node with different configuration.

👀Observability:

• Check the new node logs, ensuring the behaviour matches the expected.

Connect external spam node

By using the [nwaku-spammer] (https://github.com/waku-org/nwaku/pull/2821), you can connect a node to the network that spams the other nodes, sending messages exceeding its rate limit. It will register an RLN membership at startup. It should be configured with the same contract and rln-relay-user-message-limit as the waku nodes. If a node spams enough for the peer-score to go below the threshold, then the peers will disconnect from the spamming node.

• ⚠️ change staticnode to the node you wish. Note that the multiaddress is logged by every peer at startup.

docker run -it --network waku-simulator_simulation quay.io/wakuorg/nwaku-pr:2821 \
      --relay=true \
      --rln-relay=true \
      --rln-relay-dynamic=true \
      --rln-relay-eth-client-address=http://foundry:8545 \
      --rln-relay-eth-contract-address=0xCf7Ed3AccA5a467e9e704C703E8D87F634fB0Fc9 \
      --rln-relay-epoch-sec=1 \
      --rln-relay-user-message-limit=1 \
      --log-level=DEBUG \
      --staticnode=/ip4/10.2.0.16/tcp/60000/p2p/16Uiu2HAmAA99YfoLitSXgY1bHaqjaTKhyrU4M4y3D1rVj1bmcgL8 \
      --pubsub-topic=/waku/2/rs/66/0 \
      --cluster-id=66 \
      --rln-relay-eth-private-key=0xac0974bec39a17e36ba4a6b4d238ff944bacb478cbed5efcae784d7bf4f2ff80 \
      --rln-relay-cred-path=/keystore.json \
      --rln-relay-cred-password=password123 \
      --spammer=true \
      --spammer-delay-between-msg=200

An alternative config for the spamming node is to have it connect to the discv5 network instead of a single static node. The staticnode config can be replaced with the below config. The bootstrap node will log its ENR.

 --discv5-discovery=true \
 --discv5-enr-auto-update=True \
 --discv5-bootstrap-node=BOOTSTRAP_ENR\

You can also try to connect multiple spamming nodes, but it might be necessary to use a different private-key for each one to avoid the limitation of multiple contract transactions with the same nonce. Otherwise add a delay before running a new node. Note the &. Remember to kill the new nodes once you are done.

for i in {1..5}; do
docker run -it --network waku-simulator_simulation quay.io/wakuorg/nwaku-pr:2821 \
      --relay=true \
      --rln-relay=true \
      --rln-relay-dynamic=true \
      --rln-relay-eth-client-address=http://foundry:8545 \
      --rln-relay-eth-contract-address=0xCf7Ed3AccA5a467e9e704C703E8D87F634fB0Fc9 \
      --rln-relay-epoch-sec=1 \
      --rln-relay-user-message-limit=1 \
      --log-level=DEBUG \
      --staticnode=/ip4/10.2.0.16/tcp/60000/p2p/16Uiu2HAmAA99YfoLitSXgY1bHaqjaTKhyrU4M4y3D1rVj1bmcgL8 \
      --pubsub-topic=/waku/2/rs/66/0 \
      --cluster-id=66 \
      --rln-relay-eth-private-key=0xac0974bec39a17e36ba4a6b4d238ff944bacb478cbed5efcae784d7bf4f2ff80 \
      --rln-relay-cred-path=/keystore.json \
      --rln-relay-cred-password=password123 \
      --spammer=true \
      --spammer-delay-between-msg=200 &
done

The spammer node also provides a method to test burst messaging. It will send the total user-message-limit of messages sequentially without pauses, then rests for the epoch period and repeats.

docker run -it --network waku-simulator_simulation quay.io/wakuorg/nwaku-pr:2821 \
      --relay=true \
      --rln-relay=true \
      --rln-relay-dynamic=true \
      --rln-relay-eth-client-address=http://foundry:8545 \
      --rln-relay-eth-contract-address=0xCf7Ed3AccA5a467e9e704C703E8D87F634fB0Fc9 \
      --rln-relay-epoch-sec=1 \
      --rln-relay-user-message-limit=1 \
      --log-level=DEBUG \
      --staticnode=/ip4/10.2.0.16/tcp/60000/p2p/16Uiu2HAmAA99YfoLitSXgY1bHaqjaTKhyrU4M4y3D1rVj1bmcgL8 \
      --pubsub-topic=/waku/2/rs/66/0 \
      --cluster-id=66 \
      --rln-relay-eth-private-key=0xac0974bec39a17e36ba4a6b4d238ff944bacb478cbed5efcae784d7bf4f2ff80 \
      --rln-relay-cred-path=/keystore.json \
      --rln-relay-cred-password=password123 \
      --spammer=true \
      --spammer-burst=true

🎯Goals:

• Connect a spamming node(s) to the network where spam messages are rejected and misbehaving peers are disconnected for a time.
• See how the network reacts to different spamming rates.

👀Observability:

• Check the logs of the new node logs, ensuring the behaviour matches the expected.
• Check grafana metrics at localhost:3001.
• Check that the RLN membership was registered in the block explorer localhost:3000.

Connect external light node

By using go-waku-light, you can connect one or multiple light clients to the network. This utility can be configured to send messages at a given rate using a given peer as light-push. It will register a RLN membership at startup. Bear in mind that it should be configured with the same contract and user-message-limit as the waku nodes. You should modify the lightpush-peer.

Note that if you spin up multiple services like this using the same priv-key some of the transactions registering the RLN membership may fail due to the nonce being repeated. This can be fixed by using multiple keys or waiting for the registration to be completed before spinning up the next process.

• ⚠️ change lightpush-peer to the node you wish. Note that the multiaddress is logged by every peer at startup.

docker run --network waku-simulator_simulation alrevuelta/go-waku-light:4fabb22 \
--eth-endpoint=http://foundry:8545 \
--contract-address=0xCf7Ed3AccA5a467e9e704C703E8D87F634fB0Fc9 \
send-messages-loop \
--priv-key=ac0974bec39a17e36ba4a6b4d238ff944bacb478cbed5efcae784d7bf4f2ff80 \
--user-message-limit=1 \
--message="light client sending a rln message" \
--content-topic=/basic2/1/test/proto \
--pubsub-topic=/waku/2/rs/66/0 \
--cluster-id=66 \
--lightpush-peer=/ip4/10.2.0.16/tcp/60000/p2p/16Uiu2HAmAA99YfoLitSXgY1bHaqjaTKhyrU4M4y3D1rVj1bmcgL8 \
--message-every-secs=5 \
--epoch-size-secs=1

Note that in some examples, it could be interesting to run multiple instances, either in parallel or one after the other. For example if you set amount-message-to-send=1 this will send just 1 message and exit. You can for example run this 100 times, where a fresh RLN membership will be created on every run, create a new peerId, send a message and exit.

for i in {1..5}; do
    docker run --rm --network waku-simulator_simulation alrevuelta/go-waku-light:4fabb22 \
    --eth-endpoint=http://foundry:8545 \
    --contract-address=0xCf7Ed3AccA5a467e9e704C703E8D87F634fB0Fc9 \
    send-messages-loop \
    --priv-key=ac0974bec39a17e36ba4a6b4d238ff944bacb478cbed5efcae784d7bf4f2ff80 \
    --user-message-limit=1 \
    --message="light client sending a rln message" \
    --content-topic=/basic2/1/test/proto \
    --pubsub-topic=/waku/2/rs/66/0 \
    --cluster-id=66 \
    --lightpush-peer=/ip4/10.2.0.16/tcp/60000/p2p/16Uiu2HAm6a4kUT7YutsbwgQcmWw5VLzN3zj1StwiBVf2LUH9kb4A \
    --message-every-secs=5 \
    --epoch-size-secs=1 \
    --amount-message-to-send=1
    
    if [ $? -ne 0 ]; then
        echo "Command failed at iteration $i"
        break
    fi
done

🎯Goals:

• Tests lightpush end to end, where proofs are fetched directly from the contract

👀Observability:

• Check the logs of the node you provided as lightpush-peer.
• Check grafana metrics at localhost:3001.
• Check that the RLN membership was registered in the block explorer localhost:3000.

Connect external store node

One or more external store nodes can be connected to the waku-simulator network by using configuration similar to that shown below. The store node(s) can use the DB as backend from an existing staging or production system DB or a custom one. When connecting to any DB ensure that the store-message-retention-policy matches that of the system. The staticnode that the store node connects to can be any existing node in the waku-simulator network, alternatively use the discv5 configuration.

 --discv5-discovery=true \
 --discv5-enr-auto-update=True \
 --discv5-bootstrap-node=BOOTSTRAP_ENR\

The store node can be queried via the waku rest API. An easy way to find a message hash to query is by checking the store node logs for the message archived message:

docker logs nwaku_storenode_1 | grep -i "message archived"

The store node will need to have log-level set to DEBUG for this logging to be available.

The message-finder tool could also be useful for store node testing.

docker run -it --network waku-simulator_simulation --name nwaku_storenode_1 -d harbor.status.im/wakuorg/nwaku:latest \
    --relay=true\
    --rest=true\
    --rest-address=0.0.0.0\
    --rest-port=8645\
    --log-level=DEBUG\
    --pubsub-topic=/waku/2/rs/66/0\
    --cluster-id=66\
    --metrics-server=true\
    --discv5-discovery=true\
    --discv5-enr-auto-update=true\
    --store=true\
    --store-message-db-url=<e.g. postgres://pguser:pgpasswrd@127.0.0.1:5432/postgres>\
    --store-message-retention-policy=size:120GB\
    --staticnode=/ip4/0.0.0.0/tcp/60000/p2p/16Uiu2HAmC8Fe4Egsq6AKubmBPr52TQmwc26yoCkszwo6dfQncZ4m\
    --nodekey=5978783f8b1a16795032371fff7a526af352d9dca38179af7d71c0122942fa20

🎯Goals:

• Connect a store node(s) to the network for ad hoc store protocol testing.
• Verify that store queries get the expected results.

👀Observability:

• Check the logs of the newly connected store node, ensuring the behaviour matches the expected.

Register memberships

The go-waku-light tool can be used as well to register multiple RLN memberships. This can be useful to stress test the nodes, forcing a large amount of memberships. Set amount to the amount of memberships that you want to register. Note that it takes some time, since memberships are registered one after the other. You can spin up multiple services like this, but in that case you must provide different priv-key to each. Note that these memberships are kind of thrown away and not used to send messages.

docker run --network waku-simulator_simulation alrevuelta/go-waku-light:07b8f32 \
--eth-endpoint=http://foundry:8545 \
--contract-address=0xCf7Ed3AccA5a467e9e704C703E8D87F634fB0Fc9 \
register \
--priv-key=ac0974bec39a17e36ba4a6b4d238ff944bacb478cbed5efcae784d7bf4f2ff80 \
--user-message-limit=1 \
--amount=100

The foundry service in the waku-simulator generates deterministic accounts that can be used to register memberships. It is recommended to use different accounts for registering multiple memberships at the same time. The last 20 accounts generated are provided for ad-hoc testing purposes.

🎯Goals:

• Tests how the network reacts to a massive amount of memberships.
• Tests that nodes pick-up new memberships in real time and match.

👀Observability:

• Check the RLN memberships being registered in the block explorer localhost:3000.
• Check in grafana localhost:3001 the metric “RLN Registered Memberships” and verify the nodes are picking up the new registrations.