MWEB Progress Update Thread 2

I’m starting this new topic for monthly MWEB updates which will continue on from the previous one.

April 2022 Progress:

Testing

Testing of the final RCs went really well. All major workflows seem to be working smoothly.

The majority of mining pools have reported to us that they’ve successfully mined MWEB blocks in testnet, and are ready for activation on mainnet.

Activation

MWEB officially locked in at height 2,257,920! :rocket:

Starting at block 2,265,984 (~May 19th), MWEB will be activated on mainnet, and everyone who has updated to v0.21.2 will be able to start sending and receiving LTC using the MWEB.

Final v0.21.2 Release

The final release will be available later this evening at https://litecoin.org/. Anyone interested in using the MWEB, and especially those who installed one of the earlier release candidates, should upgrade to the official v0.21.2 before MWEB activates. If you wait until after MWEB activation to upgrade, you’ll unfortunately be required to resync the blockchain from scratch.

While the consensus logic has been solid for a while now, the wallet has undergone drastic changes these past few months as we worked to resolve issues found during testing. The major workflows all seem to be working well now, and I expect most people to be able to use MWEB without issue, but this was an enormous change, so don’t hesitate to report any bugs or behavior that seems off to you.

24 Likes

I was away last week celebrating my 10th wedding anniversary (yep, I’m ancient), so I’m even later than usual this month. Sorry about that! :grimacing:

May (and part of June) 2022 Progress:

Critical Wallet Bug

Shortly after MWEB went live, a few users reported not receiving coins sent to their MWEB address. After some investigation, it was found that old pre-HD wallets that were upgraded using the upgradewallet command were generating stealth addresses incorrectly. https://twitter.com/DavidBurkett38/status/1528575311335591937

Fortunately, all funds were safu, but a fix was needed in order for those users to recover their coins.

v0.21.2.1 Release

A new version of Litecoin Core (v0.21.2.1) was released with the fix. You can download it here.

If you’re one of the few unlucky individuals whose wallets are not receiving correctly, upgrade to the new version and then run rescanblockchain 2265984 from the Litecoin console to recover your coins.


Other than this wallet bug, MWEB seems to be working really well. If you have any issues though, please report them on github, or in the MWEB telegram channel.

Now that activation is behind us, it’s time to move to the next big task. Starting this week, we’ll begin working on the design for mobile wallets and other light clients. MWEB presents many technical challenges for light clients, so it may take some time before we settle on a design, but I’ll continue to keep you updated on the progress each month.

8 Likes

June 2022 Progress:

Light Clients

After a meeting with @coblee and @losh11, we agreed on a rough design for supporting light clients. The following is subject to change, but here’s my current thinking, taken from the LIP (Litecoin Improvement Proposal) I’m writing to help standardize the light client sync process:


1. Download and verify all headers for the longest chain.
Headers can be requested from peers using getheaders messages, which will be returned in headers messages using the process described here.

2. Download and verify the HogEx transaction and MWEB header for the most recent block.
This data can be requested using a getdata message with type MSG_MWEB_COMMITMENT, which will be returned in the following 3 messages:

  • merkleblock - Contains the hash of the HogEx transaction, and enough to validate that it is the correct transaction according to the block’s tx merkle root.
  • tx - The serialized HogEx transaction.
  • mwebheader - The serialized MWEB header.

The light client shall validate that the hash of the HogEx transaction in the tx message matches the hash in the merkleblock message, and that it’s the last transaction committed to by the merkle root of the block. It shall then validate that the pubkey script of the first output contains the HogAddr, which shall consist of <OP_8><0x20> followed by the 32-byte hash of the MWEB header. Finally, it shall validate that the blake3 hash of the MWEB header matches the hash contained in the HogAddr.

3. Download and verify the UTXO leafset bitmap.
The leafset can be requested using a getdata message with type MSG_MWEB_LEAFSET. Verify that the hash of the bitmap matches the leafset_root value in the MWEB header.

4. Download the compact UTXOs and PMMR parent hashes.
These can be requested piecemeal from multiple peers in parallel using getmwebutxos messages. As compact UTXOs are downloaded and verified to belong to the longest chain, wallets can check to see if they own the outputs using the process described in LIP-0004 (Output Identification). Any UTXOs determined to not belong to the wallet may simply be discarded.

NOTE: A compact UTXO is an unspent MWEB output sans the rangeproof. When designing MWEB, we chose to hash them in a way where we would only need the hash of the rangeproof to verify the output hash, meaning light wallets can avoid downloading the nearly 1KB rangeproof for each UTXO.


Once I’m finished with my first draft of the LIP, I’ll submit it for review and then start making the P2P protocol changes necessary to support the design.

10 Likes

July 2022 Progress:

Light Clients

Last month, I described the LIP for standardizing the light client sync process. The LIP is just about ready for review, and I’ve made good progress implementing the changes.

Here is the status of each step:

1. Download and verify all headers for the longest chain.

Light clients already download the header chain, so no changes are needed for this step.

2. Download and verify the HogEx transaction and MWEB header for the most recent block.

I’ve written the code for responding to an MSG_MWEB_COMMITMENT getdata request with a new message containing the Merkle Block, HogEx transaction, and MWEB header. Tests were also written and everything appears to be working, so it’s ready to be submitted for review.

3. Download and verify the UTXO leafset bitmap.

The new getdata message type was added, as well as a new message for responding with the actual leafset.

Peers will be able to request a leafset for a recent block, but it may not be the most recent block. We only keep one working version of the UTXO leafset, so if an older leafset is requested, we need to “rewind” the leafset to the block height it’s requested for. Currently, I’m working on the code for performing this rewind.

4. Download the compact UTXOs and PMMR parent hashes.

The changes for this step have not yet been started.


I intend to wrap up the LIP this week and get it submitted for review. I should also be able to submit a PR for steps 2 & hopefully 3 before next month’s update.

11 Likes

August 2022 Progress:

Light Clients

The first draft of LIP-0006, which details the changes and additions to the p2p protocol, is now out for review.

A pull request has also been submitted for downloading and verifying HogEx transactions, MWEB headers, and the UTXO leafset bitmap. This covers steps 2 and 3 of the light client sync process.

That means the only piece still missing that light clients need to be able to sync MWEB data is the ability to download the compact UTXOs and their merkle proofs. As documented in LIP-0006, we want light clients to be able to request UTXOs from peers in parallel using a getmwebutxos message, which will be returned in a mwebutxos message.

I should be able to get this last piece coded and out for review this month. After that we’ll look at creating a new release (v0.21.3) with these new changes, and then start working with third party wallet developers to hopefully get MWEB added to all major LTC wallets :crossed_fingers:

7 Likes

September 2022 Progress:

Light Clients

The code for supporting retrieval of MWEB UTXOs (unspent coins) for use by MWEB light clients is out for review It was designed so that light clients can request UTXOs in batches, and verify that each batch is committed to by a recent block. This is similar to how they can currently retrieve regular LTC transactions and verify that the transactions are committed to in a block’s merkle root. In order to do this, each batch of UTXOs has to come with enough hashes to verify the outputs belong to the Merkle Mountain Range (MMR) that commits to the MWEB UTXO set.

This was more difficult to implement than I had hoped. Since MMRs are reasonably more complex than the simpler merkle roots that regular LTC blocks use, it turned out to be quite a challenge designing the algorithm that determines the minimum set of hashes necessary to verify the UTXOs. Because of the complexity of this algorithm, I’m going to hold off on creating the new release until some of the light client dev teams have had time to understand how the new code works. Instead, I’ll host a node with the latest code that wallet developers can develop and test against, and if we need to tweak it to make it a little simpler (at a slight efficiency cost), we can do that much easier than if the code was already released.


Aside from working with wallet developers and improving documentation to assist with their development, I’ll also be backfilling some functional tests and fixing a few latent wallet display bugs this month in preparation for when we decide to go forward with the v0.21.3 release.

8 Likes

October 2022 Progress:

Light Clients

I had a chance to talk to some wallet developers at the LTC Summit, which I used to better understand what each needs in order to support MWEB. From these discussions, it appears there will be a few different approaches taken for syncing and identifying MWEB wallet transactions, each with different performance, security, and privacy tradeoffs. My goal is to make sure we provide all of the APIs, p2p messages, and libraries necessary to give each wallet developer the tools they need to go their own way.

I wrote up a quick document briefly describing the two high-level approaches (client-side and server-side filtering) that wallet devs can take to support MWEB, and the pros and cons of each: MWEB Light Clients. I’ve been focusing strictly on client-side filtering these past few months, since that offers the strongest privacy and security guarantees, and while most wallet developers were on board with the idea, one showed more interest in server-side filtering for performance reasons.

View Keys

Server-side filtering of txs & outputs is done using “view keys,” which can be shared with a server so that they can identify transactions belonging to your wallet. Right now, that also means the server would learn the amounts of your transactions, unfortunately. There is a way this can be done without revealing the amounts by adding a new output format version, so this may be worth looking into in the future sometime. In the meantime, I’ve decided to work on adding full view key support to LTC Core that can serve as an example for other wallets.


This month, I plan to finish adding in view keys and get that out for review, so I can start working on MWEB support in our raw transactions APIs. These are the APIs that some third-party wallets, in particular hardware wallets, use to build transactions. We’ve already had a few 3rd party devs request MWEB functionality for these APIs, so it would be great to be able to include that in our next release.

5 Likes

November 2022 Progress:

PSBT

I decided to take a short break from light client work, and jumped ahead a little bit to start adding MWEB components to the PSBT(Partially-Signed Bitcoin Transaction) format. PSBT (See BIP-0174 and BIP-0370) is a flexible transaction serialization format that can be used to support building transactions using advanced workflows such as sending from a multisig wallet, or in the case where most interested in: hardware wallets.

In order to add MWEB support to PSBTs, I first had to merge in PSBTv2 (BIP-0370), which can be seen here. I then defined the MWEB fields we needed to add, and added them to the PSBT data structures along with serialization and deserialization here.

The remaining tasks for PSBT are:

  • Add signing logic for inputs and outputs
  • Write the transaction finalization logic here
  • Implement component “merging” here
  • Add functional test(s) covering the hardware wallet workflow
  • Document the PSBT format additions

View Keys

Since the hardware wallet PSBT workflow depends on view key support to identify UTXOs belonging to the wallet, I will continue working on view key support in parallel

5 Likes