MWEB Progress Update Thread 2

Development
1

I’m starting this new topic for monthly MWEB updates which will continue on from the previous one.

April 2022 Progress:

Testing

Testing of the final RCs went really well. All major workflows seem to be working smoothly.

The majority of mining pools have reported to us that they’ve successfully mined MWEB blocks in testnet, and are ready for activation on mainnet.

Activation

MWEB officially locked in at height 2,257,920! :rocket:

Starting at block 2,265,984 (~May 19th), MWEB will be activated on mainnet, and everyone who has updated to v0.21.2 will be able to start sending and receiving LTC using the MWEB.

Final v0.21.2 Release

The final release will be available later this evening at https://litecoin.org/. Anyone interested in using the MWEB, and especially those who installed one of the earlier release candidates, should upgrade to the official v0.21.2 before MWEB activates. If you wait until after MWEB activation to upgrade, you’ll unfortunately be required to resync the blockchain from scratch.

While the consensus logic has been solid for a while now, the wallet has undergone drastic changes these past few months as we worked to resolve issues found during testing. The major workflows all seem to be working well now, and I expect most people to be able to use MWEB without issue, but this was an enormous change, so don’t hesitate to report any bugs or behavior that seems off to you.

29 Likes
9

I was away last week celebrating my 10th wedding anniversary (yep, I’m ancient), so I’m even later than usual this month. Sorry about that! :grimacing:

May (and part of June) 2022 Progress:

Critical Wallet Bug

Shortly after MWEB went live, a few users reported not receiving coins sent to their MWEB address. After some investigation, it was found that old pre-HD wallets that were upgraded using the upgradewallet command were generating stealth addresses incorrectly. https://twitter.com/DavidBurkett38/status/1528575311335591937

Fortunately, all funds were safu, but a fix was needed in order for those users to recover their coins.

v0.21.2.1 Release

A new version of Litecoin Core (v0.21.2.1) was released with the fix. You can download it here.

If you’re one of the few unlucky individuals whose wallets are not receiving correctly, upgrade to the new version and then run rescanblockchain 2265984 from the Litecoin console to recover your coins.

Other than this wallet bug, MWEB seems to be working really well. If you have any issues though, please report them on github, or in the MWEB telegram channel.

Now that activation is behind us, it’s time to move to the next big task. Starting this week, we’ll begin working on the design for mobile wallets and other light clients. MWEB presents many technical challenges for light clients, so it may take some time before we settle on a design, but I’ll continue to keep you updated on the progress each month.

10 Likes
10

June 2022 Progress:

Light Clients

After a meeting with @coblee and @losh11, we agreed on a rough design for supporting light clients. The following is subject to change, but here’s my current thinking, taken from the LIP (Litecoin Improvement Proposal) I’m writing to help standardize the light client sync process:

1. Download and verify all headers for the longest chain.
Headers can be requested from peers using getheaders messages, which will be returned in headers messages using the process described here.

2. Download and verify the HogEx transaction and MWEB header for the most recent block.
This data can be requested using a getdata message with type MSG_MWEB_COMMITMENT, which will be returned in the following 3 messages:

  • merkleblock - Contains the hash of the HogEx transaction, and enough to validate that it is the correct transaction according to the block’s tx merkle root.
  • tx - The serialized HogEx transaction.
  • mwebheader - The serialized MWEB header.

The light client shall validate that the hash of the HogEx transaction in the tx message matches the hash in the merkleblock message, and that it’s the last transaction committed to by the merkle root of the block. It shall then validate that the pubkey script of the first output contains the HogAddr, which shall consist of <OP_8><0x20> followed by the 32-byte hash of the MWEB header. Finally, it shall validate that the blake3 hash of the MWEB header matches the hash contained in the HogAddr.

3. Download and verify the UTXO leafset bitmap.
The leafset can be requested using a getdata message with type MSG_MWEB_LEAFSET. Verify that the hash of the bitmap matches the leafset_root value in the MWEB header.

4. Download the compact UTXOs and PMMR parent hashes.
These can be requested piecemeal from multiple peers in parallel using getmwebutxos messages. As compact UTXOs are downloaded and verified to belong to the longest chain, wallets can check to see if they own the outputs using the process described in LIP-0004 (Output Identification). Any UTXOs determined to not belong to the wallet may simply be discarded.

NOTE: A compact UTXO is an unspent MWEB output sans the rangeproof. When designing MWEB, we chose to hash them in a way where we would only need the hash of the rangeproof to verify the output hash, meaning light wallets can avoid downloading the nearly 1KB rangeproof for each UTXO.

Once I’m finished with my first draft of the LIP, I’ll submit it for review and then start making the P2P protocol changes necessary to support the design.

12 Likes
11

July 2022 Progress:

Light Clients

Last month, I described the LIP for standardizing the light client sync process. The LIP is just about ready for review, and I’ve made good progress implementing the changes.

Here is the status of each step:

1. Download and verify all headers for the longest chain.

Light clients already download the header chain, so no changes are needed for this step.

2. Download and verify the HogEx transaction and MWEB header for the most recent block.

I’ve written the code for responding to an MSG_MWEB_COMMITMENT getdata request with a new message containing the Merkle Block, HogEx transaction, and MWEB header. Tests were also written and everything appears to be working, so it’s ready to be submitted for review.

3. Download and verify the UTXO leafset bitmap.

The new getdata message type was added, as well as a new message for responding with the actual leafset.

Peers will be able to request a leafset for a recent block, but it may not be the most recent block. We only keep one working version of the UTXO leafset, so if an older leafset is requested, we need to “rewind” the leafset to the block height it’s requested for. Currently, I’m working on the code for performing this rewind.

4. Download the compact UTXOs and PMMR parent hashes.

The changes for this step have not yet been started.

I intend to wrap up the LIP this week and get it submitted for review. I should also be able to submit a PR for steps 2 & hopefully 3 before next month’s update.

12 Likes
12

August 2022 Progress:

Light Clients

The first draft of LIP-0006, which details the changes and additions to the p2p protocol, is now out for review.

A pull request has also been submitted for downloading and verifying HogEx transactions, MWEB headers, and the UTXO leafset bitmap. This covers steps 2 and 3 of the light client sync process.

That means the only piece still missing that light clients need to be able to sync MWEB data is the ability to download the compact UTXOs and their merkle proofs. As documented in LIP-0006, we want light clients to be able to request UTXOs from peers in parallel using a getmwebutxos message, which will be returned in a mwebutxos message.

I should be able to get this last piece coded and out for review this month. After that we’ll look at creating a new release (v0.21.3) with these new changes, and then start working with third party wallet developers to hopefully get MWEB added to all major LTC wallets :crossed_fingers:

8 Likes
13

September 2022 Progress:

Light Clients

The code for supporting retrieval of MWEB UTXOs (unspent coins) for use by MWEB light clients is out for review It was designed so that light clients can request UTXOs in batches, and verify that each batch is committed to by a recent block. This is similar to how they can currently retrieve regular LTC transactions and verify that the transactions are committed to in a block’s merkle root. In order to do this, each batch of UTXOs has to come with enough hashes to verify the outputs belong to the Merkle Mountain Range (MMR) that commits to the MWEB UTXO set.

This was more difficult to implement than I had hoped. Since MMRs are reasonably more complex than the simpler merkle roots that regular LTC blocks use, it turned out to be quite a challenge designing the algorithm that determines the minimum set of hashes necessary to verify the UTXOs. Because of the complexity of this algorithm, I’m going to hold off on creating the new release until some of the light client dev teams have had time to understand how the new code works. Instead, I’ll host a node with the latest code that wallet developers can develop and test against, and if we need to tweak it to make it a little simpler (at a slight efficiency cost), we can do that much easier than if the code was already released.

Aside from working with wallet developers and improving documentation to assist with their development, I’ll also be backfilling some functional tests and fixing a few latent wallet display bugs this month in preparation for when we decide to go forward with the v0.21.3 release.

9 Likes
15

October 2022 Progress:

Light Clients

I had a chance to talk to some wallet developers at the LTC Summit, which I used to better understand what each needs in order to support MWEB. From these discussions, it appears there will be a few different approaches taken for syncing and identifying MWEB wallet transactions, each with different performance, security, and privacy tradeoffs. My goal is to make sure we provide all of the APIs, p2p messages, and libraries necessary to give each wallet developer the tools they need to go their own way.

I wrote up a quick document briefly describing the two high-level approaches (client-side and server-side filtering) that wallet devs can take to support MWEB, and the pros and cons of each: MWEB Light Clients. I’ve been focusing strictly on client-side filtering these past few months, since that offers the strongest privacy and security guarantees, and while most wallet developers were on board with the idea, one showed more interest in server-side filtering for performance reasons.

View Keys

Server-side filtering of txs & outputs is done using “view keys,” which can be shared with a server so that they can identify transactions belonging to your wallet. Right now, that also means the server would learn the amounts of your transactions, unfortunately. There is a way this can be done without revealing the amounts by adding a new output format version, so this may be worth looking into in the future sometime. In the meantime, I’ve decided to work on adding full view key support to LTC Core that can serve as an example for other wallets.

This month, I plan to finish adding in view keys and get that out for review, so I can start working on MWEB support in our raw transactions APIs. These are the APIs that some third-party wallets, in particular hardware wallets, use to build transactions. We’ve already had a few 3rd party devs request MWEB functionality for these APIs, so it would be great to be able to include that in our next release.

7 Likes
16

November 2022 Progress:

PSBT

I decided to take a short break from light client work, and jumped ahead a little bit to start adding MWEB components to the PSBT(Partially-Signed Bitcoin Transaction) format. PSBT (See BIP-0174 and BIP-0370) is a flexible transaction serialization format that can be used to support building transactions using advanced workflows such as sending from a multisig wallet, or in the case where most interested in: hardware wallets.

In order to add MWEB support to PSBTs, I first had to merge in PSBTv2 (BIP-0370), which can be seen here. I then defined the MWEB fields we needed to add, and added them to the PSBT data structures along with serialization and deserialization here.

The remaining tasks for PSBT are:

  • Add signing logic for inputs and outputs
  • Write the transaction finalization logic here
  • Implement component “merging” here
  • Add functional test(s) covering the hardware wallet workflow
  • Document the PSBT format additions

View Keys

Since the hardware wallet PSBT workflow depends on view key support to identify UTXOs belonging to the wallet, I will continue working on view key support in parallel

8 Likes
17

I’ve been feeling under the weather for some time now, so I’m just now getting around to December’s update :face_with_thermometer:

December 2022 Progress:

I continued working on PSBT, and quickly discovered some limitations in my initial design. I had to redefine a lot of the MWEB fields before I could proceed with the PSBT tasks I listed last month. After doing that, I was able to complete the first 3:

  • Add signing logic for inputs and outputs
  • Write the transaction finalization logic here
  • Implement component “merging” here

I also have a basic functional test for the hardware wallet workflow (task 4), but there are a lot of scenarios I still want to add tests for.

I will be taking some time off, so there will not be a January update.

8 Likes
18

February & March 2023 Progress:

v0.21.2.2 Release

We released a new minor version of Litecoin Core (v0.21.2.2) in February which contained some important security fixes. If you haven’t upgraded yet, you can download it here.

v24 Release

I’ve started working on a new major release for Litecoin Core that will contain everything that I’ve been working on this past year:

  • PSBTs - To encourage hardware support for MWEB
  • P2P Support for Light Clients - To allow for mobile and other more user-friendly wallets
  • View Keys - To support “watch-only” wallets that don’t require having access to private spend keys in order to identify transactions you’ve sent or received.
  • Payment Proofs - For traditional litecoin transactions, where proving you sent coins to an address could be done by just signing a message with your private key, and pointing to the transaction on the blockchain which shows the source address (which you just proved to be the owner of), and the destination address and amount. With MWEB, addresses and amounts are all encrypted, so in the case that you need to prove to an arbiter that you made a payment, a more complex proof is needed. This proof is described here.
  • Descriptor wallets - The wallet code, originally written by Satoshi, was very limited, and not designed to support the many script types and features that have since been developed (P2SH, Taproot, BIP32, and now MWEB). Developers have been forced to hack in new wallet features in a messy, unsustainable way. To deal with this, BTC developers introduced “descriptor wallets,” which provide a more extensible replacement for the existing wallet backend. You can read more about them here. With Litecoin Core v24, we’ll be enabling descriptor wallets, and extending them to handle MWEB addresses, which should help solve a number of limitations we had with our initial MWEB wallet design.

So far, I’ve added the classic (pre-MWEB) litecoin code changes to the v24 bitcoin code, and am working on fixing tests. I’ll then merge in the already-released MWEB code, followed by the new features listed above, submitting everything for code review after each step. This will be a lengthy process taking us well into the summer, but the end result should be a new release that’s up-to-date with bitcoin core, and will include all of the new MWEB features everyone has been waiting for.

13 Likes
19

Thanks for the update @David . We are gearing up for the Litewallet refactor so we are eager to get to it.

Good you are feeling better! I was pretty sick last month myself.

6 Likes
20

April 2023 Progress:

v24 Release

Over the past couple of months, the focus has been primarily on integrating the classic (pre-MWEB) Litecoin code changes into the v24 Bitcoin code. This process is nearing its conclusion, with all unit-tests fixed, and just a few functional tests remaining to be fixed. Once the last few tests are resolved, I will submit the merged code for review as planned.

I’ll be updating the following task list each month as we work toward the release:

:white_square_button: Integrate pre-MWEB litecoin into bitcoin’s v24 codebase (A few tests remaining, then review)
:white_square_button: Merge in the previously-released MWEB code (Not started, expected by end of May)
:white_square_button: P2P Support for Light Clients (Implemented, in need of merging & thorough review)
:white_square_button: Enable descriptor wallets w/ MWEB address support (Not started)
:white_square_button: Finish implementing PSBTs (Mostly implemented, needs testing and review)
:white_square_button: View key support (Rough design known, implementation started)
:white_square_button: Payment Proofs (Design outlined in LIP-0004, implementation not started)
:white_square_button: Release Notes (Not started)
:white_square_button: Gitian Build & Publish (Not started)

6 Likes
21

May 2023 Progress:

v24 Release

Progress continues with the integration of Litecoin code changes into the v24 Bitcoin codebase. The month of May saw a major milestone with the resolution of all remaining functional tests for the pre-MWEB merge task. The code has been submitted for review here.

Furthermore, we’ve made significant strides in merging the previously-released MWEB code. The majority of this task is now complete, marking another key step forward. However, there are still a few changes needed on the wallet side and several tests that need to be fixed to finalize this phase.

I also took some time to work through bitcoin’s descriptor wallet code, and have a decent understanding of how it all works, and how the data is stored in the wallet database. I’ve started experimenting with different ways of supporting MWEB keys and addresses using descriptors, but a lot more thought is needed before I commit to a design.

Here’s our updated task list as we move closer to the v24 release:

:white_check_mark: Integrate pre-MWEB litecoin into bitcoin’s v24 codebase (In review)
:white_square_button: Merge in the previously-released MWEB code (Majority merged, a few changes & tests left)
:white_square_button: P2P Support for Light Clients (Implemented, in need of merging & thorough review)
:white_square_button: Enable descriptor wallets w/ MWEB address support (Design in progress)
:white_square_button: Finish implementing PSBTs (Mostly implemented, needs testing and review)
:white_square_button: View key support (Rough design known, implementation started)
:white_square_button: Payment Proofs (Design outlined in LIP-0004, implementation not started)
:white_square_button: Release Notes (Not started)
:white_square_button: GUIX Build & Publish (Not started)

6 Likes
22

June 2023 Progress:

v24 Release

The initial MWEB code has been merged into the v24.x codebase, and will be submitted for review shortly. I encountered a couple of obstacles that slowed down momentum, particularly on the wallet side.

When MWEB was implemented, I was able to make its node logic very modular, making merging that relatively easy (with the exception of the mempool, which is still rather complex). I was not as lucky with the wallet code, however, which requires quite a bit of merging. And it turns out that Bitcoin’s wallet code has been refactored quite a bit over the past few releases, so this ended up being a much slower, more tedious process than I hoped.

Updated Task List:

:white_check_mark: Integrate pre-MWEB litecoin into bitcoin’s v24 codebase (In review)
:white_check_mark: Merge in the previously-released MWEB code (Will update with review link shortly)
:white_square_button: P2P Support for Light Clients (Implemented, in need of merging & thorough review)
:white_square_button: Enable descriptor wallets w/ MWEB address support (Design in progress)
:white_square_button: Finish implementing PSBTs (Mostly implemented, needs testing and review)
:white_square_button: View key support (Rough design known, implementation started)
:white_square_button: Payment Proofs (Design outlined in LIP-0004, implementation not started)
:white_square_button: Release Notes (Not started)
:white_square_button: Gitian Build & Publish (Not started)

My goal for July is to implement MWEB address support for descriptor wallets.

12 Likes
23

It’s time for my next semi-annual “monthly update”. As a bonus for waiting a few extra days, I also have an update from @losh11 about the progress he and Hector have made with their LTC development.

Loshan’s Update

Over the past few months, significant effort has gone into bringing MWEB support to light clients. This required an update to Litecoin Core, adding new p2p messages, which light clients can fetch from full nodes and verify. This process has been described in LIP006 and has now been implemented and merged into Litecoin Core.

We plan to promptly release a new minor build of Litecoin Core v0.21.3, which includes these changes, alongside serving BIP157/BIP158 filters by default and various other bug fixes. Originally we had planned to release MWEB light client server in the next major release of Litecoin Core v24; however, we felt it necessary to expedite this new feature. A release candidate of Litecoin Core v0.21.3 will be available very shortly. Full node operators who would like to support MWEB light clients should upgrade as soon as possible.

For our client side implementation, we chose to work on top of the LTCSuite libraries. To summarise, a light client node makes a request to fetch MWEBUTXOs, HogExAndMWEBHeaders and MWEBLeafset from v0.21.3 peers (at various stages). The light client will validate this data. Then the client will identify outputs belonging to the wallet, and store it. The last step is to construct transactions to spend funds, this includes support for pegins/pegouts/mweb txs.

Developer Hector Chu has submitted pull requests to the following libraries:
ltcd: MWEB Light Client by hectorchu · Pull Request #35 · ltcsuite/ltcd · GitHub
ltcwallet: MWEB Light Client by hectorchu · Pull Request #10 · ltcsuite/ltcwallet · GitHub
neutrino: MWEB Light Client by hectorchu · Pull Request #10 · ltcsuite/neutrino · GitHub
lnd: MWEB Light Client by hectorchu · Pull Request #8 · ltcsuite/lnd · GitHub

Further code review and testing is currently in progress, before MWEB support is merged in. If you are interested in code review, and knowledgeable, please feel free to contact us.

We have also updated ltcsuite libraries to stay in sync with our upstream, btcsuite. This brings along many improvements, such as: taproot support, cpu optimisations for mempool scanning, rescanning uses batch fetching BIP158 filters for non-MWEB litecoin blocks. As of the moment, lndltc is synced with lnd v0.17.3, neutrino is synced with upstream’s v0.16. There are still some minor fixes required (to fix tests) before the MWEB pull requests can be merged and release versions tagged.

Additionally, Hector has also replaced the legacy litecoin backend for the Litecoin Foundation’s Litewallet mobile wallet, with ltcsuite. This means Litewallet (newborn - the internal alpha) can now make MWEB txs. A future release of Litewallet with MWEB is planned for the near future.

v24 Release (My Update)

The MWEB wallet code underwent significant refactoring in order to add MWEB support to PSBTs and Descriptors.

PSBTs

Partially Signed Bitcoin Transactions (PSBTs) are a standardized format for passing around in-process transactions. These are useful for hardware wallets to help separate signing from transaction building, and also useful for multisig wallets where PSBTs are passed between the signers, with each adding their signature and passing to the next.

We were able to extend the PSBT standard to include MWEB transaction components, which led to a large refactoring of much of the wallet code to support transactions in various states of completion, whereas before, they were always able to just assume MWEB transactions being passed around in the code were complete and fully signed.

Transaction building all seems to be working correctly now for v24, and is much cleaner, better tested, and more stable than before. I’ll be reaching out to hardware wallet manufacturers to make sure the PSBT design we went with works for them, but aside from that, this task can be considered complete.

Descriptors

In a nutshell, Output Descriptors are just a convenient way of representing individual or collections of output scripts and addresses. They’re used by the wallet to track and manage addresses and keys owned or watched by the wallet.
You can read all about them here, or refer to the official documentation.

Despite being released in Bitcoin Core v17, Litecoin Core intentionally never supported them because it was a complete change to how the wallet handled keys and everything related to them, which happens to cover quite a bit (generating addresses, signing transactions, backing up and restoring wallets, etc.). Descriptor wallets even have a different database (Sqlite, instead of BerkeleyDB) and a separate set of RPC APIs (the standardized bridges that allow other apps to interact with Litecoin Core). We wanted to make sure we got the design correct before enabling descriptor wallets, which I believe we finally have, so v24 will have full support for descriptor wallets.

Some boring technical details

We needed a way to describe the MWEB stealth addresses belonging to a wallet. This ultimately meant a way of describing the master_scan_key(a) and the master_spend_pubkey(B), which are the 2 keys that can be used to generate an ~infinite number of stealth subaddresses (see LIP-0004 - Subaddress Generation).

There were a number of ways we could’ve represented the 2 stealth address master keys, but after some trial and error, we finally settled on mweb([master_seed_key]/[bip32_path]/x), where:

  • [master_seed_key]` is the wallet’s seed xprv or xpub key
  • [bip32_path] is the BIP32 path to the parent key of the master_scan_key(a) and the master_spend_pubkey(B) (See: BIP-0032 - Child Key Derivation)
  • x just means the master_scan_key(a) is the hardened child key with index 0, and the master_spend_pubkey(B) is the hardened child key with index 1

As an example, a wallet with the descriptor mweb(xpub6ERApfZwUNrhLCkDtcHTcxd75RbzS1ed54G1LkBUHQVHQKqhMkhgbmJbZRkrgZw4koxb5JaHWkY4ALHY2grBGRjaDMzQLcgJvLJuZZvRcEL/0'/100'/x) describes all MWEB stealth subaddresses that can be derived from the master_scan_key(a) at xpub6ERApfZwUNrhLCkDtcHTcxd75RbzS1ed54G1LkBUHQVHQKqhMkhgbmJbZRkrgZw4koxb5JaHWkY4ALHY2grBGRjaDMzQLcgJvLJuZZvRcEL/0'/100'/0' and the master_spend_pubkey(B) at xpub6ERApfZwUNrhLCkDtcHTcxd75RbzS1ed54G1LkBUHQVHQKqhMkhgbmJbZRkrgZw4koxb5JaHWkY4ALHY2grBGRjaDMzQLcgJvLJuZZvRcEL/0'/100'/1'.

What’s Left?

The 2 main tasks remaining before building a release candidate that the community can start testing is view key support, which I’m wrapping up the implementation for, and we need a fix for syncing stability issues we’re experiencing as a result of a new presync process bitcoin added to eliminate their existing checkpointing system.

Updated Task List:

:white_check_mark: Integrate pre-MWEB litecoin into bitcoin’s v24 codebase (All commits prefixed with “Litecoin:”)
     :white_square_button: Fix syncing stability issues caused by new presync process (Not Started)
:white_check_mark: Merge in the previously-released MWEB code (Several commits, starting with this one)
:white_check_mark: P2P Support for Light Clients (Finished, to be released early in v21.0.3. Review)
:white_check_mark: Enable descriptor wallets w/ MWEB address support (Finished, see descriptor.cpp in this commit)
:white_check_mark: MWEB Support for PSBTs (Finished)
:white_square_button: View key support (Finalizing implementation, tests still needed)
:white_square_button: Payment Proofs (Design outlined in LIP-0004, excluding from release due to lack of demand)
:white_square_button: Community Beta Testing (Not started)
:white_square_button: Release Notes (Not started)
:white_square_button: Guix Build & Publish (Not started)

8 Likes