[Press Release] Soteria Topic Sharing | Min Wu: How to run and maintain blockchain networks

Claire: Hello everyone! I'm Claire Wu, founder of the “Magic Piper Technology Development Community”. Under Xi Da-da's call, many companies are studying how to use blockchain + and use the advantages of blockchain to improve the credibility of enterprises, products, or services. There are more and more blockchains sprouting up, and how to maintain blockchains is becoming more and more important. Today, I am very happy to invite Mr. Wu Min to discuss with us a topic that everyone is very concerned about: “How to operate and maintain a blockchain network”. Mr. Min Wu Wu...

Wu Min: Hello, I'm Min Wu. Today, I'm mainly going to discuss with you how to run and maintain a blockchain network.

First, let's discuss the different types of blockchains and their related systems. (Because the operation and maintenance of different blockchain systems are not the same), today we will mainly look at the differences between public chains and private chains.

Public Blockchain (Public Blockchain) vs. Private Blockchain (Private Blockchain)

A public chain refers to a blockchain where anyone in the world can participate and access the system at any time to read data, confirm transactions, and register competitively. Public chains are generally considered “completely decentralized” because no individual or agency can control the reading or writing of data, or allow others to participate.

The characteristics of public chains: low network congestion efficiency, high transaction costs, and poor scalability, such as Bitcoin blocks every 10 minutes.

Advantages of public chains: They are more thoroughly decentralized than private chains, and anyone can participate.

A private chain is a blockchain where only one organization has permission to write to it. Reading rights may be open to the outside world, or limited to some extent. Related applications are such as database management, auditing, and even a company. Although in some cases it is desirable that it be publicly auditable, in many cases, public readability is not necessary.

The characteristics of private chains: they are not open, and the rights are in the hands of a few organizations.

Advantages of private chains: fast transaction speed, good privacy, low transaction costs

Who is good or bad, public chains or private chains, when should you use public chains, and when should you use private chains are beyond the scope of today's discussion. We assume that a blockchain solution has already been selected.

Blockchain related systems and ecosystems

Both public and private chains will have wallets or clients, so end users can make transactions, transfers, DAPPs, etc. on the blockchain. With the exception of very early users and developers, most users want a mobile user interface, at least a web version.

POW's public chain will have miners. Mining is the only way to obtain bitcoins. Miners need to buy professional mining equipment (ASIC), connect to the network and power supply, and then mine. Professional mining machines are very noisy, so it is not recommended for everyone to deploy them at home.

Supernodes: These are the M nodes that are born from N backup nodes and are elected by all token holders to finally obtain accounting rights.

Due to our limited time, we will not discuss the operation and maintenance issues of wallets/applications/miners/supernodes today; we will focus on the blockchain itself.

Similar to most IT projects, the problems that need to be solved in operation and maintenance will include several aspects such as deployment/monitoring/upgrade/testing. Let's analyze them one by one below.

deploys

In blockchain projects, a large part of the deployment itself is not a technical issue; it is a matter of balancing the interests of multiple parties. For example, if it's a POW, how to cold start, how to have enough full nodes, and how to have enough people participate in mining, these are not problems that can be solved simply by technology; we won't discuss them today. Assuming none of these are problems, then it's simple. Download the source code on GitHub, compile it (note, competent participants must compile it themselves from the source code, don't use the execution files that others have already compiled), execute, and get it done.

Is it really that simple? No. For example, most public chains use peer-to-peer communication. For example, Xiaoming and Xiao Fang are two nodes. They will exchange information from 8 nodes in their known nodes, just like 8 neighbors in real life. Then they exchanged information about their neighbors with these 8 nodes. After layer by layer, we soon had the information of all nodes.

But how did Xiaoming and Xiao Fang meet in the beginning? Tinder or Momo. None of them. It's DNS. Take Bitcoin as an example. Everyone obtains other people's information through fixed DNS nodes.

For example: seed.bitcoin.sipa.be

The specific code can be found here:

https://github.com/bitcoin/bitcoin/blob/30521302f90e4856a7516867b32a4576fa6d98b3/src/chainparams.cpp#L116

vSeeds.emplace_back (“seed.bitcoin.sipa.be”);//Pieter Wuille, only supports x1, x5, x9, and xd

vSeeds.emplace_back (“dnseed.bluematt.me”);//Matt Corallo, only supports x9

vSeeds.emplace_back (“dnseed.bitcoin.dashjr.org”);//Luke Dashjr

vSeeds.emplace_back (“seed.bitcoinstats.com”);//Christian Decker, supports x1 - xf

vSeeds.emplace_back (“seed.bitcoin.jonasschnelli.ch”);//Jonas Schnelli, only supports x1, x5, x9, and xd

vSeeds.emplace_back (“seed.btc.petertodd.org”);//Peter Todd, only supports x1, x5, x9, and xd

vSeeds.emplace_back (“seed.bitcoin.sprovoost.nl”);//Sjors Provoost

vSeeds.emplace_back (“dnseed.emzy.de”);//Stephan Oeste

(Just a gossip here. If you're concerned about Bitcoin's Taproot Privacy Improvement, the bully Pieter Wuille is the sipa on GitHub, the first in the DNS above)

Returning to the deployment issue, POW's public chain is fully open, and anyone can participate. DNS was very important in the beginning, otherwise people wouldn't know where to look for an “organization.” Also, don't just have one DNS (Bitcoin has 8); otherwise, after being attacked by DDOS, it will be unsightly to temporarily prepare a new DNS.

There is also the bootstrap process. This process usually means that when a full node first joins the network (finds the organization), it needs to download previous historical data from the network. For example, today (2019/11), Bitcoin already has about 250GB of data. A 4K movie uses 7.2 GB per hour, and Bitcoin's historical data is about 35 hours of 4K movies. How to download this data from the network to the full node's hard drive (e-mails don't work; after all, it's not a real 4K movie), there will be different solutions at different stages of the project. The code is all written by myself, and it's all open source. There are many types. Remember, don't just bootstrap from one seed; there are various random distribution methods to choose from; otherwise, you won't look good at DDOS yourself.

Let's take another look at how private chain deployment is different?

It's not the same at all.

The code is not open source, and it's useless if it's open source, because the node that can verify that the transaction is generated must be deployed by itself. For example, 500 nodes (some people ask, why not 5 nodes; in fact, I don't know. As mentioned above, a large part of the deployment itself is not a technical issue; it is a matter of balancing the interests of multiple parties).

So here's the problem. These nodes require the same operating system, the same versions, the same patches, and the same settings. Then let's go to cloud computing. Everyone uses AWS (or other, usually has a cloud platform of their choice. Cross-platform is highly not recommended; you'll have a lot of trouble later); it's best to use a unified Terraform+ Dockerfile. This makes it easier to troubleshoot problems.

The initial deployment of a private chain is relatively easy to control. After all, it is controlled by permission. If there are many open applications later, adding a few more load balancers will help to dynamically adjust traffic, and can also avoid many single-point failures. Moreover, the biggest benefit is that you can do close loop testing for a period of time and then publish, so the pressure on operation and maintenance will be much less.

surveillance

Public chain monitoring is difficult; after all, those are the levels you can see. The consensus layer is the most important, and the least easy to monitor. Because the Internet is open, the software is open source. For example, you can collect and monitor log files on your own node, but others won't work; they don't necessarily have the same version of the software as you. It is recommended that everyone write some peer-to-peer crawlers (find them on GitHub). At least the nodes that publicly participate in the network can be counted, then you can guess the changes in the overall network and further generate a dash board, so you can clearly see what you need at a glance.

Private chain monitoring can be reflected at several different levels. Network usage in different regions needs to be monitored, the number of online Dockers, the CPU and memory conditions of the host, the number of abnormal requests, etc. (there are no such data chains, because all nodes are governed separately). Fortunately, many data AWS has ready-made tools that can be used (or other cloud service providers), which is why try not to cross platforms; otherwise, it is difficult to have a unified monitoring platform.

Whether it's a public chain or a private chain, transactions packaged in a blockchain need to be monitored, such as a large single transaction (examples will follow), such as blocks containing a large number of transactions above the normal level, such as fees that exceed the normal range, such as excessive number of node connections. These aren't necessarily all issues, and don't necessarily require action, but if you don't know them, they must be a problem.

upgrading

A very important aspect of the public chain upgrade is compatibility. The network is open, and not everyone (almost always) upgrades at the same time period. At this time, the dashboard mentioned above is useful. You can at least know how many people have upgraded to the new version and how many people are still on the old version.

Private chain upgrades are more about considering capacity issues. For example, the current RPS is 500K, and one container can handle 20 RPS, for a total of 25,000 containers. What you need to do to upgrade the new version is to upgrade a small portion of the container each time, upgrade it several times, and finally upgrade all the containers to the latest version. There are also issues with rolling back if the upgrade fails, and data compatibility issues after the rollback. The tests that need to be introduced below are very helpful.

tests

Why test? This is not within the scope of our discussion today; anyway, testing is a must.

Public chains and private chains are much the same in terms of testing. For example, every PR (pull request) requires testing before each release, and usually requires a testnet to test anytime, anywhere. Of course, the focus of each test is different; specific questions are analyzed in detail.

OK, that's all for now. Let's talk about two examples, all of which are Bitcoin.

The first one relates to upgrades.

Problems with the upgrade: https://github.com/bitcoin/bips/blob/master/bip-0050.mediawiki

Upgraded to 0.8 in 2013. There is a bug in the pre-0.8 code. The upgraded version doesn't have this bug. From the current point of view (Zhu Geliang later), of course everyone needs to upgrade. In fact, what happened is completely different from what everyone thinks now. Take a look at it in detail:

In the pre-0.8 code, the background data was stored using Berkeley DB. In the new 0.8 code, it was upgraded to LevelDB. Under the default settings of Berkeley DB, it cannot handle relatively large blocks (blocks that are large and have many transactions, not blocks that are large but have a small number of transactions). In the documentation, the developers wrote how to adjust the default settings to support larger blocks. How is it written? Here's the original text:

The recommended algorithm for considering the maximum number of locks, lockers, and lock objects is to run the application under dangerous conditions and then review the lock system's Statistics to evaluate the maximum number of locks, lockers, and lock objects that were used. Then, double these values for safety.

That is, test it yourself at high pressure and then multiply the observed data by 2. Well, here's the problem. Everyone's environment for every node or mining machine is different. The test conditions are different, the time is different, and the data obtained under high pressure is different. It's also different after multiplying by 2. Well, in other words, if everyone follows the documentation, the data everyone gets (that is, the consensus at this level) will not be the same. Remember, a different consensus will be very bad, and it will diverge.

What is it actually like? Most people are smart (lazy) to use the default settings, so everyone's consensus is that they are set by default, so everyone's consensus is “We don't support big blocks,” even though this consensus is actually wrong. This is why the pre-0.8 code is buggy... this is also a flaw in the Bitcoin code implementation of a single type.

The 0.8 code solved this problem. After upgrading to LevelDB, it can support larger blocks and more transactions. Well, the problem is simple; just upgrade together. In fact, about 40% of people have upgraded, and 60% haven't upgraded or are in the process of upgrading. At this moment, a large block was unearthed. 40% said it was OK, we accepted it, and 60% said we didn't understand it and couldn't handle it.

So it was forked, hard fork.

What is completely different from today's mainstream opinion is how to deal with this hard fork when it occurs. Marek Palatinus (Slush) and Michael Marsee (Eleuthria of BTCGuild) rolled back their mining rigs. They were the two main miners out of 40% of the upgraded miners at the time, and they sacrificed their interests to guarantee a network-wide consensus (although this consensus was wrong, oh, another joke, is this really similar to ETH vs. ETC?). Then there was a bunch of urgent puddings, and everyone was notified that an upgrade was needed, and only a few months later was the second successful upgrade. Since space is limited today, I won't go into detail about the specific steps.

The second example relates to transactions.

Bitcoin 2010 event https://nvd.nist.gov/vuln/detail/CVE-2010-5139..

Compared to the first example, this one is less brain-burning.

In 2010, block 74638 included a very powerful transaction. Any Bitcoin transaction had input and output (input and output). This transaction input 0.5 bitcoins, but the output was 184 billion bitcoins, or 184 billion bitcoins (the people who collected the money got rich, a total of two addresses).

You read that right, it's 184 billion BTC!

The reason is that the hacker found a bug, and the code for checking transaction records overflowed when processing huge output values, so such a large transaction should have been rejected, but instead accepted.

A few hours later, the developers released a patch. After 53 blocks, the new blockchain replaced the “bad” blockchain.

You can think about the common characteristics of the tests of operation and maintenance in these two completely different examples.

Code cannot be bug-free, consensus is not immutable, and various factors in the environment and system are constantly changing.

Establishing a more effective operation and maintenance system, recognizing problems earlier, and solving problems faster is the last word.

Finally, I would like to mention feedback and communication. You need to have regular channels, and at the same time avoid a single point of failure. For example, you should not combine the network required for transactions and the network used for feedback into one. For example, they all use AWS at the bottom. When AWS fails, you don't even have a channel to report errors.

That's all for today's sharing. If you have any questions, we can discuss them together.

Special thanks to the sponsor and organizer Soteria Community, the Magic Piper Technology Development Community, the community that participated in this live broadcast (in no particular order), and all the friends who participated in the discussion and attention.

- Magic Piper Technology Development Community

- Soteria hardcore tech community

- Soteria SSDE development community

- Digital Everything Discussion Group

- Unnamed Highland of the New Token Economy

- Fire Thieves Blockchain Application Alliance

- July offline online exchange and learning community

- Cryptocurrency and blockchain ecosystem

- Defi Research Institute Chengdu

- Metamask Chinese Community

- Quantum Computing Artificial Intelligence Blockchain 00003

- Western Electric Blockchain Interest Group

(To be continued)