Claire: On August 8, we shared a topic entitled “DAG's Past and Present Life (1)”, which received enthusiastic attention and discussion from many friends, and many of our peers joined the “Magic Piper Community” as a result. Today, the keynote speaker, Mr. Wu Min, will continue to share the theme of “DAG's Past and Present Life (2)” and further explain more technical details, difficulties, solutions and tools of BlockDAG, so that everyone can have a deeper understanding of BlockDAG. We welcome everyone to ask questions and discuss them in depth in the Q&A section later. Now let's ask Mr. Wu Min...

min:Hello everyone, I'm Min Wu, and today's topic sharing is explaining how to scale from blockchain to block diagram.The Github link is: https://github.com/soteria-dag/soterd

Today's sharing is mainly aimed at developers; there are quite a few programmers who understand slang (black language). I try to share it in a way that is easy to understand, so that everyone can benefit.

In the last issue, Ming Guo talked about “DAG's Past and Present Life (1)”,The link is here, https://news.huoxing24.com/20190812204900103888.html，我们快速的回顾一下.

1: Soteria is an infrastructure technology for an “endogenous decentralized economy” (SSDE - Self Sustainable Decentralized Economy).

2: Soteria is developing an overall solution to address some of the pressing problems of the current generation of blockchains while providing an adequate feature set for its anticipated decentralized economy — such as scalable throughput based on block graphs (BlockDAG).

3: DAG is called a directed acyclic map.

4: BlockDAG is called a block chart. There is a huge difference between BlockDAG and TransactionDAG. For details, see the section on transaction DAG in the previous sharing.

5: Soteria DAG is actually an extension of Bitcoin's Satoshi Nakamoto consensus algorithm, making it inclusive and providing flexible, elastic and expandable features on the basis of guaranteed security.

The block chart looks like this:

In this example. At the bottom is the GENESIS (GENESIS) block, and the arrow points from one block to its parent block (parent). If a block is not a parent of any other block, it (s) are called tips for the current block chart. For example, the tips in the image above are M, J, and L. Genesis, parent block (parent), and tips are all very important concepts, and we'll mention them over and over again later.

Under the premise of inclusiveness, how can safety be guaranteed? For example, what about malicious mining? Here we'll talk about Phantom and Greedy Phantom.

What is Phantom?

Phantom is a staining/sorting protocol invented by Yonatan Sompolinsky and Aviv Zohar. Its implementation consists of the following three main steps:

1: Identifies a set of well-connected blocks. This is called the blue set (set). Other blocks, such as those that only reference older blocks, or blocks that have been kept private for a while, will most likely be excluded from the blue set.

2: The block chart is topologically ordered, giving priority to blocks in the blue collection, and lagging behind blocks not in the blue set.

3: When sorting and checking transactions, the order of the block chart will be used. Legal transactions will be adopted and malicious transactions will be discarded.

Let's see what the colors of this graphic might look like when using Phantom or Greedy Phantom.

The image above shows the results after dyeing with Phantom (Greedy Phantom is similar). We looked at how we dyed it from the perspective of block M. In the middle part of the block chart, there is a collection of well-connected (well-connected) blocks; at the same time, some blocks (red) at the boundary of the block chart are not that well connected.

We used an internal system constant (k) to make this decision, and we'll explain how k was selected in a later technical share. This means that blocks in the red collection at the same level will be placed behind the blue collection block. Note that not all red blocks are behind the blue block. Blocks without color (J and L) are not part of M's “past set”; they are sorted after M.

The sorting results (from M's perspective) are:

GENESIS, C, D, E, H, B, I, K, F, M, J, L

In the process of dyeing and sorting, Phantom requires:

1: Determine whether a block is fully connected (well-connected) by analyzing the past (past) and future (future) of a block.

2: Go through the block chart and color each block.

3: Sorting.

Let's take a look at what Greedy Phantom is, by Aviv Yaish:

1: Basically the same concept as Phantom, Greedy Phantom aims to be simpler and more effective.

2: The dyeing part is inherited from the bluest parent (bluest parent).

3: Dyeing and sorting are incremental.

The dyeing part is inherited from the bluest parent (bluest parent), and the bluest parent is also known as the colouring parent (colouring parent). The collection of dyed parents will continue from the current block to the genesis block (genesis), which is also called the current block's colouring chain (colouring chain).

What is different from Phantom is that Greedy Phantom's coloring is inherited, and the order of the blocks before the bluest parent remains the same. In the Phantom algorithm, since every block needs to consider the past (past) and future (future), the color of each block is likely to change in the future, entirely depending on how connected the block is in the future.

Let's take a look at Greedy Phantom's dyeing process as blocks are added to the block chart.

Animated map

In this animation, in addition to the blue, red, and transparent blocks, there are also blocks with green and dotted borders.

Green: represents the dyeing tip

Dotted line: The current block's dyeing chain. (The bluest collection of parents)

Blue: A collection of blue blocks, from the perspective of the latest blocks (fully connected, well-connected)

Red: Blocks outside of the blue block collection, from the perspective of the most recent block (not very well connected)

What can be observed is that from the perspective of the latest blocks, some blocks change color (between red and blue). What can also be observed is that when the angle is different, the dyeing chain is also different.

At the beginning of DAG, when B, C, D, and E were added, the dyeing chain did not become these new blocks. This is because of an internal rule that breaks the game, when two or more blocks have the same level of blue (varying blue), the block with the lower hash value wins. In this example, B is the first (lowest) in the letter, so B wins.

Summarize the computational differences between Phantom and Greedy Phantom (affecting engineering practices):

Phantom uses the past (past) and future (future) of a block when dyeing. As the block graph grows, the future of a block will grow infinitely.

Greedy Phantom will only consider the past (past) of a block during dyeing, as the block chart continues to grow, making Greedy Phantom a better choice for dyeing/sequencing in engineering implementation.

If you don't want to use Phantom, you can achieve similar effects by setting the lower and upper limits of the node's past and future traversal

However, limiting the past and future of the node may affect the dyeing effect, so it's not ideal.

Our codebase includes implementations of the two, with the goal of comparing their differences in the future and providing more lessons for other developers.

Next, I will explain in detail how we can modify the code to achieve the purpose of expanding the blockchain as a block chart.

Before starting the project, we examined a number of blockchain projects that have already been open source, and finally decided to iterate at the upper level based on BTCD. The new project is called soterd.The Github link is: https://github.com/soteria-dag/soterd

The engineering implementation block diagram (blockDAG) includes the following content:

Trim the existing code, remove the code that only serves the blockchain scenario, and also includes a lot of code that complicates block graph (blockDAG) development and doesn't help much.

Implement the core module, representing the block diagram (the old code represents the blockchain), and provides the same or similar service functions as the blockchain module, such as:

The function to find tips;

Search for parent blocks based on existing blocks;

Retrieves a specified range of block charts.

The block data structure has been updated (this is the most important data structure in all blockchain projects).

A region has been added to store more parent information, because in the block chart, each block can have multiple parents (Bitcoin can only have one more).

Updated protocol layer encoding and analysis methods (encoding/decoding)

Updated and added tests (tests)

The mining program has been updated, and there are other places where it is possible to traverse the blockchain/block chart:

Added new P2P and RPC calls

getdagtips rpc call

renderdag rpc call

addrcache P2P call

Updated the code for the network sync section

System integration tests have been added,

Generate a block diagram (originally blockchain) and confirm that each block can have more than one parent block.

Confirm that the block chart can be synchronized across different full nodes.

Confirm that SOTO that has been mined can be used to trade.

(Actually, it's a bunch of long words. People who don't write programs only need to remember that changing the public chain code is not an easy task; it's better for programmers. :P... programmer friend, see our GitHub for details)

The above changes were carried out through several different stages, mainly because the amount of change at one time was too large and the risk was too high. We've also developed a number of tools to help understand+analyze block diagrams and troubleshoot. The tool will be shared later.

I know everyone has been watching the live text broadcast for so long, and it's very hard. One big question in their mind is, “A picture is worth a thousand words.” Editor, what about your picture???

The picture is coming soon.

OK, let's continue...

The challenges encountered:

During the development of the project, many blockchain codes need to be updated to support block diagrams. Generally, there are three different types of challenges:

Blockchain assumptions.

In the existing code, due to differences in system design, many places assume blockchain by default.

Peer-to-Peer (P2P) sync.

Protocol layer coding and parsing.

Blockchain assumptions:

Some code needs to be restructured. For example, in blockchain, the block height (height) is how many blocks are in the middle of the current block distance from the genesis block (genesis) block. In a block graph, a block can have multiple parent blocks, some of which may be more directly linked to Genesis than others. So in the block chart, the height of the block is now the maximum height of the parent block + 1.

In the image above, the height of the K block is 2 if it passes through the path B; if it passes through C, D, and E, it is 3. So we think its height is 3.

Handling Orphan Blocks:

Z-shaped dag

An isolated block looks like this. When a full node receives a reasonable block, but cannot find the block's past (past), so it can't connect to the existing block chart, this block is called an isolated block.

In blockchain, lone blocks are processed from top to bottom, starting with the lone block, and then the lone block's parents. There is a problem with this traversal method in the block chart. In particular, there is a Z shape (z-saped dag) in the block chart, and some blocks will be ignored. We've adjusted this algorithm to use depth-first ranking of search results to include all blocks.

(I know this passage seems very brain-burning; you just think, how pitiful orphans are; of course, they should be treated differently)

Duplicate issues (knots) in solitary block processing

For parts of the block chart with high connectivity between blocks, processing an isolated block will take longer because of each path

The connections will all be revisited. For this purpose, we have set up a cache to record the solitary blocks that have been processed and their past (past).

(In other words, too many special treatment for orphans will also have an impact on the system)

TOOLS:

Next, we will introduce some development tools. We have added a coloring function to Soterd's code for the purpose:

1: It helps to check the implementation or synchronization problems of the block chart. Being able to intuitively see the block chart is much more useful than only seeing a set of hash values in the log file.

2: Provide an interface for upper level applications (such as browsers).

dagviz:

Dagviz:https://github.com/soteria-dag/soterd/tree/master/cmd/dagviz 

Dagviz is a command line tool

It will open up a few soterd full nodes (yes, full nodes), let them mine and exchange blocks, and record the results step by step, and you can replay them (via browser). Each block is color-coded according to its miner, so it's very intuitive to see each miner's contribution to the network. Dagviz can be used to display soterd and block charts, and can also be used to measure the impact of changes in the algorithm on how fair mining is.

min:

(Remember, dagviz will let a few full nodes run on your computer, so the CPU requirements are not low)

dagparam Related links dagparam https://github.com/soteria-dag/soterd/tree/master/cmd/dagparam Dagparam is a tool for exploring different peer-to-peer network parameters. For example, targetTimeSpan and targetTimePerBlock, and how these parameters affect the composition and propagation of the block graph.

Dagparam is a tool for exploring different peer-to-peer network parameters. For example, targetTimeSpan and targetTimePerBlock, and how these parameters affect the composition and propagation of the block graph.

For example, when the block generation speed exceeds 62.5 blocks/second, that is, every 16 ms, the synchronized code will have unstable performance.

Soterdash https://github.com/soteria-dag/soterdash Soterdash is a web UI for browsing block diagrams and all nodes in the Soterd network. It can display all or part of the block chart. Clicking on each block will provide more detailed information, such as Header, MerkleRoot, etc. Soterdash helped us a lot during development, especially when troubleshooting issues.

Soterdash is a web UI for browsing block diagrams and all nodes in the Soterd network. It can display all or part of the block chart. Clicking on each block will provide more detailed information, such as Header, MerkleRoot, etc. Soterdash helped us a lot during development, especially when troubleshooting issues.

This is the last tool; the last one must be the best.

Go Repo Sync

The vast majority of our codebase is written in go. Go doesn't allow relative import paths, which makes sharing code between different fork/copy codebases a very complicated process. Our development process involved multiple versions and code sharing in different directions, so we developed this tool ourselves to help synchronize between different codebases. It automatically transfers the changed parts to the target codebase, and at the same time makes necessary changes. For example, the original references to the old codebase will be replaced with new ones. The specific steps are as follows:

- Process the current project and the predecessor projects on which it depends.

- Copy the current project and target project to a temporary area.

- Use git archive to synchronize current and target projects.

- Use git rm to remove unneeded files in the target project.

- Replace the referenced files and links.

- Update go's module name.

- Add the new files needed to the target project.

- Submit locally

- Commit to the target git tree.

- This tool is now in a private repository (private repo), but we can provide open source as needed. This project can be helpful to the vast majority of go programmers.

(Again, not only blockchain programmers, all go programmers are helpful. Of course, blockchain programmers also have many that don't need go)

Thank you all for participating in our sharing today. Technology has grown in the process of continuous discussion. That's all for today's sharing. Here's the Q&A session, please ask questions.

Here are some highlights from the Q&A:

seabook: @min @soteria sharing today is very in-depth, what is your economic model?

min: Oh, that's what we'll share next:

SSDE (SELF SUSTAINABLE DECENTRALIZED ECONOMY)... you read my mind.

seabook: Please give me a quick hint.

min:https://www.ssde.io/It is now in English, and the content shared will be in Chinese. We've also recorded two episodes of the podcast dedicated to talking about UBI.

seabook:https://medium.com/bitcoin-is-alien-technology/https-medium-com-bitcoin-is-alien-technology-bitcoin-is-alien-technology-part-1-7c5c82faf552 

@min @soteria I enjoy reading your article.

Zhu Jiang²2.2:seabook, is the economic model you mentioned a macro model or a microscopic model Macroscopic, as @min @soteria mentioned, our SSDE proposal microscopically relates to the ecology of Soteria DAG itself, such as reward mechanism incentive design, etc., which we'll cover in the technical sharing later.

Seabook: incentive design

Zhujiang²†:seabook, SSDE, and Soteria are neither UBI nor communism; they still advocate a spirit of struggle and are distributed according to work. It's just a few decentralized enhancements to the existing centralized economic system: fairness, inclusiveness, security, privacy, and scalability. Incentive design, are you also a game theory enthusiast? Keep an eye out for what we'll share later.

seabook: Will be watching.

Claire: Thank you so much for your attention, thank you to all the broadcast platforms, and welcome to continue to discuss Blockdag's technology in depth later. This is the end of this topic sharing. After sharing two very hardcore topics, we will continue to share with you other topics closely related to our lives, including the economy, finance, privacy protection, big data analysis, business models of blockchain projects, etc., so stay tuned for more topics to share... Thanks again to our guest speaker, Mr. Wu Min, for his careful preparation and meticulous sharing!

Special thanks to the community and media for sharing this theme(In no particular order):

 Magic Piper Technology Development Community

 The unnamed highland of the new token economy

 Digital Everything Discussion Group

 International community&ATOS atomic blockchain

 MetamaskChinese community

 Fire Thieves Blockchain Application Alliance

 ⑦ Monthly offline online community

 Computing power community1901

 Cryptocurrency and Blockchain Ecosystems

 Tsinghua alumni blockchain technology explorers

 DefiResearch Institute Chengdu

 Dragon Balls scattered all over the world