In Hyperledger fabric consensus is acheived by orderers but on what basis each transaction is ordered and How many orderers are present in distributed network if there are more than one again how ordering done by each orderers are identical
on what basis each transaction is ordered
Each transaction is ordered by an ordering service node, where it uses its internal implementation-specific consensus mechanism.
Hyperledger Fabric v1.0 comes with 2 ordering service implementations:
Solo orderer -- mainly used for development and testing; has 1 node and simply batches the transactions into blocks
Kafka based orderer -- the orderer nodes send the transactions to a kafka queue from which all orderer nodes "pull" the transactions in the same order, and then cut the same blocks; each orderer sends into the queue a "time to cut a block" message, and then when the first message reaches the queue, a block is cut by all ordering service nodes
In the future there will probably be a Byzantine tolerant implementation of the ordering service based on the sBFT (simplified pBFT) algorithm.
Related
Running a test node in GCP, using Docker 9.9.4, Ubuntu, Postgres db, Infura. I had issues with public/private IP, but once I cleared that up my node is up and running. I am now throwing the error below repeatedly, potentially due to the blockchain connection. How do I fix this?
[ERROR] HeadTracker: dropping head 26085153 with hash 0xf50e19099b7e343829935d70dd7d86c5bc0398286b7a4e4f32ac033ac60c3733 because queue is full. WARNING: Your node is overloaded and may start missing jobs. logger/default.go:155 stacktrace=github.com/smartcontractkit/chainlink/core/logger.Errorf
This log output is related to an overload of your blockchain connection.
This notification is usually related to the usage of public websocket connections and/or free third party NaaS Provider. To fix this connection issue you can either run an own full node or change the tier or the third party NaaS provider. Also it is recommended to use Chainlink version 0.10.8 or higher, as the HeadTracker has been revised here and performs more efficient.
In regard to the question let me try to give you a small technical overview, which may clarify the payload of a Chainlink node to it's remote full node:
Your Chainlink node establishes a connection to a full node. There the Chainlink node initiates various subscriptions, which are a special feature of the websocket protocol to enable bidirectional communication. More precisely, this means that the Chainlink node is informed if a certain "state" of the subscription changes. Basically, the node interacts with using JSON-RPC methods and uses the following methods to initiate and process various functions internally:
eth_getBlockByNumber,eth_getBalance,eth_getTransactionReceipt,eth_getTransactionCount,eth_getLogs,eth_subscribe,eth_unsubscribe,eth_sendRawTransaction and eth_Call
https://ethereum.org/uk/developers/docs/apis/json-rpc/
The high amount of interactions of the Chainlink node are especially executed during the syncing process via the internal HeadTracker service. This service initiates a "head" subscription in order to interact with every single incoming new blockheader.
During this syncing process it uses the JSON-RPC methods eth_GetBlockByNumber and eth_getBalance to get all the necessary information from the block. So these two methods are used/ executed every block. The number of requests now depends on the average blocktime of the network the Chainlink node is connected to
An example would be the Kovan Testnet:
The avg. blocktime here is 6.7sec, which means you get a daily request number of approx. 21.000
During fulfilling job requests, those request also includes following methods: eth_getTransactionReceipt, eth_sendRawTransaction, eth_getLogs, eth_subscribe, eth_unsubscribe, eth_getTransactionCount and eth_call, which increases the total number significantly depending on the number of job requests.
It should also be noted that especially with faster blockchains (e.g. polygon) there is a very high payload of the WebSocket and you have to deal with a good full node connection in detail, as many full nodes do not receive such a high number of requests permanently.
I am trying to understand how PBFT(practical byzantine fault tolerance) applied in block chain. After reading paper, I found that process for PBFT to reach a consensus is like below:
A client sends a request to invoke a service operation to the primary
The primary multicasts the request to the backups
Replicas execute the request and send a reply to the client
The client waits for f + 1 replies from different replicas with the same result; this is the result of the operation.
This is how I understand how it is applied in block chain:
At first, the elected primary node wants to write transaction A to chain, it will broadcast transaction A to other nodes.
Any node receives the transaction checks if the transaction legal. If the transaction is thought as legal, the node will broadcast a legal signal to all of nodes in this round of consensus.
Any node that receives equal or greater than f + 1 responds will write the transaction to the its own chain.
Here are my questions:
For malfunctioned nodes, if they keep failing to write block into its chain, they will hold a different chains with healthy node. In next consensus, the existing chain will be picked up at first. How do nodes know which one is the correct chain?
In step 1, the elected node send transaction to other nodes. Does "other nodes" means all nodes in the network? How to make sure if all nodes included in the consensus because there is not a centralized agency.
How do nodes know which one is the correct chain?
For tolerating Byzantine faulty nodes, It needs at least 3f+1 nodes in the network. PBFT is one of the algorithms which can tolerate Byzantine failure. So PBFT can tolerate up to f Byzantine nodes.
f number of malicious nodes can be tolerated if you use PBFT. If there are f number of malicious nodes which keep failing to write block into its chain, resulting in inconsistency with correct nodes, then one can figure that the same chains from rest 2f + 1 nodes are correct. (Correct nodes always output exactly same data to the same request in same order).
Does "other nodes" means all nodes in the network? How to make sure if all nodes included in the consensus because there is not a centralized agency.
In PBFT setup, identities of all nodes should be established. To do that, there should be central authority to determine whether a node can join the network or not. (Important: central authority only engages in identity management, not the algorithm itself)
Why this is needed? It's because PBFT works by voting mechanism and voting is not secure when anyone (including malicious node) can join the network. For example, a proposed value by the primary only can be recorded to all nodes in the way of state machine replication, which it means that there needs at least 2f + 1 agreed matching messages for the value to be accepted to the correct nodes.
Without the trusted identity management, Sybil attack is possible. And this is the main reason why PBFT is not for the open blockchain which allows any node can freely join or leave the network.
I'm going through the documentation on Hyperledger Fabric, and there's a portion in the introduction that writes:
"Hyperledger Fabric assigns network roles by node type. To provide concurrency and parallelism to the network, transaction execution is separated from transaction ordering and commitment. Executing transactions prior to ordering them enables each peer node to process multiple transactions simultaneously."
Maybe this is some fundamental misunderstanding on my part about the structure of the network and the different functions and mechanics of nodes, but how can one execute a transaction prior to ordering one?
Within a Hyperledger Fabric network, transactions start out with client applications sending transaction proposals, or, in other words, proposing a transaction to endorsing peers.
Each endorsing peer simulates the proposed transaction, without updating the ledger. The endorsing peers will capture the set of Read and Written data, called RW Sets.
These RW sets are then signed by the endorsing peer, and returned to the client application to be used in future steps of the transaction flow. Endorsing peers must hold smart contracts in order to simulate the transaction proposals.
The application then submits the endorsed transaction and the RW sets to the ordering service. Ordering happens across the network, in parallel with endorsed transactions and RW sets submitted by other applications.
The ordering service takes the endorsed transactions and RW sets, orders this information into a block, and delivers the block to all committing peers.
The committing peer validates the transaction by checking to make sure that the RW sets still match the current world state.
Committing peers are responsible for adding blocks of transactions to the shared ledger and updating the world state.
Here, you have the roles that there are in Hyperledger Fabric:
Endorser Peers: they receive a transaction. Then, they execute the transaction against the Smart Contrat and they sign the result. They send the transaction signed to the peer that has sent it.
Committer Peers: the Peers get the Blocks (with the validates transactions) and commit them to its ledger.
Orderes: nodes that sort the transactions and generate the blocks.
I take this info from an answer that I write for a question.
How can adding a new orderer download the ledger as ordering nodes are not connected with each other and kafka keeps messages only for 7 days.
And also if I shut down a orderer node for more than 7 days and if I bring it up again then it will not find the transactions that happened in those 7 days in kafka partition therefore how will it sync and update it's local ledger.
In 1.0, Kafka brokers are to be set with log.retention.ms = -1 (source: documentation, Step 4e).
This disables time-based retention and prevents segments from expiring. This means that:
A partition hosts the entire transaction history of the channel.
A new orderer service node (OSN) can be added at any point in time and use the Kafka brokers to sync up with all channels in their entirety.
In a minor release within the 1.x track we will support ledger pruning for the OSNs. This means that the brokers will only need to maintain a pruned sequence of the transaction history (that will always start from a configuration block), and any new OSN will only be able to sync back to that configuration block.
I have been reading the documentation on how HyperLedger Fabric's project is implementing a open source BlockChain solution: https://github.com/hyperledger/fabric/blob/master/docs/protocol-spec.md
I have seen that PBFT consensus algorithm is used, but I do not understand how blocks are mined and shared among all Validating Peers in the BlockChain network.
Hyperledger Validating Peers (VPs) do not mine blocks and do not share the blocks between them. Here is how it works:
A transaction is send to one trusted VP.
The VP broadcasts the transaction to all other VPs.
All VPs reach consensus (using PBFT algorithm) on the order to follow to execute the transactions.
All VPs execute the transactions "on their own" following the total order and build a block (calculating hashes mainly) with the executed transactions.
All the blocks will be the same because: the transaction execution is deterministic (should be) and the number of tx in a block is fixed.
According to Hyperledger Fabric 1.X
User through Client SDK send the transaction proposal to Endorsing Peers.
Endorsing Peer check the transaction and make endorsement proposal of transaction(with read/write set (previous value/Changed value)) and send to again client SDK.
Client SDK wait for all endorsement, once it get all endorsement proposal it make one invocation request and send to Orderer.
Orderer verify invocation request rent by client SDK by checking Policies defined(Consensus), verify the transaction and add to the block.
According to configuration defined for block, after specified time or number of transaction it form a Hash of block by using transaction hash, metadata and previous block hash.
The blocks of transactions are “delivered” to all peers on the channel by the Orderer.
All committing peers verify the endorsing policy and ensure that there have been no changes to ledger state for read set variables since the read set was generated by the transaction execution. After this all the transactions in the block and update the ledger with new block and current state of asset.
Ledger Contains
1) Current state Database(Level BD or Couch DB)
2) Blockchain(Files)(Linked blocks)
Read the transaction flow of hyperledger fabric
Check image for reference
Hyperledger is an umbrella of blockchain technologies. Hyperledger Fabric, mentioned above, is one of them. Hyperledger Sawtooth also does not use mining and adds these consensus algorithms:
PoET Proof of Elapsed Time (optional Nakamoto-style consensus algorithm used for Sawtooth). PoET with SGX has BFT. PoET Simulator has CFT. Not CPU-intensive as with PoW-style algorithms, although it still can fork and have stale blocks . See PoET specification at https://sawtooth.hyperledger.org/docs/core/release s/latest/architecture/poet.html
RAFT Consensus algorithm that elects a leader for a term of arbitrary time. Leader replaced if it times-out. Raft is faster than PoET, but is not BFT (Raft is CFT). Also Raft does not fork.
With unpluggable consensus, another consensus algorithm can be changed without reinitializing the blockchain or even restarting the software.
For completeness, the original consensus algorithm with bitcoin (and does use mining) is:
PoW Proof of Work. Completing work (CPU-intensive Nakamoto-style consensus algorithm). Usually used in permissionless blockchains