Let's consider a scenario: suppose 2 miners pick up 2 different unconfirmed transactions and they complete the proof of work. If miner A adds the block first to blockChain then the block produced by miner B be will be wrong/useless because the last_Hash he used is now wrong and he has to use the last_hash of miner A that is added right a second before he was going to add his block.
e.g.
<b>current chian: block1--block2--block3</b>
<table>
<tr>
<td>Miner A </td>
<td>Miner B</td>
</tr>
<tr>
<td>LastHast=block3-hash<br>
currentHast=SHA(block3's_hash,timestemp,data,nonce) <br>
proof of work <br>
add---- Successfully add block4 <br>
</td>
<td>LastHast=block3-hash<br>
currentHast=SHA(block3's_hash,timestemp,data,nonce) <br>
proof of work <br>
add---- Fail because the last hash is now block4's not block3's So currentHash is wrong so the proof of work<br>
</td>
</tr>
</table>
**
**
Double spending is avoided by having other miners verify the proposed change on the ledger.
In your case two miners verified the transaction, published the result to the network and depending on your protocol this might be fine so the change on the ledger will take place.
Apart from that a timestamp server that ensures that you will always build on top of a unique chain of blocks.
The valid transactions are always on the longest chain, by longest, I mean the block number is bigger.
If minerA and minerB finished PoW at the same time, a fork is made
At the current time, both minerA and minerB transaction is stored on the fullnode, and the fullnode is waiting for the next block to be produced.
During the next block time, minerA finished PoW first, then the fork (data chain) on minerA is considered longer, thus minerA become the longest chain and considered valid, block on minerB is considered non-valid, and discared.
First of all, this is happening all the time, and the protocol is built to deal with it.
Let’s assume 2 miners, A and B, that find the PoW for a different set of transactions at the same time.
The next thing this miners will do is to broadcast their newly found blocks over the network.
This broadcast is not instantaneous (light speed is not instantaneous) and so miners near A will receive first the broadcast from A, and miners near B will receive first the broadcast from B.
At that instant, A neighbours will start working on a new block on top of the A block, while B neighbours will start working in a new block on top of the B block.
The networked has just forked, part of the network is working on a blockchain that ends on block A, other part is working on a blockchain that ends on block B.
Let’s assume now 2 new miners, C and D. C is a A neighbour and is working on a new block on top of the A block. D is a B neighbour and is working on a new block on top of the B block.
Imagine that C finds the PoW for a new block first than D. C will broadcast this block to the network. Eventually, D will receive this C block and will realize that the blockchain X -> A -> C has more PoW then the one its working on, X -> B.
By the protocol rules, D will discard immediately what it was working on, and will start to work on the PoW for a new block on top of X -> A -> C, thus resolving the network fork.
Related
Intermittently we are receiving following errors
2022-05-25 08:32:30,691 ERROR app=abc a.c.s.DDataShardCoordinator - The ShardCoordinator was unable to update a distributed state within ‘updating-state-timeout’: 2000 millis (retrying). Perhaps the ShardRegion has not started on all active nodes yet? event=ShardRegionRegistered(Actor[akka://application#10.52.174.4:25520/system/sharding/abcapp#-1665332307])
2022-05-25 08:32:31,348 WARN app=abc a.c.s.ShardRegion - abcapp: Trying to register to coordinator at [ActorSelection[Anchor(akka://application#10.52.103.132:25520/), Path(/system/sharding/abcappCoordinator/singleton/coordinator)]], but no acknowledgement. Total [22] buffered messages. [Coordinator [Member(address = akka://application#10.52.103.132:25520, status = Up)] is reachable.]
While we check cluster members by using /cluster/members we got “10.52.174.4:25520” this as
{
“node”: “akka://application#10.52.252.4:25520”,
“nodeUid”: “7353086881718190138”,
“roles”: [
“dc-default”
],
“status”: “Up”
},
Which says its healthy but problem resolves while we remove this node from the cluster using
/cluster/members/{address} (leave operation to remove 10.52.252.4 from cluster, once it’s removed cluster will create new pod and rebalance.
Need help to understand the best way of handling this error.
Thanks
You can of course implement an external control plane to parse logs and take a node exhibiting this error out of the cluster.
That said, it's better to understand what's happening here. The ShardCoordinator runs on the oldest node in the cluster, and needs to ensure that there's agreement on things like which nodes own which shards. It accomplishes this by requiring that updates be acknowledged by a majority of nodes in the cluster. If a state update isn't acknowledged, then further updates to the state (e.g. rebalances) are delayed.
I said "majority", but because in clusters where there's substantial node turnover relative to the size of the cluster simple majorities can lead to data loss, it becomes more complex. Consider a cluster of 3 nodes, N1, N2, N3. N1 (the ShardCoordinator) updates state and considers it successful when it and N3 have updated state. N1 is dropped from the cluster and replaced by N4; N2 becomes the shard coordinator (being the next oldest node) and requests state from itself and the other nodes; N4 responds first. The result becomes that the state update N1 made is lost. So two other settings come into play:
akka.cluster.coordinator-state.write-majority-plus (default 3) which adds that to the majority write requirement (rounding down)
akka.cluster.distributed-data.majority-min-cap (default 5) which requires that the majority plus the added nodes must be at least this
If the computed majority is greater than the number of nodes, the majority becomes all nodes. So in a cluster with fewer than 9 nodes with the defaults these become effectively all nodes (and the actual timeout when updating is a quarter of the configured timeout, to allow for three retries).
You don't say what your cluster size is, but if running in a cluster with fewer than 9 nodes, it can be a good idea to increase the akka.cluster.sharding.updating-state-timeout from the default 5 seconds to allow for the increased consistency level. Decreasing write-majority-plus and majority-min-cap can be an option, if you're willing to take the risks of violating cluster sharding's guarantees (e.g. multiple instances of the same entity running and potentially destroying their persistent state). Increasing the cluster size can also be helpful, paradoxically, if the reason other nodes are slow to respond is overload.
I am reading the following article about PoS
http://earlz.net/view/2017/07/27/1904/the-missing-explanation-of-proof-of-stake-version
In this article, the author says
The block hash must be signed by the public key in the staking transaction's second vout
Why is this necessary? When using PoS, because the coin stake transaction's input comes from the miner, so the miner already provides an unlocking script on the inputs of the coin stake transaction. Why does she need to sign the block?
A reference implementation can be found in PIVX:
class CBlock : public CBlockHeader
{
public:
// network and disk
std::vector<CTransaction> vtx;
// ppcoin: block signature - signed by one of the coin base txout[N]'s owner
std::vector<unsigned char> vchBlockSig;
where vchBlockSig stores the signature which is signed by
key.Sign(block.GetHash(), block.vchBlockSig)
In the PoW systems, block signing is not needed, because of block ID id generating by hashing root of Merkle Tree (Merkle Root) of payload transactions and nonce, until hash become less than target.
If do analogous approach in PoS system, them malicious minter can generate lot of attempts with different output hashes from a same kernel UTXO (transaction output, which mints coins) - just by modify nonce and/or rearrange transaction in Merkle Tree, there is lot of combinations. And by this way, he can reduce PoS to PoW (lot of hashing attempts with same data).
To prevent such degradation, PoS cryptos (PPC, EMC, etc) limits number of attempts for some specific UTXO. And hash result (which compared to a target) depends only from kernel UTXO and from current time, and independent from nonce, block payload, and so. As result, PoS minter can make a single attempt for each matured UTXO only once per second.
But, with this approach, block content is not participate in the kernel hash, which compares to target.
As result, if minter does not sign a block, malicious actor can do following attack: He can intercept from the network a freshly-minted block from a minter, modify payload transactions and Merkle Tree and block hash (for example, add double spend TX), and redistribute the modified block over the network. Such block will contains valid coinstake transaction (which spends kernel UTXO), and will be accepted by network nodes.
To prevent this "modify freshly-minted block on the fly", block is signed by address of kernel UTXO. By this signature, minter provides a proof: Block is created by same minter, who generates a coinstake TX.
Thus, with PoS, block generation is following:
Find appropriate kernel UTXO
Generate coinstake transaction, which sends coins from kernel UTXO address to itself.
Create a new block,contains this coinstake TX and payload TXes
Sign this block with coin address of kernel UTXO.
Practically, enough to sign just a header, contains Merkle Root.
I'm trying to play with Kafka Stream to aggregate some attribute of People.
I have a kafka stream test like this :
new ConsumerRecordFactory[Array[Byte], Character]("input", new ByteArraySerializer(), new CharacterSerializer())
var i = 0
while (i != 5) {
testDriver.pipeInput(
factory.create("input",
Character(123,12), 15*10000L))
i+=1;
}
val output = testDriver.readOutput....
I'm trying to group the value by key like this :
streamBuilder.stream[Array[Byte], Character](inputKafkaTopic)
.filter((key, _) => key == null )
.mapValues(character=> PersonInfos(character.id, character.id2, character.age) // case class
.groupBy((_, value) => CharacterInfos(value.id, value.id2) // case class)
.count().toStream.print(Printed.toSysOut[CharacterInfos, Long])
When i'm running the code, I got this :
[KTABLE-TOSTREAM-0000000012]: CharacterInfos(123,12), 1
[KTABLE-TOSTREAM-0000000012]: CharacterInfos(123,12), 2
[KTABLE-TOSTREAM-0000000012]: CharacterInfos(123,12), 3
[KTABLE-TOSTREAM-0000000012]: CharacterInfos(123,12), 4
[KTABLE-TOSTREAM-0000000012]: CharacterInfos(123,12), 5
Why i'm getting 5 rows instead of just one line with CharacterInfos and the count ?
Doesn't groupBy just change the key ?
If you use the TopologyTestDriver caching is effectively disabled and thus, every input record will always produce an output record. This is by design, because caching implies non-deterministic behavior what makes itsvery hard to write an actual unit test.
If you deploy the code in a real application, the behavior will be different and caching will reduce the output load -- which intermediate results you will get, is not defined (ie, non-deterministic); compare Michael Noll's answer.
For your unit test, it should actually not really matter, and you can either test for all output records (ie, all intermediate results), or put all output records into a key-value Map and only test for the last emitted record per key (if you don't care about the intermediate results) in the test.
Furthermore, you could use suppress() operator to get fine grained control over what output messages you get. suppress()—in contrast to caching—is fully deterministic and thus writing a unit test works well. However, note that suppress() is event-time driven, and thus, if you stop sending new records, time does not advance and suppress() does not emit data. For unit testing, this is important to consider, because you might need to send some additional "dummy" data to trigger the output you actually want to test for. For more details on suppress() check out this blog post: https://www.confluent.io/blog/kafka-streams-take-on-watermarks-and-triggers
Update: I didn't spot the line in the example code that refers to the TopologyTestDriver in Kafka Streams. My answer below is for the 'normal' KStreams application behavior, whereas the TopologyTestDriver behaves differently. See the answer by Matthias J. Sax for the latter.
This is expected behavior. Somewhat simplified, Kafka Streams emits by default a new output record as soon as a new input record was received.
When you are aggregating (here: counting) the input data, then the aggregation result will be updated (and thus a new output record produced) as soon as new input was received for the aggregation.
input record 1 ---> new output record with count=1
input record 2 ---> new output record with count=2
...
input record 5 ---> new output record with count=5
What to do about it: You can reduce the number of 'intermediate' outputs through configuring the size of the so-called record caches as well as the setting of the commit.interval.ms parameter. See Memory Management. However, how much reduction you will be seeing depends not only on these settings but also on the characteristics of your input data, and because of that the extent of the reduction may also vary over time (think: could be 90% in the first hour of data, 76% in the second hour of data, etc.). That is, the reduction process is deterministic but from the resulting reduction amount is difficult to predict from the outside.
Note: When doing windowed aggregations (like windowed counts) you can also use the Suppress() API so that the number of intermediate updates is not only reduced, but there will only ever be a single output per window. However, in your use case/code you the aggregation is not windowed, so cannot use the Suppress API.
To help you understand why the setup is this way: You must keep in mind that a streaming system generally operates on unbounded streams of data, which means the system doesn't know 'when it has received all the input data'. So even the term 'intermediate outputs' is actually misleading: at the time the second input record was received, for example, the system believes that the result of the (non-windowed) aggregation is '2' -- its the correct result to the best of its knowledge at this point in time. It cannot predict whether (or when) another input record might arrive.
For windowed aggregations (where Suppress is supported) this is a bit easier, because the window size defines a boundary for the input data of a given window. Here, the Suppress() API allows you to make a trade-off decision between better latency but with multiple outputs per window (default behavior, Suppress disabled) and longer latency but you'll get only a single output per window (Suppress enabled). In the latter case, if you have 1h windows, you will not see any output for a given window until 1h later, so to speak. For some use cases this is acceptable, for others it is not.
I create a round of processes in erlang and wish to measure the time that it took for the first message to pass throigh the network and the entire message series, each time the first node gets the message back it sends another one.
right now in the first node i have the following code:
receive
stop->
io:format("all processes stopped!~n"),
true;
start->
statistics(runtime),
Son!{number, 1},
msg(PID, Son, M, 1);
{_, M} ->
{Time1, _} = statistics(runtime),
io:format("The last message has arrived after ~p! ~n",[Time1*1000]),
Son!stop;
of course i start the statistics when sending the first message.
as you can see i use the Time_Since_Last_Call for the first message loop and wish to use the Total_Run_Time for the entire run, the problem is that Total_Run_Time is accumulative since i start the statistics for the first time.
The second thought i had in mind is using another process with 2 receive loops getting the times for each one adding them and printing but i'm sure that erlang can do better than this.
i guess the best method to solve this is somehow flush the Total_Run_Time, but i couldn't find how this could be done. any ideas how this can be tackled?
One way to measure round-trip times would be to send a timestamp along with each message. When the first node receives the message, it can then measure the round-trip time, calculating Total_Run_Time - Timestamp.
To calculate the total run time, I would memorize the first timestamp in the process state (or dictionary), and calculate the total run time when stopping the test.
Besides, given that you mention the network, are you sure that the CPU time (which is what statistics(runtime) calculates is what you're after? Perhaps, wall clock time would be more appropriate.
Well i have process a in my main component and process b in my other sub component(inmplemented in the main one).
both process a and b have only the clock in their sensitivity list:
process a control eneable signal called ready which if 1 process b can work , 0 process b will do nothing.
Problem is in process a , when process a changes value of enable signal to 0 , it has to take to the next clock cycle to change so process b ends up and run an extra clock cycle.
a:process(clk)
begin
if(rising_edge(clk)) then
if(output/=old_output) then
enable<='0';
end if;
end if;
end process;
b:process(clk)
begin
if(rising_edge(clk)) then
if(enable='1') then
--do anything
end if;
end if;
end process;
The reason is that the value is latched/sampled at the exact rising_edge of the clock. At that time, 'enable' is still equal to one. In that simulation delta, enabled will get the value zero, but it won't be available until AFTER the first delta.
This is also true for when enable BECOMES one (given that it is also generated on a rising clock edge), the process will latch the value exactly when clock rises, and in the simulator, enabled will look high for a whole clock period, even though "--do anything" will not happen.
You can think of this as real electrical circuits instead of a programming language. Consider that the evaluation of "output/=old_output" will consume time, and that you as a designer want that to be DONE before the next rising clock edge.
Hope this helps, but this is how the language works. I could give you a better answer if both the setting and resetting of the enable.