02-24-2023, 06:25 AM
After yesterday's outage, 25 out of 100 total Polygon validators are about to lose t
<!-- SC_OFF --><div class="md"><p>A 25 out of 100 total Polygon validators are about to lose their staking status due to the outages yesterday. In additional, the co-founder of Polygon stated that they are exploring replacing their entire <a href="https://twitter.com/mihailobjelic/status/1628926312076423170">Bor/Heimdall blockchain protocol with a single-blockchain solution</a>.</p> <p><strong>Yesterday after laying off 20% of its staff, several Polygon validators experienced sync issues, which then lead to giant cascade of outages for most validators</strong>. They ended up having to resync, often up to 12-24 hours of blocks. The <a href="https://www.coindesk.com/tech/2023/02/22/polygon-blockchain-suffers-apparent-outage/">block explorer outage</a> was a result of this cascade. The Polygon team has yet to announce the root cause of the issue, but it was attributed to a <a href="https://www.coindesk.com/tech/2023/02/22/polygon-blockchain-suffers-apparent-outage/">157-block reorg</a>. Their validators are able to correct for a 128-block reorg, but not anything higher. The reason for the reorg was <a href="https://forum.polygon.technology/t/157-block-reorg-at-block-height-39599624/11388">a bug between their Bor and Heimdall layers</a>.</p> <p>Back in Aug 2022, there was a governance proposal, <a href="https://forum.polygon.technology/t/pip-4-validator-performance-management/9956">PIP-4</a>, to move towards permissionless validation.</p> <blockquote> <p>"In its current state, the PoS validator network is not entirely permissionless in that the previously-selected set of validators has largely persisted since testnet."</p> </blockquote> <p>The proposal added an SLA requirement that 95% of median average of last 700 checkpoints are signed by the validator. If the validator fails to meet the SLA, they have another 700 checkpoints to meet it. Otherwise, they get removed from staking. After 2 months, the SLA would increase from 95% to 98%.</p> <p><strong>Due to this rule and the validator missing checkpoints yesterday, 25 out of 100 validators would get kicked off staking.</strong></p> <p>Thus, they're now voting to reduce the SLA back down to 95% in <a href="https://forum.polygon.technology/t/pip-9-performance-benchmark-adjustment/11387">PIP-9</a>. <strong>Lowering the SLA would save 18-23 of them from getting unstaked. 2 of them would be unstaked no matter the vote since they're below the 95% SLA threshold.</strong></p> <p>So there are 3 big findings from yesterday's validator outage:</p> <ol> <li>Massive > 32-blocks reorgs can still happen (even after PIP-5 reduced most them from 128 to 32 with the sprintLength update). Many validator clients cannot automatically correct for them.</li> <li>The SLA threshold has been reduced again</li> <li>The Polygon team is investigating eliminating the reorg issues permanently by replacing their dual-layer block proposer/consensus blockchain protocol with a single-layer consensus protocol.</li> </ol> <p>Obviously, this was extremely messy for Polygon.</p> <p><strong>Update</strong>: PIP-9 passed, so 18 of those validators are saved. Another 5 of them remain in Grace period 1 and might survive if they recover before the next 700 checkpoints. 2 will be removed.</p> <p>I see some of you comparing this to a Solana outage. It's not equivalent. This was a validator outage, but the network itself stayed up aside from the reorg.</p> </div><!-- SC_ON --> submitted by <a href="https://www.reddit.com/user/Maleficent_Plankton"> /u/Maleficent_Plankton </a> <br/> <span><a href="https://www.reddit.com/r/CryptoCurrency/comments/11ahkzw/after_yesterdays_outage_25_out_of_100_total/">[link]</a></span> <span><a href="https://www.reddit.com/r/CryptoCurrency/comments/11ahkzw/after_yesterdays_outage_25_out_of_100_total/">[comments]</a></span>
<!-- SC_OFF --><div class="md"><p>A 25 out of 100 total Polygon validators are about to lose their staking status due to the outages yesterday. In additional, the co-founder of Polygon stated that they are exploring replacing their entire <a href="https://twitter.com/mihailobjelic/status/1628926312076423170">Bor/Heimdall blockchain protocol with a single-blockchain solution</a>.</p> <p><strong>Yesterday after laying off 20% of its staff, several Polygon validators experienced sync issues, which then lead to giant cascade of outages for most validators</strong>. They ended up having to resync, often up to 12-24 hours of blocks. The <a href="https://www.coindesk.com/tech/2023/02/22/polygon-blockchain-suffers-apparent-outage/">block explorer outage</a> was a result of this cascade. The Polygon team has yet to announce the root cause of the issue, but it was attributed to a <a href="https://www.coindesk.com/tech/2023/02/22/polygon-blockchain-suffers-apparent-outage/">157-block reorg</a>. Their validators are able to correct for a 128-block reorg, but not anything higher. The reason for the reorg was <a href="https://forum.polygon.technology/t/157-block-reorg-at-block-height-39599624/11388">a bug between their Bor and Heimdall layers</a>.</p> <p>Back in Aug 2022, there was a governance proposal, <a href="https://forum.polygon.technology/t/pip-4-validator-performance-management/9956">PIP-4</a>, to move towards permissionless validation.</p> <blockquote> <p>"In its current state, the PoS validator network is not entirely permissionless in that the previously-selected set of validators has largely persisted since testnet."</p> </blockquote> <p>The proposal added an SLA requirement that 95% of median average of last 700 checkpoints are signed by the validator. If the validator fails to meet the SLA, they have another 700 checkpoints to meet it. Otherwise, they get removed from staking. After 2 months, the SLA would increase from 95% to 98%.</p> <p><strong>Due to this rule and the validator missing checkpoints yesterday, 25 out of 100 validators would get kicked off staking.</strong></p> <p>Thus, they're now voting to reduce the SLA back down to 95% in <a href="https://forum.polygon.technology/t/pip-9-performance-benchmark-adjustment/11387">PIP-9</a>. <strong>Lowering the SLA would save 18-23 of them from getting unstaked. 2 of them would be unstaked no matter the vote since they're below the 95% SLA threshold.</strong></p> <p>So there are 3 big findings from yesterday's validator outage:</p> <ol> <li>Massive > 32-blocks reorgs can still happen (even after PIP-5 reduced most them from 128 to 32 with the sprintLength update). Many validator clients cannot automatically correct for them.</li> <li>The SLA threshold has been reduced again</li> <li>The Polygon team is investigating eliminating the reorg issues permanently by replacing their dual-layer block proposer/consensus blockchain protocol with a single-layer consensus protocol.</li> </ol> <p>Obviously, this was extremely messy for Polygon.</p> <p><strong>Update</strong>: PIP-9 passed, so 18 of those validators are saved. Another 5 of them remain in Grace period 1 and might survive if they recover before the next 700 checkpoints. 2 will be removed.</p> <p>I see some of you comparing this to a Solana outage. It's not equivalent. This was a validator outage, but the network itself stayed up aside from the reorg.</p> </div><!-- SC_ON --> submitted by <a href="https://www.reddit.com/user/Maleficent_Plankton"> /u/Maleficent_Plankton </a> <br/> <span><a href="https://www.reddit.com/r/CryptoCurrency/comments/11ahkzw/after_yesterdays_outage_25_out_of_100_total/">[link]</a></span> <span><a href="https://www.reddit.com/r/CryptoCurrency/comments/11ahkzw/after_yesterdays_outage_25_out_of_100_total/">[comments]</a></span>