Per the Tweet Mycelium posted on January 7th, the team is committed to providing more thorough information to detail the incident, how it happened, the impact, and the plan moving forward to prevent such an issue occurring again.
The Incident
At approximately 2:50am AEST, members of the Mycelium community Discord raised an issue with the Mycelium Swaps platform.
At 3:57am AEST, the Mycelium Incident Response Team was notified and immediately began triaging the issue. It became apparent that there were issues in upstream data, causing a trader to rapidly trade against MLP, causing degradation in the MLP price as the trader was making profits.
At 4:40am AEST, the Team decided the best course of action was to prevent all trading on the platform to protect LPs. This decision was not made lightly, as it meant that traders were unable to update, open or close positions. The decision to limit trading on the platform was necessary and would limit damage while the issue was patched.
At 5:50am AEST, the Team had identified the cause of the issue, implemented a patch and were performing a full system review before re-enabling the platform for trading.
By 6:57am AEST, the team had shipped the fix to production, and fully re-enabled trading on the platform.
The final root cause of the issue was identified as the following:
- api.binance.com was being IP blocked, and hence no data was being received from Binance for the ETH/USD fast price feed. This IP blocking was rolled out in late December;
- Bitfinex started reporting very volatile prices, with the price swinging 0.5-1% in quick succession;
- Due to there only being two price feeds, the average of the two feeds was used, causing the ETH/USD price to have large “death wicks” each time the Bitfinex price changed by a large amount. A trader was able to buy each of these dips, and sell back at the higher price, causing the slow drain of MLP;
- The Chainlink price feeds, which are used as a fallback within the system when the ETH/USD price falls outside a deviation threshold, were never activated since the price deviations were not large enough; and
- These issues combined to allow traders to arbitrage between Mycelium and other perpetual swap platforms, or to buy in the low price dips and sell back when the price returned to the actual ETH/USD price median.
The Impact
As of the 5th of January, the MLP price was USD$0.877 (USD$0.825 without trading fees). The index price at this point in time was USD$0.843. After the incident, the MLP price was USD$0.842 (USD$0.774 without fees). The index price was USD$0.848. This represented a loss for MLP holders of 4.5% (the net decrease in MLP with fees, plus the missed gains from the index increasing).
A number of actions are being taken in an attempt to make LPs whole, while limiting disruption to other active Stakers and Traders.
They include:
- Fees earned on the specific day will be directed entirely towards LPs via rewards;
- There will be a claim period, where LPs will be able to claim these rewards, during a special 'rewards period', after which the rewards will expire. Details on the rewards period and how to claim will be published shortly; and
- Fees earned throughout the remaining period, will be distributed as follows:
- 15% to MYC Stakers
- 5% to Traders on Perpetual Swaps
- 10% that is normally earned by Mycelium will be directed towards LP for the next 2 rewards cycles
- 70% to LPs
Mitigating Future Events
As part of a mitigation of future issues, the following changes have been rolled out:
- Monitoring and alerting on each individual data provider, rather than just on each asset price; and
- Restored Binance data using the read only API endpoint.
During the post mortem, the team reassessed all data providers, and have made the decision to remove FXS from the MLP composition. The fast price feed for FXS was reliant on a single data source, being Crypto.com, with a fallback to the price feed Chainlink oracle. While this might not be an immediate threat, it presents an unnecessary risk to MLP. Particularly in light of it’s historic underutilisation within the pool.
For fast price data feeds, the data providers for each are as follows:
ETH: Coinbase, Binance, Bitfinex
BTC: Coinbase, Binance, Bitfinex
Chainlink: Coinbase, Binance, Bitfinex
Uniswap: Coinbase, Binance, Bitfinex
BAL: Coinbase, Binance, Bitfinex
CRV: Coinbase, Binance, Bitfinex
There is a fallback to Chainlink price feed oracles if the price falls outside of a price deviation threshold.
There will be more announcements coming soon relating to the product and enhancements, and we are excited to share these in the coming weeks.
