1. Intro

This report contains our investigation of the Levana security incident. The Levana hack, which occurred in December 2023, is one of the most complex DeFi hacks in Cosmos so far, causing more than $1M in losses.

The goal of this document is to complement the official post-mortem of the Levana team and help the project and the community build more secure DeFi protocols in the future. We also note how the Levana hack could have been mitigated using a customized set of Range security monitoring.

To make the report simpler, we only focus on the ATOM-USD Levana market. We assume that the same findings and conclusions can be extrapolated for the rest of the markets.

‍

2. Background

Oracle Architecture

The Levana perpetual markets on Osmosis use Pyth as the oracle for asset prices. Prices published to the oracle price feeds are stored in a PRICES storage variable.

Price Feed Updates

The Pyth price feeds in use publish an assets price + confidence interval and an assets ema price + confidence interval. When these updates are processed by the perpetual market, only the price feeds non-ema price is used. For details on how Pyth determines these values, see:

• https://docs.pyth.network/documentation/how-pyth-works/price-aggregation

• https://docs.pyth.network/documentation/how-pyth-works/ema-price-aggregation

• https://docs.pyth.network/documentation/how-pyth-works/cross-chain

Prices are submitted to the price feed using Wormhole as the relayer. Messages relayed through Wormhole make the message data available with a proof known as a VAA (pyth-sdk copy of the VAA data structure).

Protocol Staleness

The perps protocol tracks staleness for the last time an oracle price update was processed, with each perp market having a configurable staleness_seconds parameter, which is currently 2 hours for the ATOM-USD market.

Age Tolerance

When processing price updates, an age_tolerance_seconds parameter determines the maximum time that a price that was published to a price feed can be recorded in the PRICES variable. The ATOM-USD market sets this to 120, enforcing a maximum age of 2 minutes.

Levana Price Updates

Whenever a position is opened, the PRICES variable is appended with the latest price published to the Pyth price feed in use. This allows a trader to influence the price at which the position will be opened.

3. Incident Analysis

High-Level Attack Overview

The Pyth price feed program allows anyone to submit valid price updates leveraging the VAA payloads that are published whenever a message is relayed through Wormhole. Because the Levana program does not use any sort of moving average for prices, an attacker is able to publish price feed updates, which push the price in the direction the attacker wants. It should be noted that the attacker cannot choose the price; rather, they can “choose” when the current price is submitted. Thus, it is fair to call it a probabilistic attack.

A single attack consists of four messages:

1. update_price_feeds

2. open_position

3. update_price_feeds

4. close_position

The time between messages 2 and 3 can vary and ultimately depends on the unpublished VAA payloads that can be consumed. Upon finalization of the fourth message, a single attack is completed.

The direction in which the price is moving is only of minor importance to the attack as the attacker can simply rotate between opening Long or Short positions depending on the direction of price movement.

The attack is effective because spot pricing is used by Levana, whereas if a moving average or exponential moving average was used, the advantage the attacker has in selecting the VAA payloads to publish is heavily limited.

VAA Payload Discovery

By using https://wormholescan.io/  or the SDK (https://docs.wormhole.com/wormhole/reference/sdk-docs), the attacker can easily discover newly submitted payloads and potentially front-run the submission of other payloads by being the first to submit a payload.

Attack vs MEV?

Sometimes, the line between MEV (or profitable trading strategies) and economic attacks is thin. It could be argued that a skilled trader leveraged the variation in prices during volatile markets and the spot pricing of assets that Levana uses to perform value extraction. However, if the trader went the extra step of attacking Levana’s infrastructure or intentionally manipulated fee price markets to block 3rd party updates to Pyth, then this would definitely be considered an attack.

Enumerating Levana Infrastructure

We have investigated the feasibility of the attacker enumerating Levana infrastructure and have found it was possible using open-source red team reconnaissance tooling. The attacker could have identified all the bots hosts run by Levana and the IP address information of their RPC servers. We’re aware that the Levana team has been working on improving the resilience of their infrastructure.

With the above information, it is possible for a threat actor to not only have attacked the hosts on which Levana runs backend services but also directly attack the RPC servers and potentially the osmosisd nodes backing the RPC servers, causing a denial of service in all the systems meant to push oracle updates.

Relevant Addresses Markers

ATOM-USD Levana Market:osmo1hd7r733w49wrqnxx3daz4gy7kvdhgwsjwn28wj7msjfk4tde89aqjqhu8x

Pyth Contract: osmo13ge29x4e2s63a8ytz2px8gurtyznmue4a69n5275692v3qn3ks8q7cwck7

We have also identified 9 addresses that are related to the attack, and we have traced their connections to CEXes accounts. We’re working with the Levana team, leveraging Range Trail to recompile further evidence in this regard.

Attack Data Analysis

We complement the investigation with a data analysis section to understand anomalies and patterns in the calling patterns of Levana functions (e.g., crank), block size variability, and the frequency of Pyth price updates during the attack. For further information, please check the appendix.

TLDR:

• The timing of cranks is noticeably different during the timeframe Dec 15-30 (compared to the timeframe Nov 15-30). (s. graphs 1 and 2).

• There are big gaps between cranks as well (s. query 3).

• Blocks sizes are bigger during the time of the attacks (s. graph 5)

• The timing of num_feed_updates frequency is different during the attacks (s. graphs 6 and 7)

• No obvious deviation in suspicious address txs per day (sus = tx with update_price_feeds+open_position) (s. graph 9 )

Mitigations

Since the pausing of the markets, the Levana team has been working hard to mitigate the issues at hand and reactivate the markets. Beyond the increased resiliency in their infra (for RPCs, oracle crank bots, etc), Levana is performing a protocol improvement.

The proposed changes include the deferred execution via a queue system for trading operations. A more detailed explanation of the new mechanism can be found in Levana’s engineering update.

After reviewing the proposed mechanism, our security research team has concluded that the deferred mechanism fully addresses the economic vulnerability that was exploited in December. We also want to emphasize the quick response by the Levana team and their comprehensive explanation of the changes in their engineering update.

Using Range for defense in depth

Beyond improvements at the protocol level, Levana could have benefited from a threat prevention and monitoring layer, like the Range monitoring offering. More than 60% of crypto hacks are non-atomic (happened in more than one transaction). In the case of Levana, the attacker slowly drained the protocol for almost two weeks.

Range enables teams to define invariants and security rules, which can detect attacks in real time. Moreover, with the mempool monitoring offering, these attacks could have prevented, or at least reduced, their damage by more than 90%. In the case of Levana, customized rules could have detected anomalies in the crank and oracle price updates, large variations in price per update (e.g.,>3%), suspicious addresses interacting with the protocol in inorganic patterns, and more.

4. Conclusion

The Levana Hack of December 2023 is one of the most complex attacks on a DeFi protocol in Cosmos, as it combines a deep understanding of the smart contract protocol mechanics and a coordinated targeted denial of service of Levana infrastructure.

We have concluded that the deferred mechanism proposed and implemented by Levana fully addresses the economic vulnerability that was exploited in December. In addition to that, we recommend an in-depth security approach, combining security audits with a post-deployment threat prevention and monitoring offering such as the one provided by Range.

We appreciate Levana’s quick response addressing some of our questions and prompt collaboration on this investigation.

About Range

Range builds security infrastructure for sovereign blockchains and rollups, with a focus on the Cosmos ecosystem and bridges such as the Inter-Blockchain Communication Protocol (IBC). Range's product suite encompasses tools for monitoring, threat detection and prevention, analytics, and forensics in order to strengthen the security of the interchain and modular ecosystems.

Appendix

Historical frequency of crank execution in contract osmo1hd7r733w49wrqnxx3daz4gy7kvdhgwsjwn28wj7msjfk4tde89aqjqhu8x.

X-axis: number of blocks between two consecutive cranks -> y-axis: number of cranks with this specific gap. Most cranks are executed within the same block (same in August long before the attacks, s. second graph).

Historical frequency of crank execution in contract osmo1hd7r733w49wrqnxx3daz4gy7kvdhgwsjwn28wj7msjfk4tde89aqjqhu8x.

Same graph as before but for a different timeframe (before the attacks)

Are there gaps? -> The graphs show a long tail of gaps.

Here's a list of the top 10 biggest gaps since December 2023 (number of blocks between two consecutive cranks):

Who are the crank callers?

Top callers with their number of calls

Historical block size during the manipulation. Were blocks full? Was it congestion?‍

Blocks are bigger during the time of the attacks

Before the attacks: Historical calling activity for [.in-line-code]update-price-feeds[.in-line-code] in Pyth Network Oracle contract osmo13ge29x4e2s63a8ytz2px8gurtyznmue4a69n5275692v3qn3ks8q7cwck7.

Are there anomalies in calling frequency? -> Yes, compare this graph to the next (7).

During the attacks: Historical calling activity for update-price-feeds in Pyth Network Oracle contract osmo13ge29x4e2s63a8ytz2px8gurtyznmue4a69n5275692v3qn3ks8q7cwck7.