OneSwap Series 10 - Every Contact Leaves a Trace: In-chain and Out-of-chain Interaction

The difference and connection between the DApp and App

DApp is an application that works on the chain. At first glance, laymen will think that users need to interact directly with the blockchain without the front and back ends which are necessary in traditional Apps. But that is not true. Just like traditional Apps, the Dapp still requires a webpage or mobile application as a front-end interface and a back end interacting with the front end.

DApp needs a front end, which is easy to understand, because direct interaction with the blockchain requires handwritten scripts, a skill barrier so high that only programmers can manage. Ordinary users need the assistance of the front end to send messages to the chain.

Why does DApp need a back end when the chain itself can act as the back end? Indeed, for some simple DApps, the chain node is enough as the backend, but as the App gets more complex, the information provided by the chain node is not enough, and a dedicated, off-chain back end is essential to perform further analysis and processing based on the information on the chain to better serve the front end.

For example, UniSwap can help users find the best swap path. Sometimes, to swap A for C, the best way is not to swap directly, but to first swap A for B, and then swap B for C. UniSwap’s back end can only find the swap path with the optimal price after further data analysis of the trading pairs on the chain in real time. For another example, OneSwap needs to provide the recommended location of the new order in the single-linked list of the order book when canceling and placing orders. An improper location may consume considerable Gas. OneSwap’s back end must maintain its own order book database to suggest locations to users.

Compared to traditional Apps, DApp requires an extra on-chain part in addition to the front end and back end. Is this necessary? Yes, it is the core logic executed on the chain that makes DApp open and transparent and allows the DApp to have multiple independently developed front ends and back ends. In principle, https://app.uniswap.org and https://oneswap.net are not the only ways to use Uniswap and Oneswap. Anyone can develop their own websites or mobile applications to cooperate with the on-chain contract in UniSwap/OneSwap and provide services for users. This is an effective restriction: developers of UniSwap and OneSwap cannot control contracts on the chain, so if the website they develop does evil or is not user-friendly, it may be replaced by anyone.

How can the information on the chain be sent to the off-chain back-end program for further analysis and processing? We need to use the event mechanism and query mechanism.

The significance of the event mechanism

The blockchain records all transactions that have been executed on the chain. These transactions cannot be tampered with and remain on the chain forever. Anyone can view these transactions and obtain the information they want.

However, obtaining such information is not easy.

Imagine that we want to understand what happened to a transaction called by a smart contract. First, we run a synchronized Ethereum full node locally, then rerun all transactions in order until the transaction concerned appears, and record changes in the Ethereum state before and after the transaction. We also need to read the smart contract code carefully. After all the above, we can figure out what happened when the transaction was executed on the chain. How troublesome!

Another important issue is that we often need to know who called our smart contract, which is even more complicated.

Prosperity on the chain is impossible without a low-friction and low-cost interaction method on and off the chain. Fortunately, Ethereum already has an answer.

As a log monitoring protocol on the Ethereum chain, the event mechanism can save the log content pre-designed by the developer into the receipt of the blockchain during the execution of a smart contract. Users can query these logs by topic and obtain the log content, contract address, calling transaction hash, block ID, and other information. Such a query is quite cheap, and developers only need to connect to a node service provider to easily obtain a transaction-related log.

The low-level implementation of the event mechanism

The EVM provides a total of five instructions: LOG0, LOG1, LOG2, LOG3, and LOG4. These 5 instructions can push data segments of any length as logs to the monitoring program of the Ethereum node, together with 0, 1, 2, 3, and 4 topics at the same time. Each topic is a 256-bit hash value. The monitoring program can ask the node to only push the log containing the specified topic to itself, thereby filtering out the logs that it does not care about.

In Solidity, you need to declare an event first, and then emit:

event AnonymousEventExample(uint start, uint middle, uint end) anonymous;
  event Mint(address indexed sender, uint stockAndMoneyAmount, address indexed to);
  ...
  emit Mint(msg.sender, (moneyAmount<<112)|stockAmount, to);

If an event is not declared anonymous, the signature of the event itself (its name and parameter list) is hashed by Keccak to get the first topic, and the parameter marked as indexed in the event parameter is also used as the topic (there can only be three such indexed parameters at most). Event parameters that are not marked as indexed are encoded as a byte string, i.e. the aforementioned data segment. Finally, depending on the number of topics, Solidity uses LOG0, LOG1, LOG2, LOG3, or LOG4 instructions to emit events. The AnonymousEventExample above is an example of using LOG0 because the event is anonymous and it has no indexed parameter; Mint uses the LOG3 command because it is not anonymous and has two indexed parameters.

A log function call costs 375 Gas. 8 Gas per byte is charged for the encoded data segment. Besides, an indexed parameter is also a topic, costing an extra 375 Gas.

In other words, in a log, the longer the data segment after parameter encoding and the more indexed parameters, the more Gas is consumed. Developers may easily ignore such gas consumption, but since in a contract call multiple logs may be sent, the cumulative Gas consumption is considerable.

event compression

Integrating both the order book and AMM on the chain, OneSwap pursues the highest Gas efficiency at low transaction costs. And as Solidity does little to compress the data segment of events, OneSwap developers use some tricks to do the job.

For example, when a user creates a new market order, a programmer may possibly choose emit a NewMarketOrder log which records the user’s order information as below:

event NewMarketOrder(address indexed user, uint orderAmount, string orderSide);

Obviously, the above event will be encoded into more bytes, and the indexed tag means more Gas consumption. Specifically, the orderSide field is a string type and will occupy more encoded bytes.

Improved by OneSwap developers, it omits the redundant information that can be obtained on the chain, and compresses the remaining necessary fields according to the effective maximum number of bits to finally fit all information into one uint, as below:

event NewMarketOrder(uint data);

The compression function is as follows:

function _emitNewMarketOrder(
      uint136 addressLow, /*255~120*/
      uint112 amount,     /*119~8*/
      bool isBuy           /*7~0*/ 
      ) private {
      uint data = uint(addressLow);
      data = (data<<112) | uint(amount);
      data = data<<8;
      if(isBuy) {
          data = data | 1;
      }
      emit NewMarketOrder(data);
  }

Let’s take a look at the emitted log when the limit order is placed

event NewLimitOrder(uint data);

This log compresses more information into uint, and its compression function is like this

function _emitNewLimitOrder(
      uint64 addressLow, /*255~193*/
      uint64 totalStockAmount, /*192~128*/
      uint64 remainedStockAmount, /*127~64*/
      uint32 price, /*63~32*/
      uint32 orderID, /*31~8*/
      bool isBuy /*7~0*/) private {
      uint data = uint(addressLow);
      data = (data<<64) | uint(totalStockAmount);
      data = (data<<64) | uint(remainedStockAmount);
      data = (data<<32) | uint(price);
      data = (data<<32) | uint(orderID<<8);
      if(isBuy) {
          data = data | 1;
      }
      emit NewLimitOrder(data);
  }

Observing these two compression functions, you will find something interesting: they do not contain all, but only part of, the bits of the order initiator’s address. Is this a defect? Not really. An event is never isolated. When you find an event, it means the external function containing it is called, either by an EOA (externally owned account) or by other smart contracts. Through debug.traceTransaction(txHash, {tracer: "callTracer"}), we can query all the contract calls within a Tx, and then find the function calls and their parameters we are concerned about. By this method, we can trace the missing information in the event. These two compression functions do not need to retain any information about the order initiator's address at all, but they still do so for the convenience of debugging.

Similarly, only the order amount totalStockAmount and the pending order amount remainingStockAmount are retained in _emitNewLimitOrder. To figure out what cause the gap between the two, you need to trace the two events of OrderChanged and DealWithPool.

Data compression should not be abused. In functions that are not frequently called, an event should retain rich information and reasonable indexed parameters to facilitate off-chain query, such as LockSend event in the LockSend contract:

event Locksend(address indexed from,address indexed to,address token,uint amount,uint32 unlockTime);

Users can query related transfer behaviors based on from and to addresses.

The corresponding web3.js code is as the following, subscribing to all events sent from the 0xa01212312312231111dd1111aaaa contract.

lockSendContract.events.LockSend({
    filter: {from: '0xa01212312312231111dd1111aaaa'},
    fromBlock: 0
}, function(error, event) { console.log(event);})
.on("connected", function(subscriptionId){
    console.log(subscriptionId);
})
.on('data', function(event){
    console.log(event);
})
.on('changed', function(event){})
.on('error', function(error, receipt) {});

Query on-chain information from outside the chain

The event is the only channel for contracts to transmit information out from the chain. You may ask, can’t the return value of the called external function transmit information? Theoretically, it should, but the RPC interface eth_sendRawTransaction provided by the Ethereum node does not support the return value to be obtained outside the chain. Strange as this design may seem, developers have no other option but to face it.

The Ethereum node provides another RPC interface eth_call, which can obtain the return value of the called function, but it has a major difference from eth_sendRawTransaction: the latter broadcasts the transaction to the P2P network, and expects the transaction to be finally packaged into the block by the miner for execution, thereby changing the state on the chain; yet eth_call only executes the called function inside the node, and the modification in the node state by the function will be thrown away without taking effect, and the transaction will not be broadcasted. The only function of eth_call is to allow the caller to get the return value of the function. It even throws away the events generated during the execution of the function instead of returning them to the caller.

Smart contracts often design many read-only external functions, which do not change the internal state of the contract, only for the convenience of other contracts to query their own internal state and for the back-end programs to query on-chain information. There are also a large number of such functions in OneSwap contracts. For example:

function internalStatus() external view returns(uint[3] memory res);
  function getReserves() external view returns (uint112 reserveStock, uint112 reserveMoney, uint32 firstSellID);
  function getBooked() external view returns (uint112 bookedStock, uint112 bookedMoney, uint32 firstBuyID);
  function stock() external returns (address);
  function money() external returns (address);
  function getPrices() external returns (
      uint firstSellPriceNumerator,
      uint firstSellPriceDenominator,
      uint firstBuyPriceNumerator,
      uint firstBuyPriceDenominator,
      uint poolPriceNumerator,
      uint poolPriceDenominator);
  function getOrderList(bool isBuy, uint32 id, uint32 maxCount) external view returns (uint[] memory);

Among them, getOrderList is used to query the content of the order book. It iterates through the single-linked list of the order book starting fromidand returns maxCountorders at most. You may ask, why does it limit the number of orders returned? Isn’t it possible to return the full set of the order book? You need to consider the upper limit on Gas consumption despite the fact that eth_call is only executed on a single node. The full set of a too large order book returned may exceed the limit, leading to query failures.

You may also ask: since the back-end program can build an off-chain order book by tracking events, why do I still need to query the content of the on-chain order book? That is because the subscription and push of events are not 100% reliable, and due to network problems, some events may be dropped, making the off-chain order book out of sync with the on-chain. Therefore, it is safer to check the full set of order books on the chain regularly.

In short, it is necessary to make good use of the events returned by eth_sendRawTransaction and the function return value returned by eth_call so that the back-end program can capture enough accurate information for further analysis and better service to the front end.

Summary

This article explains the necessity of the interaction between the on-chain logic and off-chain back-end programs, and introduces event push and contract status query, two mechanisms for back-end programs to know the on-chain state.