EVM development Starter Kit
Note: This blog post is divided into two parts. Part 1 will cover the basics of the EVM and is a pre-requisite to Part 2 in which we will demystify the bytecode of a compiled smart contract. To understand the interactions between the stack, memory and storage and write more efficient code we will go step by step through the execution of two full transactions.
You most likely have heard about Web3, but in its broadest sense what is it? In simple terms, Web3 adds a decentralised trusted state layer to the internet, opening the door for a multitude of empowering innovations. Moreover this state layer does not only store data but also stores programs that can be executed trustlessly and modify the state. The most popular Web3 infrastructure is powered by the Ethereum Virtual Machine (EVM). The concept of the EVM originated from the Ethereum yellow paper and is an implementation of its operation codes. This blog post will introduce the basics of the EVM, as well as provide a detailed practical example on how the compiled bytecode gets executed at the lower level. Any feedback is greatly appreciated :)
Part 1: EVM Basics
First of all, why a virtual machine? When it comes to decentralised networks like Ethereum, it is important that you can run a program independently of an individual machine’s operating system or hardware architecture. Similarly to the way the JVM allows you to run JAVA programs on any machine, the EVM allows you to run EVM bytecode on any machine. (In other words, the resulting bytecode of these programs does not have to vary depending on the specific machine’s underlying cpu architecture like arm or x86, the VM abstracts away the hardware).
The EVM is a virtual machine that implements the opcodes defined in the Ethereum yellow paper. It offers access to persistent storage and a stack based processor that is limited to 1024 256-bit sized words. As opposed to languages like C or C++ where value types are stored on the stack and reference types on the heap, the EVM does not store any values on the stack but loads them from memory when needed.
Another important point to note, is that the EVM is completely isolated from the operating system having no access to the file system or other processes to guarantee that every machine on the network concludes on equivalent results for a computation.
Last but not least the EVM is based on accounts as opposed to Bitcoin’s UTXO model, that stores a user’s balance through the sum of all transaction receipts. This simply means that data is mapped to an account key. Two types of accounts exist, externally owned accounts (EOA’s) and contract accounts. External accounts are controlled by public-private key pairs and can initiate state transitions by sending a transaction. Contract accounts on the other hand, are accounts that store a specific program.
Before going through a practical example, lets run through the vocabulary of places where the EVM can access and store information.
The Ethereum Virtual Machine has five areas where it can store data — stack, storage, memory, calldata and logs which are explained in the following paragraphs.
Storage
Storage in the EVM is persistent, meaning it persists between transactions. Storage is represented in the form of a merkle patricia trie and various Ethereum clients use different database implementations for it. For example Ethereum’s Python and Go client implementation use leveldb to store their tries. You might have heard about this fancy term “Merkle Patricia Trie” as part of the Uniswap airdrop distribution, but in essence it allows two things. First, it allows us to efficiently verify the integrity of data and second, it enables us to verify the inclusion of a specific piece of data with only a small subset of data that makes up the tree. These are two attractive properties when it comes to sharing data across a decentralised network.
Integrity of the global state is ensured by each new block header in the form of a state root. This state root represents the root node of a merkle patricia trie made up of 160 bit account keys mapping to the corresponding account state, that includes an account balance, a code hash, a nonce and a storage root. The storage root is again a merkle patricia trie root node and stores the contract’s state variables through a mapping of 256-bit words to 256-bit words. The code hash points to the given smart contract’s source code. Note that an EOA will have both of these fields empty as part of the account state. Keep in mind that the blockchain itself only stores the blocks (including a hash of the state root in the header), while the clients store the full tries content in a database.
Memory
Temporary storage during runtime of a transaction, think of it as RAM.
Calldata
The data field of a transaction is read-only memory.
Logs
Write-only output area to emit Logs.
Stack
Stores value temporarily during runtime to be used by operations. Any operation takes words from the stack of pushed words onto the stack or both. It serves as the intermediary to read, write, and manipulate data from storage, memory, calldata or the logs.
Part 2: Understanding your code at the lower level, a practical example
Solidity is the most popular high-level EVM language used and benefits from a rich ecosystem of tools and support. In this practical example, we will write a simple program in Solidity and investigate how the compiled byte code executes. Solidity is similar to other object oriented languages in that it is based on classes called contracts which when deployed create a single instance of that class by running the constructor function. The byte code of the resulting instance (post constructor execution) is stored as the program code of the given contract account (as part of the account state, mentioned in the Storage section above).
To get started, you can either download the static binaries of the solidity compiler here, or use the JS version of the solidity compiler which can be conveniently used within your javascript project, or within a browser based IDE like Remix. Either way compilation will generate the resulting EVM bytecode of your programme along with its ABI (an interface which tells other applications how to interact with your contract).
The EVM bytecode is a hexadecimal representation of a sequence of operation codes known as EVM opcodes (full list of EVM opcodes can be found here). Each opcode has a fixed amount of gas assigned and is a measure of resource consumption. In this practical example, we will use Remix to write a very simple contract and look at how it translates to a set of operation codes that get executed by the EVM.
Below is the a very basic smart contract which allows us to get a sense of what the generated bytecode does at deployment as well as when when a function is being called thereafter. The contract Example has one public storage variable that gets assigned a value of 1 by the constructor function at deployment. Although simplistic, analysing the bytecode of this smart contract will illustrate the basic interactions between the stack, memory and storage allowing you to apply the same methodology on more complex contracts.
contract Example {
uint256 public exampleNumber;
constructor(){
exampleNumber = 1;
}
}Running the compiler will yield the following bytecode:
Note: the deployment part of the byte code is highlighted in bold, more on the difference between runtime bytecode and deployment bytecode later
“608060405234801561001057600080fd5b50600160008190555060b2806100276000396000f3fe6080604052348015600f57600080fd5b506004361060275760003560e01c80620511a014602c575b600080fd5b60326046565b604051603d91906059565b60405180910390f35b60005481565b6053816072565b82525050565b6000602082019050606c6000830184604c565b92915050565b600081905091905056fea2646970667358221220d6d4f6b7d89f1a77922be220ca2512cf050c8eb988d06ec97a32525e80f98ae464736f6c63430008070033”
Which translates to the following opcodes (you can check for yourself here):
“PUSH1 0x80 PUSH1 0x40 MSTORE CALLVALUE DUP1 ISZERO PUSH2 0x10 JUMPI PUSH1 0x0 DUP1 REVERT JUMPDEST POP PUSH1 0x1 PUSH1 0x0 DUP2 SWAP1 SSTORE POP PUSH1 0xB2 DUP1 PUSH2 0x27 PUSH1 0x0 CODECOPY PUSH1 0x0 RETURN INVALID PUSH1 0x80 PUSH1 0x40 MSTORE CALLVALUE DUP1 ISZERO PUSH1 0xF JUMPI PUSH1 0x0 DUP1 REVERT JUMPDEST POP PUSH1 0x4 CALLDATASIZE LT PUSH1 0x27 JUMPI PUSH1 0x0 CALLDATALOAD PUSH1 0xE0 SHR DUP1 PUSH3 0x511A0 EQ PUSH1 0x2C JUMPI JUMPDEST PUSH1 0x0 DUP1 REVERT JUMPDEST PUSH1 0x32 PUSH1 0x46 JUMP JUMPDEST PUSH1 0x40 MLOAD PUSH1 0x3D SWAP2 SWAP1 PUSH1 0x59 JUMP JUMPDEST PUSH1 0x40 MLOAD DUP1 SWAP2 SUB SWAP1 RETURN JUMPDEST PUSH1 0x0 SLOAD DUP2 JUMP JUMPDEST PUSH1 0x53 DUP2 PUSH1 0x72 JUMP JUMPDEST DUP3 MSTORE POP POP JUMP JUMPDEST PUSH1 0x0 PUSH1 0x20 DUP3 ADD SWAP1 POP PUSH1 0x6C PUSH1 0x0 DUP4 ADD DUP5 PUSH1 0x4C JUMP JUMPDEST SWAP3 SWAP2 POP POP JUMP JUMPDEST PUSH1 0x0 DUP2 SWAP1 POP SWAP2 SWAP1 POP JUMP INVALID LOG2 PUSH5 0x6970667358 0x22 SLT KECCAK256 0xD6 0xD4 0xF6 0xB7 0xD8 SWAP16 BYTE PUSH24 0x922BE220CA2512CF050C8EB988D06EC97A32525E80F98AE4 PUSH5 0x736F6C6343 STOP ADDMOD SMOD STOP CALLER “
This can look intimidating but it is not really once you run through it opcode by opcode which we will do below. Once you see how things are being run, you will be able to think of ways to apply different opcodes that consume less gas to achieve the same thing. Learning solidity’s syntax and types in itself will take you a week at most. However if you want to deeply understand and improve your code, it is very helpful to understand what happens at the lower level.
As mentioned before, the bytecode is a hex representation of EVM opcodes that are documented in the yellow paper. When deploying a new contract the bytecode as part of the calldata gets executed. The bytecode includes a section that is referred to as the init section which is not being stored as part of the contract’s account code state but returns the actual contracts runtime bytecode and runs the constructor function which in our case sets an initial value to our public storage variable exampleNumber. The init part of the byte code is highlighted in bold above. To create the contract as part of the blockchain state, a deployment transaction like any other has to be initiated except that the receiver address is set to be empty while the data field contains the compiled bytecode. Once the transaction is being executed an account is created for that contract and the bytecode is being executed.
With that in mind, let’s run this bytecode that runs at deployment and demystify this series of opcodes by going through them sequentially. We will go through it by executing a series of opcodes and visualise the resulting stack. When no stack is shown it indicates an empty stack, i.e all words have been popped off.
PUSH1 0x80 PUSH1 0x40 -> push 0x80 and 0x40 onto the stack
MSTORE -> store value 0x80 at location 0x40 in memory and pop both words off the stack (this is a free memory pointer that comes with compiled solidity to reference the first unused word in memory and prevents overwriting memory that is used internally by solidity)
CALLVALUE DUP1-> pushes the amount of ether that was specified as the value field within the deployment transaction on top of the stack and duplicates it with DUP1 opcode
ISZERO -> checks if the top word of the stack is zero and pops the word off. If it was zero it pushes 0x1 else 0x0. For the sake of our example, let’s follow the execution path as if we had set the value field of our deployment transaction to 0. Hence ISZERO returns 0x1 on top of the stack.
PUSH2 0x10 -> push 2 bytes 0x0010 onto the stack
JUMPI -> Jump to instruction with program counter equal to top word of stack 0x0010 if second word of stack is 0x1 (if transaction value field was 0), else continue. Thus in our case we jump to program counter 0x0010 and continue execution since our deployment transaction did set the value field to zero. If we had set the value field to zero, we would not jump at this point but continue execution with PUSH1 0x0 DUP1 REVERT. That would push 0x0 onto the stack, duplicate that word and cause a revert with the top two words indicating the reason for the revert.
Note: All that has been happening until now was to check if the constructor was payable and if the deploy transaction sent some ether along with it. Since our constructor was not payable in our exampleContract, sending ether with our deployment transaction should revert the deployment. Hence the conditional jump above.
Program counter 0x0010: JUMPDEST POP-> valid jump destinations are indicated by the JUMPDEST opcode. Since we did not send ether with our deployment transaction the init execution jumps to this program counter and we continue execution while popping off one word with the POP opcode leaving us with an empty stack.
Note: From this part onward the constructor is being executed as part of the init code.
PUSH1 0x1 PUSH1 0x0 -> push 0x01 onto stack (the value we assign to exampleNumber in the constructor). Push 0x0 on top of the stack (we only have one storage variable so it will be located at storage slot 0x0)
DUP2 SWAP1-> duplicate the second word of the stack and swap first word with second word
SSTORE -> store top word (0x1) value at storage slot second word (0x0), and pop both words off. The value 1 is now being stored as at slot 0 which is the storage pointer of our only variable exampleNumber.
POP -> remove value from stack, leaving an empty stack
Note: From this part onward the copying of the actual runtime bytecode starts and is being returned by our deployment transaction which stores it under the newly generated contract account code section of the global blockchain state.
PUSH1 0xB2 DUP1 PUSH2 0x27 PUSH1 0x0-> push 0xB2 on stack, duplicate it push 0x27 and 0x0 on stack
CODECOPY -> copies the runtime code to memory by by taking top word 0x0 as memory offset to write the code to, second word 0x27 as offset of the byte code to read from (this is 39 in decimal and if you look at the bytecode you can see that the runtime code start at the 40th byte), and third word 0xB2 as the length in bytes to copy (length runtime bytecode). All three words pop off stack.
PUSH1 0x0 RETURN -> push 0x0 on top of stack and return. The RETURN opcode will return the runtime bytecode that was stored in memory at offset 0x0 with length 0xB2.
RETURN -> end execution and return the runtime code as a result. First word of stack is the memory offset of the result and the second word is the ending offset in memory of the result.
INALID -> indicates the end of the init code and start of runtime code.
Our contract has now successfully been deployed to the blockchain and the runtime bytecode of our contract is part of the global state under our contract accounts address.
We now move on to have a look at the runtime bytecode by calling a specific function of our deployed contract by initiating a new transaction. Our Example contract only has one function which is a getter for the storage variable exampleNumber (getters get automatically generated by solidity for public state variables). Keep in mind that purely reading state does not need a transaction and can be done by simply reading from your node’s database. However to keep this illustration simple we will read the value by triggering a transaction the same way we would call any function that mutates the state.
Let’s go through the runtime bytecode’s only getter function exampleNumber() by. When calling a function on a smart contract the transaction data field will contain the first four bytes of the keccak256 hash of the function’s signature followed by the input arguments. In our case there are no input arguments and hence the data field of our transaction only contains those 4 bytes. In the case of exampleNumber() this yields 0x511A0 as input for the data field. Furthermore our transaction will have 0 as the value field since the getter is non payable. Let’s see how the exampleNumber() getter function e call executes by going step by step through the execution of the runtime bytecode.
Note: Since there will be a lot of conditional jumping in this part of the bytecode, there is a table at the bottom of this post mapping each instruction of the runtime bytecode to its program counter so you can double check we jump to the right location.
PUSH1 0x80 PUSH1 0x40 MSTORE -> setting up the free memory pointer like it was done at the start of the init code.
CALLVALUE DUP1 ISZERO -> push the transaction value field input on the stack, duplicate it and check if its zero and pop it off, if it was true pushes 0x1 else 0x0. Let’s assume again our transaction did not send any ether as part of it’s value field, hence we push 0x1 onto the stack. You can probably already guess what is happening here, we are again checking if ether has been incorrectly sent to a non payable function.
PUSH1 0xF JUMPI-> push 0xF on top of the stack and jump to instruction at program counter 0xF if the second word of stack is 0x1. If ether was sent as part of the value field with our transaction we would not jump to program counter 0xF and continue by reverting from here (PUSH1 0x0 DUP1 REVERT) because our only function exampleNumber() is not payable.
Gas optimisation tip: You can save a little gas (CALLVALUE DUP1 ISZERO PUSH1 0xF JUMPI) by making all your functions payable to avoid the value field check as is also pointed out here by Mudit Gupta.
From this part onwards the bytecode validates the data field of our transaction (calldata) is at least 4 bytes long (minimum requirement since it’s the size of a function selector as mentioned earlier).
POP PUSH1 0x4 CALLDATASIZE -> remove top word of stack and push 0X4 onto it. Opcode CALLDATASIZE pushes the transaction data field length onto the stack.
LT-> checks if calldata size (top word) is less than 4 (second word) bytes long , if it is 0x1 gets pushed on stack, otherwise 0x0. Top two words get popped off the stack. Since our transaction to call exampleNumber() had a calldata of exactly 4 bytes 0x0 gets pushed onto stack to indicate calldata size is larger or equal to four bytes.
PUSH1 0x27 JUMPI-> push 0x27 on stack and jump to that program counter if the second word of stack is 0x1 (if calldata was less than 4 bytes) from where our execution path will revert and fail. Else (if calldata is more than 4 bytes) continue. In our case we continue since we our calldata input of our transaction is a valid 4 byte function selector.
From here onwards the bytecode iterates through the function jump table by comparing our transaction’s calldata function selector to available functions in the contract and jump to the appropriate one to execute it’s runtime bytecode. Since a lot of jumping will occur we will not go through the bytecode byte by byte but instead follow the execution flow by continuing at the relevant program counter.
PUSH1 0x0 CALLDATALOAD PUSH1 0xE0-> push 0x0 onto the stack and push the 32 byte long calldata input onto the stack, push 0xE0 onto the stack.
SHR -> opcode to shift right. Here we shift the calldata input to the right 0xE0 (224) times. 32 bytes was the size of the calldata input of which the first 4 bytes were the function selector. Hence the second word of the stack has been reduced to be the function selector itself. Furthermore the SHR opcode pops the first word off stack leaving us with.
DUP1 PUSH3 0x511A0 -> duplicate the function selector and push the following three bytes onto the stack 0x511A0 (remember this hexadecimal? this was the first 4 bytes of the keccak256 of the exampleNumber(), which we calculated earlier to input as our data field).
EQ -> pushes 0x1 on stack if the second word is equal to top word, i.e if the function selector of the calldata is equal to 0x511A0. In the Example contract we only had one runtime function, a getter to retrieve exampleNumber value. Remember 0x511A0 is the first 4 bytes of the kekkak256 hash of “exampleNumber()”. Hence what the run time code is doing here is checking which function the transaction is calling. Once a match is found, the execution jumps to the relevant program counter where that function’s runtime code is located. In our case there is only one function, so the first comparison is a successful match and we push 0x1 onto the stack.
Gas optimisation tip: In case we had more functions in our smart contract, we would keep on doing this comparison until we find the relevant function to execute. You can optimise your contract for runtime gas by naming your most used functions, so that they appear early in the search order. If there are more than four functions in your smart contract binary search is applied. The order of your functions appear in ascending order. Hence you can name your most used function so that it appears first (at mid).
PUSH1 0x2C JUMPI -> push 0x2C (the location of the functions exampleNumber() run time code) onto the stack and jump to it since the top word of stack was 0x1 indicating jump should happen.
Program counter 0x2C: JUMPDEST PUSH1 0x32 PUSH1 0x46 JUMP -> we jump to program counter 0x2C indicated as a valid jump destination by the JUMPDEST opcode, if the function selector in the calldata corresponded to the the function selector of our getter on exampleNumber() . we push 0x46 on stack and jump to program counter 0x46.
Program counter 0x46: JUMPDEST PUSH1 0x0 SLOAD DUP2 JUMP -> we jump to program counter 0x46 validated by the JUMPDEST opcode. Storage slot 0 gets loaded on stack (which is the storage slot of our only state variable example number) and the second word from stack gets duplicated and we jump to that program counter 0x32
Program counter 0x32: JUMPDEST PUSH1 0x40 MLOAD PUSH1 0x3D -> Push 0x40 on stack and load that location from memory which is 0x80. Remember 0x80 was the free memory pointer set up at the start. We push 0x3D on stack
SWAP2 SWAP1PUSH1 0x59 JUMP -> swap 3rd and 1st element of stack leaving us with the stack on the left. We push 0x59 onto the stack and jump to program counter 0x59.
Program counter 0x59: JUMPDEST PUSH1 0x0 PUSH1 0x20 DUP3 -> push 0x0 and 0x20 onto stack and duplicate 3rd word on top of stack
ADD SWAP1 POP PUSH1 0x6C PUSH1 0x0 DUP4 ADD DUP5 PUSH1 0x4C -> more stack manipulations leading to below stack (as an exercise, verify don’t trust;)) and we jump to top word of stack 0x4c.
Program counter 0x4c: JUMPDEST PUSH1 0x53 DUP2 PUSH1 0x72 JUMP -> more stack manipulations… and jumping to 0x72
Program counter 0x72: JUMPDEST PUSH1 0x0 DUP2 SWAP1 POP SWAP2 SWAP1 POP JUMP -> more stack manipulations and jumping to 0x53
Program counter 0x53: JUMPDEST DUP3 MSTORE POP POP JUMP ->duplicate 3rd word from stack (the value of examplenNumber) and store it at location 0x80 (our free memory pointer) in memory and pop top two words off stack. Jump to pc 0x6C
Program counter 0x6C: JUMPDEST SWAP3 SWAP2 POP POP JUMP -> stack manipulation and jump to 0x3D
Program counter 0x3D: JUMPDEST PUSH1 0x40 MLOAD DUP1 SWAP2 SUB SWAP1 -> load free memory pointer 0x80 and some stack operations leading to:
RETURN -> return the value at memory location 80 (the free memory pointer) of length 0x20 (32byte word). Last word of stack doesn’t get used.
TL;DR: The runtime bytecode of the exampleNumber() getter function, loaded the only storage variable from slot zero, stored it in memory at our free memory pointer location and returned it with the RETURN opcode.
Tada! That was it. We went from source code to deploying the resulting bytecode and calling a function on the stored runtime bytecode. Hope that helps and gives you an idea of what happens “behind the scenes”. This was a very simplistic example to demonstrate how to debug on the EVM level and hopefully helps in getting your hands dirty with more complex cases. A more thorough example of this byte by byte analysis can be found here. Have fun!
