Artificial intelligence (AI) has been widely adopted across various industries, demonstrating its transformative potential. Now, the blockchain industry is testing AI’s capabilities in an essential area: smart contract security. While AI-based blockchain audits have shown promise, they still lack key qualities possessed by human professionals, such as intuition, nuanced judgment, and subject expertise.
OpenZeppelin, an organization specializing in smart contract security, conducted a series of experiments using OpenAI’s latest GPT-4 model to identify vulnerabilities in Solidity smart contracts. The experiments involved testing code from the Ethernaut smart contract hacking web game, designed to train auditors to detect exploits. GPT-4 successfully identified vulnerabilities in 20 out of 28 challenges, showcasing the value of AI in vulnerability detection.
However, the results were not consistently accurate. At times, the AI required leading questions or failed to identify vulnerabilities even when explicitly described. It even generated a false vulnerability. These limitations highlight the current boundaries of this technology. Despite these shortcomings, GPT-4 has made significant progress compared to its predecessor, GPT-3.5.
Coinbase also conducted similar experiments using ChatGPT for token security reviews. While the AI provided comparable results to manual reviews for some smart contracts, it struggled with others and misclassified high-risk assets as low-risk. It’s important to note that ChatGPT and GPT-4 are language models developed for natural language processing and text generation, not specifically for vulnerability detection.
To improve vulnerability detection, training data and models tailored to specific objectives would likely yield more reliable results. For example, OpenZeppelin’s AI team developed a custom machine learning model to detect reentrancy attacks, which demonstrated superior performance compared to industry-leading security tools.
The experiments conducted so far indicate that while AI models can be useful in identifying security vulnerabilities, they cannot replace the nuanced judgment and subject expertise of human security professionals. Current AI models rely on publicly available data up until 2021, limiting their ability to recognize complex or unique vulnerabilities. The future of smart contract security lies in a collaborative approach, leveraging AI tools alongside human expertise.
By using AI to identify common vulnerabilities and staying updated with the latest advances, human auditors can effectively defend against AI-armed cybercriminals. Although AI alone cannot replace humans, auditors who embrace AI tools will be more effective in ensuring the security of smart contracts.
In conclusion, the potential of AI in auditing smart contracts is evident, but its current limitations necessitate a combined effort between AI and human expertise. Continued learning about the latest advancements and vulnerabilities within the blockchain industry is crucial. By striking a balance between AI and human auditors, we can harness the power of this emerging technology in a way that enhances security and drives positive innovations in the blockchain realm.