Artificial Intelligence Auditors

The auditing of artificial intelligence is crucial for organizations because it promotes ethical values (i.e., no misinformation, discrimination, privacy violations), ensures public trust through transparency, improves operational efficiency (by identifying and addressing errors and inefficiencies), and ensures regulatory compliance (since a growing number of governments and regulatory bodies are creating laws and standards to regulate AI).

To do their work, AI auditors typically use AI auditing frameworks (or roadmaps), which help them to navigate complex AI systems and models, although these frameworks are still in development. Frameworks are offered by the Institute of Internal Auditors, National Institute of Standards and Technology, Information Commissioner’s Office (United Kingdom), and other organizations and agencies. An audit can be a one-time act, or it can be performed on a regular basis on the same AI software or system (especially if its use is critical to a business, government agency, or other entity). The auditing process can last anywhere from a few days to a few months. AI auditors perform manual audits (which can be time-consuming, but very effective in identifying issues and areas of improvement), automatic audits (which leverage other AI systems to perform audits, although this process must be carefully monitored due to potential errors), or take a hybrid approach (manual/automatic).

When working on a project, AI auditors first seek to understand the design and architecture of the AI system, including its model infrastructure, data output pipelines, decision-making algorithms, and deployment process. They do this by studying and using the technology and talking with software developers, data scientists, and others who work closely with it. Then they gather all relevant data; study the quality and integrity of the data and the AI algorithms to verify accuracy and absence of bias; evaluate the governance structure regarding its development, usage, and maintenance; and assess AI–generated results to ensure that they match expected outcomes and to identify any issues (e.g., bias, incorrect information). They also evaluate the AI’s compliance with regulations and standards, as well as identify any potential risks to the proper functioning of the technology (such as vulnerabilities that could expose sensitive data). The AI auditor then prepares a report that summarizes their findings. It identifies risks and accuracy, bias, and other issues, as well as provides recommendations for improvement. The auditor works with their employer’s tech department to implement the audit recommendations and monitor the AI systems to ensure improved accuracy and performance, as well as identify any new issues that arise.