Testing of Artificial Intelligence (AI)

Can machines think and behave like humans or animals? From this question starts the life of Artificial Intelligence (AI). Artificial Intelligence is intelligence demonstrated by machines, in contrast to the natural intelligence displayed by humans and other animals.

Nowadays, AI has been prevalently used across a wide range of activities.
1. Virtual Personal Assistants
2. Video Games
3. Banking
4. Automobile industries
5. Security services
6. Medical field health and services

AI is related to various disciplines in both the sciences and humanities.

The above depiction shows the application of AI in day-to-day life all through most of the disciplines.

AI and Testing

Testing an AI machine means challenging human intelligence. We can also say that we are conducting an exam for the entire machine that has extra powers of natural intelligence. An AI application must maintain the highest of qualities. So we have to think how we must test them, which method of testing should be used. Should we do a manual testing or automation testing or do we need to find new ways of testing them?

Based on my past experience, I believe, we not only require manual testing and automation test, but also continuous integration testing which also plays an important role. Testers will also require new skills for testing such systems.

The challenge of testing AI is that, learning systems tend to change their behaviour over time. Predicting the results of such systems may not be possible, because the outcome that is deemed as correct today, may be different from what will be deemed as correct tomorrow.

This scenario demands a tester to have the skills to interpret a system’s boundaries or tolerances. Testers not only look at the output but also at the system’s input. Because, by limiting the input, we can influence the output.

We can categorize testing of Artificial Intelligence, Robotics, Machine Intelligence and Cognitive IT into following aspects.

1. Mechanical and Electrical
Mechanical and Electrical testing mainly applies to robots and other smart devices/ machines. These two techniques have existed for a long time and are not particularly different for any new technology.

2. Information Processing
Information processing relates to traditional IT functions and systems. Different Testing Management methods are available for this. Test management approach is an approach that combines insights on how to test and what to manage, as well as techniques for the individual test consultant.

3. Machine Intelligence
Intelligent machines have machine learning capabilities that will result in different results if the same function is called at different points of time. A major difference between testing an AI system and a traditional IT system is that, it is very hard to predict a solid result on an intellectual system. Tester can define the tolerances for the outcome that sets the boundaries between which an outcome is considered to be correct. A learning machine uses its input to discover new patterns and/or new options. Hence, it is very important to control the input. If we can control the input, we can control the output to some extent.

4. Business Impact
The effect of an AI system on a business process is different from that of the effect of traditional IT systems on a business process. In the former case it is difficult to predict the effects. When robotics technology was introduced, companies started applying new technology and created robots for their daily process. This naturally had its consequences.

Nowadays, new technology is available at your arm’s reach. The challenge now will be how to be sure that the new business process will provide the expected result to the future market. Hence, assurance of the systems to work with minimal consequence should not only be active during the project development phase, but also during its first use. This type of monitoring of the effect of new technology will ensure that both desired and unwanted effects are identified as soon as possible.

5. Social Impact
AI systems can have positive or negative impacts on our social environment. Will a taxi driver become unemployed because of the introduction of self-driving cars? Will it affect the nurses’job? (It is expected that nurses will have more time to pay personal attention to elderly patients, as the robots will do the basic care-taking activities, such as distributing the right pills or food and cleaning.)

It is hard to set up test scenarios for such impacts. But digital assurance is far more than just testing. With digital assurance, we can also observe effects in another way, such as the difference between old and new situations. This will require the ability to compare these situations and identify small signs of changes that the new technology brings on individuals, organizations and on society as a whole.

What needs to be tested?
AI system must ensure that the functional and non-functional requirements are fulfilled. Since the behaviour is much harder to predict because of the complex functioning of an AI system, many more input values have to be tested to verify a robust solution.

The following are some of the aspects that we can look for while testing an AI system:

1. What are acceptance criteria?
2. How to design test cases that test the criteria with minimal effort?
3. Are there different datasets for training-data and test-data?
4. Are the test-data adequate to represent the expected data well enough?
5. IS the test- and training-data compliant with the legal framework?
6. Is the training-data biased?
7. What is the expected outcome?
8. Is the model under or over-fitted?
9. Does the solution behave unethically?
10. Is the performance and robustness of the solution good enough?

How to Test

Till now there are no methodologies defined to test AI applications. All we know are the traditional testing methods like STLC (software testing life cycle). Hence we can say that testing will be different for each and every AI application. Below are some common scenarios to test an AI application.

1) AI systems process huge amount of data to reach an output. These data can be of varying types such as text, speech, images or videos, and can come from different sources. For an AI system, data is extremely critical and forms the backbone of any output, decision or action. Testers should have a huge collection of test data before they start testing with any AI application.

2) Next part of testing will be machine learning system or core algorithms. As the system is trained to perform certain actions using these algorithms, these algorithms become the heart and living soul of an AI system. It could be very complex as multiple algorithms are combined to get the desired output. The tester should not only check the desired output of an AI but also need to concentrate on the hidden outputs to understand the system behaviour. Because, in real life scenario, the system will provide different outputs for the same input data.

3) The final stage of testing will be the interaction of an AI with a 3rd party system interface. An AI system always interacts with the real-time environment to provide the output, so it is important to see how it fits into such a system. Testers need to perform the integration testing to ensure that both the system interacts as expected. For this, we may have to use certain testing methodologies like A/B testing, metamorphic testing, functional and non-functional testing, security testing etc.

There is no doubt that we are moving to an AI-first world. Testers need to acquire the skills to interpret a system’s boundaries or tolerances, and the emphasis of testing alters from output to input to substantiate a robust solution.