Motivation: Software is one of the most complex artifacts of mankind has ever created, but complexity is the enemy of correctness. Modern software testing and validation tools use a multitude of techniques geared toward correct computer code, most of these are base on artificial intelligence. In this course, we study these techniques in details, specifically we will understand and implement:
- Execution monitoring and taint analysis
- Branch distance computation
- Hill-climbing and genetic algorithms
- Concrete and symbolic (concolic) execution
- Active state machine learning
- Genetic programming
The goal is to better understand and test software using artificial intelligence. Using the taught techniques you will be able to automatically:
- Discover which code is reachable
- Find (security) bugs in software
- Write tests that cover all reachable code
- Reverse engineer a code's functionality
- Patch code to remove bugs and failing tests
Synopsis: In this course, we learn state-of-the-art techniques for testing the correctness of software. The main content of the course will be a lab assignment in which we will apply a modern testing technique to a software component. This testing lab can be performed in three flavours:
- Black-box: testing and monitoring the software’s interactions without trying to understand its internal logic. Fuzz testing, state machine learning, and tainting are typical examples.
- White-box: directly testing the software’s internal logic through code analysis and writing specific test cases. Typical examples are unit testing, concolic testing, mutation testing, and binary analysis.
- Grey-box: testing the software’s logic using its documentation and high-level structure. For instance using repository mining or evolution monitoring.
The different testing techniques will be studied based on recent scientific literature. Pairs of students will then choose one of the flavors and apply one state-of-the-art testing technique to real software code.
Aim: To get knowledge, understanding and skills with respect to modern software testing of real software systems.
Learning outcomes: The student will:
Understand modern AI techniques for software testing.
Be able to implement several such techniques from scratch:
- smart fuzzing (probing software with input to find crashes/bugs),
- symbolic execution (using logic to construct inputs that trigger specific code branches),
- fault localization (given that a program fails, find the line of code responsible for the failure), and
- automated program repair (using a patch library and genetic programming to improve code)
Be able to apply this technology to locate bugs in real-world software implementations.
Lecturers: Dr. A. Panichella and Dr. Ir Sicco Verwer (TUD/EWI)
Examination: First three lab assignments (pass/fail). Final lab (100%).
Education Method: The main part of the course will consist of 3 lab assignments covering the theory (fuzzing&tainting, symbolic execution, automated program repair), and one lab assignment for the application to real software. The students will implement the taught techniques from scratch in the first 3 assignments, which will be scored with a pass/fail. All three assignments need to be passed to complete the course. The final lab will contain a recap from the first three assignments and an application of a state-of-the-art tool on real software. The final lab will be graded and be the final course grade.