Motivation: Software is one of the most complex artifacts of mankind has ever created, but complexity is the enemy of correctness. Modern software testing tools use a multitude of automated techniques geared toward correct computer code, amongst others:
- Concrete and symbolic (concolic) testing
- Execution monitoring and taint analysis
- Mutation testing
- Reverse-engineering and binary analysis
- Search-based and model-based test case generation
- Software repository mining
- State machine learning or inference
- Software evolution monitoring
- Web testing
Synopsis: In this course, we learn state-of-the-art techniques for testing the correctness of software. The main content of the course will be a lab assignment in which we will apply a modern testing technique to a software component. This testing lab can be performed in three flavours:
- Black-box: testing and monitoring the software’s interactions without trying to understand its internal logic. Fuzz testing, state machine learning, and tainting are typical examples.
- White-box: directly testing the software’s internal logic through code analysis and writing specific test cases. Typical examples are unit testing, concolic testing, mutation testing, and binary analysis.
- Grey-box: testing the software’s logic using its documentation and high-level structure. For instance using repository mining or evolution monitoring.
The different testing techniques will be studied based on recent scientific literature. Pairs of students will then choose one of the flavors and apply one state-of-the-art testing technique to real software code.
Aim: To get knowledge, understanding and skills with respect to modern software testing of real software systems.
Learning outcomes: The student will acquire:
- Understanding of different advanced software testing techniques
- Ability to test and improve the correctness of existing systems
- Ability to analyze and reverse-engineer software code
Lecturers: Dr Ir Sicco Verwer (TUD/EWI) and Prof Dr Ir Arie van Deursen/Dr Ir Andy Zaidman (TUD/EWI)
Examination: Lab work 50% including a written report 30% and presentation of the results 20%.
Contents: The main part of the course will consist of a lab assignment where the students will need to test a real software system using a state-of-the-art techniques described in the scientific literature. This will be supported by a few lectures covering the tools and techniques that are required for these tasks, such as SAGE, Pitest, CWSandbox, LearnLib, Torx, Sulley, Crawljax, Valgrind, and Z3. There will be many instruction sessions where students can work on their assignment and ask the teachers for assistance.
Core text: Various papers from the literature