Difference between revisions of "Armadeus Integration Test"
m (developping...) |
(→Changelog) |
||
Line 5: | Line 5: | ||
==Changelog== | ==Changelog== | ||
+ | september 2013: Installation of a test server with Hudson / Jenkins | ||
==Introduction== | ==Introduction== |
Revision as of 22:18, 30 September 2013
Contents
Armadeus Test Integration
Create automatic test & Validation tools
Changelog
september 2013: Installation of a test server with Hudson / Jenkins
Introduction
Test Strategy
- Production tests: done (out of the scope of this document)
- Compilation tests
For the time being, compilation tests are manual. The idea is to compile Armadeus project from the head of SVN for each target to check compilation success. success condition is compilation pass and binary images (u-boot, linux, and rootfs) are available in binaries directory. Test could be run every day.
- On target test run
Here again tests have to be automatized. The idea is to control a target over serial port and ethernet to run test cases and log test results. First priroty is to validate linux kernel, modules and rootfs packages present in the default configuration for the target. Yes, it is doable!
Compilation Tests
Use a build farm to run the tests periodicaly (for each commit). Hudson or buildbot are good pretenders. There are many competitors here CI Matrix
The packages have to be tested one at a time to avoid dependency problems. <- ! not for compilation ?!!
Do you mean download packages with make defconfig or test dependencies of each package?
- checkout the head of svn
- run build for each supported target
- check build result ?? point 4 is not sufficient ? : Yes, It could be. This point aims to check the makefile.
- check presence of binary images
- log error and warning (only the last xxx lines of the console ?) or at your convenience the full log of errors. It is open
- publish results on internet (pass or fail)
On target tests
Some test tools have to be added to the target to run the tests:
- capabilities to reset the board (watchdog or wired signal)
- Add some testsuites to the target rootfs. Some packages like busybox, uclibc provide their own testsuite
- Armadeus packages need their own stategy (TBD)
Compilation Test Plan
On target Test Plan
Principle:
- reset the board
- take control of the target within uboot
- upload new binary images (with test tools)
- reset the board
- pass uboot test. dont know if uboot any test strategy
- boot linux
- connect as user root (or test)
- run integrated testsuites (uclibc, busybox...) and log report
- run test cases for remaining packages present in the target.
- Log error and warning
- publish results on internet
Busybox testsuite - howto
Busibox provides its own testsuite. The following description prepare and run busybox tests:
- copy busybox .config file to the target rootfs user directory (root or /home/test tbd)
- copy busybox testsuite directory to the target rootfs (/root or..)
- build and upload new rootfs to the target (script to be finalized. uboot1.3.4 pathch 3.x provides script "update-all")
- boot linux and connect as root or user test
- cd testsuite
- run shell script: ./testsuite *
- log test results
screenshoot:
> ./runtest *
PASS: basename-does-not-remove-identical-extension
PASS: basename-works
FAIL: bunzip2-reads-from-standard-input
FAIL: bunzip2-removes-compressed-file
FAIL: bzcat-does-not-remove-compressed-file
SKIPPED: bunzip2 (not built)
...
uclibc test - howto
Seems to require compilation on target. :-(
- Within 'make menuconfig' -> toolchain enable 'Compile and install uClibc tests'