203 lines
6.6 KiB
Plaintext
203 lines
6.6 KiB
Plaintext
FFmpeg Automated Testing Environment
|
||
************************************
|
||
|
||
Table of Contents
|
||
*****************
|
||
|
||
FFmpeg Automated Testing Environment
|
||
1 Introduction
|
||
2 Using FATE from your FFmpeg source directory
|
||
3 Submitting the results to the FFmpeg result aggregation server
|
||
4 FATE makefile targets and variables
|
||
4.1 Makefile targets
|
||
4.2 Makefile variables
|
||
4.3 Examples
|
||
|
||
|
||
1 Introduction
|
||
**************
|
||
|
||
FATE is an extended regression suite on the client-side and a means for
|
||
results aggregation and presentation on the server-side.
|
||
|
||
The first part of this document explains how you can use FATE from
|
||
your FFmpeg source directory to test your ffmpeg binary. The second
|
||
part describes how you can run FATE to submit the results to FFmpeg's
|
||
FATE server.
|
||
|
||
In any way you can have a look at the publicly viewable FATE results
|
||
by visiting this website:
|
||
|
||
`http://fate.ffmpeg.org/'
|
||
|
||
This is especially recommended for all people contributing source
|
||
code to FFmpeg, as it can be seen if some test on some platform broke
|
||
with their recent contribution. This usually happens on the platforms
|
||
the developers could not test on.
|
||
|
||
The second part of this document describes how you can run FATE to
|
||
submit your results to FFmpeg's FATE server. If you want to submit your
|
||
results be sure to check that your combination of CPU, OS and compiler
|
||
is not already listed on the above mentioned website.
|
||
|
||
In the third part you can find a comprehensive listing of FATE
|
||
makefile targets and variables.
|
||
|
||
2 Using FATE from your FFmpeg source directory
|
||
**********************************************
|
||
|
||
If you want to run FATE on your machine you need to have the samples in
|
||
place. You can get the samples via the build target fate-rsync. Use
|
||
this command from the top-level source directory:
|
||
|
||
make fate-rsync SAMPLES=fate-suite/
|
||
make fate SAMPLES=fate-suite/
|
||
|
||
The above commands set the samples location by passing a makefile
|
||
variable via command line. It is also possible to set the samples
|
||
location at source configuration time by invoking configure with
|
||
`--samples=<path to the samples directory>'. Afterwards you can invoke
|
||
the makefile targets without setting the SAMPLES makefile variable.
|
||
This is illustrated by the following commands:
|
||
|
||
./configure --samples=fate-suite/
|
||
make fate-rsync
|
||
make fate
|
||
|
||
Yet another way to tell FATE about the location of the sample
|
||
directory is by making sure the environment variable FATE_SAMPLES
|
||
contains the path to your samples directory. This can be achieved by
|
||
e.g. putting that variable in your shell profile or by setting it in
|
||
your interactive session.
|
||
|
||
FATE_SAMPLES=fate-suite/ make fate
|
||
|
||
Do not put a '~' character in the samples path to indicate a home
|
||
directory. Because of shell nuances, this will cause FATE to fail.
|
||
|
||
NOTE
|
||
|
||
To use a custom wrapper to run the test, pass `--target-exec' to
|
||
`configure' or set the TARGET_EXEC Make variable.
|
||
|
||
3 Submitting the results to the FFmpeg result aggregation server
|
||
****************************************************************
|
||
|
||
To submit your results to the server you should run fate through the
|
||
shell script `tests/fate.sh' from the FFmpeg sources. This script needs
|
||
to be invoked with a configuration file as its first argument.
|
||
|
||
tests/fate.sh /path/to/fate_config
|
||
|
||
A configuration file template with comments describing the individual
|
||
configuration variables can be found at `doc/fate_config.sh.template'.
|
||
|
||
Create a configuration that suits your needs, based on the
|
||
configuration template. The `slot' configuration variable can be any
|
||
string that is not yet used, but it is suggested that you name it
|
||
adhering to the following pattern `ARCH-OS-COMPILER-COMPILER VERSION'.
|
||
The configuration file itself will be sourced in a shell script,
|
||
therefore all shell features may be used. This enables you to setup the
|
||
environment as you need it for your build.
|
||
|
||
For your first test runs the `fate_recv' variable should be empty or
|
||
commented out. This will run everything as normal except that it will
|
||
omit the submission of the results to the server. The following files
|
||
should be present in $workdir as specified in the configuration file:
|
||
|
||
* configure.log
|
||
|
||
* compile.log
|
||
|
||
* test.log
|
||
|
||
* report
|
||
|
||
* version
|
||
|
||
When you have everything working properly you can create an SSH key
|
||
pair and send the public key to the FATE server administrator who can
|
||
be contacted at the email address <fate-admin@ffmpeg.org>.
|
||
|
||
Configure your SSH client to use public key authentication with that
|
||
key when connecting to the FATE server. Also do not forget to check the
|
||
identity of the server and to accept its host key. This can usually be
|
||
achieved by running your SSH client manually and killing it after you
|
||
accepted the key. The FATE server's fingerprint is:
|
||
|
||
`RSA'
|
||
d3:f1:83:97:a4:75:2b:a6:fb:d6:e8:aa:81:93:97:51
|
||
|
||
`ECDSA'
|
||
76:9f:68:32:04:1e:d5:d4:ec:47:3f:dc:fc:18:17:86
|
||
|
||
If you have problems connecting to the FATE server, it may help to
|
||
try out the `ssh' command with one or more `-v' options. You should get
|
||
detailed output concerning your SSH configuration and the authentication
|
||
process.
|
||
|
||
The only thing left is to automate the execution of the fate.sh
|
||
script and the synchronisation of the samples directory.
|
||
|
||
4 FATE makefile targets and variables
|
||
*************************************
|
||
|
||
4.1 Makefile targets
|
||
====================
|
||
|
||
`fate-rsync'
|
||
Download/synchronize sample files to the configured samples
|
||
directory.
|
||
|
||
`fate-list'
|
||
Will list all fate/regression test targets.
|
||
|
||
`fate'
|
||
Run the FATE test suite (requires the fate-suite dataset).
|
||
|
||
4.2 Makefile variables
|
||
======================
|
||
|
||
`V'
|
||
Verbosity level, can be set to 0, 1 or 2.
|
||
* 0: show just the test arguments
|
||
|
||
* 1: show just the command used in the test
|
||
|
||
* 2: show everything
|
||
|
||
`SAMPLES'
|
||
Specify or override the path to the FATE samples at make time, it
|
||
has a meaning only while running the regression tests.
|
||
|
||
`THREADS'
|
||
Specify how many threads to use while running regression tests, it
|
||
is quite useful to detect thread-related regressions.
|
||
|
||
`THREAD_TYPE'
|
||
Specify which threading strategy test, either `slice' or `frame',
|
||
by default `slice+frame'
|
||
|
||
`CPUFLAGS'
|
||
Specify CPU flags.
|
||
|
||
`TARGET_EXEC'
|
||
Specify or override the wrapper used to run the tests. The
|
||
`TARGET_EXEC' option provides a way to run FATE wrapped in
|
||
`valgrind', `qemu-user' or `wine' or on remote targets through
|
||
`ssh'.
|
||
|
||
`GEN'
|
||
Set to `1' to generate the missing or mismatched references.
|
||
|
||
4.3 Examples
|
||
============
|
||
|
||
make V=1 SAMPLES=/var/fate/samples THREADS=2 CPUFLAGS=mmx fate
|
||
|
||
|
||
|
||
Local Variables:
|
||
coding: utf-8
|
||
End:
|