Refine config and docs for sytest
This PR adds the "passed but expected fail" tests to testfile, as well as instructions on finding which tests to add for new PRs.
MODIFY
docs/sytest.md
docs/sytest.md
# SyTest
Dendrite uses [SyTest](https://github.com/matrix-org/sytest) for its
-integration testing. When creating a new PR, add the test IDs that your PR
-should allow to pass to `testfile` in dendrite's root directory. Not all PRs
-need to make new tests pass. If we find your PR should be making a test pass we
-may ask you to add to that file, as generally Dendrite's progress can be
+integration testing. When creating a new PR, add the test IDs (see below) that
+your PR should allow to pass to `testfile` in dendrite's root directory. Not all
+PRs need to make new tests pass. If we find your PR should be making a test pass
+we may ask you to add to that file, as generally Dendrite's progress can be
tracked through the amount of SyTest tests it passes.
+
+## Finding out which tests to add
+
+We recommend you run the tests locally by manually setting up SyTest or using a
+SyTest docker image. After running the tests, a script will print the tests you
+need to add to `testfile` for you.
+
+You should proceed after you see no build problems for dendrite after running:
+
+```sh
+./build.sh
+```
+
+### Manually Setting up SyTest
+
+Make sure you have Perl v5+ installed, and get SyTest with:
+
+(Note that this guide assumes your SyTest checkout is next to your
+`dendrite` checkout.)
+
+```sh
+git clone -b develop https://github.com/matrix-org/sytest
+cd sytest
+./install-deps.pl
+```
+
+Set up the database:
+
+```sh
+sudo -u postgres psql -c "CREATE USER dendrite PASSWORD 'itsasecret'"
+sudo -u postgres psql -c "CREATE DATABASE sytest_template OWNER dendrite"
+mkdir -p "server-0"
+cat > "server-0/database.yaml" << EOF
+args:
+ user: dendrite
+ database: dendrite
+ host: 127.0.0.1
+type: pg
+EOF
+```
+
+Run the tests:
+
+```sh
+./run-tests.pl -I Dendrite::Monolith -d ../dendrite/bin -W ../dendrite/testfile -O tap --all | tee results.tap
+```
+
+where `tee` lets you see the results while they're being piped to the file.
+
+Once the tests are complete, run the helper script to see if you need to add
+any newly passing test names to `testfile` in the project's root directory:
+
+```sh
+../dendrite/show-expected-fail-tests.sh results.tap
+```
+
+If the script prints nothing/exits with 0, then you're good to go.
+
+### Using a SyTest Docker image
+
+Ensure you have the latest image for SyTest, then run the tests:
+
+```sh
+docker pull matrixdotorg/sytest-dendrite
+docker run --rm -v /path/to/dendrite/:/src/ matrixdotorg/sytest-dendrite
+```
+
+where `/path/to/dendrite/` should be replaced with the actual path to your
+dendrite source code. The output should tell you if you need to add any tests to
+`testfile`.

ADD
show-expected-fail-tests.sh
+#! /bin/bash
+
+results_file=$1
+passed_but_expected_fail=$(grep ' # TODO passed but expected fail' ${results_file} | sed -E 's/^ok [0-9]+ (\(expected fail\) )?//' | sed -E 's/( \([0-9]+ subtests\))? # TODO passed but expected fail$//')
+tests_to_add=""
+already_in_testfile=""
+
+fail_build=0
+while read -r test_id; do
+ grep "${test_id}" testfile > /dev/null 2>&1
+ if [ "$?" != "0" ]; then
+ tests_to_add="${tests_to_add}${test_id}\n"
+ fail_build=1
+ else
+ already_in_testfile="${already_in_testfile}${test_id}\n"
+ fi
+done <<< "${passed_but_expected_fail}"
+
+if [ -n "${tests_to_add}" ]; then
+ echo "ERROR: The following passed tests are not present in testfile. Please append them to the file:"
+ echo -e "${tests_to_add}"
+fi
+
+if [ -n "${already_in_testfile}" ]; then
+ echo "WARN: Tests in testfile still marked as expected fail:"
+ echo -e "${already_in_testfile}"
+fi
+
+exit ${fail_build}

Refine config and docs for sytest (#714)

This PR adds the "passed but expected fail" tests to testfile, as well as instructions on finding which tests to add for new PRs.