Add a doc describing how to write e2e tests

CHANGELOG.rst

@@ -25,6 +25,12 @@ For admins
   still be set at configure time by passing the option "-DWATCHDOG_SEC=20”
   to cmake, if you want to enable the systemd watchdog.
 
+For developers
+--------------
+- The end-to-end tests have been refactored, cleaned and simplified a lot.
+  A tutorial and a documentation have been written. It should now be easy
+  to write a test that demonstrates a bug or a missing feature.
+
 Version 8.3 - 2018-06-01
 ========================
 

doc/contributing.rst

@@ -50,35 +50,13 @@ Tests
 There are two test suites for biboumi:
 
 - unit tests that can be run simply using `make check`.
-  These tests use the Catch test framework, are written in pure C++
+  These tests use the Catch2 test framework, are written in pure C++
   and they should always succeed, in all possible build configuration.
 
 - a more complex end-to-end test suite. This test suite is written in python3,
   uses a specific IRC server (`charybdis`_), and only tests the most complete
-  biboumi configuration (when all dependencies are used). To run it, you need
+  biboumi configuration (when all dependencies are used).
-  to install various dependencies: refer to fedora’s `Dockerfile.base`_ and
+  Read more about these tests in the specific documentation TODO.
-  `Dockerfile`_ to see how to install charybdis, slixmpp, botan, a ssl
-  certificate, etc.
-
-  Once all the dependencies are correctly installed, the tests are run with
-
-.. code-block:: sh
-
-  make e2e
-
-To run one or more specific tests, you can do something like this:
-
-.. code-block:: sh
-
-  make biboumi && python3 ../tests/end_to_end  self_ping  basic_handshake_success
-
-This will run two tests, self_ping and basic_handshake_success.
-
-To write additional tests, you need to add a Scenario
-into `the __main__.py file`_. If you have problem running this end-to-end
-test suite, or if you struggle with this weird code (that would be
-completely normal…), don’t hesitate to ask for help.
-
 
 All these tests automatically run with various configurations, on various
 platforms, using gitlab CI.

doc/developper.rst

@@ -0,0 +1,240 @@
+########################
+Developper documentation
+########################
+
+End-to-end test suite
+---------------------
+
+A powerful test suite has been developped to test biboumi’s behaviour in
+many scenarios. Its goal is to simulate a real-world usage of biboumi,
+including its interactions with a real IRC server an a real XMPP client.
+
+An IRC server is started, with a specific version and configuration, then,
+for every scenario that we want to test:
+
+- Biboumi is started, with a specific configuration
+- An XMPP “client” starts, communicates with biboumi and checks that
+  biboumi responds in the expected way.
+
+The XMPP client is actually not a real client, it’s a python script that
+uses the slixmpp library to imitate an XMPP server that would transmit the
+stanzas of one client to its component (biboumi). In real life, the
+communication to biboumi is done between the XMPP server and biboumi, but
+since the server just forwards the messages that clients send unmodified,
+we’ll call that “the clients”.
+
+A scenario is a list of functions that will be executed one by one, to
+verify the behaviour of one specific feature. Ideally, they should be
+short and test one specific aspect.
+
+Available functions
+~~~~~~~~~~~~~~~~~~~
+
+.. py:function:: send_stanza(str)
+
+  sends one stanza to biboumi. The stanza is written entirely
+  as a string (with a few automatic replacements). The “from” and “to”
+  values have to be specified everytime, because each stanza can come from
+  different clients and be directed to any IRC server/channel
+
+  .. code-block:: python
+
+    send_stanza("<message from='{jid_one}/{resource_one}' to='#foo@{biboumi_host}' type='groupchat'><body>coucou</body></message>"),
+
+.. py:function:: expect_stanza(xpath[, …])
+
+  Waits for a stanza to be received by biboumi, and checks that this
+  stanza matches one or more xpath. If the stanza doesn’t match all the
+  given xpaths, then the scenario ends and we report that as an error.
+
+  .. code-block:: python
+
+    expect_stanza("/message[@from='#foo@{biboumi_host}/{nick_one}']/body[text()='coucou']",
+                  "/message/delay:delay[@from='#foo@{biboumi_host}']"),
+
+  This waits for exactly 1 stanza, that is compared against 2 xpaths. Here
+  we check that it is a message, that it has the proper `from` value, the
+  correct body, and a <delay/>.
+
+.. py:function:: expect_unordered(list_of_xpaths[, list_of_xpaths, …])
+
+  we wait for more than one stanzas, that could be received in any order.
+  For example, in certain scenario, we wait for two presence stanzas, but
+  it’s perfectly valid to receive them in any order (one is for one
+  client, the other one for an other client). To do that, we pass multiple
+  lists of xpath. Each list can contain one or more xpath (just like
+  `expect_stanza`). When a stanza is received, it is compared with all the
+  xpaths of the first list. If it doesn’t match, it is compared with the
+  xpaths of the second list, and so on. If nothing matchs, it’s an error
+  and we stop this scenario. If the stanza matches with one of the xpath
+  lists, we remove that list, and we wait for the next stanza, until there
+  are no more xpaths.
+
+  .. code-block:: python
+
+    expect_unordered(
+             [
+                  "/presence[@from='#foo%{irc_server_one}/{nick_one}'][@to='{jid_two}/{resource_one}'][@type='unavailable']/muc_user:x/muc_user:item[@nick='Bernard']",
+                  "/presence/muc_user:x/muc_user:status[@code='303']",
+             ],
+             [
+                  "/presence[@from='#foo%{irc_server_one}/{nick_three}'][@to='{jid_two}/{resource_one}']",
+             ],
+             [
+                  "/presence[@from='#foo%{irc_server_one}/{nick_one}'][@to='{jid_one}/{resource_one}'][@type='unavailable']/muc_user:x/muc_user:item[@nick='Bernard']",
+                  "/presence/muc_user:x/muc_user:status[@code='303']",
+                  "/presence/muc_user:x/muc_user:status[@code='110']",
+             ],
+             [
+                  "/presence[@from='#foo%{irc_server_one}/{nick_three}'][@to='{jid_one}/{resource_one}']",
+                  "/presence/muc_user:x/muc_user:status[@code='110']",
+             ],
+      ),
+
+  This will wait for 4 stanzas that could be received in any order.
+
+To avoid many repetitions between each tests, some helpful sequences are
+available, `sequences.connection(…)` and `sequences.connection_tls(…)`.
+They do all the steps that are needed (send and receive stanzas) to
+connect to the component, or an IRC server.
+
+It’s also possible to reuse one simple scenario into an other scenario.
+The most notable example is to start your own scenario with
+`scenarios.simple_channel_join.scenario`, if you need your client to be in
+a channel before you can start your actual scenario. For example if you
+want to test the behaviour of a topic change, you need to first join a
+channel. Since this is a very common patern, it’s simpler to just included
+this very basic scenario at the start of your own scenarios, instead of
+copy pasting the same thing over and over.
+
+Examples of a scenario
+~~~~~~~~~~~~~~~~~~~~~~
+
+First example
+^^^^^^^^^^^^^
+
+Here we’ll describe how to write your own scenario, from scratch. For this, we will take an existing scenario and explain how it was written, line by line.
+
+See for example the scenario tests/end_to_end/scenarios/self_ping_on_real_channel.py
+
+.. code-block:: python
+
+  from scenarios import *
+
+All the tests should start with this import. It imports the file
+tests/end_to_end/scenarios/__init__.py This make all the functions
+available (send_stanza, expect_stanza…) available, as well as some very
+common scenarios that you often need to re-use.
+
+.. code-block:: python
+
+  scenario = (
+    # …
+  )
+
+This is the only required element of your scenario. This object is a tuple of funcion calls OR other scenarios.
+
+.. code-block:: python
+
+  scenarios.simple_channel_join.scenario,
+
+The first line of our scenario is actually including an other existing
+scenario. You can find it at
+tests/end_to_end/scenarios/simple_channel_join.py As its name shows, it’s
+very basic: one client {jid_one}/{resource_one} just joins one room
+#foo%{irc_server_one} with the nick {nick_one}.
+
+Since we want to test the behaviour of a ping to ourself when we are in a
+room, we just join this room without repeating everything.
+
+It is possible to directly insert a scenario inside our scenario without
+having to extract all the steps: the test suite is smart enough to detect
+that and extract the inner steps automatically.
+
+.. code-block:: python
+
+  # Send a ping to ourself
+  send_stanza("<iq type='get' from='{jid_one}/{resource_one}' id='first_ping' to='#foo%{irc_server_one}/{nick_one}'><ping xmlns='urn:xmpp:ping' /></iq>"),
+  expect_stanza("/iq[@from='#foo%{irc_server_one}/{nick_one}'][@type='result'][@to='{jid_one}/{resource_one}'][@id='first_ping']"),
+
+Here we simple send an iq stanza, properly formatted, using the same JIDs
+{jid_one}/{resource_one} and #foo%{irc_server_one}/{nick_one} to ping
+ourself in the room. We them immediately expect one stanza to be received,
+that is the response to our ping. It only contains one single xpath
+because everything we need to check can be expressed in one line.
+
+Note that it is recommended to explain all the steps of your scenario with
+comments. This helps understand what is being tested, and why, without
+having to analyze all the stanza individually.
+
+.. code-block:: python
+
+  # Now join the same room, from the same bare JID, behind the same nick
+  send_stanza("<presence from='{jid_one}/{resource_two}' to='#foo%{irc_server_one}/{nick_one}' />"),
+  expect_stanza("/presence[@to='{jid_one}/{resource_two}'][@from='#foo%{irc_server_one}/{nick_one}']/muc_user:x/muc_user:item[@affiliation='admin'][@role='moderator']",
+                "/presence/muc_user:x/muc_user:status[@code='110']"),
+
+  expect_stanza("/message[@from='#foo%{irc_server_one}'][@type='groupchat'][@to='{jid_one}/{resource_two}']/subject[not(text())]"),
+
+Here we send a presence stanza to join the same channel with an other
+resource (note the {resource_two}). As a result, we expect two stanzas:
+The first stanza (our self-presence) is checked against two xpaths, and
+the second stanza (the empty subject of the room) against only one.
+
+.. code-block:: python
+
+  # And re-send a self ping
+  send_stanza("<iq type='get' from='{jid_one}/{resource_one}' id='second_ping' to='#foo%{irc_server_one}/{nick_one}'><ping xmlns='urn:xmpp:ping' /></iq>"),
+  expect_stanza("/iq[@from='#foo%{irc_server_one}/{nick_one}'][@type='result'][@to='{jid_one}/{resource_one}'][@id='second_ping']"),
+  ## And re-do exactly the same thing, just change the resource initiating the self ping
+  send_stanza("<iq type='get' from='{jid_one}/{resource_two}' id='third_ping' to='#foo%{irc_server_one}/{nick_one}'><ping xmlns='urn:xmpp:ping' /></iq>"),
+  expect_stanza("/iq[@from='#foo%{irc_server_one}/{nick_one}'][@type='result'][@to='{jid_one}/{resource_two}'][@id='third_ping']"),
+
+And finally, we test a second ping, and check that the behaviour is correct that we now have two resources in that channel.
+
+Second example
+^^^^^^^^^^^^^^
+
+Sometimes we want to do more with the received stanzas. For example we
+need to extract some values from the received stanzas, to reuse them in
+future stanzas we send. The most obvious example is iq IDs, that we need
+to extract, to reuse them in our response.
+
+Let’s use for example the tests/end_to_end/scenarios/execute_incomplete_hello_adhoc_command.py scenario:
+
+.. code-block:: python
+
+  from scenarios import *
+
+  scenario = (
+    send_stanza("<iq type='set' id='hello-command1' from='{jid_one}/{resource_one}' to='{biboumi_host}'><command xmlns='http://jabber.org/protocol/commands' node='hello' action='execute' /></iq>"),
+    expect_stanza("/iq[@type='result']/commands:command[@node='hello'][@sessionid][@status='executing']",
+                  "/iq/commands:command/commands:actions/commands:complete",
+                  after = save_value("sessionid", extract_attribute("/iq[@type='result']/commands:command[@node='hello']", "sessionid"))
+                  ),
+
+Here is where the magic happens: as an additional argument to the
+expect_stanza function, we pass an other function (callback) with the
+“after=” keyword argument. This “after” callback gets called once the
+expected stanza has been received and validated. Here we use
+`save_value(key, value)`. This function just saves a value in our global
+values that can be used with “send_stanza”, associated with the given
+“key”. For example if you do `save_value("something_important", "blah")`
+then you can use `{something_important}` in any future stanza that you
+send and it will be replaced with “blah”.
+
+But this is only useful if we can save some value that we extract from the
+stanza. That’s where `extract_attribute(xpath, attribute_name)` comes into
+play. As the first argument, you pass an xpath corresponding to one
+specific node of the XML that is received, and the second argument is just
+the name of the attribute whose value you want.
+
+Here, we extract the value of the “sessionid=” in the node `<iq
+type='result'><commands:command node='hello' sessionid='…' /></iq>`, and
+we save that value, globally, with the name “sessionid”.
+
+.. code-block:: python
+
+  send_stanza("<iq type='set' id='hello-command2' from='{jid_one}/{resource_one}' to='{biboumi_host}'><command xmlns='http://jabber.org/protocol/commands' node='hello' sessionid='{sessionid}' action='complete'><x xmlns='jabber:x:data' type='submit'></x></command></iq>"),
+
+Here we send a second iq, to continue our ad-hoc command, and we use {sessionid} to indicate that we are continuing the session we started before.

doc/index.rst

@@ -21,3 +21,4 @@ XMPP client as if these channels were XMPP MUCs.
    admin
    user
    contributing
+   developper