Testing with Hoverfly and Java Part 3: State

Previously we simulated a delay scenario using Hoverfly. Now it’s time to dive deeper and go for a state based testing. By doing a stateful simulation we can change the way the tests endpoints behave based on how the state changed.

 

Hoverfly does have a state capability. State in a hoverfly simulation is like a map. Initially it is empty but you can define how it will get populated per request.

Our strategy would be to have a request that initializes the state and then specifies other requests that change that state.

public class SimulationStateTests {

	private Hoverfly hoverfly;

	@BeforeEach
	void setUp() {
		var simulation = SimulationSource.dsl(service("http://localhost:8085")
				.get("/initialize")
				.willReturn(success("{\"initialized\":true}", "application/json")
						.andSetState("shouldSucceed", "true")
				)
				.get("/state")
				.withState("shouldSucceed", "false")
				.willReturn(serverError().andSetState("shouldSucceed", "true"))
				.get("/state")
				.withState("shouldSucceed", "true")
				.willReturn(success("{\"username\":\"test-user\"}", "application/json")
						.andSetState("shouldSucceed", "false"))

		);

		var localConfig = HoverflyConfig.localConfigs().disableTlsVerification().asWebServer().proxyPort(8085);
		hoverfly = new Hoverfly(localConfig, SIMULATE);
		hoverfly.start();
		hoverfly.simulate(simulation);
	}

	@AfterEach
	void tearDown() {
		hoverfly.close();
	}

}

Unfortunately on the state we can only specify values in a key value fashion and not by passing a function for a key.
However with the right workaround many scenarios could be simulated.

In the example we first initialize the state and the we issue requests that behave differently based on the state, but also they do change the state.

So we expect to have a continuous first succeed and then fail mode, which can be depicted in the following test.

	@Test
	void testWithState() {
		var client = HttpClient.newHttpClient();
		var initializationRequest = HttpRequest.newBuilder()
				.uri(URI.create("http://localhost:8085/initialize"))
				.build();
		var initializationResponse = client.sendAsync(initializationRequest, HttpResponse.BodyHandlers.ofString())
				.thenApply(HttpResponse::body)
				.join();
		Assertions.assertEquals("{\"initialized\":true}", initializationResponse);

		var statefulRequest = HttpRequest.newBuilder()
				.uri(URI.create("http://localhost:8085/state"))
				.build();

		for (int i = 0; i < 100; i++) {
			var response = client.sendAsync(statefulRequest, HttpResponse.BodyHandlers.ofString())
					.join();

			int statusCode = i % 2 == 0 ? 200 : 500;

			Assertions.assertEquals(statusCode, response.statusCode());
		}
	}

That’s all about stateful simulation. On the next part we shall proceed on Hoverfly matchers

Testing with Hoverfly and Java Part 2: Delays

On the previous post we implemented json and Java based Hoverfly scenarios..
Now it’s time to dive deeper and use other Ηoverfly features.

A big part of testing has to do with negative scenarios. One of them is delays. Although we always mock a server and we are successful to reproduce erroneous scenarios one thing that is key to simulate in todays microservices driven world is delay.

So let me make a server with a 30 secs delay.

public class SimulateDelayTests {

	private Hoverfly hoverfly;

	@BeforeEach
	void setUp() {
		var simulation = SimulationSource.dsl(service("http://localhost:8085")
				.get("/delay")
				.willReturn(success("{\"username\":\"test-user\"}", "application/json").withDelay(30, TimeUnit.SECONDS)));

		var localConfig = HoverflyConfig.localConfigs().disableTlsVerification().asWebServer().proxyPort(8085);
		hoverfly = new Hoverfly(localConfig, SIMULATE);
		hoverfly.start();
		hoverfly.simulate(simulation);
	}

	@AfterEach
	void tearDown() {
		hoverfly.close();
	}

}

Let’s add the Delay Test

@Test
void testWithDelay() {
   var client = HttpClient.newHttpClient();
   var request = HttpRequest.newBuilder()
         .uri(URI.create("http://localhost:8085/delay"))
         .build();
   var start = Instant.now();
   var res = client.sendAsync(request, HttpResponse.BodyHandlers.ofString())
         .thenApply(HttpResponse::body)
         .join();
   var end = Instant.now();
   Assertions.assertEquals("{\"username\":\"test-user\"}", res);

   var seconds = Duration.between(start, end).getSeconds();
   Assertions.assertTrue(seconds >= 30);
}

Delay simulation is there, up and running, so let’s try to simulate timeouts.

	@Test
	void testTimeout() {
		var client = HttpClient.newHttpClient();
		var request = HttpRequest.newBuilder()
				.uri(URI.create("http://localhost:8085/delay"))
				.timeout(Duration.ofSeconds(10))
				.build();
		assertThrows(HttpTimeoutException.class, () -&gt; {
					try {
						client.sendAsync(request, HttpResponse.BodyHandlers.ofString()).join();
					} catch (CompletionException ex) {
						throw ex.getCause();
					}
				}

		);
	}

That’s it we got delays and timeouts!
Other test scenarios should contain state which is covered on the next tutorial.

Testing with Hoverfly and Java Part 1: Get started with Simulation Mode

These days a major problem exists when it comes to testing code that has to do with various cloud services where test tools are not provided.
For example although you might have the tools for local Pub/Sub testing, including Docker images you might not have anything that can Mock BigQuery.

This causes an issue when it comes to the CI jobs, as testing is part of the requirements, however there might be blockers on testing with the actual service. The case is, you do need to cover all the pessimistic scenarios you need to be covered (for example timeouts).

And this is where Hoverfly can help.

Hoverfly is a lightweight, open source API simulation tool. Using Hoverfly, you can create realistic simulations of the APIs your application depends on

Our first examples will have to do with simulating just a web server. The first step is to add the Hoverfly dependency.

    <dependencies>
        <dependency>
            <groupId>io.specto</groupId>
            <artifactId>hoverfly-java</artifactId>
            <version>0.12.2</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

Instead of using the Hoverfly docker image we shall use the Java Library for some extra flexibility.

We got two options on configuring the Hoverfly simulation mode. One is through the Java dsl and the other one is through json.
Let’s cover both.

The example below uses the Java DSL. We spin up hoverfly on 8085 and load this configuration.

class SimulationJavaDSLTests {

	private Hoverfly hoverfly;

	@BeforeEach
	void setUp() {
		var simulation = SimulationSource.dsl(service("http://localhost:8085")
				.get("/user")
				.willReturn(success("{\"username\":\"test-user\"}", "application/json")));

		var localConfig = HoverflyConfig.localConfigs().disableTlsVerification().asWebServer().proxyPort(8085);
		hoverfly = new Hoverfly(localConfig, SIMULATE);
		hoverfly.start();
		hoverfly.simulate(simulation);
	}

	@AfterEach
	void tearDown() {
		hoverfly.close();
	}

	@Test
	void testHttpGet() {
		var client = HttpClient.newHttpClient();
		var request = HttpRequest.newBuilder()
				.uri(URI.create("http://localhost:8085/user"))
				.build();
		var res = client.sendAsync(request, HttpResponse.BodyHandlers.ofString())
				.thenApply(HttpResponse::body)
				.join();
		Assertions.assertEquals("{\"username\":\"test-user\"}",res);
	}
}

Now let’s do the same with Json. Instead of manually trying things with json we can make the code do the work for us.

var simulation = SimulationSource.dsl(service("http://localhost:8085")
			.get("/user")
			.willReturn(success("{\"username\":\"test-user\"}", "application/json")));

var simulationStr = simulation.getSimulation()
System.out.println(simulationStr);

We can get the JSON generated by the Java DSL. The result would be like this.

{
  "data": {
    "pairs": [
      {
        "request": {
          "path": [
            {
              "matcher": "exact",
              "value": "/user"
            }
          ],
          "method": [
            {
              "matcher": "exact",
              "value": "GET"
            }
          ],
          "destination": [
            {
              "matcher": "exact",
              "value": "localhost:8085"
            }
          ],
          "scheme": [
            {
              "matcher": "exact",
              "value": "http"
            }
          ],
          "query": {},
          "body": [
            {
              "matcher": "exact",
              "value": ""
            }
          ],
          "headers": {},
          "requiresState": {}
        },
        "response": {
          "status": 200,
          "body": "{\"username\":\"test-user\"}",
          "encodedBody": false,
          "templated": true,
          "headers": {
            "Content-Type": [
              "application/json"
            ]
          }
        }
      }
    ],
    "globalActions": {
      "delays": []
    }
  },
  "meta": {
    "schemaVersion": "v5"
  }
}

Let’s place this one on the resources folder of tests under the name simulation.json

And with some code changes we get exactly the same result.


public class SimulationJsonTests {

	private Hoverfly hoverfly;

	@BeforeEach
	void setUp() {
		var simulationUrl = SimulationJsonTests.class.getClassLoader().getResource("simulation.json");
		var simulation = SimulationSource.url(simulationUrl);

		var localConfig = HoverflyConfig.localConfigs().disableTlsVerification().asWebServer().proxyPort(8085);
		hoverfly = new Hoverfly(localConfig, SIMULATE);
		hoverfly.start();
		hoverfly.simulate(simulation);
	}

	@AfterEach
	void tearDown() {
		hoverfly.close();
	}

	@Test
	void testHttpGet() {
		var client = HttpClient.newHttpClient();
		var request = HttpRequest.newBuilder()
				.uri(URI.create("http://localhost:8085/user"))
				.build();
		var res = client.sendAsync(request, HttpResponse.BodyHandlers.ofString())
				.thenApply(HttpResponse::body)
				.join();
		Assertions.assertEquals("{\"username\":\"test-user\"}",res);
	}

}

Also sometimes there is the need of combining simulations regardless they json or Java ones. This can also be facilitated by loading more that one simulations.

	@Test
	void testMixedConfiguration() {
		var simulationUrl = SimulationJsonTests.class.getClassLoader().getResource("simulation.json");
		var jsonSimulation = SimulationSource.url(simulationUrl);


		var javaSimulation = SimulationSource.dsl(service("http://localhost:8085")
				.get("/admin")
				.willReturn(success("{\"username\":\"test-admin\"}", "application/json")));

		hoverfly.simulate(jsonSimulation, javaSimulation);

		var client = HttpClient.newHttpClient();
		var jsonConfigBasedRequest = HttpRequest.newBuilder()
				.uri(URI.create("http://localhost:8085/user"))
				.build();
		var userResponse = client.sendAsync(jsonConfigBasedRequest, HttpResponse.BodyHandlers.ofString())
				.thenApply(HttpResponse::body)
				.join();
		Assertions.assertEquals("{\"username\":\"test-user\"}",userResponse);

		var javaConfigBasedRequest = HttpRequest.newBuilder()
				.uri(URI.create("http://localhost:8085/admin"))
				.build();
		var adminResponse = client.sendAsync(javaConfigBasedRequest, HttpResponse.BodyHandlers.ofString())
				.thenApply(HttpResponse::body)
				.join();
		Assertions.assertEquals("{\"username\":\"test-admin\"}",adminResponse);
	}

That’s it, we are pretty setup to continues exploring Hoverfly and it’s capabilities.

Testing has become Mandatory (and there are no more excuses)

There are many posts out there about the value of automated testing and why it is a must through the life cycle of a software product.

Most people and teams totally agree with this statement, so what goes wrong?

It is just a prototype

It seems to be a valid statement, but if we consider how many times a prototype made it into production then it is not so valid.
In fact most prototype codebase ends up to production because this is the goal of a prototype.
Adding tests in a prototype phase makes absolutely sense.

Too difficult to test

Yes there cases where testing is really difficult due to limitations. For example the android and he iphone emulators do not give you the ability
to set accelerometer events, or you use a service which provides you with no test utilities at all, or a testing environment.
Even if mocking would not make an absolute test case, it really can assist you on making a specification on how things work.
The other scenario is testing being hard to implement due to the codebase.
Consider this as an indicator that things are not simple enough. If it is just your codebase then your are lucky, It is up to you to make it more testable and more simple.

Too much to do, too little time

This is our weakest spot and the one that we are most prone too succumb.
Suppose you develop a smartphone application. In case of manual testing each mistake will cost you 3-5 minutes. Starting the emulator or even worse deploy to a physical device, load the application and press some buttons to create events, and in case of an error repeat and loose other 3-5 minutes.
In case you develop a server application things might get even worse. Connect to the sever, upload the application, check that that the upload was successful and then manually test that it works ok, and in case of an error repeat and loose some time again.
Also keep in mind that in case of projects with more than one developers involved, mistakes are more common to happen.
We tend to believe that the best case will happen where everything will work as expected.
It might work most of the time but in case of a problem you loose big. We end up being hooked on an “It’s ok all it needs is another version upload” mode.
Next time this happens just count how much time you end up loosing by testing manually until everything is ok.
Then estimate how much time tests will cost you and how much time you win in such cases.

Everything changes so rapidly

Yes a project on its initial phase, will have a completely different codebase in two months than its initial one.
But since big changes are made more errors are likely to occur. A minimal amount of tests ensures that basic features would continue to work.

All in all testing is not as hard as it used to be.
Nowadays almost every service or utility that we use comes with some test utils.
As the software industry becomes more challenging, codebases without tests would become extinct.
Pick whatever methodology you want but just do it. Your life will become much easier.

Implement custom JMeter samplers

As we proceed on different architectures and implementations the need for versatile stress testing tools rises.

Apache Jmeter is one the most well known tools when it comes to load testing. It supports many protocols such as ftp http tcp and also it can be used easily for distributed testing.

Jmeter also provides you with an easy way to create custom samplers. For example if you need to load test a http endpoint that requires a specific procedure for signing the headers then a custom sampler will come in handy.

The goal is to implement a custom sampler project which will load test a simple function.

I use gradle for this example.

group 'com.gkatzioura.jmeter'
version '1.0-SNAPSHOT'

apply plugin: 'java'

sourceCompatibility = 1.6

repositories {
    mavenCentral()
}


dependencies {
    compile 'org.apache.jmeter:ApacheJMeter_java:2.11'
    compile 'org.json:json:20151123'
    testCompile group: 'junit', name: 'junit', version: '4.11'
}

task copySample(type:Copy,dependsOn:[build]) {

    copy {
        from project.buildDir.getPath()+'/libs/jmeter-sampler-1.0-SNAPSHOT.jar'
        into 'pathtoyourjmeterinstallation/apache-jmeter-2.13/lib/ext/'
    }
}

I include the ApacheJMeter dependency on the project since the sampler will have to extend the AbstractJavaSamplerClient.
The copySample task will copy the jar to the lib/ext path of Jmeter where all samplers reside.

A simple function will be called by the sampler

package com.gkatzioura.jmeter;

/**
 * Created by gkatzioura on 30/1/2016.
 */
public class FunctionalityForSampling {

    public String testFunction(String arguement1,String arguement2) throws Exception {

        if (arguement1.equals(arguement2)) {
            throw new Exception();
        }

        else return arguement1+arguement2;
    }

}

The CustomSampler class extends the AbstractJavaSamplerClient class and invokes the testFunction.
By overriding the getDefaultParameters function we can apply default parameters that can be used with the request.

package com.gkatzioura.jmeter;

import org.apache.jmeter.config.Arguments;
import org.apache.jmeter.protocol.java.sampler.AbstractJavaSamplerClient;
import org.apache.jmeter.protocol.java.sampler.JavaSamplerContext;
import org.apache.jmeter.samplers.SampleResult;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.io.Serializable;

/**
 * Created by gkatzioura on 30/1/2016.
 */
public class CustomSampler extends AbstractJavaSamplerClient implements Serializable {

    private static final String METHOD_TAG = "method";
    private static final String ARG1_TAG = "arg1";
    private static final String ARG2_TAG = "arg2";

    private static final Logger LOGGER = LoggerFactory.getLogger(CustomSampler.class);

    @Override
    public Arguments getDefaultParameters() {

        Arguments defaultParameters = new Arguments();
        defaultParameters.addArgument(METHOD_TAG,"test");
        defaultParameters.addArgument(ARG1_TAG,"arg1");
        defaultParameters.addArgument(ARG2_TAG,"arg2");

        return defaultParameters;
    }

    @Override
    public SampleResult runTest(JavaSamplerContext javaSamplerContext) {

        String method = javaSamplerContext.getParameter(METHOD_TAG);
        String arg1 = javaSamplerContext.getParameter(ARG1_TAG);
        String arg2 = javaSamplerContext.getParameter(ARG2_TAG);

        FunctionalityForSampling functionalityForSampling = new FunctionalityForSampling();

        SampleResult sampleResult = new SampleResult();
        sampleResult.sampleStart();

        try {
            String message = functionalityForSampling.testFunction(arg1,arg2);
            sampleResult.sampleEnd();;
            sampleResult.setSuccessful(Boolean.TRUE);
            sampleResult.setResponseCodeOK();
            sampleResult.setResponseMessage(message);
        } catch (Exception e) {
            LOGGER.error("Request was not successfully processed",e);
            sampleResult.sampleEnd();
            sampleResult.setResponseMessage(e.getMessage());
            sampleResult.setSuccessful(Boolean.FALSE);

        }

        return sampleResult;
    }

}

After compile is finished the jar created must be copied to the lib/ext directory of the JMeter installation home.
Also in case there are more dependencies that have to be imported they should also be copied to the lib path of the JMeter installation home

Once the process is complete by adding Java Sampler to a JMeter Thread Group we can choose our custom sampler.

Screenshot from 2016-01-31 01:30:06

You can also find the source code here.