OTel Java Instrumentation Tests

Table of Contents

Last updated: 2024-12-01

Context

In 2023 I came across an issue in the OpenTelemetry Java Instrumentation repo for converting the various instrumentation test suites from groovy to java. I had been studying the code base for a bit to better familiarize myself with the capabilities and features of the OTel java agent to compare with the commercial product I was using at work (Datadog). I was also interested in learning more about how the instrumentation worked under the hood. I had experience using opentracing to perform manual instrumentation to extend the datadog agent tracing, but wanted to learn more about the bytecode manipulation and how various libraries were instrumented.

The task of converting tests from one language to another seemed fun in the sense that it would give me an opportunity to learn more about the code base, learn more about JVM languages (I had never worked with groovy before), play with different technologies and libraries (web servers, databases, clients) in the test setups, and would be relatively “easy” in the sense that I wouldn’t be adding anything net-new, just refactoring existing code. This felt like a great, tractable way to start dipping my feet into the project.

I have now converted hundreds of these tests, and found myself with a decent collection of notes for various things that I am constantly returning to and referencing. I will document them here for easier finding and in case they might be useful to reference or point people to.

Some of these might be very obvious, and some are not necessarily specific to tests for this project, but I find them useful to review periodically and thus included them.

Other useful resources

General tips

  • Reference the style guide for all PRs
    • Check ordering of members, methods, internal classes etc. Code should be structured in a way that it can be read top down.
    • Use static imports for things like AttributeKey and other helper methods
  • Make sure visibility of tests is accurate - Intellij often defaults new classes to public, but test classes rarely should ever be public
    • static testing (InstrumentationExtensions) methods should be private
  • Setup and cleanup methods should be package private
  • The groovy spock test methods setupSpec and cleanupSpec are equivalent to BeforeAll and Afterall, where setup and cleanup are equivalent to BeforeEach and AfterEach (docs)
  • If using an abstract class to share the test setup code, and it is used for multiple tests that require spinning up docker containers or other resources, do not use static fields for things like ports, or it can cause test flakyness due to conflicts.
  • Setup up pre-commit validation. The builds take a long time to run, you don’t want to waste time waiting around for a build to fail on a linting error.
    • To run things manually: ./gradlew spotlessApply and ./gradlew checkstyleTest
  • For test code, prioritize readability over using things like helper methods to reduce code verbosity (within reason)
  • Leverage Parameterized Tests when possible
  • Always use assertThat for assertions
  • Include blank assertions when there are no attributes
    • .hasAttributes(Attributes.empty()) or .hasTotalAttributeCount(0)
  • When a span has an exception, instead of asserting all the individual exception data, you can also do .hasException(exception.getCause())
    • This won’t check the stack trace, but it will check the exception type and message
    • or use catchThrown and use that value in assertion
  • JUnit 5 test lifecycle methods need not be public! (setUp / cleanUp)
  • Converting groovy tests will sometimes need reflection to compensate for groovy’s ability to reference private methods/attributes
  • Javaagent tests are typically in the same package as javaagent code.
  • Break large refactors into smaller PRs to allow for easier reviewing.

@RegisterExtension

There are some classes that are designed to assist with test setup and cleanup, and these can be annotated with @RegisterExtension to use them in your test classes.

InstrumentationExtension

For example, InstrumentationExtensions are used for intercepting telemetry data produced by the agent or library.

The extension classes provide several useful methods, such as waitAndAssertTraces and waitAndAssertMetrics, that you can use in your test cases to verify that the correct telemetry has been produced.

It is a common pattern to have an Abstract base test class that has the tests and a method testing() of type InstrumentationExtension.

public abstract class MyAbstractTest {
  protected abstract InstrumentationExtension testing();

  @Test
  void testSomething() {
    // ...
  }
}

Then in your javaagent and library modules, you extend this class and specify the specific implementation

class MyLibraryTest extends MyAbstractTest {

  @RegisterExtension
  static final InstrumentationExtension testing = LibraryInstrumentationExtension.create();

  @Override
  protected InstrumentationExtension testing() {
    return testing;
  }
}

There are 4 different implementations you can use.

AgentInstrumentationExtension

For writing javaagent instrumentation tests:

@RegisterExtension
static final InstrumentationExtension testing =
    AgentInstrumentationExtension.create();

LibraryInstrumentationExtension

For writing library instrumentation tests:

@RegisterExtension
static final InstrumentationExtension testing =
    LibraryInstrumentationExtension.create();

HttpServerInstrumentationExtension

This implementation sets up infrastructure, such as a test HTTP client for use with AbstractHttpServerUsingTest

// For agent
@RegisterExtension
static final InstrumentationExtension testing =
    HttpServerInstrumentationExtension.forAgent();
    
// for library
@RegisterExtension
static final InstrumentationExtension testing =
    HttpServerInstrumentationExtension.forLibrary();

HttpClientInstrumentationExtension

This implementation sets up infrastructure, such as a test HTTP server for use with AbstractHttpClientTest

// For agent
@RegisterExtension
static final InstrumentationExtension testing =
    HttpClientInstrumentationExtension.forAgent();
    
// for library
@RegisterExtension
static final InstrumentationExtension testing =
    HttpClientInstrumentationExtension.forLibrary();

AutoCleanupExtension

For situations where you want to register some hooks to do something following the execution of a test, you can use AutoCleanupExtension. This is useful for things like cleaning up resources, or ensuring that certain cleanup actions are taken after a test.

Register the extension in your test class:

@RegisterExtension static final AutoCleanupExtension cleanup = AutoCleanupExtension.create();

Then where you have a closable resource (AutoCloseable), you can register it with the extension:

ConfigurableApplicationContext context = app.run();
cleanup.deferCleanup(context);

Clearing setup data

Sometimes there are things that need to be done to set up for a test that you don’t necessarily want to write trace assertions for, and therefore we want to clear them before asserting traces we actually care about.

When clearing data, it is safest to set an explicit wait condition for the number of traces you want to clear in order to avoid race conditions and flaky tests:

..code generating 2 traces..
  
// Block and wait for traces, then clear
testing.waitForTraces(2);  
testing.clearData();

testLatestDeps

You can run individual tests with testLatestDeps by passing a flag -PtestLatestDeps=true:

./gradlew -PtestLatestDeps=true :instrumentation:hibernate:hibernate-4.0:javaagent:version5Test

Http Server test helpers

There are a few different classes to help set up for server tests:

AbstractHttpServerUsingTest - provides the scaffolding of a test server alone, doesn’t include any tests.

AbstractHttpServerTest - provides scaffolding plus a suite of tests that will run on the server.

There are a few important methods to be implemented for both:

  protected abstract SERVER setupServer() throws Exception;

  protected abstract void stopServer(SERVER server) throws Exception;

  protected abstract String getContextPath();

And you have to make sure you register the right extension:

  @RegisterExtension
  public static final InstrumentationExtension testing =
      HttpServerInstrumentationExtension.forAgent();

Example if you just need the server:

class TomcatAsyncTest extends AbstractHttpServerUsingTest<Tomcat> {

  @Override
  protected Tomcat setupServer() {
    ... tomcat setup code
    return tomcat;
  }

  @Override
  public void stopServer(Tomcat server) throws LifecycleException {
    server.stop();
    server.destroy();
  }

  /* this method is only needed for AbstractHttpServerUsingTest */
  @Override
  public String getContextPath() {
    return "/tomcat-context";
  }

Example AbstractHttpServerTest:

class TomcatAsyncTest extends AbstractHttpServerTest<Tomcat> {

  @RegisterExtension
  public static final InstrumentationExtension testing =
      HttpServerInstrumentationExtension.forAgent();

  @Override
  protected Tomcat setupServer() {
    ... tomcat setup code
    return tomcat;
  }

  @Override
  public void stopServer(Tomcat server) throws LifecycleException {
    server.stop();
    server.destroy();
  }
  ...
}

Parameterized Tests

Tests can be parameterized in JUnit 5 by using the @ParameterizedTest annotation and the other variants.

Class source parameterized test

You can create a class that will generate arguments by implementing org.junit.jupiter.params.provider. ArgumentsProvider and then using @ArgumentsSource to provide the arguments to the test.

static final class ConfigArgs implements ArgumentsProvider {
  @Override
  public Stream<? extends Arguments> provideArguments(ExtensionContext context) {
    return Stream.of(
        Arguments.of("none", SpanSuppressionStrategy.NONE),
        Arguments.of("NONE", SpanSuppressionStrategy.NONE),
        Arguments.of(null, SpanSuppressionStrategy.SEMCONV));
  }
}

@ParameterizedTest
@ArgumentsSource(ConfigArgs.class)
void shouldParseConfig(String value, SpanSuppressionStrategy expectedStrategy) {
  assertEquals(expectedStrategy, SpanSuppressionStrategy.fromConfig(value));
}

Method source parameterized test

@ParameterizedTest
@MethodSource("provideArguments")
void testConfiguration(String value, List<String> expected) {
  ...
}

private static Stream<Arguments> provideArguments() {
  return Stream.of(
      Arguments.of(null, DEFAULT_ANNOTATIONS),
      Arguments.of(" ", Collections.emptyList()),
      Arguments.of("some.Invalid[]", Collections.emptyList()),
      Arguments.of("Duplicate ;Duplicate ;Duplicate; ", Collections.singletonList("Duplicate")));
}

CSV Source parameterized test

@ParameterizedTest
@CsvSource(
    value = {
      "directChannel,application.directChannel process",
      "executorChannel,executorChannel process"
    },
    delimiter = ',')
public void testShouldPropagateContext(String channelName, String interceptorSpanName) {
  ...
}

Conditional Assertions

Sometimes you will need to make assertions that have different paths based on testLatestDeps or other parameterized test inputs. Instead of putting a big if/else block and duplicating all kinds of things, you can create a list of assertions for the base case, and then conditionally add the deltas in each code path fork.

TraceAssert

List<Consumer<TraceAssert>> traceAsserts = 
    new ArrayList<>(
        Arrays.asList(
            trace ->
                trace.hasSpansSatisfyingExactly(
                    span -> span.hasName("parent").hasKind(SpanKind.INTERNAL).hasNoParent(),
                    span ->
                        span.hasName("GET")
                            .hasKind(SpanKind.CLIENT)
                            .hasParent(trace.getSpan(0))
                            .hasAttributesSatisfyingExactly(
                                equalTo(SemanticAttributes.NETWORK_TYPE, "ipv4"),
                                equalTo(NetworkAttributes.NETWORK_PEER_ADDRESS, "127.0.0.1"),
                                equalTo(NetworkAttributes.NETWORK_PEER_PORT, port),
                                equalTo(SemanticAttributes.DB_SYSTEM, "redis"),
                                equalTo(SemanticAttributes.DB_STATEMENT, "GET NON_EXISTENT_KEY"))
                            .hasEventsSatisfyingExactly(
                                event -> event.hasName("redis.encode.start"),
                                event -> event.hasName("redis.encode.end"))));

if (testCallback()) {
    traceAsserts.add(
        trace ->
            trace.hasSpansSatisfyingExactly(
                span ->
                    span.hasName("callback1")
                        .hasKind(SpanKind.INTERNAL)
                        .hasParent(trace.getSpan(0)),
                span ->
                    span.hasName("callback2")
                        .hasKind(SpanKind.INTERNAL)
                        .hasParent(trace.getSpan(0))));
}

getInstrumentationExtension()
    .waitAndAssertTraces(traceAsserts);

SpanDataAssert

List<Consumer<SpanDataAssert>> assertions =
    new ArrayList<>(
        asList(
          span -> span.hasName(spanName)
              .hasKind(SpanKind.INTERNAL)
              .hasParent(trace.getSpan(parentIndex))
    ));

if (expectedException != null) {
  assertions.add(
      span -> span.hasStatus(StatusData.error())
          .hasException(expectedException));
}

AttributeAssertion

trace -> trace.hasSpansSatisfyingExactly(
    span -> {
        List<AttributeAssertion> attributes =
            new ArrayList<>(
                asList(
                    equalTo(stringKey("aws.agent"), "java-aws-sdk"),
                    equalTo(stringKey("aws.queue.url"),
                        "http://localhost:" + sqsPort + "/000000000000/testSdkSqs"),
                    satisfies(stringKey("aws.request.id"),
                        val ->
                            val.satisfiesAnyOf(
                                v -> assertThat(v).isEqualTo(
                                    "00000000-0000-0000-0000-000000000000"),
                                v -> assertThat(v).isEqualTo("UNKNOWN"))),
                    equalTo(stringKey("rpc.system"), "aws-api"),
                    equalTo(stringKey("rpc.service"), "Sqs"),
                    equalTo(stringKey("rpc.method"), "SendMessage"),
                    equalTo(HttpAttributes.HTTP_REQUEST_METHOD, "POST"),
                    equalTo(HttpAttributes.HTTP_RESPONSE_STATUS_CODE, 200),
                    satisfies(UrlAttributes.URL_FULL,
                        v -> v.startsWith("http://localhost" + sqsPort)),
                    equalTo(ServerAttributes.SERVER_ADDRESS, "localhost"),
                    equalTo(ServerAttributes.SERVER_PORT, sqsPort),
                    equalTo(MessagingIncubatingAttributes.MESSAGING_SYSTEM, "AmazonSQS"),
                    equalTo(MessagingIncubatingAttributes.MESSAGING_DESTINATION_NAME,
                        "testSdkSqs"),
                    equalTo(MessagingIncubatingAttributes.MESSAGING_OPERATION, "publish"),
                    satisfies(MessagingIncubatingAttributes.MESSAGING_MESSAGE_ID,
                        v -> v.isInstanceOf(String.class))
                ));

        if (captureHeaders) {
        attributes.add(
            satisfies(stringKey("messaging.header.test_message_header"),
                v -> v.isEqualTo(ImmutableList.of("test"))));
        }

        span.hasName("testSdkSqs publish")
            .hasKind(SpanKind.PRODUCER)
            .hasNoParent()
            .hasAttributesSatisfyingExactly(attributes);
    }),

Experimental Attribute Assertions

Sometimes you will have test suites that will be configured with a system property that changes a configuration, which results in different telemetry. In these cases, we need a conditional check for behavior based on whether that configuration is set.

You can setup the separate tests in gradle like this:

tasks {
  withType<Test>().configureEach {
    systemProperty("collectMetadata", findProperty("collectMetadata")?.toString() ?: "false")
  }

  val testExperimental by registering(Test::class) {
    testClassesDirs = sourceSets.test.get().output.classesDirs
    classpath = sourceSets.test.get().runtimeClasspath

    jvmArgs("-Dotel.instrumentation.guava.experimental-span-attributes=true")
    systemProperty("metadataConfig", "otel.instrumentation.guava.experimental-span-attributes=true")
  }

  check {
    dependsOn(testExperimental)
  }
}

For basic assertions we can use helper methods:

class ExperimentalTestHelper {
  private static final boolean EXPERIMENTAL_ATTRIBUTES_ENABLED =
      Boolean.getBoolean("otel.instrumentation.guava.experimental-span-attributes");

  @Nullable
  static <T> T experimental(T value) {
    return EXPERIMENTAL_ATTRIBUTES_ENABLED ? value : null;
  }
  
  private ExperimentalTestHelper () {}
}

And then in the code:

.hasAttributesSatisfyingExactly(
    equalTo(stringKey("camel.uri"), experimental("direct://input"))),

Or for situations where we have a satisfies() we can do something like this:

satisfies(
    longKey("lettuce.command.results.count"),
    val -> {
      if (EXPERIMENTAL_ATTRIBUTES_ENABLED) {
        val.isGreaterThan(100);
      }
    }

satisfies(longKey("grpc.received.message_count"), val -> val.satisfiesAnyOf(
    v -> assertThat(ExperimentalTestHelper.isEnabled).isFalse(),
    v -> assertThat(v).isGreaterThan(0)
))

Or if we want to make a helper for that too:

class ExperimentalTestHelper {
  private static final boolean isEnabled =
      Boolean.getBoolean("otel.instrumentation.grpc.experimental-span-attributes");

  static final AttributeKey<Long> GRPC_RECEIVED_MESSAGE_COUNT =
      longKey("grpc.received.message_count");
  static final AttributeKey<Long> GRPC_SENT_MESSAGE_COUNT = longKey("grpc.sent.message_count");

  static AttributeAssertion experimentalSatisfies(
      AttributeKey<Long> key, Consumer<? super Long> assertion) {
    return satisfies(
        key,
        val -> {
          if (isEnabled) {
            val.satisfies(assertion::accept);
          }
        });
  }

  private ExperimentalTestHelper() {}
}

and then used like:

.hasAttributesSatisfyingExactly(
    experimentalSatisfies(
        GRPC_RECEIVED_MESSAGE_COUNT,
        v -> assertThat(v).isGreaterThan(0)),
    experimentalSatisfies(
        GRPC_SENT_MESSAGE_COUNT, v -> assertThat(v).isGreaterThan(0)),

Groovy Assertions

Groovy has some interesting ways of handling assertions for different conditions that look a bit different in java.

Regex matching

Groovy:

span(1) {
  name ~/BatchJob splitJob\.splitFlowStep[12]/
  ...
}

Java:

span ->
    span.satisfies(
            spanData ->
                assertThat(spanData.getName())
                    .matches("BatchJob splitJob.splitFlowStep[12]"))
    ...

Multiple conditions

Groovy:

"aws.requestId" { it == "00000000-0000-0000-0000-000000000000" || it == "UNKNOWN" }

Java:

satisfies(SERVER_ADDRESS, v -> v.matches("somebucket.localhost|localhost")),

or

satisfies(stringKey("aws.request.id"),
    val ->
        val.satisfiesAnyOf(
            v -> assertThat(v).isEqualTo("00000000-0000-0000-0000-000000000000"),
            v -> assertThat(v).isEqualTo("UNKNOWN"))),

Exception / Error Event Assertions

Groovy has a simplistic way of asserting exceptions, with an errorEvent helper method, for example:

assertTraces(1) {
  trace(0, 2) {
    span(0) {
      name getContextPath() + "/greeting.xhtml"
      kind SpanKind.SERVER
      hasNoParent()
      status ERROR
      errorEvent(ex.class, ex.message)
    }
    handlerSpan(it, 1, span(0), "#{greetingForm.submit()}", ex)
  }
}

In Java, you can do it verbosely with hasEventsSatisfyingExactly():

.hasEventsSatisfyingExactly(
    event ->
        event
            .hasName("exception")
            .hasAttributesSatisfyingExactly(
                equalTo(
                    ExceptionAttributes.EXCEPTION_TYPE,
                    ex.getClass().getName()),
                equalTo(
                    ExceptionAttributes.EXCEPTION_STACKTRACE,
                    ex.getClass().getName()),
                satisfies(
                    ExceptionAttributes.EXCEPTION_MESSAGE,
                    message ->
                        message.endsWith(ex.getMessage())))

Or more simply with .hasException(exception):

IllegalStateException expectedException = new IllegalStateException("submit exception");

testing.waitAndAssertTraces(
    trace ->
        trace.hasSpansSatisfyingExactly(
            span ->
                span.hasName(getContextPath() + "/greeting.xhtml")
                    .hasKind(SpanKind.SERVER)
                    .hasNoParent()
                    .hasStatus(StatusData.error())
                    .hasException(expectedException),
            span -> handlerSpan(trace, 0, "#{greetingForm.submit()}", expectedException)));

Note: the .hasException() approach does not check the stack trace, only the exception type and message.

Invoke Dynamic (indy) and breakpoints

For instrumentation that is indy compliant you can set breakpoints in your IDE and have them hit when running tests. This is useful for debugging and understanding how the instrumentation is working. In order for this to work, you must add the -PtestIndy=true flag to your test config.

Camel AWS tests

There are a handful of AWS camel tests that are set to be ignored in the test suite due to the need to run them against real world AWS services. When working on converting these tests, there were a few things I needed to work through:

  • AWS only allows the PurgeQueue command once every 60 seconds, so if multiple tests try and call that command during cleanup in succession, it can lead to unexpected state and cause tests to fail. I found that I needed to run each test in isolation as opposed to running them all at once.
  • I occasionally needed to extend the delay on the S3 tests to allow time for things to propagate for more reliable results
  • There was a TrustStore being overridden in the otel.java-conventions.gradle.kts file that was causing certificate errors when trying to interact with AWS (Unable to execute HTTP request: PKIX path building failed) that I
    needed to prevent from being set in order to work with these tests.
    • I needed to set and use the gradle project description instead of name because the name is set implicitly by gradle using the directory name, and in this case it is a generic javaagent, so that did not seem like a safe
      way to target this.