Create an openfn test runner

[josephjclark]

This is a lightweight new Runtime Manager which wraps the compiler and runtime into an API that's convenient for automated testing.

Test runner might not be the right name. It might just be like a lightweight runtime.

I want something like this:

import exec from `@openfn/test-runner`

const result = await exec(workflow, options)

The idea is I can pass in a workflow to execute in any format - a js file, a JSON workflow, a yaml project (@openfn/project can help to load the input into something executable) - and it'll be executed with the options I give.

It returns the final state object as JSON.

Maybe it includes an event emitter hook, so that users can catch any runtime events - step start/complete notifications, logs, etc.

Options includes input state, credential values.

The package might also export some test utilities, like maybe some asserting and mocking APIs

It should auto-install adaptors I guess. Alternatively it could rely on being wrapped in an npm project, and import adaptors from there.

[josephjclark]

The hardest thing here is that workflows rely on external services. So it's an integration test helper, not a unit test helper. But it's probably not suitable for CI.

We could expose hooks for mocking adaptor functions. But there's a huge risk of mocking out too much logic in the workflows there

We could also provide APIs to easily mock HTTP requests. But that could be complex for users to set up properly, not to mention that real fixtures are hard to build. And it doesn't work well with non HTTP adaptors - stuff that uses clients.

[josephjclark]

There's another story about unit testing individual functions but I don't think that overlaps with this issue. The idea here is to provided automated testing over a workflow on disk.

See also #646, which is sort of a wider epic around this

[josephjclark]

Thinking about this last night, maybe you DO want tests which run against live systems, and you DO want to run them in CI. I mean why not? It's less crazy than trying to mock out all the endpoints you need