feat(testing): add assertInlineSnapshot()

[WWRS]

closes #3301

Updating the snapshots is not as magic as in Jest since Jest uses Babel's AST parser to find where to insert the snapshot and then Prettier to format. I instead look for the magic template

`CREATE`

which cannot appear anywhere in the file except in the places that the snapshots should be added. Replace each with the corresponding test and use deno fmt to format.

I chose this template because I think it's unlikely to appear unless the intent is to create the snapshot: A user wanting a string containing CREATE would use 's or "s. The output snapshot adds quotes around strings, so it's only possible to get an input and output that are the same by using a custom serializer.

The expectedSnapshots in snapshot_test.ts were automatically generated by assertInlineSnapshot!

[codecov]

Codecov Report

Attention: Patch coverage is 74.34555% with 49 lines in your changes missing coverage. Please review.

Project coverage is 94.98%. Comparing base (68a8460) to head (e02ee4d).
Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
testing/snapshot.ts	74.34%	48 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6530      +/-   ##
==========================================
- Coverage   95.08%   94.98%   -0.11%     
==========================================
  Files         576      576              
  Lines       43337    43536     +199     
  Branches     6466     6489      +23     
==========================================
+ Hits        41208    41352     +144     
- Misses       2089     2143      +54     
- Partials       40       41       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[kt3k]

Thanks for the PR, but this seems only supporting the initial creation of snapshot. In my view the capability of updating the snapshots are essential for snapshot testing tool. Can we also support the updating somehow? (I guess we can do that only by using AST analysis as jest does)

[WWRS]

I agree that this limitation of assertInlineSnapshot means it's not as nice as Jest's.

You can update a snapshot by manually replacing the existing one with `CREATE` and running the test. This does force users to follow the commonly recommended practice of ensuring each updated snapshot should actually have been updated. Also, users always have the option of assertSnapshot() for tests that will need to be updated frequently enough that manual updating would be a hassle.

If we wanted to support updating failing snapshots automatically, it's likely that the best solution would involve AST analysis. I don't think Deno exposes its AST parser, and Jest uses Babel which is presumably something we cannot do here. So a system using AST analysis is not likely to be ready anytime soon.

I'm not sure, but I suspect that even without the ability to update automatically, users might find this feature useful. On the other hand, I do recognize the need to maintain high quality throughout the std library, and shipping an incomplete feature is not great either.

[stefnotch]

So if I understand this correctly, there are 4 different options here

• Accept the simple implementation
• Use a Deno.lint plugin to update the snapshots, since they have an AST inspection and mutation API.
  • That plugin should ideally ship with Deno by default.
  • Bonus points for getting IDE integration for free. The light bulb would be able to suggest updating the snapshot.
• Make Deno.assertSnapshot a built-in feature of Deno, just like Deno.test and Deno.bench

[WWRS]

Also an option would be to expose an AST parsing API as a built-in feature of Deno or as a library. From looking at #2355 it seems like this functionality will not be added to Deno but is in deno_swc. Would it be reasonable to include that as a dependency here?

I think adding assertSnapshot to Deno is not very likely. All the assertions are in std/assert so it doesn't make a ton of sense to have just one or two assertion functions in default Deno.

For deno_lint, I don't really know what our options look like there. Really all we need to do is find assertInlineSnapshot (and later expect.toMatchInlineSnapshot) and manipulate the third param. But like adding to default Deno, it seems unlikely we would want support for just one assertion function in the default linter.

[stefnotch]

deno_swc has not been updated in 3 years, so I wouldn't rely on it, even if denoland/deno#2355 points at it.

Deno's current parser is a Rust library, and thus a bit trickier to use from JS land. Hence my suggestion of adding the necessary plumbing to Deno itself.
https://github.com/denoland/deno_ast

[stefnotch]

I was about to write a comment about how the lint plugin option wouldn't work that well, until I realised something.

Deno.lint.runPlugin is a function that Deno provides in test mode.
Which exactly fits the bill of what we want! That's amazing!

So I quickly whipped up a prototype which makes use of that to give us inline snapshots!

Entire project is this, run it as per usual with deno test and deno test -A -- --update
temp-deno-linter-for-snapshots.zip

Video.mp4

And just the code. I'm okay with you copy-pasting and adapting this straight into Deno, and am releasing this under CC0 and MIT.

import { serialize, SnapshotOptions } from "@std/testing/snapshot";
import { equal } from "@std/assert/equal";
import { AssertionError } from "@std/assert/assertion-error";

const isUpdateMode = Deno.args.some((arg) =>
  arg === "--update" || arg === "-u"
);

interface SnapshotUpdateRequest {
  fileName: string;
  lineNumber: number;
  columnNumber: number;
  actual: string;
}

// Batch all writes until the very end, and then update all files at once
globalThis.addEventListener("unload", () => {
  updateSnapshots();
});
function updateSnapshots() {
  if (updateRequests.length === 0) {
    return;
  }
  console.log(`Updating ${updateRequests.length} snapshots...`);
  const filesToUpdate = Map.groupBy(updateRequests, (v) => v.fileName);

  for (const [fileName, requests] of filesToUpdate) {
    const fileContents = Deno.readTextFileSync(fileName);
    const pluginRunResults = Deno.lint.runPlugin(
      makeSnapshotUpdater(requests),
      "dummy.ts",
      fileContents,
    );
    const fixes = pluginRunResults.flatMap((v) => v.fix ?? []);
    if (fixes.length !== requests.length) {
      console.error(
        "Something went wrong, not all update requests found their snapshot",
      );
    }
    // Apply the fixes
    fixes.sort((a, b) => a.range[0] - b.range[0]);
    let output = "";
    let lastIndex = 0;
    for (const fix of fixes) {
      output += fileContents.slice(lastIndex, fix.range[0]);
      output += wrapForJs(fix.text ?? "");
      lastIndex = fix.range[1];
    }
    output += fileContents.slice(lastIndex);
    Deno.writeTextFileSync(fileName, output);
  }
}

const updateRequests: SnapshotUpdateRequest[] = [];

export function assertInlineSnapshot<T>(
  actual: T,
  expected: string,
  options?: SnapshotOptions<T>,
) {
  const _serialize = options?.serializer ?? serialize;
  const _actual = _serialize(actual);

  if (equal(_actual, expected)) {
    return;
  }
  if (isUpdateMode) {
    // Uses the V8 stack trace API to get the line number where this function was called
    const oldStackTrace = (Error as any).prepareStackTrace;
    try {
      const stackCatcher = { stack: null as SnapshotUpdateRequest | null };
      (Error as any).prepareStackTrace = (
        _err: unknown,
        stack: unknown[],
      ): SnapshotUpdateRequest | null => {
        const callerStackFrame = stack[0] as any;
        if (callerStackFrame.isEval()) return null;
        return {
          fileName: callerStackFrame.getFileName(),
          lineNumber: callerStackFrame.getLineNumber(),
          columnNumber: callerStackFrame.getColumnNumber(),
          actual: _actual,
        };
      };
      // Capture the stack that comes after this function.
      Error.captureStackTrace(stackCatcher, assertInlineSnapshot);
      // Forcibly access the stack, and note it down
      const request = stackCatcher.stack;
      if (request !== null) {
        const status = Deno.permissions.requestSync({
          name: "write",
          path: request.fileName,
        });
        if (status.state !== "granted") {
          console.error(
            `Please allow writing to ${request.fileName} for snapshot updating.`,
          );
        } else {
          updateRequests.push(request);
        }
      }
    } finally {
      (Error as any).prepareStackTrace = oldStackTrace;
    }
  } else {
    throw new AssertionError("Assertion failed blabla");
  }
}

/// <reference lib="deno.unstable" />
/**
 * Makes a Deno.lint plugin that can find inline snapshots.
 * Also deals with multiple `assertInlineSnapshot` in a single line.
 */
function makeSnapshotUpdater(
  updateRequests: SnapshotUpdateRequest[],
): Deno.lint.Plugin {
  const linesToUpdate = Map.groupBy(updateRequests, (v) => v.lineNumber);

  return {
    name: "snapshot-updater-plugin",
    rules: {
      "update-snapshot": {
        create(context) {
          return {
            'CallExpression[callee.name="assertInlineSnapshot"]'(
              node: Deno.lint.CallExpression,
            ) {
              // Find the update request that corresponds to this snapshot.
              // Successful snapshots don't have an update request.
              const callPosition = toLineAndColumnNumber(
                node.range[0],
                context.sourceCode.text,
              );
              const endColum = callPosition.column +
                (node.range[1] - node.range[0]);
              const lineUpdateRequests = linesToUpdate.get(callPosition.line);
              if (lineUpdateRequests === undefined) {
                return;
              }
              const updateRequest = lineUpdateRequests.find((v) =>
                callPosition.column <= v.columnNumber &&
                v.columnNumber <= endColum
              );
              if (updateRequest === undefined) {
                return;
              }

              context.report({
                node,
                message: "",
                fix(fixer) {
                  return fixer.replaceText(
                    node.arguments[1],
                    updateRequest.actual,
                  );
                },
              });
            },
          };
        },
      },
    },
  };
}

/**
 * Takes an index and returns the 1-based line number and 1-based column number */
function toLineAndColumnNumber(index: number, text: string) {
  const textBefore = text.slice(0, index);
  // TODO: Verify that Chrome's V8 uses the same logic for returning line numbers. What about the other line terminators?
  const lineBreakCount = (textBefore.match(/\n/g) || []).length;

  // Also deals with the first line by making use of the -1 return value
  const lastLineBreak = textBefore.lastIndexOf("\n");
  return {
    line: lineBreakCount + 1,
    column: (index - lastLineBreak),
  };
}

function wrapForJs(str: string) {
  return "`" + escapeStringForJs(str) + "`";
}

function escapeStringForJs(str: string) {
  return str
    .replace(/\\/g, "\\\\")
    .replace(/`/g, "\\`")
    .replace(/\$/g, "\\$");
}

[stefnotch]

@WWRS Would you be willing to integrate the Deno.lint approach into this PR? I'd really appreciate it, since that'd save me the effort of creating and polishing a pull request for this particular feature.
I'm asking since your PR already has documentation, unit tests, code cleanups and other necessary bits.

If yes, here are some quick pointers

• I think we could get away with dropping the first argument for assertInlineSnapshot. One already gets the correct file from the stack trace.
• If you have any ideas on how to simplify parts of the code, I'd be delighted to hear them. I couldn't figure out a simpler way of doing the "find the correct assertInlineSnapshot". The simple approach of counting wouldn't work here, since the linter doesn't know in which order the inline snapshot functions are called.
• Re-assigning assertInlineSnapshot to a new variable is unsupported. So assertMonochromeInlineSnapshot wouldn't work. I wonder if there's a good solution for that.
• Wrapping assertInlineSnapshot in an extra function would lead to very unexpected results. I'm not sure if one can do much about that.
• I wonder what happens if one types deno test --watch -A -- --update

[WWRS]

I'm not 100% sold on using the error throwing location as the way to find where to insert the snapshot, it seems a bit obscure and dependent on V8. As noted, we would need to double check how V8 counts line numbers, but also how it counts unicode. (For reference, Jest does use this error throwing approach.)

For finding the right snapshot to update, would the numbering system that's in the current implementation of assertSnapshot work? It seems like tests are always run from top to bottom within the same file. I'm not sure if this works when running a subset of the tests, but if it breaks here, it might have issues in assertSnapshot as well.

For whether or not we need a context argument, if we were to remove the argument here, we would probably want to make a similar change in assertSnapshot to keep the call signatures parallel. I'm not sure how messy that would get, do you think it would be possible? Also, #6541 is worth thinking about: If we were to add a context-injected expect in bdd tests, then we could make toMatchSnapshot and toMatchInlineSnapshot use that expect and get its context for free, though possibly at the cost of reduced support outside bdd tests.

I'd like to hear some other opinions on the tradeoffs of the two proposed implementations. As noted above, the magic-word implementation lacks a core capability because it requires screenshots to be manually flagged for updates. Are the missing capabilities of the lint implementation (assertMonochromeInlineSnapshot or wrapping in a helper function) also core?

(It does occur to me that if we want to use this in expect.toMatchInlineSnapshot we'll probably have to update the lint rule.)

[WWRS]

I checked, the existing assertSnapshot-style counting system does work. This means we can avoid using errors to find where to insert snapshots, though in this case we do still need the TestContext.

So I do have a working version using the lint implementation, but I'll wait to push it until we decide which we prefer.

[stefnotch]

Thank you so much for taking a look at this. Those are some very good points, so I did some reading.

Error Throwing

Yes, the error throwing approach relies on V8 APIs. I wish we had a standard API for this https://github.com/tc39/proposal-error-stacks

For now, I think it's fine, because updating the snapshots relies on filesystem access. So, it's limited to server-side JS runtimes, which generally implement the V8 APIs. Even Bun does.

And yes, we absolutely have to check how Unicode gets counted for the column number. We'll have to verify whether the following behaves as expected.

/* 🐈‍⬛🐈‍⬛🐈‍⬛🐈 */ assertInlineSnapshot(2 + 3, `'5'`)

On that note, the linter's ranges also rely on an important detail: They're reported in UTF-16 code-somethings, which matches Javascript string indexing. So at least that part is sensible.

Numbering system

assertSnapshot relies on a numbering system. It computes a stable ID for each assertSnapshot call. That ID only changes when one renames the tests, or starts calling assertSnapshot in a different order.
To update a snapshot, it runs the test, and saves the new value with the ID.

(Amusingly, it means that if(Math.random() < 0.5) { await assertSnapshot(t, 1); } else { await assertSnapshot(t, 2); }) would repeatedly break. But that's a non-deterministic test, so it deserves to break.)

This does not work for assertInlineSnapshot.
To update a snapshot, it has to find the correct place in the source code.

One quick example where the numbering system would not work is

test("foo", () => {
  if(false) {
    assertInlineSnapshot("never happens");
  }
  assertInlineSnapshot("wait, why am I not getting updated");
});

I'm sure one can also think of a counterexample with promises, like calling assertInlineSnapshot in a .then

Context argument

Good point, that'd be an odd inconsistency.

So #3964 has an open issue about implementing a expect(foo).toMatchSnapshot() API.

But clearly the normal snapshotting API needs a test context. So I looked at what Jest and Vitest do.

Vitest passes the context to expect when run synchronously, and tells you to use the correct expect when you have async tests.
https://vitest.dev/guide/snapshot.html#use-snapshots

test('math is easy', async ({ expect }) => {
  expect(2 + 2).toMatchSnapshot()
})

Jest instead assumes that it runs within Node.js, and makes use of the AsyncLocalStorage API.
jestjs/jest#14139
Deno also implements that API https://docs.deno.com/api/node/async_hooks/~/AsyncLocalStorage
The downside of it is that it doesn't work in browsers.

I think you're better than I am at judging which of the trade-offs is the best option here.

assertMonochromeInlineSnapshot

There are four options for dealing with the limitation.

1. Only expose it as expect(foo).toMatchInlineSnapshot('bar'). Now the temptation to alias it is much lower.
2. Don't look for a specific work in the syntax tree, only look for a function call at the correct spot. As in, update it as long as there is a foo(bar, ...) in the expected position.
3. Add a setting where one can pass in the name to look for, like

const assertMonochromeInlineSnapshot = createAssertInlineSnapshot<string>({
    serializer: stripAnsiCode,
    name: "assertMonochromeInlineSnapshot" // Users are very likely to forget this, and it would not survive minification or re-assignment.
  });

4. Just document the limitation. Inline snapshots are wizardry anyways.

[stefnotch]

I just found out that there's a proposal for async context tracking https://github.com/tc39/proposal-async-context

If we had that, then we'd no longer need the first argument for the regular assertSnapshot function. And we could finally perfectly re-implement the expect().toMatchSnapshot() function from Jest.

[stefnotch]

@WWRS I figured I'd check back in: How is it going? Which trade-offs should we choose? Is there anything I could do to help move this PR forward?