Skip to content

Output syntax check proposal

I have been experimenting with output syntax checking, pursuant to #12 (closed). This turns out to be quite hard, mostly because we want to be able to point the user to the syntax error in their output.

I got a thing working with the following technique:

  • Use syn's parsing to parse the output, discarding the parsed result
  • If the parsing succeeds, carry on as normal. Otherwise:
  • Write out a copy of the template expansion to a file with a generated name in $OUT_DUR
  • Instead of returning a tokenstream containing our output, return:
    1. The compile error we got from syn
    2. An include!... invocation to include the file

This seems to cause my editor, at least, to find the file and be able to jump to the syntax error. You typically get the same error twice, once from 1 and once from 2. 1 shows you the original span in your source code files (which may not be very enlightening); 2 shows you the location in the expanded output.

Difficulties that remain

This requires syn's full feature, which increases compile times. We can't make this feature-optional because an additional syntax check is not additive.

The least bad way for constructing the filename seems to involve sha3. (The business with writing out to a file can be feature-optional, so this can too.)

In order to construct the filename, derive-adhoc-macros needs access to OUT_DIR which means it has to have an (empty) build.rs, which would have a small impact on compile times (due to restricted build parallelism). I think this can't be feature-optional because a build.rs can't be optional.

For this to work, the template will have to specify what kind of syntactic element(s) it is going to expand to. We can easily support items and expr. We can support stmts but the write-to-file trick will have to be disabled for it, because the compiler expects that an include! in statement context will expand to an expression, so include! is never useable for things that contain items plus bindings plus expression statements etc. I'm imagining: define_derive_adhoc! { MyTemplate as items = ... } and a similar syntax for templates.

This probably wants to be off by default (ie, unless the template specifies .. as .. because it results in cloning the output tokenstream and parsing it an additional time. I'm not sure as is the right introducer keyword.

While reproing #15 (closed) with a prototype, I found that soemthing is laundering the path in the error messages. Either (i) this is somehow related to something trybuild is doing, and won't happen in normal use or (ii) this happens every time you get an error in a file in someone's OUT_DIR when it isn't "sufficiently you" or something. I don't see a good way to test this without uploading to crates.io. I guess I could go guddling in trybuild's source code.

error: expected `,`
 --> $OUT_DIR[derive-adhoc-macros]/derive-adhoc~expansions~/d-a-062c790cebd67d2c11c99937.rs
  |
  |             { 0 : f_0.clone(), 1 : f_1.clone(), }, AllTypes :: Struct
  |                                                    ------------------ while parsing the fields for this pattern

@nickm, opinons?

Appendix, things I discovered

Span has no useful comparison or extraction methods, not even PartialEq. I experimented with parsing the Debug output to try to divine error locations on a best effort basis, but there were other snags.

Principally, the obvious approach is to use parse_str. But that labels all of the input with exactly the same span, so it's useless. Hence the use of files, above.

An alternative might be to try to somehow match up the spans of TTs with those of the error. This would involve a custom prettyprinter which would have to abuse the debug output to try to figure out where the error was. Nightmare.

Sometimes, empirically, things with syntax errors get formatted rather poorly by the default TokenStream as Display. However, there doesn't appear to be any useful software for doing better. prettyplease can only format a syn::File which is no use if you can't parse the TS. Invoking rustfmt would be definitely a bridge too far (even if we could use a library interface for it).