Crate syn

github crates-io docs-rs


Syn is a parsing library for parsing a stream of Rust tokens into a syntax tree of Rust source code.

Currently this library is geared toward use in Rust procedural macros, but contains some APIs that may be useful more generally.


Example of a derive macro

The canonical derive macro using Syn looks like this. We write an ordinary Rust function tagged with a proc_macro_derive attribute and the name of the trait we are deriving. Any time that derive appears in the user's code, the Rust compiler passes their data structure as tokens into our macro. We get to execute arbitrary Rust code to figure out what to do with those tokens, then hand some tokens back to the compiler to compile into the user's crate.

# Cargo.toml
[package]
...

[lib]
proc-macro = true

[dependencies]
syn = "3"
quote = "1"

# extern crate proc_macro;
#
use proc_macro::TokenStream;
use quote::quote;
use syn::{parse_macro_input, DeriveInput};

# const IGNORE_TOKENS: &str = stringify! {
#[proc_macro_derive(MyMacro)]
# };
pub fn my_macro(input: TokenStream) -> TokenStream {
    // Parse the input tokens into a syntax tree
    let input = parse_macro_input!(input as DeriveInput);

    // Build the output, possibly using quasi-quotation
    let expanded = quote! {
        // ...
    };

    // Hand the output tokens back to the compiler
    TokenStream::from(expanded)
}

The heapsize example directory shows a complete working implementation of a derive macro. The example derives a HeapSize trait which computes an estimate of the amount of heap memory owned by a value.

pub trait HeapSize {
    /// Total number of bytes of heap memory owned by `self`.
    fn heap_size_of_children(&self) -> usize;
}

The derive macro allows users to write #[derive(HeapSize)] on data structures in their program.

# const IGNORE_TOKENS: &str = stringify! {
#[derive(HeapSize)]
# };
struct Demo<'a, T: ?Sized> {
    a: Box<T>,
    b: u8,
    c: &'a str,
    d: String,
}


Spans and error reporting

The token-based procedural macro API provides great control over where the compiler's error messages are displayed in user code. Consider the error the user sees if one of their field types does not implement HeapSize.

# const IGNORE_TOKENS: &str = stringify! {
#[derive(HeapSize)]
# };
struct Broken {
    ok: String,
    bad: std::thread::Thread,
}

By tracking span information all the way through the expansion of a procedural macro as shown in the heapsize example, token-based macros in Syn are able to trigger errors that directly pinpoint the source of the problem.

error[E0277]: the trait bound `Thread: HeapSize` is not satisfied
 --> src/main.rs:9:5
  |
3 | #[derive(HeapSize)]
  |          -------- required by a bound introduced by this call
...
9 |     bad: std::thread::Thread,
  |     ^^^^^^^^^^^^^^^^^^^^^^^^ the trait `HeapSize` is not implemented for `Thread`
  |
  = help: the following other types implement trait `HeapSize`:
            &'a T
            Box<T>
            Demo<'a, T>
            String
            [T]
            u8

Parsing a custom syntax

The lazy-static example directory shows the implementation of a functionlike!(...) procedural macro in which the input tokens are parsed using Syn's parsing API.

The example reimplements the popular lazy_static crate from crates.io as a procedural macro.

# macro_rules! lazy_static {
#     ($($tt:tt)*) => {}
# }
#
lazy_static! {
    static ref USERNAME: Regex = Regex::new("^[a-z0-9_-]{3,16}$").unwrap();
}

The implementation shows how to trigger custom warnings and error messages on the macro input.

warning: come on, pick a more creative name
  --> src/main.rs:10:16
   |
10 |     static ref FOO: String = "lazy_static".to_owned();
   |                ^^^

Testing

When testing macros, we often care not just that the macro can be used successfully but also that when the macro is provided with invalid input it produces maximally helpful error messages. Consider using the trybuild crate to write tests for errors that are emitted by your macro or errors detected by the Rust compiler in the expanded code following misuse of the macro. Such tests help avoid regressions from later refactors that mistakenly make an error no longer trigger or be less helpful than it used to be.


Debugging

When developing a procedural macro it can be helpful to look at what the generated code looks like. Use cargo rustc -- -Zunstable-options -Zunpretty=expanded or the cargo expand subcommand.

To show the expanded code for some crate that uses your procedural macro, run cargo expand from that crate. To show the expanded code for one of your own test cases, run cargo expand --test the_test_case where the last argument is the name of the test file without the .rs extension.

This write-up by Brandon W Maister discusses debugging in more detail: Debugging Rust's new Custom Derive system.


Optional features

Syn puts a lot of functionality behind optional features in order to optimize compile time for the most common use cases. The following features are available.


Compatibility notes

Syn is able to accommodate most kinds of Rust grammar changes without a new major release through the following mechanisms.

Modifiers

Syntax tree structs which are expected to grow over the course of future releases of the Rust language, such as ItemTrait, contain a modifiers field of one of several "Modifiers" types, in that case TraitModifiers.

All modifiers structs have the following commonalities:

Across major versions, fields may be promoted out of a "Modifiers" struct into the enclosing syntax tree node(s), typically for syntax that has already been incorporated into a stable release of Rust or is deemed sufficiently on track for stabilization.

Verbatim variants

Syntax tree enums which are expected to grow over the course of future releases of the Rust language are declared non-exhaustive and may contain a variant named Verbatim that holds TokenStream. For example Expr::Verbatim.

Unstable language syntax for which a dedicated syntax tree node does not yet exist will get parsed to Verbatim, thus allowing unstable syntax to round-trip through parsing and printing of a syntax tree. For example you might parse token input to ItemTrait (a trait definition) in order to read or modify its method signatures, or insert associated types, or insert default function bodies. All of this would work even if there is some Expr::Verbatim syntax somewhere in one of the function bodies in the macro input.

Verbatim variants are not intended to be constructed other than by Syn's parser. Do not rely on passing one containing arbitrary tokens through Syn's ToTokens implementations or through any other library, as it may panic or otherwise misbehave, such as failing to accurately parenthesize subexpressions to preserve precedence.

It is important not to write code that expects Syn's parser to continue to produce Verbatim when parsing some particular syntax construct, as that behavior changes across patch releases of Syn. Patch releases can promote something that used to be parsed as Verbatim into a new dedicated syntax tree node. Verbatim variants are specifically only for round-tripping code not acted on by the caller.

Non-exhaustive enums

Some enums in the syntax tree are declared #[non_exhaustive] and cannot be pattern-matched using an exhaustive match; a default case (_ => ...) is required. New variants of these enums will be added over time corresponding to new Rust syntax.

For testing the exhaustiveness of a match on such an enum in downstream code, it is recommended to use the following idiom.

# use syn::Expr;
#
# fn example(expr: Expr) {
match expr {
    #![cfg_attr(test, deny(non_exhaustive_omitted_patterns))]

    Expr::Array(expr) => { /*...*/ }
    Expr::Assign(expr) => { /*...*/ }
     ...
    Expr::Yield(expr) => { /*...*/ }

    _ => { /* some sane fallback */ }
}
# }

This way you will be notified by a test failure when a variant is added, so that you can add code to handle it, but your library will continue to compile and work for downstream users in the interim.

Modules

Structs

Enums

Functions

Type Aliases

Macros