atdml: a simpler replacement for atdgen (#446)
* Add atdml: simplified OCaml JSON code generator using Yojson AST
Introduces two new packages:
- atdml-runtime: small runtime library with helpers for reading/writing
primitive types, lists, options, and nullables via Yojson.Safe.t.
- atdml: code generator that takes an ATD file and produces a single
foo.ml + foo.mli module pair. Unlike atdgen, it uses Yojson.Safe.t
as the intermediate representation throughout, making the generated
code straightforward to read and the generator easy to maintain.
Naming follows the ppx_yojson_conv convention (foo_of_yojson /
yojson_of_foo). Classic variants are the default; polymorphic variants
are opt-in per sum type via <ocaml repr="poly">.
Includes a cram test, README, Makefile, and opam package definitions.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [atdml] Inline runtime into each generated .ml; remove atdml-runtime package
The atdml-runtime library was tiny (< 70 lines). Instead of shipping it as
a separate package that every user of atdml-generated code must depend on,
emit it verbatim as a private `module Atdml_runtime = struct ... end` at the
top of each generated `.ml` file.
Benefits:
- Generated files only depend on `yojson`; no second package to install.
- Simpler dune file for downstream users (no `atdml-runtime` entry).
- The runtime stays in sync with the generator automatically.
Removes the `atdml-runtime` opam package and its source directory entirely.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [atdml] Support multiline strings in Indent.Line; simplify runtime_module
Indent.to_buffer now splits a Line containing newlines into sub-lines,
each printed at the current indentation level. This allows passing a
raw multiline string literal to the Line constructor instead of one
Line node per line.
Use this to rewrite Codegen.runtime_module as a single B.Line holding
a {|...|} raw string — much easier to read and edit in place.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [atdml] Add comprehensive codegen cram tests
Adds codegen.t covering all ATD constructs and annotations supported
by atdml:
- Type aliases (string, float, list, option)
- Classic sum types: plain constructors, constructors with payloads,
<json name> and <ocaml name> renaming
- Polymorphic variants via <ocaml repr="poly">
- Records: required, optional (?), with-default (~) fields; implicit
defaults for common types; explicit <ml default="...">; <json name>
field renaming (annotations go before the ':', per ATD convention)
- All builtin types and composites: unit, bool, int, float, string,
list, option, nullable, abstract, tuple, nested
- Parametric types with one and two type variables
- Mutually recursive types (ATD types are implicitly mutually
recursive; no 'and' keyword needed)
Both .mli and .ml output are checked for the records case; .mli only
for the others to avoid repeating the inlined runtime block.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [atdml] Use <ocaml ...> exclusively, drop <ml ...> support
Aligns atdml with atdgen: the only annotation section for OCaml-specific
options is <ocaml ...>, matching what atdgen users already write.
- Rename annot_schema_ml -> annot_schema_ocaml (section "ocaml")
- Rename get_ml_default -> get_ocaml_default (reads from "ocaml" only)
- Register exactly the three annotations atdml uses, with correct
ATD node-kind contexts:
Type_expr "repr" -- <ocaml repr="poly"> on a sum type
Variant "name" -- <ocaml name="..."> on a variant constructor
Field "default"-- <ocaml default="..."> on a with-default field
Previously the "ocaml" section was absent from the schema so these
annotations were silently accepted; now the validator enforces them.
- Update codegen.t and README to use <ocaml default="...">
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [atdml] Replace cram tests with Testo snapshot tests
- Convert atdml/test/*.t (cram) to atdml/tests/test.ml (Testo)
- Rename test/ directory to tests/ and add shell wrapper at atdml/test
- Snapshots stored as flat files in tests/named-snapshots/
- Tests cover all supported ATD constructs and <ocaml ...> annotations
- Add testo dependency to atdml package and atdml.opam
- Add fpath dependency to test executable
- Update Makefile to run atdml tests via $(MAKE) -C atdml test
- Update generated file header to say "by atdml."
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Compile and run the generated code + some debugging
* Fix some tests and simplify the invocation of atdml
* Add contributing instructions for atdml
* Update gitignore
* [atdml] Compile and run the generated code + some debugging
- Run atdml from tests using the installed binary instead of calling
Codegen.run_file directly (this also tests the CLI interface)
- Extend tests to compile the generated code with ocamlfind, run it on
sample JSON input, and capture the output as part of the snapshot
- Fix Codegen.ml: use explicit universal quantification ('a. ...) for
parametric functions in let rec...and blocks to prevent OCaml from
fixing the type of one function based on how it is called by another
function in the same block
- Fix various JSON inputs in tests (invalid token 'abstract', wrong
variant name 'rect' vs 'rectangle', missing closing '}')
- Update snapshots to reflect new ATD sources and output format
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [atdml] Add submodules, rename json I/O functions, rename reserved OCaml names
- Generate a submodule (e.g. `module Foo`) for each ATD type, with
`type nonrec t = foo`, `of_yojson`, `to_yojson`, `of_json`, `to_json`
(and `make` for record types)
- Rename top-level I/O functions: `foo_of_json`/`json_of_foo` (was
`foo_of_string`/`string_of_foo`); drop `_of_channel`/`_of_file`
- Use `Atd.Unique_name` to rename ATD type names that conflict with OCaml
keywords or with the generated function naming scheme (e.g. `yojson`,
`json`); names get a `_` suffix or similar to avoid conflicts
- Add tests for problematic ATD type names (`yojson`, `json`, `module`,
including cases where `module_` etc. are already taken)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [atdml] Update --help to show current interface
Show `_of_json`/`json_of_` function names and the generated submodules
(Color, Point, etc.) with their `make`, `of_yojson`/`to_yojson`,
`of_json`/`to_json` members.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [atdml] Add Sample_interface.mli documenting the generated interface
Shows the expected .mli shape for a non-parametric record type (Foo)
and a parametric sum type (Bar), including top-level functions and
submodules.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [atdml] Read from stdin, write self-contained module snippet to stdout
When no input file is given, atdml reads ATD source from stdin and
prints to stdout:
module type Types = sig ... end
module Types : Types = struct ... end
This snippet can be copy-pasted directly into utop or ocaml.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [atdml] Use Atd.Util.tsort for minimal recursive groups
Use topological sort to emit only strictly necessary 'and'/'let rec'
groupings:
- Independent types each get their own 'type' declaration
- A self-recursive or mutually-recursive group uses 'type ... and ...'
- Non-recursive functions use plain 'let'; only recursive groups use
'let rec' (with 'and' for mutually-recursive ones)
- Each group's types and functions are emitted together, ordered by
dependency (leaf types first)
This makes the generated code easier to review and can meaningfully
improve OCaml compile times for large ATD files.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [atdml] Add blank line between foo_of_json and json_of_foo
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [atdml] Add blank line before each 'and' in type and function groups
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [atdml] Implement the 'wrap' construct
Supports the standard ATD wrap annotations on 'wrap' type expressions:
<ocaml module="M"> → M.t, M.wrap, M.unwrap
<ocaml module="M" t="T"> → T, M.wrap, M.unwrap
<ocaml t="T" wrap="f" unwrap="g"> → fully explicit
(no annotation) → identity (behaves like a type alias)
- type_expr_str: uses wrap_t (or falls back to inner type)
- reader_expr: applies wrap_fn after reading the inner value
- writer_expr: applies unwrap_fn before writing the inner value
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [atdml] Support <ocaml attr="..."> for ppx attributes on type definitions
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [atdml] Add atdml reference manual
Documents all shipped features: type mapping, generated interface,
JSON/OCaml annotations (name, attr, default, repr, wrap), optional and
default fields, mutually recursive types, parametric types, and reserved
name handling. Includes a planned-features section for adapters.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [atdml] Add changelog entry with full feature list
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [atdml] Replace List.concat_map with a 4.08-compatible version
List.concat_map was added in OCaml 4.10.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Fix syntax in docs
* [atdml] Implement JSON adapters
Supports <json adapter.ocaml="M"> (module with normalize/restore) and
<json adapter.to_ocaml="f" adapter.from_ocaml="g"> (inline expressions)
on sum types and records. Also adds extra_sources parameter to test_e2e
for supplying auxiliary OCaml modules needed by individual tests.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [atdml] Propagate <ocaml attr="..."> to the submodule type alias
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [atdml] Add <doc text="..."> support, generating ocamldoc comments
Type-level docs are prepended as (** ... *) before the type declaration.
Field and variant docs are appended inline as (** ... *) on the same line.
Multi-paragraph docs and preformatted blocks ({{{ }}}) are rendered using
standard ocamldoc conventions ({v ... v} for verbatim text).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [atdml] Post-review improvements
- Remove stale Sample_interface.mli (named-snapshots are the canonical reference)
- Add module-level <doc text="..."> support (head annotation was silently ignored)
- Expand "doc" test with mutually recursive types to show (** ... *) before 'and'
- Add comment in dune-project: build tools require dune >= 3.18, generated code targets OCaml >= 4.08
- Auto-rename record fields and variant constructors that conflict with OCaml keywords, using Atd.Unique_name per-scope; <ocaml name="..."> on fields is not supported for simplicity
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [atdml] Improve compatibility with real-world ATD files
- Accept <ocaml module="..."> / <ocaml t="..."> on type definitions and
<ocaml name="..."> on record fields with a warning (annotation ignored);
these are atdgen-isms that atdml does not implement
- Implement <json repr="object"> for (string * 'a) list: encodes as a JSON
object {"k": v} rather than an array of pairs
- ~field: user_type with no OCaml default no longer errors: warns and
treats the field as required in JSON; make_* is skipped for that type
- All annotation warnings are emitted in a single upfront pass,
deduplicated by source location (no repeated warnings for inherited fields)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
5cf847
-
Mar 06 23:15 +00:00