- Feature #357
openTechnical DWDS text format specification
0%
Description
It will be important to define an authoritative technical specification for the text file format to be used by rulemaker, rule taker & rule reserve implementations.
This is critical to support development of reference implementations of those components, and coordinate evolution of the file format.
I understand the "canonical" DWDS text file format to be the "pipe-separated value"-based format such as produced currently by rulemaker v4.
For example: https://gitlab.com/xalgorithms-alliance/data-with-direction-specification/dwds-documents/-/blob/master/Section_5-2-2_RFC9000_rule_c8842b81-a047-4c88-a707-e889cda65655.txt?ref_type=heads.
See attached file for another example.
Those examples may not represent the final canonical version though (e.g. see https://xalgorithms.redminepro.net/issues/356).
Previous file formats include:
Files
Updated by Joseph Potvin 4 months ago
- File Refinement of the DWD Data Structure_2025-12-23.odt Refinement of the DWD Data Structure_2025-12-23.odt added
Apologies that I didn't point to this explicitly & also put it into a README:
https://gitlab.com/xalgorithms-alliance/data-with-direction-specification/dwds-documents/-/blob/16918d90c276397a23e9c83bf59e5bac043cd81b/current/Refinement%20of%20the%20DWD%20Data%20Structure.pdf
Excuse: My instinct was to get some feedback first, but evidently I forgot to ask. (The ODT is attached here for editing if you prefer.)
Updated by Charles Langlois 4 months ago
- File draft-potvin-dwd-pipe-separated-format.html draft-potvin-dwd-pipe-separated-format.html added
- File draft-potvin-dwd-pipe-separated-format.md draft-potvin-dwd-pipe-separated-format.md added
- File draft-potvin-dwd-pipe-separated-format.txt draft-potvin-dwd-pipe-separated-format.txt added
Joseph Potvin wrote in #note-1:
Apologies that I didn't point to this explicitly & also put it into a README:
https://gitlab.com/xalgorithms-alliance/data-with-direction-specification/dwds-documents/-/blob/16918d90c276397a23e9c83bf59e5bac043cd81b/current/Refinement%20of%20the%20DWD%20Data%20Structure.pdfExcuse: My instinct was to get some feedback first, but evidently I forgot to ask. (The ODT is attached here for editing if you prefer.)
Thanks I'll certainly use that reference, this is a good resource to explain the design of the data format.
But my meaning of a technical specification is one oriented for programmers to implement the format, not just understand its goal and design decisions.
The concern of that technical specification is precise guidelines and prescriptions for an implementation to parse or generate the format, in order to guarantee interoperability of any implementation following the specification.
The concerns being technical details, in technical terms, I would keep it separate from the design motivations (they can reference each other though).
Note that the same design motivations can allow for multiple technical specifications, where specific technical concerns discovered in implementation and usage may lead to iteration of the technical specification without overriding the original design decisions.
Example technical specifications:
- (CSV format) https://www.ietf.org/rfc/rfc4180.txt
- (JSON) https://www.ecma-international.org/wp-content/uploads/ECMA-404_2nd_edition_december_2017.pdf / https://www.rfc-editor.org/rfc/rfc8259.txt
The important aspects:
- how the different datatypes are encoded
- the specific character sequences (with ASCII / UTF-8 encodings) to expect in the structure
- specific constraints (e.g. field lengths) that should be accounted for in parser or generator implementations
- practical recommendations for implementors (e.g. for performance, or correctness / robustness)
- a standard-form grammar (usually ABNF )
Attached is a WIP specification. The source is in markdown (.md), a text-based markup language. txt and html formats are generated from it (I used IETF RFC tooling). I had coding agents generate most of it from example files and those documents of yours. Many points I'm pretty sure we want to change or needing discussion, but that suggests the kind of information we need to coordinate implementations.
Updated by Joseph Potvin 4 months ago
Oh, yes I agree about a technical specification. We just haven't gotten there yet. Closer is Don's README: https://gitlab.com/xalgorithms-alliance/rule-networking-software/prod-impl/-/blob/master/README.md
Updated by Charles Langlois 4 months ago
Joseph Potvin wrote in #note-3:
Oh, yes I agree about a technical specification. We just haven't gotten there yet. Closer is Don's README: https://gitlab.com/xalgorithms-alliance/rule-networking-software/prod-impl/-/blob/master/README.md
Don's README is informative about the design and APIs of the implementation, but there's no discussion of the pipe-separated file format (the format supported are the ".ior" and json format).
The specification doesn't need to be final and definitive. But any format we want implemented in multiple components would benefit from a draft specification (that can evolve) to coordinate those implementations and resolve questions.