A library to validate values of nested structs with their type spec t() and associated precondition functions
Mix.install([:domo], force: true)
A library to validate values of nested structs with their type spec t()
and associated precondition functions.
🔗 JSON parsing and validation example
🔗 Commanded + Domo combo used in Event Sourcing and CQRS app
🔗 Ecto + Domo combo in example_avialia app
🔗 TypedStruct + Domo combo in example_typed_integrations app
The library adds constructor, validation, and reflection functions to a struct module. When called, constructor and validation functions guarantee the following:
t()
types.precond
macro.If the conditions described above are not met, the constructor and validation functions return an error.
The business rule expressed via the precondition function is type-associated, which affects all structs referencing the appropriate type and using Domo.
In terms of Domain Driven Design, types and associated precondition functions define the invariants relating structs to each other.
use Domo
{: .info}When you
use Domo
, theDomo
module will define the following functions:
new
andnew!
to create a valid structensure_type
andensure_type!
to validate the existing struct- reflection functions
required_fields
,typed_fields
, and__t__
See the Callbacks section for more details.
Let's say that we have a LineItem
and PurchaseOrder
structs with relating
invariant that is the sum of line item amounts should be less then order's
approved limit. That can be expressed like the following:
defmodule LineItem do
use Domo
defstruct amount: 0
@type t :: %__MODULE__{amount: non_neg_integer()}
end
defmodule PurchaseOrder do
use Domo
defstruct id: 1000, approved_limit: 200, items: []
@type id :: non_neg_integer()
precond(id: &(1000 <= &1 and &1 <= 5000))
@type t :: %__MODULE__{
id: id(),
approved_limit: pos_integer(),
items: [LineItem.t()]
}
precond(t: &validate_invariants/1)
defp validate_invariants(po) do
amounts = po.items |> Enum.map(& &1.amount) |> Enum.sum()
if amounts <= po.approved_limit do
:ok
else
{:error, "Sum of line item amounts (#{amounts}) should be <= to approved limit (#{po.approved_limit})."}
end
end
end
Then PurchaseOrder
struct can be constructed consistently with functions generated by Domo like the following:
PurchaseOrder.new()
{:ok, %PurchaseOrder{approved_limit: 200, id: 1000, items: []}}
The constructor function takes any Enumerable
as the input value:
{:ok, po} = PurchaseOrder.new(%{approved_limit: 250})
{:ok, %PurchaseOrder{approved_limit: 250, id: 1000, items: []}}
It returns the descriptive keyword list if there is an error in input arguments. And it validates nested structs automatically:
PurchaseOrder.new(id: 500, items: [%LineItem{amount: -5}])
{:error,
[
items: "Invalid value [%LineItem{amount: -5}] for field :items of %PurchaseOrder{}.
Expected the value matching the [%LineItem{}] type.
Underlying errors:
- The element at index 0 has value %LineItem{amount: -5} that is invalid.
- Value of field :amount is invalid due to Invalid value -5 for field :amount
of %LineItem{}. Expected the value matching the non_neg_integer() type.",
id: "Invalid value 500 for field :id of %PurchaseOrder{}. Expected the
value matching the non_neg_integer() type. And a true value from
the precondition function \"&(1000 <= &1 and &1 <= 5000)\"
defined for PurchaseOrder.id() type."
]}
The returned errors are verbose and are intended for debugging purposes. See the Error messages for a user section below for more options.
The manually updated struct can be validated like the following:
po
|> Map.put(:items, [LineItem.new!(amount: 150)])
|> PurchaseOrder.ensure_type()
{:ok, %PurchaseOrder{approved_limit: 200, id: 1000, items: [%LineItem{amount: 150}]}}
Domo returns the error if the precondition function validating the t()
type
as a whole fails:
updated_po = %{po | items: [LineItem.new!(amount: 180), LineItem.new!(amount: 100)]}
PurchaseOrder.ensure_type(updated_po)
{:error, [t: "Sum of line item amounts should be <= to approved limit"]}
Domo supports sum types for struct fields, for example:
defmodule FruitBasket do
use Domo
defstruct fruits: []
@type t() :: %__MODULE__{fruits: [String.t() | :banana]}
end
FruitBasket.new(fruits: [:banana, "Maracuja"])
{:ok, %FruitBasket{fruits: [:banana, "Maracuja"]}}
{:error, [fruits: message]} = FruitBasket.new(fruits: [:banana, "Maracuja", nil])
IO.puts(message)
Invalid value [:banana, "Maracuja", nil] for field :fruits of %FruitBasket{}. Expected the value matching the [<<_::_*8>> | :banana] type.
Underlying errors:
- The element at index 2 has value nil that is invalid.
- Expected the value matching the <<_::_*8>> type.
- Expected the value matching the :banana type.
Getting the list of the required fields of the struct that have type other
then nil
or any
is like that:
PurchaseOrder.required_fields()
[:approved_limit, :id, :items]
For debug purposes you can get the original type spec t()
for the struct.
That might be useful when the struct's type spec is generated by some other library such as TypedEctoSchema
.
Do it like the following:
PurchaseOrder.__t__()
%PurchaseOrder{
id: integer() | nil
...
}
See the Callbacks section for more details about functions added to the struct.
It's possible to attach error messages to types with the precond
macro to display
them later to the user. To filter such kinds of messages, pass
the maybe_filter_precond_errors: true
option to Domo generated functions like that:
defmodule Book do
use Domo
defstruct [:title, :pages]
@type title :: String.t()
precond title: &(if String.length(&1) > 1, do: :ok, else: {:error, "Book title is required."})
@type pages :: pos_integer()
precond pages: &(if &1 > 2, do: :ok, else: {:error, "Book should have more then 3 pages. Given (#{&1})."})
@type t :: %__MODULE__{title: nil | title(), pages: nil | pages()}
end
defmodule Shelf do
use Domo
defstruct books: []
@type t :: %__MODULE__{books: [Book.t()]}
end
defmodule PublicLibrary do
use Domo
defstruct shelves: []
@type t :: %__MODULE__{shelves: [Shelf.t()]}
end
library = struct!(PublicLibrary, %{shelves: [struct!(Shelf, %{books: [struct!(Book, %{title: "", pages: 1})]})]})
PublicLibrary.ensure_type(library, maybe_filter_precond_errors: true)
{:error,
[
shelves: [
"Book title is required.",
"Book should have more then 3 pages. Given (1)."
]
]}
That output contains only a flattened list of precondition error messages from the deeply nested structure.
Sometimes a default value for the struct's field must be generated during the construction.
To reuse the new(!)/1
function's name and keep the generated value validated,
instruct Domo to use another name for its constructor with gen_constructor_name
option,
like the following:
defmodule Foo do
use Domo, skip_defaults: true, gen_constructor_name: :_new
defstruct [:id, :token]
@type id :: non_neg_integer()
@type token :: String.t()
precond token: &byte_size(&1) == 8
@type t :: %__MODULE__{id: id(), token: token()}
def new(id) do
_new(id: id, token: random_string(8))
end
def new!(id) do
_new!(id: id, token: random_string(8))
end
defp random_string(length),
do: :crypto.strong_rand_bytes(length) |> Base.encode64() |> binary_part(0, length)
end
Foo.new!(15245)
%Foo{id: 15245, token: "e8K9wP0e"}
At the project's compile-time, Domo performs the following checks:
It automatically validates that the default values given with defstruct/1
conform to the struct's type and fulfil preconditions (can be disabled,
see __using__/1
for more details). You can turn off this behaviour
by specifying skip_defaults
option; see Configuration section
for details.
It ensures that the struct using Domo built with new(!)/1
function
to be a default argument for a function or a default value for a struct's field
matches its type and preconditions.
At run-time, Domo validates structs matching their t()
types.
Domo compiles TypeEnsurer
module from struct's t()
type to do all kinds
of validations. There is a generated function with pattern matchings
and guards for each struct's field. Constructor and validation functions
of the struct delegate the work to the appropriate TypeEnsurer
module.
After the compilation, the flow of control of the nested StructA
validation
can look like the following:
+--------------------+
| PurchaseOrder | +---------------------------+
| | | PurchaseOrder.TypeEnsurer |
| new(!)/1 ----------|--_ | |
| ensure_type(!)/1 --|-----|-> ensure_field_type/1 |
+--------------------+ | private functions |
+----|----------------------+
|
+-----------------------+ |
| LineItem | | +-------------------------+
| | | | LineItem.TypeEnsurer |
| new(!)/1 | | | |
| ensure_type(!)/1 | +--|-> ensure_field_type/1 |
+-----------------------+ | private functions |
+-------------------------+
In interactive mode (iex / livebook) Domo generates TypeEnsurer
module
dynamically as the last step of struct's module definition.
In mix compile mode Domo generates all TypeEnsurer
modules after elixir compiler
finishes its job. The generated code can be found
in _build/MIX_ENV/domo_generated_code
folder. However, that is for information
purposes only. The following compilation will overwrite all changes there.
Let's suppose a structure field's type depends on a type defined in
another module. When the latter type or its precondition changes,
Domo recompiles the former module automatically to update its
TypeEnsurer
to keep the type validation up to date.
Domo tracks type-depending modules and touches appropriate files during compilation.
That works for any number of intermediate modules between the module defining the struct's field and the module defining the field's final type.
Ecto schema changeset can be automatically validated to conform to t()
type
and associated preconditions. Then the changeset function can be shortened
like the following:
defmodule Customer do
use Ecto.Schema
use Domo, skip_defaults: true
import Ecto.Changeset
import Domo.Changeset
schema "customers" do
field :first_name, :string
field :last_name, :string
field :birth_date, :date
timestamps()
end
@type t :: %__MODULE__{
first_name: String.t(),
last_name: String.t(),
birth_date: Date.t()
}
def changeset(changeset, attrs) do
changeset
|> cast(attrs, __schema__(:fields))
# Domo.Changeset defines validate_type/1 function.
|> validate_type()
end
end
The Domo validation comes with the price of about 1.5x times slower than
the equivalent Ecto's validate_N
functions.
See detailed example is in the example_avialia project.
Domo is compatible with most libraries that generate t()
type for a struct
and an Ecto schema, f.e. typed_struct
and typed_ecto_schema respectfully.
Just use Domo
in the module, and that's it.
An example is in the example_typed_integrations project.
TypedStruct's submodule generation with :module
option currently is not supported.
To use Domo in a project, add the following line to mix.exs
dependencies:
{:domo, "~> 1.5"}
And the following line to the project's mix.exs
file:
compilers: [:domo_compiler] ++ Mix.compilers()
To exclude the generated TypeEnsurer
modules from mix test --coverage
add the following line, that works since Elixir v1.13, to the project's mix.exs
:
test_coverage: [ignore_modules: [~r/\.TypeEnsurer$/]]
To avoid mix format
putting extra parentheses around precond/1
macro call,
add the following import to the .formatter.exs
:
[
import_deps: [:domo]
]
To enable Phoenix hot-reload for struct's type ensurers built by Domo,
update the compilers in the mix.exs
file like the following:
compilers: [:domo_compiler] ++ Mix.compilers() ++ [:domo_phoenix_hot_reload]
And add the following line to the endpoint's configuration in the config.exs
file:
config :my_app, MyApp.Endpoint,
reloadable_compilers: [:phoenix, :domo_compiler] ++ Mix.compilers() ++ [:domo_phoenix_hot_reload]
Add the Domo dependency and compilers config as mentioned in the section above
to the mix.exs
file for each app using Domo.
You may add the same domo compiler config line to the app itself and to the root
umbrella's mix.exs
to enable recompile
command to work correctly
for iex -S mix
run in the root.
If you have the Phoenix hot-reload configured for one of the web apps in umbrella then
the :domo_phoenix_hot_reload
compiler should be added to all dependency apps
used by the given web one.
The options listed below can be set globally in the configuration
with config :domo, option: value
. The value given
with use Domo, option: value
overrides the global setting.
skip_defaults
- if set to true
, disables the validation of
default values given with defstruct/1
to conform to the t()
type
at compile time. Default is false
.
gen_constructor_name
- the name of the constructor function added
to the module. The raising error function name is generated automatically
from the given one by adding trailing !
.
Defaults are new
and new!
appropriately.
unexpected_type_error_as_warning
- if set to true
, prints warning
instead of throwing an error for field type mismatch in the raising
functions. Default is false
.
remote_types_as_any
- keyword list of type lists by modules that should
be treated as any()
. F.e. [{ExternalModule, [:t, :name]}, {OtherModule, :t}]
Default is nil
.
Run the Application.put_env(:domo, :verbose_in_iex, true)
to enable verbose
messages from domo in Interactive Elixir console.
Parametrized types are not supported because it adds lots of complexity.
Library returns {:type_not_found, :key}
error
for @type dict(key, value) :: [{key, value}]
definition.
Library returns error for type referencing parametrized type like
@type field :: container(integer())
.
Primitive types referencing themselves are not supported.
Library returns an error for @type leaf :: leaf | nil
definition.
On the other hand, structs referencing themselves are supported.
The library will build TypeEnsurer
for the following definition
@type t :: %__MODULE__{leaf: t | nil}
and validate.
Library affects the project's full recompilation time almost insignificantly.
The compilation times for the business application with 38 structs (8 fields each on average) having 158 modules in total are the following:
Mode Average (by 3 measurements) Deviation
No Domo 14.826s 11.92
With Domo 15.711s 7.34
Comparison:
No Domo 14.826s
With Domo 15.711s - 1.06x slower
The library ensures the correctness of data types at run-time, which comes with the computation price.
One of the standard tasks is translating the map with string values received as a form into a validated nested struct.
For benchmark, we use Album
struct having many Track
structs.
When comparing Domo
generated new!/1
constructor function and Ecto
changeset
using a set of validate_.../2
functions internally, both have equivalent execution
times. However, in the former case, the memory consumption is about 25% bigger.
In the case of Ecto
changeset that uses Domo.Changeset.validate_type/1
function
the computation takes about 1.5x times longer.
One of the benchmark results is shown below. The benchmark project is
/benchmark_ecto_domo folder. You can run it with mix benchmark
command.
Operating System: macOS
CPU Information: Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz
Number of Available Cores: 8
Available memory: 16 GB
Elixir 1.11.0
Erlang 24.3.3
Benchmark suite executing with the following configuration:
warmup: 2 s
time: 8 s
memory time: 2 s
parallel: 1
inputs: none specified
Estimated total run time: 36 s
Benchmarking Domo Album.new!/1...
Benchmarking Domo.Changeset validate_type/1...
Benchmarking Ecto.Changeset validate_.../1...
Name ips average deviation median 99th %
Domo Album.new!/1 5.31 188.25 ms ±1.79% 188.28 ms 196.57 ms
Ecto.Changeset validate_.../1 5.21 191.85 ms ±1.94% 191.00 ms 202.22 ms
Domo.Changeset validate_type/1 3.76 266.19 ms ±1.20% 266.58 ms 271.01 ms
Comparison:
Domo Album.new!/1 5.31
Ecto.Changeset validate_.../1 5.21 - 1.02x slower +3.59 ms
Domo.Changeset validate_type/1 3.76 - 1.41x slower +77.93 ms
Memory usage statistics:
Name Memory usage
Domo Album.new!/1 245.73 MB
Ecto.Changeset validate_.../1 186.59 MB - 0.76x memory usage -59.13956 MB
Domo.Changeset validate_type/1 238.69 MB - 0.97x memory usage -7.04444 MB
**All measurements for memory usage were the same**
To migrate to a new version of Domo, please, clean and recompile
the project with mix clean --deps && mix compile
command.
It's possible to adopt Domo library in the project having user-defined
constructor functions named new/1
by refactoring to the Domo generated one.
Here's how:
:domo
dependency to the project, configure compilers as described in
the installation sectionconfig :domo, :gen_constructor_name, :_new
option into
the confix.exs
file, to prevent conflict with user-defined constructor
function nameuse Domo
to existing struct, f.e. FirstStruct
FistStruct._new(%{...})
new/1
if it's not needed anymore
and rename _new
to new
in the whole projectconfig :domo, :gen_constructor_name
configuration
because Domo generates constructor wiht new
name by default.Constructor, validation, and reflection functions added to the struct module using Domo.
Creates a struct validating type conformance and preconditions.
The argument is any Enumerable
that emits two-element tuples
(key-value pairs) during enumeration.
Returns the instance of the struct built from the given enumerable
.
Does so only if struct's field values conform to its t()
type
and all field's type and struct's type precondition functions return ok.
Raises an ArgumentError
if conditions described above are not fulfilled.
This function will check if every given key-value belongs to the struct
and raise KeyError
otherwise.
Creates a struct validating type conformance and preconditions.
The argument is any Enumerable
that emits two-element tuples
(key-value pairs) during enumeration.
Returns the instance of the struct built from the given enumerable
in the shape of {:ok, struct_value}
. Does so only if struct's
field values conform to its t()
type and all field's type and struct's
type precondition functions return ok.
If conditions described above are not fulfilled, the function
returns an appropriate error in the shape of {:error, message_by_field}
.
message_by_field
is a keyword list where the key is the name of
the field and value is the string with the error message.
Keys in the enumerable
that don't exist in the struct
are automatically discarded.
maybe_filter_precond_errors
- when set to true
, the values in
message_by_field
instead of string become a list of error messages
from precondition functions. If there are no error messages from
precondition functions for a field's type, then all errors are returned
unfiltered. Helpful in taking one of the custom errors after executing
precondition functions in a deeply nested type to communicate
back to the user. F.e. when the field's type is another struct.
Default is false
.Ensures that struct conforms to its t()
type and all preconditions
are fulfilled.
Returns struct when it's valid. Raises an ArgumentError
otherwise.
Useful for struct validation when its fields changed with map syntax
or with Map
module functions.
Ensures that struct conforms to its t()
type and all preconditions
are fulfilled.
Returns struct when it's valid in the shape of {:ok, struct}
.
Otherwise returns the error in the shape of {:error, message_by_field}
.
Takes the same options as new/2
.
Useful for struct validation when its fields changed with map syntax
or with Map
module functions.
Returns the list of struct's fields defined with explicit types in its t()
type spec.
Does not return meta fields with __underscored__
names and fields
having any()
type by default.
Includes fields that have nil
type into the return list.
:include_any_typed
- when set to true
, adds fields with any()
type to the return list. Default is false
.
:include_meta
- when set to true
, adds fields
with __underscored__
names to the return list. Default is false
.
Returns the list of struct's fields having type others then nil
or any()
.
Does not return meta fields with __underscored__
names.
Useful for validation of the required fields for emptiness.
F.e. with validate_required/2
call in the Ecto
changeset.
:include_meta
- when set to true
, adds fields
with __underscored__
names to the return list. Default is false
.Returns the struct's type spec t()
description for debugging purposes.
Domo compiled validation functions for the given struct based on the described type spec.
Fork the repository and make a feature branch
After implementing of the feature format the code with:
mix format
run linter and tests to ensure that all works as expected with:
mix check || mix check --failed
Make a PR to this repository
t()
. Useful when you want to know the type Domo validation is built on.mix deps.update domo
, please, remove .elixir_ls
in your
project directory and reopen VSCode.ResolvePlanner
server for printing verbose debugging messages.name_of_new_function
to gen_constructor_name
. Please, update your project if you use it.new!/1/0
and new/2/1/0
composable instead of overridable, see Custom constructor function section in README.Ecto.Schema
types: belongs_to(t)
, has_one(t)
, has_many(t)
, many_to_many(t)
, embeds_one(t)
, and embeds_many(t)
.validate_type/2
function for changesets:
Ecto
schema.Ecto.Schema
types are listed above, so the user can explicitly call cast_assoc/2/3
for these fields.term()
with other types as the term()
.new!/1/0
and new/2/1/0
become overridable.mix dialyzer
pass on generated code.mix test
and mix dialyzer
without crashes.:domo_phoenix_hot_reload
after :elixir
to compilers
list in the mix.exs file and to reloadable_compilers
list in the config file.test_coverage
configuration example to exclude generated TypeEnsurer
modules from the test coverage report.beam
fileensure_struct_defaults: false
for skip_defaults: true
, the former option is supported till the next version. Please, migrate to the latter one.@type t :: %__MODULE__{left: t() | nil, right: t() | nil}
Code.can_await_module_compilation?
:maybe_filter_precond_errors
option to lift precondition error messages from the nested structsapply()
with .
for validation function calls to run fasteriex
and live book
Breaking change:
:domo_compiler
before :elixir
in mix.exs.
And do the same for reloadable_compilers
key in config file
if configured for Phoenix endpoint.precond/1
macro error message@opaque
types.Breaking changes:
new_ok
constructor function name to new
that is more convenient.
Search and replace new_ok(
-> new(
in all files of the project
using Domo to migrate.!
to the value of :name_of_new_function
option. The defaults are new
and new!
.Ecto
3.7.x:ecto
and :decimal
as optional dependencies:remote_types_as_any
option with use Domo
MapSet
should be validated with precond
function for custom user type, because parametrized t(value)
types are not supportedapply()
with Module.function calls to run fasterDecimal.t()
:inet.port_number()
t()
type spec or have the any()
typeTypeEnsurer
modules for Date
, Date.Range
, DateTime
, File.Stat
, File.Stream
, GenEvent.Stream
, IO.Stream
, Macro.Env
, NaiveDateTime
, Range
, Regex
, Task
, Time
, URI
, and Version
structs from the standard library at the first project compilationprecond
function of the user type pointing to a structprecond
function for struct referenced by a user typeDomo.has_type_ensurer?/1
that checks whether a TypeEnsurer
module was generated for the given struct.Jason
+ ExJSONPath
+ Domo
new!
to follow Elixir naming convention.
You can always change the name with the config :domo, :name_of_new_function, :new_func_name_here
app configuration.__underscored__
fields at compile-time.t()
type returns true
at compile time regarding defaults correctness check.TypedStruct
and TypedEctoSchema
.any()
type with and without precondition.typed_fields/1
and required_fields/1
functions.maybe_filter_precond_errors: true
option that filters errors from precondition functions for better output for the user.---/2
operator and tag chain functions from Domo.TaggedTuple
into tagged_tuple library.Domo.Changeset.validate_type/*
functions to validate Ecto.Changeset field changes matching the t() type.precond/1
.defstruct/1
to the
struct's t()
type at compile-time.remote_types_as_any
option to disable validation of specified complex
remote types. What can be replaced by precondition for wrapping user-defined type.Range.t()
and MapSet.t()
mix clean
command--verbose
optionprecond/1
macronew/1
calls at compile time f.e. to specify default values:domo_compiler
TypeEnsurer
modules only if struct changes or dependency type changes:reloadable_compilers
option is fully supportedTypeEnsurer
modules for all structs---/2
operator to make tag chains with Domo.TaggedTuple
module (will be removed in v1.2.9)new
constructor to a struct Check if the field values passed as an argument to the new/1
, and put/3
matches the field types defined in typedstruct/1
.
Support the keyword list as a possible argument for the new/1
.
Add module option to put a warning in the console instead of raising of the ArgumentError
exception on value type mismatch.
Make global environment configuration options to turn errors into warnings that are equivalent to module ones.
Move type resolving to the compile time.
Keep only bare minimum of generated functions that are new/1
, ensure_type!/1
and their _ok versions.
Make the new/1
and ensure_type!/1
speed to be less or equal to 1.5 times of the struct!/2
speed.
Support new/1
calls in macros to specify default values f.e. in other structures. That is to check if default value matches type at compile time.
Support precond/1
macro to specify a struct field value's contract with a boolean function.
Support struct types referencing itself for tree structures.
Evaluate full recompilation time for a project using Domo.
Add use option to specify names of the generated functions.
Add documentation to the generated for new(_ok)/1
, and ensure_type!(_ok)/1
functions in a struct.
Copyright © 2020-2022 Ivan Rublev
This project is licensed under the MIT license.