Release 0.36

HISTORY.md

@@ -1,5 +1,139 @@
 # DynamicPPL Changelog
 
+## 0.36.0
+
+**Breaking changes**
+
+### Submodels: conditioning
+
+Variables in a submodel can now be conditioned and fixed in a correct way.
+See https://github.com/TuringLang/DynamicPPL.jl/issues/857 for a full illustration, but essentially it means you can now do this:
+
+```julia
+@model function inner()
+    x ~ Normal()
+    return y ~ Normal()
+end
+@model function outer()
+    return a ~ to_submodel(inner() | (x=1.0,))
+end
+```
+
+and the `a.x` variable will be correctly conditioned.
+(Previously, you would have to condition `inner()` with the variable `a.x`, meaning that you would need to know what prefix to use before you had actually prefixed it.)
+
+### Submodel prefixing
+
+The way in which VarNames in submodels are prefixed has been changed.
+This is best explained through an example.
+Consider this model and submodel:
+
+```julia
+using DynamicPPL, Distributions
+@model inner() = x ~ Normal()
+@model outer() = a ~ to_submodel(inner())
+```
+
+In previous versions, the inner variable `x` would be saved as `a.x`.
+However, this was represented as a single symbol `Symbol("a.x")`:
+
+```julia
+julia> dump(keys(VarInfo(outer()))[1])
+VarName{Symbol("a.x"), typeof(identity)}
+  optic: identity (function of type typeof(identity))
+```
+
+Now, the inner variable is stored as a field `x` on the VarName `a`:
+
+```julia
+julia> dump(keys(VarInfo(outer()))[1])
+VarName{:a, Accessors.PropertyLens{:x}}
+  optic: Accessors.PropertyLens{:x} (@o _.x)
+```
+
+In practice, this means that if you are trying to condition a variable in the submodel, you now need to use
+
+```julia
+outer() | (@varname(a.x) => 1.0,)
+```
+
+instead of either of these (which would have worked previously)
+
+```julia
+outer() | (@varname(var"a.x") => 1.0,)
+outer() | (a.x=1.0,)
+```
+
+In a similar way, if the variable on the left-hand side of your tilde statement is not just a single identifier, any fields or indices it accesses are now properly respected.
+Consider the following setup:
+
+```julia
+using DynamicPPL, Distributions
+@model inner() = x ~ Normal()
+@model function outer()
+    a = Vector{Float64}(undef, 1)
+    a[1] ~ to_submodel(inner())
+    return a
+end
+```
+
+In this case, the variable sampled is actually the `x` field of the first element of `a`:
+
+```julia
+julia> only(keys(VarInfo(outer()))) == @varname(a[1].x)
+true
+```
+
+Before this version, it used to be a single variable called `var"a[1].x"`.
+
+Note that if you are sampling from a model with submodels, this doesn't affect the way you interact with the `MCMCChains.Chains` object, because VarNames are converted into Symbols when stored in the chain.
+(This behaviour will likely be changed in the future, in that Chains should be indexable by VarNames and not just Symbols, but that has not been implemented yet.)
+
+### AD testing utilities
+
+`DynamicPPL.TestUtils.AD.run_ad` now links the VarInfo by default.
+To disable this, pass the `linked=false` keyword argument.
+If the calculated value or gradient is incorrect, it also throws a `DynamicPPL.TestUtils.AD.ADIncorrectException` rather than a test failure.
+This exception contains the actual and expected gradient so you can inspect it if needed; see the documentation for more information.
+From a practical perspective, this means that if you need to add this to a test suite, you need to use `@test run_ad(...) isa Any` rather than just `run_ad(...)`.
+
+### SimpleVarInfo linking / invlinking
+
+Linking a linked SimpleVarInfo, or invlinking an unlinked SimpleVarInfo, now displays a warning instead of an error.
+
+### VarInfo constructors
+
+`VarInfo(vi::VarInfo, values)` has been removed. You can replace this directly with `unflatten(vi, values)` instead.
+
+The `metadata` argument to `VarInfo([rng, ]model[, sampler, context, metadata])` has been removed.
+If you were not using this argument (most likely), then there is no change needed.
+If you were using the `metadata` argument to specify a blank `VarNamedVector`, then you should replace calls to `VarInfo` with `DynamicPPL.typed_vector_varinfo` instead (see 'Other changes' below).
+
+The `UntypedVarInfo` constructor and type is no longer exported.
+If you needed to construct one, you should now use `DynamicPPL.untyped_varinfo` instead.
+
+The `TypedVarInfo` constructor and type is no longer exported.
+The _type_ has been replaced with `DynamicPPL.NTVarInfo`.
+The _constructor_ has been replaced with `DynamicPPL.typed_varinfo`.
+
+Note that the exact kind of VarInfo returned by `VarInfo(rng, model, ...)` is an implementation detail.
+Previously, it was guaranteed that this would always be a VarInfo whose metadata was a `NamedTuple` containing `Metadata` structs.
+Going forward, this is no longer the case, and you should only assume that the returned object obeys the `AbstractVarInfo` interface.
+
+**Other changes**
+
+While these are technically breaking, they are only internal changes and do not affect the public API.
+The following four functions have been added and/or reworked to make it easier to construct VarInfos with different types of metadata:
+
+ 1. `DynamicPPL.untyped_varinfo([rng, ]model[, sampler, context])`
+ 2. `DynamicPPL.typed_varinfo([rng, ]model[, sampler, context])`
+ 3. `DynamicPPL.untyped_vector_varinfo([rng, ]model[, sampler, context])`
+ 4. `DynamicPPL.typed_vector_varinfo([rng, ]model[, sampler, context])`
+
+The reason for this change is that there were several flavours of VarInfo.
+Some, like `typed_varinfo`, were easy to construct because we had convenience methods for them; however, the others were more difficult.
+This change makes it easier to access different VarInfo types, and also makes it more explicit which one you are constructing.
+
 ## 0.35.9
 
 Fixed the `isnan` check introduced in 0.35.7 for distributions which returned NamedTuple.

Project.toml

@@ -1,6 +1,6 @@
 name = "DynamicPPL"
 uuid = "366bfd00-2699-11ea-058f-f148b4cae6d8"
-version = "0.35.9"
+version = "0.36.0"
 
 [deps]
 ADTypes = "47edcb42-4c32-4615-8424-f2b9edc5f35b"
@@ -46,7 +46,7 @@ DynamicPPLMooncakeExt = ["Mooncake"]
 [compat]
 ADTypes = "1"
 AbstractMCMC = "5"
-AbstractPPL = "0.10.1"
+AbstractPPL = "0.11"
 Accessors = "0.1"
 BangBang = "0.4.1"
 Bijectors = "0.13.18, 0.14, 0.15"

benchmarks/src/DynamicPPLBenchmarks.jl

@@ -52,8 +52,8 @@ end
 
 Create a benchmark suite for `model` using the selected varinfo type and AD backend.
 Available varinfo choices:
-  • `:untyped`           → uses `VarInfo()`
-  • `:typed`             → uses `VarInfo(model)`
+  • `:untyped`           → uses `DynamicPPL.untyped_varinfo(model)`
+  • `:typed`             → uses `DynamicPPL.typed_varinfo(model)`
   • `:simple_namedtuple` → uses `SimpleVarInfo{Float64}(model())`
   • `:simple_dict`       → builds a `SimpleVarInfo{Float64}` from a Dict (pre-populated with the model’s outputs)
 
@@ -67,11 +67,9 @@ function make_suite(model, varinfo_choice::Symbol, adbackend::Symbol, islinked::
     suite = BenchmarkGroup()
 
     vi = if varinfo_choice == :untyped
-        vi = VarInfo()
-        model(rng, vi)
-        vi
+        DynamicPPL.untyped_varinfo(rng, model)
     elseif varinfo_choice == :typed
-        VarInfo(rng, model)
+        DynamicPPL.typed_varinfo(rng, model)
     elseif varinfo_choice == :simple_namedtuple
         SimpleVarInfo{Float64}(model(rng))
     elseif varinfo_choice == :simple_dict

docs/Project.toml

@@ -1,9 +1,11 @@
 [deps]
+AbstractPPL = "7a57a42e-76ec-4ea3-a279-07e840d6d9cf"
 Accessors = "7d9f7c33-5ae7-4f3b-8dc6-eff91059b697"
 DataStructures = "864edb3b-99cc-5e75-8d2d-829cb0a9cfe8"
 Distributions = "31c24e10-a181-5473-b8eb-7969acd0382f"
 Documenter = "e30172f5-a6a5-5a46-863b-614d45cd2de4"
 DocumenterMermaid = "a078cd44-4d9c-4618-b545-3ab9d77f9177"
+DynamicPPL = "366bfd00-2699-11ea-058f-f148b4cae6d8"
 FillArrays = "1a297f60-69ca-5386-bcde-b61e274b549b"
 ForwardDiff = "f6369f11-7733-5829-9624-2563aa707210"
 JET = "c3a54625-cd67-489e-a8e7-0a5a0ff4e31b"

docs/make.jl

@@ -24,7 +24,9 @@ makedocs(;
     format=Documenter.HTML(; size_threshold=2^10 * 400),
     modules=[DynamicPPL, Base.get_extension(DynamicPPL, :DynamicPPLMCMCChainsExt)],
     pages=[
-        "Home" => "index.md", "API" => "api.md", "Internals" => ["internals/varinfo.md"]
+        "Home" => "index.md",
+        "API" => "api.md",
+        "Internals" => ["internals/varinfo.md", "internals/submodel_condition.md"],
     ],
     checkdocs=:exports,
     doctest=false,

docs/src/api.md

@@ -78,9 +78,9 @@ decondition
 
 ## Fixing and unfixing
 
-We can also _fix_ a collection of variables in a [`Model`](@ref) to certain using [`fix`](@ref).
+We can also _fix_ a collection of variables in a [`Model`](@ref) to certain values using [`DynamicPPL.fix`](@ref).
 
-This might seem quite similar to the aforementioned [`condition`](@ref) and its siblings,
+This is quite similar to the aforementioned [`condition`](@ref) and its siblings,
 but they are indeed different operations:
 
   - `condition`ed variables are considered to be _observations_, and are thus
@@ -89,19 +89,19 @@ but they are indeed different operations:
   - `fix`ed variables are considered to be _constant_, and are thus not included
     in any log-probability computations.
 
-The differences are more clearly spelled out in the docstring of [`fix`](@ref) below.
+The differences are more clearly spelled out in the docstring of [`DynamicPPL.fix`](@ref) below.
 
 ```@docs
-fix
+DynamicPPL.fix
 DynamicPPL.fixed
 ```
 
-The difference between [`fix`](@ref) and [`condition`](@ref) is described in the docstring of [`fix`](@ref) above.
+The difference between [`DynamicPPL.fix`](@ref) and [`DynamicPPL.condition`](@ref) is described in the docstring of [`DynamicPPL.fix`](@ref) above.
 
-Similarly, we can [`unfix`](@ref) variables, i.e. return them to their original meaning:
+Similarly, we can revert this with [`DynamicPPL.unfix`](@ref), i.e. return the variables to their original meaning:
 
 ```@docs
-unfix
+DynamicPPL.unfix
 ```
 
 ## Predicting
@@ -149,7 +149,7 @@ In the past, one would instead embed sub-models using [`@submodel`](@ref), which
 In the context of including models within models, it's also useful to prefix the variables in sub-models to avoid variable names clashing:
 
 ```@docs
-prefix
+DynamicPPL.prefix
 ```
 
 Under the hood, [`to_submodel`](@ref) makes use of the following method to indicate that the model it's wrapping is a model over its return-values rather than something else
@@ -212,6 +212,7 @@ To test and/or benchmark the performance of an AD backend on a model, DynamicPPL
 ```@docs
 DynamicPPL.TestUtils.AD.run_ad
 DynamicPPL.TestUtils.AD.ADResult
+DynamicPPL.TestUtils.AD.ADIncorrectException
 ```
 
 ## Demo models
@@ -300,18 +301,17 @@ AbstractVarInfo
 
 But exactly how a [`AbstractVarInfo`](@ref) stores this information can vary.
 
-For constructing the "default" typed and untyped varinfo types used in DynamicPPL (see [the section on varinfo design](@ref "Design of `VarInfo`") for more on this), we have the following two methods:
+#### `VarInfo`
 
 ```@docs
-DynamicPPL.untyped_varinfo
-DynamicPPL.typed_varinfo
+VarInfo
 ```
 
-#### `VarInfo`
-
 ```@docs
-VarInfo
-TypedVarInfo
+DynamicPPL.untyped_varinfo
+DynamicPPL.typed_varinfo
+DynamicPPL.untyped_vector_varinfo
+DynamicPPL.typed_vector_varinfo
 ```
 
 One main characteristic of [`VarInfo`](@ref) is that samples are transformed to unconstrained Euclidean space and stored in a linearized form, as described in the [main Turing documentation](https://turinglang.org/docs/developers/transforms/dynamicppl/).