Other output

Next to the NetCDF output there's other ways to output data from the model, these are described below.

JLD2 Output

As an alternative to the NetCDF output, it is also possible to directly output the Variables (or one subgroup of it) to a JLD2 file. This might be interesting if you are really interested in the model internals, or also for some machine learning tasks. However, this option doesn't feature any possibilites to regrid or select variables, and it comes with the usual limitations of serialized JLD2 data: SpeedyWeather.jl always has to be in the scope when loading the data and the saved files might only load properly with exactly the same version of SpeedyWeather.jl and Julia as used when saving the data. Its usage is similar to the NetCDF output above:

using SpeedyWeather
spectral_grid = SpectralGrid(trunc=31, nlayers=1)
output = JLD2Output(output_dt=Hour(1))
model = ShallowWaterModel(spectral_grid, output=output)
model.output

JLD2Output
├ status: inactive/uninitialized
├ write restart file: true (if active)
├ path: output.jld2
├ groups: (:all,)
└ frequency: 3600 seconds

With all options shown below

@doc JLD2Output

Output writer for a JLD2 file that saves the Variables struct directly to a JLD2 file. All internal scalings and units are still applied to these outputs. Fields are

• active::Bool

• path::String: [OPTION] path to output folder, run_???? will be created within

• run_prefix::String: [OPTION] Prefix for run folder where data is stored, e.g. 'run_'

• id::String: [OPTION] run identification, added between runprefix and runnumber

• run_number::Int64: [OPTION] run identification number, automatically determined if overwrite=false

• run_digits::Int64: [OPTION] run numbers digits

• overwrite::Bool: [OPTION] Overwrite an existing run folder?

• filename::String: [OPTION] name of the output jld2 file

• write_restart::Bool: [OPTION] also write restart file if output=true?

• write_parameters_txt::Bool: [OPTION] also write parameters txt file if output=true?

• write_progress_txt::Bool: [OPTION] also write progress txt file if output=true?

• output_dt::Second: [OPTION] output frequency, time step

• merge_output::Bool: [OPTION] will reopen and resave the file to merge everything in one big vector. Turn off if the file is too large for memory.

• groups::Tuple{Vararg{Symbol}}: [OPTION] which variable groups to save, e.g. (:prognostic,) or (:prognostic, :grid). Use (:all,) for all groups.

• core::OutputWriterCore: [DERIVED] shared output writer state (run folder, counters, output frequency)

• jld2_file::Union{Nothing, JLD2.JLDFile}

Array Output (to RAM)

It's also possible to output the Variables (or a subgroup of it) directly into an array that is kept in the memory. This might be useful e.g. for low-resolution simulations you want to work with and visualize quickly, or also for simulations within ML training loops. ArrayOutput follows the exact same logic as JLD2Output except that the actual output is kept in an array output. So, save e.g. all prognostic variables to memory, you can run:

using SpeedyWeather
spectral_grid = SpectralGrid(trunc=31, nlayers=1)
output = ArrayOutput(output_dt=Hour(1), groups=(:prognostic,))
model = ShallowWaterModel(spectral_grid, output=output)
model.output

ArrayOutput
├ status: inactive/uninitialized
├ snapshots stored: 0
├ groups: (:prognostic,)
└ frequency: 3600 seconds

After a succesfull run! the result is stored in output.output.

Parameter summary

With output=true as an argument in the run!(simulation) call, the NetCDFOutput by default also writes a parameter summary into parameters.txt in the same folder. This is implemented as a Callbacks (<: SpeedyWeather.AbstractCallback) and can be added independent of NetCDFOutput too. After output=true this callback is found here as ParametersTxt

# run an example simulation with output
simulation = initialize!(model)
run!(simulation, period=Hour(4), output=true)

# now model.output will have added additional "output" writers
simulation.model.callbacks

Dict{Symbol, SpeedyWeather.AbstractCallback} with 3 entries:
  :progress_txt           => ProgressTxt <: SpeedyWeather.AbstractCallback…
  :parameters_txt         => ParametersTxt <: SpeedyWeather.AbstractCallback…
  :variables_restart_file => WriteVariablesRestartFile <: SpeedyWeather.Abstrac…

but it's activity is by default tied to activity of the NetCDFOutput with you can control with write_only_with_output. Creating such a callback independently

parameters_txt = ParametersTxt(write_only_with_output=false)

ParametersTxt <: SpeedyWeather.AbstractCallback
├ path::String = 
├ filename::String = parameters.txt
└ write_only_with_output::Bool = false

we can add it with a random or manual key as

add!(model, parameters_txt)             # random key
add!(model, :my_key => parameters_txt)  # manual key

Dict{Symbol, SpeedyWeather.AbstractCallback} with 5 entries:
  :progress_txt           => ProgressTxt <: SpeedyWeather.AbstractCallback…
  :my_key                 => ParametersTxt <: SpeedyWeather.AbstractCallback…
  :parameters_txt         => ParametersTxt <: SpeedyWeather.AbstractCallback…
  :callback_ZKxU          => ParametersTxt <: SpeedyWeather.AbstractCallback…
  :variables_restart_file => WriteVariablesRestartFile <: SpeedyWeather.Abstrac…

But note that callbacks are overwritten with identical keys, otherwise treated independently. Meaning now we have the preexisting :parameters_txt callback and then the two we just added, writing their parameters.txt files one after another, overwriting that same file two times.

Progress txt

Similarly to ParametersTxt, a callback ProgressTxt is by default added with output=true. They can be created independently too

progress_txt = ProgressTxt(write_only_with_output=false, path="myfolder", filename="letsgo.txt")

ProgressTxt <: SpeedyWeather.AbstractCallback
├ path::String = myfolder
├ filename::String = letsgo.txt
├ write_only_with_output::Bool = false
├ every_n_percent::Int64 = 5
└ file::IOStream = IOStream()

and added like

add!(model, :progress_txt => progress_txt)

Dict{Symbol, SpeedyWeather.AbstractCallback} with 5 entries:
  :progress_txt           => ProgressTxt <: SpeedyWeather.AbstractCallback…
  :my_key                 => ParametersTxt <: SpeedyWeather.AbstractCallback…
  :parameters_txt         => ParametersTxt <: SpeedyWeather.AbstractCallback…
  :callback_ZKxU          => ParametersTxt <: SpeedyWeather.AbstractCallback…
  :variables_restart_file => WriteVariablesRestartFile <: SpeedyWeather.Abstrac…

Restart files for variables

NetCDFOutput also by default writes a restart file, containing the simulation.variables.prognostic that can be read back in with the StartFromFile initial conditions. Implemented as a callback WriteVariablesRestartFile can also be created independently of NetCDFOutput, e.g.

restart_file = WriteVariablesRestartFile(write_only_with_output=false, path="folder1", filename="restart.jld2")

WriteVariablesRestartFile <: SpeedyWeather.AbstractCallback
├ filename::String = restart.jld2
├ path::String = folder1
├ compress::Bool = true
├ write_only_with_output::Bool = false
└ pkg_version::VersionNumber = 0.19.0

and added like

add!(model, :restart_file => restart_file)

Dict{Symbol, SpeedyWeather.AbstractCallback} with 6 entries:
  :restart_file           => WriteVariablesRestartFile <: SpeedyWeather.Abstrac…
  :progress_txt           => ProgressTxt <: SpeedyWeather.AbstractCallback…
  :my_key                 => ParametersTxt <: SpeedyWeather.AbstractCallback…
  :parameters_txt         => ParametersTxt <: SpeedyWeather.AbstractCallback…
  :callback_ZKxU          => ParametersTxt <: SpeedyWeather.AbstractCallback…
  :variables_restart_file => WriteVariablesRestartFile <: SpeedyWeather.Abstrac…

By default path="" will use the folder determined by NetCDFOutput but otherwise you can also provide your own. Note that WriteVariablesRestartFile will only write the prognostic variables to file. This is such that you can simulate a spin up and then change model parameters as you like, to write out specific model components and store them in a file see Model component restart file.

Model component restart file

If you modified a model component (say by applying a custom orography) you can save this to file too.

orography = ManualOrography(spectral_grid)  # an orography that is untouched at initialize!
model = ShallowWaterModel(spectral_grid; orography)
set!(model, orography=123)                  # set as you like
orography_for_restart = WriteModelComponentFile(component=model.orography, filename="my_orography.jld2")
add!(model, :my_orography => orography_for_restart)

Dict{Symbol, SpeedyWeather.AbstractCallback} with 1 entry:
  :my_orography => WriteModelComponentFile{ManualOrography{Field{Float32...} <:…

Once the simulation ran you can then load this model component from file and use it to construct a new model with it, or use its information in some other form, e.g. by writing its arrays to other arrays:

simulation = initialize!(model)
run!(simulation, steps=0, output=true)

# either pass on that same callback (which will read out the path) or provide path directly
my_orography = SpeedyWeather.load_model_component(orography_for_restart)

# construct a new model with that orography
new_model = ShallowWaterModel(spectral_grid, orography = my_orography)
all(new_model.orography.orography .== 123)  # check that the new orography is indeed as customized

true

We do not really want to encourage it, but WriteModelComponentFile can be hijacked to write out the entire model. While this easily saves everything of model into one file, it always writes many large precomputed arrays to file whereas they could just be recomputed when constructing a new model. For example, the Legendre polynomials in model.spectral_transform can easily be GBs at higher resolution, see also Precomputed polynomials and allocated memory. Nevertheless, you can write out the entire model with

add!(model, :model_writer => WriteModelComponentFile(component=model, filename="model.jld2"))

Dict{Symbol, SpeedyWeather.AbstractCallback} with 5 entries:
  :progress_txt           => ProgressTxt <: SpeedyWeather.AbstractCallback…
  :my_orography           => WriteModelComponentFile{ManualOrography{Field{Floa…
  :parameters_txt         => ParametersTxt <: SpeedyWeather.AbstractCallback…
  :model_writer           => WriteModelComponentFile{ShallowWaterModel{Spectral…
  :variables_restart_file => WriteVariablesRestartFile <: SpeedyWeather.Abstrac…

Note that this callback contains a model that also contains this callback. This self recursion is not particularly problematic as model is just a lazy reference. However, when you do load in this model from file and use it again, note that it again contains this callback which would write out its model again. You can delete! the callback though, see Adding a callback.