NCAR Model Manager (ncarmm)

The pieces barely change from one model to the next — compilers, MPI, and the libraries that read and write model data. What changes is that someone sets them up manually again for every new model, every new version.

HDF5↔ NetCDF↔ PnetCDF↔ PIO

The classic trap: all four have to be built against the same MPI — one mismatch and you lose days.

Done by hand → compiler quirks and version mismatches, every time.
Days lost per project, and the knowledge lives in one person's head.

1:15 · running 1:45

Here's the problem we kept running into. The pipeline for these models is actually pretty consistent — the same components, in more or less the same order, whichever model you're running.

But every time we stand one up, it gets done by hand. Someone compiles it, someone installs the dependencies, someone works out which versions go together. And so we hit the same things every time: compiler issues, version matching, and this chain of I/O libraries that all have to line up.

The classic trap is the last one — all four of these have to be built against the same MPI. Get that wrong and you don't get a clean error. You get a crash hours into a run, or output that quietly doesn't write. I think most people here have lost a few days to exactly that.

None of it is hard, exactly. It's just manual, and we redo it per project — so the effort gets duplicated and the knowledge ends up in one person's head instead of somewhere shared.

Don't read the library names out one by one — point at the chain. The chain is the point.

Ease

One command gets you a cloud-ready, runnable model.

No duplicate work

One reviewed build recipe, shared across every project.

Repeatable

Every dependency pinned to an exact version, so the same software stack rebuilds the same way.

Portable

Nothing in the recipe is tied to one cloud or one delivery format.

Working today

WRF 4.4.0 WRF-Chem 4.4.0 MPAS 8.3.1

Running on

Amazon Web Services

ncarmm build mpas --version 8.3.1

1:35 · running 3:20

So the idea is simple: do this once, carefully, and let everyone reuse it. Four things we care about:

Ease — one command gets you a model that's ready to run in the cloud.
No duplicate work — one build recipe, reviewed once, shared across projects.
Repeatable — every dependency pinned to an exact version, so the stack rebuilds the same way next month as it does today.
Portable — nothing in the recipe is tied to one cloud, or to one way of delivering the model.

I want to be precise about "repeatable," because this room will care. We reproduce the build — same sources, same compiler, same library versions. That is not a promise of bit-identical forecasts. Different hardware and different MPI rank counts differ at round-off, exactly as they do on any HPC system.

Today, three models build end to end on AWS: WRF 4.4.0, WRF-Chem, and MPAS 8.3.1 — and they launch straight onto HPC-class nodes.

One clarification on WRF-Chem: that's the chemistry-enabled build of the same WRF. We give you a correctly compiled binary and its dependency stack. Emissions preprocessing and mechanism setup are still yours.

And this is what using it looks like. ncarmm build mpas, version 8.3.1. For the common case, that's the whole interface.

If asked: there's also list-models, list-images and delete. The chem model is typed "wrfchem", no hyphen. Delete is registered but currently a stub — don't demo it.

manifest.yaml Recipe Names the model and every pinned dependency.

→

automated build Build in the cloud A right-sized machine compiles the whole stack and the model, unattended.

→

snapshot Machine image Saved, labeled, and findable by anyone on the team.

→

cluster Run the model Launch that image on HPC-class nodes.

What it does and doesn't do: ncarmm builds the model and the software it needs. Staging input data and moving output back out stay part of your own workflow.

1:10 · running 4:30

Four steps. You pick or write a manifest — a small YAML file that names the model and every pinned dependency.

ncarmm hands that to an automated build in the cloud. It brings up a right-sized machine, compiles the whole stack and the model unattended — that's about three hours — and then snapshots the result into a machine image, tagged so anyone on the team can find it.

From there you launch that image onto HPC-class nodes and run your forecast.

The thing I want to be honest about is scope. ncarmm builds the model and the software it needs. Staging your input data in, and getting terabytes of output back out, is still your workflow. That's a deliberate boundary — but it does mean the model binary is maybe a third of the real problem.

If asked how it really works: CloudFormation provisions an EC2 Image Builder pipeline; the build instance (about a c6i.4xlarge, ~3 h) compiles HDF5, NetCDF, PnetCDF, PIO, METIS and the model; the snapshot is a tagged AMI that AWS ParallelCluster launches directly onto EFA-enabled nodes with FSx Lustre scratch.

If asked about restricted AWS accounts: networking self-provisions by default — a temporary VPC that gets cleaned up after — and locked-down accounts are handled with pre-created instance profiles. This was built for real NCAR accounts, not a demo sandbox.

01

More models

A new model or version is a new recipe, not a rewrite — nothing in the core changes. On the radar: CheMPAS, WRF-Hydro, FastEddy. Private models can plug in the same way, without their source living here.

02

More clouds

AWS is the one implementation today, but the cloud-specific piece is a thin, swappable layer. Google, Azure and Oracle would be additions, not a rebuild.

03

Other delivery formats

The recipe isn't tied to shipping a machine image. Containers are the obvious thing to look at — but that's exploration, not a plan.

On cost, honestly: ParallelCluster already starts and stops nodes on demand — the bill is dominated by the fast HPC machines, the fast scratch storage, and moving output back out.

1:20 · running 5:50 — compress this one if you're behind.

Three directions — and these are directions, not commitments.

More models. Adding one is a new recipe, not a rewrite; nothing in the core changes. On the radar are CheMPAS, WRF-Hydro and FastEddy. And because models plug in this way, a project can register a private model and pull restricted source with a build-time token — so a proprietary model can run on this without its code ever living in our repository. The architecture supports that today; it isn't shipped.

More clouds. AWS is the only implementation right now, but the cloud-specific piece is a thin layer — Google, Azure or Oracle would be additions, not a rebuild.

And third, other ways of delivering the model. The recipe isn't tied to producing a machine image — containers are the obvious thing to look at. But that's exploration; we haven't done the testing to say more than that.

One honest note on cost, since people ask. ParallelCluster already starts and stops nodes on demand, so idle compute isn't really the problem. The bill is dominated by the fast machines themselves, the fast scratch storage, and moving output back out.

If asked how thin that cloud layer really is: two interface functions — get_image_details and build_image. If asked about the roadmap list: the README currently names CheMPAS and FastEddy; WRF-Hydro is our stated direction and carries its own coupling complexity.

ncarmm: build the model once, and run it where NCAR computes.

Takes the duplicated, manual setup work off every project.
Working today: WRF, WRF-Chem and MPAS on AWS.
Built to extend: more models, more clouds.
Primary delivery is GitHub — github.com/NCAR/ncarmm.
Also on the internal NEXUS index, so pip install ncarmm works — in evaluation now.

1:05 · running ~7:00 — leaves ~3 min for questions.

So, where things stand. ncarmm takes the duplicated, manual setup work off every project. WRF, WRF-Chem and MPAS run on AWS today. It's built to extend — more models, and more clouds.

On getting hold of it: the primary way is the GitHub repo — that's where the source lives and that's what we point people at. It's also published to our internal NEXUS index, so if you'd rather just pip install ncarmm, that works too. We're in the evaluation stage right now: David has been getting it in front of people to see who picks it up and how much interest there is. So if this sounds useful for your project, come find one of us — that feedback is exactly what we need to decide what to prioritize.

Then stop and take questions. The backup slide has the answers you're most likely to need — it's uncounted, so press right arrow to reach it.

If asked how to get it: GitHub is the primary channel — github.com/NCAR/ncarmm. NEXUS carries it as an internal package index, so "pip install ncarmm" works inside NCAR. It is not on public PyPI, and open-sourcing is possible but not committed.

Deliberately not on this slide: open-ended discussion prompts — save the broader direction-setting conversation for when David is in the room.

Not presented — reference while taking questions.

Do cloud runs reproduce our Derecho results bit-for-bit?

No, and that's expected. We reproduce the build (pinned sources/compiler/libs). Forecasts differ at round-off across hardware, AVX paths, and MPI rank counts — same as any MPI code.

Does WRF-Chem come with the emissions preprocessors?

No. We deliver the chem-enabled WRF binary + dependency stack; emissions / mechanism setup is application-specific and stays with the user.

Could this deliver containers instead of machine images?

Nothing in the build recipe ties it to one delivery format, so it's a direction we're interested in — but we haven't done the testing to claim it as a feature. Treat it as future work.

How do I get data in/out, and what's the cost?

Input staging + output egress are the user's workflow (FSx Lustre + S3). Cost is driven by EFA HPC instances, Lustre, and egress; ParallelCluster already starts and stops nodes on demand, so idle compute is not the main lever.

How do I install it — is it open source?

Primary delivery is GitHub (github.com/NCAR/ncarmm); it's also on the internal NEXUS index, so pip install ncarmm works inside NCAR. In evaluation now. Not on public PyPI; open source is possible, not committed.

Can we run a proprietary / non-public model (e.g. WRF WxMod, a private FastEddy)?

Yes, by design. A downstream project imports the model registry, registers its own manifest + Image Builder recipe, and supplies a build-time auth token (OAuth / PAT) to pull the restricted source. The proprietary code never lives in ncarmm — the framework just builds & runs it. Registry hook is a near-term design item.

How does the build actually run on AWS?

CloudFormation provisions an EC2 Image Builder pipeline; a right-sized instance (≈ c6i.4xlarge, ~3 h) compiles HDF5 / NetCDF / PnetCDF / PIO / METIS + the model, then snapshots a tagged AMI that ParallelCluster launches onto EFA-enabled nodes with FSx Lustre scratch.