ABSTRACT:

The CyberGIS-Jupyter for Water (CJW) platform aims to advance community hydrologic modelling, and support data-intensive, reproducible, and computationally scalable water science research by simplifying access to advanced cyberGIS and cyberinfrastructure capabilities through a friendly Jupyter Notebook environment. The current release has specific support for the Structure For Unifying Multiple Modeling Alternatives (SUMMA) model and the WRFHydro model.

You may open and view any notebook (*.ipynb file) with this app.

Please send comments and bug reports to help@cybergis.org.

ABSTRACT:

CyberGIS-Jupyter for Water Quarterly Release Announcement (2020 Q2)

Dear HydroShare Users,

We are pleased to announce a new quarterly release of CyberGIS-Jupyter for Water (CJW) platform at https://go.illinois.edu/cybergis-jupyter-water. This release includes new capabilities to support the geoanalytics suite of GRASS for model pre/post-processing, PostGIS database, and Landlab Earth surface modelling toolkit along with several enhancements to job submission middleware, system security as well as service infrastructure. Please refer to the following list for details and examples.

Please let us know if you have any questions or run into any problems (help@cybergis.org). Any feedback would be greatly appreciated.

Best regards,
CyberGIS-Hydro team

GRASS GIS for model pre/post-processing:
Learn how to consolidate the features of the GRASS geoanalytics suite to support pre/post-processing for SUMMA and RHESSYs models in CJW.
Example notebooks: https://www.hydroshare.org/resource/4cbcfdd6e7f943e2969dd52e780bc52d/

Manage geospatial data with PostGIS:
PostGIS is an extension to the PostgreSQL object-relational database system which allows geospatial data to be efficiently stored while providing various advanced functions for in-situ data analysis and processing.
Example notebooks: https://www.hydroshare.org/resource/bb779d4cce564dd6afcf463c8910786f/

Security and service infrastructure enhancements
Trusted group: Starting from this release, all users are required to join the “CyberGIS-Jupyter for Water” trusted group at https://www.hydroshare.org/group/157 in order to access the CJW platform, which is a preventive measure to protect the shared computing resources from being abused by malicious users. A complete user profile page is highly recommended to expedite the approval process.
User metric submission to XSEDE: CJW, as a science gateway, is now sending unique user usage metrics to XSEDE to comply with its requirements.

Landlab for enabling collaborative numerical modeling in Earth sciences using knowledge infrastructure
Example notebooks: https://www.hydroshare.org/resource/370c288b61b84794b847ef85c4dd4ffb/
https://www.hydroshare.org/resource/6add6bee06bb4050bfe23e1081627614/

Job submission enhancements
Refactored the structure of the cyberGIS job submission system
Data-driven implementation for avoiding excessive data transmission between HydroShare and CJW
Add the specification of input parameters into a JSON file to improve the flexibility and generality of model management
Enable HPC-SUMMA object that can directly call SUMMA
Example notebooks: https://www.hydroshare.org/resource/4a4a22a69f92497ead81cc48700ba8f8/

ABSTRACT:

We are pleased to announce a new quarterly release of the CyberGIS-Jupyter for Water (CJW) platform at https://go.illinois.edu/cybergis-jupyter-water. This release includes several new capabilities and features summarized as below.

1) Modeling CAMELS Basins with SUMMA: CAMELS (Catchment Attributes and Meteorology for Large-sample Studies: https://ral.ucar.edu/solutions/products/camels) is a large-sample hydrometeorological dataset that provides catchment attributes and forcings for 671 small- to medium-sized basins across the CONUS. In this release, CJW has included enhancements and new features that support the end-to-end workflow of CAMELS basin modeling with SUMMA. An example notebook is provided to walk users through several essential steps including basin data retrieval and subsetting, setup of single and ensemble models, computation job submission and execution, and model output visualization.

2) RHESSys support via CyberGIS Computing Service: CJW now supports running ensemble RHESSys models on HPC (High-Performance Computing) resources through the newly upgraded CyberGIS Computing Service. Also, the RHESSys Jupyter kernel has been updated to include the latest version of “pyRHESSys” (https://github.com/uva-hydroinformatics/pyRHESSys) and other new tools for model configuration, output analysis, and visualization. See the example notebook below for more details.

3) User testing of Kubernetes-based CJW instance: A newly deployed CJW instance powered by Kubernetes (Aka K8s: https://kubernetes.io/) is now available for user testing at https://go.illinois.edu/cjw-k8s. The adoption of this most sought-after and cutting-edge cloud application deployment technology is expected to significantly enhance the availability and scalability of CJW as we have observed increasing user demand and a surge in new user registrations. We welcome all users to join this testing process and would greatly appreciate your feedback. We anticipate the user testing on the new CJW instance will take 1-3 months, during which the current production CJW (http://go.illinois.edu/cybergis-jupyter-water) will continue to be available in parallel until a final migration plan will be implemented before the next quarterly release of CJW.

Please refer to the following HydroShare resources for details and examples:
Modeling CAMELS Basins with SUMMA:
https://www.hydroshare.org/resource/17bc4f0031554944b8ec7558fd9ee3c2/

Run Ensemble RHESSys models on HPC through CyberGIS Computing Service:
https://www.hydroshare.org/resource/631914af4b8344e5a78e647255cf1d13/

Direct Access to Kubernetes-based CJW:
https://go.illinois.edu/cjw-k8s

Set up OpenWith for Kubernetes-based CJW:
https://www.hydroshare.org/resource/e9686eadd4474b6587d83d9330d25854/

See Release Notes on HydroShare
https://www.hydroshare.org/resource/54f3ec517ba44a83bb486e7d6c4edceb/

Please let us know if you have any questions or run into any problems (help@cybergis.org). Any feedback would be greatly appreciated.

ABSTRACT:

The HydroShare project is pleased to bring you this notebook that can set up a run-time environment on the CyberGIS-Jupyter for Water (CJW) platform for WRFHydro Hands-on Training v5.2.x (Nov 2020). In contrast to the Docker-based local setup, this HydroShare solution does not require installation or downloading of any software or data onto your local computer, and it enables you to access to more powerful computing resources in a clould-based CJW environment. All necessary materials required to complete this training are remotely accessible through a browser (Google Chrome recommended).

This notebook will retrieve the WRFHydro model codes and relevant data from different official repos on Github and Google Drive managed by the NCAR/UCAR WRFHydro Development Team, and put them in a similar directory structure as the Docker-based local setup.

How to run the notebook:
1) Click on the OpenWith button in the upper-right corner;
2) Select "CyberGIS-Jupyter for Water";
3) Open the notebook and follow instructions;

ABSTRACT:

The HydroShare project is pleased to bring you this notebook that can set up a run-time environment on the CyberGIS-Jupyter for Water (CJW) platform for WRF&WRF-Hydro Coupled Testcase Online Lesson (v5.1.2). In contrast to the Docker-based local setup, this HydroShare solution does not require installation or downloading of any software or data onto your local computer, and it enables you to access to more powerful computing resources in a clould-based CJW environment. All necessary materials required to complete this training are remotely accessible through a browser (Google Chrome recommended).

This notebook retrieves the WRFHydro model codes and relevant data from different official repos on Github and Google Drive managed by the NCAR/UCAR WRFHydro Development Team, and puts them in certain directory structure (same as the Docker-based local setup) required by the training notebooks. Specifically, two new folders will be created (wrf-hydro-training, and WRF_WPS) alongside. The training notebooks are stored in wrf-hydro-training --> lessons as shown below.

How to run the notebook:
1) Click on the OpenWith button in the upper-right corner;
2) Select "CyberGIS-Jupyter for Water";
3) Open the notebook and follow instructions;

ABSTRACT:

This is a collection that holds all the WRFHydro official training materials you can run on CyberGIS-Jupyter for Water without installing or downloading any software or data onto your local computer. This collection will expand as new training lessons be added.

ABSTRACT:

We are pleased to announce a new quarterly release of the CyberGIS-Jupyter for Water (CJW) platform at https://go.illinois.edu/cybergis-jupyter-water. This release includes several new capabilities and features summarized as below.

1) Support for NCAR/UCAR WRFHydro Training Notebooks: Users are now able to set up the necessary environment to run WRFHydro hands-on training notebooks (https://ral.ucar.edu/projects/wrf_hydro/training-materials) through simple clicks on CJW. For details on how to run these training notebooks please review the following HydroShare resources: (a) WRFHydro Hands-on Training v5.2.x on CJW; and (b) WRF&WRFHydro Coupled Training v5.1.2 on CJW. This provides an alternative solution to the traditional Docker-based local setup (https://hub.docker.com/r/wrfhydro/training/) that makes it easy for users to complete this training as it does not require installation or downloading of any software or data onto the user’s local computer. Additionally, it enables users to access more powerful computing resources in a cloud-based Jupyter environment. All necessary materials required to complete the training are retrieved from official data sources managed by the NCAR/UCAR WRFHydro Development Team and accessible on CJW via a browser.

2) Transition from XSEDE Comet to Expanse: Comet HPC will be decommissioned on July 31, 2021 (https://portal.xsede.org/sdsc-comet), and all Comet allocations and resources awarded to HydroShare/CJW will be transferred to the Expanse HPC (https://portal.xsede.org/sdsc-expanse). As a result, the CyberGIS-Compute service will also drop the support on Comet and replace it with Expanse. Despite this change, we do not expect any action is required for the majority of users. The CyberGIS-Compute service and its SDK will redirect all jobs submitted to Comet to Expanse with a warning message showing up. All previously developed notebooks that use Comet will continue to run after this transition. Please contact us for solutions if you have models or notebooks that access Comet without going through the CyberGIS-Compute service.

3) Extended user testing of Kubernetes-based CJW instance: In the previous release, we announced a Kubernetes-based (Aka K8s: https://kubernetes.io/) CJW instance deployed at https://go.illinois.edu/cjw-k8s for user testing with a preliminary migration plan on deprecation of the current DockerSwarm-based CJW. Due to the growing complexity of K8s and more features being developed and added for a smooth user experience, we have decided to continue conducting extensive testing in this release. We encourage all users to join this testing process and would greatly appreciate your feedback. The current production CJW (http://go.illinois.edu/cybergis-jupyter-water) will continue to be available in parallel until a final migration plan is implemented.

Please refer to the following HydroShare resources for details and examples:
Run WRFHydro Hands-on Training v5.2.x on CJW
https://www.hydroshare.org/resource/d2c6618090f34ee898e005969b99cf90/
Run WRF&WRFHydro Coupled Training v5.1.2 on CJW
https://www.hydroshare.org/resource/c2389a2f05564da08ab218e59bdf1e81/
User testing on Kubernetes-based CJW:
https://go.illinois.edu/cjw-k8s
Set up OpenWith for Kubernetes-based CJW:
https://www.hydroshare.org/resource/e9686eadd4474b6587d83d9330d25854/
See Release Notes on HydroShare
https://www.hydroshare.org/resource/6a1ddb17155a4b27b885f442ad14e344/

Please let us know if you have any questions or run into any problems (help@cybergis.org). Any feedback would be greatly appreciated.

Best regards,
CyberGIS-Hydro team

ABSTRACT:

This notebook demonstrates how to use Globus within CyberGIS-Compute to retrieve a large number of outputs generated by a model executed on HPC, which is often needed for postprocessing work performed on CJW. A new “data transfer” job type is provided for moving data from HPC back to the CJW Jupyter environment. Under the hood, this new job type utilizes the Globus service (https://www.globus.org/) to perform a point-to-point data transfer between HPC and CJW.

In this demo, we will first prepare a 60-member ensemble SUMMA mode and submit it to the XSEDE Expanse HPC for execution using the CyberGIS-Compute. When the model run is finished, we won't use the regular "download" function in the Compute SDK to retrieve the results. Instead, we submit another Globus job to the Compute, which will hand it off to the Globus scheduler and monitor the process (just like talking Slurm scheduler on HPC in the case of regular model submission). Please refer to the example notebook below for more details.

How to run the notebook:
1) Click on the OpenWith button in the upper-right corner;
2) Select "CyberGIS-Jupyter for Water";
3) Open the notebook and follow instructions;

ABSTRACT:

Most of this notebook is going over advanced options and technical details behind our new design. There are however a few key things all users should know:

1 What do the different kernel names/versions mean?
2 Paths to some executables might have changed.
3 We have a new cjw command to manage kernel versions.

How to run the notebook:
1) Click on the OpenWith button in the upper-right corner;
2) Select "CyberGIS-Jupyter for Water";
3) Open the notebook and follow instructions;

ABSTRACT:

An example notebook walks you through how to setup customized kernels on CyberGIS-Jupyter for Water (CJW)

How to run the notebook:
1) Click on the OpenWith button in the upper-right corner;
2) Select "CyberGIS-Jupyter for Water";
3) Open the notebook and follow instructions;

ABSTRACT:

This notebook demonstrates how to configure ensemble runs for SUMMA 3.0 model and submit to a High-Performance Computing (HPC) resource for parallel execution though the CyberGIS-Compute V2 service. The content is intended for end users who want to use the SUMMA model in CJW environment. For developers who may want to contribute new models to CyberGIS-Compute, the implementation of SUMMA on Github (https://github.com/cybergis/cybergis-compute-v2-summa) can serve as an example.

Some steps in this notebook require user interaction. "Run cell by cell" is recommended. "Run All" may not work as expected.

How to run the notebook:
1) Click on the OpenWith button in the upper-right corner;
2) Select "CyberGIS-Jupyter for Water";
3) Open the notebook and follow instructions;

ABSTRACT:

Dear CUAHSI community members,

We are pleased to announce a new quarterly release of the CyberGIS-Jupyter for Water (CJW) platform at https://go.illinois.edu/cybergis-jupyter-water. This release includes several new capabilities and features summarized as follows.

Transition to CyberGIS-Compute V2:
CyberGIS-Compute V2 is a new development phase of the CyberGIS-Compute framework that was initially released (denoted as V1) one year ago through CJW 2020-Q4. Compared with V1, V2 includes several major enhancements: 1) a new workflow for model contribution to facilitate adding new hydrologic models by community developers; 2) a GUI in the notebook environment to simplify and guide users through the job submission process; 3) transparent and bi-directional data transfers between CJW and high-performance computing (HPC) resources using Globus by default, and 4) detailed tracking of usage and metrics. It is worth noting that due to the upgraded architecture in V2, existing models implemented in V1 would need to be migrated. For a smooth transition and backward compatibility, services in V1 will remain available in parallel to those in V2, and all the old notebooks that use V1 remain functional.

SUMMA Model Migrated to CyberGIS-Compute V2:
We have migrated the SUMMA model to CyberGIS-Compute V2, and end users can now benefit from the new features mentioned above in SUMMA modeling work. Please refer to the example notebook below for details. In addition, the implementation of SUMMA in CyberGIS-Compute V2 is accessible on a Github repo (https://github.com/cybergis/cybergis-compute-v2-summa), which can serve as a real-world example to model developers who may want to contribute their models for sharing with the community. A “HelloWorld” implementation is also available serving as a model-agnostic example (https://github.com/cybergis/cybergis-compute-mpi-helloworld).

New Modules and Kernel Customization:
Upon user requests, two new easybuild-based modules have been added to the CJW toolbox and are now ready to use: NCL (https://www.ncl.ucar.edu/) for scientific data analysis and visualization (e.g., NetCDF, GRID, HDF); and CDO (https://code.mpimet.mpg.de/projects/cdo) for manipulation of climate and Numerical Weather Prediction (NWP) data. Furthermore, for advanced users who may want to customize the provided software environment and kernels, an example notebook (see below) is available for users to walk through the basics on how to install new libraries on top of existing environments or set up a Conda environment from scratch.

New UI Elements on CJW:
CJW has further customized the Jupyter Notebook user interface to include a virtual Announcement Board (in the header area) for timely communicating with users on upcoming events including downtimes and new releases, and a Bug Report button (at the upper-right corner) that opens an issue tracker page in a publicly accessible Github repo for quick bug reporting.

Please refer to the following resources for details and examples:
Run ensemble SUMMA 3.0 model with CyberGIS-Compute V2
https://www.hydroshare.org/resource/deac1b0b5b46415aaedb886b9dc16f45/

Customize Software Environment on CJW
https://www.hydroshare.org/resource/461a8a853d8e42a8ae170c68c4cfa8f1/

Implementation of SUMMA model using CyberGIS-Compute V2
https://github.com/cybergis/cybergis-compute-v2-summa

Implementation of HelloWorld model using CyberGIS-Compute V2
https://github.com/cybergis/cybergis-compute-mpi-helloworld

See Release Notes on HydroShare
https://www.hydroshare.org/resource/2086b241b16b453d827db847e8640475/

Please let us know if you have any questions or run into any problems (help@cybergis.org). Any feedback would be greatly appreciated.

Best regards,
CyberGIS-Hydro team

ABSTRACT:

This resource references the github repo (https://github.com/cybergis/cybergis-compute-v2-summa) implemented support for running ensemble SUMMA models on HPC resources via CyberGIS-Compute V2.

Model developers who may want to contribute other models to CyberGIS-Compute can use this repo as an example.

For end-users (mode users), please refer to the following resource for submitting an ensemble summa model
https://www.hydroshare.org/resource/deac1b0b5b46415aaedb886b9dc16f45/

ABSTRACT:

This resource references the github repo that implemented a MPI-based HelloWorld toy model for CyberGIS-Compute V2.
Model developers who may want to contribute their own models to CyberGIS-Compute can use this as an example.

For end-users, an example notebook is provided for running the toy model on a supported HPC resource:
https://github.com/cybergis/cybergis-compute-mpi-helloworld

ABSTRACT:

This notebook demonstrates how to prepare a WRFHydro model on CyberGIS-Jupyter for Water (CJW) for execution on a supported High-Performance Computing (HPC) resource via the CyberGIS-Compute service. First-time users are highly encouraged to go through the [NCAR WRFHydro Hands-on Training on CJW](https://www.hydroshare.org/resource/d2c6618090f34ee898e005969b99cf90/) to get familiar WRFHydro model basics including compilation of source code, preparation of forcing data and typical model configurations. This notebook will not cover those topics and assume users already have hands-on experience with local model runs.

CyberGIS-Compute is a CyberGIS-enabled web service sits between CJW and HPC resources. It acts as a middleman that takes user requests (eg. submission of a model) originated from CJW, carries out the actual job submission of model on the target HPC resource, monitors job status, and retrieves outputs when the model execution has completed. The functionality of CyberGIS-Compute is exposed as a series of REST APIs. A Python client, [CyberGIS-Compute SDK](https://github.com/cybergis/cybergis-compute-python-sdk), has been developed for use in the CJW environment that provides a simple GUI to guide users through the job submission process. Prior to job submission, model configuration and input data should be prepared and arranged in a certain way that meets specific requirements, which vary by models and their implementation in CyberGIS-Compute. We will walk through the requirements for WRFHydro below.

The general workflow for WRFHydro in CyberGIS-Compute works as follows:

1. User picks a Model_Version of WRFHydro to use;
2. User prepares configuration files and data for the model on CJW;
3. User submits configuration files and data to CyberGIS-Compute;
4. CyberGIS-Compute transfers configuration files and data to target HPC;
5. CyberGIS-Compute downloads the chosen Model_Version of WRFhydro codebase on HPC;
6. CyberGIS-Compute applies compile-time configuration files to the codebase, and compiles the source code on the fly;
7. CyberGIS-Compute applies run-time configuration files and data to the model;
8. CyberGIS-Compute submits the model job to HPC scheduler for model execution;
9. CyberGIS-Compute monitors job status;
10. CyberGIS-Compute transfers model outputs from HPC to CJW upon user request;
11. User performs post-processing work on CJW;

Some steps in this notebook require user interaction. "Run cell by cell" is recommended. "Run All" may not work as expected.

How to run the notebook:
1) Click on the OpenWith button in the upper-right corner;
2) Select "CyberGIS-Jupyter for Water";
3) Open the notebook and follow instructions;

ABSTRACT:

CyberGIS-Jupyter for Water Quarterly Release Announcement (2022 Q1) [full-version]

Dear CUAHSI community members,

We are pleased to announce a new release of the CyberGIS-Jupyter for Water (CJW) platform at https://go.illinois.edu/cybergis-jupyter-water. This release includes several new capabilities and features summarized as follows.

1) Integration of WRFHydro model with CyberGIS-Compute V2 to simplify access to High-Performance Computing (HPC) environments: A newly developed computation job template in CyberGIS-Compute enables users to configure a WRFHydro model and submit it to a HPC resource for execution. The client tool of the CyberGIS-Compute suite, CyberGIS-Compute SDK, walks users through job configuration, data transfer, job submission, and job status monitoring in a guided graphical interface. Since the overhead of HPC access is handled by CyberGIS-Compute, users can now focus on the modeling work. Currently, the implementation allows users to change almost every setting and configuration for a WRFHydro 5.x “offline run”. The whole process described above can be accomplished entirely within a notebook environment on CJW. Please refer to the example notebooks below for additional details.

2) Transition to JupyterLab: Starting with this release, CJW will launch the “next-generation notebook interface”, JupyterLab, as the default user environment. Although the new interface is different from the classic Notebook interface in many places, we anticipate this transition would be easy and smooth for most users. All existing notebooks should continue to run without modification, and the bug report and announcement UI elements have been migrated to the Lab interface. In addition, we have integrated the CUAHSI “HydroShare-on-Jupyter” extension - a handy tool that enables users to move data between CJW and HydroShare through a simple graphical user interface.

3) The “cjw” Command Line Interface (CLI): The “cjw” CLI is designed to help users manage different kernels on CJW for advanced use cases. For example, users can use this capability to set up personal kernels that will persist between sessions. For a quick start, open a terminal on CJW and try out the "cjw -h" command. Check out the documentation and examples below.

4) New Modules and Kernels: To support the latest RHESSys codebase, we have added Clang, a new C family compiler supplementing the existing GCC suite, to the CJW Easybuild-based toolbox. Accordingly, a new versioned RHESSys (2022-03) kernel has been created with Clang and other development tools pre-activated that are necessary for compilation of the RHESSys source code. Upon user requests, a new versioned WRFHydro (2022-03) kernel has been created to include the hvPlot toolset for advanced data visualization and updated versions of all the libraries from the previous WRFHydro (2021-09) kernel.

Please refer to the following resources for details and examples:
Run WRFHydro 5.x model on HPC with CyberGIS-Compute V2
https://www.hydroshare.org/resource/cc28d769943046fdac0f9b0c0fc2afc6/

Implementation of WRFHydro 5.x model using CyberGIS-Compute V2
https://www.hydroshare.org/resource/329ede31b88942c489aca3111b076446/

Customize Software Environment on CJW
https://www.hydroshare.org/resource/461a8a853d8e42a8ae170c68c4cfa8f1/

“cjw” Command Line Interface Documentation
https://cybergis.github.io/cybergisx-cli/cjw/

See Release Notes on HydroShare
https://www.hydroshare.org/resource/b0d094eef336427ab605066e166135d3/

Please let us know if you have any questions or run into any problems (help@cybergis.org). Any feedback would be greatly appreciated.

Best regards,
CyberGIS-Hydro team

ABSTRACT:

This resource references the github repo (https://github.com/cybergis/cybergis-compute-v2-wrfhydro) implemented support for running WRFHydro models on HPC resources via CyberGIS-Compute V2.

Model developers who may want to contribute other models to CyberGIS-Compute can use this repo as an example.

For end-users (mode users), please refer to the following resource for submitting an ensemble summa model
https://www.hydroshare.org/resource/cc28d769943046fdac0f9b0c0fc2afc6/

ABSTRACT:

(This collection holds major CJW announcements with full-text of the most recent and important ones repeated in the Abstract section)
(For the latest features and example notebooks please refer to the links to Release Announcement in "Collection Content" down below.)
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Updated on 07/13/2022
CJW 2022-Q2 release is live. Check it out at http://go.illinois.edu/cybergis-jupyter-water
For release notes: https://www.hydroshare.org/resource/34b04302d8b34cc6aab826f79b5e3802/

---------
5/18/2022 (Updated on 12PM CT)

Globus service interruption has been resolved on SDSC Expanse HPC. Job submission to Expanse is back online.

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
03/2022
CyberGIS-Jupyter for Water Quarterly Release Announcement (2022 Q1) [full-version]

Dear CUAHSI community members,

We are pleased to announce a new release of the CyberGIS-Jupyter for Water (CJW) platform at https://go.illinois.edu/cybergis-jupyter-water. This release includes several new capabilities and features summarized as follows.

1) Integration of WRFHydro model with CyberGIS-Compute V2 to simplify access to High-Performance Computing (HPC) environments: A newly developed computation job template in CyberGIS-Compute enables users to configure a WRFHydro model and submit it to a HPC resource for execution. The client tool of the CyberGIS-Compute suite, CyberGIS-Compute SDK, walks users through job configuration, data transfer, job submission, and job status monitoring in a guided graphical interface. Since the overhead of HPC access is handled by CyberGIS-Compute, users can now focus on the modeling work. Currently, the implementation allows users to change almost every setting and configuration for a WRFHydro 5.x “offline run”. The whole process described above can be accomplished entirely within a notebook environment on CJW. Please refer to the example notebooks below for additional details.

2) Transition to JupyterLab: Starting with this release, CJW will launch the “next-generation notebook interface”, JupyterLab, as the default user environment. Although the new interface is different from the classic Notebook interface in many places, we anticipate this transition would be easy and smooth for most users. All existing notebooks should continue to run without modification, and the bug report and announcement UI elements have been migrated to the Lab interface. In addition, we have integrated the CUAHSI “HydroShare-on-Jupyter” extension - a handy tool that enables users to move data between CJW and HydroShare through a simple graphical user interface.

3) The “cjw” Command Line Interface (CLI): The “cjw” CLI is designed to help users manage different kernels on CJW for advanced use cases. For example, users can use this capability to set up personal kernels that will persist between sessions. For a quick start, open a terminal on CJW and try out the "cjw -h" command. Check out the documentation and examples below.

4) New Modules and Kernels: To support the latest RHESSys codebase, we have added Clang, a new C family compiler supplementing the existing GCC suite, to the CJW Easybuild-based toolbox. Accordingly, a new versioned RHESSys (2022-03) kernel has been created with Clang and other development tools pre-activated that are necessary for compilation of the RHESSys source code. Upon user requests, a new versioned WRFHydro (2022-03) kernel has been created to include the hvPlot toolset for advanced data visualization and updated versions of all the libraries from the previous WRFHydro (2021-09) kernel.

Please refer to the following resources for details and examples:
Run WRFHydro 5.x model on HPC with CyberGIS-Compute V2
https://www.hydroshare.org/resource/cc28d769943046fdac0f9b0c0fc2afc6/

Implementation of WRFHydro 5.x model using CyberGIS-Compute V2
https://www.hydroshare.org/resource/329ede31b88942c489aca3111b076446/

Customize Software Environment on CJW
https://www.hydroshare.org/resource/461a8a853d8e42a8ae170c68c4cfa8f1/

“cjw” Command Line Interface Documentation
https://cybergis.github.io/cybergisx-cli/cjw/

See Release Notes on HydroShare
https://www.hydroshare.org/resource/b0d094eef336427ab605066e166135d3/

Please let us know if you have any questions or run into any problems (help@cybergis.org). Any feedback would be greatly appreciated.

Best regards,
CyberGIS-Hydro team

ABSTRACT:

CyberGIS-Jupyter for Water Quarterly Release Announcement (2022-Q2)

Dear CJW users,

We are pleased to announce a new release of the CyberGIS-Jupyter for Water (CJW) platform at https://go.illinois.edu/cybergis-jupyter-water. This release includes several new capabilities and features summarized as follows.

1) CJW moved to a new home. Jetstream-1, an NSF-funded high-performance cloud computing resource where CJW was hosted for the past 3 years, was permanently shut down on July 31, 2022. Its successor, Jetstream-2, which offers much more powerful capabilities, has become the new home of CJW. All existing CJW user data and notebooks have been migrated to Jetstream-2. We do not expect users to experience any change in usage due to this transition but to enjoy a faster and smoother Jupyter environment backed by the latest hardware and cloud technology. In exceptional cases, the previous CJW instance on Jetstream-1 could be accessible upon user request.

2) Improved user experience in CyberGIS-Compute job submission: Have you ever had a long-running job submitted to high-performance computing (HPC) resources but found your Jupyter session died after the browser was idle for too long? The latest CyberGIS-Compute SDK now allows you to reinstate job submission sessions for all previous jobs you submitted. Just switch to the new “Your Jobs” tab page in the user environment and “Restore” the jobs you are interested in. This also gives you a chance to re-download model outputs from previous jobs.

3) WRFHydro model integration supports merging model outputs: A new option “Merge_Output” is added to the WRFHydro workflow supported by CyberGIS-Compute. If enabled, single-timestep NetCDF files can be merged on the “Time” dimension after model execution. Currently supported output types include CHANOBS, LDASOUT, GWOUT, LAKEOUT, RTOUT, and LSMOUT. This optional merging step can reduce data transfer size and speed up post-processing work on CJW. The merged files are put into a separate folder called “Outputs_Merged” alongside the original model outputs. Users can choose to download either or both. Please refer to the example notebook for more information.

4) Enhanced support for user customization to CJW kernels: While CJW has pre-installed a large collection of common libraries and tools to support a suite of hydrologic analysis and modeling workflows, users may still want to install something specific to certain use cases. CJW now allows users to directly use “!pip install XXX” in notebook cells to customize existing kernels. CJW supports flexible additions or changes on a per-kernel basis, which does not affect other existing kernels. Please refer to this example notebook for more information.

5) Updates on CJW backend (kernel, plugin, and bugfix): A new general-purpose kernel, Python3-2022-06, is added, which incorporates a rich set of new geospatial packages. The ‘StickyLand” JupyterLab plugin is installed that allows users to create customizable dashboards and linear notebooks; A bug specific to Apple Safari browser in the OpenWith operation has been fixed.

Please refer to the following resources for details and examples:

Run WRFHydro model on HPC resources using CyberGIS-Compute V2 (updated 2022-07)
https://www.hydroshare.org/resource/cc28d769943046fdac0f9b0c0fc2afc6/

Customization to CJW Kernels with Pip
https://www.hydroshare.org/resource/d18886d2aedf4a2e8c6302165b8fe10f/

CyberGIS-Compute SDK new features
https://cybergis.github.io/cybergis-compute-python-sdk/release-notes.html

CJW 2022-Q2 Release Notes on HydroShare
https://www.hydroshare.org/resource/34b04302d8b34cc6aab826f79b5e3802/

Please let us know if you have any questions or run into any problems (help@cybergis.org). Any feedback would be greatly appreciated.

Best regards,
CyberGIS-Hydro team

ABSTRACT:

CyberGIS-Compute is a scalable middleware framework for enabling high-performance and data-intensive geospatial research and education on CyberGISX. This API can be used to send supported jobs to various supported HPC & computing resources.

ABSTRACT:

CyberGIS-Jupyter for Water Quarterly Release Announcement (2022-Q3)

Dear CJW users,

We are pleased to announce a new release of the CyberGIS-Jupyter for Water (CJW) platform at https://go.illinois.edu/cybergis-jupyter-water. This release includes several new capabilities and features summarized as follows.

(1) Cern Virtual Machine File System (CVMFS): We have redesigned how we deliver software within CyberGIS-Jupyter. This new design drastically increases computational performance and reproducibility, and allows the platform to make the software environment available in a variety of settings. From an end-user perspective, there should be no change to your accessing and utilizing the CJW services.

(2) Improved user experience for CyberGIS-Compute: In previous versions, we introduced the capability for users to “Restore” their previously submitted jobs of interest. Based on user feedback, we’ve further refined the interface to support viewing and downloading outputs of all previously submitted jobs by simply navigating to the “Past Results” section. The result/output of any completed job can be accessed with a single click.

(3) Support for new High Performance Computing (HPC) backend in CyberGIS-Compute: Anvil is now available as a new HPC resource for CyberGIS-Compute. Supported by NSF, Anvil is a HPC system hosted at Purdue University that contains 1000 CPU nodes based on the third generation AMD EPYC "Milan" processor, delivering a peak performance of 5.3 petaflops. Allocations on Anvil are managed by NSF's ACCESS program (https://access-ci.org/). The large numbers of CPU nodes and cores (i.e., 128) enable superior computational performance for scalable codes, short queuing times, and fast execution for hydrologic models via CyberGIS-Compute. For more information on Anvil, refer to the documentation at: https://www.rcac.purdue.edu/anvil. The WRFHydro model is supported on Anvil via CyberGIS-Compute. Please refer to the example notebook below.

Please refer to the following resources for details and examples:

A Brief Overview Of Cern Virtual Machine File System (CVMFS)
http://www.hydroshare.org/resource/ab1555c0c8d34d3496997353ba8060d9

CyberGIS-Compute updates - 2022-Q3
http://www.hydroshare.org/resource/3b472641c3504161bb13a19d4c9fbc87

Submission of WRFHydro model to Anvil HPC
https://www.hydroshare.org/resource/cc28d769943046fdac0f9b0c0fc2afc6/

See Release Notes on HydroShare
http://www.hydroshare.org/resource/bf463f07e1244de4a17b3ea7b2d95916

Please let us know if you have any questions or run into any problems (help@cybergis.org). Any feedback would be greatly appreciated.
Best regards,
CyberGIS-Hydro team