HYDROTERRE: TOWARDS AN EXPERT SYSTEM  FOR SCALING HYDROLOGICAL DATA AND MODELS FROM  HILL-SLOPES TO MAJOR-RIVER BASINS

Abstract

                      

 

The purpose of this research is to develop HydroTerre, an expert system, as a resource for the larger water research community to improve the hydrological modeling process. The expert system provides modelers with access to hydrological data and model results that scale from hill-slopes to major-river basins anywhere in the continental US. HydroTerre consists of three processes:

• Dataworkflows that automate the functionality of collecting and processing essential variables for hydrological models anywhere in the US.
• Modelworkflows that consume the data processes and automatically scale and transform the data into model inputs and generate efficient Penn State Integrated Hydrological Models (PIHM).
• Visualanalytic workflows that disseminate the PIHM model process results appropriately per scale and make it feasible for modelers to analyze both the data and model results, finding new features and details otherwise not possible.

 

The expert system, collectively, captures the critical thinking made by the hydrological modelers via the user interface, using provenance datasets that are shared amongst the modeling community. The expert system has been evaluated at the level HUC-12 catchment scale everywhere in the Continental United States of America (CONUS) with all three workflows. Chapter 2 demonstrates data workflows that provide data bundles of elevation, soil, geology, land cover, and one climate normal (30 years) of forcing data within minutes anywhere in the CONUS using distributed compute resources and High Performance Computing (HPC). The data bundles use federal national data products that would normally take a modeler days to weeks to retrieve using existing national cyber-infrastructure from these agencies.

 

 

Data to model workflows transform these data bundles into PIHM inputs and with HPC resources distributes PIHM model workflows dynamically as shown in Chapter 3. These transformations were evaluated millions of times to create database repositories (provenance) for modelers to conduct, share, and reproduce model studies.

Additionally, these model results (poor and good) help identify opportunities to improve data and model processes at the level HUC-12 before scaling towards major river basins.

 

The analysis of model workflows using HPC and web based visual analytic applications is shown in Chapter 4 to explain provenance, reproducibility, and scalability for all three processes at the HUC-12 scale. Using the HydroTerre expert system, it is feasible at the HUC-12 scale to select a catchment via a web application, define model parameters, and retrieve data, model, and visualization results within minutes using distributed computing and HPC environments. The expert user can achieve these modeling steps entirely online without handling data or model arrangements. Hence, HydroTerre increases reproducibility, provenance, and a modeler’s ability to create hydrologically correct models.

 

The focus of Chapter 5is to identify issues learned from hill-slope modeling using HydroTerre workflows that future research will be required to address, in order to generate hydrologically correct models anywhere in the CONUS at any catchment scale. From the millions of workflows evaluated, missing data, in particular soils, requires new strategies to either replace or find suitable values. The remaining data issue is stream delineation techniques. Both the use of national data products and TauDEM strategies require new visual analytical applications to guide the expert user to correct stream data. To scale both model and visual analytic workflows requires new data structures and domain decomposition strategies that scale both catchment, cyberinfrastructure, and capability of HPC environments.

 

Table of Contents

 

List of Figures ………………………………………………………………………………………………. xii

List of Tables ……………………………………………………………………………………………… xviii

Glossary …………………………………………………………………………………………………….. xxii

Acknowledgements …………………………………………………………………………………….. xxiv

Chapter 1 ………………………………………………………………………………………………………… 1 Introduction …………………………………………………………………………………………………… 1

1.1 Workflows and Use-Cases …………………………………………………………………………. 1

1.2 Why do we Need Hydrological Models?……………………………………………………….. 2

1.3 Types of Hydrological Models …………………………………………………………………….. 3

1.4 Resolving Data and Model Workflows …………………………………………………………. 4

1.5 Visual Analytic Workflows ………………………………………………………………………….. 5

1.6 Provenance and Reproducibility …………………………………………………………………. 6

1.7 What is an Expert System? ………………………………………………………………………… 6

1.7.1 Overview of Data Processes (Data Scale Workflows) …………………………….. 10

1.7.2 Overview of Model Processes (Data and Model Workflows) ……………………. 11

1.7.3 Overview of Visual Analytics Processes (Data, Model, and Visualization

Workflows) ……………………………………………………………………………………………….. 12

Chapter 2 ………………………………………………………………………………………………………. 14

Essential Terrestrial Variable Data Workflows for Distributed Water Resources

Modeling …………………………………………………………………………………………………….. 14

2.1 Abstract …………………………………………………………………………………………………. 14

2.2 Introduction to the Data Scale Problem………………………………………………………. 15

2.2.1 Spatially Distributed Watershed Models ……………………………………………….. 16

2.2.2 Related Work ……………………………………………………………………………………. 17

2.3 Essential Terrestrial Variables (ETV) National Data Products, CONUS ………….. 18

2.3.1 Access to ETV Data …………………………………………………………………………… 19

2.3.2 Hardware and Software for ETV Data Access via Web Applications …………. 22

2.4 The ETV Data Workflow…………………………………………………………………………… 25

2.4.1 User Interface to Access ETV data ………………………………………………………. 25

2.4.2 Deriving ETV Data and Steps Taken for Faster Accessibility …………………… 26

2.4.3 Consuming HydroTerre ETV Data Within a Model Workflow ……………………. 30

2.5 Scaling up the ETV Data Workflow to Model Large River Basins …………………… 32

2.6 Conclusion to Data Scale Issues ………………………………………………………………. 34

2.7 Future Directions …………………………………………………………………………………….. 35

Chapter 3 ………………………………………………………………………………………………………. 37

Automating Data-Model Workflows at a Level 12 HUC Scale: Watershed Modeling in

a Distributed Computing Environment …………………………………………………………….. 37

3.1 Abstract …………………………………………………………………………………………………. 37

3.2 Introduction to Data-Model Workflows ……………………………………………………….. 38

3.2.1 Why Data-Model Workflows? ………………………………………………………………. 393.2.2 What is Automated Within the Data-Model Workflows? …………………………… 40

3.2.3 Constraints ……………………………………………………………………………………….. 41

3.2.4 Related work …………………………………………………………………………………….. 42

3.3 System Design ……………………………………………………………………………………….. 42

3.3.1 Hardware and Administration Layers ……………………………………………………. 43

3.3.2 Overview Map of Service-Oriented Architecture …………………………………….. 46

3.4 Workflow Services ………………………………………………………………………………….. 50

3.4.1 ETV Workflow Services ……………………………………………………………………… 51

3.4.2 Data-Model Workflow ………………………………………………………………………… 53

3.4.3 PIHM Model Workflow ……………………………………………………………………….. 55

3.5 Prototype to Create Data-Model Workflows ………………………………………………… 58

3.5.1 User Credentials ……………………………………………………………………………….. 583.5.2 Selecting HUC-12s ……………………………………………………………………………. 59

3.5.3 Setup Data-Model and Model Workflows ………………………………………………. 60 3.6 Creating Provenance Data-Model Workflows using Distributed Compute

Environments ………………………………………………………………………………………………. 64

3.6.1 Distributed Compute Environments ……………………………………………………… 64

3.6.2 Evaluation of CONUS Data-Model Workflow Provenance ……………………….. 67

3.7 Conclusion …………………………………………………………………………………………….. 71

3.8 Future Direction ……………………………………………………………………………………… 72

Chapter 4 ………………………………………………………………………………………………………. 73

Visualization Workflows for Level 12 HUC Scales: Towards an Expert System for

Watershed Analysis in a Distributed Computing Environment …………………………….. 73

4.1 Abstract …………………………………………………………………………………………………. 73

4.2 Introduction ……………………………………………………………………………………………. 74

4.2.1 What is Automated Within the End-to-End Workflows? …………………………… 75

4.2.2 Why Visualization Workflows? …………………………………………………………….. 76

4.2.3 Constraints ……………………………………………………………………………………….. 77

4.2.4 Related Work ……………………………………………………………………………………. 78

4.3 System Design ……………………………………………………………………………………….. 79

4.3.1 Hardware and Administration Layers ……………………………………………………. 79

4.3.2 Overview Map of Service-Oriented Architecture …………………………………….. 81

4.3.3 Data, Data-Model, and Model Workflows ……………………………………………… 85

4.4 Workflow Services ………………………………………………………………………………….. 86

4.4.1 Visualization Workflow Overview …………………………………………………………. 87

4.4.2 Executing and Retrieving Visualization Workflow Results ……………………….. 88

4.5 Prototype to Create Data-Model-Visualization Workflows ……………………………… 90

4.5.1 Setup Data-Model and Model Workflows ………………………………………………. 91

4.5.2 Setup Visualization Workflows …………………………………………………………….. 92

4.6 Demonstration of Using Expert System to Analyze and Share Hydrological Models

at HUC-12 Scales ………………………………………………………………………………………… 94

4.6.1 Initiating Workflows & Provenance ………………………………………………………. 94

4.6.2 Spatial Analysis ………………………………………………………………………………… 96

4.6.3 Time Series Data Analysis ………………………………………………………………… 100

4.7 Conclusion …………………………………………………………………………………………… 102

4.8 Future Direction ……………………………………………………………………………………. 103

Chapter 5 …………………………………………………………………………………………………….. 104

Demonstrating Workflows to Scale From Hill-Slopes to Major-River Basins ………… 104

5.1 HUC-12s to Major-River Basins Using National Datasets ……………………………. 104

5.1.1 Introduction …………………………………………………………………………………….. 104

5.1.2 Data Processes ………………………………………………………………………………. 106

5.1.3 Model Processes …………………………………………………………………………….. 108

5.1.4 Visual Analytic Processes …………………………………………………………………. 115

5.1.5 Discussion ……………………………………………………………………………………… 117

5.1.6 Next Steps ……………………………………………………………………………………… 119

5.2 Santa Rosa Island California …………………………………………………………………… 124

5.2.1 Abstract………………………………………………………………………………………….. 124

5.2.2 Introduction …………………………………………………………………………………….. 124

5.2.3 Climate History ……………………………………………………………………………….. 125

5.2.4 Present Physiography Conditions, ETV Workflows ………………………………. 127

5.2.5 Reconstructing Climate History ………………………………………………………….. 129

5.2.6 ETV, Data-Model and Model Workflows ……………………………………………… 131

5.2.7 Results …………………………………………………………………………………………… 139

5.2.8 Discussion ……………………………………………………………………………………… 143

5.2.9 Conclusion ……………………………………………………………………………………… 146

5.3 Towards Major River Basins …………………………………………………………………… 147

5.3.1 Abstract………………………………………………………………………………………….. 147

5.3.1 Introduction …………………………………………………………………………………….. 147

5.3.2 Data Processes ………………………………………………………………………………. 150

5.3.3 Model Processes …………………………………………………………………………….. 164

5.3.4 Visualization Processes ……………………………………………………………………. 168

5.3.5 Conclusion ……………………………………………………………………………………… 171

Chapter 6 …………………………………………………………………………………………………….. 172

Contributions and Conclusions …………………………………………………………………….. 172

6.1 Scope, Objectives, Conclusions, and Future Work for Data Workflows …………. 173

6.2 Scope, Objectives, Conclusions, and Future Work for Data-Model Workflows .. 175

6.3 Scope, Objectives, Conclusions, and Future Work for Visualization Workflows 176

References ………………………………………………………………………………………………….. 178Appendix A …………………………………………………………………………………………………. 190

Chapter 1

 

Introduction

 

Have you ever been in the situation where you tried to reproduce somebody’s model results from an article or official report (i.e. Environmental Protection Agency (EPA)) and you cannot because you are missing either the code, data, “tweaks”, and access to the modeler? How about when you have these components, but you cannot figure out why the modeler made certain decisions? As a decision maker, stakeholder, modeler, or student, it is difficult to have confidence in this model software and cyber-infrastructure. With many choices in models, would you be willing to invest time and money into such a model? As an experienced user, or “domain expert”, if reproducibility and accessibility are not certain, how can experts be expected to improve model results and domain science? In this research, I demonstrate that workflows contribute to automating complex and time-consuming tasks to improve Hydrological science by creating cyberinfrastructure, called HydroTerre, which empowers the Hydrologist expert to create hydrological models with PIHM, anywhere in the CONUS.

 

1.1 Workflows and Use-Cases

Extreme weather events such as droughts and flooding cause major financial losses for all countries around the world. These events are difficult to predict and simulate because distributed hydrological models, such as the Penn State Integrated

Hydrological Model (PIHM), require large amounts of data, involve uncertainty in both data and model, and are computationally rigorous (Qu and Duffy 2007a).  This dissertation begins to address these issues at the United States (US) continental scale (CONUS) through research that creates an expert system enabling data, model, and visual analytic workflows within High Performance Computing (HPC) environments. Such data scaling will facilitate new hydrological modeling strategies and help to visualize PIHM results, which will enable modelers to improve the art of understanding flooding and drought events for both the scientific and the general community. These workflows will be evaluated at three scales and purposes (use cases). The first is at the HUC-12 scale with all 90,762 catchments within the CONUS, averaging approximately

40 square kilometers in area, to evaluate the workflow infrastructure. Next, the Santa Rosa Island catchment, California, USA, with two HUC-12s. To determine and modify the HydroTerre workflows built for only one HUC-12, before scaling to major river basins. Finally, the Juniata catchment, Pennsylvania, USA, approximately 10,000 square kilometers, or 148 level HUC-12s and the Susquehanna catchment,

Pennsylvania, USA, approximately 100,000 square kilometers in area (1045 HUC-12s) to evaluate the feasibility of HydroTerre infrastructure to support these scales. As well as to evaluate High Performance Computing (HPC) workflows aimed towards EPA water resource managers for flooding and droughts.

 

1.2 Why do we Need Hydrological Models?

The Federal Emergency Management Agency (FEMA) has approved more than 8 billion

dollars to citizens in New York affected by Hurricane Sandy flooding (Federal Emergency Management Agency 2013a). This contrasts to only 1.3 billion dollars collected by FEMA for flood coverage insurance (Federal Emergency Management Agency 2013b). Thus, every tax-paying citizen in the US contributes to these financial bailouts by the Federal Government. Clearly, with financial losses at this scale, there is interest in finding strategies to protect citizens from, and minimize the costs of, extreme weather events. Consequently, many hydrological models are used to predict the potential impacts from extreme climate events.

 

 

1.3 Types of Hydrological Models

Hydrological models are of two spatial representation categories; lumped models and distributed finite element/volume/difference models. Lumped models predict at scales from small (100s of acres) to global size regions at the expense of simplifying representation of the hydrological processes to effectively manage the computation complexity (Shah et al. 1996). Generally, lumped models use parameters that are spatially-averaged characteristics in a hydrological system, and these averaged parameters are difficult to compare with actual field measurements (Johnson and Miller 1997). Often, lumped models consider the catchment as one unit with a small number of parameters and variables, rather than multiple catchments; each with many variables (Refsgaard and Knudsen 1996). Watersheds modeled in the logic of a network of lumped models are referred to as semi-distributed models. Another type of model is climate grid hydrological models that are averaged over climate grid raster cells, but are vertically distributed for regional assessments (Bell et al. 2007).

 

Microscale distributed finite element/volume models are often restricted to small and medium (county size) watersheds. These models capture the physical processes (land parcels, topography, soils, etc.) in more detail  by representing the spatial and temporal characteristics that govern the transformation of precipitation into surface runoff and groundwater movements (Vieux et al. 2004). These models attempt to quantify the detailed spatial variability of hydrological parameters at desired locations within a catchment (Smith 1993) by taking into account spatial variability of inputs and outputs of hydrologic variables and of the hydrologic responses at ungauged sites within a catchment (Smith et al. 2004b), (Smith et al. 2004a), (Carpenter and Georgakakos 2006). However, distributed models can be computationally demanding depending on the discretization of the model and the number of physical equations required at a particular spatial resolution (Cloke and Hannah 2011).

 

In this research a state-of-the-art in hydrologic simulation model is recast as an expert system that will: 1) automate the process of setting up models from national Essential Terrestrial Variable (ETV) data (Leonard & Duffy, 2013); 2)  Design a HPC strategy to dynamically adapt unstructured meshes (hill-slopes to major-river basins) to capture physical details that a lumped model cannot. The goal is to demonstrate that scalable, distributed finite volume models can be implemented in a “model as a service framework” for automating the process of flood and drought simulations from national geospatial data and climate reanalysis products (Arguez and Vose 2011).

 

1.4 Resolving Data and Model Workflows

Accessing national datasets for many Hydrological models is a time consuming process. By using data workflows, the author demonstrates that access to ETV data can be accessed rapidly via HPC services to 100s of terabytes of data as model inputs for hydrological models anywhere in CONUS. Streaming, or coupling, climate models is a long-term goal for forecasting, but the first step is to demonstrate that a finite volume model like PIHM (development led by Dr. Duffy, Civil and Environmental Engineering, Penn State University) can simulate, for example, evolution of a major storm on the catchment. The author has developed HydroTerre (Leonard and Duffy 2013)

(www.hydroterre.psu.edu) to provide rapid access to 250 terabytes of Essential Terrestrial Variables (ETV) for any level-12 Hydrological Unit Code (HUC) catchment in the CONUS (Chapter 2). Multiple times, each of the 90,762 level-12 HUCs (Chapter 3) have been evaluated with PIHM for a 30-day duration of time-simulation on Cyberstar (http://www.ics.psu.edu/infrast), a HPC resource at Penn State University (PSU), to establish the feasibility of streaming data inputs to remote HPC. In addition, streaming these data to compute PIHM at each level-12 HUC demonstrates the validity of the input data as both a service and as a research prototype to transform PIHM software from desktop to HPC environment. However, using these local facilities is restrictive to only being able to use a short period of data; as the resources at PSU are inadequate to continue this type of HPC research. To model each level-12 HUC once requires 2.6 petabytes of storage to evaluate 30 years of North American Land Data Assimilation System (NLDAS) climate data. This is a relatively easy problem to solve as each of the 90,762 level-12 HUCs can individually, within a large HPC environment, be simulated if the flux exchanges between HUCs are calculated in an ordered sequence. However, as demonstrated in Appendix A, national datasets do not support this scaling natively. Meanwhile, to scale up from level-12 HUCs to modeling the entire Mississippi catchment, as one simulation, requires only 2600 GBs of data inputs, but requires 10s of petabytes of storage for the model results. Appendix B demonstrates how the HydroTerre distributed computing environment has been designed and tuned to address these catchment-scaling issues.

 

1.5 Visual Analytic Workflows

 

Visual analytics strives to assist the analytical reasoning process with software that maximizes user capacity to understand, distinguish, and reason about complex and dynamic data and states (Thomas and Cook 2005), (Thomas and Cook 2006), (Andrienko et al. 2011), (MacEachren and Miksch 2012). The author demonstrates in Chapter 4, visual analytic workflows (with process automation) and web services to share both model results, and for expert Hydrologists to drill-down and interrogate both data and models. HydroTerre enable modelers, via the world-wide-web, to have access to both model inputs and outputs for rapid model prototyping at level-12 HUC scales anywhere in the CONUS. HydroTerre’s interface has been designed to help expert users identify and highlight where data inputs need improving, where models fail, and include metrics to inform users where uncertainty occurs. In addition, demonstrating hydrological model results that contain 100s of millions of mesh elements representing the terrain that a user can drill down to investigate hydrological values is a state of the art advancement in hydrological modeling (Chung et al. 2012). Why is this important? A fundamental advantage of finite element/volume models is the ability to retrieve data at each mesh element to allow further investigation of the hydrological results for that element. For example, a farmer can look at water budgets with the mesh that represents his or her property (small scale), but also investigate budgets at county or state scales (large scale). Additionally, we can use this visualization pipeline to study where the HPC model accurately depicts extreme weather events and use this knowledge to improve the modeling processes by using data mining strategies to identify potential problems and to capture modeler techniques (van Wijk 2005), (Fayyad et al. 2002).

 

1.6 Provenance and Reproducibility

 

The term end-to-end workflow represents the combination of data, data-model, model, and visualization workflows within HydroTerre that enables web users to create hydrological models anywhere in the CONUS at the level-12 HUC scale. There are two significant advantages of using end-to-end workflows; provenance and reproducibility. Provenance provides a record of the history of data, derived data, user inputs, software, and hardware (Deelman et al. 2009) (Gil et al. 2007) (Taylor et al. 2007). A complete provenance record allows other modelers to reproduce the same scientific result using the HydroTerre end-to-end workflows.

 

The use of scientific workflows is becoming prevalent in many science domains such as biology, astronomy, and medicine to accelerate scientific progress. VisTrails is an open source, commonly used scientific workflow and provenance management system capable of handling large volumes of data and designed for HPC environments (University of Utah 2013) (Silva et al. 2007) (Callahan et al. 2006). The main reason for not using VisTrails in HydroTerre is the lack of resources and functionality for spatial datasets and interactive online mapping. Instead, HydroTerre uses ESRI’s ArcGIS functionality and custom tools for workflows, provenance, and reproducibility.

 

 

1.7 What is an Expert System?

Expert and knowledge-based systems have been used by hydrologists since the 1980s

for flow measurement in open channels (Simonovic 1991). An expert system is computer software that emulates decision making of a human expert (Jackson 1999). The system operates as an interactive organization that responds to questions, makes recommendations, asks for clarification, and aids users in the decision-making process. An expert system that can interpret, identify, predict, diagnose, design, monitor, train and debug will reduce the difficulty of hydrological modeling by focusing the modeler where he or she needs to improve the data inputs and model results (Benfer et al. 1991). These same features will also enable software developers to focus where code improvements within the hydrological model can occur, thus improving the hydrological science.

 

This dissertation focuses on the automation to scale and integrate large amounts of data as both model inputs and outputs for Hydrological modeling. Without the ability to emulate decision making of the expert Hydrologist, large-scale hydrological modeling is difficult. Here, the expert system model consists of the three processes (data, model, and visual analytics) as presented in Figure 1.1, and the research serves two targeted audiences; the public and hydrological modelers. The emphasis of this dissertation will be toward hydrological modelers that address all three processes. The public is under the “big picture” in Figure 1.1 and will be briefly addressed in the visual analytics process. The public will have access to data and results; however, they will not have direct involvement in the development of the expert system itself.

 

The fundamental purpose of the expert system is to manage data and model scalability issues that will arise when spatially scaling from hill-slope to major-river basins. To manage these issues requires many different, or altered, workflows within each process to account for spatial scaling (see sections 1.8.1 to 1.8.3). The purposes of these workflows are to minimize the interaction of modelers to the models and to automatically scale the model depending on either the catchment size (hill-slope to major-river basin) or the degree of detail (coarse (kilometer) to fine (meter) resolution). At first, the expert system will require more input from the hydrological modelers, but with a system that captures modeler’s methodology and decisions via the user interface, these inputs can be “data mined” to define “standard” inputs. In essence, the user interface, coupled with the workflows, is capturing the critical thinking engaged in by the modelers across each of the processes. Attached with versioning, each process could be improved and eventually reduce the essential inputs by modelers in the three processes. This is not to say that inputs will be removed in the processes, rather the artificial intelligence of the expert system improves so the expert, the hydrologist, can focus on the essential inputs; and, if they desire, they can change the default values selected by the expert system and retrace steps by using provenance.

 

To demonstrate the expert system, four scales will be used to assist Hydrologists to create PIHM models. These will be at the following locations: (1) Every CONUS HUC12 (~40 sq. km.), (2) Santa Rose Island, California, with two HUC-12s (~215 sq. km ) (3) Juniata River, Pennsylvania, USA (~ 10,000 sq. km.), and (4) Susquehanna Basin, Pennsylvania, USA (~100,000 sq. km.).

 

 

 

Figure 1.1. Towards an expert system for scaling hydrological models from hill-slopes to major-river basins

9

 

 

1.7.1 Overview of Data Processes (Data Scale Workflows)

 

The purpose of data processes (Figure 1.2) is to encapsulate the essential data sets and handle the preparation steps commonly used by hydrological models. Rather than going to multiple sources to collect data inputs for a model, this process would take care of preparing data, and a user could simply get all the data by selecting a watershed via a web browser. This concept is effective for catchments that contain less than 10 gigabytes of data, or approximately 10,000 square kilometers in size. However, as spatial and data scales increase from small watershed to continental sizes, accessibility of  users to data via a one-stop web browser will be reduced, as the complexity of retrieving data increases (Appendix B). The main reasons are due to network restrictions, as only so much data can be streamed instantly without error and due to hard drive capabilities. Thus, multiple data workflows are necessary to accommodate the three case studies identified. Further details about these issues and other competing issues are described in Chapter 2 and Appendix B.

 

Figure 1.2. Data processes overview.

 

1.7.2 Overview of Model Processes (Data and Model Workflows)

The purpose of model processes (Figure 1.3) has two parts. The first part is to consume the results from data workflows and transform the data into model formats. In this dissertation (Chapter 3), this process has been demonstrated using PIHM. The transformation process has similar HPC issues as outlined in the overview of data processes section, but are compounded further by PIHM file format specifications. As both the spatial and data watershed scales increase, so too does the complexity of modifying PIHM file inputs, algorithms, and assumptions when consuming 1000s of gigabytes of data.

 

The second part involves generating hydrological models using the expert system. The amount of model results and petabytes of data compounds the difficulty of sharing and consuming data. It is this intersection of when to “simplify” hydrological processes versus “simplifying” data processes that requires testing and refining. The general trend will be simple to complex changes and rapid to slow performance (Figure 1.3). Slow may mean not feasible without significant changes to the hydrological processes included in the simulation model. Chapter 3 discusses these details further.

 

Figure 1.3. Model processes overview.

1.7.3 Overview of Visual Analytics Processes (Data, Model, and Visualization Workflows)

 

The purpose of visual analytics processes (Figure 1.4) has three parts. The first, and simplest, part is to provide visual stimulation to the public and modelers via movies and diagrams, thereby helping to explain model results. The second part is the interface to the expert system that focuses on collecting data about the hydrological modeler’s critical thinking. These data, coupled with provenance and reproducibility, will increase confidence in the results and provide insight from other modelers’ experience. Multiple visual analytic workflows will be necessary to address data scaling issues at the three watershed case studies identified. The last part is to provide tools to identify and predict flooding and droughts by gaining insight to both the data and model process results. As both the spatial and data scales increase, the amount of data PIHM generates significantly increases thereby compounding how model results can be visually analyzed. In addition, visualization depends on interactivity needs and where the data and model results are stored. Further details about the methods used are described in Chapter 4.

Figure 1.4. Visual analytics processes overview.

 

Overview of Application of Expert System: Hydrology Scaling Examples

To demonstrate the expert system, three case studies ranging in size from 1 to 100,000 square kilometers will assess the workflows to generate PIHM using reanalysis climate products for 30 years. In Chapter 5, the results pertaining to each of the three processes will be described in detail explaining how workflows can be either created or modified to address the accessibility and scalability issues. Lessons learnt from evaluating every CONUS HUC-12 tens of times, to create a data repository with millions of workflow results will be used to explain issues that require addressing to scale to major river basins.

HYDROTERRE: TOWARDS AN EXPERT SYSTEM  FOR SCALING HYDROLOGICAL DATA AND MODELS FROM  HILL-SLOPES TO MAJOR-RIVER BASINS

Sharing is caring!