Award Abstract # 2033405
Improving Weather Forecasting through non-Gaussian Data Assimilation with Machine Learning

NSF Org: AGS
Division of Atmospheric and Geospace Sciences
Recipient: COLORADO STATE UNIVERSITY
Initial Amendment Date: August 4, 2020
Latest Amendment Date: August 4, 2020
Award Number: 2033405
Award Instrument: Standard Grant
Program Manager: Yu Gu
ygu@nsf.gov
 (703)292-8796
AGS
 Division of Atmospheric and Geospace Sciences
GEO
 Directorate for Geosciences
Start Date: August 15, 2020
End Date: July 31, 2024 (Estimated)
Total Intended Award Amount: $582,475.00
Total Awarded Amount to Date: $582,475.00
Funds Obligated to Date: FY 2020 = $582,475.00
History of Investigator:
  • Steven Fletcher (Principal Investigator)
    fletcher@cira.colostate.edu
Recipient Sponsored Research Office: Colorado State University
601 S HOWES ST
FORT COLLINS
CO  US  80521-2807
(970)491-6355
Sponsor Congressional District: 02
Primary Place of Performance: Colorado State University
200 W. Lake Street
Fort Collins
CO  US  80523-1375
Primary Place of Performance
Congressional District:
02
Unique Entity Identifier (UEI): LT9CXX8L19G1
Parent UEI:
NSF Program(s): Physical & Dynamic Meteorology
Primary Program Source: 01002021DB NSF RESEARCH & RELATED ACTIVIT
Program Reference Code(s):
Program Element Code(s): 152500
Award Agency Code: 4900
Fund Agency Code: 4900
Assistance Listing Number(s): 47.050

ABSTRACT

The research project is to advance techniques for using a mathematical discipline to optimally combine theory with observations to improve the accuracy of weather forecasts. The team will use different forms of machine learning mechanism to detect changes in the behavior of moisture fields in the atmosphere such that the new techniques are able to change parts of the weather prediction scheme to better capture these fields in different locations and at different times. To achieve the research goal, a large amount of observations and model results is required to train computers to detect moisture changes. The research will investigate how much data are needed to reliably detect changes through the machine learning techniques. As part of this research project, the research team will develop a website for the research community to view atmospheric moisture changes in the past 24 hours. This research will also test how well a new component of the weather prediction scheme works when the machine learning techniques have detected moisture changes from its normal behavior. The project will also involve training a new scientist to learn the latest research method.

The research team will investigate the ability of machine learning techniques to detect changes away from Gaussian behavior for the moisture fields and to be capable to switch the cost function in variational data assimilation between Gaussian and non-Gaussian. The scheme is important to ensure that the model-observation errors are being model consistently. The error changes are commonly assumed to be toward lognormal; recent work has indicated that the behavior of moisture fields has another probability density function?the reverse lognormal. This distribution has a right skewness and enables analysis to increase the moisture state if the background is too dry. Using the proper type of error distribution schemes will aid not only cloud prediction but also cloud retention in forecast models after the data assimilation scheme has finished. This team will also investigate a new ensemble smoother, as well as non-Gaussian versions of the Maximum Likelihood Ensemble Filter as a more consistent ensemble filter for hybrid data assimilation schemes, especially for the lognormal and reverse lognormal behavior. In addition, the skewness of the moisture field at different locations and heights will be displayed at the team?s website for the general public and forecasters to view how the moisture distribution is changing.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

PUBLICATIONS PRODUCED AS A RESULT OF THIS RESEARCH

Note:  When clicking on a Digital Object Identifier (DOI) number, you will be taken to an external site maintained by the publisher. Some full text articles may not yet be available without a charge during the embargo (administrative interval).

Some links on this page may take you to non-federal websites. Their policies may differ from this site.

Fletcher, Steven J. and Zupanski, Milija and Goodliff, Michael R. and Kliewer, Anton J. and Jones, Andrew S. and Forsythe, John M. and Wu, Ting-Chi and Hossen, Md. Jakir and Van Loon, Senne "Lognormal and Mixed GaussianLognormal Kalman Filters" Monthly Weather Review , v.151 , 2023 https://doi.org/10.1175/MWR-D-22-0072.1 Citation Details
Goodliff, Michael R. and Fletcher, Steven J. and Kliewer, Anton J. and Jones, Andrew S. and Forsythe, John M. "NonGaussian Detection Using Machine Learning With Data Assimilation Applications" Earth and Space Science , v.9 , 2022 https://doi.org/10.1029/2021EA001908 Citation Details
Van Loon, Senne and Fletcher, Steven J. "A dynamical Gaussian, lognormal, and reverse lognormal Kalman filter" Quarterly Journal of the Royal Meteorological Society , v.150 , 2023 https://doi.org/10.1002/qj.4595 Citation Details
Van Loon, Senne and Fletcher, Steven J. "Foundations for Universal NonGaussian Data Assimilation" Geophysical Research Letters , v.50 , 2023 https://doi.org/10.1029/2023GL105148 Citation Details
Van_Loon, Senne and Fletcher, Steven J and Zupanski, Milija "Dynamical Gaussian, lognormal, and reverse lognormal maximumlikelihood ensemble filter" Quarterly Journal of the Royal Meteorological Society , v.150 , 2024 https://doi.org/10.1002/qj.4706 Citation Details

PROJECT OUTCOMES REPORT

Disclaimer

This Project Outcomes Report for the General Public is displayed verbatim as submitted by the Principal Investigator (PI) for this award. Any opinions, findings, and conclusions or recommendations expressed in this Report are those of the PI and do not necessarily reflect the views of the National Science Foundation; NSF has not approved or endorsed its content.

This award investigated the ability to extend the theory that underlies numerical weather prediction/forecasts to be more consistent with the different variables involved.  There are several variables in different geophysical models that do not follow the standard assumption that they are what is referred to as Gaussian distributed.  An important variable in weather prediction is humidity.  It has been shown on numerous occasions that humidity can be skewed to the left (dry) or the right (very wet) but fitting a Gaussian distribution either over or underestimate the most likely state.

This award developed new versions of variational, Kalman Filter, and ensemble based data assimilation systems to allow for lognormal (dry), as well as reverse-lognormal (very wet).  However, the distribution can change in time and so machine learning techniques were adopted to determine when the distribution had changed as well as to then select the more consistent data assimilation method to minimize the associated errors at that time.

Given the positive results from this work it lays the foundations for the operational numerical weather and ocean prediction centers to start to research the operational feasibility to implement this work to test its impacts of improving, but also creating more reliable, weather forecasts so that the public has more faith in what is being predicted but also take head if severe weather is forecasted.

 


Last Modified: 11/05/2024
Modified by: Steven J Fletcher

Please report errors in award information by writing to: awardsearch@nsf.gov.

Print this page

Back to Top of page