RESEARCH: Surface Water Balance (SWB), Atmospheric Water Balance (AWB) and Standardized Precipitation Index (SPI) in Colombia - SAWB

Keywords: research-and-development swb awb spi era5 cru

• P: precipitation in millimeters
• E: evaporation in millimeters
• Q: vapor flux vector in millimeters

Pending tasks

• AWB post-processing - year list from Q (vapor flux) folder
• General statistics over the data source file .nc and the study zone. Pmin, Pmax, Pavg, Pstd, Emin, Emax, Eavg, Estd.
• Rain and evaporation time series over point and polygon. P-E. Graph and table
• POI GeoJson map like GISMobile
• Download ERA5 1 hour 0.25dd with full Earth coverage. 1 day aggregation
• For ERA5 global collections, drop expver values and feature. expver is used to tell the difference between the initial release (expver=5, called ERA5T) and validated ERA5 data (expver=1). https://confluence.ecmwf.int/display/CKB/ERA5%3A+data+documentation
• Compress .nc & .csv into .zip files for GitHub massive uploads (95MB part files)

sawb.py features

• Execution over different data sources as CRU or ERA5 reanalysis
• Maximum precipitation value for map plots
• Spatial zone to evaluate
• Temporal time serie segmentation
• Export the original features and the SPI features in NetCFD (.nc) and comma separated values (.csv)
• SPI calculation over multiple moving window of n months defined by the user
• Precipitation & SPI yearly maps per month (for a range of accumulation periods). Classification and palette color for SPI Classification following McKee et al. (1993)
• Exception control for years outside the available data limit, e.g. the datafile .nc contains values between 1980-2022 and the user set a range between 1970-2022.
• Research point and polygon poi.py contains the parameters required for the execution, like description, order #, date and description, data source, point location and polygon limits, year ranges, mobile average times, units conversions multipliers and the maximum values used as references for the map global palettes
• Units conversion, e.g. m to mm for rain or evaporation parameters
• Polygon processing on demand with limits defined by user
• Point processing on demand with coordinates defined by user
• Creates POI folder structures for new case studies. The file: file/poi/1/poi.py is used as template
• Purge last results before a new running
• Global definitions dictionary as spi_dictionary.py
• Validate and fix range year_min > year_max
• Integrated script sawb.py for run all the proceses. SPI & AWS post-processing are integrated
• Gif animations for time-series data map
• Scientific Markdown report for each POI and data source. (Note: SPI can run with CRU or EAR5 datasets. The current AWB script only works with EAR-5 monthly datasets). AWB results links. AWB variable description obtained from Model Builder
• General localization map (polygon and point)
• Plot atmospheric basin and atmospheric watershed areas
• Merged atmospheric basin shapefiles into a unique shapefile
• Plot merged atmospheric watershed shapefile borders
• Location map, world. Rectangle data source, rectangle study zone, point. See https://docs.xarray.dev/en/stable/user-guide/plotting.html
• SPI - Classification reference table
• SPI - Technical documentation for scientific articles. See https://edo.jrc.ec.europa.eu/documents/factsheets/factsheet_spi_ado.pdf
• AWB - Technical documentation

System configuration for big NetCDF files over Windows 11

1. Press the Windows + R keys on your keyboard to open the Run dialog.
2. Type sysdm.cpl in the text box and press Enter. The System Properties menu will open.
3. Navigate to the Advanced tab. You are now in the Advanced System Settings menu.
4. Under Performance, clic on Settings.
5. Under Performance Options, navigate to the Advanced tab.
6. Under the Virtual memory section, clic on Change.
7. Uncheck the box Automatically manage paging file size for all drives.
8. Select the Drive C: and set a virtual memory custom size with Initial size in 5120Mb and Maximum size in 10240Mb.
9. Clic on OK, Apply, Ok.
10. Restart your Windows system.

You can also set a custom size virtual memory for the drives where your data processing is running.

The specific virtual memory size depends on the size of your NetCDF files.

Execution procedure

1. From ArcGIS for Desktop 10.2.2, run the ArcToolBox SAWB.tbx/a. Monthly Balance
2. From Python run the script .src/spi.py
3. From Python run the script .src/sawb_monthly_poi.py

Python requires a Virtual Environment for the script executions and the libraries described in the file Requirements.txt. From the Terminal execute pip install -r requirements.txt

File compression

Windows CMD sample: "C:\Program Files\7-Zip\7z.exe" a -v97m "D:\R.SAWB.nc\File.zip" "D:\R.SAWB.nc\cru_ts4.03.1901.2018.pre.dat.nc"

References

• Conjoint Analysis of Surface and Atmospheric Water Balances in the Andes-Amazon System
• An Atmospheric Water Balance Perspective on Extreme Rainfall Potential for the Contiguous US
• Atmospheric water balance over oceanic regions as estimated from satellite, merged, and reanalysis data
• Global Atmospheric Water Balance and Runoff from Large River Basins
• https://stackoverflow.com/questions/41898561/pandas-transform-a-dbf-table-into-a-dataframe
• https://sparkbyexamples.com/pandas/sort-pandas-dataframe-by-date/
• https://www.dataquest.io/blog/plot-dataframe-pandas/
• https://www.geeksforgeeks.org/numpy-linspace-python/
• https://www.caee.utexas.edu/prof/maidment/gishyd97/atmos/atmos.htm
• https://edo.jrc.ec.europa.eu/documents/factsheets/factsheet_spi_ado.pdf
• Standardized Precipitation Index (SPI) | Drought & Flood Monitor
• https://climatedataguide.ucar.edu/climate-data/standardized-precipitation-index-spi
• https://crudata.uea.ac.uk/cru/data/hrg/cru_ts_4.03/cruts.1905011326.v4.03/pre/
• https://stackoverflow.com/questions/57507832/unable-to-allocate-array-with-shape-and-data-type
• https://ado.eurac.edu/spi-6
• https://matplotlib.org/stable/tutorials/colors/colormaps.html
• Read and plot netCDF file in python | easy method to handle netcdf files - Include vector limits
• Xarray Basics | Fundamentals of Xarray That Could Be Helpful for Data Science and Analytics
• Timeseries Analysis using Python Xarray
• https://docs.xarray.dev/en/stable/user-guide/plotting.html
• https://es.wikipedia.org/wiki/Crisis_energ%C3%A9tica_de_1992_en_Colombia
• https://realpython.com/python-zipfile/
• https://github.com/jeffjay88/Climate_Indices
• https://www.blog.pythonlibrary.org/2021/06/23/creating-an-animated-gif-with-python/
• https://stackoverflow.com/questions/11067097/7zip-commands-from-python
• https://www.datacamp.com/tutorial/python-subprocess
• https://www.kdnuggets.com/2020/09/geographical-plots-python.html
• https://jakevdp.github.io/PythonDataScienceHandbook/04.13-geographic-data-with-basemap.html
• https://notebook.community/milancurcic/lunch-bytes/Fall_2019/LB33/Basemap_v_Cartopy
• https://spatial-dev.guru/2022/06/05/merging-multiple-shapefiles-into-one-shapefile-using-python-and-geopandas/
• https://gis.stackexchange.com/questions/433708/writing-empty-geodataframe-to-shapefile-using-python
• https://towardsdatascience.com/plotting-maps-with-geopandas-428c97295a73
• https://matplotlib.org/stable/tutorials/colors/colormaps.html
• https://matplotlib.org/2.0.2/examples/color/named_colors.html
• https://matplotlib.org/basemap/users/examples.html
• https://stackoverflow.com/questions/12251189/how-to-draw-rectangles-on-a-basemap
• https://geopandas.org/en/stable/docs/user_guide/mapping.html
• https://matplotlib.org/basemap/api/basemap_api.html
• https://basemaptutorial.readthedocs.io/en/latest/shapefile.html
• https://www.physicsread.com/latex-vector