Missing comid data from get_catchment_characteristics() for BASIN_CHAR variables

[mjcashman]

I was getting a number of missing/NA values for a bunch of catchments across the country when trying to pull basic catchment attributes such as drainage area, slope, and catchment elevations. This occurred whether trying to access local catchment CAT or via accumulated values ACC/TOT.

It seems like the issue is with the source parquet files on Sciencebase. All of these catchments are missing from the parquet files, yet the original source .zip/.txt files contained on sciencebase do contain values for all of these catchments. They must have been lost somewhere in translation.

This results in 32,111 COMIDs that are missing when comparing the two datasets (parquet versus zip).

A release revision with an update to the parquet files, newly created from the source data, should fix the problem I would imagine.

[dblodgett-usgs]

OK. If this isn't some other glitch and is a real issue with the parquet copy of the data, this'll be something to address with Mike Wieczorek and have the sciencebase team update. Can you share a reprex that shows what comids are problematic?

[mjcashman]

Sorry for not providing a reprex. I'll add it at the end here. I don't believe this is a glitch, because I was able to recreate the issue not just through the function pull, but with direct comparisons to downloaded files too, including filtering a local parquet both in and out of memory.

This might have been caused by an inner_join() during parquet creation for the entire science item, as many of these flowlines might be missing attributes such as sinuosity, which would be omitted for coastal flowlines? Can't say this issue doesn't exist for any other datasets, but when comparing to at least a subset of other data I'm using in my model, this is the item that has the biggest gaps by far.

#A representative subset of the types of comids missing values
missing_comid_regex <- c(948070456, 904040040, 167578596, 30726946,
                         28125120, 22783330, 16933590, 11079691, 7734231, 2683860,
                         800257, 478779, -6006, -70044, -127420, -276281, -464439)

#No values are returned for any local or routed variables in the select topographic attributes category
nhdplusTools::get_catchment_characteristics(varname = c("CAT_BASIN_AREA","ACC_BASIN_AREA","TOT_BASIN_AREA",
                                                        "CAT_BASIN_SLOPE","ACC_BASIN_SLOPE", "TOT_BASIN_SLOPE",
                                                        "CAT_ELEV_MEAN","ACC_ELEV_MEAN","TOT_ELEV_MEAN"),
                                            ids = missing_comid_regex)

#They do return values for NLCD data, even if it might be 100% no data
categorynhdplusTools::get_catchment_characteristics(varname = c("CAT_NLCD19_81", "ACC_NLCD19_81", "TOT_NLCD19_81"),
                                            ids = missing_comid_regex)

#Example for comids that work as normal
contained_comid_regex <- c(948091104, 27807501, 91666643, 30833954, 101)
nhdplusTools::get_catchment_characteristics(varname = c("CAT_BASIN_AREA","ACC_BASIN_AREA","TOT_BASIN_AREA",
                                                        "CAT_BASIN_SLOPE","ACC_BASIN_SLOPE", "TOT_BASIN_SLOPE",
                                                        "CAT_ELEV_MEAN","ACC_ELEV_MEAN","TOT_ELEV_MEAN"),
                                            ids = reprex_contained_list)

[dblodgett-usgs]

Your guess is as good as mine as to why these didn't make it into those parquet files. I could dust off my old code that did the conversions and see about regenerating that particular set -- I'll see what it would take to do.

[dblodgett-usgs]

OK -- so, yes -- the issue is that in some records, not all attributes have the same number of catchments. The rewrite code assumed that within a sciencebase record, the number of rows of data was the same. Fixable -- but likely not something we'll update on sciencebase. I am re-upping my project that created the parquet data in the first place and we'll get this fixed on a new hosting location -- the sciencebase S3 is something I don't feel confident in being able to use long run. Stay tuned.