ValueError in `finalize_catalog()` when only one hpix

[rancesol]

When I run finalize_catalog() but there is only one hpix (i.e. one foo_hpix.fit) I get the issue

Traceback (most recent call last):
  File "/pbs/throng/lsst/users/rsolomon/redmapper/data_prep/lib/createPixFilesParr_GCR.py", line 148, in <module>
    maker.finalize_catalog()
  File "/sps/lsst/users/rsolomon/conda_envs/redmapper/lib/python3.11/site-packages/redmapper/galaxy.py", line 889, in finalize_catalog
    tab.to_fits_file(self.filename, header=hdr, clobber=True)
  File "/sps/lsst/users/rsolomon/conda_envs/redmapper/lib/python3.11/site-packages/redmapper/catalog.py", line 146, in to_fits_file
    fitsio.write(filename, temp_array, clobber=clobber, header=header, extname=extname)
  File "/sps/lsst/users/rsolomon/conda_envs/redmapper/lib/python3.11/site-packages/fitsio/fitslib.py", line 409, in write
    fits.write(
  File "/sps/lsst/users/rsolomon/conda_envs/redmapper/lib/python3.11/site-packages/fitsio/fitslib.py", line 761, in write
    self.write_table(data, units=units,
  File "/sps/lsst/users/rsolomon/conda_envs/redmapper/lib/python3.11/site-packages/fitsio/fitslib.py", line 1182, in write_table
    self[-1].write(data, names=names)
  File "/sps/lsst/users/rsolomon/conda_envs/redmapper/lib/python3.11/site-packages/fitsio/hdu/table.py", line 291, in write
    self._verify_column_data(tcolnum, tdata)
  File "/sps/lsst/users/rsolomon/conda_envs/redmapper/lib/python3.11/site-packages/fitsio/hdu/table.py", line 417, in _verify_column_data
    raise ValueError(
ValueError: bad input shape for column 'filenames': expected '(1,)', got '()'

Seems like a list of foo_hpix.fit file names is expected but if there is only one hpix then it does not receive a list and breaks.
This is easy enough to ignore in cosmoDC2 since the footprint is in hpix format but DP0.2 is trickier and would require really high resolution to avoid this issue.

[erykoff]

I've had a chance to look at this, and this appears to be a bug deep in the bowels of fitsio with a single element array of a string column. So far I haven't been able to figure out a workaround, and I will file an issue upstream.

Meanwhile, I don't know why DP0.2 would be a problem. This only affects a catalog that is contained within a single nside 128 pixel (by default) and that is only 0.21 deg2 which isn't enough to do a redmapper run.

[rancesol]

Thanks. I was trying to parallelize the data ingestion by running on each healpix of nside=64. This is also too small for redmapper to run on but I will then combine everything afterwards, including recalculating the master files, so the full area will be 300deg2 in the end.

I think (to be confirmed) the issue may be with the edges of the DP0.2 footprint. In the default case of nside = 128, if less than 0.21 deg2 of a healpix is included in the DP0.2 footprint then when you go to create the master file you only have one file to concatenate. But if that is the case then it would have also been a problem during the production of the redmapper.DC2 catalog.

[erykoff]

Are you using the parallel ingestion feature (not well documented)?

redmapper/tests/test_galaxycatalog.py

Line 188 in 0804fd4

def test_galaxycatalog_create_parallel(self):

You should be able to make a bunch of catalog makers (using multiprocessing) and have them each do the ingestion and it puts together everything at the end. It makes sure that only one writer does the finalization which makes sure that everything is put together properly.

I strongly do not recommend running finalize separately without the parallel mode because then combining the "master" catalogs which describe the full thing is going to be very challenging. (Not just because of this bug, but because you're going to have to carefully re-engineer the file format).

Also, if you ingest the catalog with bigger nside there will be larger i/o and memory overheads when running the cluster finder.

[rancesol]

No, I'm not using the parallel ingestion feature because of the "not well documented" warning. I figured I would try the same method as I was doing for WaZP but maybe the two are too different. Good to know for the future.

But I'll start wading through the parallel ingestion feature. Thanks!