Consolidate completion check

[TimidRobot]

Problem

There are multiple completion checks (individual fetch scripts, shared.check_completion_file_exists). Not all of them support --force.

Description

I expect a single shared function with appropriate arguments can satisfy all completion checks

• probably two arguments:
  • args - argparse Namespace
  • files - dictionary: {file_path: minimum lines}
• Shouldn't check without --enable-save
• No need to parse file with csv to count lines
• --force needs to be supported by scripts that use it.

      parser.add_argument(
          "--force",
          action="store_true",
          help="Generate new output files even if they already exist",
      )

  • All scripts that support --force should have same argument stanza and arguments should be ordered normally
• args.quarter man need to be added to scripts that use it
• Recommend a minimum of two PRs:
  1. for scripts that use shared.check_completion_file_exists
  2. scripts that have their own completion check

Additional context

I worked on this briefly before giving up due to scope of changes, different priorites, etc. The following function should be a good starting place:

def check_for_completion(args, files):
    """ 
    Check if files exist and have at least the specified number of lines (if
    specified). Can be overrideend on the command line with the --force option.
    If fiels exist and have minimum number of specified lines, the script exits
    early by raising a QuantifyingException with an exit status code of 0.
    """
    if not args.enable_save or args.force:
        return
    all_files_exist = True
    all_files_complete = True
    for path, minimum_lines in files.items():
        if not os.path.exists(path):
            all_files_exist = False
            print(f"{path} does not exist")
        elif minimum_lines is not None and minimum_lines > 0:
            with open(path, "r", encoding="utf-8") as file_obj:
                reader = csv.DictReader(file_obj, dialect="unix")
                if len(list(reader)) < minimum_lines:
                    all_files_complete = False
                    print(f"{path} has too few lines")
    if all_files_exist and all_files_complete:
        raise QuantifyingException(
            "All output files are already present and appear complete for"
            f" {args.quarter}",
            0,
        )

Implementation

• I would be interested in implementing this feature.