Skip to content

file_format not correctly set for Databricks 2.9.3 #509

@StLWallace

Description

@StLWallace

I believe there's a small bug in dbt_project.yml where the file_format key gets set. When trying to upgrade, we get the following errors:

Compilation Error in model seed_executions (models/sources/seed_executions.sql)
  Invalid file format provided: 
      Expected one of: text, csv, json, jdbc, parquet, orc, hive, delta, libsvm, hudi
  
  > in macro dbt_databricks_validate_get_file_format (macros/materializations/incremental/validate.sql)
  > called by macro materialization_incremental_databricks (macros/materializations/incremental/incremental.sql)
  > called by model seed_executions (models/sources/seed_executions.sql)

And similar for each dbt artifacts model.

I believe the fix is to change the piece below to

+file_format: '{{ "delta" if target.type == "databricks" else "" }}'

+file_format: '{{ "delta" if target.name == "databricks" else "" }}'

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions