Add Options for DataFrameWriter

[pequalsnp]

What changes were proposed in this pull request?

• This pipes options through DataFrameWriter into the WriterOperation proto message.

Why are the changes needed?

• Write options are supported in the API already, this just exposes the ability to set them in golang
• Write options are needed for many output formats, in my specific case Opensearch

Does this PR introduce any user-facing change?

Yes!

It adds the ability to set options on DataFrameWriter

How was this patch tested?

Added a call to Option in the unit tests.
Added an integration test for writer options

[caldempsey]

Can you add an integration test to demonstrate usage and use a Dataframe option which would affect the displayed results? Maybe something like NULLVALUE -> NA.

See https://github.com/apache/spark-connect-go/tree/master/internal/tests/integration

[pequalsnp]

➜  spark-connect-go git:(data-fram-write-options) ✗ SPARK_HOME=~/spark-4.0.1-bin-hadoop3 make integration
>> TEST, "integration"
ok      github.com/apache/spark-connect-go/internal/tests/integration   6.281s

[pequalsnp]

Can you add an integration test to demonstrate usage and use a Dataframe option which would affect the displayed results? Maybe something like NULLVALUE -> NA.

See https://github.com/apache/spark-connect-go/tree/master/internal/tests/integration

I used the header option on the CSV reader/writer. Created a df, wrote it with a header, read it back with a header, verified there were only 2 lines