It seems that, when running eclairjs-node against the eclairjs/minimal-gateway docker container, the URL of the spark cluster that you want to talk to can be overridden by the Scala SparkContext that Toree creates by default. This results in jobs never reaching the targeted cluster and being executed locally instead.
// Launch Spark instance
var spark = require('eclairjs');
var sparkContext = new spark.SparkContext('spark://X.X.X.X:8443', "dataframe test");The above, when run with JUPYTER_PORT and JUPYTER_HOST pointing to the eclairjs/minimal-gateway docker container, runs locally (on the docker container) instead of on the specified cluster.
It seems that, when running eclairjs-node against the eclairjs/minimal-gateway docker container, the URL of the spark cluster that you want to talk to can be overridden by the Scala SparkContext that Toree creates by default. This results in jobs never reaching the targeted cluster and being executed locally instead.
The above, when run with JUPYTER_PORT and JUPYTER_HOST pointing to the eclairjs/minimal-gateway docker container, runs locally (on the docker container) instead of on the specified cluster.