What is Spark demand currently?

I have used Spark on Databricks quite long, without understanding it properly (my known language is Python, so I use pyspark but would like to dig deeper into spark/scala). I like the idea of Spark being open source so it can be relevant for understanding other tools such as Databricks better in-depth and my impression is that big data processing/ML in the academia/research is often done directly on Spark. I have one foot in research and could work in that …

databricks datascience demand language open source pyspark python scala spark tools understanding