Apache Spark has taken over the big data world. Spark is implemented with Scala and is well know for its performance.
Submitting spark jobs implemented with Scala is pretty easy and convenient. All we need is to submit our file as our input to the spark command.
First we have to download and setup a spark version locally.
Then will shall download a text file for testing. In my case the script from MGS2 did the work.
Now on to the WordCount script. For local testing we will use a file from our file system.
val text = sc.textFile("mytextfile.txt") val counts = text.flatMap(line => line.split(" ")).map(word => (word,1)).reduceByKey(_+_) counts.collect
Next step is to run the script
spark-shell -i WordCountscala.scala
Once finished a Spark command prompt will appear and we are free to do some experiments with the word count results
Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 2.1.0 /_/ Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_111) Type in expressions to have them evaluated. Type :help for more information. scala> res0.length res1: Int = 20159
Thus we detected 20159 different words.
Our next step is to run our job to a spark cluster on HDInsight.