Getting Started with Spark on Windows 7 (64 bit)

Lets get started on Apache Spark 1.6 on Windows 7 (64 Bit). [ Mac, Ubuntu, other OS steps are similar except winutils step that is only for Windows OS ]

-  Download and install Java  (Needs Java 1.7 or 1.8, Ignore if already installed)
-  Download & Install Anaconda Python 3.5+.  (Extract to C:\Anaconda3 or any folder )
-  Download Spark  ( Download 7-zip to unzip .gz files) : Extract to C:\BigData\Spark making sure that all 15 folders go under C:\BigData\Spark folder and not in long folder name with version number
-  Download winutils.exe  ( Put in C:\BigData\Hadoop\bin )  -- This is for 64-bit
-  Download Sample Data  (Extract to C:\BigData\Data)

1. Create Environment Variables :-

    SPARK_HOME : C:\BigData\Spark
    HADOOP_HOME : C:\BigData\Hadoop

    JAVA_HOME :  Make sure JAVA_HOME is Environment variable is defined and pointing to Java home directory.

2. Add to Environment variable PATH at end:

   ;%SPARK_HOME%\bin;%HADOOP_HOME%\bin;%JAVA_HOME%\bin;

3. Create folder  

    c:\tmp\hive

 
4. On Command prompt in Admin Mode : (One time only)

winutils.exe chmod -R 777 \tmp\hive

5. On command prompt in Adminstrator mode start Spark using :-

pyspark --packages com.databricks:spark-csv_2.11:1.3.0

[ If everything is set you should see Welcome to Spark version 1.6 using Python 3.5.1 message ]

6.  Type following commands on Pyspark shell :-

sc.setLogLevel("WARN")

movies=sqlContext.read.format("com.databricks.spark.csv").options(delimiter="|").options(header="false").load("file:///BigData/Data/ml-100k/u.item")

movies.registerTempTable("movies")

movies.cache()

movies.show()

ratings=sqlContext.read.format("com.databricks.spark.csv").options(delimiter="\t").options(header="false").load("file:///BigData/Data/ml-100k/u.data")

ratings.registerTempTable("ratings")

ratings.cache()

ratings.show()

# To view Ratings distribution :

ratingsDistribution=sqlContext.sql("Select c2 as ratings, count(*) as cnt from ratings group by c2")

TopMovies.show()

#Top Most Watched Movies :

TopMovies=sqlContext.sql("Select c1 as movieId, Count(*) as Cnt from ratings group by c1 Order by Cnt Desc")

TopMovies.show()

# Top Most Watched Movies by Name

TopMovieNames=sqlContext.sql("Select movies.c1 as MovieName, count(*) as Cnt from ratings, movies where ratings.c1=movies.c0 group by movies.c1 Order by Cnt desc")

TopMovieNames.show()

---------------------------------------------------------------
Some more troubleshooting commands/tips :-
---------------------------------------------------------------

sqlContext._get_hive_ctx()      - If this runs clean with no errors, then Winutils.exe version, HADOOP_HOME path etc is correct. 

sc.appName="myFirstApp"  - appName appears in Jobs, easy to differentiate. 

winutils.exe ls \tmp\hive  : This command on windows Command propmt will display access level to \tmp\hive folder. If you see any errors, then probably you may not have proper access to C:\ drive (especially work laptop that has restrictions to C:\ drive. In that case try to run winutils and pyspark command from D:\ prompt. If \tmp\hive permissions are not set properly, you may receive error like this ( java.lang.RuntimeException: java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: rw-rw-rw-)

type sc. and hit Tab key will list all commands, options. Same with sqlContext.

ratings.unpersist()   - to remove from Cache. ( ratings.cache() to put in Memory ) 

create C:\BigData\Spark\bin\myspark.bat with pyspark --packages com.databricks:spark-csv_2.11:1.3.0  in it. Next time just type myspark on command line to open pyspark with CSV package.

Download larger Movie/Ratings data sets to slice n dice data in different ways and evaluate performance (memory/cpu) implications when Data is cached vs not cached. Explore other related data sets at this Link