Linear_Regression_Consulting_Project_SOLUTIONS.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Linear Regression Project - SOLUTIONS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Congratulations! You've been contracted by Hyundai Heavy Industries to help them build a predictive model for some ships. [Hyundai Heavy Industries](http://www.hyundai.eu/en) is one of the world's largest ship manufacturing companies and builds cruise liners.\n",
    "\n",
    "You've been flown to their headquarters in Ulsan, South Korea to help them give accurate estimates of how many crew members a ship will require.\n",
    "\n",
    "They are currently building new ships for some customers and want you to create a model and use it to predict how many crew members the ships will need.\n",
    "\n",
    "Here is what the data looks like so far:\n",
    "\n",
    "    Description: Measurements of ship size, capacity, crew, and age for 158 cruise\n",
    "    ships.\n",
    "\n",
    "\n",
    "    Variables/Columns\n",
    "    Ship Name     1-20\n",
    "    Cruise Line   21-40\n",
    "    Age (as of 2013)   46-48\n",
    "    Tonnage (1000s of tons)   50-56\n",
    "    passengers (100s)   58-64\n",
    "    Length (100s of feet)  66-72\n",
    "    Cabins  (100s)   74-80\n",
    "    Passenger Density   82-88\n",
    "    Crew  (100s)   90-96\n",
    "    \n",
    "It is saved in a csv file for you called \"cruise_ship_info.csv\". Your job is to create a regression model that will help predict how many crew members will be needed for future ships. The client also mentioned that they have found that particular cruise lines will differ in acceptable crew counts, so it is most likely an important feature to include in your analysis! \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from pyspark.sql import SparkSession"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "spark = SparkSession.builder.appName('cruise').getOrCreate()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df = spark.read.csv('cruise_ship_info.csv',inferSchema=True,header=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- Ship_name: string (nullable = true)\n",
      " |-- Cruise_line: string (nullable = true)\n",
      " |-- Age: integer (nullable = true)\n",
      " |-- Tonnage: double (nullable = true)\n",
      " |-- passengers: double (nullable = true)\n",
      " |-- length: double (nullable = true)\n",
      " |-- cabins: double (nullable = true)\n",
      " |-- passenger_density: double (nullable = true)\n",
      " |-- crew: double (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df.printSchema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-----------+-----------+---+------------------+----------+------+------+-----------------+----+\n",
      "|  Ship_name|Cruise_line|Age|           Tonnage|passengers|length|cabins|passenger_density|crew|\n",
      "+-----------+-----------+---+------------------+----------+------+------+-----------------+----+\n",
      "|    Journey|    Azamara|  6|30.276999999999997|      6.94|  5.94|  3.55|            42.64|3.55|\n",
      "|      Quest|    Azamara|  6|30.276999999999997|      6.94|  5.94|  3.55|            42.64|3.55|\n",
      "|Celebration|   Carnival| 26|            47.262|     14.86|  7.22|  7.43|             31.8| 6.7|\n",
      "|   Conquest|   Carnival| 11|             110.0|     29.74|  9.53| 14.88|            36.99|19.1|\n",
      "|    Destiny|   Carnival| 17|           101.353|     26.42|  8.92| 13.21|            38.36|10.0|\n",
      "|    Ecstasy|   Carnival| 22|            70.367|     20.52|  8.55|  10.2|            34.29| 9.2|\n",
      "|    Elation|   Carnival| 15|            70.367|     20.52|  8.55|  10.2|            34.29| 9.2|\n",
      "|    Fantasy|   Carnival| 23|            70.367|     20.56|  8.55| 10.22|            34.23| 9.2|\n",
      "|Fascination|   Carnival| 19|            70.367|     20.52|  8.55|  10.2|            34.29| 9.2|\n",
      "|    Freedom|   Carnival|  6|110.23899999999999|      37.0|  9.51| 14.87|            29.79|11.5|\n",
      "|      Glory|   Carnival| 10|             110.0|     29.74|  9.51| 14.87|            36.99|11.6|\n",
      "|    Holiday|   Carnival| 28|            46.052|     14.52|  7.27|  7.26|            31.72| 6.6|\n",
      "|Imagination|   Carnival| 18|            70.367|     20.52|  8.55|  10.2|            34.29| 9.2|\n",
      "|Inspiration|   Carnival| 17|            70.367|     20.52|  8.55|  10.2|            34.29| 9.2|\n",
      "|     Legend|   Carnival| 11|              86.0|     21.24|  9.63| 10.62|            40.49| 9.3|\n",
      "|   Liberty*|   Carnival|  8|             110.0|     29.74|  9.51| 14.87|            36.99|11.6|\n",
      "|    Miracle|   Carnival|  9|              88.5|     21.24|  9.63| 10.62|            41.67|10.3|\n",
      "|   Paradise|   Carnival| 15|            70.367|     20.52|  8.55|  10.2|            34.29| 9.2|\n",
      "|      Pride|   Carnival| 12|              88.5|     21.24|  9.63| 11.62|            41.67| 9.3|\n",
      "|  Sensation|   Carnival| 20|            70.367|     20.52|  8.55|  10.2|            34.29| 9.2|\n",
      "+-----------+-----------+---+------------------+----------+------+------+-----------------+----+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+------------------+------------------+-----------------+-----------------+------------------+-----------------+-----------------+\n",
      "|summary|               Age|           Tonnage|       passengers|           length|            cabins|passenger_density|             crew|\n",
      "+-------+------------------+------------------+-----------------+-----------------+------------------+-----------------+-----------------+\n",
      "|  count|               158|               158|              158|              158|               158|              158|              158|\n",
      "|   mean|15.689873417721518| 71.28467088607599|18.45740506329114|8.130632911392404| 8.830000000000005|39.90094936708861|7.794177215189873|\n",
      "| stddev| 7.615691058751413|37.229540025907866|9.677094775143416|1.793473548054825|4.4714172221480615| 8.63921711391542|3.503486564627034|\n",
      "|    min|                 4|             2.329|             0.66|             2.79|              0.33|             17.7|             0.59|\n",
      "|    max|                48|             220.0|             54.0|            11.82|              27.0|            71.43|             21.0|\n",
      "+-------+------------------+------------------+-----------------+-----------------+------------------+-----------------+-----------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df.describe().show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dealing with the Cruise_line categorical variable\n",
    "Ship Name is a useless arbitrary string, but the cruise_line itself may be useful. Let's make it into a categorical variable!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-----------------+-----+\n",
      "|      Cruise_line|count|\n",
      "+-----------------+-----+\n",
      "|            Costa|   11|\n",
      "|              P&O|    6|\n",
      "|           Cunard|    3|\n",
      "|Regent_Seven_Seas|    5|\n",
      "|              MSC|    8|\n",
      "|         Carnival|   22|\n",
      "|          Crystal|    2|\n",
      "|           Orient|    1|\n",
      "|         Princess|   17|\n",
      "|        Silversea|    4|\n",
      "|         Seabourn|    3|\n",
      "| Holland_American|   14|\n",
      "|         Windstar|    3|\n",
      "|           Disney|    2|\n",
      "|        Norwegian|   13|\n",
      "|          Oceania|    3|\n",
      "|          Azamara|    2|\n",
      "|        Celebrity|   10|\n",
      "|             Star|    6|\n",
      "|  Royal_Caribbean|   23|\n",
      "+-----------------+-----+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df.groupBy('Cruise_line').count().show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Row(Ship_name='Journey', Cruise_line='Azamara', Age=6, Tonnage=30.276999999999997, passengers=6.94, length=5.94, cabins=3.55, passenger_density=42.64, crew=3.55, cruise_cat=16.0),\n",
       " Row(Ship_name='Quest', Cruise_line='Azamara', Age=6, Tonnage=30.276999999999997, passengers=6.94, length=5.94, cabins=3.55, passenger_density=42.64, crew=3.55, cruise_cat=16.0),\n",
       " Row(Ship_name='Celebration', Cruise_line='Carnival', Age=26, Tonnage=47.262, passengers=14.86, length=7.22, cabins=7.43, passenger_density=31.8, crew=6.7, cruise_cat=1.0),\n",
       " Row(Ship_name='Conquest', Cruise_line='Carnival', Age=11, Tonnage=110.0, passengers=29.74, length=9.53, cabins=14.88, passenger_density=36.99, crew=19.1, cruise_cat=1.0),\n",
       " Row(Ship_name='Destiny', Cruise_line='Carnival', Age=17, Tonnage=101.353, passengers=26.42, length=8.92, cabins=13.21, passenger_density=38.36, crew=10.0, cruise_cat=1.0)]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from pyspark.ml.feature import StringIndexer\n",
    "indexer = StringIndexer(inputCol=\"Cruise_line\", outputCol=\"cruise_cat\")\n",
    "indexed = indexer.fit(df).transform(df)\n",
    "indexed.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from pyspark.ml.linalg import Vectors\n",
    "from pyspark.ml.feature import VectorAssembler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Ship_name',\n",
       " 'Cruise_line',\n",
       " 'Age',\n",
       " 'Tonnage',\n",
       " 'passengers',\n",
       " 'length',\n",
       " 'cabins',\n",
       " 'passenger_density',\n",
       " 'crew',\n",
       " 'cruise_cat']"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "indexed.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "assembler = VectorAssembler(\n",
    "  inputCols=['Age',\n",
    "             'Tonnage',\n",
    "             'passengers',\n",
    "             'length',\n",
    "             'cabins',\n",
    "             'passenger_density',\n",
    "             'cruise_cat'],\n",
    "    outputCol=\"features\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "output = assembler.transform(indexed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+----+\n",
      "|            features|crew|\n",
      "+--------------------+----+\n",
      "|[6.0,30.276999999...|3.55|\n",
      "|[6.0,30.276999999...|3.55|\n",
      "|[26.0,47.262,14.8...| 6.7|\n",
      "|[11.0,110.0,29.74...|19.1|\n",
      "|[17.0,101.353,26....|10.0|\n",
      "|[22.0,70.367,20.5...| 9.2|\n",
      "|[15.0,70.367,20.5...| 9.2|\n",
      "|[23.0,70.367,20.5...| 9.2|\n",
      "|[19.0,70.367,20.5...| 9.2|\n",
      "|[6.0,110.23899999...|11.5|\n",
      "|[10.0,110.0,29.74...|11.6|\n",
      "|[28.0,46.052,14.5...| 6.6|\n",
      "|[18.0,70.367,20.5...| 9.2|\n",
      "|[17.0,70.367,20.5...| 9.2|\n",
      "|[11.0,86.0,21.24,...| 9.3|\n",
      "|[8.0,110.0,29.74,...|11.6|\n",
      "|[9.0,88.5,21.24,9...|10.3|\n",
      "|[15.0,70.367,20.5...| 9.2|\n",
      "|[12.0,88.5,21.24,...| 9.3|\n",
      "|[20.0,70.367,20.5...| 9.2|\n",
      "+--------------------+----+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "output.select(\"features\", \"crew\").show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "final_data = output.select(\"features\", \"crew\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train_data,test_data = final_data.randomSplit([0.7,0.3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from pyspark.ml.regression import LinearRegression\n",
    "# Create a Linear Regression Model object\n",
    "lr = LinearRegression(labelCol='crew')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Fit the model to the data and call this model lrModel\n",
    "lrModel = lr.fit(train_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Coefficients: [-0.0145423814068,0.0137445818936,-0.111000735058,0.422234330769,0.705574105078,-0.00631202648669,0.0306212943631] Intercept: -0.5598623529951635\n"
     ]
    }
   ],
   "source": [
    "# Print the coefficients and intercept for linear regression\n",
    "print(\"Coefficients: {} Intercept: {}\".format(lrModel.coefficients,lrModel.intercept))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "test_results = lrModel.evaluate(test_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE: 1.3174339720092743\n",
      "MSE: 1.7356322706041332\n",
      "R2: 0.8671622449217978\n"
     ]
    }
   ],
   "source": [
    "print(\"RMSE: {}\".format(test_results.rootMeanSquaredError))\n",
    "print(\"MSE: {}\".format(test_results.meanSquaredError))\n",
    "print(\"R2: {}\".format(test_results.r2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# R2 of 0.86 is pretty good, let's check the data a little closer\n",
    "from pyspark.sql.functions import corr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+----------------------+\n",
      "|corr(crew, passengers)|\n",
      "+----------------------+\n",
      "|    0.9152341306065384|\n",
      "+----------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df.select(corr('crew','passengers')).show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------------------+\n",
      "|corr(crew, cabins)|\n",
      "+------------------+\n",
      "|0.9508226063578497|\n",
      "+------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df.select(corr('crew','cabins')).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Okay, so maybe it does make sense! Well that is good news for us, this is information we can bring to the company!\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hope you enjoyed your first consulting gig!\n",
    "# Great Job!"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [conda root]",
   "language": "python",
   "name": "conda-root-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}