Logistic_Regression_Consulting_Project.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Logistic Regression Consulting Project"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Binary Customer Churn\n",
    "\n",
    "A marketing agency has many customers that use their service to produce ads for the client/customer websites. They've noticed that they have quite a bit of churn in clients. They basically randomly assign account managers right now, but want you to create a machine learning model that will help predict which customers will churn (stop buying their service) so that they can correctly assign the customers most at risk to churn an account manager. Luckily they have some historical data, can you help them out? Create a classification algorithm that will help classify whether or not a customer churned. Then the company can test this against incoming data for future customers to predict which customers will churn and assign them an account manager.\n",
    "\n",
    "The data is saved as customer_churn.csv. Here are the fields and their definitions:\n",
    "\n",
    "    Name : Name of the latest contact at Company\n",
    "    Age: Customer Age\n",
    "    Total_Purchase: Total Ads Purchased\n",
    "    Account_Manager: Binary 0=No manager, 1= Account manager assigned\n",
    "    Years: Totaly Years as a customer\n",
    "    Num_sites: Number of websites that use the service.\n",
    "    Onboard_date: Date that the name of the latest contact was onboarded\n",
    "    Location: Client HQ Address\n",
    "    Company: Name of Client Company\n",
    "    \n",
    "Once you've created the model and evaluated it, test out the model on some new data (you can think of this almost like a hold-out set) that your client has provided, saved under new_customers.csv. The client wants to know which customers are most likely to churn given this data (they don't have the label yet)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.sql import SparkSession\n",
    "from pyspark.ml.classification import LogisticRegression\n",
    "from pyspark.ml.feature import VectorAssembler\n",
    "from pyspark.ml.evaluation import BinaryClassificationEvaluator\n",
    "from pyspark.sql.functions import col, isnull, isnan, year, month, day, dayofmonth, concat, dayofweek, dayofyear, weekofyear, weekday, hour, minute, second"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "            <div>\n",
       "                <p><b>SparkSession - in-memory</b></p>\n",
       "                \n",
       "        <div>\n",
       "            <p><b>SparkContext</b></p>\n",
       "\n",
       "            <p><a href=\"http://DESKTOP-BJS1Q67:4044\">Spark UI</a></p>\n",
       "\n",
       "            <dl>\n",
       "              <dt>Version</dt>\n",
       "                <dd><code>v3.5.0</code></dd>\n",
       "              <dt>Master</dt>\n",
       "                <dd><code>local[*]</code></dd>\n",
       "              <dt>AppName</dt>\n",
       "                <dd><code>customer_churn</code></dd>\n",
       "            </dl>\n",
       "        </div>\n",
       "        \n",
       "            </div>\n",
       "        "
      ],
      "text/plain": [
       "<pyspark.sql.session.SparkSession at 0x21fe2f60910>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "spark = SparkSession.builder.appName('customer_churn').getOrCreate()\n",
    "spark"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- Names: string (nullable = true)\n",
      " |-- Age: double (nullable = true)\n",
      " |-- Total_Purchase: double (nullable = true)\n",
      " |-- Account_Manager: integer (nullable = true)\n",
      " |-- Years: double (nullable = true)\n",
      " |-- Num_Sites: double (nullable = true)\n",
      " |-- Onboard_date: timestamp (nullable = true)\n",
      " |-- Location: string (nullable = true)\n",
      " |-- Company: string (nullable = true)\n",
      " |-- Churn: integer (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df = spark.read.csv('customer_churn.csv',inferSchema=True,header=True)\n",
    "df.printSchema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "900"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------------------+----+--------------+---------------+-----+---------+-------------------+--------------------+--------------------+-----+\n",
      "|              Names| Age|Total_Purchase|Account_Manager|Years|Num_Sites|       Onboard_date|            Location|             Company|Churn|\n",
      "+-------------------+----+--------------+---------------+-----+---------+-------------------+--------------------+--------------------+-----+\n",
      "|   Cameron Williams|42.0|       11066.8|              0| 7.22|      8.0|2013-08-30 07:00:40|10265 Elizabeth M...|          Harvey LLC|    1|\n",
      "|      Kevin Mueller|41.0|      11916.22|              0|  6.5|     11.0|2013-08-13 00:38:46|6157 Frank Garden...|          Wilson PLC|    1|\n",
      "|        Eric Lozano|38.0|      12884.75|              0| 6.67|     12.0|2016-06-29 06:20:07|1331 Keith Court ...|Miller, Johnson a...|    1|\n",
      "|      Phillip White|42.0|       8010.76|              0| 6.71|     10.0|2014-04-22 12:43:12|13120 Daniel Moun...|           Smith Inc|    1|\n",
      "|     Cynthia Norton|37.0|       9191.58|              0| 5.56|      9.0|2016-01-19 15:31:15|765 Tricia Row Ka...|          Love-Jones|    1|\n",
      "|   Jessica Williams|48.0|      10356.02|              0| 5.12|      8.0|2009-03-03 23:13:37|6187 Olson Mounta...|        Kelly-Warren|    1|\n",
      "|        Eric Butler|44.0|      11331.58|              1| 5.23|     11.0|2016-12-05 03:35:43|4846 Savannah Roa...|   Reynolds-Sheppard|    1|\n",
      "|      Zachary Walsh|32.0|       9885.12|              1| 6.92|      9.0|2006-03-09 14:50:20|25271 Roy Express...|          Singh-Cole|    1|\n",
      "|        Ashlee Carr|43.0|       14062.6|              1| 5.46|     11.0|2011-09-29 05:47:23|3725 Caroline Str...|           Lopez PLC|    1|\n",
      "|     Jennifer Lynch|40.0|       8066.94|              1| 7.11|     11.0|2006-03-28 15:42:45|363 Sandra Lodge ...|       Reed-Martinez|    1|\n",
      "|       Paula Harris|30.0|      11575.37|              1| 5.22|      8.0|2016-11-13 13:13:01|Unit 8120 Box 916...|Briggs, Lamb and ...|    1|\n",
      "|     Bruce Phillips|45.0|       8771.02|              1| 6.64|     11.0|2015-05-28 12:14:03|Unit 1895 Box 094...|    Figueroa-Maynard|    1|\n",
      "|       Craig Garner|45.0|       8988.67|              1| 4.84|     11.0|2011-02-16 08:10:47|897 Kelley Overpa...|     Abbott-Thompson|    1|\n",
      "|       Nicole Olson|40.0|       8283.32|              1|  5.1|     13.0|2012-11-22 05:35:03|11488 Weaver Cape...|Smith, Kim and Ma...|    1|\n",
      "|     Harold Griffin|41.0|       6569.87|              1|  4.3|     11.0|2015-03-28 02:13:44|1774 Peter Row Ap...|Snyder, Lee and M...|    1|\n",
      "|       James Wright|38.0|      10494.82|              1| 6.81|     12.0|2015-07-22 08:38:40|45408 David Path ...|      Sanders-Pierce|    1|\n",
      "|      Doris Wilkins|45.0|       8213.41|              1| 7.35|     11.0|2006-09-03 06:13:55|28216 Wright Moun...|Andrews, Adams an...|    1|\n",
      "|Katherine Carpenter|43.0|      11226.88|              0| 8.08|     12.0|2006-10-22 04:42:38|Unit 4948 Box 481...|Morgan, Phillips ...|    1|\n",
      "|     Lindsay Martin|53.0|       5515.09|              0| 6.85|      8.0|2015-10-07 00:27:10|69203 Crosby Divi...|      Villanueva LLC|    1|\n",
      "|        Kathy Curry|46.0|        8046.4|              1| 5.69|      8.0|2014-11-06 23:47:14|9569 Caldwell Cre...|Berry, Orr and Ca...|    1|\n",
      "+-------------------+----+--------------+---------------+-----+---------+-------------------+--------------------+--------------------+-----+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+-------------+-----------------+-----------------+------------------+-----------------+------------------+--------------------+--------------------+-------------------+\n",
      "|summary|        Names|              Age|   Total_Purchase|   Account_Manager|            Years|         Num_Sites|            Location|             Company|              Churn|\n",
      "+-------+-------------+-----------------+-----------------+------------------+-----------------+------------------+--------------------+--------------------+-------------------+\n",
      "|  count|          900|              900|              900|               900|              900|               900|                 900|                 900|                900|\n",
      "|   mean|         NULL|41.81666666666667|10062.82403333334|0.4811111111111111| 5.27315555555555| 8.587777777777777|                NULL|                NULL|0.16666666666666666|\n",
      "| stddev|         NULL|6.127560416916251|2408.644531858096|0.4999208935073339|1.274449013194616|1.7648355920350969|                NULL|                NULL| 0.3728852122772358|\n",
      "|    min|   Aaron King|             22.0|            100.0|                 0|              1.0|               3.0|00103 Jeffrey Cre...|     Abbott-Thompson|                  0|\n",
      "|    max|Zachary Walsh|             65.0|         18026.01|                 1|             9.15|              14.0|Unit 9800 Box 287...|Zuniga, Clark and...|                  1|\n",
      "+-------+-------------+-----------------+-----------------+------------------+-----------------+------------------+--------------------+--------------------+-------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df.describe().show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------------------+----+--------------+---------------+-----+---------+--------------------+--------------------+-----+\n",
      "|              Names| Age|Total_Purchase|Account_Manager|Years|Num_Sites|            Location|             Company|Churn|\n",
      "+-------------------+----+--------------+---------------+-----+---------+--------------------+--------------------+-----+\n",
      "|   Cameron Williams|42.0|       11066.8|              0| 7.22|      8.0|10265 Elizabeth M...|          Harvey LLC|    1|\n",
      "|      Kevin Mueller|41.0|      11916.22|              0|  6.5|     11.0|6157 Frank Garden...|          Wilson PLC|    1|\n",
      "|        Eric Lozano|38.0|      12884.75|              0| 6.67|     12.0|1331 Keith Court ...|Miller, Johnson a...|    1|\n",
      "|      Phillip White|42.0|       8010.76|              0| 6.71|     10.0|13120 Daniel Moun...|           Smith Inc|    1|\n",
      "|     Cynthia Norton|37.0|       9191.58|              0| 5.56|      9.0|765 Tricia Row Ka...|          Love-Jones|    1|\n",
      "|   Jessica Williams|48.0|      10356.02|              0| 5.12|      8.0|6187 Olson Mounta...|        Kelly-Warren|    1|\n",
      "|        Eric Butler|44.0|      11331.58|              1| 5.23|     11.0|4846 Savannah Roa...|   Reynolds-Sheppard|    1|\n",
      "|      Zachary Walsh|32.0|       9885.12|              1| 6.92|      9.0|25271 Roy Express...|          Singh-Cole|    1|\n",
      "|        Ashlee Carr|43.0|       14062.6|              1| 5.46|     11.0|3725 Caroline Str...|           Lopez PLC|    1|\n",
      "|     Jennifer Lynch|40.0|       8066.94|              1| 7.11|     11.0|363 Sandra Lodge ...|       Reed-Martinez|    1|\n",
      "|       Paula Harris|30.0|      11575.37|              1| 5.22|      8.0|Unit 8120 Box 916...|Briggs, Lamb and ...|    1|\n",
      "|     Bruce Phillips|45.0|       8771.02|              1| 6.64|     11.0|Unit 1895 Box 094...|    Figueroa-Maynard|    1|\n",
      "|       Craig Garner|45.0|       8988.67|              1| 4.84|     11.0|897 Kelley Overpa...|     Abbott-Thompson|    1|\n",
      "|       Nicole Olson|40.0|       8283.32|              1|  5.1|     13.0|11488 Weaver Cape...|Smith, Kim and Ma...|    1|\n",
      "|     Harold Griffin|41.0|       6569.87|              1|  4.3|     11.0|1774 Peter Row Ap...|Snyder, Lee and M...|    1|\n",
      "|       James Wright|38.0|      10494.82|              1| 6.81|     12.0|45408 David Path ...|      Sanders-Pierce|    1|\n",
      "|      Doris Wilkins|45.0|       8213.41|              1| 7.35|     11.0|28216 Wright Moun...|Andrews, Adams an...|    1|\n",
      "|Katherine Carpenter|43.0|      11226.88|              0| 8.08|     12.0|Unit 4948 Box 481...|Morgan, Phillips ...|    1|\n",
      "|     Lindsay Martin|53.0|       5515.09|              0| 6.85|      8.0|69203 Crosby Divi...|      Villanueva LLC|    1|\n",
      "|        Kathy Curry|46.0|        8046.4|              1| 5.69|      8.0|9569 Caldwell Cre...|Berry, Orr and Ca...|    1|\n",
      "+-------------------+----+--------------+---------------+-----+---------+--------------------+--------------------+-----+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df.drop('Onboard_date').show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Names': 'string',\n",
       " 'Age': 'double',\n",
       " 'Total_Purchase': 'double',\n",
       " 'Account_Manager': 'int',\n",
       " 'Years': 'double',\n",
       " 'Num_Sites': 'double',\n",
       " 'Onboard_date': 'timestamp',\n",
       " 'Location': 'string',\n",
       " 'Company': 'string',\n",
       " 'Churn': 'int'}"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_types = dict(df.dtypes)\n",
    "data_types"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Age', 'Total_Purchase', 'Account_Manager', 'Years', 'Num_Sites', 'Churn']"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "numerical_types = {\"int\", \"bigint\", \"float\", \"double\", \"decimal\", \"long\"}\n",
    "numerical_columns = [col_name for col_name, col_type in data_types.items() if col_type in numerical_types]\n",
    "numerical_columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Names', 'Location', 'Company']"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "categorical_cols = [col_name for col_name, col_type in data_types.items() if col_type == 'string']\n",
    "categorical_cols"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "null_cols = [column for column in numerical_columns if df.filter(col(column).isNull() | isnan(col(column))).count() > 0]\n",
    "null_cols"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "null_cols = [column for column in categorical_cols if df.filter(col(column).isNull() | isnan(col(column))).count() > 0]\n",
    "null_cols"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.filter(col('Onboard_date').isNull()).count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, there are no null values in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "900"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "final_df = df.dropDuplicates()\n",
    "final_df = final_df.dropna()\n",
    "final_df.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Names',\n",
       " 'Age',\n",
       " 'Total_Purchase',\n",
       " 'Account_Manager',\n",
       " 'Years',\n",
       " 'Num_Sites',\n",
       " 'Onboard_date',\n",
       " 'Location',\n",
       " 'Company',\n",
       " 'Churn',\n",
       " 'Year',\n",
       " 'Month',\n",
       " 'Day',\n",
       " 'Weekday',\n",
       " 'Week of Year',\n",
       " 'Day of Month',\n",
       " 'Day of Week',\n",
       " 'Day of Year',\n",
       " 'Hour',\n",
       " 'Minute',\n",
       " 'Second']"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "final_df = final_df.withColumn('Year',year('Onboard_date'))\n",
    "final_df = final_df.withColumn('Month',month('Onboard_date'))\n",
    "final_df = final_df.withColumn('Day',day('Onboard_date'))\n",
    "final_df = final_df.withColumn('Weekday',weekday('Onboard_date'))\n",
    "final_df = final_df.withColumn('Week of Year',weekofyear('Onboard_date'))\n",
    "final_df = final_df.withColumn('Day of Month',dayofmonth('Onboard_date'))\n",
    "final_df = final_df.withColumn('Day of Week',dayofweek('Onboard_date'))\n",
    "final_df = final_df.withColumn('Day of Year',dayofyear('Onboard_date'))\n",
    "final_df = final_df.withColumn('Hour',hour('Onboard_date'))\n",
    "final_df = final_df.withColumn('Minute',minute('Onboard_date'))\n",
    "final_df = final_df.withColumn('Second',second('Onboard_date'))\n",
    "final_df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- Names: string (nullable = true)\n",
      " |-- Age: double (nullable = true)\n",
      " |-- Total_Purchase: double (nullable = true)\n",
      " |-- Account_Manager: integer (nullable = true)\n",
      " |-- Years: double (nullable = true)\n",
      " |-- Num_Sites: double (nullable = true)\n",
      " |-- Onboard_date: timestamp (nullable = true)\n",
      " |-- Location: string (nullable = true)\n",
      " |-- Company: string (nullable = true)\n",
      " |-- Churn: integer (nullable = true)\n",
      " |-- Year: integer (nullable = true)\n",
      " |-- Month: integer (nullable = true)\n",
      " |-- Day: integer (nullable = true)\n",
      " |-- Weekday: integer (nullable = true)\n",
      " |-- Week of Year: integer (nullable = true)\n",
      " |-- Day of Month: integer (nullable = true)\n",
      " |-- Day of Week: integer (nullable = true)\n",
      " |-- Day of Year: integer (nullable = true)\n",
      " |-- Hour: integer (nullable = true)\n",
      " |-- Minute: integer (nullable = true)\n",
      " |-- Second: integer (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "final_df.printSchema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+-----+\n",
      "|             Company|count|\n",
      "+--------------------+-----+\n",
      "|            Soto PLC|    2|\n",
      "|            Todd LLC|    1|\n",
      "|           Smith PLC|    1|\n",
      "|          Obrien PLC|    1|\n",
      "|          Hall Group|    1|\n",
      "|Hunter, Reyes and...|    1|\n",
      "|Smith, Marshall a...|    1|\n",
      "|Miller, Johnson a...|    1|\n",
      "|       Smith-Carroll|    1|\n",
      "|          Cannon Inc|    1|\n",
      "|Hall, Hernandez a...|    1|\n",
      "|Freeman, Lam and ...|    1|\n",
      "|        Robinson PLC|    1|\n",
      "|Wilson, Collins a...|    1|\n",
      "|Jennings, Gates a...|    1|\n",
      "|     Campbell-Willis|    1|\n",
      "|    Martinez-Roberts|    1|\n",
      "|        White-Dennis|    1|\n",
      "|      Jackson-Garcia|    1|\n",
      "|          Hayes-Hill|    1|\n",
      "+--------------------+-----+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "final_df.groupby('Company').count().show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Names',\n",
       " 'Age',\n",
       " 'Total_Purchase',\n",
       " 'Account_Manager',\n",
       " 'Total_Years',\n",
       " 'Num_Sites',\n",
       " 'Onboard_date',\n",
       " 'Location',\n",
       " 'Company',\n",
       " 'Churn',\n",
       " 'Year',\n",
       " 'Month',\n",
       " 'Day',\n",
       " 'Weekday',\n",
       " 'Week of Year',\n",
       " 'Day of Month',\n",
       " 'Day of Week',\n",
       " 'Day of Year',\n",
       " 'Hour',\n",
       " 'Minute',\n",
       " 'Second']"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "final_df = final_df.withColumnRenamed(existing='Years',new='Total_Years')\n",
    "final_df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- Age: double (nullable = true)\n",
      " |-- Total_Purchase: double (nullable = true)\n",
      " |-- Account_Manager: integer (nullable = true)\n",
      " |-- Total_Years: double (nullable = true)\n",
      " |-- Num_Sites: double (nullable = true)\n",
      " |-- Year: integer (nullable = true)\n",
      " |-- Month: integer (nullable = true)\n",
      " |-- Day: integer (nullable = true)\n",
      " |-- Weekday: integer (nullable = true)\n",
      " |-- Week of Year: integer (nullable = true)\n",
      " |-- Day of Month: integer (nullable = true)\n",
      " |-- Day of Week: integer (nullable = true)\n",
      " |-- Day of Year: integer (nullable = true)\n",
      " |-- Hour: integer (nullable = true)\n",
      " |-- Minute: integer (nullable = true)\n",
      " |-- Second: integer (nullable = true)\n",
      " |-- Churn: integer (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "cols = ['Age','Total_Purchase','Account_Manager','Total_Years','Num_Sites','Year','Month','Day','Weekday','Week of Year','Day of Month','Day of Week','Day of Year','Hour','Minute','Second','Churn']\n",
    "\n",
    "final_df = final_df.select(cols)\n",
    "final_df.printSchema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "900"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "final_df = final_df.na.drop()\n",
    "final_df.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Age',\n",
       " 'Total_Purchase',\n",
       " 'Account_Manager',\n",
       " 'Total_Years',\n",
       " 'Num_Sites',\n",
       " 'Year',\n",
       " 'Month',\n",
       " 'Day',\n",
       " 'Weekday',\n",
       " 'Week of Year',\n",
       " 'Day of Month',\n",
       " 'Day of Week',\n",
       " 'Day of Year',\n",
       " 'Hour',\n",
       " 'Minute',\n",
       " 'Second',\n",
       " 'Churn']"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "final_df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "VectorAssembler_146745618628"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "assembler = VectorAssembler(inputCols=['Age',\n",
    " 'Total_Purchase',\n",
    " 'Account_Manager',\n",
    " 'Total_Years',\n",
    " 'Num_Sites',\n",
    " 'Year',\n",
    " 'Month',\n",
    " 'Day',\n",
    " 'Weekday',\n",
    " 'Week of Year',\n",
    " 'Day of Month',\n",
    " 'Day of Week',\n",
    " 'Day of Year',\n",
    " 'Hour',\n",
    " 'Minute',\n",
    " 'Second'],outputCol='features')\n",
    "assembler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DataFrame[Age: double, Total_Purchase: double, Account_Manager: int, Total_Years: double, Num_Sites: double, Year: int, Month: int, Day: int, Weekday: int, Week of Year: int, Day of Month: int, Day of Week: int, Day of Year: int, Hour: int, Minute: int, Second: int, Churn: int, features: vector]"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "final_data = assembler.transform(final_df)\n",
    "final_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+-----+\n",
      "|            features|Churn|\n",
      "+--------------------+-----+\n",
      "|[45.0,12309.39,1....|    1|\n",
      "|[38.0,8825.87,1.0...|    1|\n",
      "|[39.0,6351.79,0.0...|    1|\n",
      "|[33.0,10309.71,1....|    0|\n",
      "|[39.0,11490.75,0....|    0|\n",
      "|[44.0,11794.91,0....|    0|\n",
      "|[42.0,9091.99,0.0...|    0|\n",
      "|[41.0,6204.86,1.0...|    0|\n",
      "|[32.0,8617.98,1.0...|    1|\n",
      "|[33.0,12638.51,1....|    0|\n",
      "|[46.0,8603.6,0.0,...|    0|\n",
      "|[37.0,9499.3,1.0,...|    0|\n",
      "|[29.0,11274.46,1....|    0|\n",
      "|[33.0,10306.21,1....|    0|\n",
      "|[60.0,9621.04,1.0...|    0|\n",
      "|[39.0,12019.93,0....|    0|\n",
      "|[53.0,9655.25,0.0...|    0|\n",
      "|[39.0,5846.65,1.0...|    0|\n",
      "|[45.0,11599.27,1....|    1|\n",
      "|[48.0,6495.01,1.0...|    1|\n",
      "+--------------------+-----+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "final_data.select('features','Churn').show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "900"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "final_data.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df, test_df = final_data.randomSplit([0.7,0.3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr = LogisticRegression(featuresCol='features',labelCol='Churn')\n",
    "lr_model = lr.fit(train_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<pyspark.ml.classification.BinaryLogisticRegressionTrainingSummary at 0x21fe60bc190>"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr_model.summary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<pyspark.ml.classification.BinaryLogisticRegressionSummary at 0x21fe5fbc610>"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_results = lr_model.evaluate(test_df)\n",
    "test_results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.915057915057915"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_results.accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.902054794520547"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_results.areaUnderROC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.933920704845815, 0.78125]"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_results.precisionByLabel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.9680365296803652, 0.625]"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_results.recallByLabel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.9506726457399104, 0.6944444444444444]"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_results.fMeasureByLabel()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9150579150579151"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_results.weightedRecall"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9103422176109401"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_results.weightedPrecision"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9111007227599157"
      ]
     },
     "execution_count": 101,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_results.weightedFMeasure()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9150579150579151"
      ]
     },
     "execution_count": 102,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_results.weightedTruePositiveRate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.32202138537754976"
      ]
     },
     "execution_count": 103,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_results.weightedFalsePositiveRate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------------------+-----+\n",
      "|               FPR|  TPR|\n",
      "+------------------+-----+\n",
      "|               0.0|  0.0|\n",
      "|               0.0|0.025|\n",
      "|               0.0| 0.05|\n",
      "|               0.0|0.075|\n",
      "|               0.0|  0.1|\n",
      "|               0.0|0.125|\n",
      "|               0.0| 0.15|\n",
      "|               0.0|0.175|\n",
      "|               0.0|  0.2|\n",
      "|               0.0|0.225|\n",
      "|               0.0| 0.25|\n",
      "|0.0045662100456621| 0.25|\n",
      "|0.0091324200913242| 0.25|\n",
      "|0.0091324200913242|0.275|\n",
      "|0.0091324200913242|  0.3|\n",
      "|0.0091324200913242|0.325|\n",
      "|0.0136986301369863|0.325|\n",
      "|0.0136986301369863| 0.35|\n",
      "|0.0136986301369863|0.375|\n",
      "|0.0182648401826484|0.375|\n",
      "+------------------+-----+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "test_results.roc.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------+------------------+\n",
      "|recall|         precision|\n",
      "+------+------------------+\n",
      "|   0.0|               1.0|\n",
      "| 0.025|               1.0|\n",
      "|  0.05|               1.0|\n",
      "| 0.075|               1.0|\n",
      "|   0.1|               1.0|\n",
      "| 0.125|               1.0|\n",
      "|  0.15|               1.0|\n",
      "| 0.175|               1.0|\n",
      "|   0.2|               1.0|\n",
      "| 0.225|               1.0|\n",
      "|  0.25|               1.0|\n",
      "|  0.25|0.9090909090909091|\n",
      "|  0.25|0.8333333333333334|\n",
      "| 0.275|0.8461538461538461|\n",
      "|   0.3|0.8571428571428571|\n",
      "| 0.325|0.8666666666666667|\n",
      "| 0.325|            0.8125|\n",
      "|  0.35|0.8235294117647058|\n",
      "| 0.375|0.8333333333333334|\n",
      "| 0.375|0.7894736842105263|\n",
      "+------+------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "test_results.pr.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+----+--------------+---------------+-----------+---------+----+-----+---+-------+------------+------------+-----------+-----------+----+------+------+-----+--------------------+--------------------+--------------------+----------+\n",
      "| Age|Total_Purchase|Account_Manager|Total_Years|Num_Sites|Year|Month|Day|Weekday|Week of Year|Day of Month|Day of Week|Day of Year|Hour|Minute|Second|Churn|            features|       rawPrediction|         probability|prediction|\n",
      "+----+--------------+---------------+-----------+---------+----+-----+---+-------+------------+------------+-----------+-----------+----+------+------+-----+--------------------+--------------------+--------------------+----------+\n",
      "|22.0|      11254.38|              1|       4.96|      8.0|2012|    6| 23|      5|          25|          23|          7|        175|   8|    40|    29|    0|[22.0,11254.38,1....|[4.75126142947075...|[0.99143323502473...|       0.0|\n",
      "|25.0|       9672.03|              0|       5.49|      8.0|2006|    1|  2|      0|           1|           2|          2|          2|   4|    16|    13|    0|[25.0,9672.03,0.0...|[4.47948790231666...|[0.98878791779387...|       0.0|\n",
      "|28.0|      11128.95|              1|       5.12|      8.0|2009|    4| 13|      0|          16|          13|          2|        103|  23|    47|    15|    0|[28.0,11128.95,1....|[4.58458607289063...|[0.98989517551766...|       0.0|\n",
      "|28.0|      11204.23|              0|       3.67|     11.0|2012|   10| 25|      3|          43|          25|          5|        299|  20|    54|    44|    0|[28.0,11204.23,0....|[1.61680644162063...|[0.83435422804815...|       0.0|\n",
      "|29.0|       9617.59|              0|       5.49|      8.0|2013|    2| 19|      1|           8|          19|          3|         50|   3|    23|    16|    0|[29.0,9617.59,0.0...|[4.14324192388248...|[0.98437664870392...|       0.0|\n",
      "|30.0|       6744.87|              0|       5.14|      9.0|2014|    4| 10|      3|          15|          10|          5|        100|   3|    33|    41|    0|[30.0,6744.87,0.0...|[3.96690801829183...|[0.98141987689984...|       0.0|\n",
      "|30.0|       8403.78|              1|       4.13|      7.0|2006|    6| 28|      2|          26|          28|          4|        179|   9|    26|    57|    0|[30.0,8403.78,1.0...|[5.91530986774999...|[0.99730943496934...|       0.0|\n",
      "|31.0|       8829.83|              1|       4.52|      8.0|2006|    2|  2|      3|           5|           2|          5|         33|  21|    26|     7|    0|[31.0,8829.83,1.0...|[4.03443864737234...|[0.98261207907808...|       0.0|\n",
      "|31.0|      12264.68|              1|       5.88|      8.0|2014|    1| 14|      1|           3|          14|          3|         14|  14|    18|     4|    0|[31.0,12264.68,1....|[3.39796350191679...|[0.96764082905419...|       0.0|\n",
      "|32.0|       6367.22|              1|       2.84|     10.0|2015|    1|  3|      5|           1|           3|          7|          3|  18|    57|    56|    0|[32.0,6367.22,1.0...|[3.37297970623259...|[0.96684932879156...|       0.0|\n",
      "|32.0|       8575.71|              0|       4.26|      9.0|2006|   11| 15|      2|          46|          15|          4|        319|  14|    58|    19|    0|[32.0,8575.71,0.0...|[3.80225459458006...|[0.97816693100098...|       0.0|\n",
      "|32.0|       9472.72|              1|        1.0|     10.0|2009|    3| 26|      3|          13|          26|          5|         85|   5|    25|    30|    0|[32.0,9472.72,1.0...|[4.22197331295584...|[0.98554241982395...|       0.0|\n",
      "|32.0|       9885.12|              1|       6.92|      9.0|2006|    3|  9|      3|          10|           9|          5|         68|  14|    50|    20|    1|[32.0,9885.12,1.0...|[2.43210167537635...|[0.91924268938256...|       0.0|\n",
      "|32.0|      11715.72|              0|       4.73|      9.0|2010|    1| 11|      0|           2|          11|          2|         11|   4|    57|    14|    0|[32.0,11715.72,0....|[3.59288684107306...|[0.97321822818851...|       0.0|\n",
      "|32.0|      12547.91|              0|       7.78|     10.0|2011|   12| 22|      3|          51|          22|          5|        356|  17|    51|    38|    0|[32.0,12547.91,0....|[0.48873277343920...|[0.61980786074619...|       0.0|\n",
      "|33.0|       5738.82|              0|       7.66|      7.0|2014|    8|  2|      5|          31|           2|          7|        214|   9|     5|    53|    0|[33.0,5738.82,0.0...|[4.37827960954970...|[0.98760854894045...|       0.0|\n",
      "|33.0|       7720.61|              1|       4.97|     10.0|2012|    4| 16|      0|          16|          16|          2|        107|   7|     5|    14|    0|[33.0,7720.61,1.0...|[1.48252989667316...|[0.81495440258023...|       0.0|\n",
      "|33.0|       7750.54|              1|       4.57|      8.0|2012|    7| 19|      3|          29|          19|          5|        201|  13|    34|    46|    0|[33.0,7750.54,1.0...|[4.39559367861171...|[0.98781865774828...|       0.0|\n",
      "|33.0|      10306.21|              1|       6.36|      9.0|2011|   12| 20|      1|          51|          20|          3|        354|  17|     1|     0|    0|[33.0,10306.21,1....|[1.65033521833328...|[0.83893635099184...|       0.0|\n",
      "|33.0|      10709.39|              1|       5.22|      6.0|2006|    2|  2|      3|           5|           2|          5|         33|  12|    29|    48|    0|[33.0,10709.39,1....|[5.94502582600313...|[0.99738800566797...|       0.0|\n",
      "+----+--------------+---------------+-----------+---------+----+-----+---+-------+------------+------------+-----------+-----------+----+------+------+-----+--------------------+--------------------+--------------------+----------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "test_results.predictions.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BinaryClassificationEvaluator_a10aa7b6853a"
      ]
     },
     "execution_count": 110,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluator = BinaryClassificationEvaluator(rawPredictionCol='prediction',labelCol='Churn')\n",
    "evaluator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7965182648401826"
      ]
     },
     "execution_count": 111,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "area_under_roc = evaluator.evaluate(test_results.predictions)\n",
    "area_under_roc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.663723153957529"
      ]
     },
     "execution_count": 112,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluator = BinaryClassificationEvaluator(rawPredictionCol='prediction',labelCol='Churn',metricName='areaUnderPR')\n",
    "area_under_pr = evaluator.evaluate(test_results.predictions)\n",
    "area_under_pr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------+----+--------------+---------------+-----+---------+-------------------+--------------------+----------------+\n",
      "|         Names| Age|Total_Purchase|Account_Manager|Years|Num_Sites|       Onboard_date|            Location|         Company|\n",
      "+--------------+----+--------------+---------------+-----+---------+-------------------+--------------------+----------------+\n",
      "| Andrew Mccall|37.0|       9935.53|              1| 7.71|      8.0|2011-08-29 18:37:54|38612 Johnny Stra...|        King Ltd|\n",
      "|Michele Wright|23.0|       7526.94|              1| 9.28|     15.0|2013-07-22 18:19:54|21083 Nicole Junc...|   Cannon-Benson|\n",
      "|  Jeremy Chang|65.0|         100.0|              1|  1.0|     15.0|2006-12-11 07:48:13|085 Austin Views ...|Barron-Robertson|\n",
      "|Megan Ferguson|32.0|        6487.5|              0|  9.4|     14.0|2016-10-28 05:32:13|922 Wright Branch...|   Sexton-Golden|\n",
      "|  Taylor Young|32.0|      13147.71|              1| 10.0|      8.0|2012-03-20 00:36:46|Unit 0789 Box 073...|        Wood LLC|\n",
      "| Jessica Drake|22.0|       8445.26|              1| 3.46|     14.0|2011-02-04 19:29:27|1148 Tina Straven...|   Parks-Robbins|\n",
      "+--------------+----+--------------+---------------+-----+---------+-------------------+--------------------+----------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "unlabeled_data = spark.read.csv('new_customers.csv',inferSchema=True,header=True)\n",
    "unlabeled_data.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 157,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6"
      ]
     },
     "execution_count": 157,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "unlabeled_data.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- Names: string (nullable = true)\n",
      " |-- Age: double (nullable = true)\n",
      " |-- Total_Purchase: double (nullable = true)\n",
      " |-- Account_Manager: integer (nullable = true)\n",
      " |-- Years: double (nullable = true)\n",
      " |-- Num_Sites: double (nullable = true)\n",
      " |-- Onboard_date: timestamp (nullable = true)\n",
      " |-- Location: string (nullable = true)\n",
      " |-- Company: string (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "unlabeled_data.printSchema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Names',\n",
       " 'Age',\n",
       " 'Total_Purchase',\n",
       " 'Account_Manager',\n",
       " 'Years',\n",
       " 'Num_Sites',\n",
       " 'Onboard_date',\n",
       " 'Location',\n",
       " 'Company',\n",
       " 'Year',\n",
       " 'Month',\n",
       " 'Day',\n",
       " 'Weekday',\n",
       " 'Week of Year',\n",
       " 'Day of Month',\n",
       " 'Day of Week',\n",
       " 'Day of Year',\n",
       " 'Hour',\n",
       " 'Minute',\n",
       " 'Second']"
      ]
     },
     "execution_count": 159,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "unlabeled_data = unlabeled_data.withColumn('Year',year('Onboard_date'))\n",
    "unlabeled_data = unlabeled_data.withColumn('Month',month('Onboard_date'))\n",
    "unlabeled_data = unlabeled_data.withColumn('Day',day('Onboard_date'))\n",
    "unlabeled_data = unlabeled_data.withColumn('Weekday',weekday('Onboard_date'))\n",
    "unlabeled_data = unlabeled_data.withColumn('Week of Year',weekofyear('Onboard_date'))\n",
    "unlabeled_data = unlabeled_data.withColumn('Day of Month',dayofmonth('Onboard_date'))\n",
    "unlabeled_data = unlabeled_data.withColumn('Day of Week',dayofweek('Onboard_date'))\n",
    "unlabeled_data = unlabeled_data.withColumn('Day of Year',dayofyear('Onboard_date'))\n",
    "unlabeled_data = unlabeled_data.withColumn('Hour',hour('Onboard_date'))\n",
    "unlabeled_data = unlabeled_data.withColumn('Minute',minute('Onboard_date'))\n",
    "unlabeled_data = unlabeled_data.withColumn('Second',second('Onboard_date'))\n",
    "unlabeled_data.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 160,
   "metadata": {},
   "outputs": [],
   "source": [
    "unlabeled_data = unlabeled_data.withColumnRenamed(existing='Years',new='Total_Years')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Age',\n",
       " 'Total_Purchase',\n",
       " 'Account_Manager',\n",
       " 'Total_Years',\n",
       " 'Num_Sites',\n",
       " 'Year',\n",
       " 'Month',\n",
       " 'Day',\n",
       " 'Weekday',\n",
       " 'Week of Year',\n",
       " 'Day of Month',\n",
       " 'Day of Week',\n",
       " 'Day of Year',\n",
       " 'Hour',\n",
       " 'Minute',\n",
       " 'Second']"
      ]
     },
     "execution_count": 134,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# unlabeled_data = unlabeled_data.select(['Age',\n",
    "#  'Total_Purchase',\n",
    "#  'Account_Manager',\n",
    "#  'Total_Years',\n",
    "#  'Num_Sites',\n",
    "#  'Year',\n",
    "#  'Month',\n",
    "#  'Day',\n",
    "#  'Weekday',\n",
    "#  'Week of Year',\n",
    "#  'Day of Month',\n",
    "#  'Day of Week',\n",
    "#  'Day of Year',\n",
    "#  'Hour',\n",
    "#  'Minute',\n",
    "#  'Second'])\n",
    "# unlabeled_data.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 161,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- Names: string (nullable = true)\n",
      " |-- Age: double (nullable = true)\n",
      " |-- Total_Purchase: double (nullable = true)\n",
      " |-- Account_Manager: integer (nullable = true)\n",
      " |-- Total_Years: double (nullable = true)\n",
      " |-- Num_Sites: double (nullable = true)\n",
      " |-- Onboard_date: timestamp (nullable = true)\n",
      " |-- Location: string (nullable = true)\n",
      " |-- Company: string (nullable = true)\n",
      " |-- Year: integer (nullable = true)\n",
      " |-- Month: integer (nullable = true)\n",
      " |-- Day: integer (nullable = true)\n",
      " |-- Weekday: integer (nullable = true)\n",
      " |-- Week of Year: integer (nullable = true)\n",
      " |-- Day of Month: integer (nullable = true)\n",
      " |-- Day of Week: integer (nullable = true)\n",
      " |-- Day of Year: integer (nullable = true)\n",
      " |-- Hour: integer (nullable = true)\n",
      " |-- Minute: integer (nullable = true)\n",
      " |-- Second: integer (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "unlabeled_data.printSchema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 162,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------+----+--------------+---------------+-----------+---------+-------------------+--------------------+----------------+----+-----+---+-------+------------+------------+-----------+-----------+----+------+------+--------------------+\n",
      "|         Names| Age|Total_Purchase|Account_Manager|Total_Years|Num_Sites|       Onboard_date|            Location|         Company|Year|Month|Day|Weekday|Week of Year|Day of Month|Day of Week|Day of Year|Hour|Minute|Second|            features|\n",
      "+--------------+----+--------------+---------------+-----------+---------+-------------------+--------------------+----------------+----+-----+---+-------+------------+------------+-----------+-----------+----+------+------+--------------------+\n",
      "| Andrew Mccall|37.0|       9935.53|              1|       7.71|      8.0|2011-08-29 18:37:54|38612 Johnny Stra...|        King Ltd|2011|    8| 29|      0|          35|          29|          2|        241|  18|    37|    54|[37.0,9935.53,1.0...|\n",
      "|Michele Wright|23.0|       7526.94|              1|       9.28|     15.0|2013-07-22 18:19:54|21083 Nicole Junc...|   Cannon-Benson|2013|    7| 22|      0|          30|          22|          2|        203|  18|    19|    54|[23.0,7526.94,1.0...|\n",
      "|  Jeremy Chang|65.0|         100.0|              1|        1.0|     15.0|2006-12-11 07:48:13|085 Austin Views ...|Barron-Robertson|2006|   12| 11|      0|          50|          11|          2|        345|   7|    48|    13|[65.0,100.0,1.0,1...|\n",
      "|Megan Ferguson|32.0|        6487.5|              0|        9.4|     14.0|2016-10-28 05:32:13|922 Wright Branch...|   Sexton-Golden|2016|   10| 28|      4|          43|          28|          6|        302|   5|    32|    13|[32.0,6487.5,0.0,...|\n",
      "|  Taylor Young|32.0|      13147.71|              1|       10.0|      8.0|2012-03-20 00:36:46|Unit 0789 Box 073...|        Wood LLC|2012|    3| 20|      1|          12|          20|          3|         80|   0|    36|    46|[32.0,13147.71,1....|\n",
      "| Jessica Drake|22.0|       8445.26|              1|       3.46|     14.0|2011-02-04 19:29:27|1148 Tina Straven...|   Parks-Robbins|2011|    2|  4|      4|           5|           4|          6|         35|  19|    29|    27|[22.0,8445.26,1.0...|\n",
      "+--------------+----+--------------+---------------+-----------+---------+-------------------+--------------------+----------------+----+-----+---+-------+------------+------------+-----------+-----------+----+------+------+--------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "final_unlabeled_data = assembler.transform(unlabeled_data)\n",
    "final_unlabeled_data.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 163,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DataFrame[Names: string, Age: double, Total_Purchase: double, Account_Manager: int, Total_Years: double, Num_Sites: double, Onboard_date: timestamp, Location: string, Company: string, Year: int, Month: int, Day: int, Weekday: int, Week of Year: int, Day of Month: int, Day of Week: int, Day of Year: int, Hour: int, Minute: int, Second: int, features: vector, rawPrediction: vector, probability: vector, prediction: double]"
      ]
     },
     "execution_count": 163,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_predictions = lr_model.transform(final_unlabeled_data)\n",
    "test_predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 164,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------+----+--------------+---------------+-----------+---------+-------------------+--------------------+----------------+----+-----+---+-------+------------+------------+-----------+-----------+----+------+------+--------------------+--------------------+--------------------+----------+\n",
      "|         Names| Age|Total_Purchase|Account_Manager|Total_Years|Num_Sites|       Onboard_date|            Location|         Company|Year|Month|Day|Weekday|Week of Year|Day of Month|Day of Week|Day of Year|Hour|Minute|Second|            features|       rawPrediction|         probability|prediction|\n",
      "+--------------+----+--------------+---------------+-----------+---------+-------------------+--------------------+----------------+----+-----+---+-------+------------+------------+-----------+-----------+----+------+------+--------------------+--------------------+--------------------+----------+\n",
      "| Andrew Mccall|37.0|       9935.53|              1|       7.71|      8.0|2011-08-29 18:37:54|38612 Johnny Stra...|        King Ltd|2011|    8| 29|      0|          35|          29|          2|        241|  18|    37|    54|[37.0,9935.53,1.0...|[2.26304667766878...|[0.90576998883679...|       0.0|\n",
      "|Michele Wright|23.0|       7526.94|              1|       9.28|     15.0|2013-07-22 18:19:54|21083 Nicole Junc...|   Cannon-Benson|2013|    7| 22|      0|          30|          22|          2|        203|  18|    19|    54|[23.0,7526.94,1.0...|[-5.8626752975869...|[0.00283556267212...|       1.0|\n",
      "|  Jeremy Chang|65.0|         100.0|              1|        1.0|     15.0|2006-12-11 07:48:13|085 Austin Views ...|Barron-Robertson|2006|   12| 11|      0|          50|          11|          2|        345|   7|    48|    13|[65.0,100.0,1.0,1...|[-3.6953955380633...|[0.02423567092575...|       1.0|\n",
      "|Megan Ferguson|32.0|        6487.5|              0|        9.4|     14.0|2016-10-28 05:32:13|922 Wright Branch...|   Sexton-Golden|2016|   10| 28|      4|          43|          28|          6|        302|   5|    32|    13|[32.0,6487.5,0.0,...|[-5.1635736931872...|[0.00568867111021...|       1.0|\n",
      "|  Taylor Young|32.0|      13147.71|              1|       10.0|      8.0|2012-03-20 00:36:46|Unit 0789 Box 073...|        Wood LLC|2012|    3| 20|      1|          12|          20|          3|         80|   0|    36|    46|[32.0,13147.71,1....|[1.51737674357774...|[0.82015186788852...|       0.0|\n",
      "| Jessica Drake|22.0|       8445.26|              1|       3.46|     14.0|2011-02-04 19:29:27|1148 Tina Straven...|   Parks-Robbins|2011|    2|  4|      4|           5|           4|          6|         35|  19|    29|    27|[22.0,8445.26,1.0...|[-1.8603237696495...|[0.13466531842712...|       1.0|\n",
      "+--------------+----+--------------+---------------+-----------+---------+-------------------+--------------------+----------------+----+-----+---+-------+------------+------------+-----------+-----------+----+------+------+--------------------+--------------------+--------------------+----------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "test_predictions.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 165,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------+----------------+----------+\n",
      "|         Names|         Company|prediction|\n",
      "+--------------+----------------+----------+\n",
      "| Andrew Mccall|        King Ltd|       0.0|\n",
      "|Michele Wright|   Cannon-Benson|       1.0|\n",
      "|  Jeremy Chang|Barron-Robertson|       1.0|\n",
      "|Megan Ferguson|   Sexton-Golden|       1.0|\n",
      "|  Taylor Young|        Wood LLC|       0.0|\n",
      "| Jessica Drake|   Parks-Robbins|       1.0|\n",
      "+--------------+----------------+----------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "test_predictions.select(['Names','Company','prediction']).show()"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}