Machine Learning For Absolute Beginners: A Plain English Introduction

minicourse

2 WHAT IS MACHINE LEARNING?

A key characteristic of machine learning is the concept of self-learning. This refers to the application of statistical modeling to detect patterns and improve performance based on data and empirical information; all without direct programming commands.

By decoding complex patterns in the input data, the model uses machine learning to find connections without human help.

Another distinct feature of machine learning is the ability to improve predictions based on experience.

• Spam detection system is lured into producing a false-positive based on previous input data.
• Traditional programming is highly susceptible to this problem because the model is rigidly defined according to pre-set rules.

Machine learning, on the other hand, emphasizes exposure to data as a way to refine the model, adjust weak assumptions, and respond appropriately to unique data points

Training & Test Data

#training_data the initial reserve of data used to develop the model.

After you have developed a model based on patterns extracted from the training data and you are satisfied with the accuracy of its predictions, you can test the model on the remaining data, known as the test data.

The Anatomy of Machine Learning

Machine learning, data mining, artificial intelligence, and computer programming all fall under the umbrella of computer science,

• Figure 4: Visual representation of the relationship between data-related fields

3 MACHINE LEARNING CATEGORIES

Supervised Learning

#Supervised_learning imitates our own ability to extract patterns from known examples and use that extracted insight to engineer a repeatable outcome.

• Input data is referred to as the #independent_variable (uppercase X),
• Output data is called the #dependent_variable (lowercase y).

| —— | Input– | Input— | Input- | Output- | | —— | ——- | ——– | —— | ——- | | | Brand | Mileage | Year | Price | | Car 1 | Lexus | 51715 | 2012 | 15985 | | Car 2 | Lexus | 7980 | 2013 | 19600 | | Car 3 | Lexus | 82497 | 2012 | 14095 | | Car 4 | Lexus | 85199 | 2011 | 12490 | | Car 5 | Audi | 62948 | 2008 | 13985 |

• Table 2: Extract of a used car dataset

With access to the selling price of other similar cars, the supervised learning model can work backward to determine the relationship between a car’s value (output) and its characteristics (input). The input features of your own car can then be inputted into the model to generate a price prediction.

• Figure 5: Inputs (X) are fed to the model to generate a new prediction (y)

When building a supervised learning model, each item (i.e. car, product, customer) must have labeled input and output values—known in data science as a #labeled_dataset

• Figure 6: Labeled data vs. unlabeled data

#Common_algorithms used in supervised learning include:

• regression analysis (i.e. linear regression, logistic regression, non-linear regression),
• decision trees,
• k-nearest neighbors,
• neural networks,
• support vector machines,

Unsupervised Learning

#Unsupervised_learning: the output variables are unlabeled, and combinations of input and output variables aren’t known.

Unsupervised learning instead focuses on analyzing relationships between input variables and uncovering hidden patterns that can be extracted to create new labels regarding possible outputs.

Unsupervised learning is especially compelling in the domain of fraud detection—where the most dangerous attacks are those yet to be classified.

One interesting example is DataVisor; a company that has built its business model on unsupervised learning. Founded in 2013 in California, DataVisor protects customers from fraudulent online activities, including spam, fake reviews, fake app installs, and fraudulent transactions.

• traditional solutions analyze chains of activity for a specific type of attack and then create rules to predict and detect repeat attacks. In this case, the dependent variable (output) is the event of an attack, and the independent variables (input) are the common predictor variables of an attack.
• a model that monitors combinations of independent variables, such as a large purchasing order from the other side of the globe or a landslide number of book reviews that reuse existing user content generally leads to a better prediction.
• In supervised learning, the model deconstructs and classifies what these common variables are and designs a detection system to identify and prevent repeat offenses.
• Sophisticated cybercriminals, though, learn to evade these simple classification-based rule engines by modifying their tactics.
• leverage unsupervised learning techniques to address these limitations.

The drawback, though, of using unsupervised learning is that because the dataset is unlabeled, there aren’t any known output observations to check and validate the model, and predictions are therefore more subjective than those coming from supervised learning.

Semi-supervised Learning

Used for datasets that contain a mix of labeled and unlabeled cases.

One technique is to build the initial model using the labeled cases (supervised learning) and then use the same model to label the remaining cases (that are unlabeled) in the dataset.

Reinforcement Learning

The goal of reinforcement learning is to achieve a specific goal (output) by randomly trialing a vast number of possible input combinations and grading their performance.

Q-learning

A specific algorithmic example of reinforcement learning

• you start with a set environment of states, represented as “S.”
• In the game Pac-Man, states could be the challenges, obstacles, or pathways
• The set of possible actions to respond to these states is referred to as “A.” In Pac-Man, actions are limited to left, right, up, and down movements, as well as multiple combinations thereof.
• The third important symbol is “Q,” which is the model’s starting value and has an initial value of “0.”
• As Pac-Man explores the space inside the game, two main things happen:
  1. Q drops as negative things occur after a given state/action.
  2. Q increases as positive things occur after a given state/action.

A more comprehensive explanation of reinforcement learning and Q-learning using the Pac-Man case study

4 THE MACHINE LEARNING TOOLBOX

Standard Toolbox

Compartment 1: Data

As a beginner, it’s best to start with (analyzing) structured data. This means that the data is defined, organized, and labeled in a table, as shown in Table 3.

Images, videos, email messages, and audio recordings are examples of unstructured data as they don’t fit into the organized structure of rows and columns.

Contained in each column is a feature. A #feature is also known as a variable, a dimension or an attribute

• Rows are sometimes referred to as a case or value
• Each column is known also as a vector.

• Figure 8: The y value is often but not always expressed in the far-right vector

Scatterplots, including 2-D, 3-D, and 4-D plots, are also packed into the first compartment of the toolbox with the data.

Compartment 2: Infrastructure

Platforms and tools for processing data: Jupyter Notebook, machine learning libraries, including NumPy, Pandas, and Scikit-learn, server. In addition, you may need specialized libraries for data visualization such as Seaborn and Matplotlib, or a standalone software program like Tableau,

Compartment 3: Algorithms

You can find hundreds of interesting datasets in CSV format from kaggle.com

Beginners typically start out using simple supervised learning algorithms such as linear regression, logistic regression, decision trees, and k-nearest neighbors. Beginners are also likely to apply unsupervised learning in the form of k-means clustering and descending dimension algorithms.

Visualization

The visual story conveyed through graphs, scatterplots, heatmaps, box plots, and the representation of numbers as shapes make for quick and easy storytelling.

The Advanced Toolbox

Beginners work with small datasets that are easy to handle and downloaded directly to one’s desktop as a simple CSV file. Advanced users, though, will be eager to tackle massive datasets, well in the vicinity of big data.

Compartment 1: Big Data

Big data is also less likely to fit into standard rows and columns and may contain numerous data types, such as structured data and a range of unstructured data, i.e. images, videos, email messages, and audio files.

Compartment 2: Infrastructure

In 2009, Andrew Ng and a team at Stanford University made a discovery to link inexpensive GPU clusters to run neural networks consisting of hundreds of millions of connected nodes.

TensorFlow is only compatible with the Nvidia GPU card.

Compartment 3: Advanced Algorithms

While Scikit-learn offers a range of popular shallow algorithms, TensorFlow is the machine learning library of choice for deep learning/neural networks.

Written in Python, Keras is an open-source deep learning library that runs on top of TensorFlow, Theano, and other frameworks, which allows users to perform fast experimentation in fewer lines of code.

It is, however, less flexible in comparison to TensorFlow and other libraries.

Developers, therefore, will sometimes utilize Keras to validate their decision model before switching to TensorFlow to build a more customized model.

5 DATA SCRUBBING

Datasets need upfront cleaning and human manipulation before they’re ready for consumption.

Feature Selection

#feature_selection identify which variables are most relevant to your hypothesis or objective.

• Table 4: Endangered languages, database: https://www.kaggle.com/the-guardian/extinct-languages

Let’s say our goal is to identify variables that contribute to a language becoming endangered.

• Based on the purpose of our analysis, it’s unlikely that a language’s “Name in Spanish” will lead to any relevant insight. We can therefore delete this vector (column) from the dataset.
• Secondly, the dataset contains duplicated information in the form of separate vectors for “Countries” and “Country Code.”
• Another method to reduce the number of features is to roll multiple features into one,

• Table 5: Sample product inventory

For instance, we can remove individual product names and replace the eight product items with fewer categories or subtypes.

• Table 6: Synthesized product inventory

The downside to this transformation is that we have less information about the relationships between specific products.

Row Compression

In addition to feature selection, you may need to reduce the number of rows

• Table 7: Example of row merge

Non-numeric and categorical row values can be problematic to merge while preserving the true value of the original data. Also, row compression is usually less attainable than feature compression and especially for datasets with a high number of features.

One-hot Encoding

You next want to look for text-based values that can be converted into numbers.

Most algorithms are not compatible with non-numeric data.

#one-hot_encoding: transforms values into binary form

• Table 8: Endangered languages

• Table 9: Example of one-hot encoding

Using one-hot encoding, the dataset has expanded to five columns, and we have created three new features from the original feature

Binning

(also called #bucketing)

#binning: used for converting continuous numeric values into multiple binary features called bins or buckets according to their range of values.

Let’s take house price evaluation as an example. The exact measurements of a tennis court might not matter much when evaluating house property prices;

• numeric measurements of the tennis court with a True/False feature or a categorical value such as “small,” “medium,” and “large.”
• Another alternative would be to apply one-hot encoding with “0” for homes that do not have a tennis court

Normalization

#normalization and standardization help to improve model accuracy when used with the right algorithm.

The former (normalization) rescales the range of values for a given feature into a set range with a prescribed minimum and maximum, such as [0, 1] or [−1, 1].

This technique helps to normalize the variance among the dataset’s features which may otherwise be exaggerated by another factor.

Normalization, however, usually isn’t recommended for rescaling features with an extreme range as the normalized range is too narrow to emphasize extremely high or low feature values.

Standardization

#standardization: This technique converts unit variance to a standard normal distribution with a mean of zero and a standard deviation (σ) of one.

• Figure 11: Examples of rescaled data using normalization and standardization

Standardization is generally more effective than normalization when the variability of the feature reflects a bell-curve shape of normal distribution and is often used in unsupervised learning.

Standardization is generally recommended when preparing data for support vector machines (SVM), principal component analysis (PCA), and k-nearest neighbors (k-NN).

Missing Data

Missing values in your dataset can be equally frustrating and interfere with your analysis and the model’s predictions. There are, however, strategies to minimize the negative impact of missing data.

One approach is to approximate missing values using the mode value. This works best with categorical and binary variable types, such as one to five-star rating systems and positive/negative drug tests respectively.

• Figure 12: A visual example of the mode and median respectively

The second approach is to approximate missing values using the median value. This works best with continuous variables, which have an infinite number of possible values, such as house prices.

As a last resort, rows with missing values can be removed altogether. The obvious downside to this approach is having less data to analyze and potentially less comprehensive insight.

6 SETTING UP YOUR DATA

After cleaning your dataset, the next job is to split the data into two segments for training and testing, also known as #split_validation.

Usually 70/30 or 80/20.

• training data should account for 70 percent to 80 percent
• 20 percent to 30 percent of rows are left for your test data.

Before you split your data, it’s essential that you randomize the row order. This helps to avoid bias in your model.

After randomizing the data, you can begin to design your model and apply it to the training data.

The next step is to measure how well the model performed.

Four examples of performance metrics used with classification tasks

• Area under the curve (AUC)
• Receiver Operating Characteristic (ROC)[16],
• Confusion matrix,
• recall, and accuracy

Mean absolute error and root mean square error (RMSE) are commonly used to assess models that provide a numeric output

Using Scikit-learn, mean absolute error is found by inputting the X values from the training data into the model and generating a prediction for each row in the dataset.

You’ll know that the model is accurate when the error rate for the training and test dataset is low, which means the model has learned the dataset’s underlying trends and patterns.

Cross Validation

While split validation can be effective for developing models using existing data, question marks naturally arise over whether the model can remain accurate when used on new data.

Rather than split the data into two segments (one for training and one for testing), you can implement what’s called cross validation.

#Cross_validation maximizes the availability of training data by splitting data into various combinations and testing each specific combination.

• The first method is exhaustive cross validation, which involves finding and testing all possible combinations
• The alternative and more common method is non-exhaustive cross validation, known as #k-fold_validation.

K-Fold Validation

Figure 14: k-fold validation

This method, though, is slower because the training process is multiplied by the number of validation sets.

How Much Data Do I Need?

Machine learning works best when your training dataset includes a full range of feature combinations.

At an absolute minimum, a basic machine learning model should contain ten times as many data points as the total number of features.

There is a natural diminishing rate of return after an adequate volume of training data (that’s widely representative of the problem) has been reached.

• For datasets with less than 10,000 samples, clustering and dimensionality reduction algorithms can be highly effective,
• whereas regression analysis and classification algorithms are more suitable for datasets with less than 100,000 samples.

Neural networks require even more samples to run effectively and are more cost-effective and time-efficient for working with massive quantities of data.

Scikit-learn has a cheat sheet for matching algorithms to different datasets at http://scikit-learn.org/stable/tutorial/machine_learning_map/.

7 LINEAR REGRESSION

As the “Hello World” of supervised learning algorithms, regression analysis is a simple technique for predicting an unknown variable using the results you do know.

Using the Seinfeld TV sitcom series as our data, let’s start by plotting the two following variables, with season number as the x coordinate and the number of viewers per season (in millions) as the y coordinate.

• Table 11: Seinfeld dataset

The goal of linear regression is to split the data in a way that minimizes the distance between the hyperplane and the observed values.

• Figure 17: Error is the distance between the hyperplane and the observed value

The Slope

As one variable increases, the other variable will increase by the average value denoted by the hyperplane.

• Figure 18: Using the slope/hyperplane to make a prediction

Linear Regression Formula

$y = bx + a$

• $a$ point where the hyperplane crosses the y-axis, known as the y-intercept
• $b$ dictates the steepness of the slope and explains the relationship between x and y

Calculation Example

• Table 12: Sample dataset

\(\begin{align*} a &= \frac {(\sum y) (\sum x^2) - (\sum y) (\sum xy)} {n(\sum x^2) - (\sum x)^2} \\ \\ b &= \frac {n(\sum xy) - (\sum x) (\sum y)} {n(\sum x^2) - (\sum x)^2} \end{align*}\)  Where:  - $\sum x$ = Total sum of all x values (1 + 2 + 1 + 4 + 3 = 11)  - $\sum y$ = Total sum of all y values (3 + 4 + 2 + 7 + 5 = 21)  - $\sum xy$ = Total sum of xy for each row (3 + 8 + 2 + 28 + 15 = 56)  - $\sum x^2$ = Total sum of xx for each row (1 + 4 + 1 + 16 + 9 = 31)  - $n$ = Total number of rows. In the case of this example, n is equal to 5.

$y_i = b(x_i) + a = 1.441 (x_i) + 1.029$

• Figure 19: y = 1.441x + 1.029 plotted on the scatterplot

Multiple Linear Regression

The y-intercept is still expressed as a, but now there are multiple independent variables (represented as x1, x2, x3, etc.) each with their own respective coefficient (b1, b2, b3, etc).

Discrete Variables

The output (dependent variable) of linear regression must be continuous in the form of a floating-point or integer

The input (independent variables) can be continuous or categorical.

For categorical variables, i.e. gender, these variables must be expressed numerically using one-hot encoding

Variable Selection

On the one hand, adding more variables helps to account for more potential factors that control patterns in the data.

On the other hand, this rationale only holds if the variables are relevant and possess some correlation/linear relationship with the dependent variable.

In multiple linear regression, not only are the independent variables potentially related to the dependent variable, but they are also potentially related to each other.

• Figure 20: Simple linear regression (above) and multiple linear regression (below)

If a strong linear correlation exists between two independent variables, this can lead to a problem called #multi-collinearity.

Multi-Collinearity

When two independent variables are strongly correlated, they have a tendency to cancel each other out and provide the model with little to no unique information.

Example of two multi-collinear variables are liters of fuel consumed and liters of fuel in the tank to predict how far a jet plane will fly. In this case negatively correlated; as one variables increases, the other variable decreases and vice versa.

To avoid multi-collinearity, we need to check the relationship between each combination of independent variables using a scatterplot, pairplot (a matrix of relationships between variables), or correlation score.

• Figure 21: Pairplot with three variables

• Figure 22: Heatmap with three variables

We can also use a pairplot, heatmap or correlation score to check if the independent variables are correlated to the dependent variable (and therefore relevant to the prediction outcome).

CHAPTER QUIZ

Using multiple linear regression, your task is to create a model to predict the tip amount guests will leave the restaurant when paying for their meal.

1. The dependent variable for this model should be which variable?
   • A size
   • B total_bill and tip
   • C total_bill
   • D tip
2. From looking only at the data preview above, which variable(s) appear to have a linear relationship with total_bill?
   • A smoker
   • B total_bill and size
   • C time
   • D sex
3. It’s important for the independent variables to be strongly correlated with the dependent variable and one or more of the other independent variables.
   • True or False?

8 LOGISTIC REGRESSION

Linear regression is useful for quantifying relationships between variables to predict a continuous outcome. Total bill and size (number of guests) are both examples of continuous variables.

However, what if we want to predict a categorical variable such as “new customer” or “returning customer”? Unlike linear regression, the dependent variable (y) is no longer a continuous variable (such as total tip) but rather a discrete categorical variable.

Logistic regression is still a supervised learning technique but produces a qualitative prediction rather than a quantitative prediction. This algorithm is often used to predict two discrete classes, e.g., pregnant or not pregnant.

Using the sigmoid function, logistic regression finds the probability of independent variables (X) producing a discrete dependent variable (y) such as “spam” or “non-spam.”

\(\begin{align*} y = \frac {1}{1 + e^{-x}} \end{align*}\) Where:

• $x$ = the independent variable you wish to transform
• $e$ = Euler’s constant, 2.718

• Figure 23: A sigmoid function used to classify data points

Sigmoid Function

The #sigmoid_function produces an S-shaped curve that can convert any number and map it into a numerical value between 0 and 1 but without ever reaching those exact limits. Applying this formula, the sigmoid function converts independent variables into an expression of probability between 0 and 1 in relation to the dependent variable.

Based on the found probabilities of the independent variables, logistic regression assigns each data point to a discrete class.

• Figure 24: An example of logistic regression

Although logistic regression shares a visual resemblance to linear regression, the logistic hyperplane represents a classification/decision boundary rather than a prediction trendline. Instead of using the hyperplane to make numeric predictions, the hyperplane is used to divide the dataset into classes.

The other distinction between logistic and linear regression is that the dependent variable (y) isn’t placed along the y-axis in logistic regression. Instead, independent variables can be plotted along both axes, and the class (output) of the dependent variable is determined by the position of the data point in relation to the decision boundary.

Multinomial Logistic Regression

For classification scenarios with more than two possible discrete outcomes, multinomial logistic regression can be used

Figure 25: An example of multinomial logistic regression

Two tips to remember when using logistic regression are that

1. the dataset should be free of missing values
2. that all independent variables are independent and not strongly correlated with each other.

Statistics 101: Logistic Regression, An Introduction - YouTube

CHAPTER QUIZ

Using logistic regression, your task is to classify penguins into different classes based on the following dataset.

1. Which three variables (in their current form) could we use as the dependent variable to classify penguins?  

1. Which row(s) contains missing values?  
2. Which variable in the dataset preview is binary?

9 k-NEAREST NEIGHBORS

#k-NN classifies new data points based on their position to nearby data points. Similar to a voting system

• Figure 26: An example of k-NN clustering used to predict the class of a new data point
• set $k$ to determine how many data points we want to use to classify the new data
• If we set k to 3, k-NN analyzes the new data point’s position with respect to the three nearest data points (neighbors).
• The outcome of selecting the three closest neighbors returns two Class B data points and one Class A data point.
• It’s therefore useful to test numerous k combinations to find the best fit and avoid setting k too low or too high.
• Setting k too low will increase bias and lead to misclassification
• setting k too high will make it computationally expensive.

Five is the default number of neighbors for this algorithm using Scikit-learn.

This algorithm works best with continuous variables.

While k-NN is generally accurate and easy to comprehend, storing an entire dataset and calculating the distance between each new data point and all existing data points puts a heavy burden on computing resources. NN is generally not recommended for analyzing large datasets.

Another downside is that it can be challenging to apply k-NN to high-dimensional data with a high number of features.

CHAPTER QUIZ

Classify penguins into different species using the k-nearest neighbors algorithm, with k set to 5 (neighbors).

1. Which of the following variables should we consider removing from our k-NN model?
   • A. sex
   • B. species
   • C. body_mass_g
   • D. bill_depth_mm
2. If we wanted to reduce the processing time of our model, which of the following methods is recommended?
   • A. Increase k from 5 to 10
   • B. Reduce k from 10 to 5
   • C. Re-run the model and hope for a faster result
   • D. Increase the size of the training data
3. To include the variable ‘sex’ in our model, which data scrubbing technique do we need to use?