Image from Source
Deep Learning has introduced several technological revolutions and automation in the present world scenario. One of the most life-changing contributions of Deep Learning is its use in healthcare. In this tutorial, we will look into one of the amazing applications of Deep Learning in healthcare for diagnosing hypothyroid disease. We will be going through the code and workflow on how to diagnose hypothyroidism using neural networks, i.e., whether or not a person is suffering from Hypothyroid disease based on various metrics as described in this GitHub project.
Hypothyroid disease or Hypothyroidism is one of the most difficult diseases to diagnose. It is a condition that causes your thyroid gland to not produce enough of certain crucial hormones. It usually may not cause noticeable symptoms in the early stages and hence becomes extremely difficult to diagnose. Applying deep learning algorithms can help assist healthcare workers in identifying signs and symptoms based on patterns shown in past cases of patients’ data.
In this tutorial, we will go through the process of using a deep neural network to classify whether or not a person is suffering from Hypothyroidism. The concept illustrated in this tutorial is primarily aimed at diagnosing hypothyroidism, however, if you wish to, you can use this concept to diagnose almost any disease you want based on the availability of data.
Deep Learning: It is a kind of machine learning technique that enables learning through the use of neural networks that mimic the human brain.
Hypothyroidism: also known as an underactive thyroid is a condition when a person’s body does not produce enough thyroid hormones.
How can Deep Learning help diagnose hypothyroid disease?
The thyroid is a small butterfly-shaped gland in humans located in front of the windpipe and below the larynx. The thyroid gland is responsible for absorbing the iodine in food and generates two hormones: T3 and T4. The presence of these hormones is essential for the normal growth of the brain and neural system, especially during the first three years of life, i.e., in infants. Not having enough of these hormones is diagnosed as hypothyroidism, which is a class of thyroid disease that might even cause mental retardation in infants.
Thyroid diseases are said to be more common among women than men. Although various diagnosis methods such as Elisa, Electrochemiluminescence, etc., have been proposed for conducting hormone tests for diagnosing the type of thyroid disease, it still remains a difficult disease to diagnose.
Deep Learning tools and techniques can be effectively used in assisting healthcare experts in reducing human errors by predicting the possibility of the disease with high accuracy. We will be using neural networks, a class of deep learning algorithms that are proficient in accurately modeling the data to find hidden patterns and factors that affect the prediction or the diagnosis.
Classifying hypothyroidism using a Deep Learning model
We will now look into building a Deep Learning model to predict (diagnose) if a patient is suffering from hypothyroidism or not, based on several factors provided to it.
All the steps that we are going to discuss hereafter are available as a notebook here.
Step 1: Getting the data
For the training data, we are using the hypothyroid data from here. The dataset contains the value of different hormone levels of different patients along with our target variable, i.e., if the patient was suffering from hypothyroidism or not. To start using it, you can download the dataset and save it into a folder named “data”.
Upon reading in our data, it looks like this.
Step 2: Performing Data Engineering
After getting the data, cleaning, and formatting it is essential to avoid potential faults in our results due to inconsistencies or errors in our data.
We will now look into cleaning the data.
The value ‘hypothyroid’ means that the patient was diagnosed with hypothyroidism and ‘‘negative” means that their hypothyroidism was not diagnosed. Also, the target value in the dataset looks imbalanced as the number of negative samples is much higher than the positive ones.
Converting categorical columns into binary
We will be mapping the value ‘‘hypothyroid” as ‘1′ and ‘‘negative” as ‘0′.
We will then look into the value counts of some other columns in the dataset and replace the categorical values with binary values.
Hence, after binary conversion, the head of the dataset looks like this.
Now, some of the features contain ‘?’ values. So we will be replacing them with NaN values and count them.
Here, we can see that the feature column “TBG” contains an extremely high number of null values. So, we will be not be using this column for our model.
Converting columns into numeric values
This will help us in performing mathematical and statistical operations on the columns.
Step 3: Exploring the data
Next, we will be looking into the relationships between different features present in our dataset by visualizing them. We will visualize the patterns in different columns (features) of the data using the
hist() function of the Pandas library.
Replacing null values in our data
We will now replace or impute the null values in our data with their appropriate statistical metric. The columns containing null values are
['Age', 'Gender', 'TSH', 'T3', 'TT4', 'T4U', 'FTI']
Based on the statistics plotted above, we can see that,
- The features “Age” and “T4U” show a normal distribution, which is why we will be replacing the missing values with mean.
- The features “TSH”, “T3”, “TT4” and “FTI” show a skewed distribution, which is why we will be replacing the missing values with the median.
- The feature “gender” is binary and contains imbalanced data with the values “1” much greater than “0”. So, we will be replacing the missing values with “0”.
Let us now visualize the imputed data.
Hence, we’ve successfully filled in all the missing values in our dataset. Our data is now cleaned and formatted and we are now ready to model the data for prediction using Deep Learning.
Step 4: Modeling the data
For modeling the data, we will be implementing a deep neural network using Keras. We will be using the Neural network for predicting the hypothyroidism (target) based on our input data (features).
1. Defining the features and target variables
We will be separating our dataset into features (X) and the target variable (y).
2. Splitting the data into the train (80%) and test (20%) set
In order to efficiently evaluate our model, we will be training our model on 80% of the data, and keep the rest 20% for testing the model.
3. Scaling the train and test data
Scaling the data helps in normalizing the data within a particular range and also helps in speeding up the calculations in an algorithm. For scaling the data, we will be using the
StandardScaler class from scikit-learn.
4. Building the neural network
The scope of our project is to design a feed-forward neural network for the classiﬁcation of our target variable. Several factors determine the performance of the model such as the input, hidden and output layer conﬁgurations, and the training methodology used. Most of the neural network architectures are determined by experimentation in practice. Here, we are defining a neural network containing a sequential model with one hidden layer and one output layer.
We have 24 feature columns due to which we have specified the input dimension as input_dim=24. Here, we’ve implemented the ReLU activation function in the hidden layer, and sigmoid function for the output layer to return a binary value of output, i.e. hypothyroid (1) or negative (0).
We are then training the model with a validation split of 0.2 for 100 epochs.
Step 5: Evaluating the model
We will now evaluate the performance of our trained model on test data to find metrics such as Accuracy, F1-score, recall, and precision.
We are getting a quite good accuracy of 98.10% from our Deep Learning model. However, our data is imbalanced and hence accuracy is not a good measure. Hence, we are using a metric called F1-score.
We are getting a quite good value of F1-score, recall, and precision as well, and hence, our model seems to be performing well.
Training and validation loss
We can see that both the losses have converged at a very good rate.
Step 6: Saving the model
Finally, we will be saving the trained model for future use. It can be easily loaded and used for predictions.
Learning tools and strategies
- Deep Learning is a type of machine learning algorithm that uses neural networks for performing its predictions. The key to learning about neural networks effectively is to learn and visualize the whole architecture of the system. By doing this, we can easily understand how the data is being processed step-by-step.
- Also, it is a good practice to print or log important messages and errors to help with debugging.
- Neural networks are very sensitive to hyperparameters. Therefore, it’s very important to tune them precisely so as to increase the model’s accuracy and improve its performance.
Overall, this project was an interesting take on leveraging the use of Artificial Intelligence in healthcare. With the growing use of technology, its scope has as well grown ten-folds. Additionally, this project covers all the different processes from start to end of a deep learning project. Also, instead of a general machine learning algorithm, I opted to use an artificial neural network as they can be easily modified and works well with any kind of data which makes them so versatile to use and create despite their complex architectures.
Conclusions and Future Directions
In this project, we have illustrated the potential use of neural networks in diagnosing hypothyroid disease. The identification of thyroid disease identiﬁcation remains to be an essential yet difﬁcult task in both clinical diagnosis and statistical classification. The diagnosis involves the use of a large number of interrelated patient attributes along with extremely unbalanced groups that result in a complicated relationship between the input features.
Artiﬁcial neural networks show a promising result due to its flexible nature in modeling complex patterns of data for diagnosis. The architecture of the neural network implemented for this project was fairly simple for giving an overview. However, we can implement more complex architectures of neural networks with different hyperparameters comprised of several hidden layers and number of computations. Moreover, this knowledge can be further extended for many other disease diagnoses where such problems exist.
- The dataset for training our model was derived from here.
Also, the code for this project on Diagnosing hypothyroid disease using Deep Learning is available on GitHub.
If you’re looking to do similar innovative projects in Deep Learning, you might be interested in this project on How to generate unique architectures using GANs.