"""
 <b><p style="text-align:center;"><font size ="6" color ="Black">
    Quality Prediction in a Mining Process
    </font></b>
    
 
 <b><p style="text-align:center;"><font size ="3" color ="Black">A Mineral Processing Engineer Approch</font></b> 

"""
"""
<h1>Table of Contents</h1>
<div class="alert alert-block alert-info" style="margin-top: 20px">
    <ul>
        <li>
            1. Foreword    
        </li>
        <br>
        <li>
            2. CRISP-DM   
        </li>
        <br>
        <li>
            3. CRISP-DM STEP-1: Business Understanding   
        </li>
        <br>
        <li>
            4.CRISP-DM STEP-2: Data Understanding
            <ul>
                <li>4.1. Examine Data as Data Scientist</li>
                <li>4.2. Examine Data as Mineral Processing Engineer</li>
            </ul>
        </li>
        <br>
        <li>
            5. CRISP-DM STEP-3: DATA PREPARATION 
            <ul>
                <li>5.1. Grouping Rows with Hourly Frequency</li>
                <li>5.2. Divide Data</li>
            </ul>
        </li>
        <br>
        <li>
            6. CRISP-DM STEP-4: MODELING
             <ul>
                <li>6.1. P-Value</li>
                <li>6.2. Confidence Interval</li>
            </ul>
        </li>
        <br>
        <li>
            7. CRISP-DM STEP-5: Evaluation
            <ul>
                <li>7.1. R-squared in Regression Analysis</li>
                <li>7.2. Multi Linear Regression Analysis</li>
                <li>7.3. Random Forest Regressor</li>
            </ul>
        </li>
        <br>
        <li>
            8. CRISP-DM STEP-6: Deployment   
            <ul>
                <li>8.1. First Predictions</li>
                <li>8.2. Second Predictions</li>
            </ul>
        </li>
        <br>
        <li>
            9. Conclusion    
        </li>
    </ul>
</div>
"""
"""
<hr style="border: none; border-bottom: 2px solid lightblue;">



<h1><a id="Foreword">1. FOREWORD</a></h1>

<h4>I am a Mineral Processing Engineer and I train my-self about ML and AI. I prepared this notebook to gain experience. Thanks for EduardoMagalhãesOliveira for giving me this opportunity with this data. 

<h4> You will see a Mineral Processing approach in this report. I know, I am an amateur about machine learning. Please feel free to comment or reach me for anything. 
    

<b><p style="text-align:right;"><font size ="3" color ="Black">Aydin AKTAR</font></b>
<b><p style="text-align:right;"><font size ="3" color ="Black">aktaraydin@gmail.com</font></b>
<b><p style="text-align:right;"><a href="www.linkedin.com/in/aktaraydin">Linkedin</a></font></b>

"""
"""
<h4> You can reach the more detail from report below linked.

<h3><p style="text-align:center;"><a href="https://drive.google.com/file/d/1kFGw01qVLsAaZ7kcKm27s0Hl3b4Mtze8/view?usp=sharing">Detailed Report Address</a></p></h3>
"""
"""
<hr style="border: none; border-bottom: 2px solid lightblue;">
<h1><a id="CRISP-DM">2. CRISP-DM </a></h1>

"""
"""
<h2><p style="text-align:center;"><a href="https://en.wikipedia.org/wiki/Cross-industry_standard_process_for_data_mining">Definition</a></p></h2>


<img src="https://upload.wikimedia.org/wikipedia/commons/b/b9/CRISP-DM_Process_Diagram.png" width="500" height="500">



<h3><b><p style="text-align:center;">I will do the project over the CRISP-DM cycle.</b></h3>


"""
"""
<hr style="border: none; border-bottom: 2px solid lightblue;">
<h1><a id="step1">3. CRISP-DM STEP-1: BUSINESS UNDERSTANDING</a></h1>
"""
"""
<h4>We cannot use the Mineral or Element in the ore extracted from the earth directly in industry. We
have to carry out an enrichment operation beforehand. The data we obtained was taken from a
Flotation Plant, method is Reverse Cationic Flotation </h4>
<br>
<h3><p style="text-align:center;"><a href=" https://en.wikipedia.org/wiki/Froth_flotation">Froth Flotation</a></p></h3>
<h3><p style="text-align:center;"><a href=" https://iopscience.iop.org/article/10.1088/1742-6596/879/1/012016/pdf">Reverse Cationic Flotation</a></p></h3>


"""
"""
<hr style="border: none; border-bottom: 2px solid lightblue;">
<h1><a id="step2">4.CRISP-DM STEP-2: DATA UNDERSTANDING</a></h1>

"""
"""

<h3><a id="step21">4.1. Examine Data as Data Scientist</a></h3>
"""
#Library Imports

import pandas as pd 
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
from matplotlib.pyplot import figure
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LinearRegression
from sklearn.ensemble import RandomForestRegressor 
import statsmodels.api as sm 
from sklearn.metrics import r2_score
from pylab import *
#Upload Data 
url ='../input/quality-prediction-in-a-mining-process/MiningProcess_Flotation_Plant_Database.csv'
mainData = pd.read_csv(url,decimal=',')

 
print('Shape of Main Data = ', mainData.shape)
mainData = mainData.dropna()
print('Shape of Main Data after drop null values = ', mainData.shape)

"""
<h4>Data consists of 24 columns and 737453 rows, covers the dates between March 2017 and
September 2017 and some of the data were recorded at 10-second intervals and some at 1-hour
intervals.</h4> 

"""
mainData.head(10)
mainData.columns
mainData.dtypes
"""
<h4>In order to perform mathematical calculations on data, the data must be in "int" or "float"
format. The data type of columns containing numbers is "float64", sufficient for calculation.
Date column is in "object" data type, we will need to change the unit before calculations. 

Column Definitions: 

- Date: Date collection date and time.
- % Iron Feed: Feed grade of iron-containing ore.
- % Silica Feed: Feed grade of silica-containing ore..
- Starch Flow: Depressant chemical for Iron(Fe) containing ore.
- Amina Flow: Collector chemical for Silica containing ore.
- Ore Pulp Flow: The amount of pulp flow fed to the Flotation Columns as the
product of the previous process step.
- Ore Pulp pH: pH.
- Ore Pulp Density: The solid percent of ore fed to Flotation Columns.
- Flotation Column 01,02,03,04,05,06,07 Air Flow: The amount of air fed to the
Flotations Columns to frothing.
- Flotation Column 01,02,03,04,05,06,07 Level: Showing float thickness of Flotation
Columns.
- % Iron Concentrate: Concentrate grade of iron-containing ore.
- % Silica Concentrate: Concentrate grade of silica-containing ore
"""
mainData.describe()
"""
<h3><a id="step22">4.2. Examine Data as Mineral Processing Engineer:
</a></h3>
"""
"""
<h3><p style="text-align:center;">The plant flowsheet is probably (untold) like:</h3>

<img src="https://i.ibb.co/v48rqYr/ak-m-semas.jpg" width="400" height="500">
"""
"""
<h4>Let’s deep inside to Data:

    
- Data can be record at 10 second frequency.
    
- Flows can be measured.
- pH can be measured.
- Feed and Concentrate grade can be analyzed instantly.
- Columns has level sensor.
    
In order to obtain this data, plant must be operating with established automation system. It is not
possible collect this data with this frequency by human hand.
In order to fully analyze a Flotation Plant, we need to obtain the minimum data below, which is
available in a plant operated with Scada.


Missing data in the data that should be a minimum for analysis.

1. Recovery 
2. Liberation Degree, Grinding Performance
3. Pumps Amperes, Pressures, Flows
4. Feed Tons
5. Feed and Tail Grades for Every Columns 
6. Mineralogy


"""
"""
<hr style="border: none; border-bottom: 2px solid lightblue;">

<h1><a id="step3">5.CRISP-DM STEP-3: DATA PREPARATION</a></h1>


"""
"""
<h4>There is no need for unit conversions in data. I just changed the "date" column from the "object"
data type to the "datatime64".
"""
#change date columns as datetime
mainData['date'] = pd.to_datetime(mainData['date'])
"""

<h3><a id="step31">5.1. Grouping Rows with Hourly Frequency</a></h3>
"""
"""
<h4>I indicated about the need to consider the plant data as a totality. So how do we do this on
data? By doing loop-based analysis. My solution is to consider each row of data on a loop. We
feed 100 tons of ore to the plant, enrich it and complete the cycle. In this data I set each cycle as
1 hour. That is, we take a photograph of the plant every hour and analyze it on this photograph.
If we had more regular data, we could do these cycles even for 1 minute, then we would have
more cycles for machine learning.


"""
mainData['date'] = pd.to_datetime(mainData['date'])
#grouping the data according to the hours and get their average values. 
cycle_data = mainData.groupby(pd.Grouper(key='date',freq='H')).mean()
# cycle_data.insert(0,'Date',cycle_data.index)
cycle_data.reset_index(inplace = True)

#some rows have 'null' values because of timing. We need to drop them 
print('Shape of Cycle Data = ', cycle_data.shape)
cycle_data = cycle_data.dropna()
print('Shape of Cycle Data after drop null values = ', cycle_data.shape)
mainData.dtypes
"""
<h4>The grouping process has a number of advantages and disadvantages.
Advantages:
    
- Each analysis will be able to do each data cycle on an hourly frequency.
- Date column can be dropped. The number of columns fell to 23.
- The number of rows fell to 4097 from 737453. Every rows means calculations on
computer.
    
Disadvantage:
    
- The number of rows fell to 4097 from 737453. The more rows we have for machine
learning, the better results we get. This large data loss will adversely affect our
estimation results.
"""
data = cycle_data.copy()
data.head()
"""

<h3><a id="step32">5.2. Divide Data</a></h3>
"""
"""
<h4>Flotation plant product (at least for this missing data) Iron and Silica concentrates namely the
last two columns. We will develop separate models for silica and iron in machine learning, so we
will separate the columns from the main data as two data types named "iron_concentrate" and
"silica_concentrate".
The remaining columns are essentially Flotation conditions required to enrich the concentrate.
We also separate them as "flotation_conditions".
"""
#seperate data as flotation_conditions and concentrates
flotation_conditions = data.iloc[:,1:22]
concentrates = data.iloc[:,22:]
silica_concentrate = concentrates.iloc[:,1].values
iron_concentrate = concentrates.iloc[:,0].values

print('Shape of flotation_conditions = ', flotation_conditions.shape)
"""
<hr style="border: none; border-bottom: 2px solid lightblue;">
<h1><a id="step4">6. CRISP-DM STEP-4: MODELING</a></h1>
"""
"""

<h3><a id="step41">6.1. P-Value:</a></h3>
<h3><p style="text-align:center;"><a href="https://medium.com/@ODSC/the-importance-of-p-values-in-data-science-6cb7c7380881">p-Value Definition</a></p></h3>
    
"""
'''P-value'''

b_zero = np.append(arr = np.ones((len(data.iloc[:,1].values),1)).astype(int),
                                                                     values = flotation_conditions,
                                                                     axis = 1)

silica_concentrate = concentrates.iloc[:,1].values
iron_concentrate = concentrates.iloc[:,0].values

model_iron = sm.OLS(endog = iron_concentrate,exog = b_zero).fit()
print(model_iron.summary())

model_silica = sm.OLS(endog = silica_concentrate,exog = b_zero).fit()
print(model_silica.summary())
#extract p-value from regression results
pValue_fe = model_iron.pvalues
pValue_si = model_silica.pvalues
pValue_fe_list = []
pValue_si_list = []
for i in range(len(pValue_fe)):
    if i > 0:
        pValue_fe_list.append(pValue_fe[i])
        pValue_si_list.append(pValue_si[i])
width_in_inches = 15
height_in_inches = 13
dots_per_inch = 60

plt.figure(
    figsize=(width_in_inches, height_in_inches),
    dpi=dots_per_inch)

t = flotation_conditions.columns
fe = pValue_fe_list
si = pValue_si_list
plot(t, fe, label = 'Fe', color = 'blue',marker='o')
plot(t, si, label = 'Si',color = 'red', marker='x')
plt.xticks(rotation=90)
plt.legend(loc="upper right",prop={"size":25})
ylabel('P Value',fontsize = 20)
title('P Value of Si adn Fe Cocentrates',fontsize = 25)
grid(True)
plt.figure(figsize=(60,3))
plt.savefig("p_values.png")
show()

"""
<h4>When we examine the P values, we see that some values do not affect the concentrated grades.
But now we know that this is due to our lack of data. What would you think if you saw that the
cocoa had no effect on the chocolate produced in the chocolate factory?
"""
"""

<h3><a id="step42">6.2. Confidence Interval:</a></h3>
<h3><p style="text-align:center;"><a href="http://mlwiki.org/index.php/Confidence_Intervals">Confidence Interval Definition</a></p></h3>
"""
#Confidence Intervals

m_fe = iron_concentrate.mean()
se_fe = iron_concentrate.std()/math.sqrt(len(iron_concentrate))
ci_fe = [m_fe - se_fe*1.96, m_fe + se_fe*1.96]


m_si = silica_concentrate.mean()
se_si = silica_concentrate.std()/math.sqrt(len(silica_concentrate))
ci_si = [m_si - se_si*1.96, m_si + se_si*1.96]

print  ('Confidence interval of Fe Concentrate:' ,ci_fe)
print  ('Confidence interval of Silica Concentrate:' ,ci_si)
"""
<h4>Metallurgical engineers apply the pouring process to the concentrate we sell and obtain pure
iron. There is a grade range they demand from us for this operation:
%67 – 68 Fe Grade
%1.6 – 1.7 Si Grade
In the model we developed according to the confidence interval, the silica values are more than
demanded. In the first stage, we need to warn the engineers in the field. (Assuming our data is
reliable.)</h4>


"""
"""
<hr style="border: none; border-bottom: 2px solid lightblue;">
<h1><a id="step5">7. CRISP-DM STEP-5: EVALUATION</a></h1>
"""
"""

<h3> <a id="step51">7.1. R-squared in Regression Analysis</a></h3>
<h3><p style="text-align:center;"><a href="https://www.geeksforgeeks.org/ml-r-squared-in-regression-analysis/">Confidence Interval Definition</a></p></h3>

"""
"""

<h3><a id="step52">7.2. Multi Linear Regression Analysis</a></h3>
"""
width_in_inches = 15
height_in_inches = 13
dots_per_inch = 60

plt.figure(
    figsize=(width_in_inches, height_in_inches),
    dpi=dots_per_inch)

x_axis = data['% Iron Concentrate']
y_axis = data['% Iron Feed']
plot(x_axis, y_axis,marker='o')
plt.xticks(rotation=90)
plt.legend(loc="upper right",prop={"size":25})
xlabel('% Iron Concentrate',fontsize = 20)
ylabel('% Iron Feed',fontsize = 20)
title('% Iron Concentrate vs % Iron Feed',fontsize = 25)
grid(True)
plt.figure(figsize=(60,3))
show()
"""
<h4>As you can see in plot, there is no Linear Reletionship betweenn Iron Feed and Iron Concentration, this is just example. The other columns relation ships are same. I am waiting for a low result of R2-Score from Multi Linear Regression Analysis
"""
#train test split for regression training and testing 
x_train, x_test, y_train, y_test = train_test_split(flotation_conditions,
                                                    concentrates,
                                                    test_size = 0.15,
                                                    random_state = 0)
'''MLR'''
regressor_mlr = LinearRegression()
regressor_mlr.fit(x_train,y_train) #x trainden y traini öğren 

y_pred_mlr = regressor_mlr.predict(x_test)
print('R2 Score of Multi Linear Regression',r2_score(y_test,y_pred_mlr))
"""
<h4>It has become clear that the Multi Linear Regression model cannot be used. I will focus on
Random Forest machine learning model development.
"""
"""

<h3> <a id="step53">7.3. Random Forest Regressor:</a></h3>
<h3><p style="text-align:center;"><a href="https://en.wikipedia.org/wiki/Random_forest">Random Forest Regressor Definition</a></p></h3> 
"""
"""
<h4>The first time I divided the data as train test split at 0.33 degree and applied Random Forest
machine learning, but I got very low r-Square values. Finally, I used all the data for training and
got the following results. Despite the lack of data, the result is satisfactory.
"""
'''RANDOM FOREST with Train-Test Split'''

regressor_randForest = RandomForestRegressor(random_state = 0, n_estimators = 100)  
regressor_randForest.fit(x_train,y_train) 
y_pred_rf = regressor_randForest.predict(x_test)

regressor_randForest2 = RandomForestRegressor(random_state = 0, n_estimators = 100)  
regressor_randForest2.fit(flotation_conditions,concentrates)
y_pred_rf2 = regressor_randForest.predict(flotation_conditions)
 
print('R2 Score of Random Forest Regression with Train-Test Split',r2_score(y_test,y_pred_rf))
print('R2 Score of Random Forest Regression with Whole Data',r2_score(concentrates,y_pred_rf2))
"""
<h4>As I mentioned before I divided concentrations and again applied Random Forest separately to Iron and Silica Concentration grades. R2-Score result is marvelous. 
"""
'''Iron Random Forest Model'''
regressor_Fe = RandomForestRegressor(random_state = 0, n_estimators = 100)  
regressor_Fe.fit(flotation_conditions,iron_concentrate) 
y_pred_Fe = regressor_Fe.predict(flotation_conditions)

print('R2 Score of Random Forest Regression with Only Iron',r2_score(iron_concentrate,y_pred_Fe))
'''Silica Random Forest Model'''
regressor_Si = RandomForestRegressor(random_state = 0, n_estimators = 100)  
regressor_Si.fit(flotation_conditions,silica_concentrate) 
y_pred_Si = regressor_Si.predict(flotation_conditions)

print('R2 Score of Random Forest Regression with Only Silica',r2_score(silica_concentrate,y_pred_Si))
"""
<hr style="border: none; border-bottom: 2px solid lightblue;">
<h1><a id="step6">8. CRISP-DM STEP-6: DEPLOYMENT</a></h1>
"""
"""
<h3><a id="step61">8.1. First Predictions:</a><h3>

"""
"""
<h4>I will feed to ML random values as conditions and demand to predict concentrate
values</h4>
"""
#feel free change values and see results
predictions = {'% Iron Feed':50.5,
          '% Silica Feed':13.3,
          'Starch Flow':3500.0,
          'Amina Flow':580.0,
          'Ore Pulp Flow':400.0,
          'Ore Pulp pH':10.11,
          'Ore Pulp Density':1.69,
          'Flotation Column 01 Air Flow':250.0,
          'Flotation Column 02 Air Flow':150.0,
          'Flotation Column 03 Air Flow':270.0,
          'Flotation Column 04 Air Flow':190.0,
          'Flotation Column 05 Air Flow':230.0,
          'Flotation Column 06 Air Flow':200.0,
          'Flotation Column 07 Air Flow':240.0,
          'Flotation Column 01 Level':480,
          'Flotation Column 02 Level':210.0,
          'Flotation Column 03 Level':550.0,
          'Flotation Column 04 Level':620.0,
          'Flotation Column 05 Level':610.0,
          'Flotation Column 06 Level':615.0,
          'Flotation Column 07 Level':616.0,
          }
#to see predictions, run this code
predict_values = []
for value in predictions.values(): 
    predict_values.append(value)

predict_Fe = regressor_Fe.predict([predict_values])
predict_Si = regressor_Si.predict([predict_values])

print('Predicted Fe Concentrate =',predict_Fe,'%')
print('Predicted Silica Concentrate =',predict_Si,'%')
"""
<h3><a id="step62">8.2. Second Predictions:</a></h3>

<h4>I will feed to ML mean values of the main data and demand to predict
concentrate values.</h4>
"""
predict_from_means = flotation_conditions.describe().mean()
predict_Fe = regressor_Fe.predict([predict_from_means])
predict_Si = regressor_Si.predict([predict_from_means])

print('Predicted Fe Concentrate from Conditions mean  =',predict_Fe,'%')
print('Predicted Silica Concentrate from Conditions mean =',predict_Si,'%')
"""
<h4>We obtained the values we determined in the Confidence Intervals. The results are also very close to
the average tenor values.
After all, we have a working ML
application.


"""
"""
<hr style="border: none; border-bottom: 2px solid lightblue;">
<h1><a id="conc">9. CONCLUSION</a><h1>
"""
"""
<h4>With the analysis made over the same data set, we can write thousands more questions and get
results.

Our touchstone data on the road to IoT (internet of things) and how we store them. As long as we
take care of our data and enrich them, the future is not far away. The more data you get, the more
accurate results are inevitable.

Artificial intelligence, which analyzes the ore coming from the mine and gives us the facility
parameters as ready and calculate equipment maintenance possibilities, is now very close.</h4>
<br>

<b><p style="text-align:right;"><font size ="3" color ="Black">See you on the next project.</font></b> 
<b><p style="text-align:right;"><font size ="3" color ="Black">Aydin AKTAR</font></b> 

"""