"""
### Задание для курсового проекта

Метрика:
R2 - коэффициент детерминации (sklearn.metrics.r2_score)

Сдача проекта:
1. Соревнование заканчивается 02.03.21
2. Прислать в раздел Задания Урока 10 ("Вебинар. Консультация по итоговому проекту")
ссылку на ноутбук в github или public kaggle notebook.
3. Необходимо получить R2 > 0.6 на Private Leaderboard.
4. Укажите свой ник на kaggle 


Примечание:
Все файлы csv должны содержать названия полей (header - то есть "шапку"),
разделитель - запятая. В файлах не должны содержаться индексы из датафрейма.
____________
Рекомендации для файла с кодом (ipynb):
1. Файл должен содержать заголовки и комментарии
2. Повторяющиеся операции лучше оформлять в виде функций
3. По возможности добавлять графики, описывающие данные (около 3-5)
4. Добавлять только лучшую модель, то есть не включать в код все варианты решения проекта
5. Скрипт проекта должен отрабатывать от начала и до конца (от загрузки данных до выгрузки предсказаний)
6. Весь проект должен быть в одном скрипте (файл ipynb).
7. При использовании статистик (среднее, медиана и т.д.) в качестве признаков, лучше считать их на трейне, и потом на валидационных и тестовых данных не считать статистики заново, а брать их с трейна.
8. Проект должен полностью отрабатывать за разумное время (не больше 10 минут), поэтому в финальный вариант лучше не включать GridSearch с перебором большого количества сочетаний параметров. 
"""
"""
**План занятия**
* [Загрузка данных](#load)
* [1. EDA](#eda)
* [2. Обработка выбросов](#outlier)
* [3. Обработка пропусков](#nan)
* [4. Построение новых признаков](#feature)
* [5. Отбор признаков](#feature_selection)
* [6. Разбиение на train и test](#split)
* [7. Построение модели](#modeling)
* [8. Прогнозирование на тестовом датасете](#prediction)
"""
"""
**Подключение библиотек и скриптов**
"""
# This Python 3 environment comes with many helpful analytics libraries installed
# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python
# For example, here's several helpful packages to load

import numpy as np # linear algebra
import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)

# Input data files are available in the read-only "../input/" directory
# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory

import os
for dirname, _, filenames in os.walk('/kaggle/input'):
    for filename in filenames:
        print(os.path.join(dirname, filename))

# You can write up to 5GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using "Save & Run All" 
# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session
import numpy as np
import pandas as pd
import random

from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.preprocessing import StandardScaler, RobustScaler
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import r2_score as r2
from sklearn.model_selection import KFold, GridSearchCV

from datetime import datetime

import matplotlib
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
import warnings
warnings.filterwarnings('ignore')
matplotlib.rcParams.update({'font.size': 14})
def evaluate_preds(train_true_values, train_pred_values, test_true_values, test_pred_values):
    print("Train R2:\t" + str(round(r2(train_true_values, train_pred_values), 3)))
    print("Test R2:\t" + str(round(r2(test_true_values, test_pred_values), 3)))
    
    plt.figure(figsize=(18,10))
    
    plt.subplot(121)
    sns.scatterplot(x=train_pred_values, y=train_true_values)
    plt.xlabel('Predicted values')
    plt.ylabel('True values')
    plt.title('Train sample prediction')
    
    plt.subplot(122)
    sns.scatterplot(x=test_pred_values, y=test_true_values)
    plt.xlabel('Predicted values')
    plt.ylabel('True values')
    plt.title('Test sample prediction')

    plt.show()
"""
**Пути к директориям и файлам**
"""
TRAIN_DATASET_PATH = '/kaggle/input/real-estate-price-prediction-moscow/train.csv'
TEST_DATASET_PATH = '/kaggle/input/real-estate-price-prediction-moscow/test.csv'
RNDM = 21
"""
### Загрузка данных <a class='anchor' id='load'>
"""
"""
**Описание датасета**

* **Id** - идентификационный номер квартиры
* **DistrictId** - идентификационный номер района
* **Rooms** - количество комнат
* **Square** - площадь
* **LifeSquare** - жилая площадь
* **KitchenSquare** - площадь кухни
* **Floor** - этаж
* **HouseFloor** - количество этажей в доме
* **HouseYear** - год постройки дома
* **Ecology_1, Ecology_2, Ecology_3** - экологические показатели местности
* **Social_1, Social_2, Social_3** - социальные показатели местности
* **Healthcare_1, Helthcare_2** - показатели местности, связанные с охраной здоровья
* **Shops_1, Shops_2** - показатели, связанные с наличием магазинов, торговых центров
* **Price** - цена квартиры
"""
train_df = pd.read_csv(TRAIN_DATASET_PATH)
train_df.tail()
train_df.dtypes
test_df = pd.read_csv(TEST_DATASET_PATH)
test_df.tail()
print('Строк в трейне:', train_df.shape[0])
print('Строк в тесте', test_df.shape[0])
train_df.shape[1] - 1 == test_df.shape[1]
"""
### Приведение типов
"""
train_df.dtypes
train_df['Id'] = train_df['Id'].astype(str)
train_df['DistrictId'] = train_df['DistrictId'].astype(str)
"""
## 1. EDA  <a class='anchor' id='eda'>
Делаем EDA для:
- Исправления выбросов
- Заполнения NaN
- Идей для генерации новых фич
"""
"""
**Целевая переменная**
"""
plt.figure(figsize = (16, 8))

train_df['Price'].hist(bins=30)
plt.ylabel('Count')
plt.xlabel('Price')

plt.title('Target distribution')
plt.show()
"""
**Количественные переменные**
"""
train_df.describe()
df_num_features = train_df.select_dtypes(include=['float64', 'int64'])
df_num_features.drop('Price', axis=1, inplace=True)
df_num_features.hist(figsize=(16,16), bins=20, grid=False);
"""
**Номинативные переменные**
"""
train_df.select_dtypes(include='object').columns.tolist()
train_df['DistrictId'].value_counts()
train_df['Ecology_2'].value_counts()
train_df['Ecology_3'].value_counts()
train_df['Shops_2'].value_counts()
"""
### 2. Обработка выбросов  <a class='anchor' id='outlier'>
Что можно делать с ними?
1. Выкинуть эти данные (только на трейне, на тесте ничего не выкидываем)
2. Заменять выбросы разными методами (медианы, средние значения, np.clip и т.д.)
3. Делать/не делать дополнительную фичу
4. Ничего не делать
"""
"""
**Square**
"""
"""
Для начала обрежем квадратные метры
"""
train_df.loc[train_df['Square'] < 3, 'Square'] = 3
"""
**Rooms**
"""
"""
выразим выбросы количества комнат через общую площадь
"""
train_df['Rooms'].value_counts()
train_df['Rooms_outlier'] = 0
train_df.loc[(train_df['Rooms'] == 0) | (train_df['Rooms'] >= 6), 'Rooms_outlier'] = 1
train_df.head()
train_df.loc[train_df['Rooms'] == 0, 'Rooms'] = 1
temp_df = train_df.loc[train_df['Rooms'] < 6]
Square_Rooms_K = (temp_df['Square'] / temp_df['Rooms']).median()
Square_Rooms_K
train_df.loc[train_df['Rooms'] >= 6, 'Rooms'] = round(train_df['Square'] / Square_Rooms_K)
train_df['Rooms'].value_counts()
"""
**KitchenSquare** 
"""
train_df['KitchenSquare'].value_counts()
train_df['KitchenSquare'].quantile(.975), train_df['KitchenSquare'].quantile(.025)
"""
выразим выбросы площади кухни через общую площадь
"""
temp_df = train_df.loc[train_df['KitchenSquare'] < train_df['Square']]
Square_KitchenSquare_K = (temp_df['Square'] / temp_df['KitchenSquare']).median()
Square_KitchenSquare_K
condition = (train_df['KitchenSquare'].isna()) \
             | (train_df['KitchenSquare'] > train_df['KitchenSquare'].quantile(.975))
train_df.loc[condition, 'KitchenSquare'] = round(train_df['Square'] / Square_KitchenSquare_K)

train_df.loc[train_df['KitchenSquare'] < 3, 'KitchenSquare'] = 3
train_df['KitchenSquare'].value_counts()
"""
**HouseFloor, Floor**
"""
train_df['HouseFloor'].sort_values().unique()
train_df['Floor'].sort_values().unique()
(train_df['Floor'] > train_df['HouseFloor']).sum()
train_df['HouseFloor_outlier'] = 0
train_df.loc[train_df['HouseFloor'] == 0, 'HouseFloor_outlier'] = 1
train_df.loc[train_df['Floor'] > train_df['HouseFloor'], 'HouseFloor_outlier'] = 1
train_df.loc[train_df['HouseFloor'] == 0, 'HouseFloor'] = train_df['HouseFloor'].median()
floor_outliers = train_df.loc[train_df['Floor'] > train_df['HouseFloor']].index
floor_outliers
# train_df.loc[floor_outliers, 'Floor'] = train_df.loc[floor_outliers, 'HouseFloor']\
#                                                 .apply(lambda x: random.randint(1, x))
# исправил этажность, а не этаж

train_df.loc[floor_outliers, 'HouseFloor'] = train_df.loc[floor_outliers, 'Floor']
(train_df['Floor'] > train_df['HouseFloor']).sum()
"""
**HouseYear**
"""
train_df['HouseYear'].sort_values(ascending=False)
train_df.loc[train_df['HouseYear'] > 2020, 'HouseYear'] = 2020
train_df.loc[train_df['HouseYear'] < 1900, 'HouseYear'] = 1900
"""
### 3. Обработка пропусков  <a class='anchor' id='nan'>
"""
train_df.isna().sum()
train_df[['Square', 'LifeSquare', 'KitchenSquare']].head(10)
"""
**LifeSquare**
"""
train_df['LifeSquare_nan'] = train_df['LifeSquare'].isna() * 1

condition = (train_df['LifeSquare'].isna()) \
             & (~train_df['Square'].isna()) \
             & (~train_df['KitchenSquare'].isna())
        
train_df.loc[condition, 'LifeSquare'] = train_df.loc[condition, 'Square'] \
                                            - train_df.loc[condition, 'KitchenSquare'] - 3
"""
**Healthcare_1**
"""
train_df.drop('Healthcare_1', axis=1, inplace=True)
class DataPreprocessing:
    """Подготовка исходных данных"""

    def __init__(self):
        """Параметры класса"""
        self.medians=None
        self.kitchen_square_quantile = None
        self.temp_df = None
        
        
    def fit(self, X):
        """Сохранение статистик"""       
        # Расчет медиан
        self.medians = X.median()
        self.kitchen_square_quantile = X['KitchenSquare'].quantile(.975)
        
    
    def transform(self, X):
        """Трансформация данных"""

        # Square
        X.loc[X['Square'] < 3, 'Square'] = 3
        
        # Rooms
        X['Rooms_outlier'] = 0
        X.loc[(X['Rooms'] == 0) | (X['Rooms'] >= 6), 'Rooms_outlier'] = 1
        
        X.loc[X['Rooms'] == 0, 'Rooms'] = 1
        
        self.temp_df = X.loc[X['Rooms'] < 6]
        Square_Rooms_K = (self.temp_df['Square'] / self.temp_df['Rooms']).median()
        X.loc[X['Rooms'] >= 6, 'Rooms'] = round(X['Square'] / Square_Rooms_K)
                
        # KitchenSquare
        self.temp_df = X.loc[X['KitchenSquare'] < X['Square']]
        Square_KitchenSquare_K = (self.temp_df['Square'] / self.temp_df['KitchenSquare']).median()
        condition = (X['KitchenSquare'].isna()) \
                    | (X['KitchenSquare'] > self.kitchen_square_quantile)
        
        X.loc[condition, 'KitchenSquare'] = round(X['Square'] / Square_KitchenSquare_K)
        X.loc[X['KitchenSquare'] < 3, 'KitchenSquare'] = 3
        
        # HouseFloor, Floor
        X['HouseFloor_outlier'] = 0
        X.loc[X['HouseFloor'] == 0, 'HouseFloor_outlier'] = 1
        X.loc[X['Floor'] > X['HouseFloor'], 'HouseFloor_outlier'] = 1
        
        X.loc[X['HouseFloor'] == 0, 'HouseFloor'] = self.medians['HouseFloor']
        
        floor_outliers = X.loc[X['Floor'] > X['HouseFloor']].index
#         X.loc[floor_outliers, 'Floor'] = X.loc[floor_outliers, 'HouseFloor']\
#                                             .apply(lambda x: random.randint(1, x))
        X.loc[floor_outliers, 'HouseFloor'] = X.loc[floor_outliers, 'Floor']
        
        # HouseYear
        current_year = datetime.now().year
        
        X['HouseYear_outlier'] = 0
        X.loc[X['HouseYear'] > current_year, 'HouseYear_outlier'] = 1
        X.loc[X['HouseYear'] < 1900, 'HouseYear_outlier'] = 1
        
        X.loc[X['HouseYear'] > current_year, 'HouseYear'] = current_year
        X.loc[X['HouseYear'] < 1900, 'HouseYear'] = 1900
        
        # Healthcare_1
        if 'Healthcare_1' in X.columns:
            X.drop('Healthcare_1', axis=1, inplace=True)
            
        # LifeSquare
        X['LifeSquare_nan'] = X['LifeSquare'].isna() * 1
        condition = (X['LifeSquare'].isna()) & \
                      (~X['Square'].isna()) & \
                      (~X['KitchenSquare'].isna())
        
        X.loc[condition, 'LifeSquare'] = X.loc[condition, 'Square'] - X.loc[condition, 'KitchenSquare'] - 3
        
        
        X.fillna(self.medians, inplace=True)
        
        return X
"""
### 4. Построение новых признаков  <a class='anchor' id='feature'>
"""
train_df_corr = train_df.corr()
import seaborn as sns
plt.figure(figsize = (16,8))
sns.set(font_scale=0.8)
train_df_corr_round = np.round(train_df_corr, 2)
train_df_corr_round[np.abs(train_df_corr) < 0.3] = 0
sns.heatmap(train_df_corr_round, annot=True, linewidths=.5, cmap='coolwarm')
plt.show()
"""
#### можно попробовать не использовать social_1 или social_2 (из-за кореляции)
"""
"""
**Dummies**
"""
binary_to_numbers = {'A': 0, 'B': 1}

train_df['Ecology_2'] = train_df['Ecology_2'].replace(binary_to_numbers)
train_df['Ecology_3'] = train_df['Ecology_3'].replace(binary_to_numbers)
train_df['Shops_2'] = train_df['Shops_2'].replace(binary_to_numbers)
"""
**DistrictSize, IsDistrictLarge**
"""
district_size = train_df['DistrictId'].value_counts().reset_index()\
                    .rename(columns={'index':'DistrictId', 'DistrictId':'DistrictSize'})

district_size.head()
train_df = train_df.merge(district_size, on='DistrictId', how='left')
train_df.head()
(train_df['DistrictSize'] > 100).value_counts()
train_df['IsDistrictLarge'] = (train_df['DistrictSize'] > 100).astype(int)
"""
**MedPriceByDistrict**
"""
med_price_by_district = train_df.groupby(['DistrictId', 'Rooms'], as_index=False).agg({'Price':'median'})\
                            .rename(columns={'Price':'MedPriceByDistrict'})

med_price_by_district.head()
train_df = train_df.merge(med_price_by_district, on=['DistrictId', 'Rooms'], how='left')
train_df.head()
"""
**MedPriceByFloorYear**
"""
def floor_to_cat(X):

    X['floor_cat'] = 0

    X.loc[X['Floor'] <= 3, 'floor_cat'] = 1  
    X.loc[(X['Floor'] > 3) & (X['Floor'] <= 5), 'floor_cat'] = 2
    X.loc[(X['Floor'] > 5) & (X['Floor'] <= 9), 'floor_cat'] = 3
    X.loc[(X['Floor'] > 9) & (X['Floor'] <= 15), 'floor_cat'] = 4
    X.loc[X['Floor'] > 15, 'floor_cat'] = 5

    return X


def floor_to_cat_pandas(X):
    bins = [0, 3, 5, 9, 15, X['Floor'].max()]
    X['floor_cat'] = pd.cut(X['Floor'], bins=bins, labels=False)
    
    X['floor_cat'].fillna(-1, inplace=True)
    return X


def year_to_cat(X):

    X['year_cat'] = 0

    X.loc[X['HouseYear'] <= 1941, 'year_cat'] = 1
    X.loc[(X['HouseYear'] > 1941) & (X['HouseYear'] <= 1945), 'year_cat'] = 2
    X.loc[(X['HouseYear'] > 1945) & (X['HouseYear'] <= 1980), 'year_cat'] = 3
    X.loc[(X['HouseYear'] > 1980) & (X['HouseYear'] <= 2000), 'year_cat'] = 4
    X.loc[(X['HouseYear'] > 2000) & (X['HouseYear'] <= 2010), 'year_cat'] = 5
    X.loc[(X['HouseYear'] > 2010), 'year_cat'] = 6

    return X


def year_to_cat_pandas(X):
    bins = [0, 1941, 1945, 1980, 2000, 2010, X['HouseYear'].max()]
    X['year_cat'] = pd.cut(X['HouseYear'], bins=bins, labels=False)
    
    X['year_cat'].fillna(-1, inplace=True)
    return X
bins = [0, 3, 5, 9, 15, train_df['Floor'].max()]
pd.cut(train_df['Floor'], bins=bins, labels=False)
bins = [0, 3, 5, 9, 15, train_df['Floor'].max()]
pd.cut(train_df['Floor'], bins=bins)
train_df = year_to_cat(train_df)
train_df = floor_to_cat(train_df)
train_df.head()
med_price_by_floor_year = train_df.groupby(['year_cat', 'floor_cat'], as_index=False).agg({'Price':'median'}).\
                                            rename(columns={'Price':'MedPriceByFloorYear'})
med_price_by_floor_year.head()
train_df = train_df.merge(med_price_by_floor_year, on=['year_cat', 'floor_cat'], how='left')
train_df.head()
"""
#### Нормализация
"""
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
colnames = train_df.columns
train_df_scaled = pd.DataFrame(scaler.fit_transform(train_df), columns=colnames)
"""
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
train_df_scaled = pd.DataFrame(scaler.fit_transform(train_df), columns=['Rooms', 'Square', 'LifeSquare', 'KitchenSquare', 'Floor', 'HouseFloor', 'HouseYear',
                 'Ecology_1', 'Ecology_2', 'Ecology_3', 'Social_1', 'Social_2', 'Social_3',
                 'Helthcare_2', 'Shops_1', 'Shops_2'])
"""
train_df_scaled.head()
from sklearn.manifold import TSNE
def reduce_dims(df, dims=2, method='pca', perplexity=30):
    
    assert method in ['pca', 'tsne'], 'Неверно указан метод'
    
    if method=='pca':
        dim_reducer = PCA(n_components=dims, random_state=42)
        components = dim_reducer.fit_transform(df)
    elif method == 'tsne':
        dim_reducer = TSNE(n_components=dims, learning_rate=250, random_state=42, perplexity=perplexity)
        components = dim_reducer.fit_transform(df)
    else:
        print('Error')
        
    colnames = ['component_' + str(i) for i in range(1, dims+1)]
    return dim_reducer, pd.DataFrame(data = components, columns = colnames) 
def display_components_in_2D_space(components_df, labels=None):
    components_with_labels_df = pd.concat([components_df, pd.DataFrame(labels)], axis=1)

    figsize = (10, 7)
    if labels is not None:
        components_with_labels_df.plot(kind='scatter', x='component_1', y='component_2', 
                                         c=components_with_labels_df.iloc[:, -1], cmap=plt.get_cmap('jet'),
                                         alpha=0.5, figsize=figsize)
    else:
        components_with_labels_df.plot(kind='scatter', x='component_1', y='component_2', alpha=0.5, figsize=figsize)

    plt.xlabel('component_1')
    plt.ylabel('component_2')
    plt.title('2D mapping of objects')    
    plt.show()

def display_components_in_3D_space(components_df, labels=None):
    components_with_labels_df = pd.concat([components_df, pd.DataFrame(labels)], axis=1)

    fig = plt.figure(figsize=(10,10))
    ax = fig.add_subplot(111, projection='3d')
    
    if labels is not None:
        ax.scatter(components_with_labels_df['component_1'], 
                   components_with_labels_df['component_2'], 
                   components_with_labels_df['component_3'], 
                   c=components_with_labels_df.iloc[:, -1], 
                   cmap=plt.get_cmap('jet'), alpha=0.5)
    else:
        ax.scatter(components_with_labels_df['component_1'], 
                   components_with_labels_df['component_2'], 
                   components_with_labels_df['component_3'], 
                   alpha=0.5)

    ax.set_xlabel('component_1')
    ax.set_ylabel('component_2')
    ax.set_zlabel('component_3')
    plt.title('3D mapping of objects')
    plt.show()
"""
%%time
dim_reducer2d_tsne, components_2d_tsne = reduce_dims(train_df_scaled, dims=2, method='tsne', perplexity=50)
display_components_in_2D_space(components_2d_tsne, train_df['Price'])
"""
"""
![image.png](attachment:image.png)
"""
from sklearn.cluster import KMeans
from scipy.spatial.distance import cdist
def apply_elbow_method(X):
    """Визуализация для метода 'локтя'"""
    
    distortions = []
    K = range(2,30)
    for k in K:
        kmeanModel = KMeans(n_clusters=k, random_state=RNDM).fit(X)
        distortions.append(sum(np.min(cdist(X, kmeanModel.cluster_centers_, 'euclidean'), axis=1)) / X.shape[0])

    plt.figure(figsize=(10, 8))
    plt.plot(K, distortions, 'bx-')
    plt.xlabel('k')
    plt.ylabel('Distortion')
    plt.title('The Elbow Method showing the optimal k')
    plt.show()
"""
apply_elbow_method(train_df_scaled)
"""
def display_clusters_distribution(unique_labels, labels_counts):
    """Визуализация распределения классов по кластерам"""
    plt.figure(figsize=(8,5))

    plt.bar(unique, counts)

    plt.xlabel('Clusters')
    plt.xticks(unique)
    plt.ylabel('Count')
    plt.title('Clusters distribution')
    plt.show()
"""
![image.png](attachment:image.png)
"""
kmeans_n = KMeans(n_clusters=8, random_state=RNDM)
labels_clast_n = kmeans_n.fit_predict(train_df_scaled)
labels_clast_n = pd.Series(labels_clast_n, name='clusters_n')

unique, counts = np.unique(labels_clast_n, return_counts=True)
display_clusters_distribution(unique, counts)
clusters_n_dummies = pd.get_dummies(labels_clast_n, drop_first=True, prefix='clusters')

train_df_cluster = pd.concat([train_df_scaled, clusters_n_dummies], axis=1)
train_df_cluster.head()
"""
Все это круто, но кластеризация не помогла
"""
class FeatureGenetator():
    """Генерация новых фич"""
    
    def __init__(self):
        self.DistrictId_counts = None
        self.binary_to_numbers = None
        self.med_price_by_district = None
        self.med_price_by_floor_year = None
        self.house_year_max = None
        self.floor_max = None
        
    def fit(self, X, y=None):
        
        X = X.copy()
        
        # Binary features
        self.binary_to_numbers = {'A': 0, 'B': 1}
        
        # DistrictID
        self.district_size = X['DistrictId'].value_counts().reset_index() \
                               .rename(columns={'index':'DistrictId', 'DistrictId':'DistrictSize'})
                
        # Target encoding
        ## District, Rooms
        df = X.copy()
        
        if y is not None:
            df['Price'] = y.values
            
            self.med_price_by_district = df.groupby(['DistrictId', 'Rooms'], as_index=False).agg({'Price':'median'})\
                                            .rename(columns={'Price':'MedPriceByDistrict'})
            
            self.med_price_by_district_median = self.med_price_by_district['MedPriceByDistrict'].median()
            
        ## floor, year
        if y is not None:
            self.floor_max = df['Floor'].max()
            self.house_year_max = df['HouseYear'].max()
            df['Price'] = y.values
            df = self.floor_to_cat(df)
            df = self.year_to_cat(df)
            self.med_price_by_floor_year = df.groupby(['year_cat', 'floor_cat'], as_index=False).agg({'Price':'median'}).\
                                            rename(columns={'Price':'MedPriceByFloorYear'})
            self.med_price_by_floor_year_median = self.med_price_by_floor_year['MedPriceByFloorYear'].median()
        

        
    def transform(self, X):
        
        # Binary features
        X['Ecology_2'] = X['Ecology_2'].map(self.binary_to_numbers)  # self.binary_to_numbers = {'A': 0, 'B': 1}
        X['Ecology_3'] = X['Ecology_3'].map(self.binary_to_numbers)
        X['Shops_2'] = X['Shops_2'].map(self.binary_to_numbers)
        
        # DistrictId, IsDistrictLarge
        X = X.merge(self.district_size, on='DistrictId', how='left')
        
        X['new_district'] = 0
        X.loc[X['DistrictSize'].isna(), 'new_district'] = 1
        
        X['DistrictSize'].fillna(5, inplace=True)
        
        X['IsDistrictLarge'] = (X['DistrictSize'] > 100).astype(int)
        
        # More categorical features
        X = self.floor_to_cat(X)  # + столбец floor_cat
        X = self.year_to_cat(X)   # + столбец year_cat
        X = self.last_floor(X)   # + столбец last_floor
        X = self.first_floor(X)   # + столбец first_floor
        
        
        # Target encoding
        if self.med_price_by_district is not None:
            X = X.merge(self.med_price_by_district, on=['DistrictId', 'Rooms'], how='left')
            X.fillna(self.med_price_by_district_median, inplace=True)
            
        if self.med_price_by_floor_year is not None:
            X = X.merge(self.med_price_by_floor_year, on=['year_cat', 'floor_cat'], how='left')
            X.fillna(self.med_price_by_floor_year_median, inplace=True)
            
            
        # normalize
#         scaler = StandardScaler()
#         colnames = X.columns
#         X = pd.DataFrame(scaler.fit_transform(X), columns=colnames)
        # clusters
        kmeans_n = KMeans(n_clusters=8, random_state=RNDM)
        labels_clast_n = kmeans_n.fit_predict(X)
        labels_clast_n = pd.Series(labels_clast_n, name='clusters_n')
        # dummy clusters
        clusters_n_dummies = pd.get_dummies(labels_clast_n, drop_first=True, prefix='clusters')
        X = pd.concat([X, clusters_n_dummies], axis=1)
        
        return X
    
    def floor_to_cat(self, X):
        bins = [0, 1, 4, 5, 8, 9, 13, 15, self.floor_max]
        X['floor_cat'] = pd.cut(X['Floor'], bins=bins, labels=False)

        X['floor_cat'].fillna(-1, inplace=True) 
        return X
     
    def year_to_cat(self, X):
        bins = [0, 1941, 1945, 1980, 2000, 2010, self.house_year_max]
        X['year_cat'] = pd.cut(X['HouseYear'], bins=bins, labels=False)

        X['year_cat'].fillna(-1, inplace=True)
        return X
            
    def last_floor(self, X):
        X['last_floor'] = 0
        X.loc[(X['Floor'] == X['HouseFloor']), 'last_floor'] = 1
        return X
    
    def first_floor(self, X):
        X['first_floor'] = 0
        X.loc[(X['Floor'] == 1), 'first_floor'] = 1
        return X
"""
### 5. Отбор признаков  <a class='anchor' id='feature_selection'>
"""
train_df_cluster.columns.tolist()
"""
feature_names = ['Rooms', 'Square', 'LifeSquare', 'KitchenSquare', 'Floor', 'HouseFloor', 'HouseYear',
                 'Ecology_1', 'Ecology_2', 'Ecology_3', 'Social_1', 'Social_2', 'Social_3',
                 'Helthcare_2', 'Shops_1', 'Shops_2']

new_feature_names = ['Rooms_outlier', 'HouseFloor_outlier', 'HouseYear_outlier', 'LifeSquare_nan', 'DistrictSize',
                     'new_district', 'IsDistrictLarge',  'MedPriceByDistrict', 'MedPriceByFloorYear',
#                      'last_floor', 'first_floor',
#                      'clusters_1','clusters_2','clusters_3', 'clusters_4', 'clusters_5', 
#                      'clusters_6', 'clusters_7'
                    ]

target_name = 'Price'
"""
feature_names = ['DistrictId', 'Rooms', 'Square', 'LifeSquare', 'KitchenSquare', 'Floor', 'HouseFloor', 'HouseYear',
                 'Ecology_1', 'Ecology_2', 'Ecology_3', 'Social_1', 'Social_2', 'Social_3', 
                 'Helthcare_2', 'Shops_1', 'Shops_2'] 

new_feature_names = ['Rooms_outlier'] 

target_name = 'Price'
"""
### 6. Разбиение на train и test  <a class='anchor' id='split'>
"""
train_df = pd.read_csv(TRAIN_DATASET_PATH)
test_df = pd.read_csv(TEST_DATASET_PATH)

X = train_df.drop(columns=target_name)
y = train_df[target_name]
X_train, X_valid, y_train, y_valid = train_test_split(X, y, test_size=0.33, shuffle=True, random_state=RNDM)
preprocessor = DataPreprocessing()
preprocessor.fit(X_train)

X_train = preprocessor.transform(X_train)
X_valid = preprocessor.transform(X_valid)
test_df = preprocessor.transform(test_df)

X_train.shape, X_valid.shape, test_df.shape
features_gen = FeatureGenetator()
features_gen.fit(X_train, y_train)

X_train = features_gen.transform(X_train)
X_valid = features_gen.transform(X_valid)
test_df = features_gen.transform(test_df)

X_train.shape, X_valid.shape, test_df.shape
X_train = X_train[feature_names + new_feature_names]
X_valid = X_valid[feature_names + new_feature_names]
test_df = test_df[feature_names + new_feature_names]
X_train.isna().sum().sum(), X_valid.isna().sum().sum(), test_df.isna().sum().sum()
"""
### 7. Построение модели  <a class='anchor' id='modeling'>
"""
"""
**Обучение**
"""
%%time
from sklearn.ensemble import GradientBoostingRegressor
# from sklearn.ensemble import BaggingRegressor

gb = GradientBoostingRegressor(
                               max_depth=4,
                               min_samples_leaf=20,
                               random_state=RNDM,  
                               n_estimators=150
                              )

# br = BaggingRegressor(base_estimator=gb, n_estimators=25, random_state=RNDM)
# br.fit(X_train, y_train)
%%time
rf = RandomForestRegressor(
    random_state=RNDM, 
    max_depth=40,  # gridsearch
    criterion='mse',
    min_samples_leaf=5,  # gridsearch
    n_jobs=-1,
    n_estimators=1000  # gridsearch
)
# rf.fit(X_train, y_train)
%%time
rf2 = RandomForestRegressor(
    random_state=RNDM, 
    max_depth=17, 
    criterion='mse',
    max_features=7, 
    n_jobs=-1,
    n_estimators=200  
)
# rf.fit(X_train, y_train)
%%time
rf3 = RandomForestRegressor(
    max_depth=20,
    random_state=RNDM, 
)
%%time
rf4 = RandomForestRegressor(
    random_state=RNDM, 
)
%%time
import xgboost as xgb

xg=xgb.XGBRegressor(
                  random_state=RNDM,
                  n_estimators=5, 
                  n_jobs=-1,
                  subsample=0.5,
                  colsample_bynode=0.5,
                  num_parallel_tree=100,
                  learning_rate=0.5,
                  max_depth=3
                  )
"""
%%time
from sklearn.ensemble import BaggingRegressor

br = BaggingRegressor(
                       base_estimator=rf, 
                       n_estimators=5, 
                       random_state=RNDM
                     )
br.fit(X_train, y_train)
"""
"""
%%time
from sklearn.model_selection import GridSearchCV

params = {'n_estimators':[5, 10, 20, 50, 100, 200, 400, 1000], 
          'max_depth':[3, 5, 7, 10, 20, 40],
          'min_samples_leaf': [1, 2, 5, 10, 20, 40, 100]
         }

gs = GridSearchCV(rf, params, 
                  scoring='r2', # метрика 
                  cv=KFold(n_splits=5,   # k (кол-во разбиений/итераций) в кросс-валидации
                           random_state=RNDM, 
                           shuffle=True),
                  n_jobs=-1
                  )
gs.fit(X_train, y_train)
res = pd.DataFrame(gs.cv_results_)
res.head(2), gs.best_params_, gs.best_score_
"""
%%time
from sklearn.ensemble import StackingRegressor

stack = StackingRegressor([
                           ('rf', rf),
                           ('rf2', rf2),
#                            ('rf3', rf3),
                           ('rf4', rf4),
                           ('gb', gb), 
                           ('xg', xg),
                          ],
                          cv=5,
                          n_jobs=-1,
                          final_estimator=GradientBoostingRegressor(
                               
                               max_depth=1,
                               random_state=RNDM,  
                               n_estimators=85,
                               
                          )
                         )
                             
stack.fit(X_train, y_train)
"""
**Оценка модели**
"""
final_model = stack
%%time
y_train_preds = final_model.predict(X_train)
y_test_preds = final_model.predict(X_valid)

evaluate_preds(y_train, y_train_preds, y_valid, y_test_preds)
"""
[](http://)**Кросс-валидация**  (отключил, она есть в стеке уже)
"""
"""
%%time
Кросс-валидация (отключил, она есть в стеке уже)
cv_score = cross_val_score(final_model, X_train, y_train, scoring='r2', cv=KFold(n_splits=3, shuffle=True, random_state=RNDM))
cv_score, cv_score.mean()
"""
"""
### 8. Прогнозирование на тестовом датасете  <a class='anchor' id='prediction'>

1. Выполнить для тестового датасета те же этапы обработки и постронияния признаков
2. Не потерять и не перемешать индексы от примеров при построении прогнозов
3. Прогнозы должны быть для все примеров из тестового датасета (для всех строк)
"""
test_df.shape, test_df
submit = pd.read_csv('/kaggle/input/real-estate-price-prediction-moscow/sample_submission.csv')
submit.head()
predictions = final_model.predict(test_df)
predictions
submit['Price'] = predictions
submit.head()
submit.to_csv('rf_submit.csv', index=False)