Day 7 [Python ML] Machine Learning的处理流程

Step 1: 蒐集数据

要先将自己需要的数据下载好，并且确认资料格式而去做不同的处理

Step 2: 准备资料

读取csv

import pandas as pd
    
# 读取资料
melbourne_file_path = './Dataset/melb_data.csv'
melbourne_data = pd.read_csv(melbourne_file_path) 

选择目标(target)以及特徵(feature)

# 选择目标以及特徵
y = filtered_melbourne_data.Price
melbourne_features = ['Rooms', 'Bathroom', 'Landsize', 'BuildingArea', 
                        'YearBuilt', 'Lattitude', 'Longtitude']
X = filtered_melbourne_data[melbourne_features]

特徵工程

取得资料中的状态

# 取得某一个资料的全部状态
print('Unique values in `state` column:', list(ks.state.unique()))

Unique values in `state` column: ['failed', 'canceled', 'successful', 'live', 'undefined', 'suspended']

One hot encoding

get_dummies可以将dataframe自动转为one hot encoding的模式，就可以跑这些资料了

features = ["Pclass", "Sex", "SibSp", "Parch"]
X = pd.get_dummies(train_data[features])

原始资料

处理後资料

drop掉缺失值(missing values)

# 将有缺失值的row drop掉
filtered_melbourne_data = melbourne_data.dropna(axis=0)

将资料切成train以及test

from sklearn.model_selection import train_test_split
train_X, val_X, train_y, val_y = train_test_split(X, y,random_state = 0)

Step 3: Select a model && Step 4: Train the model

Decision Tree的模型建法

from sklearn.tree import DecisionTreeRegressor

dt_model = DecisionTreeRegressor(max_leaf_nodes=100, random_state=0)

dt_model(train_X, train_y)

Random Forest的模型建法

from sklearn.ensemble import RandomForestRegressor

# Define the model. Set random_state to 1
rf_model = RandomForestRegressor(random_state=1)

rf_model(train_X, train_y)

LightGBM的模型建法

pip install lightgbm
要先将资料转成lgb.Dataset，才能做後续的处理
须将目标属性放在label中
param为设定参数，利用dictionary的方式设定完後，再将参数丢入model中
num_round为模型跑的次数，early_stopping_rounds则是若有几次结果是类似的，就会停止运算

import lightgbm as lgb

dtrain = lgb.Dataset(train[feature_cols], label=train['is_attributed'])
dvalid = lgb.Dataset(valid[feature_cols], label=valid['is_attributed'])
dtest = lgb.Dataset(test[feature_cols], label=test['is_attributed'])

param = {'num_leaves': 64, 'objective': 'binary'}
param['metric'] = 'auc'
num_round = 1000
bst = lgb.train(param, dtrain, num_round, valid_sets=[dvalid], early_stopping_rounds=10)

Step 5: 评估模型

不同评估(Evaluate)的方法

平均绝对误差

from sklearn.metrics import mean_absolute_error

# 对测试资料做验证
# 从验证资料中取得预测价格
val_predictions = melbourne_model.predict(val_X)
print(mean_absolute_error(val_y, val_predictions))

ROC AUC 分数

from sklearn import metrics

ypred = bst.predict(test[feature_cols])
score = metrics.roc_auc_score(test['is_attributed'], ypred)
print(f"Test score: {score}")

Step 6: 调整参数

不同的参数会有不同的结果，训练各种不同的参数以达到最好的结果

写一只函式(function)是可以自动训练不同参数的

from sklearn.metrics import mean_absolute_error
from sklearn.tree import DecisionTreeRegressor

def get_mae(max_leaf_nodes, train_X, val_X, train_y, val_y):
    model = DecisionTreeRegressor(max_leaf_nodes=max_leaf_nodes, random_state=0)
    model.fit(train_X, train_y)
    preds_val = model.predict(val_X)
    mae = mean_absolute_error(val_y, preds_val)
    return(mae)

将不同的参数丢入这个函式(function)中，并且判断其中最好的值，将其值用於训练全部的资料

for max_leaf_nodes in [5, 50, 500, 5000]:
    my_mae = get_mae(max_leaf_nodes, train_X, val_X, train_y, val_y)
    print("Max leaf nodes: %d  \t\t Mean Absolute Error:  %d" %(max_leaf_nodes, my_mae))
    
# 可以用dic来纪录资料，并取出其中最好的值
mae_result = {}
for max_leaf in candidate_max_leaf_nodes:
    mae_result[max_leaf] = get_mae(max_leaf, train_X, val_X, train_y, val_y)
    
# 使用lambda语法，取出dic中最小的值
best_tree_size = min(mae_result, key=mae_result.get)
print(best_tree_size)

Max leaf nodes: 5  		 Mean Absolute Error:  347380
Max leaf nodes: 50  		 Mean Absolute Error:  258171
Max leaf nodes: 500  		 Mean Absolute Error:  243495
Max leaf nodes: 5000  		 Mean Absolute Error:  254983
500

final_model = DecisionTreeRegressor(max_leaf_nodes=500)
final_model.fit(X, y)

Step 7: 取得预测结果

当模型(Model)训练好後，我们可以将比赛中的测试资料丢入计算预测值，并将结果转成csv档案来提交比赛

# 读取测试资料的资料夹路径
test_data_path = './Dataset/test.csv'

# 用pandas读取测试资料
test_data = pd.read_csv(test_data_path)

# 创建test_X来取得用於预测的columns
test_X = test_data[features]

# 取得测试资料的预测结果
test_preds = rf_model.predict(test_X)

# 整理资料格式并且输出成csv
output = pd.DataFrame({'Id': test_data.Id,
                      'SalePrice': test_preds})
output.to_csv('submission.csv', index=False)