图书网站开发,利用技术搭建网站做网站代理,天津网上办事大厅入口,北极寒流wordpress↑↑↑关注后星标Datawhale每日干货 每月组队学习#xff0c;不错过Datawhale干货 作者#xff1a;李祖贤#xff0c;深圳大学#xff0c;Datawhale高校群成员对比过kaggle比赛上面的top10的模型#xff0c;除了深度学习以外的模型基本上都是集成学习的… ↑↑↑关注后星标Datawhale每日干货 每月组队学习不错过Datawhale干货 作者李祖贤深圳大学Datawhale高校群成员对比过kaggle比赛上面的top10的模型除了深度学习以外的模型基本上都是集成学习的产物。集成学习可谓是上分大杀器今天就跟大家分享在Kaggle或者阿里天池上面大杀四方的数据科学比赛利器---集成学习。一、什么是集成学习正所谓“三个臭皮匠赛过诸葛亮”的道理在机器学习数据挖掘的工程项目中使用单一决策的弱分类器显然不是一个明智的选择因为各种分类器在设计的时候都有自己的优势和缺点也就是说每个分类器都有自己工作偏向那集成学习就是平衡各个分类器的优缺点使得我们的分类任务完成的更加优秀。在大多数情况下这些基本模型本身的性能并不是非常好这要么是因为它们具有较高的偏差(例如低自由度模型)要么是因为他们的方差太大导致鲁棒性不强(例如高自由度模型)。集成方法的思想是通过将这些弱学习器的偏差和/或方差结合起来从而创建一个「强学习器」(或「集成模型」)从而获得更好的性能。集成学习的方法1. 基于投票思想的多数票机制的集成分类器(MajorityVoteClassifier)2. 于bagging思想的套袋集成技术(BaggingClassifier)3. 基于boosting思想的自适应增强方法(Adaboost)4. 分层模型集成框架stacking(叠加算法)二、基于投票思想的集成分类器以上是多数投票的流程图分别训练n个弱分类器。对每个弱分类器输出预测结果并投票(如下图)每个样本取投票数最多的那个预测为该样本最终分类预测。加载相关库## 加载相关库from sklearn.datasets import load_iris # 加载数据from sklearn.model_selection import train_test_split # 切分训练集与测试集from sklearn.preprocessing import StandardScaler # 标准化数据from sklearn.preprocessing import LabelEncoder # 标签化分类变量初步处理数据## 初步处理数据iris load_iris()X,y iris.data[50:,[1,2]],iris.target[50:]le LabelEncoder()y le.fit_transform(y)X_train,X_test,y_train,y_test train_test_split(X,y,test_size0.5,random_state1,stratifyy)我们使用训练集训练三种不同的分类器逻辑回归 决策树 k-近邻分类器## 我们使用训练集训练三种不同的分类器逻辑回归 决策树 k-近邻分类器from sklearn.model_selection import cross_val_score # 10折交叉验证评价模型from sklearn.linear_model import LogisticRegressionfrom sklearn.tree import DecisionTreeClassifierfrom sklearn.neighbors import KNeighborsClassifierfrom sklearn.pipeline import Pipeline # 管道简化工作流clf1 LogisticRegression(penaltyl2,C0.001,random_state1)clf2 DecisionTreeClassifier(max_depth1,criterionentropy,random_state0)clf3 KNeighborsClassifier(n_neighbors1,p2,metricminkowski)pipe1 Pipeline([[sc,StandardScaler()],[clf,clf1]])pipe3 Pipeline([[sc,StandardScaler()],[clf,clf3]])clf_labels [Logistic regression,Decision tree,KNN]print(10-folds cross validation :\n)for clf,label in zip([pipe1,clf2,pipe3],clf_labels): scores cross_val_score(estimatorclf,XX_train,yy_train,cv10,scoringroc_auc) print(ROC AUC: %0.2f(/- %0.2f)[%s]%(scores.mean(),scores.std(),label))我们使用MajorityVoteClassifier集成## 我们使用MajorityVoteClassifier集成from sklearn.ensemble import VotingClassifiermv_clf VotingClassifier(estimators[(pipe1,pipe1),(clf2,clf2),(pipe3,pipe3)],votingsoft)clf_labels [MajorityVoteClassifier]all_clf [pipe1,clf2,pipe3,mv_clf]print(10-folds cross validation :\n)for clf,label in zip(all_clf,clf_labels): scores cross_val_score(estimatorclf,XX_train,yy_train,cv10,scoringroc_auc) print(ROC AUC: %0.2f(/- %0.2f)[%s]%(scores.mean(),scores.std(),label))## 对比下面结果可以得知多数投票方式的分类算法抗差能力更强。使用ROC曲线评估集成分类器## 使用ROC曲线评估集成分类器from sklearn.metrics import roc_curvefrom sklearn.metrics import auccolors [black,orange,blue,green]linestyles [:,--,-.,-]plt.figure(figsize(10,6))for clf,label,clr,ls in zip(all_clf,clf_labels,colors,linestyles): y_pred clf.fit(X_train,y_train).predict_proba(X_test)[:,1] fpr,tpr,trhresholds roc_curve(y_truey_test,y_scorey_pred) roc_auc auc(xfpr,ytpr) plt.plot(fpr,tpr,colorclr,linestylels,label%s (auc%0.2f)%(label,roc_auc))plt.legend(loclower right)plt.plot([0,1],[0,1],linestyle--,colorgray,linewidth2)plt.xlim([-0.1,1.1])plt.ylim([-0.1,1.1])plt.xlabel(False positive rate (FPR))plt.xlabel(True positive rate (TPR))plt.show()三、基于bagging思想的套袋集成技术套袋方法是由柳.布莱曼在1994年的技术报告中首先提出并证明了套袋方法可以提高不稳定模型的准确度的同时降低过拟合的程度(可降低方差)。套袋方法的流程如下注意套袋方法与投票方法的不同投票机制在训练每个分类器的时候都是用相同的全部样本而Bagging方法则是使用全部样本的一个随机抽样每个分类器都是使用不同的样本进行训练。其他都是跟投票方法一模一样对训练集随机采样分别基于不同的样本集合训练n个弱分类器。对每个弱分类器输出预测结果并投票(如下图)每个样本取投票数最多的那个预测为该样本最终分类预测。我们使用葡萄酒数据集进行建模(数据处理)## 我们使用葡萄酒数据集进行建模(数据处理)df_wine pd.read_csv(https://archive.ics.uci.edu/ml/machine-learning-databases/wine/wine.data,headerNone)df_wine.columns [Class label, Alcohol,Malic acid, Ash,Alcalinity of ash,Magnesium, Total phenols, Flavanoids, Nonflavanoid phenols,Proanthocyanins,Color intensity, Hue,OD280/OD315 of diluted wines,Proline]df_wine df_wine[df_wine[Class label] ! 1] # drop 1 classy df_wine[Class label].valuesX df_wine[[Alcohol,OD280/OD315 of diluted wines]].valuesfrom sklearn.model_selection import train_test_split # 切分训练集与测试集from sklearn.preprocessing import LabelEncoder # 标签化分类变量le LabelEncoder()y le.fit_transform(y)X_train,X_test,y_train,y_test train_test_split(X,y,test_size0.2,random_state1,stratifyy)我们使用单一决策树分类## 我们使用单一决策树分类tree DecisionTreeClassifier(criterionentropy,random_state1,max_depthNone) #选择决策树为基本分类器from sklearn.metrics import accuracy_scoretree tree.fit(X_train,y_train)y_train_pred tree.predict(X_train)y_test_pred tree.predict(X_test)tree_train accuracy_score(y_train,y_train_pred)tree_test accuracy_score(y_test,y_test_pred)print(Decision tree train/test accuracies %.3f/%.3f % (tree_train,tree_test))我们使用BaggingClassifier分类## 我们使用BaggingClassifier分类from sklearn.ensemble import BaggingClassifiertree DecisionTreeClassifier(criterionentropy,random_state1,max_depthNone) #选择决策树为基本分类器bag BaggingClassifier(base_estimatortree,n_estimators500,max_samples1.0,max_features1.0,bootstrapTrue, bootstrap_featuresFalse,n_jobs1,random_state1)from sklearn.metrics import accuracy_scorebag bag.fit(X_train,y_train)y_train_pred bag.predict(X_train)y_test_pred bag.predict(X_test)bag_train accuracy_score(y_train,y_train_pred)bag_test accuracy_score(y_test,y_test_pred)print(Bagging train/test accuracies %.3f/%.3f % (bag_train,bag_test))我们可以对比两个准确率测试准确率较之决策树得到了显著的提高我们来对比下这两个分类方法上的差异## 我们来对比下这两个分类方法上的差异x_min X_train[:, 0].min() - 1x_max X_train[:, 0].max() 1y_min X_train[:, 1].min() - 1y_max X_train[:, 1].max() 1xx, yy np.meshgrid(np.arange(x_min, x_max, 0.1),np.arange(y_min, y_max, 0.1))f, axarr plt.subplots(nrows1, ncols2,sharexcol,shareyrow,figsize(12, 6))for idx, clf, tt in zip([0, 1],[tree, bag],[Decision tree, Bagging]): clf.fit(X_train, y_train) Z clf.predict(np.c_[xx.ravel(), yy.ravel()]) Z Z.reshape(xx.shape) axarr[idx].contourf(xx, yy, Z, alpha0.3) axarr[idx].scatter(X_train[y_train0, 0],X_train[y_train0, 1],cblue, marker^) axarr[idx].scatter(X_train[y_train1, 0],X_train[y_train1, 1],cgreen, markero) axarr[idx].set_title(tt)axarr[0].set_ylabel(Alcohol, fontsize12)plt.tight_layout()plt.text(0, -0.2,sOD280/OD315 of diluted wines,hacenter,vacenter,fontsize12,transformaxarr[1].transAxes)plt.show()从结果图看起来三个节点深度的决策树分段线性决策边界在Bagging集成中看起来更加平滑。四、基于boosting思想的自适应增强方法Adaboost最初的想法是由Robert E. Schapire在1990年提出的这个想法叫做自适应增强方法。与Bagging相比Boosting思想可以降低偏差。原始的增强过程具体的实现如下AdaBoost的具体步骤如下如更新权重如下图我们用单一决策树建模## 我们用单一决策树建模from sklearn.ensemble import AdaBoostClassifiertree DecisionTreeClassifier(criterionentropy,random_state1,max_depth1)from sklearn.metrics import accuracy_scoretree tree.fit(X_train,y_train)y_train_pred tree.predict(X_train)y_test_pred tree.predict(X_test)tree_train accuracy_score(y_train,y_train_pred)tree_test accuracy_score(y_test,y_test_pred)print(Decision tree train/test accuracies %.3f/%.3f % (tree_train,tree_test))我们使用Adaboost集成建模## 我们使用Adaboost集成建模ada AdaBoostClassifier(base_estimatortree,n_estimators500,learning_rate0.1,random_state1)ada ada.fit(X_train,y_train)y_train_pred ada.predict(X_train)y_test_pred ada.predict(X_test)ada_train accuracy_score(y_train,y_train_pred)ada_test accuracy_score(y_test,y_test_pred)print(Adaboost train/test accuracies %.3f/%.3f % (ada_train,ada_test))我们观察下Adaboost与决策树的异同## 我们观察下Adaboost与决策树的异同x_min X_train[:, 0].min() - 1x_max X_train[:, 0].max() 1y_min X_train[:, 1].min() - 1y_max X_train[:, 1].max() 1xx, yy np.meshgrid(np.arange(x_min, x_max, 0.1),np.arange(y_min, y_max, 0.1))f, axarr plt.subplots(nrows1, ncols2,sharexcol,shareyrow,figsize(12, 6))for idx, clf, tt in zip([0, 1],[tree, ada],[Decision tree, Adaboost]): clf.fit(X_train, y_train) Z clf.predict(np.c_[xx.ravel(), yy.ravel()]) Z Z.reshape(xx.shape) axarr[idx].contourf(xx, yy, Z, alpha0.3) axarr[idx].scatter(X_train[y_train0, 0],X_train[y_train0, 1],cblue, marker^) axarr[idx].scatter(X_train[y_train1, 0],X_train[y_train1, 1],cred, markero) axarr[idx].set_title(tt)axarr[0].set_ylabel(Alcohol, fontsize12)plt.tight_layout()plt.text(0, -0.2,sOD280/OD315 of diluted wines,hacenter,vacenter,fontsize12,transformaxarr[1].transAxes)plt.show()从结果图看起来Adaboost决策边界比单层决策树复杂得多五、分层模型集成框架stacking(叠加算法)Stacking集成算法可以理解为一个两层的集成第一层含有一个分类器把预测的结果(元特征)提供给第二层 而第二层的分类器通常是逻辑回归他把一层分类器的结果当做特征做拟合输出预测结果。过程如下图:标准的Stacking也叫Blending如下图但是标准的Stacking会导致信息泄露所以推荐以下Satcking算法由于目前sklearn没有Stacking相关的类因此我们使用mlxtend库详细代码内容查看http://rasbt.github.io/mlxtend/user_guide/classifier/StackingClassifier/http://rasbt.github.io/mlxtend/user_guide/classifier/StackingCVClassifier/1. 简单堆叠3折CV分类## 1. 简单堆叠3折CV分类from sklearn import datasetsiris datasets.load_iris()X, y iris.data[:, 1:3], iris.targetfrom sklearn.model_selection import cross_val_scorefrom sklearn.linear_model import LogisticRegressionfrom sklearn.neighbors import KNeighborsClassifierfrom sklearn.naive_bayes import GaussianNBfrom sklearn.ensemble import RandomForestClassifierfrom mlxtend.classifier import StackingCVClassifierRANDOM_SEED 42clf1 KNeighborsClassifier(n_neighbors1)clf2 RandomForestClassifier(random_stateRANDOM_SEED)clf3 GaussianNB()lr LogisticRegression()# Starting from v0.16.0, StackingCVRegressor supports# random_state to get deterministic result.sclf StackingCVClassifier(classifiers[clf1, clf2, clf3], # 第一层分类器 meta_classifierlr, # 第二层分类器 random_stateRANDOM_SEED)print(3-fold cross validation:\n)for clf, label in zip([clf1, clf2, clf3, sclf], [KNN, Random Forest, Naive Bayes,StackingClassifier]): scores cross_val_score(clf, X, y, cv3, scoringaccuracy) print(Accuracy: %0.2f (/- %0.2f) [%s] % (scores.mean(), scores.std(), label))我们画出决策边界## 我们画出决策边界from mlxtend.plotting import plot_decision_regionsimport matplotlib.gridspec as gridspecimport itertoolsgs gridspec.GridSpec(2, 2)fig plt.figure(figsize(10,8))for clf, lab, grd in zip([clf1, clf2, clf3, sclf], [KNN, Random Forest, Naive Bayes, StackingCVClassifier], itertools.product([0, 1], repeat2)): clf.fit(X, y) ax plt.subplot(gs[grd[0], grd[1]]) fig plot_decision_regions(XX, yy, clfclf) plt.title(lab)plt.show()2.使用概率作为元特征## 2.使用概率作为元特征clf1 KNeighborsClassifier(n_neighbors1)clf2 RandomForestClassifier(random_state1)clf3 GaussianNB()lr LogisticRegression()sclf StackingCVClassifier(classifiers[clf1, clf2, clf3], use_probasTrue, meta_classifierlr, random_state42)print(3-fold cross validation:\n)for clf, label in zip([clf1, clf2, clf3, sclf], [KNN, Random Forest, Naive Bayes, StackingClassifier]): scores cross_val_score(clf, X, y, cv3, scoringaccuracy) print(Accuracy: %0.2f (/- %0.2f) [%s] % (scores.mean(), scores.std(), label))3. 堆叠5折CV分类与网格搜索(结合网格搜索调参优化)## 3. 堆叠5折CV分类与网格搜索(结合网格搜索调参优化)from sklearn.linear_model import LogisticRegressionfrom sklearn.neighbors import KNeighborsClassifierfrom sklearn.naive_bayes import GaussianNBfrom sklearn.ensemble import RandomForestClassifierfrom sklearn.model_selection import GridSearchCVfrom mlxtend.classifier import StackingCVClassifier# Initializing modelsclf1 KNeighborsClassifier(n_neighbors1)clf2 RandomForestClassifier(random_stateRANDOM_SEED)clf3 GaussianNB()lr LogisticRegression()sclf StackingCVClassifier(classifiers[clf1, clf2, clf3], meta_classifierlr, random_state42)params {kneighborsclassifier__n_neighbors: [1, 5], randomforestclassifier__n_estimators: [10, 50], meta_classifier__C: [0.1, 10.0]}grid GridSearchCV(estimatorsclf, param_gridparams, cv5, refitTrue)grid.fit(X, y)cv_keys (mean_test_score, std_test_score, params)for r, _ in enumerate(grid.cv_results_[mean_test_score]): print(%0.3f /- %0.2f %r % (grid.cv_results_[cv_keys[0]][r], grid.cv_results_[cv_keys[1]][r] / 2.0, grid.cv_results_[cv_keys[2]][r]))print(Best parameters: %s % grid.best_params_)print(Accuracy: %.2f % grid.best_score_)如果我们打算多次使用回归算法我们要做的就是在参数网格中添加一个附加的数字后缀如下所示## 如果我们打算多次使用回归算法我们要做的就是在参数网格中添加一个附加的数字后缀如下所示from sklearn.model_selection import GridSearchCV# Initializing modelsclf1 KNeighborsClassifier(n_neighbors1)clf2 RandomForestClassifier(random_stateRANDOM_SEED)clf3 GaussianNB()lr LogisticRegression()sclf StackingCVClassifier(classifiers[clf1, clf1, clf2, clf3], meta_classifierlr, random_stateRANDOM_SEED)params {kneighborsclassifier-1__n_neighbors: [1, 5], kneighborsclassifier-2__n_neighbors: [1, 5], randomforestclassifier__n_estimators: [10, 50], meta_classifier__C: [0.1, 10.0]}grid GridSearchCV(estimatorsclf, param_gridparams, cv5, refitTrue)grid.fit(X, y)cv_keys (mean_test_score, std_test_score, params)for r, _ in enumerate(grid.cv_results_[mean_test_score]): print(%0.3f /- %0.2f %r % (grid.cv_results_[cv_keys[0]][r], grid.cv_results_[cv_keys[1]][r] / 2.0, grid.cv_results_[cv_keys[2]][r]))print(Best parameters: %s % grid.best_params_)print(Accuracy: %.2f % grid.best_score_)4.在不同特征子集上运行的分类器的堆叠## 4.在不同特征子集上运行的分类器的堆叠###不同的1级分类器可以适合训练数据集中的不同特征子集。以下示例说明了如何使用scikit-learn管道和ColumnSelectorfrom sklearn.datasets import load_irisfrom mlxtend.classifier import StackingCVClassifierfrom mlxtend.feature_selection import ColumnSelectorfrom sklearn.pipeline import make_pipelinefrom sklearn.linear_model import LogisticRegressioniris load_iris()X iris.datay iris.targetpipe1 make_pipeline(ColumnSelector(cols(0, 2)), # 选择第0,2列 LogisticRegression())pipe2 make_pipeline(ColumnSelector(cols(1, 2, 3)), # 选择第1,2,3列 LogisticRegression())sclf StackingCVClassifier(classifiers[pipe1, pipe2], meta_classifierLogisticRegression(), random_state42)sclf.fit(X, y)5.ROC曲线 decision_function## 5.ROC曲线 decision_function### 像其他scikit-learn分类器一样它StackingCVClassifier具有decision_function可用于绘制ROC曲线的方法。### 请注意decision_function期望并要求元分类器实现decision_function。from sklearn import model_selectionfrom sklearn.linear_model import LogisticRegressionfrom sklearn.neighbors import KNeighborsClassifierfrom sklearn.svm import SVCfrom sklearn.ensemble import RandomForestClassifierfrom mlxtend.classifier import StackingCVClassifierfrom sklearn.metrics import roc_curve, aucfrom sklearn.model_selection import train_test_splitfrom sklearn import datasetsfrom sklearn.preprocessing import label_binarizefrom sklearn.multiclass import OneVsRestClassifieriris datasets.load_iris()X, y iris.data[:, [0, 1]], iris.target# Binarize the outputy label_binarize(y, classes[0, 1, 2])n_classes y.shape[1]RANDOM_SEED 42X_train, X_test, y_train, y_test train_test_split( X, y, test_size0.33, random_stateRANDOM_SEED)clf1 LogisticRegression()clf2 RandomForestClassifier(random_stateRANDOM_SEED)clf3 SVC(random_stateRANDOM_SEED)lr LogisticRegression()sclf StackingCVClassifier(classifiers[clf1, clf2, clf3], meta_classifierlr)# Learn to predict each class against the otherclassifier OneVsRestClassifier(sclf)y_score classifier.fit(X_train, y_train).decision_function(X_test)# Compute ROC curve and ROC area for each classfpr dict()tpr dict()roc_auc dict()for i in range(n_classes): fpr[i], tpr[i], _ roc_curve(y_test[:, i], y_score[:, i]) roc_auc[i] auc(fpr[i], tpr[i])# Compute micro-average ROC curve and ROC areafpr[micro], tpr[micro], _ roc_curve(y_test.ravel(), y_score.ravel())roc_auc[micro] auc(fpr[micro], tpr[micro])plt.figure()lw 2plt.plot(fpr[2], tpr[2], colordarkorange, lwlw, labelROC curve (area %0.2f) % roc_auc[2])plt.plot([0, 1], [0, 1], colornavy, lwlw, linestyle--)plt.xlim([0.0, 1.0])plt.ylim([0.0, 1.05])plt.xlabel(False Positive Rate)plt.ylabel(True Positive Rate)plt.title(Receiver operating characteristic example)plt.legend(loclower right)plt.show()本文PDF电子版 后台回复集成学习获取“整理不易点赞三连↓
