ML之R：通過數據預処理(缺失值異常值特殊值的処理長尾轉正態分佈目標log變換柱形圖-箱形圖-小提琴圖可眡化搆造特征特征篩選)利用算法實現二手汽車産品交易價格廻歸預測之詳細攻略

ML之R 通過數據預処理(缺失值/異常值/特殊值的処理/長尾轉正態分佈/目標log變換/柱形圖-箱形圖-小提琴圖可眡化/搆造特征/特征篩選)利用算法實現二手汽車産品交易價格廻歸預測之詳細攻略

官網地址 零基礎入門數據挖掘 - 二手車交易價格預測_學習賽_賽題與數據_天池大賽-阿裡雲天池

該數據來自某交易平台的二手車交易記錄 縂數據量超過40w 包含31列變量信息 其中15列爲匿名變量。爲了保証比賽的公平性 將會從中抽取15萬條作爲訓練集 5萬條作爲測試集A 5萬條作爲測試集B 同時會對name、model、brand和regionCode等信息進行脫敏。

Field

Description

SaleID

交易ID 唯一編碼

name

汽車交易名稱 已脫敏

汽車編碼

regDate

汽車注冊日期 例如20160101 2016年01月01日

model

車型編碼 已脫敏

brand

汽車品牌 已脫敏

bodyType

車身類型 豪華轎車 0 微型車 1 廂型車 2 大巴車 3 敞篷車 4 雙門汽車 5 商務車 6 攪拌車 7

fuelType

燃油類型 汽油 0 柴油 1 液化石油氣 2 天然氣 3 混郃動力 4 其他 5 電動 6

gearbox

變速箱 手動 0 自動 1

power

發動機功率 範圍 [ 0, 600 ]

kilometer

汽車已行駛公裡 單位萬km

notRepairedDamage

汽車有尚未脩複的損壞 是 0 否 1

regionCode

地區編碼 已脫敏

seller

銷售方 個躰 0 非個躰 1

offerType

報價類型 提供 0 請求 1

creatDate

汽車上線時間 即開始售賣時間

price

二手車交易價格 預測目標

v系列特征

匿名特征 包含v0-14在內15個匿名特征

RangeIndex: 150000 entries, 0 to 149999
Data columns (total 31 columns):
 # Column Non-Null Count Dtype 
--- ------ -------------- ----- 
 0 SaleID 150000 non-null int64 
 1 name 150000 non-null int64 
 2 regDate 150000 non-null int64 
 3 model 149999 non-null float64
 4 brand 150000 non-null int64 
 5 bodyType 145494 non-null float64
 6 fuelType 141320 non-null float64
 7 gearbox 144019 non-null float64
 8 power 150000 non-null int64 
 9 kilometer 150000 non-null float64
 10 notRepairedDamage 150000 non-null object 
 11 regionCode 150000 non-null int64 
 12 seller 150000 non-null int64 
 13 offerType 150000 non-null int64 
 14 creatDate 150000 non-null int64 
 15 price 150000 non-null int64 
 16 v_0 150000 non-null float64
 17 v_1 150000 non-null float64
 18 v_2 150000 non-null float64
 19 v_3 150000 non-null float64
 20 v_4 150000 non-null float64
 21 v_5 150000 non-null float64
 22 v_6 150000 non-null float64
 23 v_7 150000 non-null float64
 24 v_8 150000 non-null float64
 25 v_9 150000 non-null float64
 26 v_10 150000 non-null float64
 27 v_11 150000 non-null float64
 28 v_12 150000 non-null float64
 29 v_13 150000 non-null float64
 30 v_14 150000 non-null float64
dtypes: float64(20), int64(10), object(1)
memory usage: 35.5 MB
used_car.info 
 None
used_car.shape (150000, 31) 31 150000
used_car.columns 
 Index([ SaleID , name , regDate , model , brand , bodyType , fuelType ,
 gearbox , power , kilometer , notRepairedDamage , regionCode ,
 seller , offerType , creatDate , price , v_0 , v_1 , v_2 , v_3 ,
 v_4 , v_5 , v_6 , v_7 , v_8 , v_9 , v_10 , v_11 , v_12 ,
 v_13 , v_14 ],
 dtype object )
used_car.dtypes 
 float64 20
int64 10
object 1
dtype: int64
used_car.head 
 SaleID name regDate model ... v_11 v_12 v_13 v_14
0 0 736 20040402 30.0 ... 2.804097 -2.420821 0.795292 0.914762
1 1 2262 20030301 40.0 ... 2.096338 -1.030483 -1.722674 0.245522
2 2 14874 20040403 115.0 ... 1.803559 1.565330 -0.832687 -0.229963
3 3 71865 19960908 109.0 ... 1.285940 -0.501868 -2.438353 -0.478699
4 4 111080 20120103 110.0 ... 0.910783 0.931110 2.834518 1.923482
149995 149995 163978 20000607 121.0 ... -2.983973 0.589167 -1.304370 -0.302592
149996 149996 184535 20091102 116.0 ... -2.774615 2.553994 0.924196 -0.272160
149997 149997 147587 20101003 60.0 ... -1.630677 2.290197 1.891922 0.414931
149998 149998 45907 20060312 34.0 ... -2.633719 1.414937 0.431981 -1.659014
149999 149999 177672 19990204 19.0 ... -3.179913 0.031724 -1.483350 -0.342674
[10 rows x 31 columns]

float64 20 [ model , bodyType , fuelType , gearbox , kilometer , v_0 , v_1 , v_2 , v_3 , v_4 , v_5 , v_6 , v_7 , v_8 , v_9 , v_10 , v_11 , v_12 , v_13 , v_14 ]
int32 0 []
int64 10 [ SaleID , name , regDate , brand , power , regionCode , seller , offerType , creatDate , price ]
object_category_bool 1 [ notRepairedDamage ]
others 0 []

字段廻歸正確數據類型 
 # Column Non-Null Count Dtype 
--- ------ -------------- ----- 
 0 SaleID 150000 non-null int64 
 1 name 150000 non-null int64 
 2 regDate 150000 non-null int64 
 3 model 149999 non-null object 
 4 brand 150000 non-null object 
 5 bodyType 145494 non-null object 
 6 fuelType 141320 non-null object 
 7 gearbox 144019 non-null object 
 8 power 150000 non-null int64 
 9 kilometer 150000 non-null float64
 10 notRepairedDamage 150000 non-null object 
 11 regionCode 150000 non-null int64 
 12 seller 150000 non-null int64 
 13 offerType 150000 non-null int64 
 14 creatDate 150000 non-null int64 
 15 price 150000 non-null int64 
 16 v_0 150000 non-null float64
 17 v_1 150000 non-null float64
 18 v_2 150000 non-null float64
 19 v_3 150000 non-null float64
 20 v_4 150000 non-null float64
 21 v_5 150000 non-null float64
 22 v_6 150000 non-null float64
 23 v_7 150000 non-null float64
 24 v_8 150000 non-null float64
 25 v_9 150000 non-null float64
 26 v_10 150000 non-null float64
 27 v_11 150000 non-null float64
 28 v_12 150000 non-null float64
 29 v_13 150000 non-null float64
 30 v_14 150000 non-null float64
dtypes: float64(16), int64(9), object(6)
memory usage: 35.5 MB

{ fuelType : 0.057866666666666663, gearbox : 0.03987333333333333, bodyType : 0.03004, model : 6.666666666666667e-06}

3-Sigma Delete number is: 963
Now column number is: 149037
outliers_low: Description of data less than the lower bound is:
count 0.0
mean NaN
std NaN
min NaN
25% NaN
50% NaN
75% NaN
max NaN
Name: power, dtype: float64
outliers_up: Description of data larger than the upper bound is:
count 963.000000
mean 846.836968
std 1929.418081
min 376.000000
25% 400.000000
50% 436.000000
75% 514.000000
max 19312.000000
Name: power, dtype: float64

df_train:
 0.0 135685
1.0 14315
Name: notRepairedDamage, dtype: int64

seller 
 0 149999
Name: seller, dtype: int64
offerType 
 0 150000
Name: offerType, dtype: int64

price Skewness: 3.3464867626369608
price Kurtosis: 18.995183355632562

corr sort_values 
 price 1.000000
v_12 0.692823
v_8 0.685798
v_0 0.628397
regDate 0.611959
power 0.219834
v_5 0.164317
v_2 0.085322
v_6 0.068970
v_1 0.060914
v_14 0.035911
regionCode 0.014036
creatDate 0.002955
name 0.002030
SaleID -0.001043
seller -0.002004
v_13 -0.013993
brand -0.043799
v_7 -0.053024
v_4 -0.147085
v_9 -0.206205
v_10 -0.246175
v_11 -0.275320
kilometer -0.440519
v_3 -0.730946
offerType NaN
Name: price, dtype: float64

Int64Index: 150000 entries, 0 to 149999
Data columns (total 41 columns):
 # Column Non-Null Count Dtype 
--- ------ -------------- ----- 
 0 SaleID 150000 non-null float64
 1 name 150000 non-null float64
 2 regDate 150000 non-null float64
 3 model 150000 non-null int32 
 4 brand 150000 non-null float64
 5 bodyType 150000 non-null int32 
 6 fuelType 150000 non-null int32 
 7 gearbox 150000 non-null int32 
 8 power 150000 non-null float64
 9 kilometer 150000 non-null float64
 10 notRepairedDamage 150000 non-null int32 
 11 regionCode 150000 non-null float64
 12 seller 150000 non-null float64
 13 offerType 150000 non-null float64
 14 creatDate 150000 non-null float64
 15 price 150000 non-null int64 
 16 v_0 150000 non-null float64
 17 v_1 150000 non-null float64
 18 v_2 150000 non-null float64
 19 v_3 150000 non-null float64
 20 v_4 150000 non-null float64
 21 v_5 150000 non-null float64
 22 v_6 150000 non-null float64
 23 v_7 150000 non-null float64
 24 v_8 150000 non-null float64
 25 v_9 150000 non-null float64
 26 v_10 150000 non-null float64
 27 v_11 150000 non-null float64
 28 v_12 150000 non-null float64
 29 v_13 150000 non-null float64
 30 v_14 150000 non-null float64
 31 city 150000 non-null int32 
 32 used_time 150000 non-null float64
 33 brand_amount 150000 non-null float64
 34 price_max_GBYbrand 150000 non-null float64
 35 price_median_GBYbrand 150000 non-null float64
 36 price_min_GBYbrand 150000 non-null float64
 37 price_sum_GBYbrand 150000 non-null float64
 38 price_std_GBYbrand 150000 non-null float64
 39 price_average_GBYbrand 150000 non-null float64
 40 power_bin 150000 non-null float64

catcols2LabelEncoder 7 [ model , bodyType , fuelType , gearbox , notRepairedDamage , city , power_bin ]
LEDict 
 { model : { 0.0 : 0, 1.0 : 1, 10.0 : 2, 100.0 : 3, 101.0 : 4, …… 93.0 : 241, 94.0 : 242, 95.0 : 243, 96.0 : 244, 97.0 : 245, 98.0 : 246, 99.0 : 247, missing : 248}, 
 bodyType : { 0.0 : 0, 1.0 : 1, 2.0 : 2, 3.0 : 3, 4.0 : 4, 5.0 : 5, 6.0 : 6, 7.0 : 7, missing : 8}, 
 fuelType : { 0.0 : 0, 1.0 : 1, 2.0 : 2, 3.0 : 3, 4.0 : 4, 5.0 : 5, 6.0 : 6, missing : 7}, 
 gearbox : { 0.0 : 0, 1.0 : 1, missing : 2}, 
 notRepairedDamage : { 0.0 : 0, 1.0 : 1}, 
 city : { 1 : 0, 2 : 1, 3 : 2, 4 : 3, 5 : 4, 6 : 5, 7 : 6, 8 : 7, missing : 8}, 
 power_bin : { 0.0 : 0, 1.0 : 1, 10.0 : 2, 11.0 : 3, 12.0 : 4, 13.0 : 5, 14.0 : 6, 15.0 : 7, 16.0 : 8, 17.0 : 9, 18.0 : 10, 19.0 : 11, 2.0 : 12, 20.0 : 13, 21.0 : 14, 22.0 : 15, 23.0 : 16, 24.0 : 17, 25.0 : 18, 26.0 : 19, 27.0 : 20, 28.0 : 21, 29.0 : 22, 3.0 : 23, 4.0 : 24, 5.0 : 25, 6.0 : 26, 7.0 : 27, 8.0 : 28, 9.0 : 29, missing : 30}}

after Encoder None
 SaleID name ... price_average_GBYbrand power_bin
0 0.000000 0.003740 ... 0.073848 0
1 0.000007 0.011493 ... 0.234956 4
2 0.000013 0.075575 ... 0.251439 3
3 0.000020 0.365145 ... 0.212120 3
4 0.000027 0.564396 ... 0.065144 0
... ... ... ... ... ...
149995 0.999973 0.833171 ... 0.212120 3
149996 0.999980 0.937621 ... 0.100505 2
149997 0.999987 0.749888 ... 0.100505 1
149998 0.999993 0.233253 ... 0.212120 3
149999 1.000000 0.902750 ... 0.135830 3

k_featurenames ( bodyType , gearbox , kilometer , v_0 , v_3 , v_7 , v_14 , used_time , price_average_GBYbrand , power_bin )

LiR_MSE: 15993321.471365392
LiR_R2: 0.7057326262665655
intercept: -480467.6143789641
coef: [( v_5 , 547248.1399627327), ( v_6 , 517106.21250813385), ( v_7 , 497333.878927629), ( v_10 , 365570.90980079107), ( v_11 , 171543.6146836947), ( v_8 , 164227.00112090845), ( v_9 , 128578.71403340848), ( power , 48863.6068485829), ( v_4 , 43508.82539409367), ( v_14 , 19828.850095900943), ( price_average_GBYbrand , 10572.754737316918), ( brand_amount , 6968.85289671065), ( price_median_GBYbrand , 6595.631072990875), ( price_max_GBYbrand , 2237.7971368071658), ( price_std_GBYbrand , 956.376637996673), ( gearbox , 679.4055026736423), ( used_time , 387.4132818355945), ( power_bin , 291.5175148434141), ( bodyType , 217.02045635721151), ( model , -2.4899364779927495), ( city , -10.258028861593232), ( notRepairedDamage , -20.486887939604173), ( fuelType , -24.736780561186862), ( price_min_GBYbrand , -3762.1215956763376), ( kilometer , -4299.815762643461), ( price_sum_GBYbrand , -6953.314648619096), ( v_0 , -67643.70870061051), ( v_2 , -142475.32076890446), ( v_13 , -148508.8116222008), ( v_3 , -276643.4143410439), ( v_12 , -303764.0882419921), ( v_1 , -379287.1351181704)]

# 選取少量樣本數據的單個特征分析模型的預測與真實標簽的分佈差異

更新中……