pandas知識點（處理缺失資料）
阿新 • • 發佈：2018-12-17
pandas使用浮點值NaN表示浮點和非浮點陣列中的缺失資料：
In [14]: string_data = Series(['aardvark','artichoke',np.nan,'avocado']) In [15]: string_data Out[15]: 0 aardvark 1 artichoke 2 NaN 3 avocado dtype: object In [16]: string_data.isnull() Out[16]: 0 False 1 False 2 True 3 False dtype: bool
python內建的None值也會被當作NA處理：
In [17]: string_data[0] = None In [18]: string_data.isnull() Out[18]: 0 True 1 False 2 True 3 False dtype: bool
1.濾除缺失資料 dropna用於返回一個僅含非空資料和索引值的Series：
In [20]: data = Series([1,NA,3.5,NA,7]) In [21]: data.dropna() Out[21]: 0 1.0 2 3.5 4 7.0 dtype: float64
通過布林型索引亦可以：
In [22]: data[data.notnull()] Out[22]: 0 1.0 2 3.5 4 7.0 dtype: float64
對於DataFrame，dropna預設丟棄含有缺失值的行
In [23]: data = DataFrame([[1.,6.5,3.],[1.,NA,NA],[NA,NA,NA],[NA,6.5,3.]]) In [24]: cleaned = data.dropna() In [25]: data Out[25]: 0 1 2 0 1.0 6.5 3.0 1 1.0 NaN NaN 2 NaN NaN NaN 3 NaN 6.5 3.0 In [26]: cleaned Out[26]: 0 1 2 0 1.0 6.5 3.0
傳入how="all"丟棄全為NaN的行：
In [27]: data.dropna(how="all") Out[27]: 0 1 2 0 1.0 6.5 3.0 1 1.0 NaN NaN 3 NaN 6.5 3.0
丟棄含有NaN的列：
In [28]: data[4] = NA In [29]: data Out[29]: 0 1 2 4 0 1.0 6.5 3.0 NaN 1 1.0 NaN NaN NaN 2 NaN NaN NaN NaN 3 NaN 6.5 3.0 NaN In [30]: data.dropna(axis=1,how="all") Out[30]: 0 1 2 0 1.0 6.5 3.0 1 1.0 NaN NaN 2 NaN NaN NaN 3 NaN 6.5 3.0
thresh可以選擇過濾含有多少有用資料的行
In [41]: df Out[41]: 0 1 2 0 -0.184676 NaN NaN 1 0.565214 NaN NaN 2 0.440203 NaN NaN 3 0.188283 NaN 0.146847 4 1.696903 NaN 0.554640 5 -1.287915 0.139527 -0.494558 6 0.854922 0.299511 0.773247 In [42]: df.dropna(thresh=2) # thresh=2表示至少有兩個非空資料 Out[42]: 0 1 2 3 0.188283 NaN 0.146847 4 1.696903 NaN 0.554640 5 -1.287915 0.139527 -0.494558 6 0.854922 0.299511 0.773247 In [43]: df.dropna(thresh=1) Out[43]: 0 1 2 0 -0.184676 NaN NaN 1 0.565214 NaN NaN 2 0.440203 NaN NaN 3 0.188283 NaN 0.146847 4 1.696903 NaN 0.554640 5 -1.287915 0.139527 -0.494558 6 0.854922 0.299511 0.773247
2.填充缺失資料 fillna是最主要的填充預設資料的方法：
In [9]: df.fillna(0) Out[9]: 0 1 2 0 0.863556 0.000000 0.000000 1 -0.099558 0.000000 0.000000 2 -0.605804 0.000000 0.000000 3 -0.934688 0.000000 -1.198976 4 0.741383 0.000000 0.229845 5 -1.415495 0.511485 -0.086808 6 -0.748325 0.437964 -2.458319
通過字典呼叫fillna可以實現不同列填充不同值：
In [11]: df.fillna({1:0.5,2:-1})
Out[11]:
0 1 2
0 0.863556 0.500000 -1.000000
1 -0.099558 0.500000 -1.000000
2 -0.605804 0.500000 -1.000000
3 -0.934688 0.500000 -1.198976
4 0.741383 0.500000 0.229845
5 -1.415495 0.511485 -0.086808
6 -0.748325 0.437964 -2.458319