b/2015/山田/実験結果
をテンプレートにして作成
[
トップ
] [
新規
|
一覧
|
単語検索
|
最終更新
|
ヘルプ
|
ログイン
]
開始行:
このページに詳しい実験結果をまとめていくことにします。&br;
#contents
備考
-ex20151130から、ネットワークへ入力する画像から教師画像の平均画像を引く正規化処理を導入
*CIFAR10 [#c92a4213]
**ex20151130_入力画像から「訓練用データの」平均画像を引く正規化を行った際の学習への影響を見る実験[#r416d0c0]
-比較対象は、正規化の有無以外同条件である[[b/2015/山田/実験結果#w8778ff0]]の「C[0.01]-P-C[0.01]-P-F[0.01]-F[0.01]」。
-学習推移の比較( https://drive.google.com/open?id=0B9W18yqQO6JAclhsdk1maW9mblU )
-正規化なしとは誤差レベルの違いしか見られず。少なくとも悪影響は無さそうなので、先人に倣って今後はこれを導入して実験を行っていく。
-[[#ex20151201]]も参照。有効な結果を確認できた。
#pre{{
with meansubnormalize
############# training condition ############
200 epoch training. input_size: (28, 28) , minibatch_size: 100
, learning_rate: 0.01 , momentum: 0.9 , weight_decay: 0.0005
#############################################
########## architecture of the CNN #########
layer1 - Convolution
< kernel_size: (50, 2, 2) , stride: (1, 1) , act_func: ReLU
, border_mode: valid , out_size: (50, 27, 27) , iniW_σ: 0.010000 >
layer2 - Convolution
< kernel_size: (50, 2, 2) , stride: (1, 1) , act_func: ReLU
, border_mode: valid , out_size: (50, 26, 26) , iniW_σ: 0.010000 >
layer3 - Pooling
< downscale: (4, 4) , stride: None , out_size: (50, 6, 6) >
layer4 - fully-connected
< number_of_units: 1000 , drop_rate: 0.8 , act_func: ReLU , iniW_σ: 0.010000 >
layer5 - fully-connected
< number_of_units: 10 , drop_rate: 0.5 , act_func: softmax , iniW_σ: 0.010000 >
###########################################
epoch XE(L) error(L) XE(T) error(T)
1 2.301241 0.890720 2.292954 0.871600
2 2.168675 0.793220 2.040033 0.743200
3 2.024148 0.738300 1.988409 0.730100
4 1.983140 0.723320 1.930618 0.694700
5 1.943590 0.710660 1.917150 0.694300
6 1.915704 0.696460 1.898298 0.694300
7 1.910534 0.694820 1.880623 0.686200
8 1.886874 0.683620 1.851273 0.660600
9 1.855617 0.668620 1.812055 0.645800
10 1.814704 0.653100 1.737229 0.618800
11 1.757508 0.627840 1.655818 0.587100
12 1.691635 0.604380 1.590292 0.563700
13 1.621381 0.579320 1.519763 0.539800
14 1.559681 0.557720 1.484703 0.532600
15 1.511344 0.542200 1.427634 0.510600
16 1.490780 0.532740 1.389791 0.492400
17 1.421070 0.507820 1.333367 0.470100
18 1.416972 0.507580 1.379368 0.490700
19 1.361172 0.484840 1.304148 0.462600
20 1.387499 0.496060 1.308432 0.461300
21 1.328294 0.474740 1.262993 0.450000
22 1.330144 0.473340 1.223739 0.428200
23 1.307318 0.465260 1.235810 0.444800
24 1.256990 0.446860 1.219263 0.424700
25 1.253362 0.444860 1.181059 0.412200
26 1.226874 0.435160 1.179139 0.414000
27 1.228463 0.437640 1.153660 0.402300
28 1.202043 0.426640 1.124669 0.391800
29 1.198916 0.424820 1.102292 0.391800
30 1.130790 0.399460 1.040838 0.362700
31 1.091429 0.385860 1.014462 0.350100
32 1.081484 0.381800 1.005059 0.348500
33 1.051955 0.369900 0.999547 0.350100
34 1.039517 0.364420 0.950747 0.325100
35 1.052473 0.366600 0.954961 0.329400
36 1.014120 0.355900 0.919145 0.318900
37 0.953343 0.334160 0.910735 0.316300
38 0.989915 0.347160 0.918302 0.317700
39 0.939300 0.329020 0.863239 0.295000
40 0.948976 0.331720 0.856169 0.293500
41 0.927161 0.323480 0.854197 0.297600
42 0.926378 0.323800 0.841668 0.290400
43 0.880868 0.307880 0.840756 0.290400
44 0.832429 0.292900 0.811874 0.279500
45 0.900572 0.314300 0.818376 0.285300
46 0.845470 0.296120 0.799914 0.274000
47 0.885975 0.311020 0.794399 0.271900
48 0.817159 0.286700 0.793099 0.272200
49 0.822162 0.287260 0.767546 0.265200
50 0.846227 0.295780 0.749083 0.254400
51 0.781471 0.270340 0.749934 0.260200
52 0.768634 0.267900 0.734875 0.250900
53 0.767662 0.270420 0.736494 0.252000
54 0.784999 0.273580 0.731604 0.248100
55 0.784041 0.272800 0.742787 0.253100
56 0.741014 0.256840 0.722091 0.246900
57 0.722639 0.252700 0.714930 0.245900
58 0.779533 0.270560 0.720464 0.250400
59 0.716955 0.249340 0.704772 0.240200
60 0.743085 0.258100 0.714782 0.240600
61 0.724373 0.252660 0.709176 0.239400
62 0.694776 0.242980 0.690473 0.236400
63 0.659292 0.230280 0.692154 0.236800
64 0.707746 0.247520 0.690970 0.233300
65 0.726773 0.254160 0.677306 0.232100
66 0.670460 0.235280 0.687865 0.235300
67 0.774429 0.270280 0.709229 0.243500
68 0.630434 0.220900 0.687857 0.234900
69 0.804328 0.279860 0.695814 0.236000
70 0.621518 0.218140 0.705632 0.240000
71 0.591258 0.207580 0.677732 0.231600
72 0.711016 0.249800 0.697816 0.239600
73 0.664425 0.232780 0.681117 0.230300
74 0.566506 0.198280 0.672950 0.230500
75 0.676259 0.236580 0.672809 0.228100
76 0.702371 0.245780 0.665470 0.227700
77 0.560746 0.196580 0.670936 0.229600
78 0.628251 0.219180 0.652131 0.223700
79 0.538038 0.188640 0.658130 0.222300
80 0.571354 0.201200 0.656089 0.222700
81 0.640938 0.224640 0.668623 0.232200
82 0.578572 0.202680 0.673226 0.228000
83 0.680907 0.239160 0.658503 0.226400
84 0.606439 0.211320 0.659322 0.225100
85 0.571947 0.200340 0.642748 0.222300
86 0.575176 0.202420 0.654706 0.222200
87 0.576259 0.201920 0.640902 0.215800
88 0.555670 0.196180 0.638271 0.219000
89 0.729043 0.254560 0.647124 0.218200
90 0.668291 0.234420 0.656433 0.223500
91 0.522506 0.185740 0.646257 0.219200
92 0.665435 0.229840 0.650611 0.220400
93 0.653686 0.227200 0.643800 0.219100
94 0.639696 0.222820 0.631171 0.216400
95 0.636667 0.223300 0.627640 0.213700
96 0.564452 0.199560 0.619440 0.211000
97 0.596317 0.208160 0.656351 0.223800
98 0.559324 0.196600 0.644581 0.217300
99 0.579977 0.203060 0.625384 0.211800
100 0.552774 0.195240 0.623098 0.211900
101 0.471780 0.167000 0.621501 0.212100
102 0.458696 0.163740 0.624754 0.210700
103 0.643440 0.225180 0.626696 0.212900
104 0.559983 0.197260 0.627611 0.215200
105 0.526930 0.185320 0.631127 0.214200
106 0.565367 0.200220 0.621152 0.212800
107 0.528703 0.184320 0.612028 0.209400
108 0.528290 0.187840 0.618490 0.215200
109 0.508079 0.179640 0.616074 0.211700
110 0.470439 0.166120 0.623196 0.209700
111 0.461730 0.162100 0.622962 0.212600
112 0.646683 0.226200 0.632754 0.218700
113 0.590706 0.206960 0.623041 0.213900
114 0.586474 0.205080 0.634192 0.212600
115 0.559189 0.197520 0.636560 0.215400
116 0.446040 0.156540 0.619566 0.210000
117 0.592400 0.209120 0.629333 0.214900
118 0.513011 0.179940 0.644279 0.218200
119 0.513977 0.180880 0.603127 0.204400
120 0.449013 0.160020 0.605747 0.208700
121 0.571244 0.201660 0.630140 0.215100
122 0.511436 0.179580 0.620652 0.209400
123 0.626654 0.221100 0.642481 0.217600
124 0.450823 0.159140 0.602686 0.202000
125 0.494492 0.175680 0.609670 0.206300
126 0.512142 0.181080 0.603773 0.202400
127 0.420901 0.146580 0.631388 0.211400
128 0.533896 0.188200 0.615401 0.211500
129 0.487292 0.172520 0.611321 0.204900
130 0.599748 0.210160 0.626388 0.211600
131 0.419864 0.148900 0.609263 0.206000
132 0.545272 0.192700 0.622831 0.212300
133 0.478314 0.170580 0.615337 0.210600
134 0.503807 0.178920 0.609677 0.203400
135 0.472619 0.167720 0.594770 0.199500
136 0.370442 0.132380 0.613169 0.203400
137 0.435198 0.154660 0.611876 0.206900
138 0.503113 0.179100 0.613414 0.203600
139 0.496871 0.175760 0.609763 0.205500
140 0.433943 0.154180 0.605862 0.205300
141 0.411215 0.145440 0.611845 0.202300
142 0.590320 0.209060 0.620617 0.209400
143 0.413181 0.145480 0.611651 0.204000
144 0.447773 0.159780 0.607428 0.202000
145 0.424696 0.150420 0.599058 0.202000
146 0.494046 0.175420 0.603054 0.204000
147 0.470818 0.167180 0.610959 0.202500
148 0.361096 0.128560 0.614718 0.198300
149 0.558248 0.197700 0.617627 0.209700
150 0.395579 0.140160 0.614893 0.201500
151 0.366426 0.130840 0.613265 0.201500
152 0.422831 0.149820 0.613052 0.204900
153 0.491266 0.172360 0.612104 0.203500
154 0.489052 0.172480 0.603320 0.204000
155 0.352787 0.125720 0.610067 0.200400
156 0.390451 0.138740 0.604489 0.199900
157 0.381451 0.135400 0.605064 0.200000
158 0.449463 0.160180 0.613838 0.203000
159 0.452143 0.160840 0.607294 0.204500
160 0.439858 0.157000 0.605615 0.204200
161 0.428808 0.152200 0.622550 0.205300
162 0.400631 0.142540 0.628669 0.207700
163 0.356724 0.126400 0.631125 0.209200
164 0.334470 0.119520 0.638453 0.204500
165 0.531309 0.188160 0.636315 0.216200
166 0.463861 0.164600 0.614799 0.205200
167 0.298837 0.105720 0.617777 0.200500
168 0.283677 0.100900 0.641540 0.201200
169 0.461111 0.162160 0.611954 0.200300
170 0.417671 0.149380 0.605191 0.201000
171 0.364417 0.128500 0.622437 0.204400
172 0.349503 0.125100 0.617939 0.201100
173 0.332663 0.119720 0.625480 0.198300
174 0.637518 0.221040 0.615185 0.205800
175 0.468763 0.164660 0.603146 0.200500
176 0.367030 0.130540 0.610282 0.197400
real 35m4.599s
user 47m11.476s
sys 55m53.392s
}};
**ex20151127-2_層ごとの重み初期値の標準偏差の組み合わせを色々変えて性能を比べる実験 [#w8778ff0]
-教師の推移比較( https://drive.google.com/open?id=0B9W18yqQO6JAREFpQV93ZG5LLVU )
-テストの推移比較( https://drive.google.com/open?id=0B9W18yqQO6JAQXAtVWpoaDdZN3c )
-ILSVRCのようにFCの標準偏差を大きくした方が早くなったりするかと思ったが、そうでもないらしい。パラメータ数やクラス数との兼ね合いの方が大事だということなのかも。Convの標準偏差を大きくしたものとFCのそれを大きくしたものでも結果に大差はないので、この値をそこまで慎重に決める必要はなさそう。
***C[0.001]-P-C[0.001]-P-F[0.001]-F[0.001] [#zb2ae618]
#pre{{
############# training condition ############
500 epoch training. input_size: (28, 28) , minibatch_size: 100
, learning_rate: 0.01 , momentum: 0.9 , weight_decay: 0.0005
#############################################
########## architecture of the CNN #########
layer1 - Convolution
< kernel_size: (50, 2, 2) , stride: (1, 1) , act_func: ReLU
, border_mode: valid , out_size: (50, 27, 27) , iniW_σ: 0.001000 >
layer2 - Convolution
< kernel_size: (50, 2, 2) , stride: (1, 1) , act_func: ReLU
, border_mode: valid , out_size: (50, 26, 26) , iniW_σ: 0.001000 >
layer3 - Pooling
< downscale: (4, 4) , stride: None , out_size: (50, 6, 6) >
layer4 - fully-connected
< number_of_units: 1000 , drop_rate: 0.8 , act_func: ReLU , iniW_σ: 0.001000 >
layer5 - fully-connected
< number_of_units: 10 , drop_rate: 0.5 , act_func: softmax , iniW_σ: 0.001000 >
###########################################
epoch XE(L) error(L) XE(T) error(T)
1 2.302816 0.902240 2.302728 0.900000
2 2.302879 0.901100 2.302805 0.900000
3 2.302906 0.902040 2.302703 0.900000
4 2.302792 0.899660 2.302803 0.900000
5 2.302885 0.901500 2.302837 0.900000
6 2.302877 0.901500 2.302654 0.900000
7 2.302864 0.899980 2.302733 0.900000
8 2.302880 0.901100 2.302828 0.900000
9 2.302816 0.899180 2.302854 0.900000
10 2.302942 0.900980 2.302740 0.900000
11 2.302877 0.900660 2.302797 0.900000
12 2.302951 0.900980 2.302666 0.900000
13 2.302896 0.901380 2.302770 0.900000
14 2.302930 0.903420 2.302701 0.900000
15 2.302886 0.901700 2.302700 0.900000
16 2.302876 0.902320 2.302740 0.900000
17 2.302918 0.900680 2.302705 0.900000
18 2.302859 0.900840 2.302680 0.900000
19 2.302854 0.899220 2.302746 0.900000
20 2.302887 0.902240 2.302685 0.900000
21 2.302853 0.901640 2.302783 0.900000
22 2.302877 0.900240 2.302672 0.900000
23 2.302900 0.903340 2.302670 0.900000
24 2.302900 0.900340 2.302646 0.900000
25 2.302869 0.901900 2.302734 0.900000
26 2.302863 0.899500 2.302859 0.900000
27 2.302896 0.901400 2.302732 0.900000
28 2.302926 0.903920 2.302740 0.900000
29 2.302812 0.898960 2.302872 0.900000
30 2.302941 0.901460 2.302744 0.900000
31 2.302809 0.901120 2.302862 0.900000
32 2.302873 0.903100 2.302745 0.900000
33 2.302815 0.900080 2.302847 0.900000
34 2.302873 0.900560 2.302701 0.900000
35 2.302857 0.901700 2.302606 0.900000
36 2.302829 0.901980 2.302828 0.900000
37 2.302889 0.900820 2.302719 0.900000
38 2.302913 0.901760 2.302720 0.900000
39 2.302820 0.900860 2.302830 0.900000
40 2.302861 0.902580 2.302886 0.900000
41 2.302895 0.901320 2.302770 0.900000
42 2.302911 0.902060 2.302693 0.900000
43 2.302888 0.900960 2.302736 0.900000
44 2.302922 0.900560 2.302662 0.900000
45 2.302896 0.899620 2.302675 0.900000
46 2.302884 0.900000 2.302700 0.900000
47 2.302867 0.902160 2.302755 0.900000
48 2.302843 0.900780 2.302807 0.900000
49 2.302875 0.900600 2.302703 0.900000
50 2.302860 0.902980 2.302767 0.900000
51 2.302904 0.902740 2.302663 0.900000
52 2.302875 0.901020 2.302791 0.900000
53 2.302802 0.900060 2.302787 0.900000
54 2.302905 0.899540 2.302714 0.900000
55 2.302918 0.901180 2.302813 0.900000
56 2.302894 0.903820 2.302661 0.900000
57 2.302897 0.902660 2.302698 0.900000
58 2.302806 0.900980 2.302812 0.900000
59 2.302901 0.902020 2.302679 0.900000
60 2.302860 0.902800 2.302660 0.900000
61 2.302864 0.901540 2.302682 0.900000
62 2.302851 0.901940 2.302773 0.900000
63 2.302877 0.900800 2.302760 0.900000
64 2.302901 0.900280 2.302729 0.900000
65 2.302923 0.901180 2.302698 0.900000
66 2.302878 0.900340 2.302655 0.900000
67 2.302845 0.901600 2.302739 0.900000
68 2.302807 0.900180 2.302745 0.900000
69 2.302879 0.899780 2.302640 0.900000
70 2.302851 0.900020 2.302733 0.900000
71 2.302906 0.902960 2.302744 0.900000
72 2.302900 0.902320 2.302689 0.900000
73 2.302818 0.900200 2.302743 0.900000
74 2.302934 0.902860 2.302675 0.900000
75 2.302927 0.902560 2.302680 0.900000
76 2.302844 0.900960 2.302713 0.900000
77 2.302856 0.901220 2.302770 0.900000
78 2.302773 0.900220 2.302903 0.900000
79 2.302886 0.901360 2.302787 0.900000
80 2.302900 0.899900 2.302687 0.900000
81 2.302858 0.901160 2.302636 0.900000
82 2.302866 0.899960 2.302754 0.900000
83 2.302939 0.901420 2.302622 0.900000
84 2.302832 0.900940 2.302782 0.900000
85 2.302873 0.902020 2.302694 0.900000
86 2.302882 0.900640 2.302688 0.900000
87 2.302813 0.900920 2.302709 0.900000
88 2.302846 0.900920 2.302531 0.900000
89 2.302660 0.900640 2.302236 0.900000
90 2.299378 0.890580 2.284871 0.878900
91 2.168390 0.819720 2.091465 0.776200
92 2.059146 0.761820 2.008044 0.739500
93 2.017725 0.740320 1.989615 0.730800
94 1.997479 0.730460 1.997901 0.730700
95 1.968781 0.717040 1.956786 0.722600
96 1.945655 0.710800 1.912005 0.692400
97 1.931816 0.703500 1.900630 0.701600
98 1.921936 0.699940 1.869197 0.674700
99 1.901791 0.686000 1.873437 0.676900
100 1.885371 0.680580 1.862281 0.672200
101 1.881802 0.682380 1.870674 0.670800
102 1.875387 0.677840 1.838516 0.664500
103 1.861597 0.672400 1.851301 0.662300
104 1.880659 0.685500 1.808609 0.652700
105 1.863076 0.675220 1.824427 0.652400
106 1.847045 0.668020 1.815635 0.649700
107 1.839112 0.662980 1.795213 0.637600
108 1.840379 0.663240 1.784321 0.636000
109 1.825963 0.656440 1.749073 0.617000
110 1.766159 0.632060 1.668041 0.596600
111 1.686609 0.603240 1.595677 0.571300
112 1.616457 0.581920 1.574964 0.565200
113 1.573660 0.563640 1.508577 0.541600
114 1.500858 0.538540 1.428491 0.514200
115 1.434960 0.513880 1.368229 0.481800
116 1.422711 0.507820 1.333455 0.463200
117 1.383827 0.494820 1.329217 0.467700
118 1.377169 0.491680 1.310349 0.461300
119 1.336277 0.476600 1.269550 0.444800
120 1.308907 0.466440 1.221570 0.428600
121 1.259891 0.447300 1.186137 0.417600
122 1.236259 0.439420 1.163775 0.403500
123 1.219182 0.431340 1.171127 0.404900
124 1.217761 0.433320 1.113039 0.385100
125 1.201874 0.424660 1.119029 0.391700
126 1.154899 0.407680 1.121279 0.394500
127 1.142473 0.402260 1.079211 0.374900
128 1.109837 0.389520 1.046161 0.361900
129 1.113610 0.389820 1.012330 0.354200
130 1.081986 0.380940 1.000698 0.349100
131 1.002081 0.350760 0.957500 0.333600
132 0.992582 0.349380 0.945118 0.323600
133 0.967429 0.338860 0.946332 0.329500
134 0.968382 0.341340 0.909104 0.314600
135 0.926163 0.322280 0.909532 0.314900
136 1.016451 0.355060 0.923451 0.320100
137 0.881288 0.310020 0.876305 0.306400
138 0.905802 0.318080 0.863423 0.297400
139 0.895989 0.313400 0.846558 0.292700
140 0.841693 0.294200 0.849797 0.295500
141 0.955008 0.334160 0.831881 0.285500
142 0.836120 0.294480 0.821438 0.286000
143 0.790014 0.276680 0.836626 0.287200
144 0.922634 0.322660 0.803645 0.277300
145 0.808569 0.283180 0.797457 0.273000
146 0.867673 0.300880 0.795595 0.272000
147 0.851220 0.297840 0.803278 0.278500
148 0.841037 0.292420 0.811309 0.281300
149 0.858974 0.299640 0.766421 0.262100
150 0.797129 0.277620 0.779408 0.267600
151 0.831622 0.291940 0.746741 0.256300
152 0.753381 0.263660 0.758080 0.262200
153 0.810279 0.282840 0.765352 0.263800
154 0.842484 0.295760 0.771300 0.263900
155 0.792305 0.277680 0.729488 0.250700
156 0.712752 0.248260 0.728300 0.248900
157 0.698934 0.244480 0.734653 0.251700
158 0.811315 0.283540 0.741308 0.254600
159 0.761607 0.266960 0.729404 0.248800
160 0.764513 0.268940 0.724224 0.248400
161 0.707847 0.247220 0.706810 0.242500
162 0.688261 0.241220 0.713286 0.246000
163 0.748346 0.260660 0.731541 0.254300
164 0.691880 0.242800 0.719255 0.244800
165 0.696499 0.243320 0.697712 0.237700
166 0.757661 0.267740 0.697511 0.239600
167 0.693566 0.243340 0.688757 0.233500
168 0.660016 0.229700 0.693834 0.238200
169 0.621602 0.216960 0.698628 0.237900
170 0.693565 0.241460 0.683942 0.233800
171 0.791773 0.276660 0.718621 0.248300
172 0.694664 0.241320 0.679286 0.230000
173 0.749198 0.260960 0.688338 0.236900
174 0.690450 0.240540 0.695980 0.237200
175 0.670440 0.234040 0.697503 0.239100
176 0.625189 0.219900 0.699741 0.240600
177 0.589703 0.208140 0.681859 0.231800
178 0.643510 0.224760 0.680289 0.235600
179 0.644631 0.226000 0.677113 0.228600
180 0.573221 0.200240 0.667921 0.226300
181 0.591352 0.207860 0.669414 0.225000
182 0.644971 0.225320 0.660583 0.222400
183 0.558789 0.196260 0.649809 0.218600
184 0.719270 0.253800 0.685986 0.233600
185 0.604209 0.212560 0.647094 0.219500
186 0.578746 0.202120 0.651767 0.220500
187 0.594220 0.207120 0.649317 0.219500
188 0.566839 0.198320 0.654618 0.218600
189 0.550815 0.190560 0.640559 0.214600
190 0.736706 0.258080 0.658022 0.223000
191 0.653457 0.231120 0.648714 0.216400
192 0.663878 0.233520 0.654263 0.223300
193 0.556408 0.197080 0.648131 0.219600
194 0.672649 0.235160 0.659517 0.221800
195 0.634843 0.221960 0.657571 0.221900
196 0.532282 0.187980 0.669375 0.226700
197 0.610847 0.216980 0.634669 0.214200
198 0.602794 0.211580 0.637142 0.215600
199 0.500774 0.178000 0.644303 0.216100
200 0.556853 0.197860 0.653069 0.219800
201 0.473643 0.166640 0.637706 0.215500
202 0.543367 0.192500 0.630976 0.210900
203 0.547610 0.193560 0.647321 0.218400
204 0.549161 0.193160 0.652783 0.216800
205 0.555136 0.194700 0.638940 0.214200
206 0.681273 0.239900 0.652195 0.219600
207 0.538329 0.188140 0.632762 0.216300
208 0.536182 0.188480 0.635168 0.214900
209 0.542705 0.191000 0.627548 0.210600
210 0.575116 0.203320 0.639326 0.214100
211 0.507712 0.177960 0.623835 0.210300
212 0.571738 0.201600 0.637558 0.215800
213 0.528358 0.185160 0.655310 0.222300
214 0.590359 0.208460 0.658555 0.221200
215 0.530377 0.187640 0.635255 0.214300
216 0.494095 0.174020 0.635528 0.210600
217 0.482562 0.170700 0.641069 0.218300
218 0.561212 0.199020 0.645154 0.217900
219 0.461704 0.163360 0.621216 0.209100
220 0.497201 0.176980 0.635099 0.216300
221 0.519957 0.184700 0.638311 0.211700
222 0.607542 0.215080 0.627521 0.206700
223 0.566220 0.201660 0.630709 0.208700
224 0.516641 0.182200 0.640494 0.213500
225 0.467458 0.167020 0.632075 0.211200
226 0.561692 0.199120 0.636891 0.208600
227 0.527434 0.186860 0.636249 0.214700
228 0.493206 0.175560 0.642024 0.217900
229 0.435306 0.153840 0.624948 0.207200
230 0.465713 0.163760 0.625984 0.206600
231 0.613709 0.216140 0.627374 0.209800
232 0.585429 0.206380 0.639690 0.216400
233 0.486990 0.173220 0.627945 0.212600
234 0.611468 0.214740 0.634425 0.215000
235 0.508671 0.181200 0.627543 0.208800
236 0.505741 0.179560 0.623797 0.208300
237 0.546465 0.192600 0.624793 0.209700
238 0.471171 0.169480 0.617415 0.204800
239 0.493997 0.174900 0.624413 0.207300
240 0.411756 0.145880 0.627248 0.202000
241 0.553179 0.194820 0.638805 0.215800
242 0.422501 0.149560 0.622032 0.208500
243 0.531075 0.188420 0.621820 0.206800
244 0.489139 0.173960 0.627738 0.207800
245 0.419156 0.147180 0.629689 0.202700
246 0.489858 0.172860 0.618313 0.203900
247 0.510623 0.182660 0.631426 0.212700
248 0.496845 0.174860 0.625647 0.211500
249 0.420132 0.149140 0.620893 0.204800
250 0.512875 0.180660 0.614870 0.202300
251 0.524712 0.184420 0.615267 0.205400
252 0.381032 0.134060 0.630305 0.204900
253 0.426079 0.150820 0.614480 0.200200
254 0.413228 0.147720 0.626044 0.206800
255 0.356428 0.126460 0.636710 0.206000
256 0.434676 0.153440 0.636250 0.210000
257 0.442012 0.157880 0.629783 0.206400
258 0.473236 0.169440 0.630550 0.211200
259 0.447313 0.158560 0.638433 0.212900
260 0.414860 0.147860 0.628668 0.208500
261 0.392432 0.140000 0.641313 0.205600
262 0.511654 0.179940 0.627193 0.212100
263 0.507849 0.178880 0.652576 0.219200
264 0.566355 0.199360 0.638171 0.212300
265 0.425083 0.149560 0.615276 0.202100
266 0.474922 0.167860 0.627144 0.205600
267 0.431780 0.151740 0.625363 0.204400
268 0.454628 0.162160 0.629383 0.205700
269 0.457794 0.162160 0.623723 0.204100
270 0.548920 0.193420 0.619574 0.208200
271 0.415419 0.148140 0.630262 0.207600
272 0.334148 0.117640 0.632166 0.205000
273 0.523316 0.185060 0.632792 0.207100
274 0.472590 0.166780 0.625767 0.207700
275 0.366252 0.130020 0.629547 0.204900
276 0.409062 0.144900 0.625694 0.203800
277 0.469791 0.165200 0.634042 0.208000
278 0.431191 0.152160 0.630712 0.201600
279 0.345246 0.123700 0.628148 0.198100
280 0.465820 0.164240 0.623144 0.207100
281 0.325065 0.115620 0.630440 0.201900
282 0.420623 0.150320 0.624133 0.202800
283 0.398687 0.142820 0.623236 0.204400
284 0.585759 0.203360 0.628997 0.209400
285 0.481718 0.171620 0.631858 0.206900
286 0.406801 0.145260 0.642828 0.208700
287 0.424924 0.151320 0.623796 0.201900
288 0.415299 0.147700 0.636585 0.206300
289 0.472887 0.167360 0.639477 0.208900
290 0.389044 0.136600 0.635800 0.201700
291 0.349180 0.124700 0.628040 0.197800
292 0.448326 0.158620 0.630691 0.203900
293 0.358445 0.128200 0.636178 0.201000
294 0.522634 0.184980 0.632877 0.210200
295 0.458186 0.162820 0.626756 0.204600
296 0.382808 0.135500 0.631453 0.207800
297 0.438598 0.156740 0.628119 0.206600
298 0.371808 0.133780 0.651291 0.211200
299 0.429471 0.152240 0.631111 0.205000
300 0.536832 0.189640 0.631903 0.210000
301 0.334097 0.117920 0.631833 0.200600
302 0.430021 0.151060 0.619491 0.202800
303 0.413743 0.148440 0.627023 0.205200
304 0.324145 0.115500 0.642452 0.200700
305 0.355051 0.125440 0.637457 0.202900
306 0.451619 0.159740 0.636109 0.208000
real 59m12.227s
user 80m49.696s
sys 85m0.752s
}};
***C[0.001]-P-C[0.001]-P-F[0.01]-F[0.01]
#pre{{
############# training condition ############
500 epoch training. input_size: (28, 28) , minibatch_size: 100
, learning_rate: 0.01 , momentum: 0.9 , weight_decay: 0.0005
#############################################
########## architecture of the CNN #########
layer1 - Convolution
< kernel_size: (50, 2, 2) , stride: (1, 1) , act_func: ReLU
, border_mode: valid , out_size: (50, 27, 27) , iniW_σ: 0.001000 >
layer2 - Convolution
< kernel_size: (50, 2, 2) , stride: (1, 1) , act_func: ReLU
, border_mode: valid , out_size: (50, 26, 26) , iniW_σ: 0.001000 >
layer3 - Pooling
< downscale: (4, 4) , stride: None , out_size: (50, 6, 6) >
layer4 - fully-connected
< number_of_units: 1000 , drop_rate: 0.8 , act_func: ReLU , iniW_σ: 0.010000 >
layer5 - fully-connected
< number_of_units: 10 , drop_rate: 0.5 , act_func: softmax , iniW_σ: 0.010000 >
###########################################
epoch XE(L) error(L) XE(T) error(T)
1 2.302872 0.898900 2.302660 0.900000
2 2.302858 0.901900 2.302781 0.900000
3 2.302803 0.901360 2.302714 0.900000
4 2.302769 0.898940 2.302536 0.900000
5 2.302424 0.897020 2.301566 0.900000
6 2.276309 0.858520 2.169257 0.787100
7 2.072670 0.758680 2.005378 0.719300
8 2.014385 0.733880 1.987759 0.727200
9 1.984420 0.721800 1.926013 0.696000
10 1.946731 0.713860 1.890811 0.678500
11 1.916479 0.695900 1.856166 0.667600
12 1.896306 0.690600 1.887214 0.685200
13 1.887320 0.688060 1.844342 0.662100
14 1.874480 0.683780 1.821667 0.656600
15 1.855282 0.674580 1.806978 0.649200
16 1.841777 0.668900 1.829868 0.653600
17 1.827406 0.664220 1.799668 0.649300
18 1.829521 0.663860 1.766902 0.639000
19 1.830559 0.664520 1.746905 0.619000
20 1.729513 0.624260 1.631033 0.582600
21 1.638321 0.585920 1.563860 0.555500
22 1.602554 0.574500 1.489464 0.526200
23 1.543845 0.551720 1.468083 0.523900
24 1.513750 0.540320 1.430057 0.500200
25 1.462747 0.521020 1.395763 0.497300
26 1.430794 0.508380 1.368149 0.482800
27 1.419554 0.505040 1.352399 0.479500
28 1.397583 0.495760 1.304829 0.457500
29 1.360754 0.484840 1.284288 0.453700
30 1.310166 0.463500 1.259930 0.439600
31 1.287544 0.456280 1.203224 0.420400
32 1.256335 0.444860 1.197972 0.421400
33 1.207174 0.423920 1.179540 0.411500
34 1.214620 0.430120 1.125907 0.389500
35 1.164442 0.408680 1.134102 0.395600
36 1.203204 0.427800 1.088162 0.370600
37 1.161411 0.407680 1.066622 0.369100
38 1.144375 0.399700 1.048877 0.360600
39 1.078669 0.378340 0.987294 0.339000
40 1.059620 0.373000 0.994588 0.345600
41 1.068687 0.372000 0.982816 0.340900
42 1.015443 0.356140 0.937158 0.326100
43 1.005130 0.352380 0.927095 0.319600
44 0.992993 0.346840 0.918870 0.316800
45 1.004931 0.349700 0.921541 0.319900
46 1.014245 0.353240 0.945760 0.324800
47 0.958723 0.333060 0.916614 0.320300
48 0.954285 0.333700 0.868280 0.297900
49 0.871379 0.304040 0.837763 0.291400
50 0.951212 0.329500 0.855732 0.293300
51 0.925132 0.323060 0.841287 0.291300
52 0.826395 0.289580 0.811994 0.280500
53 0.850422 0.296380 0.814791 0.282200
54 0.806513 0.281620 0.818986 0.280900
55 0.890272 0.310980 0.818280 0.280500
56 0.832098 0.291500 0.800061 0.272900
57 0.802854 0.280280 0.794087 0.273000
58 0.780356 0.272840 0.793135 0.269800
59 0.870416 0.302300 0.813302 0.280400
60 0.775174 0.271740 0.755276 0.259200
61 0.741149 0.258020 0.763465 0.259500
62 0.795234 0.277920 0.807325 0.280800
63 0.791168 0.277520 0.764574 0.261000
64 0.777582 0.268900 0.735795 0.250100
65 0.849017 0.296300 0.761325 0.259900
66 0.750857 0.261180 0.732703 0.252300
67 0.782823 0.272460 0.730523 0.248700
68 0.720695 0.252040 0.729317 0.252700
69 0.721221 0.254040 0.707432 0.238900
70 0.663256 0.231140 0.719061 0.246600
71 0.702100 0.246580 0.712442 0.239400
72 0.706973 0.246760 0.715738 0.247200
73 0.681144 0.238480 0.704208 0.240300
74 0.694613 0.242520 0.707136 0.238800
75 0.687925 0.242060 0.699915 0.234200
76 0.699514 0.245760 0.689479 0.231500
77 0.696203 0.244640 0.706988 0.236800
78 0.665517 0.232720 0.701321 0.234500
79 0.706234 0.247340 0.695606 0.241300
80 0.661712 0.231800 0.688041 0.234000
81 0.640701 0.223980 0.684495 0.232200
82 0.660487 0.229720 0.693339 0.234800
83 0.673713 0.237440 0.682402 0.230800
84 0.620994 0.217000 0.680232 0.236400
85 0.646409 0.226920 0.676887 0.229400
86 0.645745 0.227860 0.664853 0.220100
87 0.689577 0.241760 0.673440 0.228400
88 0.601237 0.208440 0.667513 0.224600
89 0.609765 0.215080 0.673499 0.229300
90 0.642425 0.224980 0.662619 0.224400
91 0.618559 0.215380 0.668127 0.227800
92 0.655401 0.230220 0.655188 0.217500
93 0.669924 0.234000 0.659074 0.220400
94 0.560884 0.196020 0.677943 0.230300
95 0.649010 0.225580 0.662177 0.227000
96 0.690835 0.241180 0.664829 0.227900
97 0.650865 0.227640 0.643818 0.213200
98 0.566280 0.198040 0.644339 0.217400
99 0.613401 0.214320 0.639799 0.216100
100 0.667891 0.235540 0.641766 0.213800
101 0.580818 0.205480 0.638238 0.216000
102 0.536167 0.189780 0.649514 0.214000
103 0.709693 0.246580 0.663111 0.225000
104 0.631701 0.221200 0.652446 0.226100
105 0.619709 0.216680 0.654964 0.222000
106 0.533599 0.186440 0.641212 0.214100
107 0.493039 0.173280 0.651675 0.217000
108 0.645964 0.224960 0.632432 0.209600
109 0.618259 0.217500 0.645378 0.216700
110 0.514565 0.181820 0.637972 0.213700
111 0.534446 0.186740 0.645896 0.214100
112 0.522354 0.183960 0.638978 0.212200
113 0.600494 0.211620 0.643802 0.214100
114 0.549375 0.195360 0.645564 0.213000
115 0.501327 0.177980 0.626850 0.211100
116 0.616277 0.216520 0.656425 0.224100
117 0.573093 0.203100 0.636134 0.215400
118 0.668564 0.232860 0.651479 0.218300
119 0.566857 0.200380 0.663800 0.217600
120 0.589800 0.208400 0.650366 0.219400
121 0.547857 0.193800 0.634174 0.214200
122 0.538350 0.191180 0.632722 0.211500
123 0.509841 0.180000 0.622958 0.209700
124 0.447756 0.157080 0.628028 0.210800
125 0.555008 0.195080 0.632165 0.212900
126 0.485475 0.168980 0.620688 0.205200
127 0.506353 0.179820 0.629752 0.211600
128 0.575891 0.202620 0.650257 0.217900
129 0.613002 0.217260 0.650063 0.221000
130 0.456279 0.162080 0.630299 0.211000
131 0.458614 0.161660 0.636779 0.209600
132 0.510723 0.180580 0.649673 0.218800
133 0.470076 0.164560 0.622362 0.203900
134 0.565348 0.200940 0.653620 0.220300
135 0.523974 0.185100 0.636000 0.211200
136 0.418222 0.149120 0.650506 0.215900
137 0.498014 0.175140 0.626513 0.205400
138 0.436453 0.155120 0.630010 0.205600
139 0.431618 0.154040 0.630019 0.206800
140 0.459956 0.163020 0.634649 0.209500
141 0.384548 0.135240 0.645044 0.207900
142 0.515291 0.183680 0.652195 0.215700
143 0.506902 0.177940 0.633119 0.209500
144 0.425628 0.150740 0.653576 0.210200
145 0.607325 0.213460 0.640440 0.213700
146 0.500311 0.176800 0.628680 0.207900
147 0.538828 0.189120 0.638379 0.212300
148 0.554753 0.194980 0.627643 0.210500
149 0.519375 0.182780 0.615895 0.204200
150 0.378655 0.134740 0.637201 0.206600
151 0.453305 0.160340 0.626257 0.205900
152 0.512213 0.181380 0.634887 0.213200
153 0.552009 0.195980 0.634409 0.208700
154 0.483465 0.172520 0.613985 0.201000
155 0.481577 0.172640 0.623544 0.204200
156 0.455515 0.161200 0.617180 0.202800
157 0.603864 0.211500 0.634859 0.213600
158 0.388869 0.136280 0.617869 0.203500
159 0.520007 0.184220 0.621469 0.208500
160 0.573866 0.201960 0.621250 0.204100
161 0.545355 0.190700 0.628192 0.209200
162 0.396342 0.139500 0.615180 0.198300
163 0.502700 0.177000 0.629004 0.207400
164 0.547316 0.192460 0.625879 0.211100
165 0.474462 0.168240 0.608074 0.202300
166 0.429055 0.151400 0.612654 0.200300
167 0.444818 0.159240 0.637688 0.205700
168 0.462765 0.163040 0.616273 0.205200
169 0.485579 0.171260 0.619473 0.204200
170 0.389189 0.137680 0.619296 0.200600
171 0.438430 0.156140 0.630109 0.207800
172 0.481412 0.170000 0.620443 0.200400
173 0.470107 0.166080 0.631968 0.212400
174 0.479419 0.170640 0.629643 0.205400
175 0.365037 0.129640 0.624014 0.201400
176 0.372456 0.132940 0.629099 0.204800
177 0.504970 0.178740 0.619571 0.202600
178 0.357786 0.129120 0.619522 0.200700
179 0.338896 0.119140 0.640979 0.205900
180 0.406746 0.144440 0.637314 0.208800
181 0.374679 0.134800 0.634658 0.206700
182 0.362342 0.128320 0.617033 0.202900
183 0.336896 0.120900 0.639205 0.204800
184 0.506341 0.178800 0.632390 0.206700
185 0.323573 0.115340 0.640153 0.202300
186 0.360581 0.127460 0.633997 0.204900
187 0.523081 0.183560 0.642037 0.216100
188 0.439722 0.156820 0.628661 0.200200
189 0.429584 0.152000 0.634667 0.206500
190 0.478491 0.171140 0.611366 0.199200
191 0.526997 0.185360 0.654256 0.217300
192 0.314695 0.112120 0.651739 0.204400
193 0.373587 0.134240 0.632306 0.202100
194 0.332759 0.118540 0.642136 0.205300
195 0.505420 0.178400 0.636079 0.207300
196 0.306033 0.107780 0.647592 0.206400
197 0.289547 0.104060 0.663207 0.206300
198 0.417031 0.147400 0.639258 0.209100
199 0.365849 0.130020 0.656354 0.210400
200 0.529010 0.185280 0.639793 0.208900
real 38m49.140s
user 52m52.912s
sys 56m25.160s
}};
***C[0.01]-P-C[0.01]-P-F[0.001]-F[0.001]
#pre{{
############# training condition ############
500 epoch training. input_size: (28, 28) , minibatch_size: 100
, learning_rate: 0.01 , momentum: 0.9 , weight_decay: 0.0005
#############################################
########## architecture of the CNN #########
layer1 - Convolution
< kernel_size: (50, 2, 2) , stride: (1, 1) , act_func: ReLU
, border_mode: valid , out_size: (50, 27, 27) , iniW_σ: 0.010000 >
layer2 - Convolution
< kernel_size: (50, 2, 2) , stride: (1, 1) , act_func: ReLU
, border_mode: valid , out_size: (50, 26, 26) , iniW_σ: 0.010000 >
layer3 - Pooling
< downscale: (4, 4) , stride: None , out_size: (50, 6, 6) >
layer4 - fully-connected
< number_of_units: 1000 , drop_rate: 0.8 , act_func: ReLU , iniW_σ: 0.001000 >
layer5 - fully-connected
< number_of_units: 10 , drop_rate: 0.5 , act_func: softmax , iniW_σ: 0.001000 >
###########################################
epoch XE(L) error(L) XE(T) error(T)
1 2.302901 0.901300 2.302623 0.900000
2 2.302820 0.901660 2.302581 0.900000
3 2.302217 0.897120 2.299640 0.882000
4 2.230180 0.848140 2.112987 0.776600
5 2.071772 0.765080 2.018637 0.734000
6 2.024891 0.741720 1.993755 0.715000
7 2.005046 0.731740 1.987875 0.728900
8 1.983595 0.721320 1.941262 0.704800
9 1.970283 0.715120 1.931210 0.689400
10 1.939247 0.706840 1.880771 0.676300
11 1.925168 0.697200 1.859701 0.661500
12 1.911540 0.695640 1.872053 0.680600
13 1.900941 0.688160 1.855476 0.669400
14 1.885428 0.682280 1.845671 0.658300
15 1.887153 0.683660 1.838827 0.655800
16 1.862930 0.671900 1.818500 0.650200
17 1.863536 0.673000 1.809071 0.647100
18 1.789304 0.640780 1.697487 0.596600
19 1.705984 0.606700 1.618248 0.577700
20 1.645008 0.589960 1.572515 0.560100
21 1.600171 0.577340 1.623008 0.572500
22 1.538985 0.551240 1.459121 0.524300
23 1.512821 0.544740 1.440079 0.511300
24 1.489921 0.537300 1.437246 0.506300
25 1.421916 0.509780 1.369165 0.487300
26 1.388289 0.496780 1.308086 0.462300
27 1.383351 0.494060 1.288331 0.453800
28 1.367467 0.489860 1.292792 0.453500
29 1.322461 0.471560 1.244533 0.438400
30 1.283352 0.455240 1.231320 0.432500
31 1.273257 0.452500 1.193134 0.419400
32 1.274518 0.453240 1.225659 0.435000
33 1.236957 0.440480 1.176855 0.415100
34 1.235800 0.436800 1.161239 0.402700
35 1.224157 0.436300 1.111506 0.381400
36 1.197053 0.422840 1.118210 0.388300
37 1.150130 0.406760 1.069545 0.369300
38 1.118568 0.395240 1.060744 0.366100
39 1.122202 0.396220 1.064548 0.372600
40 1.091638 0.385880 1.019541 0.352700
41 1.081617 0.379300 0.967701 0.342300
42 1.060177 0.368940 0.999182 0.351200
43 0.989399 0.346460 0.958429 0.334500
44 1.002523 0.351440 0.922679 0.320000
45 0.970988 0.339800 0.929425 0.324700
46 1.027290 0.360400 0.910248 0.317300
47 0.921182 0.322060 0.879655 0.308700
48 0.890499 0.311240 0.865614 0.304200
49 0.903701 0.314560 0.899403 0.308900
50 0.906859 0.317240 0.866775 0.301300
51 0.844980 0.293460 0.841168 0.295700
52 0.956383 0.332780 0.845176 0.290800
53 0.854559 0.299100 0.834348 0.295500
54 0.820551 0.288820 0.832715 0.291300
55 0.827537 0.290020 0.823778 0.288000
56 0.794404 0.278280 0.802516 0.279200
57 0.919831 0.321780 0.804460 0.274800
58 0.947838 0.331940 0.807488 0.277000
59 0.818844 0.283660 0.803254 0.280300
60 0.814942 0.284740 0.760469 0.260300
61 0.766425 0.267640 0.765969 0.264000
62 0.849141 0.293940 0.753863 0.253700
63 0.834052 0.291400 0.755548 0.258800
64 0.818990 0.288020 0.763077 0.265100
65 0.867316 0.301760 0.759022 0.260300
66 0.836070 0.289560 0.751518 0.257900
67 0.783445 0.271820 0.738580 0.254000
68 0.808771 0.281320 0.763343 0.266100
69 0.798330 0.277260 0.774414 0.265000
70 0.751814 0.261960 0.747813 0.259100
71 0.756193 0.264800 0.716244 0.247800
72 0.719148 0.250520 0.723906 0.248000
73 0.756608 0.263240 0.719750 0.241700
74 0.771815 0.267760 0.713564 0.245900
75 0.706465 0.247160 0.695171 0.237700
76 0.665230 0.230280 0.694496 0.240200
77 0.679877 0.238100 0.703050 0.238900
78 0.641064 0.225080 0.694722 0.238800
79 0.643016 0.226540 0.685286 0.234500
80 0.665971 0.233140 0.701992 0.240600
81 0.750654 0.261080 0.681645 0.232000
82 0.725471 0.253200 0.673750 0.231300
83 0.678729 0.235480 0.672499 0.236400
84 0.676068 0.236100 0.664762 0.225300
85 0.601115 0.211680 0.668561 0.227800
86 0.595666 0.209800 0.670929 0.229000
87 0.626114 0.218340 0.654591 0.223100
88 0.674411 0.236280 0.661875 0.226100
89 0.661216 0.230800 0.644898 0.222900
90 0.734703 0.255400 0.682345 0.233500
91 0.646937 0.227020 0.655013 0.223400
92 0.632959 0.221040 0.644096 0.219000
93 0.639466 0.225200 0.656427 0.223200
94 0.715922 0.249580 0.657482 0.228800
95 0.655604 0.228600 0.663985 0.227900
96 0.628675 0.218700 0.634387 0.215600
97 0.636521 0.225440 0.640979 0.218300
98 0.658399 0.230140 0.645956 0.220100
99 0.573348 0.201100 0.630046 0.213300
100 0.550218 0.192360 0.638988 0.217000
101 0.541562 0.191540 0.628448 0.212800
102 0.609731 0.213660 0.637119 0.219500
103 0.541422 0.189760 0.627562 0.217200
104 0.594789 0.206520 0.632211 0.216100
105 0.533965 0.188240 0.619938 0.212100
106 0.614995 0.215720 0.638859 0.217700
107 0.518768 0.181300 0.632164 0.212200
108 0.510290 0.178900 0.637924 0.218600
109 0.514615 0.181840 0.613340 0.209400
110 0.653121 0.228860 0.650518 0.222600
111 0.616260 0.217580 0.634341 0.217700
112 0.521380 0.183800 0.630964 0.211400
113 0.484702 0.170940 0.617360 0.210900
114 0.607867 0.214240 0.637023 0.219400
115 0.515181 0.180100 0.613999 0.208600
116 0.519477 0.183900 0.637284 0.213500
117 0.475232 0.167840 0.629038 0.211400
118 0.557439 0.197300 0.638357 0.217600
119 0.598365 0.210760 0.627784 0.211400
120 0.512726 0.181080 0.613917 0.207900
121 0.577865 0.201420 0.615097 0.210300
122 0.528882 0.186140 0.610660 0.206500
123 0.603721 0.210940 0.628577 0.215500
124 0.565278 0.199160 0.622649 0.209200
125 0.491616 0.173440 0.619445 0.210800
126 0.435459 0.154320 0.617538 0.207900
127 0.438711 0.155760 0.625713 0.212500
128 0.508624 0.178940 0.615024 0.205800
129 0.487787 0.172260 0.632125 0.213400
130 0.420390 0.149080 0.628939 0.210400
131 0.519737 0.183280 0.611218 0.211000
132 0.488809 0.173080 0.610631 0.205500
133 0.411985 0.143900 0.612675 0.200700
134 0.425841 0.150760 0.607322 0.202800
135 0.564123 0.197460 0.620712 0.211400
136 0.619336 0.217700 0.627627 0.215600
137 0.500103 0.178560 0.605096 0.207300
138 0.601750 0.211960 0.614226 0.209400
139 0.409697 0.144840 0.628751 0.214100
140 0.426528 0.150440 0.612627 0.207400
141 0.418140 0.149120 0.626981 0.211200
142 0.508491 0.177940 0.616579 0.206600
143 0.400348 0.140980 0.611378 0.199900
144 0.558010 0.194900 0.610655 0.207500
145 0.436419 0.153820 0.620998 0.207500
146 0.592183 0.207060 0.618941 0.211600
147 0.492563 0.174560 0.608695 0.205800
148 0.460964 0.161960 0.594691 0.197900
149 0.533392 0.188640 0.610247 0.203900
150 0.387436 0.136380 0.607173 0.200900
151 0.532460 0.186620 0.616157 0.207400
152 0.362419 0.129580 0.616162 0.202900
153 0.466150 0.166000 0.616198 0.202600
154 0.503470 0.178000 0.613591 0.203800
155 0.528610 0.186960 0.612632 0.203200
156 0.515903 0.182660 0.608241 0.201600
157 0.361893 0.127440 0.625812 0.204100
158 0.332449 0.118520 0.619963 0.200700
159 0.571487 0.201460 0.613481 0.209000
160 0.382678 0.134820 0.621906 0.206000
161 0.434988 0.155560 0.621866 0.205700
162 0.305630 0.108640 0.626495 0.200200
163 0.505812 0.178360 0.611657 0.210200
164 0.444961 0.157260 0.610047 0.206000
165 0.483167 0.171960 0.616183 0.210300
166 0.450604 0.158760 0.617694 0.205400
167 0.413043 0.145360 0.608303 0.200700
168 0.298992 0.106240 0.620169 0.200600
169 0.434270 0.153600 0.616022 0.205200
170 0.365111 0.130840 0.613091 0.199300
171 0.358076 0.128580 0.614925 0.201400
172 0.300774 0.106440 0.615542 0.197600
173 0.363072 0.129160 0.622786 0.202700
174 0.488701 0.173240 0.616393 0.203400
175 0.462220 0.162760 0.612299 0.207500
176 0.323003 0.113780 0.622196 0.203600
177 0.438456 0.154780 0.616680 0.204500
178 0.363259 0.128480 0.620704 0.202300
179 0.496789 0.175420 0.625555 0.207600
180 0.403707 0.140840 0.608423 0.200000
181 0.506264 0.178940 0.611625 0.205800
182 0.338968 0.120360 0.603149 0.198300
183 0.392754 0.139340 0.608893 0.197800
184 0.530670 0.187840 0.603998 0.200700
185 0.397160 0.139960 0.597075 0.196000
186 0.447407 0.159900 0.599321 0.198900
187 0.487685 0.172500 0.605278 0.202000
188 0.490796 0.173180 0.598128 0.199600
189 0.381923 0.136320 0.603520 0.197300
190 0.483876 0.170920 0.605777 0.200200
191 0.417605 0.147780 0.610298 0.204100
192 0.354936 0.126120 0.601664 0.195900
193 0.475200 0.168520 0.613556 0.197200
194 0.445222 0.156600 0.620630 0.199900
195 0.407889 0.143760 0.609373 0.199100
196 0.340630 0.121420 0.610759 0.196500
197 0.314984 0.112240 0.615860 0.197400
198 0.364542 0.129200 0.609140 0.197400
199 0.375407 0.134680 0.606543 0.194400
200 0.400660 0.142460 0.608066 0.196600
real 38m47.191s
user 52m58.352s
sys 55m38.200s
}};
***C[0.01]-P-C[0.01]-P-F[0.01]-F[0.01]
#pre{{
############# training condition ############
500 epoch training. input_size: (28, 28) , minibatch_size: 100
, learning_rate: 0.01 , momentum: 0.9 , weight_decay: 0.0005
#############################################
########## architecture of the CNN #########
layer1 - Convolution
< kernel_size: (50, 2, 2) , stride: (1, 1) , act_func: ReLU
, border_mode: valid , out_size: (50, 27, 27) , iniW_σ: 0.010000 >
layer2 - Convolution
< kernel_size: (50, 2, 2) , stride: (1, 1) , act_func: ReLU
, border_mode: valid , out_size: (50, 26, 26) , iniW_σ: 0.010000 >
layer3 - Pooling
< downscale: (4, 4) , stride: None , out_size: (50, 6, 6) >
layer4 - fully-connected
< number_of_units: 1000 , drop_rate: 0.8 , act_func: ReLU , iniW_σ: 0.010000 >
layer5 - fully-connected
< number_of_units: 10 , drop_rate: 0.5 , act_func: softmax , iniW_σ: 0.010000 >
###########################################
epoch XE(L) error(L) XE(T) error(T)
1 2.295846 0.889220 2.262589 0.856300
2 2.115511 0.770380 2.014915 0.732600
3 2.020671 0.734360 1.971737 0.706800
4 1.980240 0.718320 1.931875 0.707900
5 1.944661 0.706700 1.904450 0.687500
6 1.922105 0.698880 1.896850 0.683000
7 1.905911 0.692200 1.871081 0.674900
8 1.886066 0.682940 1.843626 0.666900
9 1.882368 0.681560 1.838893 0.664500
10 1.869553 0.676960 1.815571 0.650500
11 1.864278 0.673200 1.837251 0.659500
12 1.873286 0.676240 1.812946 0.645400
13 1.818123 0.653520 1.732392 0.612100
14 1.749607 0.624340 1.656842 0.594700
15 1.653327 0.590760 1.561860 0.555800
16 1.594014 0.569740 1.514609 0.540300
17 1.548157 0.556760 1.528837 0.542500
18 1.503404 0.537180 1.415669 0.507300
19 1.459075 0.522320 1.394656 0.497300
20 1.423965 0.506960 1.362277 0.477700
21 1.404101 0.499880 1.330899 0.472400
22 1.351620 0.482200 1.290235 0.453500
23 1.325897 0.472540 1.253288 0.439300
24 1.328091 0.472800 1.234367 0.431100
25 1.285086 0.458520 1.195523 0.418900
26 1.278986 0.454940 1.167001 0.407300
27 1.230778 0.437560 1.169653 0.409800
28 1.254427 0.445320 1.151704 0.397200
29 1.179577 0.420480 1.113861 0.388000
30 1.180674 0.415880 1.091560 0.378900
31 1.220288 0.430180 1.122305 0.389400
32 1.162690 0.409680 1.106598 0.383200
33 1.158491 0.407460 1.094822 0.381700
34 1.121711 0.394400 1.023304 0.353700
35 1.075405 0.378720 1.005997 0.351100
36 1.051160 0.369500 0.972907 0.337700
37 1.090402 0.382400 0.959699 0.327700
38 1.056046 0.371380 0.957709 0.335300
39 1.019505 0.356000 0.963410 0.336700
40 0.984273 0.344980 0.923884 0.319400
41 0.965589 0.338220 0.907603 0.311500
42 1.063812 0.374860 0.911057 0.312200
43 0.951267 0.335120 0.897823 0.305400
44 0.959741 0.334940 0.895456 0.305600
45 0.884838 0.311280 0.839286 0.285100
46 0.978875 0.341900 0.854322 0.296700
47 0.932763 0.324100 0.848260 0.295400
48 0.869802 0.303320 0.828480 0.281100
49 0.928031 0.323380 0.859657 0.297800
50 0.910751 0.316660 0.829283 0.287900
51 0.828789 0.290060 0.814298 0.280700
52 0.855133 0.300180 0.799630 0.274500
53 0.867939 0.302020 0.788236 0.267700
54 0.876353 0.305140 0.797260 0.269900
55 0.836990 0.291780 0.765439 0.263200
56 0.798550 0.278140 0.764460 0.261800
57 0.849203 0.296000 0.776326 0.267700
58 0.816710 0.285240 0.751844 0.256900
59 0.788163 0.277620 0.752705 0.257200
60 0.812437 0.285180 0.754142 0.258300
61 0.810678 0.282800 0.758790 0.258300
62 0.783860 0.273800 0.750935 0.258500
63 0.783673 0.271280 0.758504 0.262100
64 0.776376 0.269860 0.734984 0.255000
65 0.780174 0.273660 0.726331 0.246000
66 0.737401 0.258100 0.710941 0.240400
67 0.822316 0.286380 0.721608 0.244900
68 0.732870 0.257040 0.721644 0.246400
69 0.752138 0.260680 0.689774 0.232800
70 0.722814 0.251820 0.720069 0.243400
71 0.806843 0.283540 0.712912 0.241800
72 0.693675 0.241860 0.696936 0.239000
73 0.732292 0.257680 0.723535 0.250400
74 0.743348 0.258320 0.692517 0.235200
75 0.698793 0.243020 0.692085 0.235500
76 0.656595 0.233220 0.679415 0.228100
77 0.686947 0.238740 0.688888 0.233200
78 0.695408 0.244160 0.689658 0.233500
79 0.676750 0.237000 0.673825 0.230100
80 0.607276 0.210120 0.688524 0.236800
81 0.763574 0.265000 0.701790 0.240700
82 0.696277 0.240880 0.704207 0.241700
83 0.709175 0.247520 0.698019 0.238200
84 0.656472 0.232080 0.675286 0.230800
85 0.736603 0.257940 0.696097 0.237500
86 0.665403 0.231300 0.655941 0.222900
87 0.689938 0.241400 0.692603 0.238300
88 0.637561 0.221680 0.692464 0.241100
89 0.603042 0.210800 0.670211 0.225600
90 0.719143 0.250720 0.673726 0.228600
91 0.638884 0.224260 0.670063 0.228500
92 0.669008 0.233440 0.689416 0.235800
93 0.580914 0.204200 0.662864 0.229100
94 0.691088 0.242320 0.671204 0.231300
95 0.610035 0.211500 0.660191 0.221400
96 0.580456 0.204240 0.643380 0.220500
97 0.624617 0.218240 0.657044 0.220700
98 0.615550 0.217620 0.667579 0.225100
99 0.582741 0.205900 0.663710 0.227600
100 0.634705 0.221020 0.672440 0.226300
101 0.591286 0.208640 0.679087 0.234200
102 0.601860 0.212240 0.658298 0.224400
103 0.618291 0.217400 0.653479 0.219400
104 0.607953 0.213280 0.635141 0.214300
105 0.581175 0.203140 0.650927 0.223000
106 0.555254 0.193520 0.654571 0.220400
107 0.618177 0.217460 0.656775 0.223200
108 0.572511 0.201120 0.657990 0.224000
109 0.585749 0.208040 0.642138 0.220100
110 0.527424 0.185120 0.644592 0.221200
111 0.499200 0.174620 0.675473 0.227800
112 0.490442 0.172340 0.651364 0.218500
113 0.628758 0.221420 0.662762 0.222200
114 0.635877 0.225240 0.647096 0.218200
115 0.563327 0.200540 0.659880 0.226200
116 0.531017 0.185880 0.659357 0.225600
117 0.480397 0.171980 0.642966 0.212400
118 0.510882 0.180440 0.659652 0.225000
119 0.672867 0.233560 0.662867 0.227400
120 0.536841 0.189740 0.632090 0.213800
121 0.444981 0.156820 0.633691 0.208900
122 0.441651 0.157400 0.638043 0.212900
123 0.497385 0.177060 0.637389 0.214100
124 0.416272 0.148220 0.657665 0.220100
125 0.632245 0.222040 0.651039 0.221600
126 0.409079 0.145640 0.648700 0.216600
127 0.559476 0.195820 0.630556 0.211000
128 0.553231 0.192740 0.631859 0.208100
129 0.516978 0.182720 0.629586 0.209100
130 0.508360 0.178520 0.625875 0.207300
131 0.506639 0.177960 0.627617 0.208100
132 0.532023 0.187800 0.622849 0.210200
133 0.513279 0.181620 0.624994 0.210900
134 0.529050 0.188320 0.624003 0.209400
135 0.566943 0.199420 0.626655 0.211400
136 0.495513 0.176260 0.617666 0.203600
137 0.512096 0.180500 0.620363 0.209200
138 0.555502 0.196840 0.635807 0.216100
139 0.532674 0.188540 0.633788 0.213300
140 0.503084 0.176920 0.636918 0.211100
141 0.497505 0.176940 0.612319 0.206200
142 0.528009 0.187120 0.623163 0.209100
143 0.482022 0.171140 0.622657 0.208500
144 0.487121 0.173200 0.630329 0.213600
145 0.492959 0.175460 0.627584 0.206700
146 0.536013 0.190000 0.629574 0.210800
147 0.561766 0.199160 0.635424 0.212200
148 0.493793 0.174240 0.626691 0.207900
149 0.425548 0.149520 0.630802 0.210700
150 0.475931 0.168100 0.623399 0.209500
151 0.483713 0.170360 0.640011 0.209300
152 0.465127 0.164960 0.634834 0.214400
153 0.460989 0.163280 0.626684 0.206300
154 0.449315 0.158220 0.617004 0.204700
155 0.416660 0.145660 0.626971 0.202600
156 0.506524 0.179720 0.638759 0.211700
157 0.533466 0.188700 0.624977 0.208900
158 0.617077 0.216740 0.635375 0.212300
159 0.505615 0.177840 0.618484 0.206700
160 0.472271 0.166580 0.613608 0.203400
161 0.426948 0.151200 0.628950 0.210300
162 0.473673 0.168160 0.616989 0.204500
163 0.361854 0.129540 0.629883 0.204900
164 0.442030 0.156980 0.630763 0.208200
165 0.442348 0.156800 0.624419 0.209700
166 0.546094 0.191400 0.631525 0.210100
167 0.424228 0.150240 0.630012 0.207500
168 0.363285 0.128780 0.636904 0.209200
169 0.453622 0.161140 0.625986 0.208600
170 0.482874 0.172320 0.620622 0.207600
171 0.363466 0.129120 0.643209 0.206600
172 0.329619 0.116680 0.641204 0.205200
173 0.484069 0.172160 0.626184 0.207400
174 0.321387 0.111680 0.647512 0.207900
175 0.460836 0.163940 0.625833 0.204800
176 0.404673 0.144360 0.637315 0.207100
177 0.307870 0.109080 0.646901 0.202200
178 0.539832 0.188320 0.625369 0.209900
179 0.398925 0.142280 0.619288 0.203000
180 0.479182 0.169500 0.629842 0.207300
181 0.409125 0.145880 0.624094 0.202800
182 0.377236 0.134020 0.633550 0.206800
183 0.324477 0.116260 0.616041 0.199100
184 0.496429 0.175200 0.626253 0.206800
185 0.371767 0.133680 0.636571 0.204600
186 0.357283 0.130060 0.623920 0.206700
187 0.450721 0.158980 0.616588 0.201800
188 0.366726 0.132320 0.627136 0.204200
189 0.429640 0.153620 0.613509 0.200100
190 0.345599 0.123720 0.644716 0.204800
191 0.375227 0.134880 0.632093 0.206000
192 0.425309 0.150060 0.621840 0.202100
193 0.516373 0.181600 0.615686 0.202600
194 0.531388 0.186700 0.615578 0.208300
195 0.413314 0.146800 0.616822 0.196400
196 0.409600 0.145720 0.633601 0.205600
197 0.451619 0.160000 0.635755 0.206500
198 0.435021 0.153900 0.620861 0.200800
199 0.400606 0.142060 0.621494 0.200200
200 0.391516 0.138520 0.614794 0.196900
201 0.365059 0.129860 0.619348 0.200800
202 0.402264 0.142960 0.615285 0.199200
203 0.410062 0.145700 0.630870 0.208300
real 39m10.866s
user 54m16.480s
sys 56m10.956s
}};
***C[0.01]-P-C[0.01]-P-F[0.1]-F[0.1]
#pre{{
############# training condition ############
500 epoch training. input_size: (28, 28) , minibatch_size: 100
, learning_rate: 0.01 , momentum: 0.9 , weight_decay: 0.0005
#############################################
########## architecture of the CNN #########
layer1 - Convolution
< kernel_size: (50, 2, 2) , stride: (1, 1) , act_func: ReLU
, border_mode: valid , out_size: (50, 27, 27) , iniW_σ: 0.010000 >
layer2 - Convolution
< kernel_size: (50, 2, 2) , stride: (1, 1) , act_func: ReLU
, border_mode: valid , out_size: (50, 26, 26) , iniW_σ: 0.010000 >
layer3 - Pooling
< downscale: (4, 4) , stride: None , out_size: (50, 6, 6) >
layer4 - fully-connected
< number_of_units: 1000 , drop_rate: 0.8 , act_func: ReLU , iniW_σ: 0.100000 >
layer5 - fully-connected
< number_of_units: 10 , drop_rate: 0.5 , act_func: softmax , iniW_σ: 0.100000 >
###########################################
epoch XE(L) error(L) XE(T) error(T)
1 2.172126 0.806540 2.027054 0.733900
2 2.035170 0.747220 1.983551 0.720100
3 1.988547 0.728440 1.891935 0.679600
4 1.929771 0.706340 1.841547 0.665000
5 1.854730 0.676500 1.812250 0.662500
6 1.780028 0.643160 1.674585 0.600200
7 1.699687 0.617240 1.555386 0.559100
8 1.654096 0.600640 1.525662 0.546600
9 1.611351 0.580120 1.495098 0.536100
10 1.573158 0.564400 1.458604 0.524000
11 1.540977 0.554460 1.446038 0.515000
12 1.560106 0.562840 1.413125 0.501200
13 1.514473 0.544700 1.421622 0.507300
14 1.496816 0.538960 1.378435 0.493700
15 1.455022 0.524800 1.385545 0.497000
16 1.441190 0.517100 1.352296 0.480000
17 1.435965 0.514920 1.292696 0.454400
18 1.412946 0.508140 1.295221 0.457200
19 1.414605 0.507120 1.279375 0.446100
20 1.413595 0.507600 1.291859 0.452400
21 1.396855 0.500060 1.294704 0.453100
22 1.382536 0.491640 1.307064 0.463800
23 1.400341 0.500040 1.308915 0.473800
24 1.339820 0.478880 1.234676 0.422900
25 1.357593 0.480360 1.235378 0.435700
26 1.346123 0.482560 1.219342 0.421800
27 1.315418 0.467780 1.202096 0.416400
28 1.327973 0.472600 1.226551 0.429700
29 1.290800 0.457280 1.187783 0.414900
30 1.265084 0.449300 1.189499 0.414900
31 1.269951 0.450260 1.162423 0.402100
32 1.229706 0.437520 1.139199 0.397700
33 1.256525 0.445140 1.143627 0.403000
34 1.228535 0.434140 1.112442 0.382400
35 1.205356 0.427200 1.081512 0.373300
36 1.227064 0.431580 1.135518 0.388100
37 1.203071 0.426680 1.066740 0.367700
38 1.174142 0.416060 1.093697 0.378100
39 1.206996 0.425060 1.083049 0.372200
40 1.177429 0.414880 1.070377 0.374600
41 1.181964 0.419780 1.078487 0.372800
42 1.154330 0.406840 1.092380 0.377600
43 1.156735 0.409060 1.095766 0.374700
44 1.136484 0.401520 1.042007 0.359600
45 1.165397 0.412300 1.030762 0.355600
46 1.111314 0.391160 1.003724 0.347200
47 1.117155 0.395880 1.034887 0.354800
48 1.160369 0.408580 1.037490 0.357200
49 1.111658 0.392120 1.008933 0.342800
50 1.089677 0.382500 1.009640 0.346600
51 1.125150 0.396400 1.015790 0.354300
52 1.086868 0.382360 1.017911 0.348000
53 1.079408 0.380480 0.987813 0.340500
54 1.101633 0.387140 0.978916 0.338000
55 1.103292 0.388400 1.017744 0.349900
56 1.055742 0.371920 0.999023 0.349200
57 1.095965 0.386880 0.991891 0.340600
58 1.075035 0.376700 1.039214 0.357900
59 1.105706 0.389480 0.962715 0.332700
60 1.058415 0.373280 0.946698 0.324500
61 1.046448 0.366340 0.974094 0.334900
62 1.060476 0.372560 0.945780 0.327400
63 1.017735 0.360360 0.925158 0.318500
64 1.049510 0.369400 0.968878 0.335600
65 1.041503 0.367640 0.958483 0.328700
66 1.079477 0.377540 0.952297 0.328200
67 1.024882 0.361820 0.931482 0.313400
68 1.014564 0.357060 0.912452 0.309900
69 1.090831 0.383480 0.928553 0.311700
70 1.057769 0.370600 0.953148 0.328400
71 1.048245 0.370240 0.910431 0.311900
72 1.061504 0.372140 0.933928 0.324300
73 1.039348 0.365600 0.908720 0.311000
74 1.017609 0.358460 0.909951 0.317000
75 1.016555 0.357520 0.906251 0.312700
76 1.038671 0.366300 0.897406 0.305500
77 0.980130 0.343920 0.895771 0.310800
78 1.013186 0.355400 0.902160 0.316300
79 0.970512 0.341200 0.889874 0.306300
80 0.985979 0.347140 0.877723 0.297400
81 0.985099 0.346940 0.871658 0.296300
82 0.987575 0.346300 0.887024 0.298000
83 0.973258 0.342320 0.872484 0.300300
84 0.979816 0.346480 0.865057 0.291500
85 0.987033 0.344540 0.860676 0.294000
86 0.989011 0.347220 0.881951 0.305300
87 0.983126 0.343620 0.885246 0.303000
88 0.948252 0.331220 0.855348 0.296700
89 1.005005 0.352520 0.883245 0.305200
90 0.958306 0.336200 0.883274 0.301500
91 0.930092 0.326480 0.851001 0.289300
92 0.951223 0.333120 0.848586 0.286800
93 0.950430 0.333220 0.861223 0.296200
94 0.927446 0.325760 0.871467 0.299800
95 0.933366 0.327760 0.874397 0.303900
96 0.913233 0.320140 0.861355 0.292000
97 0.905543 0.316420 0.855736 0.297700
98 0.975903 0.342480 0.874594 0.301200
99 0.911865 0.317940 0.833164 0.290000
100 0.978378 0.343840 0.873178 0.298500
101 0.975264 0.343500 0.855746 0.291100
102 0.936860 0.326420 0.867001 0.300800
103 0.896566 0.316180 0.825730 0.282700
104 0.954344 0.333120 0.864847 0.299100
105 0.943417 0.331460 0.845261 0.291600
106 0.915229 0.321920 0.842217 0.292300
107 0.892967 0.312880 0.834767 0.293200
108 0.923368 0.323500 0.834013 0.278800
109 0.952158 0.331680 0.876165 0.300700
110 0.960994 0.334260 0.835252 0.282000
111 0.871473 0.304960 0.807815 0.276900
112 0.964376 0.338580 0.857544 0.297400
113 0.916684 0.320640 0.878934 0.298600
114 0.892944 0.312360 0.803190 0.276600
115 0.866723 0.303060 0.851757 0.294400
116 0.912606 0.318980 0.829088 0.284500
117 0.891724 0.313700 0.818643 0.280300
118 0.907253 0.318040 0.821649 0.287600
119 0.921999 0.323040 0.829584 0.280100
120 0.900529 0.313660 0.818153 0.279500
121 0.937879 0.328620 0.839500 0.286800
122 0.911937 0.319420 0.832967 0.288600
123 0.902038 0.314560 0.805792 0.275200
124 0.961827 0.335600 0.816764 0.279600
125 0.939841 0.329580 0.829886 0.285800
126 0.891077 0.312300 0.807790 0.271300
127 0.867737 0.303160 0.806412 0.274100
128 0.882219 0.308840 0.796532 0.271600
129 0.911777 0.318700 0.794350 0.271700
130 0.857981 0.299900 0.796901 0.272500
131 0.834586 0.292920 0.790472 0.270600
132 0.905096 0.316020 0.816004 0.276700
133 0.952341 0.333660 0.822376 0.276400
134 0.849635 0.299660 0.788939 0.268100
135 0.877486 0.307080 0.781232 0.264600
136 0.900104 0.315540 0.810521 0.275700
137 0.875975 0.306880 0.795411 0.272500
138 0.891001 0.312740 0.788374 0.270100
139 0.913581 0.317280 0.821446 0.279300
140 0.835725 0.293100 0.778462 0.265000
141 0.884017 0.308620 0.799846 0.274100
142 0.870504 0.304900 0.807875 0.276700
143 0.874864 0.306060 0.823310 0.283600
144 0.871382 0.303760 0.802308 0.271100
145 0.904468 0.314620 0.797340 0.273400
146 0.862642 0.302380 0.808056 0.278400
147 0.827082 0.288960 0.787023 0.269100
148 0.907789 0.317740 0.800022 0.273600
149 0.910789 0.319600 0.818200 0.280400
150 0.866458 0.302640 0.808086 0.277100
151 0.920660 0.322460 0.782074 0.260800
152 0.885272 0.309180 0.780229 0.267400
153 0.859376 0.298380 0.800715 0.273600
154 0.801656 0.281420 0.769792 0.262200
155 0.848805 0.295900 0.770786 0.261100
156 0.813184 0.284280 0.773820 0.264500
157 0.866589 0.302500 0.772845 0.258900
158 0.843960 0.293660 0.766434 0.256700
159 0.805899 0.280620 0.754112 0.255900
160 0.809272 0.285080 0.750416 0.254300
161 0.836713 0.293200 0.776605 0.261800
162 0.819470 0.285040 0.764839 0.264300
163 0.796412 0.279000 0.744179 0.248900
164 0.805700 0.282580 0.758193 0.257200
165 0.863856 0.301800 0.764254 0.261500
166 0.833535 0.292280 0.760476 0.252900
167 0.886558 0.307960 0.773528 0.263100
168 0.787533 0.274660 0.764123 0.260300
169 0.806697 0.283460 0.778052 0.267300
170 0.792651 0.277600 0.745825 0.254400
171 0.791425 0.277540 0.737426 0.252800
172 0.890823 0.309980 0.779712 0.267800
173 0.855596 0.299080 0.764133 0.263100
174 0.915325 0.317660 0.799692 0.272900
175 0.788956 0.274880 0.753961 0.260100
176 0.774234 0.268720 0.765237 0.264900
177 0.820336 0.286560 0.735296 0.253000
178 0.827865 0.289300 0.759258 0.261800
179 0.831426 0.289600 0.822596 0.282300
180 0.865377 0.299880 0.758460 0.257600
181 0.796654 0.277280 0.741971 0.254100
182 0.765785 0.265420 0.730033 0.248200
183 0.844058 0.294440 0.772839 0.264700
184 0.831631 0.290800 0.751584 0.253000
185 0.806699 0.280100 0.744356 0.256600
186 0.828892 0.291180 0.769724 0.262600
187 0.837089 0.292120 0.757070 0.260200
188 0.769261 0.268180 0.746825 0.254200
189 0.769149 0.269240 0.732405 0.249100
190 0.760422 0.265360 0.735797 0.245100
191 0.857432 0.298640 0.785864 0.268900
192 0.818788 0.284740 0.759145 0.262900
193 0.767524 0.268940 0.725755 0.248500
194 0.803127 0.281080 0.749890 0.253200
195 0.816030 0.286400 0.758032 0.258100
196 0.860487 0.299180 0.775654 0.268400
197 0.754009 0.263060 0.744933 0.253700
198 0.843668 0.293780 0.766283 0.258300
199 0.800195 0.279700 0.758428 0.259300
200 0.801760 0.279920 0.749372 0.253300
}};
***C[0.1]-P-C[0.1]-P-F[0.1]-F[0.1]
#pre{{
############# training condition ############
500 epoch training. input_size: (28, 28) , minibatch_size: 100
, learning_rate: 0.01 , momentum: 0.9 , weight_decay: 0.0005
#############################################
########## architecture of the CNN #########
layer1 - Convolution
< kernel_size: (50, 2, 2) , stride: (1, 1) , act_func: ReLU
, border_mode: valid , out_size: (50, 27, 27) , iniW_σ: 0.100000 >
layer2 - Convolution
< kernel_size: (50, 2, 2) , stride: (1, 1) , act_func: ReLU
, border_mode: valid , out_size: (50, 26, 26) , iniW_σ: 0.100000 >
layer3 - Pooling
< downscale: (4, 4) , stride: None , out_size: (50, 6, 6) >
layer4 - fully-connected
< number_of_units: 1000 , drop_rate: 0.8 , act_func: ReLU , iniW_σ: 0.100000 >
layer5 - fully-connected
< number_of_units: 10 , drop_rate: 0.5 , act_func: softmax , iniW_σ: 0.100000 >
###########################################
epoch XE(L) error(L) XE(T) error(T)
1 2.469696 0.903120 2.302834 0.900000
2 2.302945 0.900160 2.302872 0.900000
3 2.302946 0.901760 2.302638 0.900000
4 2.302876 0.903000 2.302803 0.900000
5 2.302969 0.902140 2.302640 0.900000
6 2.302881 0.899880 2.302644 0.900000
7 2.302930 0.900580 2.302656 0.900000
8 2.302901 0.903200 2.302741 0.900000
9 2.302881 0.900880 2.302691 0.900000
10 2.302883 0.900360 2.302758 0.900000
11 2.302919 0.899480 2.302720 0.900000
12 2.302891 0.900400 2.302767 0.900000
13 2.302844 0.900360 2.302721 0.900000
14 2.302869 0.900680 2.302830 0.900000
15 2.302972 0.901260 2.302669 0.900000
16 2.302851 0.901600 2.302821 0.900000
17 2.302944 0.902180 2.302710 0.900000
18 2.302902 0.902100 2.302707 0.900000
19 2.302866 0.900760 2.302859 0.900000
20 2.302912 0.902300 2.302680 0.900000
21 2.302888 0.901260 2.302673 0.900000
22 2.302860 0.900320 2.302694 0.900000
23 2.302874 0.900360 2.302740 0.900000
24 2.302860 0.901140 2.302673 0.900000
25 2.302795 0.901300 2.302747 0.900000
26 2.302883 0.900760 2.302735 0.900000
27 2.302845 0.899840 2.302753 0.900000
28 2.302856 0.900980 2.302781 0.900000
29 2.302876 0.900140 2.302671 0.900000
30 2.302877 0.899980 2.302692 0.900000
31 2.302934 0.903280 2.302656 0.900000
32 2.302863 0.900760 2.302713 0.900000
33 2.302915 0.902880 2.302714 0.900000
34 2.302928 0.901880 2.302650 0.900000
35 2.302866 0.901380 2.302725 0.900000
36 2.302825 0.900720 2.302807 0.900000
37 2.302892 0.902460 2.302619 0.900000
38 2.302845 0.901600 2.302765 0.900000
39 2.302914 0.902720 2.302743 0.900000
40 2.302921 0.901600 2.302677 0.900000
41 2.302836 0.900940 2.302792 0.900000
42 2.302791 0.899040 2.302865 0.900000
43 2.302911 0.901320 2.302741 0.900000
44 2.302819 0.899380 2.302865 0.900000
45 2.302945 0.901820 2.302727 0.900000
46 2.302858 0.898500 2.302843 0.900000
47 2.302933 0.902640 2.302636 0.900000
48 2.302869 0.902680 2.302715 0.900000
49 2.302831 0.901920 2.302749 0.900000
50 2.302902 0.900260 2.302702 0.900000
51 2.302915 0.901580 2.302758 0.900000
52 2.302837 0.900320 2.302890 0.900000
53 2.302885 0.900160 2.302726 0.900000
54 2.302865 0.902600 2.302754 0.900000
55 2.302849 0.900520 2.302711 0.900000
56 2.302866 0.903040 2.302749 0.900000
57 2.302907 0.902060 2.302714 0.900000
58 2.302898 0.904120 2.302728 0.900000
59 2.302899 0.900840 2.302761 0.900000
60 2.302974 0.901780 2.302660 0.900000
61 2.302901 0.899720 2.302698 0.900000
62 2.302916 0.902500 2.302723 0.900000
63 2.302907 0.901860 2.302665 0.900000
64 2.302834 0.902040 2.302671 0.900000
65 2.302908 0.903960 2.302636 0.900000
66 2.302846 0.900720 2.302781 0.900000
67 2.302897 0.901640 2.302807 0.900000
68 2.302890 0.901000 2.302656 0.900000
69 2.302878 0.899900 2.302637 0.900000
70 2.302908 0.902440 2.302684 0.900000
71 2.302904 0.901340 2.302690 0.900000
72 2.302887 0.902000 2.302632 0.900000
73 2.302851 0.901180 2.302742 0.900000
74 2.302876 0.901320 2.302730 0.900000
75 2.302831 0.899560 2.302755 0.900000
76 2.302867 0.902080 2.302701 0.900000
77 2.302783 0.901000 2.303005 0.900000
78 2.302869 0.899620 2.302806 0.900000
79 2.302857 0.899900 2.302774 0.900000
80 2.302936 0.901880 2.302664 0.900000
81 2.302874 0.900220 2.302776 0.900000
82 2.302942 0.902880 2.302651 0.900000
83 2.302881 0.902540 2.302710 0.900000
84 2.302884 0.900680 2.302822 0.900000
85 2.302892 0.901160 2.302698 0.900000
86 2.302854 0.901020 2.302746 0.900000
87 2.302849 0.902120 2.302697 0.900000
88 2.302851 0.901900 2.302635 0.900000
89 2.302851 0.900780 2.302721 0.900000
90 2.302825 0.900340 2.302800 0.900000
91 2.302910 0.901940 2.302648 0.900000
92 2.302930 0.903180 2.302644 0.900000
93 2.302940 0.902940 2.302687 0.900000
94 2.302902 0.902080 2.302719 0.900000
95 2.302903 0.902620 2.302718 0.900000
96 2.302908 0.900440 2.302715 0.900000
97 2.302918 0.901280 2.302743 0.900000
98 2.302900 0.901520 2.302664 0.900000
99 2.302875 0.900100 2.302705 0.900000
100 2.302955 0.902540 2.302623 0.900000
101 2.302834 0.900540 2.302717 0.900000
102 2.302931 0.901340 2.302675 0.900000
103 2.302928 0.902440 2.302671 0.900000
104 2.302883 0.902920 2.302754 0.900000
105 2.302936 0.903300 2.302649 0.900000
106 2.302850 0.900940 2.302701 0.900000
107 2.302882 0.901440 2.302749 0.900000
108 2.302863 0.901020 2.302780 0.900000
109 2.302911 0.902860 2.302698 0.900000
110 2.302854 0.900220 2.302695 0.900000
111 2.302906 0.901480 2.302693 0.900000
112 2.302827 0.901020 2.302732 0.900000
113 2.302868 0.901000 2.302690 0.900000
114 2.302870 0.900520 2.302764 0.900000
115 2.302846 0.900280 2.302795 0.900000
116 2.302924 0.901420 2.302691 0.900000
117 2.302839 0.900840 2.302810 0.900000
118 2.302927 0.900620 2.302779 0.900000
119 2.302907 0.901980 2.302638 0.900000
120 2.302848 0.901200 2.302821 0.900000
121 2.302883 0.901360 2.302833 0.900000
122 2.302876 0.901760 2.302695 0.900000
123 2.302830 0.899780 2.302757 0.900000
124 2.302893 0.900940 2.302753 0.900000
125 2.302882 0.902880 2.302826 0.900000
126 2.302941 0.902640 2.302663 0.900000
127 2.302845 0.900620 2.302707 0.900000
128 2.302872 0.901720 2.302686 0.900000
129 2.302925 0.900740 2.302679 0.900000
130 2.302902 0.904020 2.302697 0.900000
131 2.302914 0.902560 2.302687 0.900000
132 2.302895 0.899900 2.302668 0.900000
133 2.302915 0.901800 2.302649 0.900000
134 2.302854 0.901300 2.302746 0.900000
135 2.302883 0.900820 2.302828 0.900000
136 2.302909 0.900720 2.302699 0.900000
137 2.302908 0.900320 2.302631 0.900000
138 2.302826 0.899820 2.302702 0.900000
139 2.302849 0.900100 2.302845 0.900000
140 2.302842 0.901300 2.302676 0.900000
141 2.302844 0.900100 2.302851 0.900000
142 2.302861 0.899400 2.302719 0.900000
143 2.302889 0.900700 2.302653 0.900000
144 2.302943 0.901700 2.302664 0.900000
145 2.302849 0.901600 2.302750 0.900000
146 2.302887 0.901880 2.302637 0.900000
147 2.302866 0.901120 2.302795 0.900000
148 2.302899 0.901120 2.302794 0.900000
149 2.302916 0.901800 2.302687 0.900000
150 2.302869 0.901600 2.302766 0.900000
151 2.302911 0.901280 2.302631 0.900000
152 2.302876 0.901760 2.302697 0.900000
153 2.302907 0.901040 2.302706 0.900000
154 2.302895 0.902340 2.302698 0.900000
155 2.302866 0.901980 2.302753 0.900000
156 2.302911 0.901380 2.302764 0.900000
157 2.302881 0.901400 2.302782 0.900000
158 2.302922 0.903480 2.302663 0.900000
159 2.302916 0.902040 2.302689 0.900000
160 2.302912 0.903460 2.302759 0.900000
161 2.302892 0.902000 2.302633 0.900000
162 2.302868 0.902920 2.302685 0.900000
163 2.302875 0.902540 2.302739 0.900000
164 2.302891 0.901720 2.302747 0.900000
165 2.302893 0.903380 2.302692 0.900000
166 2.302901 0.902320 2.302729 0.900000
167 2.302912 0.901540 2.302785 0.900000
168 2.302803 0.900920 2.302877 0.900000
169 2.302902 0.903200 2.302750 0.900000
170 2.302833 0.901080 2.302715 0.900000
171 2.302826 0.900980 2.302698 0.900000
172 2.302876 0.898880 2.302729 0.900000
173 2.302833 0.900020 2.302843 0.900000
174 2.302900 0.900860 2.302693 0.900000
175 2.302847 0.900600 2.302878 0.900000
176 2.302976 0.902760 2.302691 0.900000
177 2.302902 0.902420 2.302791 0.900000
178 2.302938 0.904280 2.302684 0.900000
179 2.302875 0.900340 2.302723 0.900000
180 2.302863 0.900860 2.302689 0.900000
181 2.302883 0.899860 2.302677 0.900000
182 2.302908 0.902560 2.302744 0.900000
183 2.302885 0.901200 2.302707 0.900000
184 2.302864 0.900340 2.302755 0.900000
185 2.302864 0.900280 2.302783 0.900000
186 2.302891 0.900180 2.302772 0.900000
187 2.302919 0.902740 2.302653 0.900000
188 2.302705 0.900640 2.303075 0.900000
189 2.303016 0.902080 2.302681 0.900000
190 2.302799 0.899260 2.302814 0.900000
191 2.302957 0.902220 2.302677 0.900000
192 2.302808 0.900160 2.302747 0.900000
193 2.302941 0.899640 2.302764 0.900000
194 2.302920 0.901400 2.302660 0.900000
195 2.302916 0.902700 2.302655 0.900000
196 2.302792 0.900780 2.302863 0.900000
197 2.302958 0.902280 2.302696 0.900000
198 2.302820 0.899840 2.302737 0.900000
199 2.302860 0.902080 2.302677 0.900000
200 2.302882 0.902160 2.302703 0.900000
201 2.302827 0.898820 2.302696 0.900000
202 2.302886 0.901280 2.302664 0.900000
203 2.302820 0.902560 2.302757 0.900000
204 2.302895 0.903140 2.302790 0.900000
205 2.302906 0.900200 2.302672 0.900000
206 2.302863 0.900160 2.302667 0.900000
207 2.302852 0.898640 2.302745 0.900000
208 2.302830 0.902600 2.302786 0.900000
209 2.302871 0.900420 2.302814 0.900000
210 2.302969 0.903040 2.302597 0.900000
211 2.302858 0.899980 2.302673 0.900000
212 2.302862 0.899840 2.302731 0.900000
213 2.302937 0.901440 2.302684 0.900000
214 2.302936 0.901320 2.302697 0.900000
215 2.302861 0.901240 2.302782 0.900000
216 2.302873 0.900540 2.302659 0.900000
217 2.302881 0.903020 2.302690 0.900000
218 2.302816 0.901500 2.302875 0.900000
219 2.302922 0.902760 2.302714 0.900000
220 2.302896 0.902720 2.302733 0.900000
221 2.302908 0.901320 2.302749 0.900000
222 2.302837 0.901660 2.302840 0.900000
223 2.302945 0.901600 2.302691 0.900000
224 2.302934 0.903160 2.302655 0.900000
real 42m3.781s
user 58m0.592s
sys 60m36.236s
}};
*ILSVRC2010 - 10クラス [#l3dbaa07]
**ex20151205-2 【学習係数ηの下げ方をいろいろ試す】 [#ex201512052]
-[[#ex201512052stasis579]]では、バリデーションの最低誤識別率がN epoch連続で下がらなかった場合にηを0.1倍している。Nは5から始まり、ηを下げる毎に5,7,9と大きくする。
-毎epoch下げる方法では、停滞時に0.1倍する方法よりも良い結果は得られなかった。[[岡田さんが行ったとき>m/2015/okada/diary/2015-12-03#tee792ca]]は良い結果が出ているので、今回の条件では学習の進行とηの減少がうまくかみ合わなかったのだろう。調整が難しそう。
***学習停滞時にηを下げる(5,7,9) [#ex201512052stasis579]
#pre{{
ex201512051600 eta*0.1_in_stasis5,7,9
############# training condition ############
100 epoch training. input_size: (227, 227) , minibatch_size: 100
, learning_rate: 0.01 , momentum: 0.9 , weight_decay: 0.0005
#############################################
########## architecture of the CNN #########
layer1 - Convolution
< kernel_size: (100, 11, 11) , stride: (4, 4) , act_func: ReLU
, border_mode: valid , out_size: (100, 55, 55) , iniW_σ: 0.010000 >
layer2 - Pooling
< downscale: (2, 2) , stride: None , out_size: (100, 27, 27) >
layer3 - Convolution
< kernel_size: (200, 5, 5) , stride: (1, 1) , act_func: ReLU
, border_mode: same , out_size: (200, 27, 27) , iniW_σ: 0.010000 >
layer4 - Pooling
< downscale: (2, 2) , stride: None , out_size: (200, 13, 13) >
layer5 - Convolution
< kernel_size: (200, 3, 3) , stride: (1, 1) , act_func: ReLU
, border_mode: same , out_size: (200, 13, 13) , iniW_σ: 0.010000 >
layer6 - Pooling
< downscale: (2, 2) , stride: None , out_size: (200, 6, 6) >
layer7 - fully-connected
< number_of_units: 4000 , drop_rate: 1.0 , act_func: ReLU , iniW_σ: 0.010000 >
layer8 - fully-connected
< number_of_units: 10 , drop_rate: 0.5 , act_func: softmax , iniW_σ: 0.010000 >
###########################################
##### CNN initialized #####
##### start training #####
epoch XE(L) error(L) XE(V) error(V) top-5(V) LE(ε)
1 2.277367 0.858319 2.251666 0.836000 0.428000 0.010000
2 1.947574 0.709496 1.879266 0.686000 0.170000 0.010000
3 1.722490 0.607395 1.663454 0.570000 0.120000 0.010000
4 1.535354 0.525966 1.477183 0.518000 0.082000 0.010000
5 1.398474 0.477227 1.365852 0.466000 0.064000 0.010000
6 1.278411 0.431765 1.285098 0.466000 0.056000 0.010000
7 1.183516 0.401849 1.216781 0.418000 0.058000 0.010000
8 1.103606 0.367731 1.100320 0.372000 0.034000 0.010000
9 1.041006 0.345882 1.085061 0.392000 0.044000 0.010000
10 0.994406 0.335210 1.086543 0.382000 0.054000 0.010000
11 0.943029 0.318067 1.052583 0.362000 0.040000 0.010000
12 0.917502 0.307059 1.090705 0.376000 0.060000 0.010000
13 0.853778 0.290084 0.947194 0.316000 0.038000 0.010000
14 0.813730 0.275630 1.032880 0.320000 0.050000 0.010000
15 0.779904 0.263361 0.889893 0.294000 0.040000 0.010000
16 0.738899 0.253445 0.896218 0.308000 0.038000 0.010000
17 0.709131 0.237059 0.903660 0.314000 0.042000 0.010000
18 0.658844 0.224790 0.973461 0.332000 0.036000 0.010000
19 0.633154 0.213529 0.982704 0.342000 0.048000 0.010000
20 0.613156 0.203697 0.854572 0.286000 0.036000 0.010000
21 0.569365 0.195462 0.836357 0.316000 0.034000 0.010000
22 0.572420 0.196387 0.830615 0.260000 0.030000 0.010000
23 0.512127 0.177311 0.817538 0.278000 0.038000 0.010000
24 0.480103 0.162773 0.847216 0.266000 0.040000 0.010000
25 0.451453 0.152689 0.872383 0.260000 0.034000 0.010000
26 0.449348 0.152521 0.888524 0.262000 0.026000 0.010000
27 0.423916 0.144202 0.858103 0.248000 0.032000 0.010000
28 0.388746 0.133025 0.845194 0.246000 0.030000 0.010000
29 0.351130 0.121429 0.791058 0.242000 0.024000 0.010000
30 0.333022 0.112773 0.827974 0.272000 0.022000 0.010000
31 0.315010 0.109580 0.917370 0.276000 0.032000 0.010000
32 0.305722 0.106723 0.853677 0.244000 0.026000 0.010000
33 0.279666 0.097143 0.888657 0.258000 0.032000 0.010000
34 0.258500 0.086891 0.831414 0.252000 0.036000 0.010000
35 0.193057 0.061933 0.806545 0.226000 0.026000 0.001000
36 0.153609 0.050000 0.812244 0.238000 0.026000 0.001000
37 0.137934 0.042437 0.816455 0.226000 0.026000 0.001000
38 0.133661 0.041345 0.836213 0.228000 0.034000 0.001000
39 0.116838 0.035210 0.842869 0.220000 0.030000 0.001000
40 0.121436 0.036555 0.840142 0.224000 0.030000 0.001000
41 0.112238 0.034034 0.846096 0.226000 0.026000 0.001000
42 0.098387 0.028571 0.863834 0.220000 0.030000 0.001000
43 0.099678 0.030168 0.850278 0.220000 0.032000 0.001000
44 0.093421 0.028655 0.863470 0.212000 0.030000 0.001000
45 0.091027 0.029244 0.866952 0.218000 0.030000 0.001000
46 0.093821 0.031008 0.863583 0.212000 0.030000 0.001000
47 0.084964 0.026387 0.878371 0.220000 0.026000 0.001000
48 0.081341 0.024958 0.931855 0.234000 0.032000 0.001000
49 0.086145 0.027311 0.894826 0.220000 0.034000 0.001000
50 0.076297 0.022101 0.883321 0.220000 0.030000 0.001000
51 0.075838 0.022353 0.893859 0.216000 0.030000 0.001000
52 0.074963 0.023782 0.903655 0.216000 0.032000 0.000100
53 0.072232 0.022269 0.900654 0.216000 0.032000 0.000100
54 0.069701 0.020756 0.900656 0.220000 0.032000 0.000100
55 0.066896 0.018655 0.901350 0.222000 0.030000 0.000100
56 0.067094 0.019916 0.899645 0.220000 0.028000 0.000100
57 0.070243 0.019328 0.900114 0.222000 0.030000 0.000100
58 0.067378 0.020504 0.897133 0.220000 0.032000 0.000100
59 0.070497 0.021597 0.894566 0.216000 0.032000 0.000100
60 0.069522 0.020588 0.897999 0.216000 0.030000 0.000100
61 0.072902 0.021261 0.898477 0.216000 0.030000 0.000010
62 0.068760 0.021429 0.898896 0.216000 0.030000 0.000010
63 0.070049 0.021429 0.898424 0.216000 0.030000 0.000010
64 0.064309 0.018824 0.897976 0.216000 0.030000 0.000010
65 0.062635 0.017983 0.898582 0.216000 0.032000 0.000010
66 0.065354 0.019412 0.898488 0.216000 0.032000 0.000010
67 0.068760 0.021092 0.898843 0.216000 0.030000 0.000010
68 0.069648 0.020252 0.898074 0.216000 0.032000 0.000010
69 0.068220 0.019832 0.897917 0.216000 0.032000 0.000010
70 0.062939 0.018235 0.896810 0.216000 0.032000 0.000010
71 0.072348 0.021261 0.897577 0.216000 0.032000 0.000010
72 0.068491 0.020084 0.898607 0.218000 0.030000 0.000010
73 0.073683 0.021345 0.898009 0.218000 0.032000 0.000010
74 0.070746 0.020420 0.898094 0.218000 0.032000 0.000010
75 0.070437 0.021429 0.898307 0.218000 0.032000 0.000010
Terminated
real 86m40.138s
user 131m43.860s
sys 56m52.428s
}};
***1epoch終了毎にηを0.95倍 [#ex201512052095]
#pre{{
ex201512051737 eta*0.95/epoch
############# training condition ############
100 epoch training. input_size: (227, 227) , minibatch_size: 100
, learning_rate: 0.01 , momentum: 0.9 , weight_decay: 0.0005
#############################################
########## architecture of the CNN #########
layer1 - Convolution
< kernel_size: (100, 11, 11) , stride: (4, 4) , act_func: ReLU
, border_mode: valid , out_size: (100, 55, 55) , iniW_σ: 0.010000 >
layer2 - Pooling
< downscale: (2, 2) , stride: None , out_size: (100, 27, 27) >
layer3 - Convolution
< kernel_size: (200, 5, 5) , stride: (1, 1) , act_func: ReLU
, border_mode: same , out_size: (200, 27, 27) , iniW_σ: 0.010000 >
layer4 - Pooling
< downscale: (2, 2) , stride: None , out_size: (200, 13, 13) >
layer5 - Convolution
< kernel_size: (200, 3, 3) , stride: (1, 1) , act_func: ReLU
, border_mode: same , out_size: (200, 13, 13) , iniW_σ: 0.010000 >
layer6 - Pooling
< downscale: (2, 2) , stride: None , out_size: (200, 6, 6) >
layer7 - fully-connected
< number_of_units: 4000 , drop_rate: 1.0 , act_func: ReLU , iniW_σ: 0.010000 >
layer8 - fully-connected
< number_of_units: 10 , drop_rate: 0.5 , act_func: softmax , iniW_σ: 0.010000 >
###########################################
##### CNN initialized #####
##### start training #####
epoch XE(L) error(L) XE(V) error(V) top-5(V) LE(ε)
1 2.277352 0.858235 2.251584 0.836000 0.430000 0.010000
2 1.951836 0.711597 1.881063 0.684000 0.168000 0.009500
3 1.728413 0.610756 1.695229 0.596000 0.128000 0.009025
4 1.545813 0.534538 1.509981 0.504000 0.098000 0.008574
5 1.405563 0.480252 1.373952 0.462000 0.058000 0.008145
6 1.296154 0.437731 1.296158 0.460000 0.064000 0.007738
7 1.208430 0.407143 1.234515 0.408000 0.056000 0.007351
8 1.127124 0.376555 1.145219 0.404000 0.040000 0.006983
9 1.076626 0.359160 1.081067 0.376000 0.038000 0.006634
10 1.029072 0.344874 1.064646 0.384000 0.048000 0.006302
11 0.982807 0.330336 1.006307 0.346000 0.036000 0.005987
12 0.946189 0.313361 1.031065 0.354000 0.046000 0.005688
13 0.899290 0.298655 0.985757 0.342000 0.040000 0.005404
14 0.860212 0.287899 0.957702 0.320000 0.036000 0.005133
15 0.825233 0.274202 0.953047 0.338000 0.038000 0.004877
16 0.799453 0.267815 0.914362 0.320000 0.032000 0.004633
17 0.770583 0.257899 0.913052 0.314000 0.034000 0.004401
18 0.730774 0.246134 0.857517 0.298000 0.032000 0.004181
19 0.709381 0.237647 0.878458 0.298000 0.032000 0.003972
20 0.701334 0.237143 0.876736 0.304000 0.032000 0.003774
21 0.657359 0.222689 0.919825 0.358000 0.032000 0.003585
22 0.653392 0.220000 0.891270 0.312000 0.034000 0.003406
23 0.611222 0.205294 0.822501 0.276000 0.034000 0.003235
24 0.586018 0.200000 0.793090 0.262000 0.030000 0.003074
25 0.583927 0.196807 0.814655 0.280000 0.034000 0.002920
26 0.559830 0.188403 0.835614 0.278000 0.034000 0.002774
27 0.547732 0.183277 0.802821 0.270000 0.040000 0.002635
28 0.517970 0.175294 0.810674 0.286000 0.028000 0.002503
29 0.503597 0.172941 0.766553 0.266000 0.030000 0.002378
30 0.503102 0.169916 0.819078 0.286000 0.038000 0.002259
31 0.475812 0.162353 0.801845 0.262000 0.026000 0.002146
32 0.459279 0.155294 0.823232 0.272000 0.032000 0.002039
33 0.442686 0.152185 0.766325 0.264000 0.030000 0.001937
34 0.431171 0.144790 0.789724 0.248000 0.032000 0.001840
35 0.418669 0.140168 0.767989 0.256000 0.030000 0.001748
36 0.401216 0.134034 0.778885 0.278000 0.028000 0.001661
37 0.388688 0.134958 0.806615 0.272000 0.034000 0.001578
38 0.379779 0.125294 0.800105 0.262000 0.032000 0.001499
39 0.366716 0.121513 0.810097 0.270000 0.028000 0.001424
40 0.358762 0.118655 0.811743 0.264000 0.030000 0.001353
41 0.340084 0.112101 0.847037 0.276000 0.032000 0.001285
42 0.320878 0.107899 0.822449 0.262000 0.034000 0.001221
43 0.323759 0.108571 0.814669 0.258000 0.030000 0.001160
44 0.309059 0.101849 0.796212 0.260000 0.024000 0.001102
45 0.314341 0.105798 0.777682 0.248000 0.032000 0.001047
46 0.307798 0.102101 0.822097 0.278000 0.032000 0.000994
47 0.285250 0.091345 0.807610 0.260000 0.030000 0.000945
48 0.275792 0.090084 0.796285 0.278000 0.032000 0.000897
49 0.273662 0.089580 0.820993 0.256000 0.030000 0.000853
50 0.264182 0.085126 0.816834 0.252000 0.028000 0.000810
51 0.256567 0.085294 0.803669 0.266000 0.030000 0.000769
52 0.251788 0.081765 0.833863 0.268000 0.030000 0.000731
53 0.249871 0.083193 0.821690 0.268000 0.028000 0.000694
54 0.236485 0.076723 0.822134 0.272000 0.028000 0.000660
55 0.234667 0.077983 0.828458 0.258000 0.032000 0.000627
56 0.232844 0.075966 0.856038 0.258000 0.028000 0.000595
57 0.224520 0.070420 0.820648 0.268000 0.026000 0.000566
58 0.210911 0.068487 0.821500 0.252000 0.028000 0.000537
59 0.222944 0.074286 0.825125 0.264000 0.032000 0.000510
60 0.218585 0.070756 0.823545 0.252000 0.028000 0.000485
61 0.213524 0.069496 0.825661 0.270000 0.026000 0.000461
62 0.204758 0.067479 0.825810 0.264000 0.026000 0.000438
63 0.203523 0.066975 0.861448 0.260000 0.030000 0.000416
64 0.199064 0.062101 0.840673 0.266000 0.028000 0.000395
65 0.194264 0.063445 0.858491 0.258000 0.034000 0.000375
Terminated
real 78m6.722s
user 117m24.212s
sys 49m30.216s
}};
***1epoch終了毎にηを0.96倍 [#ex201512052096]
#pre{{
ex201512051900 eta*0.96/epoch
############# training condition ############
100 epoch training. input_size: (227, 227) , minibatch_size: 100
, learning_rate: 0.01 , momentum: 0.9 , weight_decay: 0.0005
#############################################
########## architecture of the CNN #########
layer1 - Convolution
< kernel_size: (100, 11, 11) , stride: (4, 4) , act_func: ReLU
, border_mode: valid , out_size: (100, 55, 55) , iniW_σ: 0.010000 >
layer2 - Pooling
< downscale: (2, 2) , stride: None , out_size: (100, 27, 27) >
layer3 - Convolution
< kernel_size: (200, 5, 5) , stride: (1, 1) , act_func: ReLU
, border_mode: same , out_size: (200, 27, 27) , iniW_σ: 0.010000 >
layer4 - Pooling
< downscale: (2, 2) , stride: None , out_size: (200, 13, 13) >
layer5 - Convolution
< kernel_size: (200, 3, 3) , stride: (1, 1) , act_func: ReLU
, border_mode: same , out_size: (200, 13, 13) , iniW_σ: 0.010000 >
layer6 - Pooling
< downscale: (2, 2) , stride: None , out_size: (200, 6, 6) >
layer7 - fully-connected
< number_of_units: 4000 , drop_rate: 1.0 , act_func: ReLU , iniW_σ: 0.010000 >
layer8 - fully-connected
< number_of_units: 10 , drop_rate: 0.5 , act_func: softmax , iniW_σ: 0.010000 >
###########################################
##### CNN initialized #####
##### start training #####
epoch XE(L) error(L) XE(V) error(V) top-5(V) LE(ε)
1 2.277385 0.858319 2.251771 0.836000 0.428000 0.010000
2 1.950632 0.710924 1.882485 0.686000 0.170000 0.009600
3 1.727466 0.609832 1.686233 0.584000 0.130000 0.009216
4 1.544110 0.530924 1.506889 0.512000 0.094000 0.008847
5 1.411264 0.484454 1.378533 0.458000 0.062000 0.008493
6 1.288491 0.434034 1.290672 0.460000 0.060000 0.008154
7 1.198105 0.404790 1.239868 0.410000 0.060000 0.007828
8 1.125188 0.374874 1.131541 0.378000 0.038000 0.007514
9 1.063416 0.353950 1.080499 0.386000 0.040000 0.007214
10 1.024589 0.341176 1.049794 0.370000 0.046000 0.006925
11 0.971587 0.326555 1.000571 0.334000 0.036000 0.006648
12 0.936320 0.308151 1.041155 0.350000 0.046000 0.006382
13 0.879419 0.291429 0.972596 0.338000 0.040000 0.006127
14 0.849814 0.285966 0.976599 0.316000 0.046000 0.005882
15 0.814663 0.273361 0.923654 0.326000 0.034000 0.005647
16 0.783737 0.262437 0.880429 0.290000 0.028000 0.005421
17 0.757869 0.253025 0.904275 0.300000 0.042000 0.005204
18 0.714593 0.240420 0.857306 0.298000 0.030000 0.004996
19 0.692006 0.230000 0.893808 0.298000 0.038000 0.004796
20 0.680247 0.229412 0.859658 0.296000 0.032000 0.004604
21 0.636739 0.214790 0.907512 0.356000 0.028000 0.004420
22 0.639926 0.217059 0.905840 0.314000 0.032000 0.004243
23 0.591634 0.200504 0.835064 0.286000 0.036000 0.004073
24 0.562105 0.191092 0.786839 0.266000 0.032000 0.003911
25 0.556643 0.190420 0.812612 0.278000 0.034000 0.003754
26 0.528213 0.178067 0.857938 0.294000 0.034000 0.003604
27 0.522009 0.174370 0.799086 0.260000 0.034000 0.003460
28 0.488593 0.165378 0.793842 0.270000 0.032000 0.003321
29 0.466996 0.159076 0.753675 0.258000 0.032000 0.003189
30 0.459218 0.154874 0.803812 0.258000 0.030000 0.003061
31 0.433331 0.144454 0.807739 0.264000 0.030000 0.002939
32 0.416626 0.140504 0.834518 0.272000 0.030000 0.002821
33 0.396543 0.131933 0.785867 0.256000 0.034000 0.002708
34 0.387115 0.133613 0.813038 0.260000 0.036000 0.002600
35 0.378541 0.128151 0.768092 0.258000 0.028000 0.002496
36 0.348519 0.117731 0.788247 0.280000 0.030000 0.002396
37 0.332313 0.113109 0.863426 0.262000 0.032000 0.002300
38 0.324902 0.109580 0.809045 0.272000 0.024000 0.002208
39 0.303767 0.097815 0.834204 0.254000 0.032000 0.002120
40 0.301304 0.099160 0.829597 0.262000 0.030000 0.002035
41 0.280919 0.092185 0.901386 0.270000 0.032000 0.001954
42 0.252134 0.083782 0.851724 0.242000 0.030000 0.001876
43 0.260497 0.087059 0.891211 0.278000 0.028000 0.001800
44 0.238548 0.080084 0.870616 0.268000 0.028000 0.001728
45 0.242894 0.083109 0.811577 0.232000 0.028000 0.001659
46 0.238153 0.080924 0.879595 0.270000 0.032000 0.001593
47 0.214860 0.069748 0.844704 0.240000 0.034000 0.001529
48 0.201777 0.065966 0.820648 0.250000 0.030000 0.001468
49 0.196741 0.062857 0.883993 0.248000 0.030000 0.001409
50 0.191576 0.063025 0.857124 0.244000 0.032000 0.001353
51 0.183070 0.060168 0.857630 0.244000 0.030000 0.001299
52 0.174940 0.055546 0.895181 0.250000 0.034000 0.001247
53 0.164991 0.053613 0.865979 0.250000 0.030000 0.001197
54 0.160129 0.051092 0.868790 0.250000 0.032000 0.001149
55 0.151970 0.046387 0.865217 0.242000 0.030000 0.001103
56 0.154774 0.050252 0.906302 0.242000 0.032000 0.001059
57 0.149370 0.046807 0.888584 0.254000 0.030000 0.001017
58 0.137899 0.043025 0.872256 0.260000 0.032000 0.000976
59 0.143911 0.046387 0.892266 0.250000 0.030000 0.000937
60 0.134742 0.041933 0.884923 0.264000 0.032000 0.000900
61 0.132981 0.042017 0.882259 0.270000 0.030000 0.000864
62 0.125207 0.039328 0.902916 0.252000 0.032000 0.000829
63 0.122665 0.037395 0.941738 0.264000 0.028000 0.000796
64 0.113982 0.035546 0.895369 0.254000 0.028000 0.000764
65 0.111341 0.034370 0.920459 0.250000 0.028000 0.000733
66 0.104054 0.030840 0.947754 0.242000 0.030000 0.000704
67 0.111851 0.035042 0.950643 0.250000 0.032000 0.000676
68 0.112251 0.034454 0.938128 0.264000 0.030000 0.000649
69 0.106461 0.032689 0.936006 0.248000 0.030000 0.000623
70 0.097583 0.026050 0.958789 0.256000 0.026000 0.000598
71 0.105913 0.032437 0.950073 0.254000 0.030000 0.000574
72 0.099856 0.030504 0.951158 0.244000 0.032000 0.000551
73 0.100141 0.030504 0.931178 0.248000 0.028000 0.000529
74 0.097282 0.030084 0.910496 0.246000 0.028000 0.000508
75 0.099303 0.030336 0.973177 0.252000 0.032000 0.000488
76 0.092883 0.027059 0.956271 0.258000 0.036000 0.000468
77 0.090073 0.026555 0.933896 0.246000 0.034000 0.000449
78 0.084194 0.023361 0.964180 0.254000 0.034000 0.000431
79 0.080961 0.023361 0.947876 0.248000 0.030000 0.000414
80 0.090716 0.027395 0.966286 0.248000 0.032000 0.000398
81 0.081042 0.023782 0.947670 0.256000 0.030000 0.000382
82 0.083237 0.023361 0.960480 0.252000 0.034000 0.000366
83 0.079486 0.022437 0.952245 0.264000 0.032000 0.000352
84 0.078496 0.022521 0.965654 0.254000 0.030000 0.000338
85 0.083684 0.023529 0.941120 0.250000 0.030000 0.000324
86 0.077509 0.022941 0.951007 0.254000 0.030000 0.000311
87 0.076359 0.022101 0.948779 0.252000 0.032000 0.000299
88 0.068875 0.018739 0.952948 0.246000 0.030000 0.000287
89 0.076159 0.021765 0.976507 0.252000 0.034000 0.000275
90 0.075564 0.022773 0.953888 0.252000 0.030000 0.000264
91 0.071741 0.019916 0.966227 0.248000 0.032000 0.000254
92 0.075323 0.021765 0.969359 0.250000 0.032000 0.000244
93 0.070876 0.019916 0.966330 0.250000 0.032000 0.000234
94 0.071192 0.019916 0.976763 0.248000 0.032000 0.000225
95 0.074919 0.022941 0.953077 0.250000 0.032000 0.000216
96 0.071012 0.020252 0.973520 0.256000 0.030000 0.000207
97 0.074735 0.022353 0.960020 0.242000 0.034000 0.000199
98 0.070708 0.018403 0.966824 0.246000 0.032000 0.000191
99 0.069960 0.019328 0.967983 0.242000 0.032000 0.000183
100 0.067620 0.018571 0.965933 0.246000 0.032000 0.000176
#elapsed time after 100 epoch : 7049.65958095 [sec]
real 117m41.328s
user 180m19.676s
sys 72m16.240s
}};
**ex20151204-2【学習係数ηを段階的に下げてみるテスト】 [#ex201512042]
-バリデーションの最低誤識別率が3epoch連続で下がらなかった場合にηを0.1倍する処理を試してみる。
-下げ方が適切かはともかくとして、効果的に働いていると判断できそう。とくにepoch19。
-ηが小さくなるほど次に0.1倍するまでの基準を緩くするのが良さげ。回数制限も設けた方が良いだろうか。
#pre{{
ex201512041734
############# training condition ############
100 epoch training. input_size: (227, 227) , minibatch_size: 100
, learning_rate: 0.01 , momentum: 0.9 , weight_decay: 0.0005
#############################################
########## architecture of the CNN #########
layer1 - Convolution
< kernel_size: (100, 11, 11) , stride: (4, 4) , act_func: ReLU
, border_mode: valid , out_size: (100, 55, 55) , iniW_σ: 0.010000 >
layer2 - Pooling
< downscale: (2, 2) , stride: None , out_size: (100, 27, 27) >
layer3 - Convolution
< kernel_size: (200, 5, 5) , stride: (1, 1) , act_func: ReLU
, border_mode: same , out_size: (200, 27, 27) , iniW_σ: 0.010000 >
layer4 - Pooling
< downscale: (2, 2) , stride: None , out_size: (200, 13, 13) >
layer5 - Convolution
< kernel_size: (200, 3, 3) , stride: (1, 1) , act_func: ReLU
, border_mode: same , out_size: (200, 13, 13) , iniW_σ: 0.010000 >
layer6 - Pooling
< downscale: (2, 2) , stride: None , out_size: (200, 6, 6) >
layer7 - fully-connected
< number_of_units: 4000 , drop_rate: 1.0 , act_func: ReLU , iniW_σ: 0.010000 >
layer8 - fully-connected
< number_of_units: 10 , drop_rate: 0.5 , act_func: softmax , iniW_σ: 0.010000 >
###########################################
##### CNN initialized #####
##### start training #####
epoch XE(L) error(L) XE(V) error(V) top-5(V) LE(ε)
1 2.277364 0.858319 2.251660 0.836000 0.428000 0.0100
2 1.947058 0.709412 1.877985 0.690000 0.170000 0.0100
3 1.724140 0.609244 1.668289 0.574000 0.118000 0.0100
4 1.534381 0.524790 1.480330 0.502000 0.090000 0.0100
5 1.383491 0.470420 1.338253 0.466000 0.056000 0.0100
6 1.286580 0.435630 1.312038 0.476000 0.070000 0.0100
7 1.182439 0.402017 1.206981 0.418000 0.054000 0.0100
8 1.103736 0.368571 1.096689 0.372000 0.038000 0.0100
9 1.041839 0.346050 1.067890 0.380000 0.050000 0.0100
10 0.997058 0.334202 1.074426 0.384000 0.052000 0.0100
11 0.943790 0.318487 1.027471 0.356000 0.032000 0.0100
12 0.910211 0.306303 1.037716 0.340000 0.044000 0.0100
13 0.850385 0.287227 0.945195 0.316000 0.030000 0.0100
14 0.806556 0.271933 0.996095 0.322000 0.048000 0.0100
15 0.781184 0.263866 0.905792 0.300000 0.036000 0.0100
16 0.732171 0.248235 0.898016 0.316000 0.034000 0.0100
17 0.710222 0.237647 0.918373 0.302000 0.042000 0.0100
18 0.656151 0.223277 0.912142 0.306000 0.032000 0.0100
19 0.631633 0.210336 0.793342 0.274000 0.034000 0.0010
20 0.539100 0.182941 0.757377 0.258000 0.034000 0.0010
21 0.517366 0.172773 0.758475 0.254000 0.028000 0.0010
22 0.502556 0.166387 0.747837 0.256000 0.028000 0.0010
23 0.486466 0.164370 0.749277 0.254000 0.032000 0.0010
24 0.476005 0.162353 0.749252 0.252000 0.032000 0.0010
25 0.463468 0.157563 0.754237 0.254000 0.034000 0.0010
26 0.457040 0.153193 0.745088 0.246000 0.034000 0.0010
27 0.456977 0.154538 0.726688 0.244000 0.034000 0.0010
28 0.447599 0.150504 0.737343 0.256000 0.032000 0.0010
29 0.433964 0.147395 0.739840 0.254000 0.032000 0.0010
30 0.433755 0.144454 0.755817 0.248000 0.030000 0.0010
31 0.422160 0.142269 0.729965 0.244000 0.030000 0.0001
32 0.412104 0.138655 0.730816 0.244000 0.030000 0.0001
33 0.411611 0.138319 0.731842 0.242000 0.030000 0.0001
34 0.403684 0.132101 0.726915 0.242000 0.030000 0.0001
35 0.410788 0.137563 0.728174 0.236000 0.030000 0.0001
36 0.411745 0.137563 0.726576 0.244000 0.030000 0.0001
37 0.405807 0.134538 0.728789 0.246000 0.030000 0.0001
38 0.413303 0.138235 0.726447 0.240000 0.030000 0.0001
39 0.402246 0.133697 0.727214 0.240000 0.030000 0.0000
Terminated
real 46m5.150s
user 68m54.232s
sys 30m7.252s
}};
**ex20151203 【Convの出力イメージのサイズを入力イメージと同サイズにできるように】[#ex20151203]
conv2dをborder_mode = ‘full’でパディングを行った時の出力イメージの大きさが元画像より大きくなる(元画像幅(X) + フィルター幅(W) - 1)のが気に食わなかったので、元画像と同じサイズになるように下記の処理を加えて実験。CNNの設計出力で「border_mode: same」となっているConv層が対象。&br;
con2d関数でborder_mode = ‘full’とした時に得られたイメージの縦横幅から(W-1-W/2) : (X+W-1-W/2)の範囲を指定して値を取り出し、これにバイアスを付け加えた値を層の出力とすることで、入力イメージと同サイズのイメージが出力に。なお、こうして指定した範囲では今のところ、Wが偶数の時に良い値を取り出せるか分かりません(理由は割愛。今回の実験ではWはすべて奇数なのでこの問題の心配はない・・・はず)。
-[[#ex20151201c4p2]]との比較( https://drive.google.com/open?id=0B9W18yqQO6JAWUN4cGt1M3BmQmc )
-若干過学習してる気がしなくもないが、ほぼ変わらないということにしておく。良くなるかと思ったけど・・・。
-パディング使って元画像より大きくしてしまうのはいろいろ問題あると思うというか気に入らないので、この方法を使うかborder_mode=‘valid’だけで今後考えていきたいが・・・100クラスでもっと条件整えて実験してみるべきだろうか。
-実行時間はおよそ1.13倍に。layer5での畳み込み計算量の違いのせいだろう。
#pre{{
ex201512031617
############# training condition ############
100 epoch training. input_size: (227, 227) , minibatch_size: 100
, learning_rate: 0.01 , momentum: 0.9 , weight_decay: 0.0005
#############################################
########## architecture of the CNN #########
layer1 - Convolution
< kernel_size: (100, 11, 11) , stride: (4, 4) , act_func: ReLU
, border_mode: valid , out_size: (100, 55, 55) , iniW_σ: 0.001000 >
layer2 - Pooling
< downscale: (2, 2) , stride: None , out_size: (100, 27, 27) >
layer3 - Convolution
< kernel_size: (200, 5, 5) , stride: (1, 1) , act_func: ReLU
, border_mode: same , out_size: (200, 27, 27) , iniW_σ: 0.001000 >
layer4 - Pooling
< downscale: (2, 2) , stride: None , out_size: (200, 13, 13) >
layer5 - Convolution
< kernel_size: (200, 3, 3) , stride: (1, 1) , act_func: ReLU
, border_mode: same , out_size: (200, 13, 13) , iniW_σ: 0.001000 >
layer6 - Pooling
< downscale: (2, 2) , stride: None , out_size: (200, 6, 6) >
layer7 - fully-connected
< number_of_units: 4000 , drop_rate: 1.0 , act_func: ReLU , iniW_σ: 0.010000 >
layer8 - fully-connected
< number_of_units: 4000 , drop_rate: 0.5 , act_func: ReLU , iniW_σ: 0.010000 >
layer9 - fully-connected
< number_of_units: 10 , drop_rate: 0.5 , act_func: softmax , iniW_σ: 0.010000 >
###########################################
##### CNN initialized #####
##### start training #####
epoch XE(L) error(L) XE(V) error(V)
1 2.261595 0.847815 2.231633 0.866000
2 2.121800 0.793529 2.115445 0.776000
3 2.049488 0.753109 2.110948 0.778000
4 2.013915 0.739496 2.093595 0.780000
5 1.992620 0.729580 2.050377 0.760000
6 1.975841 0.720756 2.043442 0.724000
7 1.962802 0.708908 2.048557 0.734000
8 1.942856 0.701176 2.060472 0.756000
9 1.944721 0.700924 2.013132 0.742000
10 1.927313 0.694874 2.041432 0.738000
11 1.914587 0.688571 2.005463 0.712000
12 1.912851 0.681345 2.000884 0.736000
13 1.885119 0.677563 1.947328 0.696000
14 1.876197 0.672857 1.942985 0.664000
15 1.846004 0.657815 1.946611 0.680000
16 1.810703 0.642521 1.959792 0.702000
17 1.786401 0.627899 1.872008 0.648000
18 1.755359 0.617731 1.825710 0.626000
19 1.723820 0.601513 1.836580 0.626000
20 1.669675 0.584538 1.714131 0.598000
21 1.610878 0.557143 1.656381 0.544000
22 1.565316 0.535210 1.620164 0.530000
23 1.514481 0.518151 1.584729 0.518000
24 1.463522 0.499916 1.507924 0.530000
25 1.404998 0.479916 1.483843 0.496000
26 1.328340 0.452941 1.406143 0.486000
27 1.281395 0.428235 1.389601 0.486000
28 1.224581 0.411849 1.331589 0.456000
29 1.188548 0.393361 1.211243 0.422000
30 1.143402 0.384538 1.229746 0.414000
31 1.105005 0.368235 1.183988 0.406000
32 1.074304 0.360756 1.090274 0.388000
33 1.024225 0.345126 1.076960 0.368000
34 0.990604 0.332185 1.154183 0.416000
35 0.974571 0.324118 1.228499 0.390000
36 0.955587 0.318151 1.059609 0.366000
37 0.893780 0.302353 1.043145 0.346000
38 0.893082 0.301765 1.077651 0.356000
39 0.856138 0.290420 1.029480 0.342000
40 0.832944 0.279244 1.009248 0.358000
41 0.810006 0.274370 0.976203 0.334000
42 0.785528 0.266975 0.956664 0.342000
43 0.747682 0.252689 0.956831 0.336000
44 0.726532 0.246134 0.987396 0.326000
45 0.714824 0.240000 0.945414 0.316000
46 0.687596 0.230672 0.982540 0.334000
47 0.662129 0.227143 0.963468 0.310000
48 0.653706 0.222101 0.875850 0.306000
49 0.646374 0.222941 0.994816 0.344000
50 0.602276 0.206723 0.991948 0.312000
51 0.569963 0.195546 0.923926 0.308000
52 0.542956 0.186555 0.921159 0.294000
53 0.533963 0.181345 0.982949 0.300000
54 0.507973 0.173529 1.057671 0.316000
55 0.487629 0.166555 0.925082 0.308000
56 0.458850 0.157983 0.910515 0.288000
57 0.445771 0.150252 1.062388 0.312000
58 0.428169 0.142605 0.966293 0.304000
59 0.401519 0.136303 0.998146 0.326000
60 0.388616 0.130420 1.031176 0.300000
61 0.385073 0.131681 0.951174 0.298000
62 0.349887 0.116891 1.062435 0.306000
63 0.340367 0.117143 0.992822 0.304000
64 0.313215 0.106639 1.093462 0.280000
65 0.319747 0.110252 1.107072 0.332000
66 0.292158 0.101261 1.047072 0.306000
67 0.283548 0.096807 1.044728 0.296000
68 0.251862 0.085042 1.161073 0.296000
69 0.256815 0.086639 1.234181 0.294000
70 0.226725 0.074538 1.140025 0.296000
71 0.222152 0.075714 1.126003 0.290000
72 0.211216 0.069748 1.180236 0.310000
73 0.190438 0.065714 1.185414 0.294000
74 0.199757 0.069580 1.241724 0.296000
75 0.184444 0.058739 1.166877 0.294000
76 0.181040 0.063782 1.324296 0.306000
77 0.179575 0.059244 1.104675 0.292000
78 0.182981 0.061008 1.323485 0.286000
79 0.144122 0.046723 1.212137 0.270000
80 0.171675 0.056218 1.315203 0.298000
81 0.150110 0.048487 1.165764 0.304000
82 0.133821 0.045714 1.341715 0.320000
83 0.130722 0.043697 1.352473 0.288000
84 0.123045 0.039664 1.235930 0.296000
85 0.112687 0.037059 1.360362 0.290000
86 0.106411 0.033950 1.350940 0.300000
87 0.110522 0.037143 1.332613 0.300000
88 0.112628 0.036303 1.392944 0.320000
89 0.114699 0.039412 1.380837 0.302000
90 0.110989 0.036134 1.198287 0.294000
91 0.098821 0.033025 1.243859 0.282000
92 0.099191 0.033025 1.237007 0.284000
93 0.091087 0.029496 1.279187 0.290000
94 0.091195 0.029832 1.363711 0.280000
95 0.100856 0.033025 1.232120 0.276000
96 0.082839 0.026723 1.570684 0.292000
97 0.076587 0.023613 1.360142 0.286000
98 0.071916 0.023193 1.490749 0.288000
99 0.083934 0.027899 1.455066 0.266000
100 0.079976 0.025294 1.455697 0.282000
#elapsed time after 100 epoch : 6456.72848797 [sec]
real 107m46.662s
user 191m42.832s
sys 49m24.920s
}};
**ex20151201 【Convのストライドを大きくするか、プーリングサイズを大きくするか】 [#ex20151201]
-CNNの最初のConvーPoolでイメージのサイズを(27,27)まで落とし込む際に、Convストライド(4,4)・Poolサイズ(2,2)か、Convストライド(1,1)・Poolサイズ(8,8)の二通りの設定が考えられる(Convストライドはconv2d関数のsubsampleではなく、subsample=(1,1)で[[takataka/note/2015-10-03]]下部2の方法で設定
)。
-学習推移の比較( https://drive.google.com/open?id=0B9W18yqQO6JAbmdpcVNubks3R00 )
-順伝播時の畳み込み計算の量は同じだから計算時間もあまり変わらないかと思ったら、3倍ほど差がついてしまっている。
-前者のバリデーションデータ最低誤識別率は0.256、後者は0.218で有意味な差が付いてしまっている。前者より後者の方が計算上の取りこぼしが少ない為か。
-1000クラスで行う場合20日近くかかることを考えると、後者で実験を続けるのは論外だが、この差は惜しい。convのストライドを(1,1)のままで、かつ計算量を抑える方法はないものか。
-なお、前者の結果と[[#d447b3c9]]は、[[ex20151130>#r416d0c0]]で導入した訓練画像の平均を入力から引く正規化の有無のみの違いとなる。比べてみると、この正規化が効いているらしいことが分かる( https://drive.google.com/open?id=0B9W18yqQO6JAcDd5TzJtS0lwTWM )。
***Conv_stride(4, 4), Pool_size(2, 2) [#ex20151201c4p2]
#pre{{
ex201511301829
############# training condition ############
100 epoch training. input_size: (227, 227) , minibatch_size: 100
, learning_rate: 0.01 , momentum: 0.9 , weight_decay: 0.0005
#############################################
########## architecture of the CNN #########
layer1 - Convolution
< kernel_size: (100, 11, 11) , stride: (4, 4) , act_func: ReLU
, border_mode: valid , out_size: (100, 55, 55) , iniW_σ: 0.001000 >
layer2 - Pooling
< downscale: (2, 2) , stride: None , out_size: (100, 27, 27) >
layer3 - Convolution
< kernel_size: (200, 5, 5) , stride: (1, 1) , act_func: ReLU
, border_mode: valid , out_size: (200, 23, 23) , iniW_σ: 0.001000 >
layer4 - Pooling
< downscale: (2, 2) , stride: None , out_size: (200, 11, 11) >
layer5 - Convolution
< kernel_size: (200, 3, 3) , stride: (1, 1) , act_func: ReLU
, border_mode: full , out_size: (200, 13, 13) , iniW_σ: 0.001000 >
layer6 - Pooling
< downscale: (2, 2) , stride: None , out_size: (200, 6, 6) >
layer7 - fully-connected
< number_of_units: 4000 , drop_rate: 1.0 , act_func: ReLU , iniW_σ: 0.010000 >
layer8 - fully-connected
< number_of_units: 4000 , drop_rate: 0.5 , act_func: ReLU , iniW_σ: 0.010000 >
layer9 - fully-connected
< number_of_units: 10 , drop_rate: 0.5 , act_func: softmax , iniW_σ: 0.010000 >
###########################################
##### CNN initialized #####
##### start training #####
epoch XE(L) error(L) XE(V) error(V)
1 2.265512 0.854118 2.240207 0.858000
2 2.126916 0.793361 2.132252 0.796000
3 2.053773 0.753529 2.114341 0.784000
4 2.022574 0.746387 2.118557 0.794000
5 2.000311 0.733529 2.071346 0.770000
6 1.986872 0.728319 2.072314 0.742000
7 1.970179 0.720588 2.056508 0.746000
8 1.951982 0.710924 2.060582 0.748000
9 1.949321 0.706555 2.019281 0.742000
10 1.938045 0.701681 2.072499 0.758000
11 1.923634 0.699412 2.044654 0.740000
12 1.917019 0.685966 1.999032 0.730000
13 1.896135 0.687899 1.983176 0.716000
14 1.894803 0.684790 1.999759 0.692000
15 1.874767 0.674370 2.021464 0.692000
16 1.856811 0.662857 2.004935 0.698000
17 1.838303 0.660252 1.953219 0.678000
18 1.826441 0.655630 1.913830 0.660000
19 1.779350 0.636891 1.879912 0.644000
20 1.739723 0.620000 1.825149 0.642000
21 1.691372 0.597647 1.752824 0.608000
22 1.655938 0.578151 1.778146 0.578000
23 1.629195 0.557647 1.714115 0.596000
24 1.565406 0.541092 1.620170 0.546000
25 1.513010 0.523109 1.658018 0.574000
26 1.463774 0.507731 1.561847 0.530000
27 1.435841 0.494958 1.564757 0.518000
28 1.394540 0.477563 1.442391 0.500000
29 1.320723 0.446134 1.298339 0.450000
30 1.282767 0.432941 1.356098 0.452000
31 1.209008 0.406975 1.243206 0.434000
32 1.160657 0.392941 1.208980 0.416000
33 1.114366 0.371849 1.130212 0.396000
34 1.093672 0.365714 1.180310 0.392000
35 1.056027 0.353445 1.139584 0.372000
36 1.032040 0.343109 1.110961 0.380000
37 0.995310 0.331008 1.095206 0.380000
38 0.979810 0.327479 1.030094 0.356000
39 0.946138 0.312689 1.039761 0.360000
40 0.934624 0.312185 1.020273 0.352000
41 0.906518 0.304958 1.040692 0.366000
42 0.887821 0.297059 1.009176 0.344000
43 0.847129 0.282437 1.010904 0.346000
44 0.837678 0.279832 0.980055 0.344000
45 0.804155 0.269664 0.953995 0.350000
46 0.793181 0.269496 0.983602 0.334000
47 0.777183 0.261092 1.045581 0.340000
48 0.755691 0.255966 0.910118 0.322000
49 0.727655 0.246975 1.054030 0.338000
50 0.713998 0.238655 0.949052 0.318000
51 0.671913 0.225378 0.960111 0.322000
52 0.655613 0.222689 0.875177 0.310000
53 0.632419 0.215546 0.921186 0.304000
54 0.609891 0.210000 0.926388 0.308000
55 0.598780 0.203109 0.956141 0.314000
56 0.580358 0.197815 0.884541 0.284000
57 0.563777 0.191176 1.036186 0.334000
58 0.540403 0.185042 0.878522 0.290000
59 0.525809 0.179832 0.866313 0.276000
60 0.496978 0.173025 0.903838 0.302000
61 0.510715 0.174874 0.933896 0.322000
62 0.470734 0.158319 0.930657 0.282000
63 0.445534 0.150756 0.908095 0.290000
64 0.449468 0.154370 1.021374 0.302000
65 0.405614 0.139748 0.982117 0.298000
66 0.389739 0.131681 0.952315 0.286000
67 0.385806 0.130252 0.993064 0.298000
68 0.346114 0.117479 0.925194 0.276000
69 0.337600 0.113529 1.001964 0.294000
70 0.316199 0.107479 0.954678 0.278000
71 0.297671 0.103109 1.017478 0.264000
72 0.278122 0.094622 1.026215 0.278000
73 0.279483 0.094874 1.075593 0.284000
74 0.274797 0.093025 1.125169 0.290000
75 0.261910 0.089748 1.063620 0.278000
76 0.247519 0.083950 1.053286 0.288000
77 0.257481 0.088151 0.964341 0.282000
78 0.233071 0.077899 1.143678 0.294000
79 0.208931 0.074202 1.090725 0.286000
80 0.204916 0.068235 1.221492 0.308000
81 0.180988 0.062605 1.038368 0.256000
82 0.176942 0.060000 1.066491 0.292000
83 0.178173 0.058151 1.248633 0.286000
84 0.189114 0.063697 1.118649 0.270000
85 0.162007 0.055798 1.152669 0.270000
86 0.135113 0.044874 1.245378 0.282000
87 0.157199 0.054454 1.214303 0.282000
88 0.142269 0.048403 1.250788 0.278000
89 0.133804 0.045042 1.334189 0.294000
90 0.144161 0.046218 1.119339 0.262000
91 0.109862 0.036387 1.400242 0.294000
92 0.119807 0.040000 1.270848 0.272000
93 0.116280 0.037143 1.310338 0.288000
94 0.103779 0.035546 1.359119 0.286000
95 0.109552 0.036050 1.254955 0.288000
96 0.111649 0.037227 1.229749 0.270000
97 0.082756 0.027563 1.275888 0.284000
98 0.096411 0.033697 1.360698 0.274000
99 0.093900 0.031345 1.337077 0.266000
100 0.096269 0.032269 1.190403 0.258000
#elapsed time after 100 epoch : 5708.08045292 [sec]
real 95m17.891s
user 178m35.332s
sys 46m2.572s
}};
***Conv_stride(1, 1), Pool_size(8, 8)
#pre{{
ex201511302330
############# training condition ############
100 epoch training. input_size: (227, 227) , minibatch_size: 100
, learning_rate: 0.01 , momentum: 0.9 , weight_decay: 0.0005
#############################################
########## architecture of the CNN #########
layer1 - Convolution
< kernel_size: (100, 11, 11) , stride: (1, 1) , act_func: ReLU
, border_mode: valid , out_size: (100, 217, 217) , iniW_σ: 0.001000 >
layer2 - Pooling
< downscale: (8, 8) , stride: None , out_size: (100, 27, 27) >
layer3 - Convolution
< kernel_size: (200, 5, 5) , stride: (1, 1) , act_func: ReLU
, border_mode: valid , out_size: (200, 23, 23) , iniW_σ: 0.001000 >
layer4 - Pooling
< downscale: (2, 2) , stride: None , out_size: (200, 11, 11) >
layer5 - Convolution
< kernel_size: (200, 3, 3) , stride: (1, 1) , act_func: ReLU
, border_mode: full , out_size: (200, 13, 13) , iniW_σ: 0.001000 >
layer6 - Pooling
< downscale: (2, 2) , stride: None , out_size: (200, 6, 6) >
layer7 - fully-connected
< number_of_units: 4000 , drop_rate: 1.0 , act_func: ReLU , iniW_σ: 0.010000 >
layer8 - fully-connected
< number_of_units: 4000 , drop_rate: 0.5 , act_func: ReLU , iniW_σ: 0.010000 >
layer9 - fully-connected
< number_of_units: 10 , drop_rate: 0.5 , act_func: softmax , iniW_σ: 0.010000 >
###########################################
##### CNN initialized #####
##### start training #####
epoch XE(L) error(L) XE(V) error(V)
1 2.266060 0.853361 2.243392 0.856000
2 2.130685 0.793025 2.135200 0.794000
3 2.054085 0.756218 2.123839 0.782000
4 2.023824 0.745546 2.125052 0.778000
5 2.004558 0.736723 2.075419 0.764000
6 1.988269 0.730084 2.080712 0.736000
7 1.978521 0.722269 2.064026 0.752000
8 1.949791 0.708992 2.083354 0.728000
9 1.946183 0.713109 2.024760 0.726000
10 1.939402 0.704370 2.067147 0.736000
11 1.924256 0.696303 2.015610 0.718000
12 1.913130 0.683445 2.023050 0.692000
13 1.890571 0.675630 1.992919 0.702000
14 1.871528 0.659244 1.986287 0.704000
15 1.837502 0.646134 1.965102 0.676000
16 1.783898 0.617983 1.883287 0.664000
17 1.727252 0.598487 1.817291 0.628000
18 1.677167 0.577563 1.763205 0.594000
19 1.646680 0.566134 1.717152 0.584000
20 1.620932 0.560252 1.631011 0.538000
21 1.569441 0.537563 1.638784 0.556000
22 1.520713 0.521765 1.531932 0.524000
23 1.446847 0.489832 1.521347 0.518000
24 1.373653 0.470084 1.373839 0.488000
25 1.316823 0.444706 1.389143 0.510000
26 1.273436 0.432353 1.357315 0.474000
27 1.231128 0.417395 1.275293 0.454000
28 1.158633 0.392185 1.244637 0.434000
29 1.145679 0.385798 1.132520 0.388000
30 1.096611 0.365882 1.151662 0.404000
31 1.078286 0.362857 1.154174 0.404000
32 1.039630 0.345714 1.099213 0.380000
33 1.006328 0.336555 1.065627 0.356000
34 0.985907 0.326807 1.064509 0.356000
35 0.938378 0.313950 1.067813 0.372000
36 0.919013 0.305546 1.016313 0.354000
37 0.892731 0.293613 1.056768 0.374000
38 0.874155 0.292185 1.016974 0.368000
39 0.855303 0.284286 0.934007 0.320000
40 0.839621 0.281597 0.950497 0.314000
41 0.812977 0.271597 1.036738 0.330000
42 0.784090 0.265630 0.900286 0.312000
43 0.737794 0.247563 0.924353 0.316000
44 0.750065 0.249244 0.895992 0.314000
45 0.718061 0.235294 0.909370 0.306000
46 0.680311 0.226807 0.938665 0.318000
47 0.670178 0.225294 0.915215 0.318000
48 0.669341 0.222941 0.815276 0.266000
49 0.643219 0.217479 0.880080 0.296000
50 0.608783 0.207983 0.871083 0.286000
51 0.586518 0.198403 0.879849 0.294000
52 0.560401 0.187815 0.862862 0.268000
53 0.552764 0.186471 0.937973 0.306000
54 0.541401 0.185546 0.878444 0.292000
55 0.528155 0.176975 0.889366 0.282000
56 0.476757 0.162017 0.818434 0.268000
57 0.464492 0.157563 0.936802 0.308000
58 0.448118 0.156387 0.828980 0.278000
59 0.437451 0.147059 0.847776 0.276000
60 0.407118 0.137479 0.857101 0.262000
61 0.410758 0.139748 0.848346 0.266000
62 0.375467 0.131849 0.912574 0.262000
63 0.380334 0.128151 0.809710 0.250000
64 0.352329 0.122353 0.956197 0.288000
65 0.331058 0.113109 0.943340 0.274000
66 0.337890 0.114454 0.789285 0.246000
67 0.301068 0.102353 0.849514 0.256000
68 0.288528 0.100252 0.899972 0.260000
69 0.259481 0.090588 0.951661 0.262000
70 0.246767 0.083277 0.930885 0.254000
71 0.230417 0.076387 0.929545 0.282000
72 0.229210 0.080084 0.900590 0.244000
73 0.226441 0.077899 0.938132 0.238000
74 0.219149 0.075462 0.895079 0.242000
75 0.206667 0.072269 0.971582 0.254000
76 0.206785 0.071261 0.916770 0.238000
77 0.177221 0.060252 0.963789 0.256000
78 0.193635 0.065546 1.013842 0.264000
79 0.179224 0.060672 0.999443 0.238000
80 0.193169 0.064370 0.982249 0.262000
81 0.156145 0.051261 1.105881 0.268000
82 0.147516 0.048739 1.030534 0.248000
83 0.130676 0.042101 0.975202 0.242000
84 0.153309 0.052521 1.076137 0.274000
85 0.128124 0.042773 1.149028 0.266000
86 0.120232 0.039496 1.078922 0.242000
87 0.108206 0.034538 0.959661 0.224000
88 0.107621 0.036471 1.261161 0.270000
89 0.111350 0.037143 1.049523 0.256000
90 0.113830 0.037647 1.140495 0.244000
91 0.096187 0.030420 1.228679 0.268000
92 0.095758 0.031933 1.192075 0.254000
93 0.100844 0.033109 1.188920 0.254000
94 0.093317 0.030000 1.074962 0.270000
95 0.089934 0.029748 1.185309 0.250000
96 0.088179 0.031176 1.101567 0.238000
97 0.078000 0.025966 1.128674 0.238000
98 0.083343 0.026807 1.140479 0.218000
99 0.073937 0.023613 1.481587 0.292000
100 0.089651 0.028403 1.117366 0.238000
#elapsed time after 100 epoch : 17126.3356249 [sec]
real 285m35.820s
user 355m27.476s
sys 70m56.932s
}};
**ex20151127-1_重みの初期値を一様分布から正規分布へと変えて、初期値の標準偏差と学習係数の適切な組み合わせを探る実験 [#febbc23c]
-一様分布のときと同様に、FCの初期値の幅をConvより広くしたものが最も良い性能を示した。
-学習係数が0.001のデータをあまり貼ってないのでアレですが、初期値に正規分布を使う場合、学習係数は0.01の方が良いと思います。0.001では総じて遅すぎたので。
-omegaの横の数値が標準偏差の値です。sigmaと間違えました。
***eta0.01, conv_omega0.001, FC_omega0.01 [#d447b3c9]
ex201511262153
#pre{{
########## architecture of the CNN #########
100 epoch learning. minibatch_size: 100 , learning_rate: 0.01 , momentum: 0.9 , weight_decay: 0.0005
layer1 - Convolution
< kernel_size: (100, 11, 11) , stride: (4, 4) , act_func: ReLU
, border_mode: valid , out_size: (100, 55, 55) , iniW_σ: 0.001000 >
layer2 - Pooling
< downscale: (2, 2) , stride: None , out_size: (100, 27, 27) >
layer3 - Convolution
< kernel_size: (200, 5, 5) , stride: (1, 1) , act_func: ReLU
, border_mode: valid , out_size: (200, 23, 23) , iniW_σ: 0.001000 >
layer4 - Pooling
< downscale: (2, 2) , stride: None , out_size: (200, 11, 11) >
layer5 - Convolution
< kernel_size: (200, 3, 3) , stride: (1, 1) , act_func: ReLU
, border_mode: full , out_size: (200, 13, 13) , iniW_σ: 0.001000 >
layer6 - Pooling
< downscale: (2, 2) , stride: None , out_size: (200, 6, 6) >
layer7 - fully-connected
< number_of_units: 4000 , drop_rate: 1.0 , act_func: ReLU , iniW_σ: 0.010000 >
layer8 - fully-connected
< number_of_units: 4000 , drop_rate: 0.5 , act_func: ReLU , iniW_σ: 0.010000 >
layer9 - fully-connected
< number_of_units: 10 , drop_rate: 0.5 , act_func: softmax , iniW_σ: 0.010000 >
###########################################
##### start training #####
epoch XE(L) error(L) XE(V) error(V)
1 2.279788 0.857227 2.296859 0.878000
2 2.191314 0.806891 2.173209 0.806000
3 2.096164 0.762017 2.119903 0.768000
4 2.056766 0.748571 2.116595 0.758000
5 2.039419 0.747899 2.087323 0.764000
6 2.048450 0.750420 2.100580 0.760000
7 2.012680 0.735798 2.086680 0.750000
8 1.998883 0.730588 2.094667 0.746000
9 1.992026 0.723193 2.157129 0.776000
10 1.985581 0.719412 2.196125 0.788000
11 1.973130 0.716050 2.102198 0.744000
12 1.976247 0.717059 2.016111 0.710000
13 1.925662 0.700336 2.028982 0.724000
14 1.938949 0.698655 2.114204 0.752000
15 1.924359 0.685378 1.994008 0.710000
16 1.882777 0.662941 2.102367 0.722000
17 1.872506 0.660924 1.966560 0.682000
18 1.864624 0.655966 1.922495 0.682000
19 1.825770 0.641429 1.902522 0.630000
20 1.793480 0.625630 1.870321 0.648000
21 1.752025 0.605882 1.862212 0.628000
22 1.738010 0.605294 1.857119 0.622000
23 1.712135 0.596807 1.788575 0.646000
24 1.679222 0.581008 1.724772 0.558000
25 1.643857 0.568824 1.674149 0.568000
26 1.615264 0.552185 1.648617 0.566000
27 1.580249 0.544958 1.699005 0.576000
28 1.544599 0.532941 1.601888 0.564000
29 1.566499 0.540168 1.604061 0.548000
30 1.482200 0.507227 1.630673 0.562000
31 1.407958 0.485378 1.467805 0.508000
32 1.356966 0.458235 1.380396 0.478000
33 1.299496 0.437647 1.337013 0.464000
34 1.281185 0.430084 1.349695 0.466000
35 1.209316 0.416639 1.271618 0.434000
36 1.208390 0.408319 1.262016 0.436000
37 1.145750 0.388487 1.228800 0.416000
38 1.145020 0.390252 1.210642 0.408000
39 1.092922 0.369160 1.250346 0.402000
40 1.090970 0.368067 1.125966 0.382000
41 1.048666 0.350168 1.124405 0.404000
42 1.004277 0.336807 1.172188 0.398000
43 0.990163 0.328571 1.079470 0.370000
44 0.956650 0.319160 1.114563 0.396000
45 0.942413 0.317479 1.080161 0.374000
46 0.927220 0.308235 1.168196 0.380000
47 0.919637 0.313193 1.077926 0.366000
48 0.891175 0.298739 0.968754 0.340000
49 0.856628 0.286639 1.008114 0.360000
50 0.833166 0.282437 1.055485 0.332000
51 0.794779 0.271597 1.056139 0.344000
52 0.788357 0.263782 1.031454 0.326000
53 0.765426 0.260756 1.075679 0.372000
54 0.752347 0.253193 1.089792 0.364000
55 0.735859 0.250672 0.985893 0.324000
56 0.721941 0.245882 1.001033 0.316000
57 0.688821 0.231597 1.064463 0.338000
58 0.672875 0.229076 0.970349 0.326000
59 0.655550 0.224118 0.975476 0.316000
60 0.617892 0.210672 1.082753 0.356000
61 0.616934 0.207731 1.003524 0.326000
62 0.587871 0.199832 1.066631 0.328000
63 0.592913 0.204118 1.016835 0.308000
64 0.558418 0.187815 1.057545 0.320000
65 0.542399 0.185378 1.098326 0.344000
66 0.515363 0.173277 1.050404 0.302000
67 0.495595 0.171597 1.025466 0.326000
68 0.464327 0.161513 1.018175 0.282000
69 0.445623 0.153193 1.056368 0.296000
70 0.420819 0.142521 1.074576 0.322000
71 0.423316 0.147647 1.230379 0.338000
72 0.398392 0.136807 1.104836 0.306000
73 0.395690 0.138487 1.109917 0.312000
74 0.385336 0.128235 1.210536 0.326000
75 0.373550 0.124874 1.257324 0.304000
76 0.358137 0.122941 1.214032 0.308000
77 0.317682 0.106975 1.183054 0.322000
78 0.333262 0.112353 1.269349 0.328000
79 0.306124 0.105126 1.303980 0.308000
80 0.296196 0.100336 1.158767 0.308000
81 0.253620 0.084958 1.222014 0.302000
82 0.277029 0.093193 1.141394 0.290000
83 0.257218 0.086387 1.152766 0.278000
84 0.253831 0.083109 1.175288 0.292000
85 0.231094 0.077311 1.344497 0.310000
86 0.228450 0.075294 1.330323 0.318000
87 0.213584 0.073109 1.187189 0.290000
88 0.215680 0.074202 1.225960 0.296000
89 0.198841 0.063782 1.419377 0.308000
90 0.188258 0.065462 1.334841 0.302000
91 0.172070 0.058151 1.257102 0.300000
92 0.174285 0.056218 1.297317 0.302000
93 0.175419 0.057647 1.259745 0.278000
94 0.164890 0.055210 1.333592 0.308000
95 0.164969 0.054706 1.301733 0.290000
96 0.152825 0.052017 1.361302 0.314000
97 0.153835 0.050504 1.367234 0.292000
98 0.139659 0.047731 1.436525 0.276000
99 0.146334 0.048235 1.337804 0.294000
100 0.141419 0.047311 1.285882 0.294000
#elapsed time after 100 epoch : 5919.28769898 [sec]
real 98m49.206s
user 175m15.312s
sys 46m14.628s
}};
***eta0.01, conv_omega0.001, FC_omega0.001
ex201511262237
#pre{{
########## architecture of the CNN #########
100 epoch learning. minibatch_size: 100 , learning_rate: 0.01 , momentum: 0.9 , weight_decay: 0.0005
layer1 - Convolution
< kernel_size: (100, 11, 11) , stride: (4, 4) , act_func: ReLU
, border_mode: valid , out_size: (100, 55, 55) , iniW_σ: 0.001000 >
layer2 - Pooling
< downscale: (2, 2) , stride: None , out_size: (100, 27, 27) >
layer3 - Convolution
< kernel_size: (200, 5, 5) , stride: (1, 1) , act_func: ReLU
, border_mode: valid , out_size: (200, 23, 23) , iniW_σ: 0.001000 >
layer4 - Pooling
< downscale: (2, 2) , stride: None , out_size: (200, 11, 11) >
layer5 - Convolution
< kernel_size: (200, 3, 3) , stride: (1, 1) , act_func: ReLU
, border_mode: full , out_size: (200, 13, 13) , iniW_σ: 0.001000 >
layer6 - Pooling
< downscale: (2, 2) , stride: None , out_size: (200, 6, 6) >
layer7 - fully-connected
< number_of_units: 4000 , drop_rate: 1.0 , act_func: ReLU , iniW_σ: 0.001000 >
layer8 - fully-connected
< number_of_units: 4000 , drop_rate: 0.5 , act_func: ReLU , iniW_σ: 0.001000 >
layer9 - fully-connected
< number_of_units: 10 , drop_rate: 0.5 , act_func: softmax , iniW_σ: 0.001000 >
###########################################
##### start training #####
epoch XE(L) error(L) XE(V) error(V)
1 2.289214 0.859748 2.314248 0.900000
2 2.279820 0.858151 2.322131 0.900000
3 2.279018 0.858319 2.325436 0.900000
4 2.278875 0.858151 2.326430 0.900000
5 2.278644 0.857899 2.326484 0.900000
6 2.278996 0.858571 2.327302 0.900000
7 2.278895 0.858487 2.327085 0.900000
8 2.278733 0.858151 2.326541 0.900000
9 2.278884 0.858235 2.328256 0.900000
10 2.278741 0.858235 2.327457 0.900000
11 2.278825 0.858235 2.326281 0.900000
12 2.278796 0.858487 2.327071 0.900000
13 2.278692 0.858067 2.326404 0.900000
14 2.278856 0.858403 2.327175 0.900000
15 2.278512 0.858151 2.326523 0.900000
16 2.278399 0.858067 2.327618 0.900000
17 2.277651 0.858067 2.324584 0.900000
18 2.266181 0.843445 2.295807 0.882000
19 2.192752 0.818908 2.217242 0.816000
20 2.126434 0.778992 2.204913 0.828000
21 2.098422 0.767311 2.173321 0.800000
22 2.083043 0.765126 2.173632 0.828000
23 2.077946 0.762101 2.161616 0.806000
24 2.062897 0.757059 2.161109 0.798000
25 2.061684 0.759160 2.175754 0.806000
26 2.071941 0.767395 2.187909 0.818000
27 2.053810 0.757647 2.147933 0.810000
28 2.034118 0.754874 2.131397 0.810000
29 2.015168 0.743361 2.099014 0.790000
30 1.977714 0.723529 2.057542 0.748000
31 1.948424 0.709664 2.063088 0.768000
32 1.921753 0.691345 2.008414 0.722000
33 1.893688 0.683950 1.997358 0.720000
34 1.907050 0.689832 1.996617 0.720000
35 1.862669 0.664538 1.956418 0.704000
36 1.839722 0.657479 1.926295 0.702000
37 1.802570 0.646807 2.116516 0.754000
38 1.821628 0.648824 1.912855 0.700000
39 1.790424 0.639748 1.869369 0.666000
40 1.787948 0.637815 1.851022 0.656000
41 1.745422 0.620840 1.966666 0.696000
42 1.742164 0.614286 1.805366 0.630000
43 1.727556 0.604706 1.782348 0.630000
44 1.704735 0.602101 1.748261 0.610000
45 1.697856 0.596639 1.770579 0.624000
46 1.669272 0.584706 1.722074 0.596000
47 1.638318 0.572941 1.674674 0.580000
48 1.610545 0.563697 1.629384 0.550000
49 1.581405 0.551092 1.666137 0.580000
50 1.535939 0.528151 1.592743 0.546000
51 1.524203 0.525462 1.598540 0.562000
52 1.478741 0.503782 1.545349 0.550000
53 1.418556 0.483361 1.524162 0.530000
54 1.374773 0.466303 1.470476 0.506000
55 1.325039 0.448067 1.362406 0.492000
56 1.297996 0.437311 1.293635 0.466000
57 1.263279 0.430168 1.315085 0.472000
58 1.224640 0.410840 1.193438 0.448000
59 1.193657 0.403109 1.171922 0.406000
60 1.129074 0.380336 1.210986 0.430000
61 1.127256 0.376218 1.214752 0.404000
62 1.078499 0.361849 1.097572 0.380000
63 1.064977 0.362773 1.111543 0.374000
64 1.046567 0.351008 1.142148 0.394000
65 1.004249 0.338487 1.072640 0.384000
66 0.995249 0.334958 1.146379 0.380000
67 0.959661 0.320756 1.055862 0.370000
68 0.936250 0.310420 1.014813 0.364000
69 0.925829 0.312353 1.079364 0.380000
70 0.864163 0.289664 0.979812 0.342000
71 0.857257 0.288824 1.006246 0.332000
72 0.846522 0.286303 0.979174 0.326000
73 0.818960 0.272605 0.959189 0.344000
74 0.796602 0.267143 0.996843 0.352000
75 0.816673 0.274370 0.981101 0.348000
76 0.770414 0.263445 0.948454 0.342000
77 0.738688 0.247731 0.988118 0.352000
78 0.730586 0.247395 0.953460 0.338000
79 0.687154 0.230672 1.004113 0.330000
80 0.671581 0.227395 0.984177 0.338000
81 0.647149 0.217563 0.909386 0.318000
82 0.632714 0.213613 0.896398 0.310000
83 0.603839 0.204202 0.971699 0.332000
84 0.574152 0.193529 0.907302 0.300000
85 0.579654 0.197647 1.017110 0.342000
86 0.554308 0.188067 0.916641 0.316000
87 0.530155 0.180840 0.998144 0.302000
88 0.492030 0.166723 0.962299 0.306000
89 0.477169 0.162857 1.006667 0.298000
90 0.482011 0.168319 0.969493 0.304000
91 0.446418 0.148908 0.962634 0.314000
92 0.416054 0.145210 1.013056 0.318000
93 0.407301 0.139076 0.943681 0.294000
94 0.402713 0.136891 1.051337 0.326000
95 0.382483 0.131261 0.986986 0.292000
96 0.374495 0.127143 1.014350 0.312000
97 0.341564 0.115462 1.042052 0.308000
98 0.323176 0.109328 0.984928 0.294000
99 0.323966 0.111513 1.026672 0.284000
100 0.315672 0.107311 0.969291 0.290000
#elapsed time after 100 epoch : 5757.69596601 [sec]
real 96m7.670s
user 174m6.028s
sys 45m8.128s
}};
***eta0.01, conv_omega0.01, FC_omega0.01
#pre{{
############# training condition ############
100 epoch training. input_size: (227, 227) , minibatch_size: 100
, learning_rate: 0.01 , momentum: 0.9 , weight_decay: 0.0005
#############################################
########## architecture of the CNN #########
layer1 - Convolution
< kernel_size: (100, 11, 11) , stride: (4, 4) , act_func: ReLU
, border_mode: valid , out_size: (100, 55, 55) , iniW_σ: 0.010000 >
layer2 - Pooling
< downscale: (2, 2) , stride: None , out_size: (100, 27, 27) >
layer3 - Convolution
< kernel_size: (200, 5, 5) , stride: (1, 1) , act_func: ReLU
, border_mode: valid , out_size: (200, 23, 23) , iniW_σ: 0.010000 >
layer4 - Pooling
< downscale: (2, 2) , stride: None , out_size: (200, 11, 11) >
layer5 - Convolution
< kernel_size: (200, 3, 3) , stride: (1, 1) , act_func: ReLU
, border_mode: full , out_size: (200, 13, 13) , iniW_σ: 0.010000 >
layer6 - Pooling
< downscale: (2, 2) , stride: None , out_size: (200, 6, 6) >
layer7 - fully-connected
< number_of_units: 4000 , drop_rate: 1.0 , act_func: ReLU , iniW_σ: 0.010000 >
layer8 - fully-connected
< number_of_units: 4000 , drop_rate: 0.5 , act_func: ReLU , iniW_σ: 0.010000 >
layer9 - fully-connected
< number_of_units: 10 , drop_rate: 0.5 , act_func: softmax , iniW_σ: 0.010000 >
###########################################
##### CNN initialized #####
##### start training #####
epoch XE(L) error(L) XE(V) error(V)
1 57.355752 0.864034 2.314639 0.900000
2 2.282321 0.858151 2.322015 0.900000
3 2.279897 0.858319 2.325741 0.900000
4 2.279249 0.858151 2.325908 0.900000
5 2.281314 0.857899 2.326870 0.900000
6 2.279741 0.858571 2.327218 0.900000
7 2.279324 0.858487 2.327748 0.900000
8 2.281921 0.858151 2.326988 0.900000
9 2.279329 0.858235 2.328779 0.900000
10 2.279096 0.858235 2.327633 0.900000
11 2.278899 0.858235 2.326507 0.900000
12 2.279134 0.858487 2.327373 0.900000
13 2.278813 0.858067 2.326532 0.900000
14 2.279033 0.858403 2.327347 0.900000
15 2.279382 0.858151 2.327028 0.900000
16 2.278789 0.858067 2.327958 0.900000
17 2.278875 0.858067 2.327753 0.900000
18 2.278697 0.858151 2.326792 0.900000
19 2.279006 0.858067 2.326805 0.900000
20 2.279178 0.858235 2.327419 0.900000
21 2.279002 0.858319 2.327464 0.900000
22 2.279026 0.858151 2.327943 0.900000
23 2.278919 0.858235 2.327733 0.900000
24 2.278769 0.857983 2.327533 0.900000
25 2.278809 0.858655 2.326716 0.900000
26 2.279008 0.858319 2.326593 0.900000
27 2.278840 0.858235 2.326860 0.900000
28 2.278622 0.858151 2.327226 0.900000
29 2.278845 0.858319 2.327433 0.900000
30 2.278707 0.858235 2.327389 0.900000
}};
***eta0.001, conv_omega0.01, FC_omega0.01
#pre{{
##### CNN initialized #####
########## architecture of the CNN #########
100 epoch learning. minibatch_size: 100 , learning_rate: 0.001 , momentum: 0.9 , weight_decay: 0.0005
layer1 - Convolution
< kernel_size: (100, 11, 11) , stride: (4, 4) , act_func: ReLU
, border_mode: valid , out_size: (100, 55, 55) , iniW_σ: 0.001000 >
layer2 - Pooling
< downscale: (2, 2) , stride: None , out_size: (100, 27, 27) >
layer3 - Convolution
< kernel_size: (200, 5, 5) , stride: (1, 1) , act_func: ReLU
, border_mode: valid , out_size: (200, 23, 23) , iniW_σ: 0.001000 >
layer4 - Pooling
< downscale: (2, 2) , stride: None , out_size: (200, 11, 11) >
layer5 - Convolution
< kernel_size: (200, 3, 3) , stride: (1, 1) , act_func: ReLU
, border_mode: full , out_size: (200, 13, 13) , iniW_σ: 0.001000 >
layer6 - Pooling
< downscale: (2, 2) , stride: None , out_size: (200, 6, 6) >
layer7 - fully-connected
< number_of_units: 4000 , drop_rate: 1.0 , act_func: ReLU , iniW_σ: 0.010000 >
layer8 - fully-connected
< number_of_units: 4000 , drop_rate: 0.5 , act_func: ReLU , iniW_σ: 0.010000 >
layer9 - fully-connected
< number_of_units: 10 , drop_rate: 0.5 , act_func: softmax , iniW_σ: 0.010000 >
###########################################
##### start training #####
epoch XE(L) error(L) XE(V) error(V)
1 2.298945 0.868151 2.302684 0.900000
2 2.291475 0.858403 2.303700 0.900000
3 2.283327 0.858319 2.306935 0.900000
4 2.276402 0.858151 2.309872 0.900000
5 2.272492 0.857647 2.307571 0.900000
6 2.268735 0.851261 2.304408 0.900000
7 2.263944 0.848487 2.299230 0.896000
8 2.258452 0.844538 2.291861 0.874000
9 2.250948 0.829496 2.284950 0.878000
10 2.240829 0.830504 2.274124 0.858000
11 2.227953 0.823277 2.255608 0.846000
12 2.207249 0.816218 2.233939 0.840000
13 2.179750 0.797983 2.210093 0.828000
14 2.156048 0.779244 2.190328 0.818000
15 2.134629 0.764958 2.166895 0.800000
16 2.116268 0.755966 2.152267 0.794000
17 2.101822 0.749916 2.135769 0.784000
18 2.089538 0.746387 2.122359 0.772000
19 2.074834 0.738487 2.114890 0.762000
20 2.063822 0.737731 2.111917 0.774000
21 2.054489 0.734790 2.107136 0.764000
22 2.049469 0.734706 2.099675 0.764000
23 2.040591 0.727983 2.095448 0.754000
24 2.029844 0.726050 2.090445 0.748000
25 2.027214 0.724622 2.094199 0.744000
26 2.016414 0.721933 2.097714 0.742000
27 2.011720 0.720756 2.089883 0.738000
28 2.003814 0.720000 2.075352 0.742000
29 2.002691 0.721092 2.076192 0.726000
30 1.994191 0.711429 2.074222 0.734000
31 1.985296 0.711429 2.070637 0.726000
32 1.980544 0.706639 2.068676 0.730000
33 1.969502 0.709160 2.074861 0.718000
34 1.972719 0.709244 2.062387 0.714000
35 1.963712 0.704790 2.058929 0.696000
36 1.962358 0.706807 2.067888 0.700000
37 1.955976 0.701933 2.055356 0.702000
38 1.948130 0.695378 2.039469 0.706000
39 1.945331 0.695630 2.038332 0.694000
40 1.939442 0.695294 2.064415 0.696000
41 1.935854 0.691261 2.045462 0.704000
42 1.934636 0.688992 2.046764 0.706000
43 1.927097 0.687311 2.023183 0.696000
44 1.926392 0.690588 2.030900 0.704000
45 1.922850 0.683782 2.020085 0.708000
46 1.919929 0.684454 2.012714 0.702000
47 1.915890 0.681933 2.019143 0.700000
48 1.916717 0.685378 2.006689 0.696000
49 1.916212 0.684118 2.023428 0.704000
50 1.908615 0.675966 2.006994 0.690000
51 1.914212 0.684118 2.002732 0.684000
52 1.907916 0.671681 2.019715 0.702000
53 1.908784 0.682101 2.029294 0.688000
54 1.904535 0.677479 2.001421 0.692000
55 1.903722 0.676807 2.012378 0.704000
56 1.893367 0.670756 1.998448 0.704000
57 1.896418 0.671849 2.020645 0.682000
58 1.889160 0.662269 1.992309 0.686000
59 1.893595 0.674958 2.002228 0.680000
60 1.891602 0.661681 2.001342 0.692000
61 1.889934 0.668235 2.006420 0.688000
62 1.891581 0.664286 1.990300 0.684000
63 1.884815 0.661849 1.986786 0.684000
64 1.890615 0.662185 1.985312 0.682000
65 1.880546 0.657395 1.997947 0.678000
66 1.877505 0.658655 1.993619 0.698000
67 1.878208 0.660000 2.003519 0.668000
68 1.873462 0.649832 1.993286 0.682000
}};
**ex20151125_[[m/2015/okada/diary/2015-11-25]]のCPCPCPFFFと(たぶん)完全に同じ構成のCNN [#d5d09647]
-学習の推移もほとんど同じになった。
-重みの初期値はConvが-0.01~0.01、FCが-0.1~0.1の一様分布
#pre{{
########## architecture of the CNN #########
60 epoch learning. minibatch_size: 100 , learning_rate: 0.001 , momentum: 0.9 , weight_decay: 0.0005
layer1 - Convolution
< kernel_size: (100, 11, 11) , stride: (4, 4)
, act_func: ReLU , border_mode: valid , out_size: (100, 55, 55) >
layer2 - Pooling
< downscale: (2, 2) , stride: None , out_size: (100, 27, 27) >
layer3 - Convolution
< kernel_size: (200, 5, 5) , stride: (1, 1)
, act_func: ReLU , border_mode: valid , out_size: (200, 23, 23) >
layer4 - Pooling
< downscale: (2, 2) , stride: None , out_size: (200, 11, 11) >
layer5 - Convolution
< kernel_size: (200, 3, 3) , stride: (1, 1)
, act_func: ReLU , border_mode: full , out_size: (200, 13, 13) >
layer6 - Pooling
< downscale: (2, 2) , stride: None , out_size: (200, 6, 6) >
layer7 - fully-connected
< number_of_units: 4000 , drop_rate: 1.0 , act_func: ReLU >
layer8 - fully-connected
< number_of_units: 4000 , drop_rate: 0.5 , act_func: ReLU >
layer9 - fully-connected
< number_of_units: 10 , drop_rate: 0.5 , act_func: softmax >
###########################################
##### start training #####
epoch XE(L) error(L) XE(V) error(V)
1 2.273499 0.851933 2.254003 0.822000
2 2.122519 0.771597 2.168369 0.804000
3 2.032852 0.738992 2.093122 0.786000
4 1.989788 0.720756 2.036462 0.740000
5 1.935234 0.700084 2.021396 0.740000
6 1.911362 0.685798 1.946361 0.716000
7 1.835096 0.648067 1.834889 0.636000
8 1.768207 0.623361 1.879292 0.674000
9 1.749459 0.613445 1.786514 0.616000
10 1.685634 0.585042 1.727242 0.598000
11 1.635825 0.570252 1.654894 0.572000
12 1.619336 0.555462 1.641253 0.552000
13 1.580660 0.547395 1.638332 0.558000
14 1.576853 0.547731 1.541338 0.530000
15 1.524050 0.528655 1.511180 0.538000
16 1.517430 0.523529 1.490757 0.506000
17 1.491088 0.515630 1.536197 0.540000
18 1.444853 0.493445 1.479253 0.528000
19 1.442445 0.488908 1.462722 0.492000
20 1.404088 0.478739 1.484211 0.516000
21 1.377695 0.467311 1.379101 0.484000
22 1.375965 0.467227 1.358642 0.460000
23 1.352322 0.460084 1.282201 0.448000
24 1.321630 0.447395 1.352539 0.454000
25 1.303786 0.445294 1.343048 0.446000
26 1.265659 0.426975 1.277301 0.434000
27 1.254898 0.424034 1.256850 0.442000
28 1.245315 0.420000 1.253533 0.444000
29 1.240361 0.417143 1.178082 0.394000
30 1.199977 0.410672 1.193307 0.400000
31 1.210327 0.407227 1.180300 0.410000
32 1.179978 0.398655 1.209459 0.404000
33 1.156352 0.389328 1.133693 0.392000
34 1.147514 0.388824 1.258485 0.400000
35 1.105322 0.373529 1.074826 0.374000
36 1.118110 0.371429 1.151573 0.386000
37 1.095125 0.366723 1.098699 0.382000
38 1.086520 0.361513 1.043424 0.370000
39 1.067529 0.362269 1.062328 0.384000
40 1.041084 0.351681 1.051441 0.330000
41 1.051023 0.353613 1.056013 0.344000
42 1.023138 0.340756 1.056894 0.362000
43 1.018674 0.342185 1.055755 0.364000
44 0.999104 0.333445 0.984605 0.332000
45 0.976297 0.332185 0.960260 0.332000
46 0.995175 0.331176 0.994270 0.344000
47 0.955052 0.317395 0.932399 0.340000
48 0.954418 0.319496 0.940519 0.324000
49 0.927824 0.312605 0.972910 0.350000
50 0.941172 0.314706 1.058312 0.350000
51 0.921515 0.305042 0.935366 0.314000
52 0.907579 0.303697 0.931843 0.322000
53 0.896925 0.305378 1.008476 0.352000
54 0.884030 0.295294 0.934060 0.326000
55 0.900392 0.302353 0.954390 0.318000
56 0.879308 0.293277 0.997523 0.328000
57 0.886057 0.295630 0.897006 0.314000
58 0.861598 0.288571 0.957117 0.314000
59 0.861517 0.285882 0.871357 0.302000
60 0.853959 0.282185 0.889827 0.304000
}}
*ILSVRC2010 - 100クラス [#i7e9bcb2]
**ex20151207 【CPCPCCCPFFF (学習停滞時に学習係数ηを下げる-その2)】 [#ex20151207]
[[前回>#ex201512051]]よりもηを下げる基準をキツくして実験を行う。最初はバリデーションの最低誤識別率が4epoch連続で下がらなかった場合にηを0.1倍し、一回これを行う毎に次に0.1倍するまでの基準を8epoch、16epochと厳しくしていく。
-[[前回>#ex201512051]、[[前々回>#b3ff8545]]との比較…誤識別率 ( https://drive.google.com/open?id=0B9W18yqQO6JAN2xpcEFEV0JQUkk )、交差エントロピー( https://drive.google.com/open?id=0B9W18yqQO6JAblRjRHdEekR0Yk0 )
-2回目以降のη低下に悪あがき程度の効果しか見られない。実質的な学習は50epochもあれば済んでしまっている。2回目に下げるまでの期間をもっと長くすべきか…？
-[[前回>#ex201512051]]よりもtop1の誤識別率は僅かに良くなっているが、top5および交差エントロピーは微妙に悪くなっている。
-Alexの方法ではテストの交差エントロピーは(おそらく)一切上昇してなかったので、誤識別率よりそちらを見てηを調整していくべきなのかもしれない。
#pre{{
ex201512052123
############# training condition ############
100 epoch training. input_size: (231, 231) , minibatch_size: 128
, learning_rate: 0.01 , momentum: 0.9 , weight_decay: 0.0005
#############################################
########## architecture of the CNN #########
layer1 - Convolution
< kernel_size: (96, 11, 11) , stride: (4, 4) , act_func: ReLU
, border_mode: valid , out_size: (96, 56, 56) , iniW_σ: 0.010000 >
layer2 - Pooling
< downscale: (2, 2) , stride: None , out_size: (96, 28, 28) >
layer3 - Convolution
< kernel_size: (256, 5, 5) , stride: (1, 1) , act_func: ReLU
, border_mode: same , out_size: (256, 28, 28) , iniW_σ: 0.010000 >
layer4 - Pooling
< downscale: (2, 2) , stride: None , out_size: (256, 14, 14) >
layer5 - Convolution
< kernel_size: (384, 3, 3) , stride: (1, 1) , act_func: ReLU
, border_mode: same , out_size: (384, 14, 14) , iniW_σ: 0.010000 >
layer6 - Convolution
< kernel_size: (384, 3, 3) , stride: (1, 1) , act_func: ReLU
, border_mode: same , out_size: (384, 14, 14) , iniW_σ: 0.010000 >
layer7 - Convolution
< kernel_size: (256, 3, 3) , stride: (1, 1) , act_func: ReLU
, border_mode: valid , out_size: (256, 12, 12) , iniW_σ: 0.010000 >
layer8 - Pooling
< downscale: (2, 2) , stride: None , out_size: (256, 6, 6) >
layer9 - fully-connected
< number_of_units: 4096 , drop_rate: 1.0 , act_func: ReLU , iniW_σ: 0.010000 >
layer10 - fully-connected
< number_of_units: 4096 , drop_rate: 0.5 , act_func: ReLU , iniW_σ: 0.010000 >
layer11 - fully-connected
< number_of_units: 100 , drop_rate: 0.5 , act_func: softmax , iniW_σ: 0.010000 >
###########################################
##### CNN initialized #####
##### start training #####
epoch XE(L) error(L) XE(V) error(V) top-5(V) LE(ε)
1 4.584246 0.981340 4.621274 0.989984 0.949920 0.010000
2 4.571351 0.981150 4.632951 0.989984 0.949920 0.010000
3 4.562931 0.981000 4.548244 0.984776 0.919071 0.010000
4 4.264742 0.950954 4.155408 0.935296 0.788862 0.010000
5 3.900478 0.904689 3.675923 0.872196 0.644832 0.010000
6 3.383855 0.817349 3.087507 0.764022 0.471955 0.010000
7 2.952254 0.731412 2.862256 0.714744 0.410056 0.010000
8 2.646342 0.666535 2.549279 0.637821 0.348157 0.010000
9 2.414317 0.617243 2.350138 0.596755 0.304287 0.010000
10 2.219808 0.574021 2.268665 0.581130 0.294471 0.010000
11 2.059470 0.537692 2.050703 0.529647 0.251202 0.010000
12 1.923878 0.504493 1.899849 0.490585 0.223157 0.010000
13 1.796782 0.474954 1.819390 0.474760 0.209936 0.010000
14 1.682050 0.448197 1.810292 0.464343 0.202524 0.010000
15 1.574019 0.423499 1.691068 0.440104 0.188902 0.010000
16 1.481506 0.400719 1.684501 0.437099 0.187300 0.010000
17 1.387527 0.377583 1.671349 0.432091 0.182893 0.010000
18 1.308860 0.359153 1.587274 0.408454 0.169872 0.010000
19 1.223489 0.338909 1.600092 0.411659 0.169271 0.010000
20 1.149080 0.322341 1.624053 0.413662 0.170272 0.010000
21 1.072990 0.303348 1.575932 0.396034 0.164263 0.010000
22 1.013205 0.286804 1.534901 0.388421 0.162861 0.010000
23 0.947663 0.271758 1.573840 0.399439 0.164663 0.010000
24 0.885571 0.254318 1.561206 0.396434 0.157853 0.010000
25 0.819125 0.238416 1.544524 0.385817 0.155048 0.010000
26 0.773512 0.226515 1.555939 0.380609 0.153045 0.010000
27 0.725640 0.212047 1.605225 0.385216 0.160457 0.010000
28 0.682721 0.202650 1.545164 0.380208 0.155248 0.010000
29 0.642045 0.191604 1.585771 0.386819 0.158454 0.010000
30 0.604303 0.181233 1.596926 0.380208 0.157452 0.010000
31 0.581983 0.173301 1.571747 0.379207 0.157452 0.010000
32 0.540536 0.164490 1.631092 0.378205 0.160657 0.010000
33 0.506684 0.152843 1.625295 0.382812 0.153245 0.010000
34 0.486952 0.148889 1.641772 0.377604 0.155849 0.010000
35 0.463988 0.141473 1.660071 0.372596 0.156450 0.010000
36 0.440627 0.134675 1.625053 0.372997 0.155849 0.010000
37 0.413489 0.126371 1.665282 0.374199 0.155649 0.010000
38 0.396151 0.121197 1.629789 0.367588 0.164062 0.010000
39 0.380485 0.116934 1.687949 0.374800 0.155048 0.010000
40 0.364581 0.112632 1.722068 0.372796 0.156651 0.010000
41 0.351045 0.108258 1.689238 0.373397 0.154647 0.010000
42 0.335098 0.103567 1.807631 0.373197 0.160657 0.010000
43 0.215545 0.067333 1.716054 0.351162 0.146635 0.001000
44 0.182056 0.057580 1.730659 0.345753 0.145232 0.001000
45 0.167415 0.052350 1.746588 0.345553 0.143229 0.001000
46 0.154355 0.048618 1.770620 0.343550 0.144631 0.001000
47 0.146220 0.046122 1.805363 0.342348 0.142628 0.001000
48 0.134141 0.043452 1.812618 0.345954 0.144431 0.001000
49 0.132238 0.042596 1.829691 0.343149 0.143630 0.001000
50 0.130362 0.041820 1.841428 0.341146 0.144631 0.001000
51 0.126805 0.039966 1.841386 0.342348 0.144631 0.001000
52 0.119223 0.037929 1.865575 0.343349 0.142829 0.001000
53 0.116597 0.037193 1.839309 0.342147 0.143029 0.001000
54 0.111315 0.035687 1.870408 0.344952 0.142027 0.001000
55 0.109173 0.035125 1.878980 0.342949 0.142829 0.001000
56 0.107391 0.034657 1.918847 0.340946 0.142829 0.001000
57 0.105666 0.034221 1.890377 0.343750 0.141026 0.001000
58 0.104033 0.033714 1.873100 0.339744 0.140625 0.001000
59 0.097992 0.031725 1.902423 0.342348 0.140024 0.001000
60 0.095191 0.031179 1.910197 0.342348 0.140825 0.001000
61 0.091494 0.029713 1.922204 0.341747 0.142027 0.001000
62 0.093082 0.030481 1.922195 0.342949 0.142829 0.001000
63 0.091475 0.029594 1.941977 0.339944 0.139022 0.001000
64 0.085844 0.027867 1.947466 0.341747 0.141627 0.001000
65 0.087982 0.028318 1.964257 0.340545 0.140825 0.001000
66 0.084643 0.027201 1.943214 0.340345 0.139022 0.001000
67 0.080769 0.026599 1.950320 0.339944 0.139223 0.000100
68 0.079915 0.025561 1.952375 0.340345 0.140224 0.000100
69 0.079247 0.025822 1.953625 0.341346 0.140224 0.000100
70 0.079155 0.025513 1.954775 0.342147 0.139223 0.000100
71 0.077192 0.025125 1.957436 0.341346 0.140825 0.000100
72 0.074155 0.024222 1.960327 0.340345 0.139623 0.000100
73 0.075652 0.024412 1.963825 0.341346 0.140024 0.000100
74 0.077644 0.024586 1.953252 0.340144 0.139824 0.000100
75 0.076401 0.024523 1.961390 0.340745 0.139623 0.000100
76 0.075255 0.024396 1.962003 0.338742 0.139623 0.000100
77 0.074594 0.024555 1.962883 0.339343 0.139423 0.000100
78 0.074231 0.023865 1.969501 0.339944 0.140224 0.000100
79 0.076269 0.024927 1.968261 0.340545 0.140825 0.000100
80 0.076875 0.024832 1.965102 0.340545 0.140425 0.000100
81 0.075533 0.024341 1.966429 0.339944 0.140625 0.000100
82 0.073920 0.024000 1.965175 0.339143 0.141627 0.000100
83 0.073000 0.023374 1.973192 0.340545 0.141026 0.000100
84 0.077539 0.025244 1.969003 0.339744 0.141426 0.000100
85 0.073741 0.023794 1.973915 0.337941 0.141226 0.000100
86 0.072854 0.023279 1.970303 0.339744 0.140024 0.000100
87 0.074940 0.024238 1.979327 0.339343 0.140825 0.000100
88 0.070790 0.023113 1.975582 0.339143 0.141426 0.000100
89 0.073999 0.023834 1.977978 0.338542 0.141226 0.000100
90 0.070613 0.022978 1.976582 0.338742 0.140625 0.000100
91 0.075713 0.024674 1.973902 0.339343 0.139824 0.000100
92 0.070920 0.022629 1.978591 0.339944 0.140625 0.000100
93 0.073148 0.023976 1.986253 0.338942 0.141226 0.000100
94 0.069378 0.022812 1.986268 0.339744 0.141026 0.000100
95 0.071320 0.023057 1.985336 0.338341 0.141026 0.000100
96 0.069574 0.022558 1.989404 0.339744 0.140224 0.000100
97 0.073174 0.024016 1.981400 0.340144 0.141426 0.000100
98 0.072548 0.023723 1.983688 0.338742 0.140024 0.000100
99 0.070839 0.023025 1.984810 0.338942 0.141226 0.000100
100 0.071473 0.023057 1.991602 0.339143 0.139824 0.000100
#elapsed time after 100 epoch : 88787.538543 [sec]
real 1480m2.110s
user 2279m24.212s
sys 933m33.988s
}};
**ex20151205-1 【CPCPCCCPFFF (学習停滞時に学習係数ηを下げる)】[#ex201512051]
バリデーションの最低誤識別率がN epoch連続で更新されなかったとき、学習係数ηを0.1倍するようにして実験を行った。Nは3から開始し、ηを下げる毎に3,4,5と増えていく。ηを下げる回数は最大3回までに限定。
-[[η=0.01固定>#b3ff8545]]との比較( https://drive.google.com/open?id=0B9W18yqQO6JAMERycHlwWTBkaUE )。縦線はηを0.1倍したタイミング。
-初回のη減少は明らかに良い効果があった模様。2回目もわずかに効いてるっぽいが、それ以降は下げるタイミングが早すぎた感もあって効果は見られず。
#pre{{
ex201512041855
############# training condition ############
100 epoch training. input_size: (231, 231) , minibatch_size: 128
, learning_rate: 0.01 , momentum: 0.9 , weight_decay: 0.0005
#############################################
########## architecture of the CNN #########
layer1 - Convolution
< kernel_size: (96, 11, 11) , stride: (4, 4) , act_func: ReLU
, border_mode: valid , out_size: (96, 56, 56) , iniW_σ: 0.010000 >
layer2 - Pooling
< downscale: (2, 2) , stride: None , out_size: (96, 28, 28) >
layer3 - Convolution
< kernel_size: (256, 5, 5) , stride: (1, 1) , act_func: ReLU
, border_mode: same , out_size: (256, 28, 28) , iniW_σ: 0.010000 >
layer4 - Pooling
< downscale: (2, 2) , stride: None , out_size: (256, 14, 14) >
layer5 - Convolution
< kernel_size: (384, 3, 3) , stride: (1, 1) , act_func: ReLU
, border_mode: same , out_size: (384, 14, 14) , iniW_σ: 0.010000 >
layer6 - Convolution
< kernel_size: (384, 3, 3) , stride: (1, 1) , act_func: ReLU
, border_mode: same , out_size: (384, 14, 14) , iniW_σ: 0.010000 >
layer7 - Convolution
< kernel_size: (256, 3, 3) , stride: (1, 1) , act_func: ReLU
, border_mode: valid , out_size: (256, 12, 12) , iniW_σ: 0.010000 >
layer8 - Pooling
< downscale: (2, 2) , stride: None , out_size: (256, 6, 6) >
layer9 - fully-connected
< number_of_units: 4096 , drop_rate: 1.0 , act_func: ReLU , iniW_σ: 0.010000 >
layer10 - fully-connected
< number_of_units: 4096 , drop_rate: 0.5 , act_func: ReLU , iniW_σ: 0.010000 >
layer11 - fully-connected
< number_of_units: 100 , drop_rate: 0.5 , act_func: softmax , iniW_σ: 0.010000 >
###########################################
##### CNN initialized #####
##### start training #####
epoch XE(L) error(L) XE(V) error(V) top-5(V) LE(ε)
1 4.584245 0.981332 4.621276 0.989984 0.949920 0.010000
2 4.571351 0.981150 4.632952 0.989984 0.949920 0.010000
3 4.561820 0.980944 4.539330 0.984375 0.916266 0.010000
4 4.265857 0.950471 4.140177 0.933494 0.784255 0.010000
5 3.898917 0.904364 3.662564 0.871194 0.637620 0.010000
6 3.373583 0.814964 3.099520 0.756210 0.468349 0.010000
7 2.948426 0.730524 2.838661 0.703325 0.397436 0.010000
8 2.648571 0.667652 2.562404 0.647035 0.350962 0.010000
9 2.414502 0.616799 2.379750 0.603966 0.306090 0.010000
10 2.217295 0.573498 2.240499 0.574920 0.285657 0.010000
11 2.057729 0.537660 2.036490 0.527244 0.248798 0.010000
12 1.915242 0.501965 1.939623 0.492788 0.232171 0.010000
13 1.787862 0.472569 1.859350 0.485978 0.212740 0.010000
14 1.676458 0.445526 1.782328 0.462941 0.195913 0.010000
15 1.577050 0.422826 1.701397 0.444912 0.191707 0.010000
16 1.477317 0.399816 1.658040 0.434095 0.178285 0.010000
17 1.386620 0.378470 1.642021 0.432692 0.179688 0.010000
18 1.300428 0.357774 1.595300 0.412460 0.171074 0.010000
19 1.222153 0.338624 1.605401 0.413862 0.167869 0.010000
20 1.142201 0.317943 1.588153 0.409856 0.167468 0.010000
21 1.072498 0.302144 1.540821 0.396234 0.159455 0.010000
22 1.007504 0.285616 1.572307 0.397837 0.164263 0.010000
23 0.949187 0.270989 1.543926 0.389623 0.157252 0.010000
24 0.879827 0.253209 1.547989 0.394631 0.165064 0.010000
25 0.819547 0.239066 1.550430 0.383614 0.158854 0.010000
26 0.778321 0.228789 1.578135 0.386819 0.153846 0.010000
27 0.728134 0.213901 1.559589 0.373197 0.151042 0.010000
28 0.685571 0.203093 1.560373 0.377404 0.156851 0.010000
29 0.641609 0.191810 1.578929 0.378405 0.152043 0.010000
30 0.600765 0.179521 1.573359 0.383013 0.161058 0.010000
31 0.415692 0.126696 1.590206 0.356971 0.144030 0.001000
32 0.351503 0.107679 1.595305 0.357171 0.143830 0.001000
33 0.323913 0.099376 1.615260 0.351763 0.142428 0.001000
34 0.308306 0.095255 1.632192 0.347556 0.139022 0.001000
35 0.296006 0.092823 1.640612 0.348758 0.142829 0.001000
36 0.280420 0.087221 1.648830 0.347957 0.140625 0.001000
37 0.267182 0.083885 1.654571 0.351362 0.141226 0.001000
38 0.259883 0.081255 1.674374 0.347356 0.142027 0.001000
39 0.247045 0.077206 1.685620 0.347155 0.140425 0.001000
40 0.240777 0.076002 1.696517 0.348958 0.139824 0.001000
41 0.230222 0.073236 1.715194 0.345152 0.140825 0.001000
42 0.225075 0.071612 1.700519 0.345353 0.140024 0.001000
43 0.212415 0.067801 1.729592 0.344551 0.140425 0.001000
44 0.211383 0.068292 1.764561 0.343950 0.140425 0.001000
45 0.203443 0.063673 1.769579 0.343950 0.141026 0.001000
46 0.200575 0.064275 1.773398 0.342748 0.140024 0.001000
47 0.190406 0.061129 1.785921 0.342748 0.138221 0.001000
48 0.182409 0.058348 1.795420 0.344351 0.138622 0.001000
49 0.186650 0.059893 1.771866 0.344952 0.137821 0.001000
50 0.179944 0.057912 1.800079 0.340745 0.138622 0.001000
51 0.174807 0.056795 1.791111 0.342949 0.138221 0.001000
52 0.167797 0.054489 1.807459 0.344351 0.138622 0.001000
53 0.164205 0.052453 1.833803 0.342748 0.138822 0.001000
54 0.164386 0.053071 1.810848 0.346154 0.140825 0.001000
55 0.151607 0.049688 1.816693 0.340946 0.138021 0.000100
56 0.149181 0.048476 1.828310 0.340345 0.136819 0.000100
57 0.146591 0.046606 1.830688 0.343149 0.136418 0.000100
58 0.148974 0.047723 1.827465 0.342949 0.137420 0.000100
59 0.145522 0.047105 1.827852 0.340946 0.135617 0.000100
60 0.141144 0.045932 1.835013 0.341747 0.136418 0.000100
61 0.141460 0.046122 1.832027 0.341947 0.137821 0.000100
62 0.142015 0.046392 1.835050 0.342748 0.136619 0.000010
63 0.144502 0.046827 1.837035 0.342748 0.136218 0.000010
64 0.139197 0.044324 1.839591 0.343349 0.136418 0.000010
65 0.142461 0.046186 1.839009 0.342949 0.136619 0.000010
66 0.143740 0.046455 1.837929 0.342748 0.136218 0.000010
67 0.141635 0.046083 1.838704 0.342748 0.136619 0.000010
68 0.142662 0.045488 1.839136 0.342748 0.136218 0.000010
69 0.140663 0.045734 1.839786 0.341747 0.136418 0.000010
70 0.140814 0.045861 1.841710 0.342348 0.136418 0.000010
71 0.138502 0.045552 1.842531 0.342147 0.136619 0.000010
72 0.138867 0.044292 1.841794 0.342348 0.135817 0.000010
73 0.142738 0.046233 1.841789 0.342348 0.136018 0.000010
74 0.141308 0.045829 1.841042 0.341747 0.136619 0.000010
75 0.140935 0.045940 1.841952 0.341546 0.136418 0.000010
76 0.142914 0.046423 1.842410 0.341947 0.136018 0.000010
77 0.141811 0.045599 1.842060 0.342748 0.136418 0.000010
78 0.139229 0.044815 1.842689 0.342949 0.136819 0.000010
79 0.142700 0.046463 1.842244 0.341947 0.136819 0.000010
80 0.142348 0.045774 1.843348 0.341947 0.137019 0.000010
Terminated
real 1197m14.062s
user 1840m22.340s
sys 752m1.900s
}};
**ex20151204-1 【CPCPCCCPFFF(eta=0.01)】 [#b3ff8545]
-AlexNetに非常に近い構成。重複プーリングはなし。DataAugmentationで切り取るサイズが調整のため若干大きくなっている。
-得られた汎化性能は、同層数の[[m/2015/okada/diary/2015-11-29#c37b9fbd]]と同程度、Cが1つ少ない[[m/2015/okada/diary/2015-11-29#se03f1b4]]以下。
-学習係数が10倍であるおかげか、上記2つに比べて学習の進行は非常に早い。早さと汎化性能の兼ね合いで学習係数を段階的に下げていくのが適切だと思う。
-1epochの学習に14.5分ほどかかっていることが気になる。このまま1000クラスで行うことを考えると10日と少しかかる計算。
#pre{{
############# training condition ############
150 epoch training. input_size: (231, 231) , minibatch_size: 128
, learning_rate: 0.01 , momentum: 0.9 , weight_decay: 0.0005
#############################################
########## architecture of the CNN #########
layer1 - Convolution
< kernel_size: (96, 11, 11) , stride: (4, 4) , act_func: ReLU
, border_mode: valid , out_size: (96, 56, 56) , iniW_σ: 0.010000 >
layer2 - Pooling
< downscale: (2, 2) , stride: None , out_size: (96, 28, 28) >
layer3 - Convolution
< kernel_size: (256, 5, 5) , stride: (1, 1) , act_func: ReLU
, border_mode: same , out_size: (256, 28, 28) , iniW_σ: 0.010000 >
layer4 - Pooling
< downscale: (2, 2) , stride: None , out_size: (256, 14, 14) >
layer5 - Convolution
< kernel_size: (384, 3, 3) , stride: (1, 1) , act_func: ReLU
, border_mode: same , out_size: (384, 14, 14) , iniW_σ: 0.010000 >
layer6 - Convolution
< kernel_size: (384, 3, 3) , stride: (1, 1) , act_func: ReLU
, border_mode: same , out_size: (384, 14, 14) , iniW_σ: 0.010000 >
layer7 - Convolution
< kernel_size: (256, 3, 3) , stride: (1, 1) , act_func: ReLU
, border_mode: valid , out_size: (256, 12, 12) , iniW_σ: 0.010000 >
layer8 - Pooling
< downscale: (2, 2) , stride: None , out_size: (256, 6, 6) >
layer9 - fully-connected
< number_of_units: 4096 , drop_rate: 1.0 , act_func: ReLU , iniW_σ: 0.010000 >
layer10 - fully-connected
< number_of_units: 4096 , drop_rate: 0.5 , act_func: ReLU , iniW_σ: 0.010000 >
layer11 - fully-connected
< number_of_units: 100 , drop_rate: 0.5 , act_func: softmax , iniW_σ: 0.010000 >
###########################################
##### CNN initialized #####
##### start training #####
epoch XE(L) error(L) XE(V) error(V)
1 4.584245 0.981340 4.621274 0.989984
2 4.571352 0.981150 4.632952 0.989984
3 4.563417 0.981008 4.552187 0.984976
4 4.267677 0.951208 4.143998 0.939503
5 3.898794 0.904840 3.676194 0.875401
6 3.390031 0.819385 3.107175 0.774239
7 2.966760 0.735247 2.856739 0.705529
8 2.660312 0.671606 2.594077 0.648838
9 2.429676 0.621308 2.432343 0.616186
10 2.230778 0.576144 2.252805 0.574519
11 2.055965 0.536606 2.002277 0.515425
12 1.921568 0.504786 1.960560 0.503806
13 1.792159 0.474178 1.854432 0.490184
14 1.676039 0.444465 1.816310 0.463942
15 1.567097 0.420504 1.699283 0.441106
16 1.470421 0.397780 1.673308 0.434095
17 1.382050 0.378043 1.619874 0.415465
18 1.304656 0.358329 1.610165 0.420873
19 1.219439 0.336104 1.589374 0.407853
20 1.136920 0.318593 1.589943 0.402043
21 1.065958 0.300250 1.562457 0.393429
22 1.003505 0.284055 1.550190 0.398237
23 0.943692 0.270514 1.574947 0.396835
24 0.871596 0.251093 1.547367 0.391426
25 0.820359 0.238463 1.542040 0.384415
26 0.772067 0.225976 1.570934 0.379006
27 0.721574 0.212427 1.591359 0.389423
28 0.678220 0.201857 1.560925 0.384615
29 0.636746 0.190154 1.611045 0.384415
30 0.587623 0.176978 1.650638 0.382612
31 0.579415 0.175393 1.616405 0.381210
32 0.540307 0.163199 1.652461 0.379607
33 0.506116 0.153271 1.670217 0.384014
34 0.486215 0.147986 1.648316 0.380809
35 0.462831 0.139904 1.632836 0.375000
36 0.438179 0.133843 1.642773 0.378405
37 0.419966 0.129976 1.642937 0.381611
38 0.396922 0.121339 1.694674 0.372997
39 0.381463 0.117710 1.663578 0.371795
40 0.362087 0.111269 1.650016 0.373197
41 0.344702 0.106103 1.678225 0.367388
42 0.336862 0.104470 1.771848 0.374599
43 0.323751 0.099518 1.753931 0.374599
44 0.311679 0.095945 1.724282 0.375801
45 0.297819 0.092054 1.822172 0.384215
46 0.290038 0.089392 1.768434 0.375801
47 0.277375 0.086579 1.784120 0.369792
48 0.267387 0.082729 1.797198 0.367388
49 0.262976 0.080882 1.814224 0.358974
50 0.257811 0.080050 1.724297 0.362179
51 0.251058 0.077309 1.769808 0.370192
52 0.235611 0.072872 1.829191 0.363582
53 0.231322 0.072040 1.792821 0.365585
54 0.224361 0.069346 1.823692 0.362981
55 0.215018 0.066676 1.814038 0.360978
56 0.212135 0.065503 1.867576 0.363181
57 0.208964 0.065194 1.844547 0.370593
58 0.210177 0.064877 1.875207 0.364583
59 0.195149 0.060416 1.825801 0.363181
60 0.187702 0.059370 1.894739 0.364383
Terminated
real 874m31.305s
user 1387m44.580s
sys 468m21.692s
}};
*1000クラス [#ff5ad5c2]
**20151210start [#vdae7816]
#pre{{
Using gpu device 0: GeForce GTX TITAN X
##### using all categories #####
##### initializing ILSVRC2010_L dir = /IntelSSD750/data/ILSVRC2010
# reading the meta data from /IntelSSD750/data/ILSVRC2010/work/meta20150919.pickle
# ncat = 1000
# ndat = 1261406
##### initializing ILSVRC2010_V dir = /IntelSSD750/data/ILSVRC2010
# ndat = 50000
##### initializing ILSVRC2010_T dir = /IntelSSD750/data/ILSVRC2010
# ndat = 150000
/IntelSSD750/data/ILSVRC2010/train/n07711080/n07711080_32735.JPEG (226, 500, 3) 0 french fries, french-fried potatoes, fries, chips
/IntelSSD750/data/ILSVRC2010/val/ILSVRC2010_val_00000001.JPEG (333, 500, 3) 77 seashore, coast, seacoast, sea-coast
/IntelSSD750/data/ILSVRC2010/test/ILSVRC2010_test_00000001.JPEG (335, 500, 3) 541 thermometer
############# training condition ############
100 epoch training. input_size: (231, 231) , minibatch_size: 128
, learning_rate: 0.01 , momentum: 0.9 , weight_decay: 0.0005
#############################################
########## architecture of the CNN #########
layer1 - Convolution
< kernel_size: (96, 11, 11) , stride: (4, 4) , act_func: ReLU
, border_mode: valid , out_size: (96, 56, 56) , iniW_σ: 0.010000 >
layer2 - Pooling
< downscale: (2, 2) , stride: None , out_size: (96, 28, 28) >
layer3 - Convolution
< kernel_size: (256, 5, 5) , stride: (1, 1) , act_func: ReLU
, border_mode: same , out_size: (256, 28, 28) , iniW_σ: 0.010000 >
layer4 - Pooling
< downscale: (2, 2) , stride: None , out_size: (256, 14, 14) >
layer5 - Convolution
< kernel_size: (384, 3, 3) , stride: (1, 1) , act_func: ReLU
, border_mode: same , out_size: (384, 14, 14) , iniW_σ: 0.010000 >
layer6 - Convolution
< kernel_size: (384, 3, 3) , stride: (1, 1) , act_func: ReLU
, border_mode: same , out_size: (384, 14, 14) , iniW_σ: 0.010000 >
layer7 - Convolution
< kernel_size: (256, 3, 3) , stride: (1, 1) , act_func: ReLU
, border_mode: valid , out_size: (256, 12, 12) , iniW_σ: 0.010000 >
layer8 - Pooling
< downscale: (2, 2) , stride: None , out_size: (256, 6, 6) >
layer9 - fully-connected
< number_of_units: 4096 , drop_rate: 1.0 , act_func: ReLU , iniW_σ: 0.010000 >
layer10 - fully-connected
< number_of_units: 4096 , drop_rate: 0.5 , act_func: ReLU , iniW_σ: 0.010000 >
layer11 - fully-connected
< number_of_units: 1000 , drop_rate: 0.5 , act_func: softmax , iniW_σ: 0.010000 >
###########################################
##### CNN initialized #####
##### start training #####
epoch XE(L) error(L) XE(V) error(V) top-5(V) LE(ε)
1 6.884752 0.997537 6.812517 0.997636 0.988662 0.010000
2 6.159330 0.981330 5.382997 0.952224 0.846575 0.010000
3 4.820724 0.895189 4.237020 0.831370 0.624159 0.010000
4 4.077025 0.808215 3.691015 0.749700 0.512981 0.010000
5 3.647175 0.747388 3.302666 0.697175 0.448558 0.010000
6 3.375887 0.705914 3.151889 0.667468 0.416326 0.010000
7 3.191690 0.676516 3.022518 0.646494 0.391567 0.010000
8 3.063357 0.655295 2.879581 0.625060 0.368450 0.010000
9 2.961918 0.638250 2.768079 0.603145 0.346955 0.010000
10 2.889310 0.625960 2.752685 0.601522 0.344752 0.010000
11 2.830659 0.615426 2.708251 0.587881 0.331470 0.010000
12 2.783030 0.607403 2.669339 0.584996 0.330970 0.010000
13 2.743840 0.600204 2.623211 0.577123 0.321454 0.010000
14 2.711900 0.594776 2.593437 0.572736 0.315505 0.010000
15 2.683638 0.589933 2.612056 0.572015 0.321214 0.010000
16 2.659777 0.585942 2.540284 0.561018 0.308874 0.010000
17 2.640581 0.582593 2.523216 0.554046 0.301743 0.010000
18 2.624222 0.578919 2.563521 0.561078 0.311699 0.010000
19 2.608824 0.576539 2.534935 0.559796 0.308213 0.010000
20 2.594859 0.574089 2.503979 0.554748 0.304367 0.010000
21 2.580792 0.571413 2.487949 0.551262 0.299479 0.010000
22 2.573882 0.569963 2.512330 0.555709 0.302704 0.010000
23 2.560682 0.568329 2.496245 0.552885 0.300741 0.010000
24 2.555082 0.566847 2.489083 0.548918 0.297636 0.010000
25 2.546908 0.565279 2.442227 0.545733 0.293830 0.010000
26 2.539257 0.564075 2.460876 0.547516 0.297055 0.010000
27 2.531815 0.562679 2.475089 0.548858 0.296354 0.010000
28 2.524805 0.561128 2.441060 0.543530 0.290966 0.010000
29 2.519792 0.560358 2.455679 0.540365 0.292107 0.010000
30 2.514656 0.559700 2.432604 0.539583 0.290405 0.010000
31 2.511469 0.559645 2.443431 0.544271 0.294291 0.010000
32 2.504787 0.557393 2.439221 0.539503 0.292388 0.010000
33 2.499973 0.556842 2.421732 0.539183 0.289663 0.010000
34 2.498350 0.556177 2.449258 0.546214 0.293429 0.010000
35 2.493292 0.555558 2.387780 0.530429 0.282732 0.010000
36 2.488641 0.555068 2.428506 0.538381 0.286699 0.010000
37 2.488250 0.554925 2.476041 0.542508 0.293670 0.010000
38 2.483451 0.553555 2.432560 0.540825 0.290665 0.010000
39 2.481457 0.553781 2.433631 0.539323 0.286318 0.010000
40 2.477724 0.552969 2.464033 0.542568 0.294291 0.010000
41 2.477396 0.552845 2.418973 0.536138 0.285938 0.010000
42 2.473406 0.551768 2.433758 0.539864 0.289203 0.010000
43 2.470789 0.552038 2.406371 0.538101 0.285517 0.010000
44 2.468073 0.550811 2.401807 0.535677 0.287240 0.010000
45 2.463922 0.550596 2.415541 0.538401 0.287981 0.010000
46 2.462944 0.549769 2.404986 0.534115 0.283494 0.010000
47 2.463471 0.550736 2.398261 0.532712 0.284856 0.010000
48 2.461038 0.550083 2.380089 0.530529 0.281170 0.010000
49 2.456632 0.549224 2.458168 0.541506 0.294091 0.010000
50 2.456645 0.549156 2.425614 0.539663 0.290585 0.010000
51 1.962958 0.455524 1.978884 0.450581 0.221735 0.001000
52 1.862928 0.435947 1.946839 0.446374 0.215765 0.001000
53 1.823379 0.428164 1.943885 0.444631 0.216306 0.001000
54 1.796986 0.423433 1.926847 0.441506 0.213341 0.001000
55 1.775713 0.419142 1.927886 0.441627 0.213301 0.001000
56 1.760049 0.416159 1.915538 0.438962 0.212640 0.001000
57 1.744787 0.413618 1.908974 0.436939 0.210056 0.001000
58 1.730326 0.410906 1.916117 0.439423 0.211699 0.001000
59 1.718417 0.408489 1.911490 0.439183 0.211218 0.001000
60 1.705688 0.406548 1.903501 0.435236 0.208734 0.001000
61 1.695307 0.404181 1.907159 0.436899 0.210797 0.001000
62 1.684335 0.402614 1.895269 0.433754 0.208454 0.001000
63 1.674639 0.400454 1.887359 0.432953 0.206651 0.001000
64 1.664605 0.398936 1.888616 0.433974 0.207332 0.001000
65 1.654332 0.397447 1.904917 0.434275 0.209155 0.001000
66 1.645336 0.395801 1.914080 0.437139 0.210897 0.001000
67 1.637503 0.394795 1.875011 0.429988 0.206150 0.001000
68 1.627407 0.392400 1.887011 0.432051 0.206170 0.001000
69 1.618667 0.391106 1.898288 0.433994 0.209896 0.001000
70 1.610144 0.389192 1.894694 0.435737 0.207993 0.001000
71 1.602037 0.387736 1.880388 0.431691 0.205649 0.001000
72 1.593667 0.387269 1.885249 0.431530 0.208233 0.001000
73 1.585427 0.385001 1.883168 0.432171 0.205288 0.001000
74 1.578055 0.384353 1.892626 0.433193 0.208033 0.001000
75 1.570413 0.382946 1.883466 0.431130 0.207732 0.001000
76 1.563408 0.381818 1.882975 0.432853 0.207913 0.001000
77 1.556648 0.380900 1.887011 0.431170 0.206951 0.001000
78 1.548848 0.378898 1.877993 0.430108 0.205389 0.001000
79 1.542193 0.378167 1.890434 0.433073 0.206791 0.001000
80 1.536980 0.377049 1.904177 0.435697 0.209475 0.001000
81 1.529383 0.375725 1.889776 0.431891 0.206931 0.001000
82 1.523214 0.374542 1.890501 0.431811 0.208353 0.001000
83 1.515065 0.373358 1.892736 0.432712 0.206911 0.001000
84 1.509030 0.371804 1.899701 0.432372 0.209175 0.001000
85 1.502982 0.370819 1.892647 0.431330 0.206330 0.001000
86 1.498615 0.370613 1.875204 0.426943 0.205148 0.001000
87 1.492169 0.368866 1.896035 0.433133 0.208734 0.001000
88 1.487142 0.368000 1.889295 0.431711 0.208013 0.001000
89 1.482072 0.367115 1.880413 0.429587 0.206370 0.001000
90 1.477492 0.366487 1.904637 0.433133 0.210837 0.001000
91 1.470232 0.365266 1.889451 0.429908 0.206791 0.001000
92 1.466913 0.364462 1.883390 0.430088 0.207572 0.001000
93 1.462341 0.364112 1.885805 0.430168 0.206751 0.001000
94 1.456559 0.362902 1.924685 0.433934 0.211639 0.001000
95 1.451973 0.361927 1.891140 0.428626 0.207772 0.001000
96 1.445586 0.361249 1.890103 0.430929 0.207412 0.001000
97 1.443212 0.360517 1.893173 0.430489 0.207993 0.001000
98 1.439502 0.359880 1.906369 0.433534 0.210056 0.001000
99 1.434970 0.359085 1.884951 0.429407 0.208133 0.001000
100 1.429950 0.357831 1.893761 0.430108 0.208053 0.001000
#elapsed time after 50 epoch : 615972.886903 [sec]
WARNING (theano.gof.cmodule): Removing key file /home/yamada/.theano/compiledir_Linux-3.19--generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpkQlc59/key.pkl because the corresponding module is gone from the file system.
WARNING (theano.gof.cmodule): Removing key file /home/yamada/.theano/compiledir_Linux-3.19--generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpRzh86h/key.pkl because the corresponding module is gone from the file system.
WARNING (theano.gof.cmodule): A module that was loaded by this ModuleCache can no longer be read from file /home/yamada/.theano/compiledir_Linux-3.19--generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmppDLvKe/f90a7a397ede0965c66895e171db1485.so... this could lead to problems.
WARNING (theano.gof.cmodule): Removing key file /home/yamada/.theano/compiledir_Linux-3.19--generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpYfdkIF/key.pkl because the corresponding module is gone from the file system.
WARNING (theano.gof.cmodule): Removing key file /home/yamada/.theano/compiledir_Linux-3.19--generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpNEavvz/key.pkl because the corresponding module is gone from the file system.
WARNING (theano.gof.cmodule): Removing key file /home/yamada/.theano/compiledir_Linux-3.19--generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpTxp6Pk/key.pkl because the corresponding module is gone from the file system.
WARNING (theano.gof.cmodule): Removing key file /home/yamada/.theano/compiledir_Linux-3.19--generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmphXxEHX/key.pkl because the corresponding module is gone from the file system.
WARNING (theano.gof.cmodule): Removing key file /home/yamada/.theano/compiledir_Linux-3.19--generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpbed2uK/key.pkl because the corresponding module is gone from the file system.
WARNING (theano.gof.cmodule): Removing key file /home/yamada/.theano/compiledir_Linux-3.19--generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmp1EDqiq/key.pkl because the corresponding module is gone from the file system.
WARNING (theano.gof.cmodule): Removing key file /home/yamada/.theano/compiledir_Linux-3.19--generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpVPuKT3/key.pkl because the corresponding module is gone from the file system.
WARNING (theano.gof.cmodule): Removing key file /home/yamada/.theano/compiledir_Linux-3.19--generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpaOcUcz/key.pkl because the corresponding module is gone from the file system.
WARNING (theano.gof.cmodule): Removing key file /home/yamada/.theano/compiledir_Linux-3.19--generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmp71Q7GZ/key.pkl because the corresponding module is gone from the file system.
WARNING (theano.gof.cmodule): Removing key file /home/yamada/.theano/compiledir_Linux-3.19--generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmp_mxp3s/key.pkl because the corresponding module is gone from the file system.
WARNING (theano.gof.cmodule): Removing key file /home/yamada/.theano/compiledir_Linux-3.19--generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpK5d4Up/key.pkl because the corresponding module is gone from the file system.
real 10267m53.554s
user 11948m50.384s
sys 7092m16.120s
}};
}}
終了行:
このページに詳しい実験結果をまとめていくことにします。&br;
#contents
備考
-ex20151130から、ネットワークへ入力する画像から教師画像の平均画像を引く正規化処理を導入
*CIFAR10 [#c92a4213]
**ex20151130_入力画像から「訓練用データの」平均画像を引く正規化を行った際の学習への影響を見る実験[#r416d0c0]
-比較対象は、正規化の有無以外同条件である[[b/2015/山田/実験結果#w8778ff0]]の「C[0.01]-P-C[0.01]-P-F[0.01]-F[0.01]」。
-学習推移の比較( https://drive.google.com/open?id=0B9W18yqQO6JAclhsdk1maW9mblU )
-正規化なしとは誤差レベルの違いしか見られず。少なくとも悪影響は無さそうなので、先人に倣って今後はこれを導入して実験を行っていく。
-[[#ex20151201]]も参照。有効な結果を確認できた。
#pre{{
with meansubnormalize
############# training condition ############
200 epoch training. input_size: (28, 28) , minibatch_size: 100
, learning_rate: 0.01 , momentum: 0.9 , weight_decay: 0.0005
#############################################
########## architecture of the CNN #########
layer1 - Convolution
< kernel_size: (50, 2, 2) , stride: (1, 1) , act_func: ReLU
, border_mode: valid , out_size: (50, 27, 27) , iniW_σ: 0.010000 >
layer2 - Convolution
< kernel_size: (50, 2, 2) , stride: (1, 1) , act_func: ReLU
, border_mode: valid , out_size: (50, 26, 26) , iniW_σ: 0.010000 >
layer3 - Pooling
< downscale: (4, 4) , stride: None , out_size: (50, 6, 6) >
layer4 - fully-connected
< number_of_units: 1000 , drop_rate: 0.8 , act_func: ReLU , iniW_σ: 0.010000 >
layer5 - fully-connected
< number_of_units: 10 , drop_rate: 0.5 , act_func: softmax , iniW_σ: 0.010000 >
###########################################
epoch XE(L) error(L) XE(T) error(T)
1 2.301241 0.890720 2.292954 0.871600
2 2.168675 0.793220 2.040033 0.743200
3 2.024148 0.738300 1.988409 0.730100
4 1.983140 0.723320 1.930618 0.694700
5 1.943590 0.710660 1.917150 0.694300
6 1.915704 0.696460 1.898298 0.694300
7 1.910534 0.694820 1.880623 0.686200
8 1.886874 0.683620 1.851273 0.660600
9 1.855617 0.668620 1.812055 0.645800
10 1.814704 0.653100 1.737229 0.618800
11 1.757508 0.627840 1.655818 0.587100
12 1.691635 0.604380 1.590292 0.563700
13 1.621381 0.579320 1.519763 0.539800
14 1.559681 0.557720 1.484703 0.532600
15 1.511344 0.542200 1.427634 0.510600
16 1.490780 0.532740 1.389791 0.492400
17 1.421070 0.507820 1.333367 0.470100
18 1.416972 0.507580 1.379368 0.490700
19 1.361172 0.484840 1.304148 0.462600
20 1.387499 0.496060 1.308432 0.461300
21 1.328294 0.474740 1.262993 0.450000
22 1.330144 0.473340 1.223739 0.428200
23 1.307318 0.465260 1.235810 0.444800
24 1.256990 0.446860 1.219263 0.424700
25 1.253362 0.444860 1.181059 0.412200
26 1.226874 0.435160 1.179139 0.414000
27 1.228463 0.437640 1.153660 0.402300
28 1.202043 0.426640 1.124669 0.391800
29 1.198916 0.424820 1.102292 0.391800
30 1.130790 0.399460 1.040838 0.362700
31 1.091429 0.385860 1.014462 0.350100
32 1.081484 0.381800 1.005059 0.348500
33 1.051955 0.369900 0.999547 0.350100
34 1.039517 0.364420 0.950747 0.325100
35 1.052473 0.366600 0.954961 0.329400
36 1.014120 0.355900 0.919145 0.318900
37 0.953343 0.334160 0.910735 0.316300
38 0.989915 0.347160 0.918302 0.317700
39 0.939300 0.329020 0.863239 0.295000
40 0.948976 0.331720 0.856169 0.293500
41 0.927161 0.323480 0.854197 0.297600
42 0.926378 0.323800 0.841668 0.290400
43 0.880868 0.307880 0.840756 0.290400
44 0.832429 0.292900 0.811874 0.279500
45 0.900572 0.314300 0.818376 0.285300
46 0.845470 0.296120 0.799914 0.274000
47 0.885975 0.311020 0.794399 0.271900
48 0.817159 0.286700 0.793099 0.272200
49 0.822162 0.287260 0.767546 0.265200
50 0.846227 0.295780 0.749083 0.254400
51 0.781471 0.270340 0.749934 0.260200
52 0.768634 0.267900 0.734875 0.250900
53 0.767662 0.270420 0.736494 0.252000
54 0.784999 0.273580 0.731604 0.248100
55 0.784041 0.272800 0.742787 0.253100
56 0.741014 0.256840 0.722091 0.246900
57 0.722639 0.252700 0.714930 0.245900
58 0.779533 0.270560 0.720464 0.250400
59 0.716955 0.249340 0.704772 0.240200
60 0.743085 0.258100 0.714782 0.240600
61 0.724373 0.252660 0.709176 0.239400
62 0.694776 0.242980 0.690473 0.236400
63 0.659292 0.230280 0.692154 0.236800
64 0.707746 0.247520 0.690970 0.233300
65 0.726773 0.254160 0.677306 0.232100
66 0.670460 0.235280 0.687865 0.235300
67 0.774429 0.270280 0.709229 0.243500
68 0.630434 0.220900 0.687857 0.234900
69 0.804328 0.279860 0.695814 0.236000
70 0.621518 0.218140 0.705632 0.240000
71 0.591258 0.207580 0.677732 0.231600
72 0.711016 0.249800 0.697816 0.239600
73 0.664425 0.232780 0.681117 0.230300
74 0.566506 0.198280 0.672950 0.230500
75 0.676259 0.236580 0.672809 0.228100
76 0.702371 0.245780 0.665470 0.227700
77 0.560746 0.196580 0.670936 0.229600
78 0.628251 0.219180 0.652131 0.223700
79 0.538038 0.188640 0.658130 0.222300
80 0.571354 0.201200 0.656089 0.222700
81 0.640938 0.224640 0.668623 0.232200
82 0.578572 0.202680 0.673226 0.228000
83 0.680907 0.239160 0.658503 0.226400
84 0.606439 0.211320 0.659322 0.225100
85 0.571947 0.200340 0.642748 0.222300
86 0.575176 0.202420 0.654706 0.222200
87 0.576259 0.201920 0.640902 0.215800
88 0.555670 0.196180 0.638271 0.219000
89 0.729043 0.254560 0.647124 0.218200
90 0.668291 0.234420 0.656433 0.223500
91 0.522506 0.185740 0.646257 0.219200
92 0.665435 0.229840 0.650611 0.220400
93 0.653686 0.227200 0.643800 0.219100
94 0.639696 0.222820 0.631171 0.216400
95 0.636667 0.223300 0.627640 0.213700
96 0.564452 0.199560 0.619440 0.211000
97 0.596317 0.208160 0.656351 0.223800
98 0.559324 0.196600 0.644581 0.217300
99 0.579977 0.203060 0.625384 0.211800
100 0.552774 0.195240 0.623098 0.211900
101 0.471780 0.167000 0.621501 0.212100
102 0.458696 0.163740 0.624754 0.210700
103 0.643440 0.225180 0.626696 0.212900
104 0.559983 0.197260 0.627611 0.215200
105 0.526930 0.185320 0.631127 0.214200
106 0.565367 0.200220 0.621152 0.212800
107 0.528703 0.184320 0.612028 0.209400
108 0.528290 0.187840 0.618490 0.215200
109 0.508079 0.179640 0.616074 0.211700
110 0.470439 0.166120 0.623196 0.209700
111 0.461730 0.162100 0.622962 0.212600
112 0.646683 0.226200 0.632754 0.218700
113 0.590706 0.206960 0.623041 0.213900
114 0.586474 0.205080 0.634192 0.212600
115 0.559189 0.197520 0.636560 0.215400
116 0.446040 0.156540 0.619566 0.210000
117 0.592400 0.209120 0.629333 0.214900
118 0.513011 0.179940 0.644279 0.218200
119 0.513977 0.180880 0.603127 0.204400
120 0.449013 0.160020 0.605747 0.208700
121 0.571244 0.201660 0.630140 0.215100
122 0.511436 0.179580 0.620652 0.209400
123 0.626654 0.221100 0.642481 0.217600
124 0.450823 0.159140 0.602686 0.202000
125 0.494492 0.175680 0.609670 0.206300
126 0.512142 0.181080 0.603773 0.202400
127 0.420901 0.146580 0.631388 0.211400
128 0.533896 0.188200 0.615401 0.211500
129 0.487292 0.172520 0.611321 0.204900
130 0.599748 0.210160 0.626388 0.211600
131 0.419864 0.148900 0.609263 0.206000
132 0.545272 0.192700 0.622831 0.212300
133 0.478314 0.170580 0.615337 0.210600
134 0.503807 0.178920 0.609677 0.203400
135 0.472619 0.167720 0.594770 0.199500
136 0.370442 0.132380 0.613169 0.203400
137 0.435198 0.154660 0.611876 0.206900
138 0.503113 0.179100 0.613414 0.203600
139 0.496871 0.175760 0.609763 0.205500
140 0.433943 0.154180 0.605862 0.205300
141 0.411215 0.145440 0.611845 0.202300
142 0.590320 0.209060 0.620617 0.209400
143 0.413181 0.145480 0.611651 0.204000
144 0.447773 0.159780 0.607428 0.202000
145 0.424696 0.150420 0.599058 0.202000
146 0.494046 0.175420 0.603054 0.204000
147 0.470818 0.167180 0.610959 0.202500
148 0.361096 0.128560 0.614718 0.198300
149 0.558248 0.197700 0.617627 0.209700
150 0.395579 0.140160 0.614893 0.201500
151 0.366426 0.130840 0.613265 0.201500
152 0.422831 0.149820 0.613052 0.204900
153 0.491266 0.172360 0.612104 0.203500
154 0.489052 0.172480 0.603320 0.204000
155 0.352787 0.125720 0.610067 0.200400
156 0.390451 0.138740 0.604489 0.199900
157 0.381451 0.135400 0.605064 0.200000
158 0.449463 0.160180 0.613838 0.203000
159 0.452143 0.160840 0.607294 0.204500
160 0.439858 0.157000 0.605615 0.204200
161 0.428808 0.152200 0.622550 0.205300
162 0.400631 0.142540 0.628669 0.207700
163 0.356724 0.126400 0.631125 0.209200
164 0.334470 0.119520 0.638453 0.204500
165 0.531309 0.188160 0.636315 0.216200
166 0.463861 0.164600 0.614799 0.205200
167 0.298837 0.105720 0.617777 0.200500
168 0.283677 0.100900 0.641540 0.201200
169 0.461111 0.162160 0.611954 0.200300
170 0.417671 0.149380 0.605191 0.201000
171 0.364417 0.128500 0.622437 0.204400
172 0.349503 0.125100 0.617939 0.201100
173 0.332663 0.119720 0.625480 0.198300
174 0.637518 0.221040 0.615185 0.205800
175 0.468763 0.164660 0.603146 0.200500
176 0.367030 0.130540 0.610282 0.197400
real 35m4.599s
user 47m11.476s
sys 55m53.392s
}};
**ex20151127-2_層ごとの重み初期値の標準偏差の組み合わせを色々変えて性能を比べる実験 [#w8778ff0]
-教師の推移比較( https://drive.google.com/open?id=0B9W18yqQO6JAREFpQV93ZG5LLVU )
-テストの推移比較( https://drive.google.com/open?id=0B9W18yqQO6JAQXAtVWpoaDdZN3c )
-ILSVRCのようにFCの標準偏差を大きくした方が早くなったりするかと思ったが、そうでもないらしい。パラメータ数やクラス数との兼ね合いの方が大事だということなのかも。Convの標準偏差を大きくしたものとFCのそれを大きくしたものでも結果に大差はないので、この値をそこまで慎重に決める必要はなさそう。
***C[0.001]-P-C[0.001]-P-F[0.001]-F[0.001] [#zb2ae618]
#pre{{
############# training condition ############
500 epoch training. input_size: (28, 28) , minibatch_size: 100
, learning_rate: 0.01 , momentum: 0.9 , weight_decay: 0.0005
#############################################
########## architecture of the CNN #########
layer1 - Convolution
< kernel_size: (50, 2, 2) , stride: (1, 1) , act_func: ReLU
, border_mode: valid , out_size: (50, 27, 27) , iniW_σ: 0.001000 >
layer2 - Convolution
< kernel_size: (50, 2, 2) , stride: (1, 1) , act_func: ReLU
, border_mode: valid , out_size: (50, 26, 26) , iniW_σ: 0.001000 >
layer3 - Pooling
< downscale: (4, 4) , stride: None , out_size: (50, 6, 6) >
layer4 - fully-connected
< number_of_units: 1000 , drop_rate: 0.8 , act_func: ReLU , iniW_σ: 0.001000 >
layer5 - fully-connected
< number_of_units: 10 , drop_rate: 0.5 , act_func: softmax , iniW_σ: 0.001000 >
###########################################
epoch XE(L) error(L) XE(T) error(T)
1 2.302816 0.902240 2.302728 0.900000
2 2.302879 0.901100 2.302805 0.900000
3 2.302906 0.902040 2.302703 0.900000
4 2.302792 0.899660 2.302803 0.900000
5 2.302885 0.901500 2.302837 0.900000
6 2.302877 0.901500 2.302654 0.900000
7 2.302864 0.899980 2.302733 0.900000
8 2.302880 0.901100 2.302828 0.900000
9 2.302816 0.899180 2.302854 0.900000
10 2.302942 0.900980 2.302740 0.900000
11 2.302877 0.900660 2.302797 0.900000
12 2.302951 0.900980 2.302666 0.900000
13 2.302896 0.901380 2.302770 0.900000
14 2.302930 0.903420 2.302701 0.900000
15 2.302886 0.901700 2.302700 0.900000
16 2.302876 0.902320 2.302740 0.900000
17 2.302918 0.900680 2.302705 0.900000
18 2.302859 0.900840 2.302680 0.900000
19 2.302854 0.899220 2.302746 0.900000
20 2.302887 0.902240 2.302685 0.900000
21 2.302853 0.901640 2.302783 0.900000
22 2.302877 0.900240 2.302672 0.900000
23 2.302900 0.903340 2.302670 0.900000
24 2.302900 0.900340 2.302646 0.900000
25 2.302869 0.901900 2.302734 0.900000
26 2.302863 0.899500 2.302859 0.900000
27 2.302896 0.901400 2.302732 0.900000
28 2.302926 0.903920 2.302740 0.900000
29 2.302812 0.898960 2.302872 0.900000
30 2.302941 0.901460 2.302744 0.900000
31 2.302809 0.901120 2.302862 0.900000
32 2.302873 0.903100 2.302745 0.900000
33 2.302815 0.900080 2.302847 0.900000
34 2.302873 0.900560 2.302701 0.900000
35 2.302857 0.901700 2.302606 0.900000
36 2.302829 0.901980 2.302828 0.900000
37 2.302889 0.900820 2.302719 0.900000
38 2.302913 0.901760 2.302720 0.900000
39 2.302820 0.900860 2.302830 0.900000
40 2.302861 0.902580 2.302886 0.900000
41 2.302895 0.901320 2.302770 0.900000
42 2.302911 0.902060 2.302693 0.900000
43 2.302888 0.900960 2.302736 0.900000
44 2.302922 0.900560 2.302662 0.900000
45 2.302896 0.899620 2.302675 0.900000
46 2.302884 0.900000 2.302700 0.900000
47 2.302867 0.902160 2.302755 0.900000
48 2.302843 0.900780 2.302807 0.900000
49 2.302875 0.900600 2.302703 0.900000
50 2.302860 0.902980 2.302767 0.900000
51 2.302904 0.902740 2.302663 0.900000
52 2.302875 0.901020 2.302791 0.900000
53 2.302802 0.900060 2.302787 0.900000
54 2.302905 0.899540 2.302714 0.900000
55 2.302918 0.901180 2.302813 0.900000
56 2.302894 0.903820 2.302661 0.900000
57 2.302897 0.902660 2.302698 0.900000
58 2.302806 0.900980 2.302812 0.900000
59 2.302901 0.902020 2.302679 0.900000
60 2.302860 0.902800 2.302660 0.900000
61 2.302864 0.901540 2.302682 0.900000
62 2.302851 0.901940 2.302773 0.900000
63 2.302877 0.900800 2.302760 0.900000
64 2.302901 0.900280 2.302729 0.900000
65 2.302923 0.901180 2.302698 0.900000
66 2.302878 0.900340 2.302655 0.900000
67 2.302845 0.901600 2.302739 0.900000
68 2.302807 0.900180 2.302745 0.900000
69 2.302879 0.899780 2.302640 0.900000
70 2.302851 0.900020 2.302733 0.900000
71 2.302906 0.902960 2.302744 0.900000
72 2.302900 0.902320 2.302689 0.900000
73 2.302818 0.900200 2.302743 0.900000
74 2.302934 0.902860 2.302675 0.900000
75 2.302927 0.902560 2.302680 0.900000
76 2.302844 0.900960 2.302713 0.900000
77 2.302856 0.901220 2.302770 0.900000
78 2.302773 0.900220 2.302903 0.900000
79 2.302886 0.901360 2.302787 0.900000
80 2.302900 0.899900 2.302687 0.900000
81 2.302858 0.901160 2.302636 0.900000
82 2.302866 0.899960 2.302754 0.900000
83 2.302939 0.901420 2.302622 0.900000
84 2.302832 0.900940 2.302782 0.900000
85 2.302873 0.902020 2.302694 0.900000
86 2.302882 0.900640 2.302688 0.900000
87 2.302813 0.900920 2.302709 0.900000
88 2.302846 0.900920 2.302531 0.900000
89 2.302660 0.900640 2.302236 0.900000
90 2.299378 0.890580 2.284871 0.878900
91 2.168390 0.819720 2.091465 0.776200
92 2.059146 0.761820 2.008044 0.739500
93 2.017725 0.740320 1.989615 0.730800
94 1.997479 0.730460 1.997901 0.730700
95 1.968781 0.717040 1.956786 0.722600
96 1.945655 0.710800 1.912005 0.692400
97 1.931816 0.703500 1.900630 0.701600
98 1.921936 0.699940 1.869197 0.674700
99 1.901791 0.686000 1.873437 0.676900
100 1.885371 0.680580 1.862281 0.672200
101 1.881802 0.682380 1.870674 0.670800
102 1.875387 0.677840 1.838516 0.664500
103 1.861597 0.672400 1.851301 0.662300
104 1.880659 0.685500 1.808609 0.652700
105 1.863076 0.675220 1.824427 0.652400
106 1.847045 0.668020 1.815635 0.649700
107 1.839112 0.662980 1.795213 0.637600
108 1.840379 0.663240 1.784321 0.636000
109 1.825963 0.656440 1.749073 0.617000
110 1.766159 0.632060 1.668041 0.596600
111 1.686609 0.603240 1.595677 0.571300
112 1.616457 0.581920 1.574964 0.565200
113 1.573660 0.563640 1.508577 0.541600
114 1.500858 0.538540 1.428491 0.514200
115 1.434960 0.513880 1.368229 0.481800
116 1.422711 0.507820 1.333455 0.463200
117 1.383827 0.494820 1.329217 0.467700
118 1.377169 0.491680 1.310349 0.461300
119 1.336277 0.476600 1.269550 0.444800
120 1.308907 0.466440 1.221570 0.428600
121 1.259891 0.447300 1.186137 0.417600
122 1.236259 0.439420 1.163775 0.403500
123 1.219182 0.431340 1.171127 0.404900
124 1.217761 0.433320 1.113039 0.385100
125 1.201874 0.424660 1.119029 0.391700
126 1.154899 0.407680 1.121279 0.394500
127 1.142473 0.402260 1.079211 0.374900
128 1.109837 0.389520 1.046161 0.361900
129 1.113610 0.389820 1.012330 0.354200
130 1.081986 0.380940 1.000698 0.349100
131 1.002081 0.350760 0.957500 0.333600
132 0.992582 0.349380 0.945118 0.323600
133 0.967429 0.338860 0.946332 0.329500
134 0.968382 0.341340 0.909104 0.314600
135 0.926163 0.322280 0.909532 0.314900
136 1.016451 0.355060 0.923451 0.320100
137 0.881288 0.310020 0.876305 0.306400
138 0.905802 0.318080 0.863423 0.297400
139 0.895989 0.313400 0.846558 0.292700
140 0.841693 0.294200 0.849797 0.295500
141 0.955008 0.334160 0.831881 0.285500
142 0.836120 0.294480 0.821438 0.286000
143 0.790014 0.276680 0.836626 0.287200
144 0.922634 0.322660 0.803645 0.277300
145 0.808569 0.283180 0.797457 0.273000
146 0.867673 0.300880 0.795595 0.272000
147 0.851220 0.297840 0.803278 0.278500
148 0.841037 0.292420 0.811309 0.281300
149 0.858974 0.299640 0.766421 0.262100
150 0.797129 0.277620 0.779408 0.267600
151 0.831622 0.291940 0.746741 0.256300
152 0.753381 0.263660 0.758080 0.262200
153 0.810279 0.282840 0.765352 0.263800
154 0.842484 0.295760 0.771300 0.263900
155 0.792305 0.277680 0.729488 0.250700
156 0.712752 0.248260 0.728300 0.248900
157 0.698934 0.244480 0.734653 0.251700
158 0.811315 0.283540 0.741308 0.254600
159 0.761607 0.266960 0.729404 0.248800
160 0.764513 0.268940 0.724224 0.248400
161 0.707847 0.247220 0.706810 0.242500
162 0.688261 0.241220 0.713286 0.246000
163 0.748346 0.260660 0.731541 0.254300
164 0.691880 0.242800 0.719255 0.244800
165 0.696499 0.243320 0.697712 0.237700
166 0.757661 0.267740 0.697511 0.239600
167 0.693566 0.243340 0.688757 0.233500
168 0.660016 0.229700 0.693834 0.238200
169 0.621602 0.216960 0.698628 0.237900
170 0.693565 0.241460 0.683942 0.233800
171 0.791773 0.276660 0.718621 0.248300
172 0.694664 0.241320 0.679286 0.230000
173 0.749198 0.260960 0.688338 0.236900
174 0.690450 0.240540 0.695980 0.237200
175 0.670440 0.234040 0.697503 0.239100
176 0.625189 0.219900 0.699741 0.240600
177 0.589703 0.208140 0.681859 0.231800
178 0.643510 0.224760 0.680289 0.235600
179 0.644631 0.226000 0.677113 0.228600
180 0.573221 0.200240 0.667921 0.226300
181 0.591352 0.207860 0.669414 0.225000
182 0.644971 0.225320 0.660583 0.222400
183 0.558789 0.196260 0.649809 0.218600
184 0.719270 0.253800 0.685986 0.233600
185 0.604209 0.212560 0.647094 0.219500
186 0.578746 0.202120 0.651767 0.220500
187 0.594220 0.207120 0.649317 0.219500
188 0.566839 0.198320 0.654618 0.218600
189 0.550815 0.190560 0.640559 0.214600
190 0.736706 0.258080 0.658022 0.223000
191 0.653457 0.231120 0.648714 0.216400
192 0.663878 0.233520 0.654263 0.223300
193 0.556408 0.197080 0.648131 0.219600
194 0.672649 0.235160 0.659517 0.221800
195 0.634843 0.221960 0.657571 0.221900
196 0.532282 0.187980 0.669375 0.226700
197 0.610847 0.216980 0.634669 0.214200
198 0.602794 0.211580 0.637142 0.215600
199 0.500774 0.178000 0.644303 0.216100
200 0.556853 0.197860 0.653069 0.219800
201 0.473643 0.166640 0.637706 0.215500
202 0.543367 0.192500 0.630976 0.210900
203 0.547610 0.193560 0.647321 0.218400
204 0.549161 0.193160 0.652783 0.216800
205 0.555136 0.194700 0.638940 0.214200
206 0.681273 0.239900 0.652195 0.219600
207 0.538329 0.188140 0.632762 0.216300
208 0.536182 0.188480 0.635168 0.214900
209 0.542705 0.191000 0.627548 0.210600
210 0.575116 0.203320 0.639326 0.214100
211 0.507712 0.177960 0.623835 0.210300
212 0.571738 0.201600 0.637558 0.215800
213 0.528358 0.185160 0.655310 0.222300
214 0.590359 0.208460 0.658555 0.221200
215 0.530377 0.187640 0.635255 0.214300
216 0.494095 0.174020 0.635528 0.210600
217 0.482562 0.170700 0.641069 0.218300
218 0.561212 0.199020 0.645154 0.217900
219 0.461704 0.163360 0.621216 0.209100
220 0.497201 0.176980 0.635099 0.216300
221 0.519957 0.184700 0.638311 0.211700
222 0.607542 0.215080 0.627521 0.206700
223 0.566220 0.201660 0.630709 0.208700
224 0.516641 0.182200 0.640494 0.213500
225 0.467458 0.167020 0.632075 0.211200
226 0.561692 0.199120 0.636891 0.208600
227 0.527434 0.186860 0.636249 0.214700
228 0.493206 0.175560 0.642024 0.217900
229 0.435306 0.153840 0.624948 0.207200
230 0.465713 0.163760 0.625984 0.206600
231 0.613709 0.216140 0.627374 0.209800
232 0.585429 0.206380 0.639690 0.216400
233 0.486990 0.173220 0.627945 0.212600
234 0.611468 0.214740 0.634425 0.215000
235 0.508671 0.181200 0.627543 0.208800
236 0.505741 0.179560 0.623797 0.208300
237 0.546465 0.192600 0.624793 0.209700
238 0.471171 0.169480 0.617415 0.204800
239 0.493997 0.174900 0.624413 0.207300
240 0.411756 0.145880 0.627248 0.202000
241 0.553179 0.194820 0.638805 0.215800
242 0.422501 0.149560 0.622032 0.208500
243 0.531075 0.188420 0.621820 0.206800
244 0.489139 0.173960 0.627738 0.207800
245 0.419156 0.147180 0.629689 0.202700
246 0.489858 0.172860 0.618313 0.203900
247 0.510623 0.182660 0.631426 0.212700
248 0.496845 0.174860 0.625647 0.211500
249 0.420132 0.149140 0.620893 0.204800
250 0.512875 0.180660 0.614870 0.202300
251 0.524712 0.184420 0.615267 0.205400
252 0.381032 0.134060 0.630305 0.204900
253 0.426079 0.150820 0.614480 0.200200
254 0.413228 0.147720 0.626044 0.206800
255 0.356428 0.126460 0.636710 0.206000
256 0.434676 0.153440 0.636250 0.210000
257 0.442012 0.157880 0.629783 0.206400
258 0.473236 0.169440 0.630550 0.211200
259 0.447313 0.158560 0.638433 0.212900
260 0.414860 0.147860 0.628668 0.208500
261 0.392432 0.140000 0.641313 0.205600
262 0.511654 0.179940 0.627193 0.212100
263 0.507849 0.178880 0.652576 0.219200
264 0.566355 0.199360 0.638171 0.212300
265 0.425083 0.149560 0.615276 0.202100
266 0.474922 0.167860 0.627144 0.205600
267 0.431780 0.151740 0.625363 0.204400
268 0.454628 0.162160 0.629383 0.205700
269 0.457794 0.162160 0.623723 0.204100
270 0.548920 0.193420 0.619574 0.208200
271 0.415419 0.148140 0.630262 0.207600
272 0.334148 0.117640 0.632166 0.205000
273 0.523316 0.185060 0.632792 0.207100
274 0.472590 0.166780 0.625767 0.207700
275 0.366252 0.130020 0.629547 0.204900
276 0.409062 0.144900 0.625694 0.203800
277 0.469791 0.165200 0.634042 0.208000
278 0.431191 0.152160 0.630712 0.201600
279 0.345246 0.123700 0.628148 0.198100
280 0.465820 0.164240 0.623144 0.207100
281 0.325065 0.115620 0.630440 0.201900
282 0.420623 0.150320 0.624133 0.202800
283 0.398687 0.142820 0.623236 0.204400
284 0.585759 0.203360 0.628997 0.209400
285 0.481718 0.171620 0.631858 0.206900
286 0.406801 0.145260 0.642828 0.208700
287 0.424924 0.151320 0.623796 0.201900
288 0.415299 0.147700 0.636585 0.206300
289 0.472887 0.167360 0.639477 0.208900
290 0.389044 0.136600 0.635800 0.201700
291 0.349180 0.124700 0.628040 0.197800
292 0.448326 0.158620 0.630691 0.203900
293 0.358445 0.128200 0.636178 0.201000
294 0.522634 0.184980 0.632877 0.210200
295 0.458186 0.162820 0.626756 0.204600
296 0.382808 0.135500 0.631453 0.207800
297 0.438598 0.156740 0.628119 0.206600
298 0.371808 0.133780 0.651291 0.211200
299 0.429471 0.152240 0.631111 0.205000
300 0.536832 0.189640 0.631903 0.210000
301 0.334097 0.117920 0.631833 0.200600
302 0.430021 0.151060 0.619491 0.202800
303 0.413743 0.148440 0.627023 0.205200
304 0.324145 0.115500 0.642452 0.200700
305 0.355051 0.125440 0.637457 0.202900
306 0.451619 0.159740 0.636109 0.208000
real 59m12.227s
user 80m49.696s
sys 85m0.752s
}};
***C[0.001]-P-C[0.001]-P-F[0.01]-F[0.01]
#pre{{
############# training condition ############
500 epoch training. input_size: (28, 28) , minibatch_size: 100
, learning_rate: 0.01 , momentum: 0.9 , weight_decay: 0.0005
#############################################
########## architecture of the CNN #########
layer1 - Convolution
< kernel_size: (50, 2, 2) , stride: (1, 1) , act_func: ReLU
, border_mode: valid , out_size: (50, 27, 27) , iniW_σ: 0.001000 >
layer2 - Convolution
< kernel_size: (50, 2, 2) , stride: (1, 1) , act_func: ReLU
, border_mode: valid , out_size: (50, 26, 26) , iniW_σ: 0.001000 >
layer3 - Pooling
< downscale: (4, 4) , stride: None , out_size: (50, 6, 6) >
layer4 - fully-connected
< number_of_units: 1000 , drop_rate: 0.8 , act_func: ReLU , iniW_σ: 0.010000 >
layer5 - fully-connected
< number_of_units: 10 , drop_rate: 0.5 , act_func: softmax , iniW_σ: 0.010000 >
###########################################
epoch XE(L) error(L) XE(T) error(T)
1 2.302872 0.898900 2.302660 0.900000
2 2.302858 0.901900 2.302781 0.900000
3 2.302803 0.901360 2.302714 0.900000
4 2.302769 0.898940 2.302536 0.900000
5 2.302424 0.897020 2.301566 0.900000
6 2.276309 0.858520 2.169257 0.787100
7 2.072670 0.758680 2.005378 0.719300
8 2.014385 0.733880 1.987759 0.727200
9 1.984420 0.721800 1.926013 0.696000
10 1.946731 0.713860 1.890811 0.678500
11 1.916479 0.695900 1.856166 0.667600
12 1.896306 0.690600 1.887214 0.685200
13 1.887320 0.688060 1.844342 0.662100
14 1.874480 0.683780 1.821667 0.656600
15 1.855282 0.674580 1.806978 0.649200
16 1.841777 0.668900 1.829868 0.653600
17 1.827406 0.664220 1.799668 0.649300
18 1.829521 0.663860 1.766902 0.639000
19 1.830559 0.664520 1.746905 0.619000
20 1.729513 0.624260 1.631033 0.582600
21 1.638321 0.585920 1.563860 0.555500
22 1.602554 0.574500 1.489464 0.526200
23 1.543845 0.551720 1.468083 0.523900
24 1.513750 0.540320 1.430057 0.500200
25 1.462747 0.521020 1.395763 0.497300
26 1.430794 0.508380 1.368149 0.482800
27 1.419554 0.505040 1.352399 0.479500
28 1.397583 0.495760 1.304829 0.457500
29 1.360754 0.484840 1.284288 0.453700
30 1.310166 0.463500 1.259930 0.439600
31 1.287544 0.456280 1.203224 0.420400
32 1.256335 0.444860 1.197972 0.421400
33 1.207174 0.423920 1.179540 0.411500
34 1.214620 0.430120 1.125907 0.389500
35 1.164442 0.408680 1.134102 0.395600
36 1.203204 0.427800 1.088162 0.370600
37 1.161411 0.407680 1.066622 0.369100
38 1.144375 0.399700 1.048877 0.360600
39 1.078669 0.378340 0.987294 0.339000
40 1.059620 0.373000 0.994588 0.345600
41 1.068687 0.372000 0.982816 0.340900
42 1.015443 0.356140 0.937158 0.326100
43 1.005130 0.352380 0.927095 0.319600
44 0.992993 0.346840 0.918870 0.316800
45 1.004931 0.349700 0.921541 0.319900
46 1.014245 0.353240 0.945760 0.324800
47 0.958723 0.333060 0.916614 0.320300
48 0.954285 0.333700 0.868280 0.297900
49 0.871379 0.304040 0.837763 0.291400
50 0.951212 0.329500 0.855732 0.293300
51 0.925132 0.323060 0.841287 0.291300
52 0.826395 0.289580 0.811994 0.280500
53 0.850422 0.296380 0.814791 0.282200
54 0.806513 0.281620 0.818986 0.280900
55 0.890272 0.310980 0.818280 0.280500
56 0.832098 0.291500 0.800061 0.272900
57 0.802854 0.280280 0.794087 0.273000
58 0.780356 0.272840 0.793135 0.269800
59 0.870416 0.302300 0.813302 0.280400
60 0.775174 0.271740 0.755276 0.259200
61 0.741149 0.258020 0.763465 0.259500
62 0.795234 0.277920 0.807325 0.280800
63 0.791168 0.277520 0.764574 0.261000
64 0.777582 0.268900 0.735795 0.250100
65 0.849017 0.296300 0.761325 0.259900
66 0.750857 0.261180 0.732703 0.252300
67 0.782823 0.272460 0.730523 0.248700
68 0.720695 0.252040 0.729317 0.252700
69 0.721221 0.254040 0.707432 0.238900
70 0.663256 0.231140 0.719061 0.246600
71 0.702100 0.246580 0.712442 0.239400
72 0.706973 0.246760 0.715738 0.247200
73 0.681144 0.238480 0.704208 0.240300
74 0.694613 0.242520 0.707136 0.238800
75 0.687925 0.242060 0.699915 0.234200
76 0.699514 0.245760 0.689479 0.231500
77 0.696203 0.244640 0.706988 0.236800
78 0.665517 0.232720 0.701321 0.234500
79 0.706234 0.247340 0.695606 0.241300
80 0.661712 0.231800 0.688041 0.234000
81 0.640701 0.223980 0.684495 0.232200
82 0.660487 0.229720 0.693339 0.234800
83 0.673713 0.237440 0.682402 0.230800
84 0.620994 0.217000 0.680232 0.236400
85 0.646409 0.226920 0.676887 0.229400
86 0.645745 0.227860 0.664853 0.220100
87 0.689577 0.241760 0.673440 0.228400
88 0.601237 0.208440 0.667513 0.224600
89 0.609765 0.215080 0.673499 0.229300
90 0.642425 0.224980 0.662619 0.224400
91 0.618559 0.215380 0.668127 0.227800
92 0.655401 0.230220 0.655188 0.217500
93 0.669924 0.234000 0.659074 0.220400
94 0.560884 0.196020 0.677943 0.230300
95 0.649010 0.225580 0.662177 0.227000
96 0.690835 0.241180 0.664829 0.227900
97 0.650865 0.227640 0.643818 0.213200
98 0.566280 0.198040 0.644339 0.217400
99 0.613401 0.214320 0.639799 0.216100
100 0.667891 0.235540 0.641766 0.213800
101 0.580818 0.205480 0.638238 0.216000
102 0.536167 0.189780 0.649514 0.214000
103 0.709693 0.246580 0.663111 0.225000
104 0.631701 0.221200 0.652446 0.226100
105 0.619709 0.216680 0.654964 0.222000
106 0.533599 0.186440 0.641212 0.214100
107 0.493039 0.173280 0.651675 0.217000
108 0.645964 0.224960 0.632432 0.209600
109 0.618259 0.217500 0.645378 0.216700
110 0.514565 0.181820 0.637972 0.213700
111 0.534446 0.186740 0.645896 0.214100
112 0.522354 0.183960 0.638978 0.212200
113 0.600494 0.211620 0.643802 0.214100
114 0.549375 0.195360 0.645564 0.213000
115 0.501327 0.177980 0.626850 0.211100
116 0.616277 0.216520 0.656425 0.224100
117 0.573093 0.203100 0.636134 0.215400
118 0.668564 0.232860 0.651479 0.218300
119 0.566857 0.200380 0.663800 0.217600
120 0.589800 0.208400 0.650366 0.219400
121 0.547857 0.193800 0.634174 0.214200
122 0.538350 0.191180 0.632722 0.211500
123 0.509841 0.180000 0.622958 0.209700
124 0.447756 0.157080 0.628028 0.210800
125 0.555008 0.195080 0.632165 0.212900
126 0.485475 0.168980 0.620688 0.205200
127 0.506353 0.179820 0.629752 0.211600
128 0.575891 0.202620 0.650257 0.217900
129 0.613002 0.217260 0.650063 0.221000
130 0.456279 0.162080 0.630299 0.211000
131 0.458614 0.161660 0.636779 0.209600
132 0.510723 0.180580 0.649673 0.218800
133 0.470076 0.164560 0.622362 0.203900
134 0.565348 0.200940 0.653620 0.220300
135 0.523974 0.185100 0.636000 0.211200
136 0.418222 0.149120 0.650506 0.215900
137 0.498014 0.175140 0.626513 0.205400
138 0.436453 0.155120 0.630010 0.205600
139 0.431618 0.154040 0.630019 0.206800
140 0.459956 0.163020 0.634649 0.209500
141 0.384548 0.135240 0.645044 0.207900
142 0.515291 0.183680 0.652195 0.215700
143 0.506902 0.177940 0.633119 0.209500
144 0.425628 0.150740 0.653576 0.210200
145 0.607325 0.213460 0.640440 0.213700
146 0.500311 0.176800 0.628680 0.207900
147 0.538828 0.189120 0.638379 0.212300
148 0.554753 0.194980 0.627643 0.210500
149 0.519375 0.182780 0.615895 0.204200
150 0.378655 0.134740 0.637201 0.206600
151 0.453305 0.160340 0.626257 0.205900
152 0.512213 0.181380 0.634887 0.213200
153 0.552009 0.195980 0.634409 0.208700
154 0.483465 0.172520 0.613985 0.201000
155 0.481577 0.172640 0.623544 0.204200
156 0.455515 0.161200 0.617180 0.202800
157 0.603864 0.211500 0.634859 0.213600
158 0.388869 0.136280 0.617869 0.203500
159 0.520007 0.184220 0.621469 0.208500
160 0.573866 0.201960 0.621250 0.204100
161 0.545355 0.190700 0.628192 0.209200
162 0.396342 0.139500 0.615180 0.198300
163 0.502700 0.177000 0.629004 0.207400
164 0.547316 0.192460 0.625879 0.211100
165 0.474462 0.168240 0.608074 0.202300
166 0.429055 0.151400 0.612654 0.200300
167 0.444818 0.159240 0.637688 0.205700
168 0.462765 0.163040 0.616273 0.205200
169 0.485579 0.171260 0.619473 0.204200
170 0.389189 0.137680 0.619296 0.200600
171 0.438430 0.156140 0.630109 0.207800
172 0.481412 0.170000 0.620443 0.200400
173 0.470107 0.166080 0.631968 0.212400
174 0.479419 0.170640 0.629643 0.205400
175 0.365037 0.129640 0.624014 0.201400
176 0.372456 0.132940 0.629099 0.204800
177 0.504970 0.178740 0.619571 0.202600
178 0.357786 0.129120 0.619522 0.200700
179 0.338896 0.119140 0.640979 0.205900
180 0.406746 0.144440 0.637314 0.208800
181 0.374679 0.134800 0.634658 0.206700
182 0.362342 0.128320 0.617033 0.202900
183 0.336896 0.120900 0.639205 0.204800
184 0.506341 0.178800 0.632390 0.206700
185 0.323573 0.115340 0.640153 0.202300
186 0.360581 0.127460 0.633997 0.204900
187 0.523081 0.183560 0.642037 0.216100
188 0.439722 0.156820 0.628661 0.200200
189 0.429584 0.152000 0.634667 0.206500
190 0.478491 0.171140 0.611366 0.199200
191 0.526997 0.185360 0.654256 0.217300
192 0.314695 0.112120 0.651739 0.204400
193 0.373587 0.134240 0.632306 0.202100
194 0.332759 0.118540 0.642136 0.205300
195 0.505420 0.178400 0.636079 0.207300
196 0.306033 0.107780 0.647592 0.206400
197 0.289547 0.104060 0.663207 0.206300
198 0.417031 0.147400 0.639258 0.209100
199 0.365849 0.130020 0.656354 0.210400
200 0.529010 0.185280 0.639793 0.208900
real 38m49.140s
user 52m52.912s
sys 56m25.160s
}};
***C[0.01]-P-C[0.01]-P-F[0.001]-F[0.001]
#pre{{
############# training condition ############
500 epoch training. input_size: (28, 28) , minibatch_size: 100
, learning_rate: 0.01 , momentum: 0.9 , weight_decay: 0.0005
#############################################
########## architecture of the CNN #########
layer1 - Convolution
< kernel_size: (50, 2, 2) , stride: (1, 1) , act_func: ReLU
, border_mode: valid , out_size: (50, 27, 27) , iniW_σ: 0.010000 >
layer2 - Convolution
< kernel_size: (50, 2, 2) , stride: (1, 1) , act_func: ReLU
, border_mode: valid , out_size: (50, 26, 26) , iniW_σ: 0.010000 >
layer3 - Pooling
< downscale: (4, 4) , stride: None , out_size: (50, 6, 6) >
layer4 - fully-connected
< number_of_units: 1000 , drop_rate: 0.8 , act_func: ReLU , iniW_σ: 0.001000 >
layer5 - fully-connected
< number_of_units: 10 , drop_rate: 0.5 , act_func: softmax , iniW_σ: 0.001000 >
###########################################
epoch XE(L) error(L) XE(T) error(T)
1 2.302901 0.901300 2.302623 0.900000
2 2.302820 0.901660 2.302581 0.900000
3 2.302217 0.897120 2.299640 0.882000
4 2.230180 0.848140 2.112987 0.776600
5 2.071772 0.765080 2.018637 0.734000
6 2.024891 0.741720 1.993755 0.715000
7 2.005046 0.731740 1.987875 0.728900
8 1.983595 0.721320 1.941262 0.704800
9 1.970283 0.715120 1.931210 0.689400
10 1.939247 0.706840 1.880771 0.676300
11 1.925168 0.697200 1.859701 0.661500
12 1.911540 0.695640 1.872053 0.680600
13 1.900941 0.688160 1.855476 0.669400
14 1.885428 0.682280 1.845671 0.658300
15 1.887153 0.683660 1.838827 0.655800
16 1.862930 0.671900 1.818500 0.650200
17 1.863536 0.673000 1.809071 0.647100
18 1.789304 0.640780 1.697487 0.596600
19 1.705984 0.606700 1.618248 0.577700
20 1.645008 0.589960 1.572515 0.560100
21 1.600171 0.577340 1.623008 0.572500
22 1.538985 0.551240 1.459121 0.524300
23 1.512821 0.544740 1.440079 0.511300
24 1.489921 0.537300 1.437246 0.506300
25 1.421916 0.509780 1.369165 0.487300
26 1.388289 0.496780 1.308086 0.462300
27 1.383351 0.494060 1.288331 0.453800
28 1.367467 0.489860 1.292792 0.453500
29 1.322461 0.471560 1.244533 0.438400
30 1.283352 0.455240 1.231320 0.432500
31 1.273257 0.452500 1.193134 0.419400
32 1.274518 0.453240 1.225659 0.435000
33 1.236957 0.440480 1.176855 0.415100
34 1.235800 0.436800 1.161239 0.402700
35 1.224157 0.436300 1.111506 0.381400
36 1.197053 0.422840 1.118210 0.388300
37 1.150130 0.406760 1.069545 0.369300
38 1.118568 0.395240 1.060744 0.366100
39 1.122202 0.396220 1.064548 0.372600
40 1.091638 0.385880 1.019541 0.352700
41 1.081617 0.379300 0.967701 0.342300
42 1.060177 0.368940 0.999182 0.351200
43 0.989399 0.346460 0.958429 0.334500
44 1.002523 0.351440 0.922679 0.320000
45 0.970988 0.339800 0.929425 0.324700
46 1.027290 0.360400 0.910248 0.317300
47 0.921182 0.322060 0.879655 0.308700
48 0.890499 0.311240 0.865614 0.304200
49 0.903701 0.314560 0.899403 0.308900
50 0.906859 0.317240 0.866775 0.301300
51 0.844980 0.293460 0.841168 0.295700
52 0.956383 0.332780 0.845176 0.290800
53 0.854559 0.299100 0.834348 0.295500
54 0.820551 0.288820 0.832715 0.291300
55 0.827537 0.290020 0.823778 0.288000
56 0.794404 0.278280 0.802516 0.279200
57 0.919831 0.321780 0.804460 0.274800
58 0.947838 0.331940 0.807488 0.277000
59 0.818844 0.283660 0.803254 0.280300
60 0.814942 0.284740 0.760469 0.260300
61 0.766425 0.267640 0.765969 0.264000
62 0.849141 0.293940 0.753863 0.253700
63 0.834052 0.291400 0.755548 0.258800
64 0.818990 0.288020 0.763077 0.265100
65 0.867316 0.301760 0.759022 0.260300
66 0.836070 0.289560 0.751518 0.257900
67 0.783445 0.271820 0.738580 0.254000
68 0.808771 0.281320 0.763343 0.266100
69 0.798330 0.277260 0.774414 0.265000
70 0.751814 0.261960 0.747813 0.259100
71 0.756193 0.264800 0.716244 0.247800
72 0.719148 0.250520 0.723906 0.248000
73 0.756608 0.263240 0.719750 0.241700
74 0.771815 0.267760 0.713564 0.245900
75 0.706465 0.247160 0.695171 0.237700
76 0.665230 0.230280 0.694496 0.240200
77 0.679877 0.238100 0.703050 0.238900
78 0.641064 0.225080 0.694722 0.238800
79 0.643016 0.226540 0.685286 0.234500
80 0.665971 0.233140 0.701992 0.240600
81 0.750654 0.261080 0.681645 0.232000
82 0.725471 0.253200 0.673750 0.231300
83 0.678729 0.235480 0.672499 0.236400
84 0.676068 0.236100 0.664762 0.225300
85 0.601115 0.211680 0.668561 0.227800
86 0.595666 0.209800 0.670929 0.229000
87 0.626114 0.218340 0.654591 0.223100
88 0.674411 0.236280 0.661875 0.226100
89 0.661216 0.230800 0.644898 0.222900
90 0.734703 0.255400 0.682345 0.233500
91 0.646937 0.227020 0.655013 0.223400
92 0.632959 0.221040 0.644096 0.219000
93 0.639466 0.225200 0.656427 0.223200
94 0.715922 0.249580 0.657482 0.228800
95 0.655604 0.228600 0.663985 0.227900
96 0.628675 0.218700 0.634387 0.215600
97 0.636521 0.225440 0.640979 0.218300
98 0.658399 0.230140 0.645956 0.220100
99 0.573348 0.201100 0.630046 0.213300
100 0.550218 0.192360 0.638988 0.217000
101 0.541562 0.191540 0.628448 0.212800
102 0.609731 0.213660 0.637119 0.219500
103 0.541422 0.189760 0.627562 0.217200
104 0.594789 0.206520 0.632211 0.216100
105 0.533965 0.188240 0.619938 0.212100
106 0.614995 0.215720 0.638859 0.217700
107 0.518768 0.181300 0.632164 0.212200
108 0.510290 0.178900 0.637924 0.218600
109 0.514615 0.181840 0.613340 0.209400
110 0.653121 0.228860 0.650518 0.222600
111 0.616260 0.217580 0.634341 0.217700
112 0.521380 0.183800 0.630964 0.211400
113 0.484702 0.170940 0.617360 0.210900
114 0.607867 0.214240 0.637023 0.219400
115 0.515181 0.180100 0.613999 0.208600
116 0.519477 0.183900 0.637284 0.213500
117 0.475232 0.167840 0.629038 0.211400
118 0.557439 0.197300 0.638357 0.217600
119 0.598365 0.210760 0.627784 0.211400
120 0.512726 0.181080 0.613917 0.207900
121 0.577865 0.201420 0.615097 0.210300
122 0.528882 0.186140 0.610660 0.206500
123 0.603721 0.210940 0.628577 0.215500
124 0.565278 0.199160 0.622649 0.209200
125 0.491616 0.173440 0.619445 0.210800
126 0.435459 0.154320 0.617538 0.207900
127 0.438711 0.155760 0.625713 0.212500
128 0.508624 0.178940 0.615024 0.205800
129 0.487787 0.172260 0.632125 0.213400
130 0.420390 0.149080 0.628939 0.210400
131 0.519737 0.183280 0.611218 0.211000
132 0.488809 0.173080 0.610631 0.205500
133 0.411985 0.143900 0.612675 0.200700
134 0.425841 0.150760 0.607322 0.202800
135 0.564123 0.197460 0.620712 0.211400
136 0.619336 0.217700 0.627627 0.215600
137 0.500103 0.178560 0.605096 0.207300
138 0.601750 0.211960 0.614226 0.209400
139 0.409697 0.144840 0.628751 0.214100
140 0.426528 0.150440 0.612627 0.207400
141 0.418140 0.149120 0.626981 0.211200
142 0.508491 0.177940 0.616579 0.206600
143 0.400348 0.140980 0.611378 0.199900
144 0.558010 0.194900 0.610655 0.207500
145 0.436419 0.153820 0.620998 0.207500
146 0.592183 0.207060 0.618941 0.211600
147 0.492563 0.174560 0.608695 0.205800
148 0.460964 0.161960 0.594691 0.197900
149 0.533392 0.188640 0.610247 0.203900
150 0.387436 0.136380 0.607173 0.200900
151 0.532460 0.186620 0.616157 0.207400
152 0.362419 0.129580 0.616162 0.202900
153 0.466150 0.166000 0.616198 0.202600
154 0.503470 0.178000 0.613591 0.203800
155 0.528610 0.186960 0.612632 0.203200
156 0.515903 0.182660 0.608241 0.201600
157 0.361893 0.127440 0.625812 0.204100
158 0.332449 0.118520 0.619963 0.200700
159 0.571487 0.201460 0.613481 0.209000
160 0.382678 0.134820 0.621906 0.206000
161 0.434988 0.155560 0.621866 0.205700
162 0.305630 0.108640 0.626495 0.200200
163 0.505812 0.178360 0.611657 0.210200
164 0.444961 0.157260 0.610047 0.206000
165 0.483167 0.171960 0.616183 0.210300
166 0.450604 0.158760 0.617694 0.205400
167 0.413043 0.145360 0.608303 0.200700
168 0.298992 0.106240 0.620169 0.200600
169 0.434270 0.153600 0.616022 0.205200
170 0.365111 0.130840 0.613091 0.199300
171 0.358076 0.128580 0.614925 0.201400
172 0.300774 0.106440 0.615542 0.197600
173 0.363072 0.129160 0.622786 0.202700
174 0.488701 0.173240 0.616393 0.203400
175 0.462220 0.162760 0.612299 0.207500
176 0.323003 0.113780 0.622196 0.203600
177 0.438456 0.154780 0.616680 0.204500
178 0.363259 0.128480 0.620704 0.202300
179 0.496789 0.175420 0.625555 0.207600
180 0.403707 0.140840 0.608423 0.200000
181 0.506264 0.178940 0.611625 0.205800
182 0.338968 0.120360 0.603149 0.198300
183 0.392754 0.139340 0.608893 0.197800
184 0.530670 0.187840 0.603998 0.200700
185 0.397160 0.139960 0.597075 0.196000
186 0.447407 0.159900 0.599321 0.198900
187 0.487685 0.172500 0.605278 0.202000
188 0.490796 0.173180 0.598128 0.199600
189 0.381923 0.136320 0.603520 0.197300
190 0.483876 0.170920 0.605777 0.200200
191 0.417605 0.147780 0.610298 0.204100
192 0.354936 0.126120 0.601664 0.195900
193 0.475200 0.168520 0.613556 0.197200
194 0.445222 0.156600 0.620630 0.199900
195 0.407889 0.143760 0.609373 0.199100
196 0.340630 0.121420 0.610759 0.196500
197 0.314984 0.112240 0.615860 0.197400
198 0.364542 0.129200 0.609140 0.197400
199 0.375407 0.134680 0.606543 0.194400
200 0.400660 0.142460 0.608066 0.196600
real 38m47.191s
user 52m58.352s
sys 55m38.200s
}};
***C[0.01]-P-C[0.01]-P-F[0.01]-F[0.01]
#pre{{
############# training condition ############
500 epoch training. input_size: (28, 28) , minibatch_size: 100
, learning_rate: 0.01 , momentum: 0.9 , weight_decay: 0.0005
#############################################
########## architecture of the CNN #########
layer1 - Convolution
< kernel_size: (50, 2, 2) , stride: (1, 1) , act_func: ReLU
, border_mode: valid , out_size: (50, 27, 27) , iniW_σ: 0.010000 >
layer2 - Convolution
< kernel_size: (50, 2, 2) , stride: (1, 1) , act_func: ReLU
, border_mode: valid , out_size: (50, 26, 26) , iniW_σ: 0.010000 >
layer3 - Pooling
< downscale: (4, 4) , stride: None , out_size: (50, 6, 6) >
layer4 - fully-connected
< number_of_units: 1000 , drop_rate: 0.8 , act_func: ReLU , iniW_σ: 0.010000 >
layer5 - fully-connected
< number_of_units: 10 , drop_rate: 0.5 , act_func: softmax , iniW_σ: 0.010000 >
###########################################
epoch XE(L) error(L) XE(T) error(T)
1 2.295846 0.889220 2.262589 0.856300
2 2.115511 0.770380 2.014915 0.732600
3 2.020671 0.734360 1.971737 0.706800
4 1.980240 0.718320 1.931875 0.707900
5 1.944661 0.706700 1.904450 0.687500
6 1.922105 0.698880 1.896850 0.683000
7 1.905911 0.692200 1.871081 0.674900
8 1.886066 0.682940 1.843626 0.666900
9 1.882368 0.681560 1.838893 0.664500
10 1.869553 0.676960 1.815571 0.650500
11 1.864278 0.673200 1.837251 0.659500
12 1.873286 0.676240 1.812946 0.645400
13 1.818123 0.653520 1.732392 0.612100
14 1.749607 0.624340 1.656842 0.594700
15 1.653327 0.590760 1.561860 0.555800
16 1.594014 0.569740 1.514609 0.540300
17 1.548157 0.556760 1.528837 0.542500
18 1.503404 0.537180 1.415669 0.507300
19 1.459075 0.522320 1.394656 0.497300
20 1.423965 0.506960 1.362277 0.477700
21 1.404101 0.499880 1.330899 0.472400
22 1.351620 0.482200 1.290235 0.453500
23 1.325897 0.472540 1.253288 0.439300
24 1.328091 0.472800 1.234367 0.431100
25 1.285086 0.458520 1.195523 0.418900
26 1.278986 0.454940 1.167001 0.407300
27 1.230778 0.437560 1.169653 0.409800
28 1.254427 0.445320 1.151704 0.397200
29 1.179577 0.420480 1.113861 0.388000
30 1.180674 0.415880 1.091560 0.378900
31 1.220288 0.430180 1.122305 0.389400
32 1.162690 0.409680 1.106598 0.383200
33 1.158491 0.407460 1.094822 0.381700
34 1.121711 0.394400 1.023304 0.353700
35 1.075405 0.378720 1.005997 0.351100
36 1.051160 0.369500 0.972907 0.337700
37 1.090402 0.382400 0.959699 0.327700
38 1.056046 0.371380 0.957709 0.335300
39 1.019505 0.356000 0.963410 0.336700
40 0.984273 0.344980 0.923884 0.319400
41 0.965589 0.338220 0.907603 0.311500
42 1.063812 0.374860 0.911057 0.312200
43 0.951267 0.335120 0.897823 0.305400
44 0.959741 0.334940 0.895456 0.305600
45 0.884838 0.311280 0.839286 0.285100
46 0.978875 0.341900 0.854322 0.296700
47 0.932763 0.324100 0.848260 0.295400
48 0.869802 0.303320 0.828480 0.281100
49 0.928031 0.323380 0.859657 0.297800
50 0.910751 0.316660 0.829283 0.287900
51 0.828789 0.290060 0.814298 0.280700
52 0.855133 0.300180 0.799630 0.274500
53 0.867939 0.302020 0.788236 0.267700
54 0.876353 0.305140 0.797260 0.269900
55 0.836990 0.291780 0.765439 0.263200
56 0.798550 0.278140 0.764460 0.261800
57 0.849203 0.296000 0.776326 0.267700
58 0.816710 0.285240 0.751844 0.256900
59 0.788163 0.277620 0.752705 0.257200
60 0.812437 0.285180 0.754142 0.258300
61 0.810678 0.282800 0.758790 0.258300
62 0.783860 0.273800 0.750935 0.258500
63 0.783673 0.271280 0.758504 0.262100
64 0.776376 0.269860 0.734984 0.255000
65 0.780174 0.273660 0.726331 0.246000
66 0.737401 0.258100 0.710941 0.240400
67 0.822316 0.286380 0.721608 0.244900
68 0.732870 0.257040 0.721644 0.246400
69 0.752138 0.260680 0.689774 0.232800
70 0.722814 0.251820 0.720069 0.243400
71 0.806843 0.283540 0.712912 0.241800
72 0.693675 0.241860 0.696936 0.239000
73 0.732292 0.257680 0.723535 0.250400
74 0.743348 0.258320 0.692517 0.235200
75 0.698793 0.243020 0.692085 0.235500
76 0.656595 0.233220 0.679415 0.228100
77 0.686947 0.238740 0.688888 0.233200
78 0.695408 0.244160 0.689658 0.233500
79 0.676750 0.237000 0.673825 0.230100
80 0.607276 0.210120 0.688524 0.236800
81 0.763574 0.265000 0.701790 0.240700
82 0.696277 0.240880 0.704207 0.241700
83 0.709175 0.247520 0.698019 0.238200
84 0.656472 0.232080 0.675286 0.230800
85 0.736603 0.257940 0.696097 0.237500
86 0.665403 0.231300 0.655941 0.222900
87 0.689938 0.241400 0.692603 0.238300
88 0.637561 0.221680 0.692464 0.241100
89 0.603042 0.210800 0.670211 0.225600
90 0.719143 0.250720 0.673726 0.228600
91 0.638884 0.224260 0.670063 0.228500
92 0.669008 0.233440 0.689416 0.235800
93 0.580914 0.204200 0.662864 0.229100
94 0.691088 0.242320 0.671204 0.231300
95 0.610035 0.211500 0.660191 0.221400
96 0.580456 0.204240 0.643380 0.220500
97 0.624617 0.218240 0.657044 0.220700
98 0.615550 0.217620 0.667579 0.225100
99 0.582741 0.205900 0.663710 0.227600
100 0.634705 0.221020 0.672440 0.226300
101 0.591286 0.208640 0.679087 0.234200
102 0.601860 0.212240 0.658298 0.224400
103 0.618291 0.217400 0.653479 0.219400
104 0.607953 0.213280 0.635141 0.214300
105 0.581175 0.203140 0.650927 0.223000
106 0.555254 0.193520 0.654571 0.220400
107 0.618177 0.217460 0.656775 0.223200
108 0.572511 0.201120 0.657990 0.224000
109 0.585749 0.208040 0.642138 0.220100
110 0.527424 0.185120 0.644592 0.221200
111 0.499200 0.174620 0.675473 0.227800
112 0.490442 0.172340 0.651364 0.218500
113 0.628758 0.221420 0.662762 0.222200
114 0.635877 0.225240 0.647096 0.218200
115 0.563327 0.200540 0.659880 0.226200
116 0.531017 0.185880 0.659357 0.225600
117 0.480397 0.171980 0.642966 0.212400
118 0.510882 0.180440 0.659652 0.225000
119 0.672867 0.233560 0.662867 0.227400
120 0.536841 0.189740 0.632090 0.213800
121 0.444981 0.156820 0.633691 0.208900
122 0.441651 0.157400 0.638043 0.212900
123 0.497385 0.177060 0.637389 0.214100
124 0.416272 0.148220 0.657665 0.220100
125 0.632245 0.222040 0.651039 0.221600
126 0.409079 0.145640 0.648700 0.216600
127 0.559476 0.195820 0.630556 0.211000
128 0.553231 0.192740 0.631859 0.208100
129 0.516978 0.182720 0.629586 0.209100
130 0.508360 0.178520 0.625875 0.207300
131 0.506639 0.177960 0.627617 0.208100
132 0.532023 0.187800 0.622849 0.210200
133 0.513279 0.181620 0.624994 0.210900
134 0.529050 0.188320 0.624003 0.209400
135 0.566943 0.199420 0.626655 0.211400
136 0.495513 0.176260 0.617666 0.203600
137 0.512096 0.180500 0.620363 0.209200
138 0.555502 0.196840 0.635807 0.216100
139 0.532674 0.188540 0.633788 0.213300
140 0.503084 0.176920 0.636918 0.211100
141 0.497505 0.176940 0.612319 0.206200
142 0.528009 0.187120 0.623163 0.209100
143 0.482022 0.171140 0.622657 0.208500
144 0.487121 0.173200 0.630329 0.213600
145 0.492959 0.175460 0.627584 0.206700
146 0.536013 0.190000 0.629574 0.210800
147 0.561766 0.199160 0.635424 0.212200
148 0.493793 0.174240 0.626691 0.207900
149 0.425548 0.149520 0.630802 0.210700
150 0.475931 0.168100 0.623399 0.209500
151 0.483713 0.170360 0.640011 0.209300
152 0.465127 0.164960 0.634834 0.214400
153 0.460989 0.163280 0.626684 0.206300
154 0.449315 0.158220 0.617004 0.204700
155 0.416660 0.145660 0.626971 0.202600
156 0.506524 0.179720 0.638759 0.211700
157 0.533466 0.188700 0.624977 0.208900
158 0.617077 0.216740 0.635375 0.212300
159 0.505615 0.177840 0.618484 0.206700
160 0.472271 0.166580 0.613608 0.203400
161 0.426948 0.151200 0.628950 0.210300
162 0.473673 0.168160 0.616989 0.204500
163 0.361854 0.129540 0.629883 0.204900
164 0.442030 0.156980 0.630763 0.208200
165 0.442348 0.156800 0.624419 0.209700
166 0.546094 0.191400 0.631525 0.210100
167 0.424228 0.150240 0.630012 0.207500
168 0.363285 0.128780 0.636904 0.209200
169 0.453622 0.161140 0.625986 0.208600
170 0.482874 0.172320 0.620622 0.207600
171 0.363466 0.129120 0.643209 0.206600
172 0.329619 0.116680 0.641204 0.205200
173 0.484069 0.172160 0.626184 0.207400
174 0.321387 0.111680 0.647512 0.207900
175 0.460836 0.163940 0.625833 0.204800
176 0.404673 0.144360 0.637315 0.207100
177 0.307870 0.109080 0.646901 0.202200
178 0.539832 0.188320 0.625369 0.209900
179 0.398925 0.142280 0.619288 0.203000
180 0.479182 0.169500 0.629842 0.207300
181 0.409125 0.145880 0.624094 0.202800
182 0.377236 0.134020 0.633550 0.206800
183 0.324477 0.116260 0.616041 0.199100
184 0.496429 0.175200 0.626253 0.206800
185 0.371767 0.133680 0.636571 0.204600
186 0.357283 0.130060 0.623920 0.206700
187 0.450721 0.158980 0.616588 0.201800
188 0.366726 0.132320 0.627136 0.204200
189 0.429640 0.153620 0.613509 0.200100
190 0.345599 0.123720 0.644716 0.204800
191 0.375227 0.134880 0.632093 0.206000
192 0.425309 0.150060 0.621840 0.202100
193 0.516373 0.181600 0.615686 0.202600
194 0.531388 0.186700 0.615578 0.208300
195 0.413314 0.146800 0.616822 0.196400
196 0.409600 0.145720 0.633601 0.205600
197 0.451619 0.160000 0.635755 0.206500
198 0.435021 0.153900 0.620861 0.200800
199 0.400606 0.142060 0.621494 0.200200
200 0.391516 0.138520 0.614794 0.196900
201 0.365059 0.129860 0.619348 0.200800
202 0.402264 0.142960 0.615285 0.199200
203 0.410062 0.145700 0.630870 0.208300
real 39m10.866s
user 54m16.480s
sys 56m10.956s
}};
***C[0.01]-P-C[0.01]-P-F[0.1]-F[0.1]
#pre{{
############# training condition ############
500 epoch training. input_size: (28, 28) , minibatch_size: 100
, learning_rate: 0.01 , momentum: 0.9 , weight_decay: 0.0005
#############################################
########## architecture of the CNN #########
layer1 - Convolution
< kernel_size: (50, 2, 2) , stride: (1, 1) , act_func: ReLU
, border_mode: valid , out_size: (50, 27, 27) , iniW_σ: 0.010000 >
layer2 - Convolution
< kernel_size: (50, 2, 2) , stride: (1, 1) , act_func: ReLU
, border_mode: valid , out_size: (50, 26, 26) , iniW_σ: 0.010000 >
layer3 - Pooling
< downscale: (4, 4) , stride: None , out_size: (50, 6, 6) >
layer4 - fully-connected
< number_of_units: 1000 , drop_rate: 0.8 , act_func: ReLU , iniW_σ: 0.100000 >
layer5 - fully-connected
< number_of_units: 10 , drop_rate: 0.5 , act_func: softmax , iniW_σ: 0.100000 >
###########################################
epoch XE(L) error(L) XE(T) error(T)
1 2.172126 0.806540 2.027054 0.733900
2 2.035170 0.747220 1.983551 0.720100
3 1.988547 0.728440 1.891935 0.679600
4 1.929771 0.706340 1.841547 0.665000
5 1.854730 0.676500 1.812250 0.662500
6 1.780028 0.643160 1.674585 0.600200
7 1.699687 0.617240 1.555386 0.559100
8 1.654096 0.600640 1.525662 0.546600
9 1.611351 0.580120 1.495098 0.536100
10 1.573158 0.564400 1.458604 0.524000
11 1.540977 0.554460 1.446038 0.515000
12 1.560106 0.562840 1.413125 0.501200
13 1.514473 0.544700 1.421622 0.507300
14 1.496816 0.538960 1.378435 0.493700
15 1.455022 0.524800 1.385545 0.497000
16 1.441190 0.517100 1.352296 0.480000
17 1.435965 0.514920 1.292696 0.454400
18 1.412946 0.508140 1.295221 0.457200
19 1.414605 0.507120 1.279375 0.446100
20 1.413595 0.507600 1.291859 0.452400
21 1.396855 0.500060 1.294704 0.453100
22 1.382536 0.491640 1.307064 0.463800
23 1.400341 0.500040 1.308915 0.473800
24 1.339820 0.478880 1.234676 0.422900
25 1.357593 0.480360 1.235378 0.435700
26 1.346123 0.482560 1.219342 0.421800
27 1.315418 0.467780 1.202096 0.416400
28 1.327973 0.472600 1.226551 0.429700
29 1.290800 0.457280 1.187783 0.414900
30 1.265084 0.449300 1.189499 0.414900
31 1.269951 0.450260 1.162423 0.402100
32 1.229706 0.437520 1.139199 0.397700
33 1.256525 0.445140 1.143627 0.403000
34 1.228535 0.434140 1.112442 0.382400
35 1.205356 0.427200 1.081512 0.373300
36 1.227064 0.431580 1.135518 0.388100
37 1.203071 0.426680 1.066740 0.367700
38 1.174142 0.416060 1.093697 0.378100
39 1.206996 0.425060 1.083049 0.372200
40 1.177429 0.414880 1.070377 0.374600
41 1.181964 0.419780 1.078487 0.372800
42 1.154330 0.406840 1.092380 0.377600
43 1.156735 0.409060 1.095766 0.374700
44 1.136484 0.401520 1.042007 0.359600
45 1.165397 0.412300 1.030762 0.355600
46 1.111314 0.391160 1.003724 0.347200
47 1.117155 0.395880 1.034887 0.354800
48 1.160369 0.408580 1.037490 0.357200
49 1.111658 0.392120 1.008933 0.342800
50 1.089677 0.382500 1.009640 0.346600
51 1.125150 0.396400 1.015790 0.354300
52 1.086868 0.382360 1.017911 0.348000
53 1.079408 0.380480 0.987813 0.340500
54 1.101633 0.387140 0.978916 0.338000
55 1.103292 0.388400 1.017744 0.349900
56 1.055742 0.371920 0.999023 0.349200
57 1.095965 0.386880 0.991891 0.340600
58 1.075035 0.376700 1.039214 0.357900
59 1.105706 0.389480 0.962715 0.332700
60 1.058415 0.373280 0.946698 0.324500
61 1.046448 0.366340 0.974094 0.334900
62 1.060476 0.372560 0.945780 0.327400
63 1.017735 0.360360 0.925158 0.318500
64 1.049510 0.369400 0.968878 0.335600
65 1.041503 0.367640 0.958483 0.328700
66 1.079477 0.377540 0.952297 0.328200
67 1.024882 0.361820 0.931482 0.313400
68 1.014564 0.357060 0.912452 0.309900
69 1.090831 0.383480 0.928553 0.311700
70 1.057769 0.370600 0.953148 0.328400
71 1.048245 0.370240 0.910431 0.311900
72 1.061504 0.372140 0.933928 0.324300
73 1.039348 0.365600 0.908720 0.311000
74 1.017609 0.358460 0.909951 0.317000
75 1.016555 0.357520 0.906251 0.312700
76 1.038671 0.366300 0.897406 0.305500
77 0.980130 0.343920 0.895771 0.310800
78 1.013186 0.355400 0.902160 0.316300
79 0.970512 0.341200 0.889874 0.306300
80 0.985979 0.347140 0.877723 0.297400
81 0.985099 0.346940 0.871658 0.296300
82 0.987575 0.346300 0.887024 0.298000
83 0.973258 0.342320 0.872484 0.300300
84 0.979816 0.346480 0.865057 0.291500
85 0.987033 0.344540 0.860676 0.294000
86 0.989011 0.347220 0.881951 0.305300
87 0.983126 0.343620 0.885246 0.303000
88 0.948252 0.331220 0.855348 0.296700
89 1.005005 0.352520 0.883245 0.305200
90 0.958306 0.336200 0.883274 0.301500
91 0.930092 0.326480 0.851001 0.289300
92 0.951223 0.333120 0.848586 0.286800
93 0.950430 0.333220 0.861223 0.296200
94 0.927446 0.325760 0.871467 0.299800
95 0.933366 0.327760 0.874397 0.303900
96 0.913233 0.320140 0.861355 0.292000
97 0.905543 0.316420 0.855736 0.297700
98 0.975903 0.342480 0.874594 0.301200
99 0.911865 0.317940 0.833164 0.290000
100 0.978378 0.343840 0.873178 0.298500
101 0.975264 0.343500 0.855746 0.291100
102 0.936860 0.326420 0.867001 0.300800
103 0.896566 0.316180 0.825730 0.282700
104 0.954344 0.333120 0.864847 0.299100
105 0.943417 0.331460 0.845261 0.291600
106 0.915229 0.321920 0.842217 0.292300
107 0.892967 0.312880 0.834767 0.293200
108 0.923368 0.323500 0.834013 0.278800
109 0.952158 0.331680 0.876165 0.300700
110 0.960994 0.334260 0.835252 0.282000
111 0.871473 0.304960 0.807815 0.276900
112 0.964376 0.338580 0.857544 0.297400
113 0.916684 0.320640 0.878934 0.298600
114 0.892944 0.312360 0.803190 0.276600
115 0.866723 0.303060 0.851757 0.294400
116 0.912606 0.318980 0.829088 0.284500
117 0.891724 0.313700 0.818643 0.280300
118 0.907253 0.318040 0.821649 0.287600
119 0.921999 0.323040 0.829584 0.280100
120 0.900529 0.313660 0.818153 0.279500
121 0.937879 0.328620 0.839500 0.286800
122 0.911937 0.319420 0.832967 0.288600
123 0.902038 0.314560 0.805792 0.275200
124 0.961827 0.335600 0.816764 0.279600
125 0.939841 0.329580 0.829886 0.285800
126 0.891077 0.312300 0.807790 0.271300
127 0.867737 0.303160 0.806412 0.274100
128 0.882219 0.308840 0.796532 0.271600
129 0.911777 0.318700 0.794350 0.271700
130 0.857981 0.299900 0.796901 0.272500
131 0.834586 0.292920 0.790472 0.270600
132 0.905096 0.316020 0.816004 0.276700
133 0.952341 0.333660 0.822376 0.276400
134 0.849635 0.299660 0.788939 0.268100
135 0.877486 0.307080 0.781232 0.264600
136 0.900104 0.315540 0.810521 0.275700
137 0.875975 0.306880 0.795411 0.272500
138 0.891001 0.312740 0.788374 0.270100
139 0.913581 0.317280 0.821446 0.279300
140 0.835725 0.293100 0.778462 0.265000
141 0.884017 0.308620 0.799846 0.274100
142 0.870504 0.304900 0.807875 0.276700
143 0.874864 0.306060 0.823310 0.283600
144 0.871382 0.303760 0.802308 0.271100
145 0.904468 0.314620 0.797340 0.273400
146 0.862642 0.302380 0.808056 0.278400
147 0.827082 0.288960 0.787023 0.269100
148 0.907789 0.317740 0.800022 0.273600
149 0.910789 0.319600 0.818200 0.280400
150 0.866458 0.302640 0.808086 0.277100
151 0.920660 0.322460 0.782074 0.260800
152 0.885272 0.309180 0.780229 0.267400
153 0.859376 0.298380 0.800715 0.273600
154 0.801656 0.281420 0.769792 0.262200
155 0.848805 0.295900 0.770786 0.261100
156 0.813184 0.284280 0.773820 0.264500
157 0.866589 0.302500 0.772845 0.258900
158 0.843960 0.293660 0.766434 0.256700
159 0.805899 0.280620 0.754112 0.255900
160 0.809272 0.285080 0.750416 0.254300
161 0.836713 0.293200 0.776605 0.261800
162 0.819470 0.285040 0.764839 0.264300
163 0.796412 0.279000 0.744179 0.248900
164 0.805700 0.282580 0.758193 0.257200
165 0.863856 0.301800 0.764254 0.261500
166 0.833535 0.292280 0.760476 0.252900
167 0.886558 0.307960 0.773528 0.263100
168 0.787533 0.274660 0.764123 0.260300
169 0.806697 0.283460 0.778052 0.267300
170 0.792651 0.277600 0.745825 0.254400
171 0.791425 0.277540 0.737426 0.252800
172 0.890823 0.309980 0.779712 0.267800
173 0.855596 0.299080 0.764133 0.263100
174 0.915325 0.317660 0.799692 0.272900
175 0.788956 0.274880 0.753961 0.260100
176 0.774234 0.268720 0.765237 0.264900
177 0.820336 0.286560 0.735296 0.253000
178 0.827865 0.289300 0.759258 0.261800
179 0.831426 0.289600 0.822596 0.282300
180 0.865377 0.299880 0.758460 0.257600
181 0.796654 0.277280 0.741971 0.254100
182 0.765785 0.265420 0.730033 0.248200
183 0.844058 0.294440 0.772839 0.264700
184 0.831631 0.290800 0.751584 0.253000
185 0.806699 0.280100 0.744356 0.256600
186 0.828892 0.291180 0.769724 0.262600
187 0.837089 0.292120 0.757070 0.260200
188 0.769261 0.268180 0.746825 0.254200
189 0.769149 0.269240 0.732405 0.249100
190 0.760422 0.265360 0.735797 0.245100
191 0.857432 0.298640 0.785864 0.268900
192 0.818788 0.284740 0.759145 0.262900
193 0.767524 0.268940 0.725755 0.248500
194 0.803127 0.281080 0.749890 0.253200
195 0.816030 0.286400 0.758032 0.258100
196 0.860487 0.299180 0.775654 0.268400
197 0.754009 0.263060 0.744933 0.253700
198 0.843668 0.293780 0.766283 0.258300
199 0.800195 0.279700 0.758428 0.259300
200 0.801760 0.279920 0.749372 0.253300
}};
***C[0.1]-P-C[0.1]-P-F[0.1]-F[0.1]
#pre{{
############# training condition ############
500 epoch training. input_size: (28, 28) , minibatch_size: 100
, learning_rate: 0.01 , momentum: 0.9 , weight_decay: 0.0005
#############################################
########## architecture of the CNN #########
layer1 - Convolution
< kernel_size: (50, 2, 2) , stride: (1, 1) , act_func: ReLU
, border_mode: valid , out_size: (50, 27, 27) , iniW_σ: 0.100000 >
layer2 - Convolution
< kernel_size: (50, 2, 2) , stride: (1, 1) , act_func: ReLU
, border_mode: valid , out_size: (50, 26, 26) , iniW_σ: 0.100000 >
layer3 - Pooling
< downscale: (4, 4) , stride: None , out_size: (50, 6, 6) >
layer4 - fully-connected
< number_of_units: 1000 , drop_rate: 0.8 , act_func: ReLU , iniW_σ: 0.100000 >
layer5 - fully-connected
< number_of_units: 10 , drop_rate: 0.5 , act_func: softmax , iniW_σ: 0.100000 >
###########################################
epoch XE(L) error(L) XE(T) error(T)
1 2.469696 0.903120 2.302834 0.900000
2 2.302945 0.900160 2.302872 0.900000
3 2.302946 0.901760 2.302638 0.900000
4 2.302876 0.903000 2.302803 0.900000
5 2.302969 0.902140 2.302640 0.900000
6 2.302881 0.899880 2.302644 0.900000
7 2.302930 0.900580 2.302656 0.900000
8 2.302901 0.903200 2.302741 0.900000
9 2.302881 0.900880 2.302691 0.900000
10 2.302883 0.900360 2.302758 0.900000
11 2.302919 0.899480 2.302720 0.900000
12 2.302891 0.900400 2.302767 0.900000
13 2.302844 0.900360 2.302721 0.900000
14 2.302869 0.900680 2.302830 0.900000
15 2.302972 0.901260 2.302669 0.900000
16 2.302851 0.901600 2.302821 0.900000
17 2.302944 0.902180 2.302710 0.900000
18 2.302902 0.902100 2.302707 0.900000
19 2.302866 0.900760 2.302859 0.900000
20 2.302912 0.902300 2.302680 0.900000
21 2.302888 0.901260 2.302673 0.900000
22 2.302860 0.900320 2.302694 0.900000
23 2.302874 0.900360 2.302740 0.900000
24 2.302860 0.901140 2.302673 0.900000
25 2.302795 0.901300 2.302747 0.900000
26 2.302883 0.900760 2.302735 0.900000
27 2.302845 0.899840 2.302753 0.900000
28 2.302856 0.900980 2.302781 0.900000
29 2.302876 0.900140 2.302671 0.900000
30 2.302877 0.899980 2.302692 0.900000
31 2.302934 0.903280 2.302656 0.900000
32 2.302863 0.900760 2.302713 0.900000
33 2.302915 0.902880 2.302714 0.900000
34 2.302928 0.901880 2.302650 0.900000
35 2.302866 0.901380 2.302725 0.900000
36 2.302825 0.900720 2.302807 0.900000
37 2.302892 0.902460 2.302619 0.900000
38 2.302845 0.901600 2.302765 0.900000
39 2.302914 0.902720 2.302743 0.900000
40 2.302921 0.901600 2.302677 0.900000
41 2.302836 0.900940 2.302792 0.900000
42 2.302791 0.899040 2.302865 0.900000
43 2.302911 0.901320 2.302741 0.900000
44 2.302819 0.899380 2.302865 0.900000
45 2.302945 0.901820 2.302727 0.900000
46 2.302858 0.898500 2.302843 0.900000
47 2.302933 0.902640 2.302636 0.900000
48 2.302869 0.902680 2.302715 0.900000
49 2.302831 0.901920 2.302749 0.900000
50 2.302902 0.900260 2.302702 0.900000
51 2.302915 0.901580 2.302758 0.900000
52 2.302837 0.900320 2.302890 0.900000
53 2.302885 0.900160 2.302726 0.900000
54 2.302865 0.902600 2.302754 0.900000
55 2.302849 0.900520 2.302711 0.900000
56 2.302866 0.903040 2.302749 0.900000
57 2.302907 0.902060 2.302714 0.900000
58 2.302898 0.904120 2.302728 0.900000
59 2.302899 0.900840 2.302761 0.900000
60 2.302974 0.901780 2.302660 0.900000
61 2.302901 0.899720 2.302698 0.900000
62 2.302916 0.902500 2.302723 0.900000
63 2.302907 0.901860 2.302665 0.900000
64 2.302834 0.902040 2.302671 0.900000
65 2.302908 0.903960 2.302636 0.900000
66 2.302846 0.900720 2.302781 0.900000
67 2.302897 0.901640 2.302807 0.900000
68 2.302890 0.901000 2.302656 0.900000
69 2.302878 0.899900 2.302637 0.900000
70 2.302908 0.902440 2.302684 0.900000
71 2.302904 0.901340 2.302690 0.900000
72 2.302887 0.902000 2.302632 0.900000
73 2.302851 0.901180 2.302742 0.900000
74 2.302876 0.901320 2.302730 0.900000
75 2.302831 0.899560 2.302755 0.900000
76 2.302867 0.902080 2.302701 0.900000
77 2.302783 0.901000 2.303005 0.900000
78 2.302869 0.899620 2.302806 0.900000
79 2.302857 0.899900 2.302774 0.900000
80 2.302936 0.901880 2.302664 0.900000
81 2.302874 0.900220 2.302776 0.900000
82 2.302942 0.902880 2.302651 0.900000
83 2.302881 0.902540 2.302710 0.900000
84 2.302884 0.900680 2.302822 0.900000
85 2.302892 0.901160 2.302698 0.900000
86 2.302854 0.901020 2.302746 0.900000
87 2.302849 0.902120 2.302697 0.900000
88 2.302851 0.901900 2.302635 0.900000
89 2.302851 0.900780 2.302721 0.900000
90 2.302825 0.900340 2.302800 0.900000
91 2.302910 0.901940 2.302648 0.900000
92 2.302930 0.903180 2.302644 0.900000
93 2.302940 0.902940 2.302687 0.900000
94 2.302902 0.902080 2.302719 0.900000
95 2.302903 0.902620 2.302718 0.900000
96 2.302908 0.900440 2.302715 0.900000
97 2.302918 0.901280 2.302743 0.900000
98 2.302900 0.901520 2.302664 0.900000
99 2.302875 0.900100 2.302705 0.900000
100 2.302955 0.902540 2.302623 0.900000
101 2.302834 0.900540 2.302717 0.900000
102 2.302931 0.901340 2.302675 0.900000
103 2.302928 0.902440 2.302671 0.900000
104 2.302883 0.902920 2.302754 0.900000
105 2.302936 0.903300 2.302649 0.900000
106 2.302850 0.900940 2.302701 0.900000
107 2.302882 0.901440 2.302749 0.900000
108 2.302863 0.901020 2.302780 0.900000
109 2.302911 0.902860 2.302698 0.900000
110 2.302854 0.900220 2.302695 0.900000
111 2.302906 0.901480 2.302693 0.900000
112 2.302827 0.901020 2.302732 0.900000
113 2.302868 0.901000 2.302690 0.900000
114 2.302870 0.900520 2.302764 0.900000
115 2.302846 0.900280 2.302795 0.900000
116 2.302924 0.901420 2.302691 0.900000
117 2.302839 0.900840 2.302810 0.900000
118 2.302927 0.900620 2.302779 0.900000
119 2.302907 0.901980 2.302638 0.900000
120 2.302848 0.901200 2.302821 0.900000
121 2.302883 0.901360 2.302833 0.900000
122 2.302876 0.901760 2.302695 0.900000
123 2.302830 0.899780 2.302757 0.900000
124 2.302893 0.900940 2.302753 0.900000
125 2.302882 0.902880 2.302826 0.900000
126 2.302941 0.902640 2.302663 0.900000
127 2.302845 0.900620 2.302707 0.900000
128 2.302872 0.901720 2.302686 0.900000
129 2.302925 0.900740 2.302679 0.900000
130 2.302902 0.904020 2.302697 0.900000
131 2.302914 0.902560 2.302687 0.900000
132 2.302895 0.899900 2.302668 0.900000
133 2.302915 0.901800 2.302649 0.900000
134 2.302854 0.901300 2.302746 0.900000
135 2.302883 0.900820 2.302828 0.900000
136 2.302909 0.900720 2.302699 0.900000
137 2.302908 0.900320 2.302631 0.900000
138 2.302826 0.899820 2.302702 0.900000
139 2.302849 0.900100 2.302845 0.900000
140 2.302842 0.901300 2.302676 0.900000
141 2.302844 0.900100 2.302851 0.900000
142 2.302861 0.899400 2.302719 0.900000
143 2.302889 0.900700 2.302653 0.900000
144 2.302943 0.901700 2.302664 0.900000
145 2.302849 0.901600 2.302750 0.900000
146 2.302887 0.901880 2.302637 0.900000
147 2.302866 0.901120 2.302795 0.900000
148 2.302899 0.901120 2.302794 0.900000
149 2.302916 0.901800 2.302687 0.900000
150 2.302869 0.901600 2.302766 0.900000
151 2.302911 0.901280 2.302631 0.900000
152 2.302876 0.901760 2.302697 0.900000
153 2.302907 0.901040 2.302706 0.900000
154 2.302895 0.902340 2.302698 0.900000
155 2.302866 0.901980 2.302753 0.900000
156 2.302911 0.901380 2.302764 0.900000
157 2.302881 0.901400 2.302782 0.900000
158 2.302922 0.903480 2.302663 0.900000
159 2.302916 0.902040 2.302689 0.900000
160 2.302912 0.903460 2.302759 0.900000
161 2.302892 0.902000 2.302633 0.900000
162 2.302868 0.902920 2.302685 0.900000
163 2.302875 0.902540 2.302739 0.900000
164 2.302891 0.901720 2.302747 0.900000
165 2.302893 0.903380 2.302692 0.900000
166 2.302901 0.902320 2.302729 0.900000
167 2.302912 0.901540 2.302785 0.900000
168 2.302803 0.900920 2.302877 0.900000
169 2.302902 0.903200 2.302750 0.900000
170 2.302833 0.901080 2.302715 0.900000
171 2.302826 0.900980 2.302698 0.900000
172 2.302876 0.898880 2.302729 0.900000
173 2.302833 0.900020 2.302843 0.900000
174 2.302900 0.900860 2.302693 0.900000
175 2.302847 0.900600 2.302878 0.900000
176 2.302976 0.902760 2.302691 0.900000
177 2.302902 0.902420 2.302791 0.900000
178 2.302938 0.904280 2.302684 0.900000
179 2.302875 0.900340 2.302723 0.900000
180 2.302863 0.900860 2.302689 0.900000
181 2.302883 0.899860 2.302677 0.900000
182 2.302908 0.902560 2.302744 0.900000
183 2.302885 0.901200 2.302707 0.900000
184 2.302864 0.900340 2.302755 0.900000
185 2.302864 0.900280 2.302783 0.900000
186 2.302891 0.900180 2.302772 0.900000
187 2.302919 0.902740 2.302653 0.900000
188 2.302705 0.900640 2.303075 0.900000
189 2.303016 0.902080 2.302681 0.900000
190 2.302799 0.899260 2.302814 0.900000
191 2.302957 0.902220 2.302677 0.900000
192 2.302808 0.900160 2.302747 0.900000
193 2.302941 0.899640 2.302764 0.900000
194 2.302920 0.901400 2.302660 0.900000
195 2.302916 0.902700 2.302655 0.900000
196 2.302792 0.900780 2.302863 0.900000
197 2.302958 0.902280 2.302696 0.900000
198 2.302820 0.899840 2.302737 0.900000
199 2.302860 0.902080 2.302677 0.900000
200 2.302882 0.902160 2.302703 0.900000
201 2.302827 0.898820 2.302696 0.900000
202 2.302886 0.901280 2.302664 0.900000
203 2.302820 0.902560 2.302757 0.900000
204 2.302895 0.903140 2.302790 0.900000
205 2.302906 0.900200 2.302672 0.900000
206 2.302863 0.900160 2.302667 0.900000
207 2.302852 0.898640 2.302745 0.900000
208 2.302830 0.902600 2.302786 0.900000
209 2.302871 0.900420 2.302814 0.900000
210 2.302969 0.903040 2.302597 0.900000
211 2.302858 0.899980 2.302673 0.900000
212 2.302862 0.899840 2.302731 0.900000
213 2.302937 0.901440 2.302684 0.900000
214 2.302936 0.901320 2.302697 0.900000
215 2.302861 0.901240 2.302782 0.900000
216 2.302873 0.900540 2.302659 0.900000
217 2.302881 0.903020 2.302690 0.900000
218 2.302816 0.901500 2.302875 0.900000
219 2.302922 0.902760 2.302714 0.900000
220 2.302896 0.902720 2.302733 0.900000
221 2.302908 0.901320 2.302749 0.900000
222 2.302837 0.901660 2.302840 0.900000
223 2.302945 0.901600 2.302691 0.900000
224 2.302934 0.903160 2.302655 0.900000
real 42m3.781s
user 58m0.592s
sys 60m36.236s
}};
*ILSVRC2010 - 10クラス [#l3dbaa07]
**ex20151205-2 【学習係数ηの下げ方をいろいろ試す】 [#ex201512052]
-[[#ex201512052stasis579]]では、バリデーションの最低誤識別率がN epoch連続で下がらなかった場合にηを0.1倍している。Nは5から始まり、ηを下げる毎に5,7,9と大きくする。
-毎epoch下げる方法では、停滞時に0.1倍する方法よりも良い結果は得られなかった。[[岡田さんが行ったとき>m/2015/okada/diary/2015-12-03#tee792ca]]は良い結果が出ているので、今回の条件では学習の進行とηの減少がうまくかみ合わなかったのだろう。調整が難しそう。
***学習停滞時にηを下げる(5,7,9) [#ex201512052stasis579]
#pre{{
ex201512051600 eta*0.1_in_stasis5,7,9
############# training condition ############
100 epoch training. input_size: (227, 227) , minibatch_size: 100
, learning_rate: 0.01 , momentum: 0.9 , weight_decay: 0.0005
#############################################
########## architecture of the CNN #########
layer1 - Convolution
< kernel_size: (100, 11, 11) , stride: (4, 4) , act_func: ReLU
, border_mode: valid , out_size: (100, 55, 55) , iniW_σ: 0.010000 >
layer2 - Pooling
< downscale: (2, 2) , stride: None , out_size: (100, 27, 27) >
layer3 - Convolution
< kernel_size: (200, 5, 5) , stride: (1, 1) , act_func: ReLU
, border_mode: same , out_size: (200, 27, 27) , iniW_σ: 0.010000 >
layer4 - Pooling
< downscale: (2, 2) , stride: None , out_size: (200, 13, 13) >
layer5 - Convolution
< kernel_size: (200, 3, 3) , stride: (1, 1) , act_func: ReLU
, border_mode: same , out_size: (200, 13, 13) , iniW_σ: 0.010000 >
layer6 - Pooling
< downscale: (2, 2) , stride: None , out_size: (200, 6, 6) >
layer7 - fully-connected
< number_of_units: 4000 , drop_rate: 1.0 , act_func: ReLU , iniW_σ: 0.010000 >
layer8 - fully-connected
< number_of_units: 10 , drop_rate: 0.5 , act_func: softmax , iniW_σ: 0.010000 >
###########################################
##### CNN initialized #####
##### start training #####
epoch XE(L) error(L) XE(V) error(V) top-5(V) LE(ε)
1 2.277367 0.858319 2.251666 0.836000 0.428000 0.010000
2 1.947574 0.709496 1.879266 0.686000 0.170000 0.010000
3 1.722490 0.607395 1.663454 0.570000 0.120000 0.010000
4 1.535354 0.525966 1.477183 0.518000 0.082000 0.010000
5 1.398474 0.477227 1.365852 0.466000 0.064000 0.010000
6 1.278411 0.431765 1.285098 0.466000 0.056000 0.010000
7 1.183516 0.401849 1.216781 0.418000 0.058000 0.010000
8 1.103606 0.367731 1.100320 0.372000 0.034000 0.010000
9 1.041006 0.345882 1.085061 0.392000 0.044000 0.010000
10 0.994406 0.335210 1.086543 0.382000 0.054000 0.010000
11 0.943029 0.318067 1.052583 0.362000 0.040000 0.010000
12 0.917502 0.307059 1.090705 0.376000 0.060000 0.010000
13 0.853778 0.290084 0.947194 0.316000 0.038000 0.010000
14 0.813730 0.275630 1.032880 0.320000 0.050000 0.010000
15 0.779904 0.263361 0.889893 0.294000 0.040000 0.010000
16 0.738899 0.253445 0.896218 0.308000 0.038000 0.010000
17 0.709131 0.237059 0.903660 0.314000 0.042000 0.010000
18 0.658844 0.224790 0.973461 0.332000 0.036000 0.010000
19 0.633154 0.213529 0.982704 0.342000 0.048000 0.010000
20 0.613156 0.203697 0.854572 0.286000 0.036000 0.010000
21 0.569365 0.195462 0.836357 0.316000 0.034000 0.010000
22 0.572420 0.196387 0.830615 0.260000 0.030000 0.010000
23 0.512127 0.177311 0.817538 0.278000 0.038000 0.010000
24 0.480103 0.162773 0.847216 0.266000 0.040000 0.010000
25 0.451453 0.152689 0.872383 0.260000 0.034000 0.010000
26 0.449348 0.152521 0.888524 0.262000 0.026000 0.010000
27 0.423916 0.144202 0.858103 0.248000 0.032000 0.010000
28 0.388746 0.133025 0.845194 0.246000 0.030000 0.010000
29 0.351130 0.121429 0.791058 0.242000 0.024000 0.010000
30 0.333022 0.112773 0.827974 0.272000 0.022000 0.010000
31 0.315010 0.109580 0.917370 0.276000 0.032000 0.010000
32 0.305722 0.106723 0.853677 0.244000 0.026000 0.010000
33 0.279666 0.097143 0.888657 0.258000 0.032000 0.010000
34 0.258500 0.086891 0.831414 0.252000 0.036000 0.010000
35 0.193057 0.061933 0.806545 0.226000 0.026000 0.001000
36 0.153609 0.050000 0.812244 0.238000 0.026000 0.001000
37 0.137934 0.042437 0.816455 0.226000 0.026000 0.001000
38 0.133661 0.041345 0.836213 0.228000 0.034000 0.001000
39 0.116838 0.035210 0.842869 0.220000 0.030000 0.001000
40 0.121436 0.036555 0.840142 0.224000 0.030000 0.001000
41 0.112238 0.034034 0.846096 0.226000 0.026000 0.001000
42 0.098387 0.028571 0.863834 0.220000 0.030000 0.001000
43 0.099678 0.030168 0.850278 0.220000 0.032000 0.001000
44 0.093421 0.028655 0.863470 0.212000 0.030000 0.001000
45 0.091027 0.029244 0.866952 0.218000 0.030000 0.001000
46 0.093821 0.031008 0.863583 0.212000 0.030000 0.001000
47 0.084964 0.026387 0.878371 0.220000 0.026000 0.001000
48 0.081341 0.024958 0.931855 0.234000 0.032000 0.001000
49 0.086145 0.027311 0.894826 0.220000 0.034000 0.001000
50 0.076297 0.022101 0.883321 0.220000 0.030000 0.001000
51 0.075838 0.022353 0.893859 0.216000 0.030000 0.001000
52 0.074963 0.023782 0.903655 0.216000 0.032000 0.000100
53 0.072232 0.022269 0.900654 0.216000 0.032000 0.000100
54 0.069701 0.020756 0.900656 0.220000 0.032000 0.000100
55 0.066896 0.018655 0.901350 0.222000 0.030000 0.000100
56 0.067094 0.019916 0.899645 0.220000 0.028000 0.000100
57 0.070243 0.019328 0.900114 0.222000 0.030000 0.000100
58 0.067378 0.020504 0.897133 0.220000 0.032000 0.000100
59 0.070497 0.021597 0.894566 0.216000 0.032000 0.000100
60 0.069522 0.020588 0.897999 0.216000 0.030000 0.000100
61 0.072902 0.021261 0.898477 0.216000 0.030000 0.000010
62 0.068760 0.021429 0.898896 0.216000 0.030000 0.000010
63 0.070049 0.021429 0.898424 0.216000 0.030000 0.000010
64 0.064309 0.018824 0.897976 0.216000 0.030000 0.000010
65 0.062635 0.017983 0.898582 0.216000 0.032000 0.000010
66 0.065354 0.019412 0.898488 0.216000 0.032000 0.000010
67 0.068760 0.021092 0.898843 0.216000 0.030000 0.000010
68 0.069648 0.020252 0.898074 0.216000 0.032000 0.000010
69 0.068220 0.019832 0.897917 0.216000 0.032000 0.000010
70 0.062939 0.018235 0.896810 0.216000 0.032000 0.000010
71 0.072348 0.021261 0.897577 0.216000 0.032000 0.000010
72 0.068491 0.020084 0.898607 0.218000 0.030000 0.000010
73 0.073683 0.021345 0.898009 0.218000 0.032000 0.000010
74 0.070746 0.020420 0.898094 0.218000 0.032000 0.000010
75 0.070437 0.021429 0.898307 0.218000 0.032000 0.000010
Terminated
real 86m40.138s
user 131m43.860s
sys 56m52.428s
}};
***1epoch終了毎にηを0.95倍 [#ex201512052095]
#pre{{
ex201512051737 eta*0.95/epoch
############# training condition ############
100 epoch training. input_size: (227, 227) , minibatch_size: 100
, learning_rate: 0.01 , momentum: 0.9 , weight_decay: 0.0005
#############################################
########## architecture of the CNN #########
layer1 - Convolution
< kernel_size: (100, 11, 11) , stride: (4, 4) , act_func: ReLU
, border_mode: valid , out_size: (100, 55, 55) , iniW_σ: 0.010000 >
layer2 - Pooling
< downscale: (2, 2) , stride: None , out_size: (100, 27, 27) >
layer3 - Convolution
< kernel_size: (200, 5, 5) , stride: (1, 1) , act_func: ReLU
, border_mode: same , out_size: (200, 27, 27) , iniW_σ: 0.010000 >
layer4 - Pooling
< downscale: (2, 2) , stride: None , out_size: (200, 13, 13) >
layer5 - Convolution
< kernel_size: (200, 3, 3) , stride: (1, 1) , act_func: ReLU
, border_mode: same , out_size: (200, 13, 13) , iniW_σ: 0.010000 >
layer6 - Pooling
< downscale: (2, 2) , stride: None , out_size: (200, 6, 6) >
layer7 - fully-connected
< number_of_units: 4000 , drop_rate: 1.0 , act_func: ReLU , iniW_σ: 0.010000 >
layer8 - fully-connected
< number_of_units: 10 , drop_rate: 0.5 , act_func: softmax , iniW_σ: 0.010000 >
###########################################
##### CNN initialized #####
##### start training #####
epoch XE(L) error(L) XE(V) error(V) top-5(V) LE(ε)
1 2.277352 0.858235 2.251584 0.836000 0.430000 0.010000
2 1.951836 0.711597 1.881063 0.684000 0.168000 0.009500
3 1.728413 0.610756 1.695229 0.596000 0.128000 0.009025
4 1.545813 0.534538 1.509981 0.504000 0.098000 0.008574
5 1.405563 0.480252 1.373952 0.462000 0.058000 0.008145
6 1.296154 0.437731 1.296158 0.460000 0.064000 0.007738
7 1.208430 0.407143 1.234515 0.408000 0.056000 0.007351
8 1.127124 0.376555 1.145219 0.404000 0.040000 0.006983
9 1.076626 0.359160 1.081067 0.376000 0.038000 0.006634
10 1.029072 0.344874 1.064646 0.384000 0.048000 0.006302
11 0.982807 0.330336 1.006307 0.346000 0.036000 0.005987
12 0.946189 0.313361 1.031065 0.354000 0.046000 0.005688
13 0.899290 0.298655 0.985757 0.342000 0.040000 0.005404
14 0.860212 0.287899 0.957702 0.320000 0.036000 0.005133
15 0.825233 0.274202 0.953047 0.338000 0.038000 0.004877
16 0.799453 0.267815 0.914362 0.320000 0.032000 0.004633
17 0.770583 0.257899 0.913052 0.314000 0.034000 0.004401
18 0.730774 0.246134 0.857517 0.298000 0.032000 0.004181
19 0.709381 0.237647 0.878458 0.298000 0.032000 0.003972
20 0.701334 0.237143 0.876736 0.304000 0.032000 0.003774
21 0.657359 0.222689 0.919825 0.358000 0.032000 0.003585
22 0.653392 0.220000 0.891270 0.312000 0.034000 0.003406
23 0.611222 0.205294 0.822501 0.276000 0.034000 0.003235
24 0.586018 0.200000 0.793090 0.262000 0.030000 0.003074
25 0.583927 0.196807 0.814655 0.280000 0.034000 0.002920
26 0.559830 0.188403 0.835614 0.278000 0.034000 0.002774
27 0.547732 0.183277 0.802821 0.270000 0.040000 0.002635
28 0.517970 0.175294 0.810674 0.286000 0.028000 0.002503
29 0.503597 0.172941 0.766553 0.266000 0.030000 0.002378
30 0.503102 0.169916 0.819078 0.286000 0.038000 0.002259
31 0.475812 0.162353 0.801845 0.262000 0.026000 0.002146
32 0.459279 0.155294 0.823232 0.272000 0.032000 0.002039
33 0.442686 0.152185 0.766325 0.264000 0.030000 0.001937
34 0.431171 0.144790 0.789724 0.248000 0.032000 0.001840
35 0.418669 0.140168 0.767989 0.256000 0.030000 0.001748
36 0.401216 0.134034 0.778885 0.278000 0.028000 0.001661
37 0.388688 0.134958 0.806615 0.272000 0.034000 0.001578
38 0.379779 0.125294 0.800105 0.262000 0.032000 0.001499
39 0.366716 0.121513 0.810097 0.270000 0.028000 0.001424
40 0.358762 0.118655 0.811743 0.264000 0.030000 0.001353
41 0.340084 0.112101 0.847037 0.276000 0.032000 0.001285
42 0.320878 0.107899 0.822449 0.262000 0.034000 0.001221
43 0.323759 0.108571 0.814669 0.258000 0.030000 0.001160
44 0.309059 0.101849 0.796212 0.260000 0.024000 0.001102
45 0.314341 0.105798 0.777682 0.248000 0.032000 0.001047
46 0.307798 0.102101 0.822097 0.278000 0.032000 0.000994
47 0.285250 0.091345 0.807610 0.260000 0.030000 0.000945
48 0.275792 0.090084 0.796285 0.278000 0.032000 0.000897
49 0.273662 0.089580 0.820993 0.256000 0.030000 0.000853
50 0.264182 0.085126 0.816834 0.252000 0.028000 0.000810
51 0.256567 0.085294 0.803669 0.266000 0.030000 0.000769
52 0.251788 0.081765 0.833863 0.268000 0.030000 0.000731
53 0.249871 0.083193 0.821690 0.268000 0.028000 0.000694
54 0.236485 0.076723 0.822134 0.272000 0.028000 0.000660
55 0.234667 0.077983 0.828458 0.258000 0.032000 0.000627
56 0.232844 0.075966 0.856038 0.258000 0.028000 0.000595
57 0.224520 0.070420 0.820648 0.268000 0.026000 0.000566
58 0.210911 0.068487 0.821500 0.252000 0.028000 0.000537
59 0.222944 0.074286 0.825125 0.264000 0.032000 0.000510
60 0.218585 0.070756 0.823545 0.252000 0.028000 0.000485
61 0.213524 0.069496 0.825661 0.270000 0.026000 0.000461
62 0.204758 0.067479 0.825810 0.264000 0.026000 0.000438
63 0.203523 0.066975 0.861448 0.260000 0.030000 0.000416
64 0.199064 0.062101 0.840673 0.266000 0.028000 0.000395
65 0.194264 0.063445 0.858491 0.258000 0.034000 0.000375
Terminated
real 78m6.722s
user 117m24.212s
sys 49m30.216s
}};
***1epoch終了毎にηを0.96倍 [#ex201512052096]
#pre{{
ex201512051900 eta*0.96/epoch
############# training condition ############
100 epoch training. input_size: (227, 227) , minibatch_size: 100
, learning_rate: 0.01 , momentum: 0.9 , weight_decay: 0.0005
#############################################
########## architecture of the CNN #########
layer1 - Convolution
< kernel_size: (100, 11, 11) , stride: (4, 4) , act_func: ReLU
, border_mode: valid , out_size: (100, 55, 55) , iniW_σ: 0.010000 >
layer2 - Pooling
< downscale: (2, 2) , stride: None , out_size: (100, 27, 27) >
layer3 - Convolution
< kernel_size: (200, 5, 5) , stride: (1, 1) , act_func: ReLU
, border_mode: same , out_size: (200, 27, 27) , iniW_σ: 0.010000 >
layer4 - Pooling
< downscale: (2, 2) , stride: None , out_size: (200, 13, 13) >
layer5 - Convolution
< kernel_size: (200, 3, 3) , stride: (1, 1) , act_func: ReLU
, border_mode: same , out_size: (200, 13, 13) , iniW_σ: 0.010000 >
layer6 - Pooling
< downscale: (2, 2) , stride: None , out_size: (200, 6, 6) >
layer7 - fully-connected
< number_of_units: 4000 , drop_rate: 1.0 , act_func: ReLU , iniW_σ: 0.010000 >
layer8 - fully-connected
< number_of_units: 10 , drop_rate: 0.5 , act_func: softmax , iniW_σ: 0.010000 >
###########################################
##### CNN initialized #####
##### start training #####
epoch XE(L) error(L) XE(V) error(V) top-5(V) LE(ε)
1 2.277385 0.858319 2.251771 0.836000 0.428000 0.010000
2 1.950632 0.710924 1.882485 0.686000 0.170000 0.009600
3 1.727466 0.609832 1.686233 0.584000 0.130000 0.009216
4 1.544110 0.530924 1.506889 0.512000 0.094000 0.008847
5 1.411264 0.484454 1.378533 0.458000 0.062000 0.008493
6 1.288491 0.434034 1.290672 0.460000 0.060000 0.008154
7 1.198105 0.404790 1.239868 0.410000 0.060000 0.007828
8 1.125188 0.374874 1.131541 0.378000 0.038000 0.007514
9 1.063416 0.353950 1.080499 0.386000 0.040000 0.007214
10 1.024589 0.341176 1.049794 0.370000 0.046000 0.006925
11 0.971587 0.326555 1.000571 0.334000 0.036000 0.006648
12 0.936320 0.308151 1.041155 0.350000 0.046000 0.006382
13 0.879419 0.291429 0.972596 0.338000 0.040000 0.006127
14 0.849814 0.285966 0.976599 0.316000 0.046000 0.005882
15 0.814663 0.273361 0.923654 0.326000 0.034000 0.005647
16 0.783737 0.262437 0.880429 0.290000 0.028000 0.005421
17 0.757869 0.253025 0.904275 0.300000 0.042000 0.005204
18 0.714593 0.240420 0.857306 0.298000 0.030000 0.004996
19 0.692006 0.230000 0.893808 0.298000 0.038000 0.004796
20 0.680247 0.229412 0.859658 0.296000 0.032000 0.004604
21 0.636739 0.214790 0.907512 0.356000 0.028000 0.004420
22 0.639926 0.217059 0.905840 0.314000 0.032000 0.004243
23 0.591634 0.200504 0.835064 0.286000 0.036000 0.004073
24 0.562105 0.191092 0.786839 0.266000 0.032000 0.003911
25 0.556643 0.190420 0.812612 0.278000 0.034000 0.003754
26 0.528213 0.178067 0.857938 0.294000 0.034000 0.003604
27 0.522009 0.174370 0.799086 0.260000 0.034000 0.003460
28 0.488593 0.165378 0.793842 0.270000 0.032000 0.003321
29 0.466996 0.159076 0.753675 0.258000 0.032000 0.003189
30 0.459218 0.154874 0.803812 0.258000 0.030000 0.003061
31 0.433331 0.144454 0.807739 0.264000 0.030000 0.002939
32 0.416626 0.140504 0.834518 0.272000 0.030000 0.002821
33 0.396543 0.131933 0.785867 0.256000 0.034000 0.002708
34 0.387115 0.133613 0.813038 0.260000 0.036000 0.002600
35 0.378541 0.128151 0.768092 0.258000 0.028000 0.002496
36 0.348519 0.117731 0.788247 0.280000 0.030000 0.002396
37 0.332313 0.113109 0.863426 0.262000 0.032000 0.002300
38 0.324902 0.109580 0.809045 0.272000 0.024000 0.002208
39 0.303767 0.097815 0.834204 0.254000 0.032000 0.002120
40 0.301304 0.099160 0.829597 0.262000 0.030000 0.002035
41 0.280919 0.092185 0.901386 0.270000 0.032000 0.001954
42 0.252134 0.083782 0.851724 0.242000 0.030000 0.001876
43 0.260497 0.087059 0.891211 0.278000 0.028000 0.001800
44 0.238548 0.080084 0.870616 0.268000 0.028000 0.001728
45 0.242894 0.083109 0.811577 0.232000 0.028000 0.001659
46 0.238153 0.080924 0.879595 0.270000 0.032000 0.001593
47 0.214860 0.069748 0.844704 0.240000 0.034000 0.001529
48 0.201777 0.065966 0.820648 0.250000 0.030000 0.001468
49 0.196741 0.062857 0.883993 0.248000 0.030000 0.001409
50 0.191576 0.063025 0.857124 0.244000 0.032000 0.001353
51 0.183070 0.060168 0.857630 0.244000 0.030000 0.001299
52 0.174940 0.055546 0.895181 0.250000 0.034000 0.001247
53 0.164991 0.053613 0.865979 0.250000 0.030000 0.001197
54 0.160129 0.051092 0.868790 0.250000 0.032000 0.001149
55 0.151970 0.046387 0.865217 0.242000 0.030000 0.001103
56 0.154774 0.050252 0.906302 0.242000 0.032000 0.001059
57 0.149370 0.046807 0.888584 0.254000 0.030000 0.001017
58 0.137899 0.043025 0.872256 0.260000 0.032000 0.000976
59 0.143911 0.046387 0.892266 0.250000 0.030000 0.000937
60 0.134742 0.041933 0.884923 0.264000 0.032000 0.000900
61 0.132981 0.042017 0.882259 0.270000 0.030000 0.000864
62 0.125207 0.039328 0.902916 0.252000 0.032000 0.000829
63 0.122665 0.037395 0.941738 0.264000 0.028000 0.000796
64 0.113982 0.035546 0.895369 0.254000 0.028000 0.000764
65 0.111341 0.034370 0.920459 0.250000 0.028000 0.000733
66 0.104054 0.030840 0.947754 0.242000 0.030000 0.000704
67 0.111851 0.035042 0.950643 0.250000 0.032000 0.000676
68 0.112251 0.034454 0.938128 0.264000 0.030000 0.000649
69 0.106461 0.032689 0.936006 0.248000 0.030000 0.000623
70 0.097583 0.026050 0.958789 0.256000 0.026000 0.000598
71 0.105913 0.032437 0.950073 0.254000 0.030000 0.000574
72 0.099856 0.030504 0.951158 0.244000 0.032000 0.000551
73 0.100141 0.030504 0.931178 0.248000 0.028000 0.000529
74 0.097282 0.030084 0.910496 0.246000 0.028000 0.000508
75 0.099303 0.030336 0.973177 0.252000 0.032000 0.000488
76 0.092883 0.027059 0.956271 0.258000 0.036000 0.000468
77 0.090073 0.026555 0.933896 0.246000 0.034000 0.000449
78 0.084194 0.023361 0.964180 0.254000 0.034000 0.000431
79 0.080961 0.023361 0.947876 0.248000 0.030000 0.000414
80 0.090716 0.027395 0.966286 0.248000 0.032000 0.000398
81 0.081042 0.023782 0.947670 0.256000 0.030000 0.000382
82 0.083237 0.023361 0.960480 0.252000 0.034000 0.000366
83 0.079486 0.022437 0.952245 0.264000 0.032000 0.000352
84 0.078496 0.022521 0.965654 0.254000 0.030000 0.000338
85 0.083684 0.023529 0.941120 0.250000 0.030000 0.000324
86 0.077509 0.022941 0.951007 0.254000 0.030000 0.000311
87 0.076359 0.022101 0.948779 0.252000 0.032000 0.000299
88 0.068875 0.018739 0.952948 0.246000 0.030000 0.000287
89 0.076159 0.021765 0.976507 0.252000 0.034000 0.000275
90 0.075564 0.022773 0.953888 0.252000 0.030000 0.000264
91 0.071741 0.019916 0.966227 0.248000 0.032000 0.000254
92 0.075323 0.021765 0.969359 0.250000 0.032000 0.000244
93 0.070876 0.019916 0.966330 0.250000 0.032000 0.000234
94 0.071192 0.019916 0.976763 0.248000 0.032000 0.000225
95 0.074919 0.022941 0.953077 0.250000 0.032000 0.000216
96 0.071012 0.020252 0.973520 0.256000 0.030000 0.000207
97 0.074735 0.022353 0.960020 0.242000 0.034000 0.000199
98 0.070708 0.018403 0.966824 0.246000 0.032000 0.000191
99 0.069960 0.019328 0.967983 0.242000 0.032000 0.000183
100 0.067620 0.018571 0.965933 0.246000 0.032000 0.000176
#elapsed time after 100 epoch : 7049.65958095 [sec]
real 117m41.328s
user 180m19.676s
sys 72m16.240s
}};
**ex20151204-2【学習係数ηを段階的に下げてみるテスト】 [#ex201512042]
-バリデーションの最低誤識別率が3epoch連続で下がらなかった場合にηを0.1倍する処理を試してみる。
-下げ方が適切かはともかくとして、効果的に働いていると判断できそう。とくにepoch19。
-ηが小さくなるほど次に0.1倍するまでの基準を緩くするのが良さげ。回数制限も設けた方が良いだろうか。
#pre{{
ex201512041734
############# training condition ############
100 epoch training. input_size: (227, 227) , minibatch_size: 100
, learning_rate: 0.01 , momentum: 0.9 , weight_decay: 0.0005
#############################################
########## architecture of the CNN #########
layer1 - Convolution
< kernel_size: (100, 11, 11) , stride: (4, 4) , act_func: ReLU
, border_mode: valid , out_size: (100, 55, 55) , iniW_σ: 0.010000 >
layer2 - Pooling
< downscale: (2, 2) , stride: None , out_size: (100, 27, 27) >
layer3 - Convolution
< kernel_size: (200, 5, 5) , stride: (1, 1) , act_func: ReLU
, border_mode: same , out_size: (200, 27, 27) , iniW_σ: 0.010000 >
layer4 - Pooling
< downscale: (2, 2) , stride: None , out_size: (200, 13, 13) >
layer5 - Convolution
< kernel_size: (200, 3, 3) , stride: (1, 1) , act_func: ReLU
, border_mode: same , out_size: (200, 13, 13) , iniW_σ: 0.010000 >
layer6 - Pooling
< downscale: (2, 2) , stride: None , out_size: (200, 6, 6) >
layer7 - fully-connected
< number_of_units: 4000 , drop_rate: 1.0 , act_func: ReLU , iniW_σ: 0.010000 >
layer8 - fully-connected
< number_of_units: 10 , drop_rate: 0.5 , act_func: softmax , iniW_σ: 0.010000 >
###########################################
##### CNN initialized #####
##### start training #####
epoch XE(L) error(L) XE(V) error(V) top-5(V) LE(ε)
1 2.277364 0.858319 2.251660 0.836000 0.428000 0.0100
2 1.947058 0.709412 1.877985 0.690000 0.170000 0.0100
3 1.724140 0.609244 1.668289 0.574000 0.118000 0.0100
4 1.534381 0.524790 1.480330 0.502000 0.090000 0.0100
5 1.383491 0.470420 1.338253 0.466000 0.056000 0.0100
6 1.286580 0.435630 1.312038 0.476000 0.070000 0.0100
7 1.182439 0.402017 1.206981 0.418000 0.054000 0.0100
8 1.103736 0.368571 1.096689 0.372000 0.038000 0.0100
9 1.041839 0.346050 1.067890 0.380000 0.050000 0.0100
10 0.997058 0.334202 1.074426 0.384000 0.052000 0.0100
11 0.943790 0.318487 1.027471 0.356000 0.032000 0.0100
12 0.910211 0.306303 1.037716 0.340000 0.044000 0.0100
13 0.850385 0.287227 0.945195 0.316000 0.030000 0.0100
14 0.806556 0.271933 0.996095 0.322000 0.048000 0.0100
15 0.781184 0.263866 0.905792 0.300000 0.036000 0.0100
16 0.732171 0.248235 0.898016 0.316000 0.034000 0.0100
17 0.710222 0.237647 0.918373 0.302000 0.042000 0.0100
18 0.656151 0.223277 0.912142 0.306000 0.032000 0.0100
19 0.631633 0.210336 0.793342 0.274000 0.034000 0.0010
20 0.539100 0.182941 0.757377 0.258000 0.034000 0.0010
21 0.517366 0.172773 0.758475 0.254000 0.028000 0.0010
22 0.502556 0.166387 0.747837 0.256000 0.028000 0.0010
23 0.486466 0.164370 0.749277 0.254000 0.032000 0.0010
24 0.476005 0.162353 0.749252 0.252000 0.032000 0.0010
25 0.463468 0.157563 0.754237 0.254000 0.034000 0.0010
26 0.457040 0.153193 0.745088 0.246000 0.034000 0.0010
27 0.456977 0.154538 0.726688 0.244000 0.034000 0.0010
28 0.447599 0.150504 0.737343 0.256000 0.032000 0.0010
29 0.433964 0.147395 0.739840 0.254000 0.032000 0.0010
30 0.433755 0.144454 0.755817 0.248000 0.030000 0.0010
31 0.422160 0.142269 0.729965 0.244000 0.030000 0.0001
32 0.412104 0.138655 0.730816 0.244000 0.030000 0.0001
33 0.411611 0.138319 0.731842 0.242000 0.030000 0.0001
34 0.403684 0.132101 0.726915 0.242000 0.030000 0.0001
35 0.410788 0.137563 0.728174 0.236000 0.030000 0.0001
36 0.411745 0.137563 0.726576 0.244000 0.030000 0.0001
37 0.405807 0.134538 0.728789 0.246000 0.030000 0.0001
38 0.413303 0.138235 0.726447 0.240000 0.030000 0.0001
39 0.402246 0.133697 0.727214 0.240000 0.030000 0.0000
Terminated
real 46m5.150s
user 68m54.232s
sys 30m7.252s
}};
**ex20151203 【Convの出力イメージのサイズを入力イメージと同サイズにできるように】[#ex20151203]
conv2dをborder_mode = ‘full’でパディングを行った時の出力イメージの大きさが元画像より大きくなる(元画像幅(X) + フィルター幅(W) - 1)のが気に食わなかったので、元画像と同じサイズになるように下記の処理を加えて実験。CNNの設計出力で「border_mode: same」となっているConv層が対象。&br;
con2d関数でborder_mode = ‘full’とした時に得られたイメージの縦横幅から(W-1-W/2) : (X+W-1-W/2)の範囲を指定して値を取り出し、これにバイアスを付け加えた値を層の出力とすることで、入力イメージと同サイズのイメージが出力に。なお、こうして指定した範囲では今のところ、Wが偶数の時に良い値を取り出せるか分かりません(理由は割愛。今回の実験ではWはすべて奇数なのでこの問題の心配はない・・・はず)。
-[[#ex20151201c4p2]]との比較( https://drive.google.com/open?id=0B9W18yqQO6JAWUN4cGt1M3BmQmc )
-若干過学習してる気がしなくもないが、ほぼ変わらないということにしておく。良くなるかと思ったけど・・・。
-パディング使って元画像より大きくしてしまうのはいろいろ問題あると思うというか気に入らないので、この方法を使うかborder_mode=‘valid’だけで今後考えていきたいが・・・100クラスでもっと条件整えて実験してみるべきだろうか。
-実行時間はおよそ1.13倍に。layer5での畳み込み計算量の違いのせいだろう。
#pre{{
ex201512031617
############# training condition ############
100 epoch training. input_size: (227, 227) , minibatch_size: 100
, learning_rate: 0.01 , momentum: 0.9 , weight_decay: 0.0005
#############################################
########## architecture of the CNN #########
layer1 - Convolution
< kernel_size: (100, 11, 11) , stride: (4, 4) , act_func: ReLU
, border_mode: valid , out_size: (100, 55, 55) , iniW_σ: 0.001000 >
layer2 - Pooling
< downscale: (2, 2) , stride: None , out_size: (100, 27, 27) >
layer3 - Convolution
< kernel_size: (200, 5, 5) , stride: (1, 1) , act_func: ReLU
, border_mode: same , out_size: (200, 27, 27) , iniW_σ: 0.001000 >
layer4 - Pooling
< downscale: (2, 2) , stride: None , out_size: (200, 13, 13) >
layer5 - Convolution
< kernel_size: (200, 3, 3) , stride: (1, 1) , act_func: ReLU
, border_mode: same , out_size: (200, 13, 13) , iniW_σ: 0.001000 >
layer6 - Pooling
< downscale: (2, 2) , stride: None , out_size: (200, 6, 6) >
layer7 - fully-connected
< number_of_units: 4000 , drop_rate: 1.0 , act_func: ReLU , iniW_σ: 0.010000 >
layer8 - fully-connected
< number_of_units: 4000 , drop_rate: 0.5 , act_func: ReLU , iniW_σ: 0.010000 >
layer9 - fully-connected
< number_of_units: 10 , drop_rate: 0.5 , act_func: softmax , iniW_σ: 0.010000 >
###########################################
##### CNN initialized #####
##### start training #####
epoch XE(L) error(L) XE(V) error(V)
1 2.261595 0.847815 2.231633 0.866000
2 2.121800 0.793529 2.115445 0.776000
3 2.049488 0.753109 2.110948 0.778000
4 2.013915 0.739496 2.093595 0.780000
5 1.992620 0.729580 2.050377 0.760000
6 1.975841 0.720756 2.043442 0.724000
7 1.962802 0.708908 2.048557 0.734000
8 1.942856 0.701176 2.060472 0.756000
9 1.944721 0.700924 2.013132 0.742000
10 1.927313 0.694874 2.041432 0.738000
11 1.914587 0.688571 2.005463 0.712000
12 1.912851 0.681345 2.000884 0.736000
13 1.885119 0.677563 1.947328 0.696000
14 1.876197 0.672857 1.942985 0.664000
15 1.846004 0.657815 1.946611 0.680000
16 1.810703 0.642521 1.959792 0.702000
17 1.786401 0.627899 1.872008 0.648000
18 1.755359 0.617731 1.825710 0.626000
19 1.723820 0.601513 1.836580 0.626000
20 1.669675 0.584538 1.714131 0.598000
21 1.610878 0.557143 1.656381 0.544000
22 1.565316 0.535210 1.620164 0.530000
23 1.514481 0.518151 1.584729 0.518000
24 1.463522 0.499916 1.507924 0.530000
25 1.404998 0.479916 1.483843 0.496000
26 1.328340 0.452941 1.406143 0.486000
27 1.281395 0.428235 1.389601 0.486000
28 1.224581 0.411849 1.331589 0.456000
29 1.188548 0.393361 1.211243 0.422000
30 1.143402 0.384538 1.229746 0.414000
31 1.105005 0.368235 1.183988 0.406000
32 1.074304 0.360756 1.090274 0.388000
33 1.024225 0.345126 1.076960 0.368000
34 0.990604 0.332185 1.154183 0.416000
35 0.974571 0.324118 1.228499 0.390000
36 0.955587 0.318151 1.059609 0.366000
37 0.893780 0.302353 1.043145 0.346000
38 0.893082 0.301765 1.077651 0.356000
39 0.856138 0.290420 1.029480 0.342000
40 0.832944 0.279244 1.009248 0.358000
41 0.810006 0.274370 0.976203 0.334000
42 0.785528 0.266975 0.956664 0.342000
43 0.747682 0.252689 0.956831 0.336000
44 0.726532 0.246134 0.987396 0.326000
45 0.714824 0.240000 0.945414 0.316000
46 0.687596 0.230672 0.982540 0.334000
47 0.662129 0.227143 0.963468 0.310000
48 0.653706 0.222101 0.875850 0.306000
49 0.646374 0.222941 0.994816 0.344000
50 0.602276 0.206723 0.991948 0.312000
51 0.569963 0.195546 0.923926 0.308000
52 0.542956 0.186555 0.921159 0.294000
53 0.533963 0.181345 0.982949 0.300000
54 0.507973 0.173529 1.057671 0.316000
55 0.487629 0.166555 0.925082 0.308000
56 0.458850 0.157983 0.910515 0.288000
57 0.445771 0.150252 1.062388 0.312000
58 0.428169 0.142605 0.966293 0.304000
59 0.401519 0.136303 0.998146 0.326000
60 0.388616 0.130420 1.031176 0.300000
61 0.385073 0.131681 0.951174 0.298000
62 0.349887 0.116891 1.062435 0.306000
63 0.340367 0.117143 0.992822 0.304000
64 0.313215 0.106639 1.093462 0.280000
65 0.319747 0.110252 1.107072 0.332000
66 0.292158 0.101261 1.047072 0.306000
67 0.283548 0.096807 1.044728 0.296000
68 0.251862 0.085042 1.161073 0.296000
69 0.256815 0.086639 1.234181 0.294000
70 0.226725 0.074538 1.140025 0.296000
71 0.222152 0.075714 1.126003 0.290000
72 0.211216 0.069748 1.180236 0.310000
73 0.190438 0.065714 1.185414 0.294000
74 0.199757 0.069580 1.241724 0.296000
75 0.184444 0.058739 1.166877 0.294000
76 0.181040 0.063782 1.324296 0.306000
77 0.179575 0.059244 1.104675 0.292000
78 0.182981 0.061008 1.323485 0.286000
79 0.144122 0.046723 1.212137 0.270000
80 0.171675 0.056218 1.315203 0.298000
81 0.150110 0.048487 1.165764 0.304000
82 0.133821 0.045714 1.341715 0.320000
83 0.130722 0.043697 1.352473 0.288000
84 0.123045 0.039664 1.235930 0.296000
85 0.112687 0.037059 1.360362 0.290000
86 0.106411 0.033950 1.350940 0.300000
87 0.110522 0.037143 1.332613 0.300000
88 0.112628 0.036303 1.392944 0.320000
89 0.114699 0.039412 1.380837 0.302000
90 0.110989 0.036134 1.198287 0.294000
91 0.098821 0.033025 1.243859 0.282000
92 0.099191 0.033025 1.237007 0.284000
93 0.091087 0.029496 1.279187 0.290000
94 0.091195 0.029832 1.363711 0.280000
95 0.100856 0.033025 1.232120 0.276000
96 0.082839 0.026723 1.570684 0.292000
97 0.076587 0.023613 1.360142 0.286000
98 0.071916 0.023193 1.490749 0.288000
99 0.083934 0.027899 1.455066 0.266000
100 0.079976 0.025294 1.455697 0.282000
#elapsed time after 100 epoch : 6456.72848797 [sec]
real 107m46.662s
user 191m42.832s
sys 49m24.920s
}};
**ex20151201 【Convのストライドを大きくするか、プーリングサイズを大きくするか】 [#ex20151201]
-CNNの最初のConvーPoolでイメージのサイズを(27,27)まで落とし込む際に、Convストライド(4,4)・Poolサイズ(2,2)か、Convストライド(1,1)・Poolサイズ(8,8)の二通りの設定が考えられる(Convストライドはconv2d関数のsubsampleではなく、subsample=(1,1)で[[takataka/note/2015-10-03]]下部2の方法で設定
)。
-学習推移の比較( https://drive.google.com/open?id=0B9W18yqQO6JAbmdpcVNubks3R00 )
-順伝播時の畳み込み計算の量は同じだから計算時間もあまり変わらないかと思ったら、3倍ほど差がついてしまっている。
-前者のバリデーションデータ最低誤識別率は0.256、後者は0.218で有意味な差が付いてしまっている。前者より後者の方が計算上の取りこぼしが少ない為か。
-1000クラスで行う場合20日近くかかることを考えると、後者で実験を続けるのは論外だが、この差は惜しい。convのストライドを(1,1)のままで、かつ計算量を抑える方法はないものか。
-なお、前者の結果と[[#d447b3c9]]は、[[ex20151130>#r416d0c0]]で導入した訓練画像の平均を入力から引く正規化の有無のみの違いとなる。比べてみると、この正規化が効いているらしいことが分かる( https://drive.google.com/open?id=0B9W18yqQO6JAcDd5TzJtS0lwTWM )。
***Conv_stride(4, 4), Pool_size(2, 2) [#ex20151201c4p2]
#pre{{
ex201511301829
############# training condition ############
100 epoch training. input_size: (227, 227) , minibatch_size: 100
, learning_rate: 0.01 , momentum: 0.9 , weight_decay: 0.0005
#############################################
########## architecture of the CNN #########
layer1 - Convolution
< kernel_size: (100, 11, 11) , stride: (4, 4) , act_func: ReLU
, border_mode: valid , out_size: (100, 55, 55) , iniW_σ: 0.001000 >
layer2 - Pooling
< downscale: (2, 2) , stride: None , out_size: (100, 27, 27) >
layer3 - Convolution
< kernel_size: (200, 5, 5) , stride: (1, 1) , act_func: ReLU
, border_mode: valid , out_size: (200, 23, 23) , iniW_σ: 0.001000 >
layer4 - Pooling
< downscale: (2, 2) , stride: None , out_size: (200, 11, 11) >
layer5 - Convolution
< kernel_size: (200, 3, 3) , stride: (1, 1) , act_func: ReLU
, border_mode: full , out_size: (200, 13, 13) , iniW_σ: 0.001000 >
layer6 - Pooling
< downscale: (2, 2) , stride: None , out_size: (200, 6, 6) >
layer7 - fully-connected
< number_of_units: 4000 , drop_rate: 1.0 , act_func: ReLU , iniW_σ: 0.010000 >
layer8 - fully-connected
< number_of_units: 4000 , drop_rate: 0.5 , act_func: ReLU , iniW_σ: 0.010000 >
layer9 - fully-connected
< number_of_units: 10 , drop_rate: 0.5 , act_func: softmax , iniW_σ: 0.010000 >
###########################################
##### CNN initialized #####
##### start training #####
epoch XE(L) error(L) XE(V) error(V)
1 2.265512 0.854118 2.240207 0.858000
2 2.126916 0.793361 2.132252 0.796000
3 2.053773 0.753529 2.114341 0.784000
4 2.022574 0.746387 2.118557 0.794000
5 2.000311 0.733529 2.071346 0.770000
6 1.986872 0.728319 2.072314 0.742000
7 1.970179 0.720588 2.056508 0.746000
8 1.951982 0.710924 2.060582 0.748000
9 1.949321 0.706555 2.019281 0.742000
10 1.938045 0.701681 2.072499 0.758000
11 1.923634 0.699412 2.044654 0.740000
12 1.917019 0.685966 1.999032 0.730000
13 1.896135 0.687899 1.983176 0.716000
14 1.894803 0.684790 1.999759 0.692000
15 1.874767 0.674370 2.021464 0.692000
16 1.856811 0.662857 2.004935 0.698000
17 1.838303 0.660252 1.953219 0.678000
18 1.826441 0.655630 1.913830 0.660000
19 1.779350 0.636891 1.879912 0.644000
20 1.739723 0.620000 1.825149 0.642000
21 1.691372 0.597647 1.752824 0.608000
22 1.655938 0.578151 1.778146 0.578000
23 1.629195 0.557647 1.714115 0.596000
24 1.565406 0.541092 1.620170 0.546000
25 1.513010 0.523109 1.658018 0.574000
26 1.463774 0.507731 1.561847 0.530000
27 1.435841 0.494958 1.564757 0.518000
28 1.394540 0.477563 1.442391 0.500000
29 1.320723 0.446134 1.298339 0.450000
30 1.282767 0.432941 1.356098 0.452000
31 1.209008 0.406975 1.243206 0.434000
32 1.160657 0.392941 1.208980 0.416000
33 1.114366 0.371849 1.130212 0.396000
34 1.093672 0.365714 1.180310 0.392000
35 1.056027 0.353445 1.139584 0.372000
36 1.032040 0.343109 1.110961 0.380000
37 0.995310 0.331008 1.095206 0.380000
38 0.979810 0.327479 1.030094 0.356000
39 0.946138 0.312689 1.039761 0.360000
40 0.934624 0.312185 1.020273 0.352000
41 0.906518 0.304958 1.040692 0.366000
42 0.887821 0.297059 1.009176 0.344000
43 0.847129 0.282437 1.010904 0.346000
44 0.837678 0.279832 0.980055 0.344000
45 0.804155 0.269664 0.953995 0.350000
46 0.793181 0.269496 0.983602 0.334000
47 0.777183 0.261092 1.045581 0.340000
48 0.755691 0.255966 0.910118 0.322000
49 0.727655 0.246975 1.054030 0.338000
50 0.713998 0.238655 0.949052 0.318000
51 0.671913 0.225378 0.960111 0.322000
52 0.655613 0.222689 0.875177 0.310000
53 0.632419 0.215546 0.921186 0.304000
54 0.609891 0.210000 0.926388 0.308000
55 0.598780 0.203109 0.956141 0.314000
56 0.580358 0.197815 0.884541 0.284000
57 0.563777 0.191176 1.036186 0.334000
58 0.540403 0.185042 0.878522 0.290000
59 0.525809 0.179832 0.866313 0.276000
60 0.496978 0.173025 0.903838 0.302000
61 0.510715 0.174874 0.933896 0.322000
62 0.470734 0.158319 0.930657 0.282000
63 0.445534 0.150756 0.908095 0.290000
64 0.449468 0.154370 1.021374 0.302000
65 0.405614 0.139748 0.982117 0.298000
66 0.389739 0.131681 0.952315 0.286000
67 0.385806 0.130252 0.993064 0.298000
68 0.346114 0.117479 0.925194 0.276000
69 0.337600 0.113529 1.001964 0.294000
70 0.316199 0.107479 0.954678 0.278000
71 0.297671 0.103109 1.017478 0.264000
72 0.278122 0.094622 1.026215 0.278000
73 0.279483 0.094874 1.075593 0.284000
74 0.274797 0.093025 1.125169 0.290000
75 0.261910 0.089748 1.063620 0.278000
76 0.247519 0.083950 1.053286 0.288000
77 0.257481 0.088151 0.964341 0.282000
78 0.233071 0.077899 1.143678 0.294000
79 0.208931 0.074202 1.090725 0.286000
80 0.204916 0.068235 1.221492 0.308000
81 0.180988 0.062605 1.038368 0.256000
82 0.176942 0.060000 1.066491 0.292000
83 0.178173 0.058151 1.248633 0.286000
84 0.189114 0.063697 1.118649 0.270000
85 0.162007 0.055798 1.152669 0.270000
86 0.135113 0.044874 1.245378 0.282000
87 0.157199 0.054454 1.214303 0.282000
88 0.142269 0.048403 1.250788 0.278000
89 0.133804 0.045042 1.334189 0.294000
90 0.144161 0.046218 1.119339 0.262000
91 0.109862 0.036387 1.400242 0.294000
92 0.119807 0.040000 1.270848 0.272000
93 0.116280 0.037143 1.310338 0.288000
94 0.103779 0.035546 1.359119 0.286000
95 0.109552 0.036050 1.254955 0.288000
96 0.111649 0.037227 1.229749 0.270000
97 0.082756 0.027563 1.275888 0.284000
98 0.096411 0.033697 1.360698 0.274000
99 0.093900 0.031345 1.337077 0.266000
100 0.096269 0.032269 1.190403 0.258000
#elapsed time after 100 epoch : 5708.08045292 [sec]
real 95m17.891s
user 178m35.332s
sys 46m2.572s
}};
***Conv_stride(1, 1), Pool_size(8, 8)
#pre{{
ex201511302330
############# training condition ############
100 epoch training. input_size: (227, 227) , minibatch_size: 100
, learning_rate: 0.01 , momentum: 0.9 , weight_decay: 0.0005
#############################################
########## architecture of the CNN #########
layer1 - Convolution
< kernel_size: (100, 11, 11) , stride: (1, 1) , act_func: ReLU
, border_mode: valid , out_size: (100, 217, 217) , iniW_σ: 0.001000 >
layer2 - Pooling
< downscale: (8, 8) , stride: None , out_size: (100, 27, 27) >
layer3 - Convolution
< kernel_size: (200, 5, 5) , stride: (1, 1) , act_func: ReLU
, border_mode: valid , out_size: (200, 23, 23) , iniW_σ: 0.001000 >
layer4 - Pooling
< downscale: (2, 2) , stride: None , out_size: (200, 11, 11) >
layer5 - Convolution
< kernel_size: (200, 3, 3) , stride: (1, 1) , act_func: ReLU
, border_mode: full , out_size: (200, 13, 13) , iniW_σ: 0.001000 >
layer6 - Pooling
< downscale: (2, 2) , stride: None , out_size: (200, 6, 6) >
layer7 - fully-connected
< number_of_units: 4000 , drop_rate: 1.0 , act_func: ReLU , iniW_σ: 0.010000 >
layer8 - fully-connected
< number_of_units: 4000 , drop_rate: 0.5 , act_func: ReLU , iniW_σ: 0.010000 >
layer9 - fully-connected
< number_of_units: 10 , drop_rate: 0.5 , act_func: softmax , iniW_σ: 0.010000 >
###########################################
##### CNN initialized #####
##### start training #####
epoch XE(L) error(L) XE(V) error(V)
1 2.266060 0.853361 2.243392 0.856000
2 2.130685 0.793025 2.135200 0.794000
3 2.054085 0.756218 2.123839 0.782000
4 2.023824 0.745546 2.125052 0.778000
5 2.004558 0.736723 2.075419 0.764000
6 1.988269 0.730084 2.080712 0.736000
7 1.978521 0.722269 2.064026 0.752000
8 1.949791 0.708992 2.083354 0.728000
9 1.946183 0.713109 2.024760 0.726000
10 1.939402 0.704370 2.067147 0.736000
11 1.924256 0.696303 2.015610 0.718000
12 1.913130 0.683445 2.023050 0.692000
13 1.890571 0.675630 1.992919 0.702000
14 1.871528 0.659244 1.986287 0.704000
15 1.837502 0.646134 1.965102 0.676000
16 1.783898 0.617983 1.883287 0.664000
17 1.727252 0.598487 1.817291 0.628000
18 1.677167 0.577563 1.763205 0.594000
19 1.646680 0.566134 1.717152 0.584000
20 1.620932 0.560252 1.631011 0.538000
21 1.569441 0.537563 1.638784 0.556000
22 1.520713 0.521765 1.531932 0.524000
23 1.446847 0.489832 1.521347 0.518000
24 1.373653 0.470084 1.373839 0.488000
25 1.316823 0.444706 1.389143 0.510000
26 1.273436 0.432353 1.357315 0.474000
27 1.231128 0.417395 1.275293 0.454000
28 1.158633 0.392185 1.244637 0.434000
29 1.145679 0.385798 1.132520 0.388000
30 1.096611 0.365882 1.151662 0.404000
31 1.078286 0.362857 1.154174 0.404000
32 1.039630 0.345714 1.099213 0.380000
33 1.006328 0.336555 1.065627 0.356000
34 0.985907 0.326807 1.064509 0.356000
35 0.938378 0.313950 1.067813 0.372000
36 0.919013 0.305546 1.016313 0.354000
37 0.892731 0.293613 1.056768 0.374000
38 0.874155 0.292185 1.016974 0.368000
39 0.855303 0.284286 0.934007 0.320000
40 0.839621 0.281597 0.950497 0.314000
41 0.812977 0.271597 1.036738 0.330000
42 0.784090 0.265630 0.900286 0.312000
43 0.737794 0.247563 0.924353 0.316000
44 0.750065 0.249244 0.895992 0.314000
45 0.718061 0.235294 0.909370 0.306000
46 0.680311 0.226807 0.938665 0.318000
47 0.670178 0.225294 0.915215 0.318000
48 0.669341 0.222941 0.815276 0.266000
49 0.643219 0.217479 0.880080 0.296000
50 0.608783 0.207983 0.871083 0.286000
51 0.586518 0.198403 0.879849 0.294000
52 0.560401 0.187815 0.862862 0.268000
53 0.552764 0.186471 0.937973 0.306000
54 0.541401 0.185546 0.878444 0.292000
55 0.528155 0.176975 0.889366 0.282000
56 0.476757 0.162017 0.818434 0.268000
57 0.464492 0.157563 0.936802 0.308000
58 0.448118 0.156387 0.828980 0.278000
59 0.437451 0.147059 0.847776 0.276000
60 0.407118 0.137479 0.857101 0.262000
61 0.410758 0.139748 0.848346 0.266000
62 0.375467 0.131849 0.912574 0.262000
63 0.380334 0.128151 0.809710 0.250000
64 0.352329 0.122353 0.956197 0.288000
65 0.331058 0.113109 0.943340 0.274000
66 0.337890 0.114454 0.789285 0.246000
67 0.301068 0.102353 0.849514 0.256000
68 0.288528 0.100252 0.899972 0.260000
69 0.259481 0.090588 0.951661 0.262000
70 0.246767 0.083277 0.930885 0.254000
71 0.230417 0.076387 0.929545 0.282000
72 0.229210 0.080084 0.900590 0.244000
73 0.226441 0.077899 0.938132 0.238000
74 0.219149 0.075462 0.895079 0.242000
75 0.206667 0.072269 0.971582 0.254000
76 0.206785 0.071261 0.916770 0.238000
77 0.177221 0.060252 0.963789 0.256000
78 0.193635 0.065546 1.013842 0.264000
79 0.179224 0.060672 0.999443 0.238000
80 0.193169 0.064370 0.982249 0.262000
81 0.156145 0.051261 1.105881 0.268000
82 0.147516 0.048739 1.030534 0.248000
83 0.130676 0.042101 0.975202 0.242000
84 0.153309 0.052521 1.076137 0.274000
85 0.128124 0.042773 1.149028 0.266000
86 0.120232 0.039496 1.078922 0.242000
87 0.108206 0.034538 0.959661 0.224000
88 0.107621 0.036471 1.261161 0.270000
89 0.111350 0.037143 1.049523 0.256000
90 0.113830 0.037647 1.140495 0.244000
91 0.096187 0.030420 1.228679 0.268000
92 0.095758 0.031933 1.192075 0.254000
93 0.100844 0.033109 1.188920 0.254000
94 0.093317 0.030000 1.074962 0.270000
95 0.089934 0.029748 1.185309 0.250000
96 0.088179 0.031176 1.101567 0.238000
97 0.078000 0.025966 1.128674 0.238000
98 0.083343 0.026807 1.140479 0.218000
99 0.073937 0.023613 1.481587 0.292000
100 0.089651 0.028403 1.117366 0.238000
#elapsed time after 100 epoch : 17126.3356249 [sec]
real 285m35.820s
user 355m27.476s
sys 70m56.932s
}};
**ex20151127-1_重みの初期値を一様分布から正規分布へと変えて、初期値の標準偏差と学習係数の適切な組み合わせを探る実験 [#febbc23c]
-一様分布のときと同様に、FCの初期値の幅をConvより広くしたものが最も良い性能を示した。
-学習係数が0.001のデータをあまり貼ってないのでアレですが、初期値に正規分布を使う場合、学習係数は0.01の方が良いと思います。0.001では総じて遅すぎたので。
-omegaの横の数値が標準偏差の値です。sigmaと間違えました。
***eta0.01, conv_omega0.001, FC_omega0.01 [#d447b3c9]
ex201511262153
#pre{{
########## architecture of the CNN #########
100 epoch learning. minibatch_size: 100 , learning_rate: 0.01 , momentum: 0.9 , weight_decay: 0.0005
layer1 - Convolution
< kernel_size: (100, 11, 11) , stride: (4, 4) , act_func: ReLU
, border_mode: valid , out_size: (100, 55, 55) , iniW_σ: 0.001000 >
layer2 - Pooling
< downscale: (2, 2) , stride: None , out_size: (100, 27, 27) >
layer3 - Convolution
< kernel_size: (200, 5, 5) , stride: (1, 1) , act_func: ReLU
, border_mode: valid , out_size: (200, 23, 23) , iniW_σ: 0.001000 >
layer4 - Pooling
< downscale: (2, 2) , stride: None , out_size: (200, 11, 11) >
layer5 - Convolution
< kernel_size: (200, 3, 3) , stride: (1, 1) , act_func: ReLU
, border_mode: full , out_size: (200, 13, 13) , iniW_σ: 0.001000 >
layer6 - Pooling
< downscale: (2, 2) , stride: None , out_size: (200, 6, 6) >
layer7 - fully-connected
< number_of_units: 4000 , drop_rate: 1.0 , act_func: ReLU , iniW_σ: 0.010000 >
layer8 - fully-connected
< number_of_units: 4000 , drop_rate: 0.5 , act_func: ReLU , iniW_σ: 0.010000 >
layer9 - fully-connected
< number_of_units: 10 , drop_rate: 0.5 , act_func: softmax , iniW_σ: 0.010000 >
###########################################
##### start training #####
epoch XE(L) error(L) XE(V) error(V)
1 2.279788 0.857227 2.296859 0.878000
2 2.191314 0.806891 2.173209 0.806000
3 2.096164 0.762017 2.119903 0.768000
4 2.056766 0.748571 2.116595 0.758000
5 2.039419 0.747899 2.087323 0.764000
6 2.048450 0.750420 2.100580 0.760000
7 2.012680 0.735798 2.086680 0.750000
8 1.998883 0.730588 2.094667 0.746000
9 1.992026 0.723193 2.157129 0.776000
10 1.985581 0.719412 2.196125 0.788000
11 1.973130 0.716050 2.102198 0.744000
12 1.976247 0.717059 2.016111 0.710000
13 1.925662 0.700336 2.028982 0.724000
14 1.938949 0.698655 2.114204 0.752000
15 1.924359 0.685378 1.994008 0.710000
16 1.882777 0.662941 2.102367 0.722000
17 1.872506 0.660924 1.966560 0.682000
18 1.864624 0.655966 1.922495 0.682000
19 1.825770 0.641429 1.902522 0.630000
20 1.793480 0.625630 1.870321 0.648000
21 1.752025 0.605882 1.862212 0.628000
22 1.738010 0.605294 1.857119 0.622000
23 1.712135 0.596807 1.788575 0.646000
24 1.679222 0.581008 1.724772 0.558000
25 1.643857 0.568824 1.674149 0.568000
26 1.615264 0.552185 1.648617 0.566000
27 1.580249 0.544958 1.699005 0.576000
28 1.544599 0.532941 1.601888 0.564000
29 1.566499 0.540168 1.604061 0.548000
30 1.482200 0.507227 1.630673 0.562000
31 1.407958 0.485378 1.467805 0.508000
32 1.356966 0.458235 1.380396 0.478000
33 1.299496 0.437647 1.337013 0.464000
34 1.281185 0.430084 1.349695 0.466000
35 1.209316 0.416639 1.271618 0.434000
36 1.208390 0.408319 1.262016 0.436000
37 1.145750 0.388487 1.228800 0.416000
38 1.145020 0.390252 1.210642 0.408000
39 1.092922 0.369160 1.250346 0.402000
40 1.090970 0.368067 1.125966 0.382000
41 1.048666 0.350168 1.124405 0.404000
42 1.004277 0.336807 1.172188 0.398000
43 0.990163 0.328571 1.079470 0.370000
44 0.956650 0.319160 1.114563 0.396000
45 0.942413 0.317479 1.080161 0.374000
46 0.927220 0.308235 1.168196 0.380000
47 0.919637 0.313193 1.077926 0.366000
48 0.891175 0.298739 0.968754 0.340000
49 0.856628 0.286639 1.008114 0.360000
50 0.833166 0.282437 1.055485 0.332000
51 0.794779 0.271597 1.056139 0.344000
52 0.788357 0.263782 1.031454 0.326000
53 0.765426 0.260756 1.075679 0.372000
54 0.752347 0.253193 1.089792 0.364000
55 0.735859 0.250672 0.985893 0.324000
56 0.721941 0.245882 1.001033 0.316000
57 0.688821 0.231597 1.064463 0.338000
58 0.672875 0.229076 0.970349 0.326000
59 0.655550 0.224118 0.975476 0.316000
60 0.617892 0.210672 1.082753 0.356000
61 0.616934 0.207731 1.003524 0.326000
62 0.587871 0.199832 1.066631 0.328000
63 0.592913 0.204118 1.016835 0.308000
64 0.558418 0.187815 1.057545 0.320000
65 0.542399 0.185378 1.098326 0.344000
66 0.515363 0.173277 1.050404 0.302000
67 0.495595 0.171597 1.025466 0.326000
68 0.464327 0.161513 1.018175 0.282000
69 0.445623 0.153193 1.056368 0.296000
70 0.420819 0.142521 1.074576 0.322000
71 0.423316 0.147647 1.230379 0.338000
72 0.398392 0.136807 1.104836 0.306000
73 0.395690 0.138487 1.109917 0.312000
74 0.385336 0.128235 1.210536 0.326000
75 0.373550 0.124874 1.257324 0.304000
76 0.358137 0.122941 1.214032 0.308000
77 0.317682 0.106975 1.183054 0.322000
78 0.333262 0.112353 1.269349 0.328000
79 0.306124 0.105126 1.303980 0.308000
80 0.296196 0.100336 1.158767 0.308000
81 0.253620 0.084958 1.222014 0.302000
82 0.277029 0.093193 1.141394 0.290000
83 0.257218 0.086387 1.152766 0.278000
84 0.253831 0.083109 1.175288 0.292000
85 0.231094 0.077311 1.344497 0.310000
86 0.228450 0.075294 1.330323 0.318000
87 0.213584 0.073109 1.187189 0.290000
88 0.215680 0.074202 1.225960 0.296000
89 0.198841 0.063782 1.419377 0.308000
90 0.188258 0.065462 1.334841 0.302000
91 0.172070 0.058151 1.257102 0.300000
92 0.174285 0.056218 1.297317 0.302000
93 0.175419 0.057647 1.259745 0.278000
94 0.164890 0.055210 1.333592 0.308000
95 0.164969 0.054706 1.301733 0.290000
96 0.152825 0.052017 1.361302 0.314000
97 0.153835 0.050504 1.367234 0.292000
98 0.139659 0.047731 1.436525 0.276000
99 0.146334 0.048235 1.337804 0.294000
100 0.141419 0.047311 1.285882 0.294000
#elapsed time after 100 epoch : 5919.28769898 [sec]
real 98m49.206s
user 175m15.312s
sys 46m14.628s
}};
***eta0.01, conv_omega0.001, FC_omega0.001
ex201511262237
#pre{{
########## architecture of the CNN #########
100 epoch learning. minibatch_size: 100 , learning_rate: 0.01 , momentum: 0.9 , weight_decay: 0.0005
layer1 - Convolution
< kernel_size: (100, 11, 11) , stride: (4, 4) , act_func: ReLU
, border_mode: valid , out_size: (100, 55, 55) , iniW_σ: 0.001000 >
layer2 - Pooling
< downscale: (2, 2) , stride: None , out_size: (100, 27, 27) >
layer3 - Convolution
< kernel_size: (200, 5, 5) , stride: (1, 1) , act_func: ReLU
, border_mode: valid , out_size: (200, 23, 23) , iniW_σ: 0.001000 >
layer4 - Pooling
< downscale: (2, 2) , stride: None , out_size: (200, 11, 11) >
layer5 - Convolution
< kernel_size: (200, 3, 3) , stride: (1, 1) , act_func: ReLU
, border_mode: full , out_size: (200, 13, 13) , iniW_σ: 0.001000 >
layer6 - Pooling
< downscale: (2, 2) , stride: None , out_size: (200, 6, 6) >
layer7 - fully-connected
< number_of_units: 4000 , drop_rate: 1.0 , act_func: ReLU , iniW_σ: 0.001000 >
layer8 - fully-connected
< number_of_units: 4000 , drop_rate: 0.5 , act_func: ReLU , iniW_σ: 0.001000 >
layer9 - fully-connected
< number_of_units: 10 , drop_rate: 0.5 , act_func: softmax , iniW_σ: 0.001000 >
###########################################
##### start training #####
epoch XE(L) error(L) XE(V) error(V)
1 2.289214 0.859748 2.314248 0.900000
2 2.279820 0.858151 2.322131 0.900000
3 2.279018 0.858319 2.325436 0.900000
4 2.278875 0.858151 2.326430 0.900000
5 2.278644 0.857899 2.326484 0.900000
6 2.278996 0.858571 2.327302 0.900000
7 2.278895 0.858487 2.327085 0.900000
8 2.278733 0.858151 2.326541 0.900000
9 2.278884 0.858235 2.328256 0.900000
10 2.278741 0.858235 2.327457 0.900000
11 2.278825 0.858235 2.326281 0.900000
12 2.278796 0.858487 2.327071 0.900000
13 2.278692 0.858067 2.326404 0.900000
14 2.278856 0.858403 2.327175 0.900000
15 2.278512 0.858151 2.326523 0.900000
16 2.278399 0.858067 2.327618 0.900000
17 2.277651 0.858067 2.324584 0.900000
18 2.266181 0.843445 2.295807 0.882000
19 2.192752 0.818908 2.217242 0.816000
20 2.126434 0.778992 2.204913 0.828000
21 2.098422 0.767311 2.173321 0.800000
22 2.083043 0.765126 2.173632 0.828000
23 2.077946 0.762101 2.161616 0.806000
24 2.062897 0.757059 2.161109 0.798000
25 2.061684 0.759160 2.175754 0.806000
26 2.071941 0.767395 2.187909 0.818000
27 2.053810 0.757647 2.147933 0.810000
28 2.034118 0.754874 2.131397 0.810000
29 2.015168 0.743361 2.099014 0.790000
30 1.977714 0.723529 2.057542 0.748000
31 1.948424 0.709664 2.063088 0.768000
32 1.921753 0.691345 2.008414 0.722000
33 1.893688 0.683950 1.997358 0.720000
34 1.907050 0.689832 1.996617 0.720000
35 1.862669 0.664538 1.956418 0.704000
36 1.839722 0.657479 1.926295 0.702000
37 1.802570 0.646807 2.116516 0.754000
38 1.821628 0.648824 1.912855 0.700000
39 1.790424 0.639748 1.869369 0.666000
40 1.787948 0.637815 1.851022 0.656000
41 1.745422 0.620840 1.966666 0.696000
42 1.742164 0.614286 1.805366 0.630000
43 1.727556 0.604706 1.782348 0.630000
44 1.704735 0.602101 1.748261 0.610000
45 1.697856 0.596639 1.770579 0.624000
46 1.669272 0.584706 1.722074 0.596000
47 1.638318 0.572941 1.674674 0.580000
48 1.610545 0.563697 1.629384 0.550000
49 1.581405 0.551092 1.666137 0.580000
50 1.535939 0.528151 1.592743 0.546000
51 1.524203 0.525462 1.598540 0.562000
52 1.478741 0.503782 1.545349 0.550000
53 1.418556 0.483361 1.524162 0.530000
54 1.374773 0.466303 1.470476 0.506000
55 1.325039 0.448067 1.362406 0.492000
56 1.297996 0.437311 1.293635 0.466000
57 1.263279 0.430168 1.315085 0.472000
58 1.224640 0.410840 1.193438 0.448000
59 1.193657 0.403109 1.171922 0.406000
60 1.129074 0.380336 1.210986 0.430000
61 1.127256 0.376218 1.214752 0.404000
62 1.078499 0.361849 1.097572 0.380000
63 1.064977 0.362773 1.111543 0.374000
64 1.046567 0.351008 1.142148 0.394000
65 1.004249 0.338487 1.072640 0.384000
66 0.995249 0.334958 1.146379 0.380000
67 0.959661 0.320756 1.055862 0.370000
68 0.936250 0.310420 1.014813 0.364000
69 0.925829 0.312353 1.079364 0.380000
70 0.864163 0.289664 0.979812 0.342000
71 0.857257 0.288824 1.006246 0.332000
72 0.846522 0.286303 0.979174 0.326000
73 0.818960 0.272605 0.959189 0.344000
74 0.796602 0.267143 0.996843 0.352000
75 0.816673 0.274370 0.981101 0.348000
76 0.770414 0.263445 0.948454 0.342000
77 0.738688 0.247731 0.988118 0.352000
78 0.730586 0.247395 0.953460 0.338000
79 0.687154 0.230672 1.004113 0.330000
80 0.671581 0.227395 0.984177 0.338000
81 0.647149 0.217563 0.909386 0.318000
82 0.632714 0.213613 0.896398 0.310000
83 0.603839 0.204202 0.971699 0.332000
84 0.574152 0.193529 0.907302 0.300000
85 0.579654 0.197647 1.017110 0.342000
86 0.554308 0.188067 0.916641 0.316000
87 0.530155 0.180840 0.998144 0.302000
88 0.492030 0.166723 0.962299 0.306000
89 0.477169 0.162857 1.006667 0.298000
90 0.482011 0.168319 0.969493 0.304000
91 0.446418 0.148908 0.962634 0.314000
92 0.416054 0.145210 1.013056 0.318000
93 0.407301 0.139076 0.943681 0.294000
94 0.402713 0.136891 1.051337 0.326000
95 0.382483 0.131261 0.986986 0.292000
96 0.374495 0.127143 1.014350 0.312000
97 0.341564 0.115462 1.042052 0.308000
98 0.323176 0.109328 0.984928 0.294000
99 0.323966 0.111513 1.026672 0.284000
100 0.315672 0.107311 0.969291 0.290000
#elapsed time after 100 epoch : 5757.69596601 [sec]
real 96m7.670s
user 174m6.028s
sys 45m8.128s
}};
***eta0.01, conv_omega0.01, FC_omega0.01
#pre{{
############# training condition ############
100 epoch training. input_size: (227, 227) , minibatch_size: 100
, learning_rate: 0.01 , momentum: 0.9 , weight_decay: 0.0005
#############################################
########## architecture of the CNN #########
layer1 - Convolution
< kernel_size: (100, 11, 11) , stride: (4, 4) , act_func: ReLU
, border_mode: valid , out_size: (100, 55, 55) , iniW_σ: 0.010000 >
layer2 - Pooling
< downscale: (2, 2) , stride: None , out_size: (100, 27, 27) >
layer3 - Convolution
< kernel_size: (200, 5, 5) , stride: (1, 1) , act_func: ReLU
, border_mode: valid , out_size: (200, 23, 23) , iniW_σ: 0.010000 >
layer4 - Pooling
< downscale: (2, 2) , stride: None , out_size: (200, 11, 11) >
layer5 - Convolution
< kernel_size: (200, 3, 3) , stride: (1, 1) , act_func: ReLU
, border_mode: full , out_size: (200, 13, 13) , iniW_σ: 0.010000 >
layer6 - Pooling
< downscale: (2, 2) , stride: None , out_size: (200, 6, 6) >
layer7 - fully-connected
< number_of_units: 4000 , drop_rate: 1.0 , act_func: ReLU , iniW_σ: 0.010000 >
layer8 - fully-connected
< number_of_units: 4000 , drop_rate: 0.5 , act_func: ReLU , iniW_σ: 0.010000 >
layer9 - fully-connected
< number_of_units: 10 , drop_rate: 0.5 , act_func: softmax , iniW_σ: 0.010000 >
###########################################
##### CNN initialized #####
##### start training #####
epoch XE(L) error(L) XE(V) error(V)
1 57.355752 0.864034 2.314639 0.900000
2 2.282321 0.858151 2.322015 0.900000
3 2.279897 0.858319 2.325741 0.900000
4 2.279249 0.858151 2.325908 0.900000
5 2.281314 0.857899 2.326870 0.900000
6 2.279741 0.858571 2.327218 0.900000
7 2.279324 0.858487 2.327748 0.900000
8 2.281921 0.858151 2.326988 0.900000
9 2.279329 0.858235 2.328779 0.900000
10 2.279096 0.858235 2.327633 0.900000
11 2.278899 0.858235 2.326507 0.900000
12 2.279134 0.858487 2.327373 0.900000
13 2.278813 0.858067 2.326532 0.900000
14 2.279033 0.858403 2.327347 0.900000
15 2.279382 0.858151 2.327028 0.900000
16 2.278789 0.858067 2.327958 0.900000
17 2.278875 0.858067 2.327753 0.900000
18 2.278697 0.858151 2.326792 0.900000
19 2.279006 0.858067 2.326805 0.900000
20 2.279178 0.858235 2.327419 0.900000
21 2.279002 0.858319 2.327464 0.900000
22 2.279026 0.858151 2.327943 0.900000
23 2.278919 0.858235 2.327733 0.900000
24 2.278769 0.857983 2.327533 0.900000
25 2.278809 0.858655 2.326716 0.900000
26 2.279008 0.858319 2.326593 0.900000
27 2.278840 0.858235 2.326860 0.900000
28 2.278622 0.858151 2.327226 0.900000
29 2.278845 0.858319 2.327433 0.900000
30 2.278707 0.858235 2.327389 0.900000
}};
***eta0.001, conv_omega0.01, FC_omega0.01
#pre{{
##### CNN initialized #####
########## architecture of the CNN #########
100 epoch learning. minibatch_size: 100 , learning_rate: 0.001 , momentum: 0.9 , weight_decay: 0.0005
layer1 - Convolution
< kernel_size: (100, 11, 11) , stride: (4, 4) , act_func: ReLU
, border_mode: valid , out_size: (100, 55, 55) , iniW_σ: 0.001000 >
layer2 - Pooling
< downscale: (2, 2) , stride: None , out_size: (100, 27, 27) >
layer3 - Convolution
< kernel_size: (200, 5, 5) , stride: (1, 1) , act_func: ReLU
, border_mode: valid , out_size: (200, 23, 23) , iniW_σ: 0.001000 >
layer4 - Pooling
< downscale: (2, 2) , stride: None , out_size: (200, 11, 11) >
layer5 - Convolution
< kernel_size: (200, 3, 3) , stride: (1, 1) , act_func: ReLU
, border_mode: full , out_size: (200, 13, 13) , iniW_σ: 0.001000 >
layer6 - Pooling
< downscale: (2, 2) , stride: None , out_size: (200, 6, 6) >
layer7 - fully-connected
< number_of_units: 4000 , drop_rate: 1.0 , act_func: ReLU , iniW_σ: 0.010000 >
layer8 - fully-connected
< number_of_units: 4000 , drop_rate: 0.5 , act_func: ReLU , iniW_σ: 0.010000 >
layer9 - fully-connected
< number_of_units: 10 , drop_rate: 0.5 , act_func: softmax , iniW_σ: 0.010000 >
###########################################
##### start training #####
epoch XE(L) error(L) XE(V) error(V)
1 2.298945 0.868151 2.302684 0.900000
2 2.291475 0.858403 2.303700 0.900000
3 2.283327 0.858319 2.306935 0.900000
4 2.276402 0.858151 2.309872 0.900000
5 2.272492 0.857647 2.307571 0.900000
6 2.268735 0.851261 2.304408 0.900000
7 2.263944 0.848487 2.299230 0.896000
8 2.258452 0.844538 2.291861 0.874000
9 2.250948 0.829496 2.284950 0.878000
10 2.240829 0.830504 2.274124 0.858000
11 2.227953 0.823277 2.255608 0.846000
12 2.207249 0.816218 2.233939 0.840000
13 2.179750 0.797983 2.210093 0.828000
14 2.156048 0.779244 2.190328 0.818000
15 2.134629 0.764958 2.166895 0.800000
16 2.116268 0.755966 2.152267 0.794000
17 2.101822 0.749916 2.135769 0.784000
18 2.089538 0.746387 2.122359 0.772000
19 2.074834 0.738487 2.114890 0.762000
20 2.063822 0.737731 2.111917 0.774000
21 2.054489 0.734790 2.107136 0.764000
22 2.049469 0.734706 2.099675 0.764000
23 2.040591 0.727983 2.095448 0.754000
24 2.029844 0.726050 2.090445 0.748000
25 2.027214 0.724622 2.094199 0.744000
26 2.016414 0.721933 2.097714 0.742000
27 2.011720 0.720756 2.089883 0.738000
28 2.003814 0.720000 2.075352 0.742000
29 2.002691 0.721092 2.076192 0.726000
30 1.994191 0.711429 2.074222 0.734000
31 1.985296 0.711429 2.070637 0.726000
32 1.980544 0.706639 2.068676 0.730000
33 1.969502 0.709160 2.074861 0.718000
34 1.972719 0.709244 2.062387 0.714000
35 1.963712 0.704790 2.058929 0.696000
36 1.962358 0.706807 2.067888 0.700000
37 1.955976 0.701933 2.055356 0.702000
38 1.948130 0.695378 2.039469 0.706000
39 1.945331 0.695630 2.038332 0.694000
40 1.939442 0.695294 2.064415 0.696000
41 1.935854 0.691261 2.045462 0.704000
42 1.934636 0.688992 2.046764 0.706000
43 1.927097 0.687311 2.023183 0.696000
44 1.926392 0.690588 2.030900 0.704000
45 1.922850 0.683782 2.020085 0.708000
46 1.919929 0.684454 2.012714 0.702000
47 1.915890 0.681933 2.019143 0.700000
48 1.916717 0.685378 2.006689 0.696000
49 1.916212 0.684118 2.023428 0.704000
50 1.908615 0.675966 2.006994 0.690000
51 1.914212 0.684118 2.002732 0.684000
52 1.907916 0.671681 2.019715 0.702000
53 1.908784 0.682101 2.029294 0.688000
54 1.904535 0.677479 2.001421 0.692000
55 1.903722 0.676807 2.012378 0.704000
56 1.893367 0.670756 1.998448 0.704000
57 1.896418 0.671849 2.020645 0.682000
58 1.889160 0.662269 1.992309 0.686000
59 1.893595 0.674958 2.002228 0.680000
60 1.891602 0.661681 2.001342 0.692000
61 1.889934 0.668235 2.006420 0.688000
62 1.891581 0.664286 1.990300 0.684000
63 1.884815 0.661849 1.986786 0.684000
64 1.890615 0.662185 1.985312 0.682000
65 1.880546 0.657395 1.997947 0.678000
66 1.877505 0.658655 1.993619 0.698000
67 1.878208 0.660000 2.003519 0.668000
68 1.873462 0.649832 1.993286 0.682000
}};
**ex20151125_[[m/2015/okada/diary/2015-11-25]]のCPCPCPFFFと(たぶん)完全に同じ構成のCNN [#d5d09647]
-学習の推移もほとんど同じになった。
-重みの初期値はConvが-0.01~0.01、FCが-0.1~0.1の一様分布
#pre{{
########## architecture of the CNN #########
60 epoch learning. minibatch_size: 100 , learning_rate: 0.001 , momentum: 0.9 , weight_decay: 0.0005
layer1 - Convolution
< kernel_size: (100, 11, 11) , stride: (4, 4)
, act_func: ReLU , border_mode: valid , out_size: (100, 55, 55) >
layer2 - Pooling
< downscale: (2, 2) , stride: None , out_size: (100, 27, 27) >
layer3 - Convolution
< kernel_size: (200, 5, 5) , stride: (1, 1)
, act_func: ReLU , border_mode: valid , out_size: (200, 23, 23) >
layer4 - Pooling
< downscale: (2, 2) , stride: None , out_size: (200, 11, 11) >
layer5 - Convolution
< kernel_size: (200, 3, 3) , stride: (1, 1)
, act_func: ReLU , border_mode: full , out_size: (200, 13, 13) >
layer6 - Pooling
< downscale: (2, 2) , stride: None , out_size: (200, 6, 6) >
layer7 - fully-connected
< number_of_units: 4000 , drop_rate: 1.0 , act_func: ReLU >
layer8 - fully-connected
< number_of_units: 4000 , drop_rate: 0.5 , act_func: ReLU >
layer9 - fully-connected
< number_of_units: 10 , drop_rate: 0.5 , act_func: softmax >
###########################################
##### start training #####
epoch XE(L) error(L) XE(V) error(V)
1 2.273499 0.851933 2.254003 0.822000
2 2.122519 0.771597 2.168369 0.804000
3 2.032852 0.738992 2.093122 0.786000
4 1.989788 0.720756 2.036462 0.740000
5 1.935234 0.700084 2.021396 0.740000
6 1.911362 0.685798 1.946361 0.716000
7 1.835096 0.648067 1.834889 0.636000
8 1.768207 0.623361 1.879292 0.674000
9 1.749459 0.613445 1.786514 0.616000
10 1.685634 0.585042 1.727242 0.598000
11 1.635825 0.570252 1.654894 0.572000
12 1.619336 0.555462 1.641253 0.552000
13 1.580660 0.547395 1.638332 0.558000
14 1.576853 0.547731 1.541338 0.530000
15 1.524050 0.528655 1.511180 0.538000
16 1.517430 0.523529 1.490757 0.506000
17 1.491088 0.515630 1.536197 0.540000
18 1.444853 0.493445 1.479253 0.528000
19 1.442445 0.488908 1.462722 0.492000
20 1.404088 0.478739 1.484211 0.516000
21 1.377695 0.467311 1.379101 0.484000
22 1.375965 0.467227 1.358642 0.460000
23 1.352322 0.460084 1.282201 0.448000
24 1.321630 0.447395 1.352539 0.454000
25 1.303786 0.445294 1.343048 0.446000
26 1.265659 0.426975 1.277301 0.434000
27 1.254898 0.424034 1.256850 0.442000
28 1.245315 0.420000 1.253533 0.444000
29 1.240361 0.417143 1.178082 0.394000
30 1.199977 0.410672 1.193307 0.400000
31 1.210327 0.407227 1.180300 0.410000
32 1.179978 0.398655 1.209459 0.404000
33 1.156352 0.389328 1.133693 0.392000
34 1.147514 0.388824 1.258485 0.400000
35 1.105322 0.373529 1.074826 0.374000
36 1.118110 0.371429 1.151573 0.386000
37 1.095125 0.366723 1.098699 0.382000
38 1.086520 0.361513 1.043424 0.370000
39 1.067529 0.362269 1.062328 0.384000
40 1.041084 0.351681 1.051441 0.330000
41 1.051023 0.353613 1.056013 0.344000
42 1.023138 0.340756 1.056894 0.362000
43 1.018674 0.342185 1.055755 0.364000
44 0.999104 0.333445 0.984605 0.332000
45 0.976297 0.332185 0.960260 0.332000
46 0.995175 0.331176 0.994270 0.344000
47 0.955052 0.317395 0.932399 0.340000
48 0.954418 0.319496 0.940519 0.324000
49 0.927824 0.312605 0.972910 0.350000
50 0.941172 0.314706 1.058312 0.350000
51 0.921515 0.305042 0.935366 0.314000
52 0.907579 0.303697 0.931843 0.322000
53 0.896925 0.305378 1.008476 0.352000
54 0.884030 0.295294 0.934060 0.326000
55 0.900392 0.302353 0.954390 0.318000
56 0.879308 0.293277 0.997523 0.328000
57 0.886057 0.295630 0.897006 0.314000
58 0.861598 0.288571 0.957117 0.314000
59 0.861517 0.285882 0.871357 0.302000
60 0.853959 0.282185 0.889827 0.304000
}}
*ILSVRC2010 - 100クラス [#i7e9bcb2]
**ex20151207 【CPCPCCCPFFF (学習停滞時に学習係数ηを下げる-その2)】 [#ex20151207]
[[前回>#ex201512051]]よりもηを下げる基準をキツくして実験を行う。最初はバリデーションの最低誤識別率が4epoch連続で下がらなかった場合にηを0.1倍し、一回これを行う毎に次に0.1倍するまでの基準を8epoch、16epochと厳しくしていく。
-[[前回>#ex201512051]、[[前々回>#b3ff8545]]との比較…誤識別率 ( https://drive.google.com/open?id=0B9W18yqQO6JAN2xpcEFEV0JQUkk )、交差エントロピー( https://drive.google.com/open?id=0B9W18yqQO6JAblRjRHdEekR0Yk0 )
-2回目以降のη低下に悪あがき程度の効果しか見られない。実質的な学習は50epochもあれば済んでしまっている。2回目に下げるまでの期間をもっと長くすべきか…？
-[[前回>#ex201512051]]よりもtop1の誤識別率は僅かに良くなっているが、top5および交差エントロピーは微妙に悪くなっている。
-Alexの方法ではテストの交差エントロピーは(おそらく)一切上昇してなかったので、誤識別率よりそちらを見てηを調整していくべきなのかもしれない。
#pre{{
ex201512052123
############# training condition ############
100 epoch training. input_size: (231, 231) , minibatch_size: 128
, learning_rate: 0.01 , momentum: 0.9 , weight_decay: 0.0005
#############################################
########## architecture of the CNN #########
layer1 - Convolution
< kernel_size: (96, 11, 11) , stride: (4, 4) , act_func: ReLU
, border_mode: valid , out_size: (96, 56, 56) , iniW_σ: 0.010000 >
layer2 - Pooling
< downscale: (2, 2) , stride: None , out_size: (96, 28, 28) >
layer3 - Convolution
< kernel_size: (256, 5, 5) , stride: (1, 1) , act_func: ReLU
, border_mode: same , out_size: (256, 28, 28) , iniW_σ: 0.010000 >
layer4 - Pooling
< downscale: (2, 2) , stride: None , out_size: (256, 14, 14) >
layer5 - Convolution
< kernel_size: (384, 3, 3) , stride: (1, 1) , act_func: ReLU
, border_mode: same , out_size: (384, 14, 14) , iniW_σ: 0.010000 >
layer6 - Convolution
< kernel_size: (384, 3, 3) , stride: (1, 1) , act_func: ReLU
, border_mode: same , out_size: (384, 14, 14) , iniW_σ: 0.010000 >
layer7 - Convolution
< kernel_size: (256, 3, 3) , stride: (1, 1) , act_func: ReLU
, border_mode: valid , out_size: (256, 12, 12) , iniW_σ: 0.010000 >
layer8 - Pooling
< downscale: (2, 2) , stride: None , out_size: (256, 6, 6) >
layer9 - fully-connected
< number_of_units: 4096 , drop_rate: 1.0 , act_func: ReLU , iniW_σ: 0.010000 >
layer10 - fully-connected
< number_of_units: 4096 , drop_rate: 0.5 , act_func: ReLU , iniW_σ: 0.010000 >
layer11 - fully-connected
< number_of_units: 100 , drop_rate: 0.5 , act_func: softmax , iniW_σ: 0.010000 >
###########################################
##### CNN initialized #####
##### start training #####
epoch XE(L) error(L) XE(V) error(V) top-5(V) LE(ε)
1 4.584246 0.981340 4.621274 0.989984 0.949920 0.010000
2 4.571351 0.981150 4.632951 0.989984 0.949920 0.010000
3 4.562931 0.981000 4.548244 0.984776 0.919071 0.010000
4 4.264742 0.950954 4.155408 0.935296 0.788862 0.010000
5 3.900478 0.904689 3.675923 0.872196 0.644832 0.010000
6 3.383855 0.817349 3.087507 0.764022 0.471955 0.010000
7 2.952254 0.731412 2.862256 0.714744 0.410056 0.010000
8 2.646342 0.666535 2.549279 0.637821 0.348157 0.010000
9 2.414317 0.617243 2.350138 0.596755 0.304287 0.010000
10 2.219808 0.574021 2.268665 0.581130 0.294471 0.010000
11 2.059470 0.537692 2.050703 0.529647 0.251202 0.010000
12 1.923878 0.504493 1.899849 0.490585 0.223157 0.010000
13 1.796782 0.474954 1.819390 0.474760 0.209936 0.010000
14 1.682050 0.448197 1.810292 0.464343 0.202524 0.010000
15 1.574019 0.423499 1.691068 0.440104 0.188902 0.010000
16 1.481506 0.400719 1.684501 0.437099 0.187300 0.010000
17 1.387527 0.377583 1.671349 0.432091 0.182893 0.010000
18 1.308860 0.359153 1.587274 0.408454 0.169872 0.010000
19 1.223489 0.338909 1.600092 0.411659 0.169271 0.010000
20 1.149080 0.322341 1.624053 0.413662 0.170272 0.010000
21 1.072990 0.303348 1.575932 0.396034 0.164263 0.010000
22 1.013205 0.286804 1.534901 0.388421 0.162861 0.010000
23 0.947663 0.271758 1.573840 0.399439 0.164663 0.010000
24 0.885571 0.254318 1.561206 0.396434 0.157853 0.010000
25 0.819125 0.238416 1.544524 0.385817 0.155048 0.010000
26 0.773512 0.226515 1.555939 0.380609 0.153045 0.010000
27 0.725640 0.212047 1.605225 0.385216 0.160457 0.010000
28 0.682721 0.202650 1.545164 0.380208 0.155248 0.010000
29 0.642045 0.191604 1.585771 0.386819 0.158454 0.010000
30 0.604303 0.181233 1.596926 0.380208 0.157452 0.010000
31 0.581983 0.173301 1.571747 0.379207 0.157452 0.010000
32 0.540536 0.164490 1.631092 0.378205 0.160657 0.010000
33 0.506684 0.152843 1.625295 0.382812 0.153245 0.010000
34 0.486952 0.148889 1.641772 0.377604 0.155849 0.010000
35 0.463988 0.141473 1.660071 0.372596 0.156450 0.010000
36 0.440627 0.134675 1.625053 0.372997 0.155849 0.010000
37 0.413489 0.126371 1.665282 0.374199 0.155649 0.010000
38 0.396151 0.121197 1.629789 0.367588 0.164062 0.010000
39 0.380485 0.116934 1.687949 0.374800 0.155048 0.010000
40 0.364581 0.112632 1.722068 0.372796 0.156651 0.010000
41 0.351045 0.108258 1.689238 0.373397 0.154647 0.010000
42 0.335098 0.103567 1.807631 0.373197 0.160657 0.010000
43 0.215545 0.067333 1.716054 0.351162 0.146635 0.001000
44 0.182056 0.057580 1.730659 0.345753 0.145232 0.001000
45 0.167415 0.052350 1.746588 0.345553 0.143229 0.001000
46 0.154355 0.048618 1.770620 0.343550 0.144631 0.001000
47 0.146220 0.046122 1.805363 0.342348 0.142628 0.001000
48 0.134141 0.043452 1.812618 0.345954 0.144431 0.001000
49 0.132238 0.042596 1.829691 0.343149 0.143630 0.001000
50 0.130362 0.041820 1.841428 0.341146 0.144631 0.001000
51 0.126805 0.039966 1.841386 0.342348 0.144631 0.001000
52 0.119223 0.037929 1.865575 0.343349 0.142829 0.001000
53 0.116597 0.037193 1.839309 0.342147 0.143029 0.001000
54 0.111315 0.035687 1.870408 0.344952 0.142027 0.001000
55 0.109173 0.035125 1.878980 0.342949 0.142829 0.001000
56 0.107391 0.034657 1.918847 0.340946 0.142829 0.001000
57 0.105666 0.034221 1.890377 0.343750 0.141026 0.001000
58 0.104033 0.033714 1.873100 0.339744 0.140625 0.001000
59 0.097992 0.031725 1.902423 0.342348 0.140024 0.001000
60 0.095191 0.031179 1.910197 0.342348 0.140825 0.001000
61 0.091494 0.029713 1.922204 0.341747 0.142027 0.001000
62 0.093082 0.030481 1.922195 0.342949 0.142829 0.001000
63 0.091475 0.029594 1.941977 0.339944 0.139022 0.001000
64 0.085844 0.027867 1.947466 0.341747 0.141627 0.001000
65 0.087982 0.028318 1.964257 0.340545 0.140825 0.001000
66 0.084643 0.027201 1.943214 0.340345 0.139022 0.001000
67 0.080769 0.026599 1.950320 0.339944 0.139223 0.000100
68 0.079915 0.025561 1.952375 0.340345 0.140224 0.000100
69 0.079247 0.025822 1.953625 0.341346 0.140224 0.000100
70 0.079155 0.025513 1.954775 0.342147 0.139223 0.000100
71 0.077192 0.025125 1.957436 0.341346 0.140825 0.000100
72 0.074155 0.024222 1.960327 0.340345 0.139623 0.000100
73 0.075652 0.024412 1.963825 0.341346 0.140024 0.000100
74 0.077644 0.024586 1.953252 0.340144 0.139824 0.000100
75 0.076401 0.024523 1.961390 0.340745 0.139623 0.000100
76 0.075255 0.024396 1.962003 0.338742 0.139623 0.000100
77 0.074594 0.024555 1.962883 0.339343 0.139423 0.000100
78 0.074231 0.023865 1.969501 0.339944 0.140224 0.000100
79 0.076269 0.024927 1.968261 0.340545 0.140825 0.000100
80 0.076875 0.024832 1.965102 0.340545 0.140425 0.000100
81 0.075533 0.024341 1.966429 0.339944 0.140625 0.000100
82 0.073920 0.024000 1.965175 0.339143 0.141627 0.000100
83 0.073000 0.023374 1.973192 0.340545 0.141026 0.000100
84 0.077539 0.025244 1.969003 0.339744 0.141426 0.000100
85 0.073741 0.023794 1.973915 0.337941 0.141226 0.000100
86 0.072854 0.023279 1.970303 0.339744 0.140024 0.000100
87 0.074940 0.024238 1.979327 0.339343 0.140825 0.000100
88 0.070790 0.023113 1.975582 0.339143 0.141426 0.000100
89 0.073999 0.023834 1.977978 0.338542 0.141226 0.000100
90 0.070613 0.022978 1.976582 0.338742 0.140625 0.000100
91 0.075713 0.024674 1.973902 0.339343 0.139824 0.000100
92 0.070920 0.022629 1.978591 0.339944 0.140625 0.000100
93 0.073148 0.023976 1.986253 0.338942 0.141226 0.000100
94 0.069378 0.022812 1.986268 0.339744 0.141026 0.000100
95 0.071320 0.023057 1.985336 0.338341 0.141026 0.000100
96 0.069574 0.022558 1.989404 0.339744 0.140224 0.000100
97 0.073174 0.024016 1.981400 0.340144 0.141426 0.000100
98 0.072548 0.023723 1.983688 0.338742 0.140024 0.000100
99 0.070839 0.023025 1.984810 0.338942 0.141226 0.000100
100 0.071473 0.023057 1.991602 0.339143 0.139824 0.000100
#elapsed time after 100 epoch : 88787.538543 [sec]
real 1480m2.110s
user 2279m24.212s
sys 933m33.988s
}};
**ex20151205-1 【CPCPCCCPFFF (学習停滞時に学習係数ηを下げる)】[#ex201512051]
バリデーションの最低誤識別率がN epoch連続で更新されなかったとき、学習係数ηを0.1倍するようにして実験を行った。Nは3から開始し、ηを下げる毎に3,4,5と増えていく。ηを下げる回数は最大3回までに限定。
-[[η=0.01固定>#b3ff8545]]との比較( https://drive.google.com/open?id=0B9W18yqQO6JAMERycHlwWTBkaUE )。縦線はηを0.1倍したタイミング。
-初回のη減少は明らかに良い効果があった模様。2回目もわずかに効いてるっぽいが、それ以降は下げるタイミングが早すぎた感もあって効果は見られず。
#pre{{
ex201512041855
############# training condition ############
100 epoch training. input_size: (231, 231) , minibatch_size: 128
, learning_rate: 0.01 , momentum: 0.9 , weight_decay: 0.0005
#############################################
########## architecture of the CNN #########
layer1 - Convolution
< kernel_size: (96, 11, 11) , stride: (4, 4) , act_func: ReLU
, border_mode: valid , out_size: (96, 56, 56) , iniW_σ: 0.010000 >
layer2 - Pooling
< downscale: (2, 2) , stride: None , out_size: (96, 28, 28) >
layer3 - Convolution
< kernel_size: (256, 5, 5) , stride: (1, 1) , act_func: ReLU
, border_mode: same , out_size: (256, 28, 28) , iniW_σ: 0.010000 >
layer4 - Pooling
< downscale: (2, 2) , stride: None , out_size: (256, 14, 14) >
layer5 - Convolution
< kernel_size: (384, 3, 3) , stride: (1, 1) , act_func: ReLU
, border_mode: same , out_size: (384, 14, 14) , iniW_σ: 0.010000 >
layer6 - Convolution
< kernel_size: (384, 3, 3) , stride: (1, 1) , act_func: ReLU
, border_mode: same , out_size: (384, 14, 14) , iniW_σ: 0.010000 >
layer7 - Convolution
< kernel_size: (256, 3, 3) , stride: (1, 1) , act_func: ReLU
, border_mode: valid , out_size: (256, 12, 12) , iniW_σ: 0.010000 >
layer8 - Pooling
< downscale: (2, 2) , stride: None , out_size: (256, 6, 6) >
layer9 - fully-connected
< number_of_units: 4096 , drop_rate: 1.0 , act_func: ReLU , iniW_σ: 0.010000 >
layer10 - fully-connected
< number_of_units: 4096 , drop_rate: 0.5 , act_func: ReLU , iniW_σ: 0.010000 >
layer11 - fully-connected
< number_of_units: 100 , drop_rate: 0.5 , act_func: softmax , iniW_σ: 0.010000 >
###########################################
##### CNN initialized #####
##### start training #####
epoch XE(L) error(L) XE(V) error(V) top-5(V) LE(ε)
1 4.584245 0.981332 4.621276 0.989984 0.949920 0.010000
2 4.571351 0.981150 4.632952 0.989984 0.949920 0.010000
3 4.561820 0.980944 4.539330 0.984375 0.916266 0.010000
4 4.265857 0.950471 4.140177 0.933494 0.784255 0.010000
5 3.898917 0.904364 3.662564 0.871194 0.637620 0.010000
6 3.373583 0.814964 3.099520 0.756210 0.468349 0.010000
7 2.948426 0.730524 2.838661 0.703325 0.397436 0.010000
8 2.648571 0.667652 2.562404 0.647035 0.350962 0.010000
9 2.414502 0.616799 2.379750 0.603966 0.306090 0.010000
10 2.217295 0.573498 2.240499 0.574920 0.285657 0.010000
11 2.057729 0.537660 2.036490 0.527244 0.248798 0.010000
12 1.915242 0.501965 1.939623 0.492788 0.232171 0.010000
13 1.787862 0.472569 1.859350 0.485978 0.212740 0.010000
14 1.676458 0.445526 1.782328 0.462941 0.195913 0.010000
15 1.577050 0.422826 1.701397 0.444912 0.191707 0.010000
16 1.477317 0.399816 1.658040 0.434095 0.178285 0.010000
17 1.386620 0.378470 1.642021 0.432692 0.179688 0.010000
18 1.300428 0.357774 1.595300 0.412460 0.171074 0.010000
19 1.222153 0.338624 1.605401 0.413862 0.167869 0.010000
20 1.142201 0.317943 1.588153 0.409856 0.167468 0.010000
21 1.072498 0.302144 1.540821 0.396234 0.159455 0.010000
22 1.007504 0.285616 1.572307 0.397837 0.164263 0.010000
23 0.949187 0.270989 1.543926 0.389623 0.157252 0.010000
24 0.879827 0.253209 1.547989 0.394631 0.165064 0.010000
25 0.819547 0.239066 1.550430 0.383614 0.158854 0.010000
26 0.778321 0.228789 1.578135 0.386819 0.153846 0.010000
27 0.728134 0.213901 1.559589 0.373197 0.151042 0.010000
28 0.685571 0.203093 1.560373 0.377404 0.156851 0.010000
29 0.641609 0.191810 1.578929 0.378405 0.152043 0.010000
30 0.600765 0.179521 1.573359 0.383013 0.161058 0.010000
31 0.415692 0.126696 1.590206 0.356971 0.144030 0.001000
32 0.351503 0.107679 1.595305 0.357171 0.143830 0.001000
33 0.323913 0.099376 1.615260 0.351763 0.142428 0.001000
34 0.308306 0.095255 1.632192 0.347556 0.139022 0.001000
35 0.296006 0.092823 1.640612 0.348758 0.142829 0.001000
36 0.280420 0.087221 1.648830 0.347957 0.140625 0.001000
37 0.267182 0.083885 1.654571 0.351362 0.141226 0.001000
38 0.259883 0.081255 1.674374 0.347356 0.142027 0.001000
39 0.247045 0.077206 1.685620 0.347155 0.140425 0.001000
40 0.240777 0.076002 1.696517 0.348958 0.139824 0.001000
41 0.230222 0.073236 1.715194 0.345152 0.140825 0.001000
42 0.225075 0.071612 1.700519 0.345353 0.140024 0.001000
43 0.212415 0.067801 1.729592 0.344551 0.140425 0.001000
44 0.211383 0.068292 1.764561 0.343950 0.140425 0.001000
45 0.203443 0.063673 1.769579 0.343950 0.141026 0.001000
46 0.200575 0.064275 1.773398 0.342748 0.140024 0.001000
47 0.190406 0.061129 1.785921 0.342748 0.138221 0.001000
48 0.182409 0.058348 1.795420 0.344351 0.138622 0.001000
49 0.186650 0.059893 1.771866 0.344952 0.137821 0.001000
50 0.179944 0.057912 1.800079 0.340745 0.138622 0.001000
51 0.174807 0.056795 1.791111 0.342949 0.138221 0.001000
52 0.167797 0.054489 1.807459 0.344351 0.138622 0.001000
53 0.164205 0.052453 1.833803 0.342748 0.138822 0.001000
54 0.164386 0.053071 1.810848 0.346154 0.140825 0.001000
55 0.151607 0.049688 1.816693 0.340946 0.138021 0.000100
56 0.149181 0.048476 1.828310 0.340345 0.136819 0.000100
57 0.146591 0.046606 1.830688 0.343149 0.136418 0.000100
58 0.148974 0.047723 1.827465 0.342949 0.137420 0.000100
59 0.145522 0.047105 1.827852 0.340946 0.135617 0.000100
60 0.141144 0.045932 1.835013 0.341747 0.136418 0.000100
61 0.141460 0.046122 1.832027 0.341947 0.137821 0.000100
62 0.142015 0.046392 1.835050 0.342748 0.136619 0.000010
63 0.144502 0.046827 1.837035 0.342748 0.136218 0.000010
64 0.139197 0.044324 1.839591 0.343349 0.136418 0.000010
65 0.142461 0.046186 1.839009 0.342949 0.136619 0.000010
66 0.143740 0.046455 1.837929 0.342748 0.136218 0.000010
67 0.141635 0.046083 1.838704 0.342748 0.136619 0.000010
68 0.142662 0.045488 1.839136 0.342748 0.136218 0.000010
69 0.140663 0.045734 1.839786 0.341747 0.136418 0.000010
70 0.140814 0.045861 1.841710 0.342348 0.136418 0.000010
71 0.138502 0.045552 1.842531 0.342147 0.136619 0.000010
72 0.138867 0.044292 1.841794 0.342348 0.135817 0.000010
73 0.142738 0.046233 1.841789 0.342348 0.136018 0.000010
74 0.141308 0.045829 1.841042 0.341747 0.136619 0.000010
75 0.140935 0.045940 1.841952 0.341546 0.136418 0.000010
76 0.142914 0.046423 1.842410 0.341947 0.136018 0.000010
77 0.141811 0.045599 1.842060 0.342748 0.136418 0.000010
78 0.139229 0.044815 1.842689 0.342949 0.136819 0.000010
79 0.142700 0.046463 1.842244 0.341947 0.136819 0.000010
80 0.142348 0.045774 1.843348 0.341947 0.137019 0.000010
Terminated
real 1197m14.062s
user 1840m22.340s
sys 752m1.900s
}};
**ex20151204-1 【CPCPCCCPFFF(eta=0.01)】 [#b3ff8545]
-AlexNetに非常に近い構成。重複プーリングはなし。DataAugmentationで切り取るサイズが調整のため若干大きくなっている。
-得られた汎化性能は、同層数の[[m/2015/okada/diary/2015-11-29#c37b9fbd]]と同程度、Cが1つ少ない[[m/2015/okada/diary/2015-11-29#se03f1b4]]以下。
-学習係数が10倍であるおかげか、上記2つに比べて学習の進行は非常に早い。早さと汎化性能の兼ね合いで学習係数を段階的に下げていくのが適切だと思う。
-1epochの学習に14.5分ほどかかっていることが気になる。このまま1000クラスで行うことを考えると10日と少しかかる計算。
#pre{{
############# training condition ############
150 epoch training. input_size: (231, 231) , minibatch_size: 128
, learning_rate: 0.01 , momentum: 0.9 , weight_decay: 0.0005
#############################################
########## architecture of the CNN #########
layer1 - Convolution
< kernel_size: (96, 11, 11) , stride: (4, 4) , act_func: ReLU
, border_mode: valid , out_size: (96, 56, 56) , iniW_σ: 0.010000 >
layer2 - Pooling
< downscale: (2, 2) , stride: None , out_size: (96, 28, 28) >
layer3 - Convolution
< kernel_size: (256, 5, 5) , stride: (1, 1) , act_func: ReLU
, border_mode: same , out_size: (256, 28, 28) , iniW_σ: 0.010000 >
layer4 - Pooling
< downscale: (2, 2) , stride: None , out_size: (256, 14, 14) >
layer5 - Convolution
< kernel_size: (384, 3, 3) , stride: (1, 1) , act_func: ReLU
, border_mode: same , out_size: (384, 14, 14) , iniW_σ: 0.010000 >
layer6 - Convolution
< kernel_size: (384, 3, 3) , stride: (1, 1) , act_func: ReLU
, border_mode: same , out_size: (384, 14, 14) , iniW_σ: 0.010000 >
layer7 - Convolution
< kernel_size: (256, 3, 3) , stride: (1, 1) , act_func: ReLU
, border_mode: valid , out_size: (256, 12, 12) , iniW_σ: 0.010000 >
layer8 - Pooling
< downscale: (2, 2) , stride: None , out_size: (256, 6, 6) >
layer9 - fully-connected
< number_of_units: 4096 , drop_rate: 1.0 , act_func: ReLU , iniW_σ: 0.010000 >
layer10 - fully-connected
< number_of_units: 4096 , drop_rate: 0.5 , act_func: ReLU , iniW_σ: 0.010000 >
layer11 - fully-connected
< number_of_units: 100 , drop_rate: 0.5 , act_func: softmax , iniW_σ: 0.010000 >
###########################################
##### CNN initialized #####
##### start training #####
epoch XE(L) error(L) XE(V) error(V)
1 4.584245 0.981340 4.621274 0.989984
2 4.571352 0.981150 4.632952 0.989984
3 4.563417 0.981008 4.552187 0.984976
4 4.267677 0.951208 4.143998 0.939503
5 3.898794 0.904840 3.676194 0.875401
6 3.390031 0.819385 3.107175 0.774239
7 2.966760 0.735247 2.856739 0.705529
8 2.660312 0.671606 2.594077 0.648838
9 2.429676 0.621308 2.432343 0.616186
10 2.230778 0.576144 2.252805 0.574519
11 2.055965 0.536606 2.002277 0.515425
12 1.921568 0.504786 1.960560 0.503806
13 1.792159 0.474178 1.854432 0.490184
14 1.676039 0.444465 1.816310 0.463942
15 1.567097 0.420504 1.699283 0.441106
16 1.470421 0.397780 1.673308 0.434095
17 1.382050 0.378043 1.619874 0.415465
18 1.304656 0.358329 1.610165 0.420873
19 1.219439 0.336104 1.589374 0.407853
20 1.136920 0.318593 1.589943 0.402043
21 1.065958 0.300250 1.562457 0.393429
22 1.003505 0.284055 1.550190 0.398237
23 0.943692 0.270514 1.574947 0.396835
24 0.871596 0.251093 1.547367 0.391426
25 0.820359 0.238463 1.542040 0.384415
26 0.772067 0.225976 1.570934 0.379006
27 0.721574 0.212427 1.591359 0.389423
28 0.678220 0.201857 1.560925 0.384615
29 0.636746 0.190154 1.611045 0.384415
30 0.587623 0.176978 1.650638 0.382612
31 0.579415 0.175393 1.616405 0.381210
32 0.540307 0.163199 1.652461 0.379607
33 0.506116 0.153271 1.670217 0.384014
34 0.486215 0.147986 1.648316 0.380809
35 0.462831 0.139904 1.632836 0.375000
36 0.438179 0.133843 1.642773 0.378405
37 0.419966 0.129976 1.642937 0.381611
38 0.396922 0.121339 1.694674 0.372997
39 0.381463 0.117710 1.663578 0.371795
40 0.362087 0.111269 1.650016 0.373197
41 0.344702 0.106103 1.678225 0.367388
42 0.336862 0.104470 1.771848 0.374599
43 0.323751 0.099518 1.753931 0.374599
44 0.311679 0.095945 1.724282 0.375801
45 0.297819 0.092054 1.822172 0.384215
46 0.290038 0.089392 1.768434 0.375801
47 0.277375 0.086579 1.784120 0.369792
48 0.267387 0.082729 1.797198 0.367388
49 0.262976 0.080882 1.814224 0.358974
50 0.257811 0.080050 1.724297 0.362179
51 0.251058 0.077309 1.769808 0.370192
52 0.235611 0.072872 1.829191 0.363582
53 0.231322 0.072040 1.792821 0.365585
54 0.224361 0.069346 1.823692 0.362981
55 0.215018 0.066676 1.814038 0.360978
56 0.212135 0.065503 1.867576 0.363181
57 0.208964 0.065194 1.844547 0.370593
58 0.210177 0.064877 1.875207 0.364583
59 0.195149 0.060416 1.825801 0.363181
60 0.187702 0.059370 1.894739 0.364383
Terminated
real 874m31.305s
user 1387m44.580s
sys 468m21.692s
}};
*1000クラス [#ff5ad5c2]
**20151210start [#vdae7816]
#pre{{
Using gpu device 0: GeForce GTX TITAN X
##### using all categories #####
##### initializing ILSVRC2010_L dir = /IntelSSD750/data/ILSVRC2010
# reading the meta data from /IntelSSD750/data/ILSVRC2010/work/meta20150919.pickle
# ncat = 1000
# ndat = 1261406
##### initializing ILSVRC2010_V dir = /IntelSSD750/data/ILSVRC2010
# ndat = 50000
##### initializing ILSVRC2010_T dir = /IntelSSD750/data/ILSVRC2010
# ndat = 150000
/IntelSSD750/data/ILSVRC2010/train/n07711080/n07711080_32735.JPEG (226, 500, 3) 0 french fries, french-fried potatoes, fries, chips
/IntelSSD750/data/ILSVRC2010/val/ILSVRC2010_val_00000001.JPEG (333, 500, 3) 77 seashore, coast, seacoast, sea-coast
/IntelSSD750/data/ILSVRC2010/test/ILSVRC2010_test_00000001.JPEG (335, 500, 3) 541 thermometer
############# training condition ############
100 epoch training. input_size: (231, 231) , minibatch_size: 128
, learning_rate: 0.01 , momentum: 0.9 , weight_decay: 0.0005
#############################################
########## architecture of the CNN #########
layer1 - Convolution
< kernel_size: (96, 11, 11) , stride: (4, 4) , act_func: ReLU
, border_mode: valid , out_size: (96, 56, 56) , iniW_σ: 0.010000 >
layer2 - Pooling
< downscale: (2, 2) , stride: None , out_size: (96, 28, 28) >
layer3 - Convolution
< kernel_size: (256, 5, 5) , stride: (1, 1) , act_func: ReLU
, border_mode: same , out_size: (256, 28, 28) , iniW_σ: 0.010000 >
layer4 - Pooling
< downscale: (2, 2) , stride: None , out_size: (256, 14, 14) >
layer5 - Convolution
< kernel_size: (384, 3, 3) , stride: (1, 1) , act_func: ReLU
, border_mode: same , out_size: (384, 14, 14) , iniW_σ: 0.010000 >
layer6 - Convolution
< kernel_size: (384, 3, 3) , stride: (1, 1) , act_func: ReLU
, border_mode: same , out_size: (384, 14, 14) , iniW_σ: 0.010000 >
layer7 - Convolution
< kernel_size: (256, 3, 3) , stride: (1, 1) , act_func: ReLU
, border_mode: valid , out_size: (256, 12, 12) , iniW_σ: 0.010000 >
layer8 - Pooling
< downscale: (2, 2) , stride: None , out_size: (256, 6, 6) >
layer9 - fully-connected
< number_of_units: 4096 , drop_rate: 1.0 , act_func: ReLU , iniW_σ: 0.010000 >
layer10 - fully-connected
< number_of_units: 4096 , drop_rate: 0.5 , act_func: ReLU , iniW_σ: 0.010000 >
layer11 - fully-connected
< number_of_units: 1000 , drop_rate: 0.5 , act_func: softmax , iniW_σ: 0.010000 >
###########################################
##### CNN initialized #####
##### start training #####
epoch XE(L) error(L) XE(V) error(V) top-5(V) LE(ε)
1 6.884752 0.997537 6.812517 0.997636 0.988662 0.010000
2 6.159330 0.981330 5.382997 0.952224 0.846575 0.010000
3 4.820724 0.895189 4.237020 0.831370 0.624159 0.010000
4 4.077025 0.808215 3.691015 0.749700 0.512981 0.010000
5 3.647175 0.747388 3.302666 0.697175 0.448558 0.010000
6 3.375887 0.705914 3.151889 0.667468 0.416326 0.010000
7 3.191690 0.676516 3.022518 0.646494 0.391567 0.010000
8 3.063357 0.655295 2.879581 0.625060 0.368450 0.010000
9 2.961918 0.638250 2.768079 0.603145 0.346955 0.010000
10 2.889310 0.625960 2.752685 0.601522 0.344752 0.010000
11 2.830659 0.615426 2.708251 0.587881 0.331470 0.010000
12 2.783030 0.607403 2.669339 0.584996 0.330970 0.010000
13 2.743840 0.600204 2.623211 0.577123 0.321454 0.010000
14 2.711900 0.594776 2.593437 0.572736 0.315505 0.010000
15 2.683638 0.589933 2.612056 0.572015 0.321214 0.010000
16 2.659777 0.585942 2.540284 0.561018 0.308874 0.010000
17 2.640581 0.582593 2.523216 0.554046 0.301743 0.010000
18 2.624222 0.578919 2.563521 0.561078 0.311699 0.010000
19 2.608824 0.576539 2.534935 0.559796 0.308213 0.010000
20 2.594859 0.574089 2.503979 0.554748 0.304367 0.010000
21 2.580792 0.571413 2.487949 0.551262 0.299479 0.010000
22 2.573882 0.569963 2.512330 0.555709 0.302704 0.010000
23 2.560682 0.568329 2.496245 0.552885 0.300741 0.010000
24 2.555082 0.566847 2.489083 0.548918 0.297636 0.010000
25 2.546908 0.565279 2.442227 0.545733 0.293830 0.010000
26 2.539257 0.564075 2.460876 0.547516 0.297055 0.010000
27 2.531815 0.562679 2.475089 0.548858 0.296354 0.010000
28 2.524805 0.561128 2.441060 0.543530 0.290966 0.010000
29 2.519792 0.560358 2.455679 0.540365 0.292107 0.010000
30 2.514656 0.559700 2.432604 0.539583 0.290405 0.010000
31 2.511469 0.559645 2.443431 0.544271 0.294291 0.010000
32 2.504787 0.557393 2.439221 0.539503 0.292388 0.010000
33 2.499973 0.556842 2.421732 0.539183 0.289663 0.010000
34 2.498350 0.556177 2.449258 0.546214 0.293429 0.010000
35 2.493292 0.555558 2.387780 0.530429 0.282732 0.010000
36 2.488641 0.555068 2.428506 0.538381 0.286699 0.010000
37 2.488250 0.554925 2.476041 0.542508 0.293670 0.010000
38 2.483451 0.553555 2.432560 0.540825 0.290665 0.010000
39 2.481457 0.553781 2.433631 0.539323 0.286318 0.010000
40 2.477724 0.552969 2.464033 0.542568 0.294291 0.010000
41 2.477396 0.552845 2.418973 0.536138 0.285938 0.010000
42 2.473406 0.551768 2.433758 0.539864 0.289203 0.010000
43 2.470789 0.552038 2.406371 0.538101 0.285517 0.010000
44 2.468073 0.550811 2.401807 0.535677 0.287240 0.010000
45 2.463922 0.550596 2.415541 0.538401 0.287981 0.010000
46 2.462944 0.549769 2.404986 0.534115 0.283494 0.010000
47 2.463471 0.550736 2.398261 0.532712 0.284856 0.010000
48 2.461038 0.550083 2.380089 0.530529 0.281170 0.010000
49 2.456632 0.549224 2.458168 0.541506 0.294091 0.010000
50 2.456645 0.549156 2.425614 0.539663 0.290585 0.010000
51 1.962958 0.455524 1.978884 0.450581 0.221735 0.001000
52 1.862928 0.435947 1.946839 0.446374 0.215765 0.001000
53 1.823379 0.428164 1.943885 0.444631 0.216306 0.001000
54 1.796986 0.423433 1.926847 0.441506 0.213341 0.001000
55 1.775713 0.419142 1.927886 0.441627 0.213301 0.001000
56 1.760049 0.416159 1.915538 0.438962 0.212640 0.001000
57 1.744787 0.413618 1.908974 0.436939 0.210056 0.001000
58 1.730326 0.410906 1.916117 0.439423 0.211699 0.001000
59 1.718417 0.408489 1.911490 0.439183 0.211218 0.001000
60 1.705688 0.406548 1.903501 0.435236 0.208734 0.001000
61 1.695307 0.404181 1.907159 0.436899 0.210797 0.001000
62 1.684335 0.402614 1.895269 0.433754 0.208454 0.001000
63 1.674639 0.400454 1.887359 0.432953 0.206651 0.001000
64 1.664605 0.398936 1.888616 0.433974 0.207332 0.001000
65 1.654332 0.397447 1.904917 0.434275 0.209155 0.001000
66 1.645336 0.395801 1.914080 0.437139 0.210897 0.001000
67 1.637503 0.394795 1.875011 0.429988 0.206150 0.001000
68 1.627407 0.392400 1.887011 0.432051 0.206170 0.001000
69 1.618667 0.391106 1.898288 0.433994 0.209896 0.001000
70 1.610144 0.389192 1.894694 0.435737 0.207993 0.001000
71 1.602037 0.387736 1.880388 0.431691 0.205649 0.001000
72 1.593667 0.387269 1.885249 0.431530 0.208233 0.001000
73 1.585427 0.385001 1.883168 0.432171 0.205288 0.001000
74 1.578055 0.384353 1.892626 0.433193 0.208033 0.001000
75 1.570413 0.382946 1.883466 0.431130 0.207732 0.001000
76 1.563408 0.381818 1.882975 0.432853 0.207913 0.001000
77 1.556648 0.380900 1.887011 0.431170 0.206951 0.001000
78 1.548848 0.378898 1.877993 0.430108 0.205389 0.001000
79 1.542193 0.378167 1.890434 0.433073 0.206791 0.001000
80 1.536980 0.377049 1.904177 0.435697 0.209475 0.001000
81 1.529383 0.375725 1.889776 0.431891 0.206931 0.001000
82 1.523214 0.374542 1.890501 0.431811 0.208353 0.001000
83 1.515065 0.373358 1.892736 0.432712 0.206911 0.001000
84 1.509030 0.371804 1.899701 0.432372 0.209175 0.001000
85 1.502982 0.370819 1.892647 0.431330 0.206330 0.001000
86 1.498615 0.370613 1.875204 0.426943 0.205148 0.001000
87 1.492169 0.368866 1.896035 0.433133 0.208734 0.001000
88 1.487142 0.368000 1.889295 0.431711 0.208013 0.001000
89 1.482072 0.367115 1.880413 0.429587 0.206370 0.001000
90 1.477492 0.366487 1.904637 0.433133 0.210837 0.001000
91 1.470232 0.365266 1.889451 0.429908 0.206791 0.001000
92 1.466913 0.364462 1.883390 0.430088 0.207572 0.001000
93 1.462341 0.364112 1.885805 0.430168 0.206751 0.001000
94 1.456559 0.362902 1.924685 0.433934 0.211639 0.001000
95 1.451973 0.361927 1.891140 0.428626 0.207772 0.001000
96 1.445586 0.361249 1.890103 0.430929 0.207412 0.001000
97 1.443212 0.360517 1.893173 0.430489 0.207993 0.001000
98 1.439502 0.359880 1.906369 0.433534 0.210056 0.001000
99 1.434970 0.359085 1.884951 0.429407 0.208133 0.001000
100 1.429950 0.357831 1.893761 0.430108 0.208053 0.001000
#elapsed time after 50 epoch : 615972.886903 [sec]
WARNING (theano.gof.cmodule): Removing key file /home/yamada/.theano/compiledir_Linux-3.19--generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpkQlc59/key.pkl because the corresponding module is gone from the file system.
WARNING (theano.gof.cmodule): Removing key file /home/yamada/.theano/compiledir_Linux-3.19--generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpRzh86h/key.pkl because the corresponding module is gone from the file system.
WARNING (theano.gof.cmodule): A module that was loaded by this ModuleCache can no longer be read from file /home/yamada/.theano/compiledir_Linux-3.19--generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmppDLvKe/f90a7a397ede0965c66895e171db1485.so... this could lead to problems.
WARNING (theano.gof.cmodule): Removing key file /home/yamada/.theano/compiledir_Linux-3.19--generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpYfdkIF/key.pkl because the corresponding module is gone from the file system.
WARNING (theano.gof.cmodule): Removing key file /home/yamada/.theano/compiledir_Linux-3.19--generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpNEavvz/key.pkl because the corresponding module is gone from the file system.
WARNING (theano.gof.cmodule): Removing key file /home/yamada/.theano/compiledir_Linux-3.19--generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpTxp6Pk/key.pkl because the corresponding module is gone from the file system.
WARNING (theano.gof.cmodule): Removing key file /home/yamada/.theano/compiledir_Linux-3.19--generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmphXxEHX/key.pkl because the corresponding module is gone from the file system.
WARNING (theano.gof.cmodule): Removing key file /home/yamada/.theano/compiledir_Linux-3.19--generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpbed2uK/key.pkl because the corresponding module is gone from the file system.
WARNING (theano.gof.cmodule): Removing key file /home/yamada/.theano/compiledir_Linux-3.19--generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmp1EDqiq/key.pkl because the corresponding module is gone from the file system.
WARNING (theano.gof.cmodule): Removing key file /home/yamada/.theano/compiledir_Linux-3.19--generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpVPuKT3/key.pkl because the corresponding module is gone from the file system.
WARNING (theano.gof.cmodule): Removing key file /home/yamada/.theano/compiledir_Linux-3.19--generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpaOcUcz/key.pkl because the corresponding module is gone from the file system.
WARNING (theano.gof.cmodule): Removing key file /home/yamada/.theano/compiledir_Linux-3.19--generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmp71Q7GZ/key.pkl because the corresponding module is gone from the file system.
WARNING (theano.gof.cmodule): Removing key file /home/yamada/.theano/compiledir_Linux-3.19--generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmp_mxp3s/key.pkl because the corresponding module is gone from the file system.
WARNING (theano.gof.cmodule): Removing key file /home/yamada/.theano/compiledir_Linux-3.19--generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpK5d4Up/key.pkl because the corresponding module is gone from the file system.
real 10267m53.554s
user 11948m50.384s
sys 7092m16.120s
}};
}}
ページ名:
![[PukiWiki]](image/blackuni3-80x80.png "[PukiWiki]")
