Skip to content

点
点头人工智能课程-v5.0-影像
Commit 471d171f by 靓靓
Options

upload files

parent 18e947f5
Showing with 650 additions and 0 deletions
6-模型训练/.gitkeep 0 → 100644
++ "b/6-\346\250\241\345\236\213\350\256\255\347\273\203/.gitkeep"
6-模型训练/6.2-大模型学习拓展/readme.md 0 → 100644
# 课上补充资料：
# 课上补充资料：
https://github.com/jindongwang/transferlearning
https://github.com/PacktPublishing/Ensemble-Machine-Learning/tree/master
7-模型改进/7.1-影像类数据增强/图像混合.ipynb 0 → 100644
This source diff could not be displayed because it is too large. You can view the blob instead.
7-模型改进/7.1-影像类数据增强/基础方法.ipynb 0 → 100644
This source diff could not be displayed because it is too large. You can view the blob instead.
7-模型改进/7.1-影像类数据增强/擦除方法.ipynb 0 → 100644
This source diff could not be displayed because it is too large. You can view the blob instead.
7-模型改进/7.4-模型参数调优/OC/OC ratio-inputs.csv 0 → 100644
38.3,5,3.4,37.3,72.1,11.9,16,500,10,80
38.3,5,3.4,37.3,72.1,11.9,16,500,10,80
41.7,5.8,3,22.7,64.6,8.7,26,800,20,40
38.3,5,3.4,37.3,72.1,11.9,16,400,10,70
35.7,5.2,3.5,25.4,60.7,4.6,29.5,650,60,10.83333333
20.3,2.7,3.4,27,50.3,3.9,45.8,600,20,90
28,3.4,4,10.3,38,7.9,46.8,575,28.75,20
23.97,3.59,4.27,5.33,37.93,4.03,58.04,650,13,50
41.7,5.8,3,22.7,64.6,8.7,26,700,20,35
34.375,5.35,36.97,2.32,73.775,5.2475,36.94709524,500,15,120
26.638,4.733,17.513,4.355,53.9876699,7.92815534,43.761,500,10,120
41.185,5.59,40.61,1.94,81.605,7.7225,36.94709524,600,15,120
38.3,5,3.4,37.3,72.1,11.9,16,300,10,60
35.5,5.5,5.1,27.2,63,12.1,24.9,700,10,60
30.2,4.3,4.8,21.4,48.8,3.1,38.1,350,35,10
41.7,5.8,3,22.7,64.6,8.7,26,900,20,45
38.1,5,5.8,15,64.8,5.3,29.9,450,11.25,40
37.78,5.47,38.79,2.13,77.69,6.485,36.94709524,700,15,120
35.7,5.2,3.5,25.4,60.7,4.6,29.5,450,5,90
41.185,5.59,40.61,1.94,81.605,7.7225,36.94709524,400,15,120
33.29,5.06,5.14,16.91,52.08,6.26,35.74,550,12.2,45
20.3,2.7,3.4,27,50.3,3.9,45.8,500,20,85
23.97,3.59,4.27,5.33,37.93,4.03,58.04,550,11,50
24.67,4.65,4.51,18.6,51.44,5.68,46.62,500,15,40
34.375,5.35,36.97,2.32,73.775,5.2475,36.94709524,700,15,120
35.7,5.2,3.5,25.4,60.7,4.6,29.5,450,60,7.5
35.8975,5.1575,30.01,3.6525,53.9876699,7.92815534,25.275,500,10,120
26.52,6.24,4.08,20.24,40.3,8.7,42,700,10,60
35.5,5.5,5.1,27.2,63,12.1,24.9,600,10,60
41.185,5.59,40.61,1.94,81.605,7.7225,36.94709524,500,15,120
41.26,5.425,31.3,3.195,53.9876699,7.92815534,18.82,500,10,120
22.3,3.6,2.8,16.6,41,5.7,53.3,500,10,60
46.615,5.6925,32.59,2.7375,53.9876699,7.92815534,12.365,500,10,120
37.78,5.47,38.79,2.13,77.69,6.485,36.94709524,400,15,120
9.1,0.86,1.5,10.2,18.8,2.8,78.4,700,20,95
52.3,8,6.7,32.3,62.3,6.5,31.2,1040,74.3,24
41.7,5.8,3,22.7,64.6,8.7,26,600,20,30
41.61,5.19,5.61,26.01,64.15,15.57,20.28,600,20,30
41.7,5.8,3,22.7,64.6,8.7,26,500,20,25
21.5,3.3,3.6,16.8,43.3,1.9,54.8,600,20,90
21.5,3.3,3.6,16.8,43.3,1.9,54.8,500,20,85
23.97,3.59,4.27,5.33,37.93,4.03,58.04,450,9,50
20.3,2.7,3.4,27,50.3,3.9,45.8,400,20,80
32.074,4.979,22.859,3.565,53.9876699,7.92815534,36.523,500,10,120
41.7,5.8,3,22.7,64.6,8.7,26,400,20,20
37.78,5.47,38.79,2.13,77.69,6.485,36.94709524,600,15,120
17.6,2.5,3.1,16.1,34.1,5.9,60,500,20,85
34.375,5.35,36.97,2.32,73.775,5.2475,36.94709524,400,15,120
9.1,0.86,1.5,10.2,18.8,2.8,78.4,600,20,90
17.6,2.5,3.1,16.1,34.1,5.9,60,400,20,80
33.292,5.102,25.532,3.17,53.9876699,7.92815534,32.904,500,10,120
26.52,6.24,4.08,20.24,40.3,8.7,42,400,10,60
35.5,5.5,5.1,27.2,63,12.1,24.9,500,10,60
24.67,4.65,4.51,18.6,51.44,5.68,46.62,300,15,40
34.375,5.35,36.97,2.32,73.775,5.2475,36.94709524,600,15,120
30.2,4.3,4.8,21.4,48.8,3.1,38.1,450,45,10
22.3,3.6,2.8,16.6,41,5.7,53.3,600,10,60
47.07,3.91,2.29,44.4,70.4,6.21,11.84,600,25,25
25.5,4.5,4.9,25.9,54.2,8.6,37.2,500,50,10
12.79,1.74,1.2,16.22,29.01,3.49,67.5,600,6.666666667,90
17.6,2.5,3.1,16.1,34.1,5.9,60,300,20,75
34.51,5.225,28.205,2.775,53.9876699,7.92815534,29.285,500,10,120
21.5,3.3,3.6,16.8,43.3,1.9,54.8,300,20,75
38.5,5.08,1.33,39.49,62.85,22.74,14.4,500,5.5,30
38.5,5.08,1.33,39.49,62.85,22.74,14.4,600,5.5,30
29.63,5.3,5.11,24.41,60.89,5.04,34.27,300,10,30
20.3,2.7,3.4,27,50.3,3.9,45.8,300,20,75
29.63,5.3,5.11,24.41,60.89,5.04,34.27,350,10,30
35.7,5.2,3.5,25.4,60.7,4.6,29.5,850,60,14.16666667
29.63,5.3,5.11,24.41,60.89,5.04,34.27,500,10,30
24.9,3.36,0.21,44.31,63.5,9.72,26.78,800,10,30
46.615,5.6925,32.59,2.7375,53.9876699,7.92815534,12.365,300,10,120
41.26,5.425,31.3,3.195,53.9876699,7.92815534,18.82,300,10,120
41.7,5.8,3,22.7,64.6,8.7,26,300,20,15
21.5,3.3,3.6,16.8,43.3,1.9,54.8,400,20,80
35.8975,5.1575,30.01,3.6525,53.9876699,7.92815534,25.275,300,10,120
9.1,0.86,1.5,10.2,18.8,2.8,78.4,500,20,85
12.18,5.82,1.26,23.06,27.12,16.65,56.23,900,35,25.71
29.63,5.3,5.11,24.41,60.89,5.04,34.27,450,10,30
29.63,5.3,5.11,24.41,60.89,5.04,34.27,400,10,30
9.1,0.86,1.5,10.2,18.8,2.8,78.4,400,20,80
24.67,4.65,4.51,18.6,51.44,5.68,46.62,700,15,40
47.07,3.91,2.29,44.4,70.4,6.21,11.84,400,25,25
38.5,5.08,1.33,39.49,62.85,22.74,14.4,400,5.5,30
9.1,0.86,1.5,10.2,18.8,2.8,78.4,300,20,75
17.6,2.5,3.1,16.1,34.1,5.9,60,700,20,95
37.78,5.47,38.79,2.13,77.69,6.485,36.94709524,500,15,120
35.8975,5.1575,30.01,3.6525,53.9876699,7.92815534,25.275,700,10,120
12.18,5.82,1.26,23.06,27.12,16.65,56.23,700,35,20
42.1,6.1,6.4,27.3,63.5,11.6,16.6,450,20,52.5
42.1,6.1,6.4,27.3,63.5,11.6,16.6,525,20,56.25
42.1,6.1,6.4,27.3,63.5,11.6,16.6,600,20,60
23.97,3.59,4.27,5.33,37.93,4.03,58.04,750,15,50
20.3,2.7,3.4,27,50.3,3.9,45.8,700,20,95
42.1,6.1,6.4,27.3,63.5,11.6,16.6,375,20,48.75
42.1,6.1,6.4,27.3,63.5,11.6,16.6,600,30,50
41.26,5.425,31.3,3.195,53.9876699,7.92815534,18.82,700,10,120
42.1,6.1,6.4,27.3,63.5,11.6,16.6,450,30,45
42.1,6.1,6.4,27.3,63.5,11.6,16.6,525,30,47.5
36.52,5.33,5.18,23.15,65.07,7.65,27.69,700,10,60
21.5,3.3,3.6,16.8,43.3,1.9,54.8,700,20,95
12.18,5.82,1.26,23.06,27.12,16.65,56.23,500,35,14.28
30.856,4.856,20.186,3.96,53.9876699,7.92815534,40.142,500,10,120
46.615,5.6925,32.59,2.7375,53.9876699,7.92815534,12.365,700,10,120
30.2,4.3,4.8,21.4,48.8,3.1,38.1,400,40,10
41.185,5.59,40.61,1.94,81.605,7.7225,36.94709524,700,15,120
17.6,2.5,3.1,16.1,34.1,5.9,60,600,20,90
42.1,6.1,6.4,27.3,63.5,11.6,16.6,300,10,60
42.1,6.1,6.4,27.3,63.5,11.6,16.6,375,10,67.5
42.1,6.1,6.4,27.3,63.5,11.6,16.6,450,10,75
42.1,6.1,6.4,27.3,63.5,11.6,16.6,525,10,82.5
30.54,2.2,1.44,8.05,23.66,19.36,56.98,850,20.73170732,41
30.54,2.2,1.44,8.05,23.66,19.36,56.98,650,15.85365854,41
42.1,6.1,6.4,27.3,63.5,11.6,16.6,600,10,90
30.54,2.2,1.44,8.05,23.66,19.36,56.98,450,10.97560976,41
42.1,6.1,6.4,27.3,63.5,11.6,16.6,300,20,45
23.97,3.59,4.27,3.53,37.93,4.03,58.04,450,10.97560976,41
23.97,3.59,4.27,3.53,37.93,4.03,58.04,650,15.85365854,41
7-模型改进/7.4-模型参数调优/OC/OC ratio-outputs.csv 0 → 100644
0.1
0.1
0.1
0.1
0.105308219
0.107142857
0.11
0.11
0.11
0.110607434
0.111058601
0.113930267
0.12
0.12
0.13
0.13
0.130239521
0.132173192
0.133333333
0.141719217
0.158350515
0.158505155
0.16
0.16
0.161218092
0.163043478
0.169734513
0.17
0.17
0.174592617
0.179504814
0.18
0.182315668
0.188382412
0.197368421
0.2
0.2
0.219606579
0.22
0.220588235
0.224522293
0.24
0.241189427
0.245472837
0.25
0.250607198
0.263157895
0.265535313
0.27
0.270220588
0.271174377
0.28
0.28
0.29
0.296411856
0.3
0.3
0.3
0.3
0.3
0.315088757
0.318587106
0.32
0.321896
0.328737
0.33
0.331395349
0.36
0.380067568
0.39
0.39
0.390319258
0.395809611
0.41
0.42
0.428995253
0.434210526
0.44
0.46
0.47
0.477272727
0.5
0.5
0.513459
0.564705882
0.6
0.632675847
0.676392573
0.686063218
0.7
0.7
0.7
0.7
0.708661417
0.71
0.74
0.753336203
0.77
0.78
0.8
0.803571429
0.831709477
0.850354314
0.855880729
0.9
0.972177806
0.975
1.29
1.3
1.32
1.36
1.454914722
1.481535649
1.5
1.550840203
1.75
2.415956014
2.948008277
7-模型改进/7.4-模型参数调优/OC/OC ratio.csv 0 → 100644
Inputs,,,,,,,,,,Outputs
Inputs,,,,,,,,,,Outputs
Ultimate analysis (sewage sludge),,Proximate analysis (sewage sludge),,,,,Operating conditions,,,Bio-char
C (wt%),H (wt%),N (wt%),O (wt%),Volatile matter (wt%),Fixed carbon (wt%),Ash (wt%),Temperature (°C),Heating rate (°C/min),Reaction time (min),OC ratio
38.3,5,3.4,37.3,72.1,11.9,16,500,10,80,0.1
41.7,5.8,3,22.7,64.6,8.7,26,800,20,40,0.1
38.3,5,3.4,37.3,72.1,11.9,16,400,10,70,0.1
35.7,5.2,3.5,25.4,60.7,4.6,29.5,650,60,10.83333333,0.105308219
20.3,2.7,3.4,27,50.3,3.9,45.8,600,20,90,0.107142857
28,3.4,4,10.3,38,7.9,46.8,575,28.75,20,0.11
23.97,3.59,4.27,5.33,37.93,4.03,58.04,650,13,50,0.11
41.7,5.8,3,22.7,64.6,8.7,26,700,20,35,0.11
34.375,5.35,26.97,2.32,73.775,5.2475,36.94709524,500,15,120,0.110607434
26.638,4.733,17.513,4.355,53.9876699,7.92815534,43.761,500,10,120,0.111058601
41.185,5.59,30.61,1.94,81.605,7.7225,36.94709524,600,15,120,0.113930267
38.3,5,3.4,37.3,72.1,11.9,16,300,10,60,0.12
35.5,5.5,5.1,27.2,63,12.1,24.9,700,10,60,0.12
30.2,4.3,4.8,21.4,48.8,3.1,38.1,350,35,10,0.13
41.7,5.8,3,22.7,64.6,8.7,26,900,20,45,0.13
38.1,5,5.8,15,64.8,5.3,29.9,450,11.25,40,0.130239521
37.78,5.47,28.79,2.13,77.69,6.485,36.94709524,700,15,120,0.132173192
35.7,5.2,3.5,25.4,60.7,4.6,29.5,450,5,90,0.133333333
41.185,5.59,30.61,1.94,81.605,7.7225,36.94709524,400,15,120,0.141719217
33.29,5.06,5.14,16.91,52.08,6.26,35.74,550,12.2,45,0.158350515
20.3,2.7,3.4,27,50.3,3.9,45.8,500,20,85,0.158505155
23.97,3.59,4.27,5.33,37.93,4.03,58.04,550,11,50,0.16
24.67,4.65,4.51,18.6,51.44,5.68,46.62,500,15,40,0.16
34.375,5.35,26.97,2.32,73.775,5.2475,36.94709524,700,15,120,0.161218092
35.7,5.2,3.5,25.4,60.7,4.6,29.5,450,60,7.5,0.163043478
35.8975,5.1575,20.01,3.6525,53.9876699,7.92815534,25.275,500,10,120,0.169734513
26.52,6.24,4.08,20.24,40.3,8.7,42,700,10,60,0.17
35.5,5.5,5.1,27.2,63,12.1,24.9,600,10,60,0.17
41.185,5.59,20.61,1.94,81.605,7.7225,36.94709524,500,15,120,0.174592617
41.26,5.425,31.3,3.195,53.9876699,7.92815534,18.82,500,10,120,0.179504814
22.3,3.6,2.8,16.6,41,5.7,53.3,500,10,60,0.18
46.615,5.6925,22.59,2.7375,53.9876699,7.92815534,12.365,500,10,120,0.182315668
37.78,5.47,28.79,2.13,77.69,6.485,36.94709524,400,15,120,0.188382412
9.1,0.86,1.5,10.2,18.8,2.8,78.4,700,20,95,0.197368421
52.3,8,6.7,32.3,62.3,6.5,31.2,1040,74.3,24,0.2
41.7,5.8,3,22.7,64.6,8.7,26,600,20,30,0.2
41.61,5.19,5.61,26.01,64.15,15.57,20.28,600,20,30,0.219606579
41.7,5.8,3,22.7,64.6,8.7,26,500,20,25,0.22
21.5,3.3,3.6,16.8,43.3,1.9,54.8,600,20,90,0.220588235
21.5,3.3,3.6,16.8,43.3,1.9,54.8,500,20,85,0.224522293
23.97,3.59,4.27,5.33,37.93,4.03,58.04,450,9,50,0.24
20.3,2.7,3.4,27,50.3,3.9,45.8,400,20,80,0.241189427
32.074,4.979,22.859,3.565,53.9876699,7.92815534,36.523,500,10,120,0.245472837
41.7,5.8,3,22.7,64.6,8.7,26,400,20,20,0.25
37.78,5.47,28.79,2.13,77.69,6.485,36.94709524,600,15,120,0.250607198
17.6,2.5,3.1,16.1,34.1,5.9,60,500,20,85,0.263157895
34.375,5.35,26.97,2.32,73.775,5.2475,36.94709524,400,15,120,0.265535313
9.1,0.86,1.5,10.2,18.8,2.8,78.4,600,20,90,0.27
17.6,2.5,3.1,16.1,34.1,5.9,60,400,20,80,0.270220588
33.292,5.102,25.532,3.17,53.9876699,7.92815534,32.904,500,10,120,0.271174377
26.52,6.24,4.08,20.24,40.3,8.7,42,400,10,60,0.28
35.5,5.5,5.1,27.2,63,12.1,24.9,500,10,60,0.28
24.67,4.65,4.51,18.6,51.44,5.68,46.62,300,15,40,0.29
34.375,5.35,26.97,2.32,73.775,5.2475,36.94709524,600,15,120,0.296411856
30.2,4.3,4.8,21.4,48.8,3.1,38.1,450,45,10,0.3
22.3,3.6,2.8,16.6,41,5.7,53.3,600,10,60,0.3
47.07,3.91,2.29,44.4,70.4,6.21,11.84,600,25,25,0.3
25.5,4.5,4.9,25.9,54.2,8.6,37.2,500,50,10,0.3
12.79,1.74,1.2,16.22,29.01,3.49,67.5,600,6.666666667,90,0.3
17.6,2.5,3.1,16.1,34.1,5.9,60,300,20,75,0.315088757
34.51,5.225,28.205,2.775,53.9876699,7.92815534,29.285,500,10,120,0.318587106
21.5,3.3,3.6,16.8,43.3,1.9,54.8,300,20,75,0.32
38.5,5.08,1.33,39.49,62.85,22.74,14.4,500,5.5,30,0.321896
38.5,5.08,1.33,39.49,62.85,22.74,14.4,600,5.5,30,0.328737
29.63,5.3,5.11,24.41,60.89,5.04,34.27,300,10,30,0.33
20.3,2.7,3.4,27,50.3,3.9,45.8,300,20,75,0.331395349
29.63,5.3,5.11,24.41,60.89,5.04,34.27,350,10,30,0.36
35.7,5.2,3.5,25.4,60.7,4.6,29.5,850,60,14.16666667,0.380067568
29.63,5.3,5.11,24.41,60.89,5.04,34.27,500,10,30,0.39
24.9,3.36,0.21,44.31,63.5,9.72,26.78,800,10,30,0.39
46.615,5.6925,22.59,2.7375,53.9876699,7.92815534,12.365,300,10,120,0.390319258
41.26,5.425,31.3,3.195,53.9876699,7.92815534,18.82,300,10,120,0.395809611
41.7,5.8,3,22.7,64.6,8.7,26,300,20,15,0.41
21.5,3.3,3.6,16.8,43.3,1.9,54.8,400,20,80,0.42
35.8975,5.1575,20.01,3.6525,53.9876699,7.92815534,25.275,300,10,120,0.428995253
9.1,0.86,1.5,10.2,18.8,2.8,78.4,500,20,85,0.434210526
12.18,5.82,1.26,23.06,27.12,16.65,56.23,900,35,25.71,0.44
29.63,5.3,5.11,24.41,60.89,5.04,34.27,450,10,30,0.46
29.63,5.3,5.11,24.41,60.89,5.04,34.27,400,10,30,0.47
9.1,0.86,1.5,10.2,18.8,2.8,78.4,400,20,80,0.477272727
24.67,4.65,4.51,18.6,51.44,5.68,46.62,700,15,40,0.5
47.07,3.91,2.29,44.4,70.4,6.21,11.84,400,25,25,0.5
38.5,5.08,1.33,39.49,62.85,22.74,14.4,400,5.5,30,0.513459
9.1,0.86,1.5,10.2,18.8,2.8,78.4,300,20,75,0.564705882
17.6,2.5,3.1,16.1,34.1,5.9,60,700,20,95,0.6
37.78,5.47,28.79,2.13,77.69,6.485,36.94709524,500,15,120,0.632675847
35.8975,5.1575,30.01,3.6525,53.9876699,7.92815534,25.275,700,10,120,0.676392573
12.18,5.82,1.26,23.06,27.12,16.65,56.23,700,35,20,0.686063218
42.1,6.1,6.4,27.3,63.5,11.6,16.6,450,20,52.5,0.7
42.1,6.1,6.4,27.3,63.5,11.6,16.6,525,20,56.25,0.7
42.1,6.1,6.4,27.3,63.5,11.6,16.6,600,20,60,0.7
23.97,3.59,4.27,5.33,37.93,4.03,58.04,750,15,50,0.7
20.3,2.7,3.4,27,50.3,3.9,45.8,700,20,95,0.708661417
42.1,6.1,6.4,27.3,63.5,11.6,16.6,375,20,48.75,0.71
42.1,6.1,6.4,27.3,63.5,11.6,16.6,600,30,50,0.74
41.26,5.425,31.3,3.195,53.9876699,7.92815534,18.82,700,10,120,0.753336203
42.1,6.1,6.4,27.3,63.5,11.6,16.6,450,30,45,0.77
42.1,6.1,6.4,27.3,63.5,11.6,16.6,525,30,47.5,0.78
36.52,5.33,5.18,23.15,65.07,7.65,27.69,700,10,60,0.8
21.5,3.3,3.6,16.8,43.3,1.9,54.8,700,20,95,0.803571429
12.18,5.82,1.26,23.06,27.12,16.65,56.23,500,35,14.28,0.831709477
30.856,4.856,20.186,3.96,53.9876699,7.92815534,40.142,500,10,120,0.850354314
46.615,5.6925,32.59,2.7375,53.9876699,7.92815534,12.365,700,10,120,0.855880729
30.2,4.3,4.8,21.4,48.8,3.1,38.1,400,40,10,0.9
41.185,5.59,30.61,1.94,81.605,7.7225,36.94709524,700,15,120,0.972177806
17.6,2.5,3.1,16.1,34.1,5.9,60,600,20,90,0.975
42.1,6.1,6.4,27.3,63.5,11.6,16.6,300,10,60,1.29
42.1,6.1,6.4,27.3,63.5,11.6,16.6,375,10,67.5,1.3
42.1,6.1,6.4,27.3,63.5,11.6,16.6,450,10,75,1.32
42.1,6.1,6.4,27.3,63.5,11.6,16.6,525,10,82.5,1.36
30.54,2.2,1.44,8.05,23.66,19.36,56.98,850,20.73170732,41,1.454914722
30.54,2.2,1.44,8.05,23.66,19.36,56.98,650,15.85365854,41,1.481535649
42.1,6.1,6.4,27.3,63.5,11.6,16.6,600,10,90,1.5
30.54,2.2,1.44,8.05,23.66,19.36,56.98,450,10.97560976,41,1.550840203
42.1,6.1,6.4,27.3,63.5,11.6,16.6,300,20,45,1.75
23.97,3.59,4.27,3.53,37.93,4.03,58.04,450,10.97560976,41,2.415956014
23.97,3.59,4.27,3.53,37.93,4.03,58.04,650,15.85365854,41,2.948008277
,,,,,,,,,,
,,,,,,,,,,
52.3,8,8.948663793,44.4,81.605,22.74,78.4,1040,74.3,120,
9.1,0.86,0.21,1.94,18.8,1.9,11.84,300,5,7.5,
,,,,,,,,,,
,,,,,,,,,,
9.1-52.3,0.86-8,0.21-8.95,1.94-44.4,18.8-81.605,1.9-22.74,11.84-78.4,300-800,5-74,,
7-模型改进/7.4-模型参数调优/hyp_tune_optuna.ipynb 0 → 100644
{
{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"# 文本分类实例"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step1 导入相关包"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from transformers import AutoTokenizer, AutoModelForSequenceClassification, Trainer, TrainingArguments\n",
"from datasets import load_dataset"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step2 加载数据集"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"dataset = load_dataset(\"csv\", data_files=\"./ChnSentiCorp_htl_all.csv\", split=\"train\")\n",
"dataset = dataset.filter(lambda x: x[\"review\"] is not None)\n",
"dataset"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step3 划分数据集"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"datasets = dataset.train_test_split(test_size=0.1)\n",
"datasets"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step4 数据集预处理"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import torch\n",
"\n",
"tokenizer = AutoTokenizer.from_pretrained(\"hfl/rbt3\")\n",
"\n",
"def process_function(examples):\n",
" tokenized_examples = tokenizer(examples[\"review\"], max_length=128, truncation=True)\n",
" tokenized_examples[\"labels\"] = examples[\"label\"]\n",
" return tokenized_examples\n",
"\n",
"tokenized_datasets = datasets.map(process_function, batched=True, remove_columns=datasets[\"train\"].column_names)\n",
"tokenized_datasets"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step5 创建模型"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def model_init():\n",
" model = AutoModelForSequenceClassification.from_pretrained(\"hfl/rbt3\")\n",
" return model"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step6 创建评估函数"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import evaluate\n",
"\n",
"acc_metric = evaluate.load(\"accuracy\")\n",
"f1_metirc = evaluate.load(\"f1\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def eval_metric(eval_predict):\n",
" predictions, labels = eval_predict\n",
" predictions = predictions.argmax(axis=-1)\n",
" acc = acc_metric.compute(predictions=predictions, references=labels)\n",
" f1 = f1_metirc.compute(predictions=predictions, references=labels)\n",
" acc.update(f1)\n",
" return acc"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step7 创建TrainingArguments"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"train_args = TrainingArguments(output_dir=\"./checkpoints\", # 输出文件夹\n",
" per_device_train_batch_size=64, # 训练时的batch_size\n",
" per_device_eval_batch_size=128, # 验证时的batch_size\n",
" logging_steps=500, # log 打印的频率\n",
" evaluation_strategy=\"epoch\", # 评估策略\n",
" save_strategy=\"epoch\", # 保存策略\n",
" save_total_limit=3, # 最大保存数\n",
" learning_rate=2e-5, # 学习率\n",
" weight_decay=0.01, # weight_decay\n",
" metric_for_best_model=\"f1\", # 设定评估指标\n",
" load_best_model_at_end=True) # 训练完成后加载最优模型"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step8 创建Trainer"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from transformers import DataCollatorWithPadding\n",
"trainer = Trainer(model_init=model_init, \n",
" args=train_args, \n",
" train_dataset=tokenized_datasets[\"train\"], \n",
" eval_dataset=tokenized_datasets[\"test\"], \n",
" data_collator=DataCollatorWithPadding(tokenizer=tokenizer),\n",
" compute_metrics=eval_metric)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step9 模型训练"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# trainer.train()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step9 模型训练（自动搜索）"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def default_hp_space_optuna(trial):\n",
" return {\n",
" \"learning_rate\": trial.suggest_float(\"learning_rate\", 1e-6, 1e-4),\n",
" \"num_train_epochs\": trial.suggest_int(\"num_train_epochs\", 1, 5),\n",
" \"seed\": trial.suggest_int(\"seed\", 1, 40),\n",
" \"per_device_train_batch_size\": trial.suggest_categorical(\"per_device_train_batch_size\", [4, 8, 16, 32, 64]),\n",
" \"optim\": trial.suggest_categorical(\"optim\", [\"sgd\", \"adamw_hf\"]),\n",
" }\n",
"\n",
"trainer.hyperparameter_search(hp_space=default_hp_space_optuna, compute_objective=lambda x: x[\"eval_f1\"], direction=\"maximize\", n_trials=10)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "transformers",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.16"
},
"orig_nbformat": 4
},
"nbformat": 4,
"nbformat_minor": 2
}
7-模型改进/7.4-模型参数调优/optuna.ipynb 0 → 100644
This source diff could not be displayed because it is too large. You can view the blob instead.
7-模型改进/7.4-模型参数调优/optuna_DL.ipynb 0 → 100644
This source diff could not be displayed because it is too large. You can view the blob instead.
7-模型改进/7.6-模型训练策略创新/各种学习策略.ipynb 0 → 100644
This source diff could not be displayed because it is too large. You can view the blob instead.
7-模型改进/7.7-影像数据模型优化/readme.md 0 → 100644
# 图像模型优化补充资料
# 图像模型优化补充资料
## 下载链接
请通过以下链接下载补充资料：
- **链接**: [https://pan.baidu.com/s/1EQuRDiDl57kFgGYLNtTh8w?pwd=q2cj](https://pan.baidu.com/s/1EQuRDiDl57kFgGYLNtTh8w?pwd=q2cj)
- **提取码**: q2cj
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment