bigscience-bloom-3b
Find this model in the BLOOM model summary
bigscience-bloom-3b Model Summary Plots
bigscience-bloom-3b Model Selected Details
layer_type
N
M
Q
alpha
D
alpha-hat
log_SN
% Rand
num_traps
num_fingers
rank_loss
layer_id
4
DENSE
7680
2560
3.00
3.40
0.02
5.85
1.72
83.81
0
0
0
5
DENSE
2560
2560
1.00
3.35
0.04
6.25
1.86
80.95
16
0
3
7
DENSE
10240
2560
4.00
5.30
0.05
11.63
2.19
88.42
1
0
0
8
DENSE
10240
2560
4.00
4.51
0.03
9.59
2.13
85.38
9
0
0
10
DENSE
7680
2560
3.00
3.71
0.02
6.44
1.74
86.05
0
0
0
11
DENSE
2560
2560
1.00
3.72
0.03
5.99
1.61
85.01
7
0
4
13
DENSE
10240
2560
4.00
2.78
0.03
5.81
2.09
84.07
1
0
0
14
DENSE
10240
2560
4.00
4.80
0.02
9.10
1.90
86.26
0
0
0
16
DENSE
7680
2560
3.00
3.43
0.02
5.72
1.67
86.14
0
0
0
17
DENSE
2560
2560
1.00
3.81
0.02
5.95
1.56
85.55
3
0
3
19
DENSE
10240
2560
4.00
2.76
0.03
5.62
2.04
82.88
0
0
0
20
DENSE
10240
2560
4.00
4.83
0.03
8.63
1.79
86.52
1
0
0
22
DENSE
7680
2560
3.00
3.38
0.02
5.67
1.68
86.17
0
0
0
23
DENSE
2560
2560
1.00
4.10
0.02
5.42
1.32
87.88
4
0
3
25
DENSE
10240
2560
4.00
2.95
0.03
5.89
2.00
82.71
0
0
0
26
DENSE
10240
2560
4.00
4.38
0.03
7.43
1.70
85.72
0
0
0
28
DENSE
7680
2560
3.00
3.38
0.04
5.52
1.63
85.62
0
0
0
29
DENSE
2560
2560
1.00
4.30
0.02
5.35
1.24
89.15
0
0
4
31
DENSE
10240
2560
4.00
3.36
0.01
6.45
1.92
82.58
0
0
0
32
DENSE
10240
2560
4.00
4.19
0.02
6.94
1.65
84.48
1
0
0
34
DENSE
7680
2560
3.00
3.52
0.03
6.03
1.71
85.92
0
0
0
35
DENSE
2560
2560
1.00
4.29
0.03
5.71
1.33
88.74
2
0
4
37
DENSE
10240
2560
4.00
3.41
0.01
6.34
1.86
82.55
0
0
0
38
DENSE
10240
2560
4.00
4.36
0.02
7.19
1.65
83.70
10
0
0
40
DENSE
7680
2560
3.00
3.56
0.03
6.02
1.69
85.99
0
0
0
41
DENSE
2560
2560
1.00
4.43
0.02
6.37
1.44
88.77
2
0
3
43
DENSE
10240
2560
4.00
3.27
0.02
7.80
2.39
78.55
0
0
0
44
DENSE
10240
2560
4.00
3.92
0.03
7.93
2.03
83.98
11
0
0
46
DENSE
7680
2560
3.00
3.65
0.03
6.07
1.67
85.91
0
0
0
47
DENSE
2560
2560
1.00
4.29
0.02
5.48
1.28
88.62
0
0
3
49
DENSE
10240
2560
4.00
3.28
0.01
7.51
2.29
79.32
0
0
0
50
DENSE
10240
2560
4.00
3.86
0.02
8.18
2.12
83.58
9
0
0
52
DENSE
7680
2560
3.00
3.50
0.03
5.70
1.63
85.70
0
0
0
53
DENSE
2560
2560
1.00
3.41
0.08
3.44
1.01
89.54
0
4
4
55
DENSE
10240
2560
4.00
3.32
0.01
6.87
2.07
82.84
1
0
0
56
DENSE
10240
2560
4.00
3.72
0.02
6.82
1.83
86.22
6
0
0
58
DENSE
7680
2560
3.00
3.41
0.04
5.56
1.63
85.44
0
0
0
59
DENSE
2560
2560
1.00
3.22
0.07
3.35
1.04
88.24
0
1
4
61
DENSE
10240
2560
4.00
3.28
0.01
6.55
2.00
85.18
0
0
0
62
DENSE
10240
2560
4.00
3.74
0.03
5.74
1.54
88.20
4
0
0
64
DENSE
7680
2560
3.00
2.64
0.05
4.31
1.63
82.04
0
0
0
65
DENSE
2560
2560
1.00
3.43
0.07
3.46
1.01
89.77
0
1
3
67
DENSE
10240
2560
4.00
3.39
0.01
7.13
2.10
86.71
0
0
0
68
DENSE
10240
2560
4.00
3.75
0.03
6.36
1.70
88.04
0
0
0
70
DENSE
7680
2560
3.00
2.37
0.06
3.78
1.59
79.43
0
0
0
71
DENSE
2560
2560
1.00
3.74
0.06
3.09
0.83
86.48
0
0
3
73
DENSE
10240
2560
4.00
3.66
0.03
7.73
2.11
87.03
0
0
0
74
DENSE
10240
2560
4.00
4.53
0.04
7.66
1.69
87.76
0
0
0
76
DENSE
7680
2560
3.00
2.29
0.06
3.71
1.62
76.55
0
0
0
77
DENSE
2560
2560
1.00
3.75
0.05
2.75
0.73
84.56
0
0
4
79
DENSE
10240
2560
4.00
3.55
0.02
6.99
1.97
85.75
0
0
0
80
DENSE
10240
2560
4.00
3.49
0.03
4.82
1.38
88.08
0
0
0
82
DENSE
7680
2560
3.00
2.41
0.08
3.81
1.58
80.25
0
7
0
83
DENSE
2560
2560
1.00
4.33
0.06
3.16
0.73
89.12
0
0
4
85
DENSE
10240
2560
4.00
3.29
0.03
6.44
1.96
84.38
0
0
0
86
DENSE
10240
2560
4.00
3.58
0.03
4.84
1.35
88.93
0
0
0
88
DENSE
7680
2560
3.00
3.03
0.07
4.72
1.56
79.01
0
0
0
89
DENSE
2560
2560
1.00
3.52
0.06
2.63
0.75
87.99
0
0
3
91
DENSE
10240
2560
4.00
3.76
0.04
7.86
2.09
84.37
0
0
0
92
DENSE
10240
2560
4.00
4.06
0.05
5.08
1.25
90.92
0
0
0
94
DENSE
7680
2560
3.00
3.10
0.06
4.90
1.58
80.46
0
0
0
95
DENSE
2560
2560
1.00
3.70
0.06
2.97
0.80
87.22
0
0
4
97
DENSE
10240
2560
4.00
3.25
0.03
7.02
2.16
83.80
0
0
0
98
DENSE
10240
2560
4.00
4.02
0.06
4.95
1.23
89.83
0
0
0
100
DENSE
7680
2560
3.00
3.04
0.05
4.68
1.54
82.26
0
0
0
101
DENSE
2560
2560
1.00
3.11
0.07
2.57
0.83
85.92
0
0
3
103
DENSE
10240
2560
4.00
3.20
0.03
7.05
2.20
82.93
0
0
0
104
DENSE
10240
2560
4.00
3.55
0.05
4.61
1.30
88.47
0
0
0
106
DENSE
7680
2560
3.00
3.24
0.04
5.04
1.55
84.34
0
0
0
107
DENSE
2560
2560
1.00
4.29
0.07
3.20
0.75
89.66
0
0
3
109
DENSE
10240
2560
4.00
3.13
0.03
6.90
2.20
82.56
0
0
0
110
DENSE
10240
2560
4.00
3.41
0.05
4.63
1.36
88.22
0
0
0
112
DENSE
7680
2560
3.00
3.20
0.05
4.97
1.55
84.08
0
0
0
113
DENSE
2560
2560
1.00
3.54
0.08
2.87
0.81
87.07
0
1
4
115
DENSE
10240
2560
4.00
3.20
0.03
7.05
2.20
83.39
0
0
0
116
DENSE
10240
2560
4.00
3.68
0.04
5.30
1.44
88.26
0
0
0
118
DENSE
7680
2560
3.00
3.06
0.05
4.74
1.55
82.78
0
0
0
119
DENSE
2560
2560
1.00
3.14
0.03
2.73
0.87
84.63
0
0
4
121
DENSE
10240
2560
4.00
3.27
0.03
7.25
2.22
84.38
0
0
0
122
DENSE
10240
2560
4.00
3.63
0.03
5.43
1.50
88.73
0
0
0
124
DENSE
7680
2560
3.00
3.24
0.04
4.94
1.52
86.66
0
0
0
125
DENSE
2560
2560
1.00
3.56
0.10
2.90
0.82
87.84
0
2
3
127
DENSE
10240
2560
4.00
3.38
0.02
7.48
2.21
85.05
0
0
0
128
DENSE
10240
2560
4.00
4.10
0.03
6.42
1.57
90.22
0
0
0
130
DENSE
7680
2560
3.00
3.40
0.04
5.31
1.56
86.70
0
0
0
131
DENSE
2560
2560
1.00
5.95
0.03
6.59
1.11
91.74
0
0
4
133
DENSE
10240
2560
4.00
3.38
0.02
7.55
2.23
85.12
0
0
0
134
DENSE
10240
2560
4.00
4.56
0.03
7.67
1.68
90.04
0
0
0
136
DENSE
7680
2560
3.00
3.67
0.04
5.76
1.57
87.10
0
0
0
137
DENSE
2560
2560
1.00
6.64
0.05
7.49
1.13
89.96
0
0
3
139
DENSE
10240
2560
4.00
3.44
0.02
7.68
2.23
85.40
0
0
0
140
DENSE
10240
2560
4.00
4.89
0.03
7.88
1.61
90.90
0
0
0
142
DENSE
7680
2560
3.00
3.86
0.04
5.91
1.53
88.76
0
0
0
143
DENSE
2560
2560
1.00
5.39
0.03
7.90
1.47
89.17
0
0
3
145
DENSE
10240
2560
4.00
3.43
0.01
7.71
2.24
85.50
0
0
0
146
DENSE
10240
2560
4.00
5.80
0.04
9.31
1.60
91.62
1
0
0
148
DENSE
7680
2560
3.00
3.98
0.02
6.42
1.61
89.85
0
0
0
149
DENSE
2560
2560
1.00
3.87
0.04
6.42
1.66
87.24
0
0
3
151
DENSE
10240
2560
4.00
3.33
0.01
7.56
2.27
84.92
0
0
0
152
DENSE
10240
2560
4.00
4.91
0.03
8.29
1.69
92.51
1
0
0
154
DENSE
7680
2560
3.00
3.88
0.03
6.10
1.57
88.58
0
0
0
155
DENSE
2560
2560
1.00
3.69
0.06
6.60
1.79
85.19
4
0
3
157
DENSE
10240
2560
4.00
3.19
0.01
7.28
2.28
84.32
0
0
0
158
DENSE
10240
2560
4.00
4.92
0.02
8.53
1.73
92.84
0
0
0
160
DENSE
7680
2560
3.00
3.60
0.02
5.76
1.60
87.77
0
0
0
161
DENSE
2560
2560
1.00
3.05
0.08
6.05
1.98
81.38
22
0
2
163
DENSE
10240
2560
4.00
3.15
0.01
7.21
2.29
84.13
0
0
0
164
DENSE
10240
2560
4.00
4.54
0.01
8.21
1.81
92.35
3
0
0
166
DENSE
7680
2560
3.00
3.57
0.02
5.82
1.63
87.13
0
0
0
167
DENSE
2560
2560
1.00
2.03
0.05
4.69
2.31
74.82
39
0
3
169
DENSE
10240
2560
4.00
3.12
0.01
7.38
2.36
83.68
0
0
0
170
DENSE
10240
2560
4.00
4.13
0.03
10.78
2.61
86.99
4
0
0
172
DENSE
7680
2560
3.00
3.40
0.01
6.40
1.88
86.50
0
0
0
173
DENSE
2560
2560
1.00
2.07
0.05
4.38
2.12
73.50
36
0
3
175
DENSE
10240
2560
4.00
3.21
0.01
7.49
2.33
84.57
0
0
0
176
DENSE
10240
2560
4.00
4.31
0.03
10.29
2.39
88.27
1
0
0
178
DENSE
7680
2560
3.00
3.18
0.01
5.95
1.87
85.24
0
0
0
179
DENSE
2560
2560
1.00
2.75
0.04
6.26
2.27
71.82
35
0
3
181
DENSE
10240
2560
4.00
3.46
0.02
7.88
2.28
85.31
0
0
0
182
DENSE
10240
2560
4.00
4.48
0.04
9.68
2.16
89.53
4
0
0
bigscience-bloom-3b Layer Plots
Layer 4
Layer=4 | N=7680 | M=2560 | Q=3.00 | alpha=3.40 | D_ks=0.02 | alpha-hat=5.85 | num traps=0
Layer 5
Layer=5 | N=2560 | M=2560 | Q=1.00 | alpha=3.35 | D_ks=0.04 | alpha-hat=6.25 | num traps=16
Layer 7
Layer=7 | N=10240 | M=2560 | Q=4.00 | alpha=5.30 | D_ks=0.05 | alpha-hat=11.63 | num traps=1
Layer 8
Layer=8 | N=10240 | M=2560 | Q=4.00 | alpha=4.51 | D_ks=0.03 | alpha-hat=9.59 | num traps=9
Layer 10
Layer=10 | N=7680 | M=2560 | Q=3.00 | alpha=3.71 | D_ks=0.02 | alpha-hat=6.44 | num traps=0
Layer 11
Layer=11 | N=2560 | M=2560 | Q=1.00 | alpha=3.72 | D_ks=0.03 | alpha-hat=5.99 | num traps=7
Layer 13
Layer=13 | N=10240 | M=2560 | Q=4.00 | alpha=2.78 | D_ks=0.03 | alpha-hat=5.81 | num traps=1
Layer 14
Layer=14 | N=10240 | M=2560 | Q=4.00 | alpha=4.80 | D_ks=0.02 | alpha-hat=9.10 | num traps=0
Layer 16
Layer=16 | N=7680 | M=2560 | Q=3.00 | alpha=3.43 | D_ks=0.02 | alpha-hat=5.72 | num traps=0
Layer 17
Layer=17 | N=2560 | M=2560 | Q=1.00 | alpha=3.81 | D_ks=0.02 | alpha-hat=5.95 | num traps=3
Layer 19
Layer=19 | N=10240 | M=2560 | Q=4.00 | alpha=2.76 | D_ks=0.03 | alpha-hat=5.62 | num traps=0
Layer 20
Layer=20 | N=10240 | M=2560 | Q=4.00 | alpha=4.83 | D_ks=0.03 | alpha-hat=8.63 | num traps=1
Layer 22
Layer=22 | N=7680 | M=2560 | Q=3.00 | alpha=3.38 | D_ks=0.02 | alpha-hat=5.67 | num traps=0
Layer 23
Layer=23 | N=2560 | M=2560 | Q=1.00 | alpha=4.10 | D_ks=0.02 | alpha-hat=5.42 | num traps=4
Layer 25
Layer=25 | N=10240 | M=2560 | Q=4.00 | alpha=2.95 | D_ks=0.03 | alpha-hat=5.89 | num traps=0
Layer 26
Layer=26 | N=10240 | M=2560 | Q=4.00 | alpha=4.38 | D_ks=0.03 | alpha-hat=7.43 | num traps=0
Layer 28
Layer=28 | N=7680 | M=2560 | Q=3.00 | alpha=3.38 | D_ks=0.04 | alpha-hat=5.52 | num traps=0
Layer 29
Layer=29 | N=2560 | M=2560 | Q=1.00 | alpha=4.30 | D_ks=0.02 | alpha-hat=5.35 | num traps=0
Layer 31
Layer=31 | N=10240 | M=2560 | Q=4.00 | alpha=3.36 | D_ks=0.01 | alpha-hat=6.45 | num traps=0
Layer 32
Layer=32 | N=10240 | M=2560 | Q=4.00 | alpha=4.19 | D_ks=0.02 | alpha-hat=6.94 | num traps=1
Layer 34
Layer=34 | N=7680 | M=2560 | Q=3.00 | alpha=3.52 | D_ks=0.03 | alpha-hat=6.03 | num traps=0
Layer 35
Layer=35 | N=2560 | M=2560 | Q=1.00 | alpha=4.29 | D_ks=0.03 | alpha-hat=5.71 | num traps=2
Layer 37
Layer=37 | N=10240 | M=2560 | Q=4.00 | alpha=3.41 | D_ks=0.01 | alpha-hat=6.34 | num traps=0
Layer 38
Layer=38 | N=10240 | M=2560 | Q=4.00 | alpha=4.36 | D_ks=0.02 | alpha-hat=7.19 | num traps=10
Layer 40
Layer=40 | N=7680 | M=2560 | Q=3.00 | alpha=3.56 | D_ks=0.03 | alpha-hat=6.02 | num traps=0
Layer 41
Layer=41 | N=2560 | M=2560 | Q=1.00 | alpha=4.43 | D_ks=0.02 | alpha-hat=6.37 | num traps=2
Layer 43
Layer=43 | N=10240 | M=2560 | Q=4.00 | alpha=3.27 | D_ks=0.02 | alpha-hat=7.80 | num traps=0
Layer 44
Layer=44 | N=10240 | M=2560 | Q=4.00 | alpha=3.92 | D_ks=0.03 | alpha-hat=7.93 | num traps=11
Layer 46
Layer=46 | N=7680 | M=2560 | Q=3.00 | alpha=3.65 | D_ks=0.03 | alpha-hat=6.07 | num traps=0
Layer 47
Layer=47 | N=2560 | M=2560 | Q=1.00 | alpha=4.29 | D_ks=0.02 | alpha-hat=5.48 | num traps=0
Layer 49
Layer=49 | N=10240 | M=2560 | Q=4.00 | alpha=3.28 | D_ks=0.01 | alpha-hat=7.51 | num traps=0
Layer 50
Layer=50 | N=10240 | M=2560 | Q=4.00 | alpha=3.86 | D_ks=0.02 | alpha-hat=8.18 | num traps=9
Layer 52
Layer=52 | N=7680 | M=2560 | Q=3.00 | alpha=3.50 | D_ks=0.03 | alpha-hat=5.70 | num traps=0
Layer 53
Layer=53 | N=2560 | M=2560 | Q=1.00 | alpha=3.41 | D_ks=0.08 | alpha-hat=3.44 | num traps=0
Layer 55
Layer=55 | N=10240 | M=2560 | Q=4.00 | alpha=3.32 | D_ks=0.01 | alpha-hat=6.87 | num traps=1
Layer 56
Layer=56 | N=10240 | M=2560 | Q=4.00 | alpha=3.72 | D_ks=0.02 | alpha-hat=6.82 | num traps=6
Layer 58
Layer=58 | N=7680 | M=2560 | Q=3.00 | alpha=3.41 | D_ks=0.04 | alpha-hat=5.56 | num traps=0
Layer 59
Layer=59 | N=2560 | M=2560 | Q=1.00 | alpha=3.22 | D_ks=0.07 | alpha-hat=3.35 | num traps=0
Layer 61
Layer=61 | N=10240 | M=2560 | Q=4.00 | alpha=3.28 | D_ks=0.01 | alpha-hat=6.55 | num traps=0
Layer 62
Layer=62 | N=10240 | M=2560 | Q=4.00 | alpha=3.74 | D_ks=0.03 | alpha-hat=5.74 | num traps=4
Layer 64
Layer=64 | N=7680 | M=2560 | Q=3.00 | alpha=2.64 | D_ks=0.05 | alpha-hat=4.31 | num traps=0
Layer 65
Layer=65 | N=2560 | M=2560 | Q=1.00 | alpha=3.43 | D_ks=0.07 | alpha-hat=3.46 | num traps=0
Layer 67
Layer=67 | N=10240 | M=2560 | Q=4.00 | alpha=3.39 | D_ks=0.01 | alpha-hat=7.13 | num traps=0
Layer 68
Layer=68 | N=10240 | M=2560 | Q=4.00 | alpha=3.75 | D_ks=0.03 | alpha-hat=6.36 | num traps=0
Layer 70
Layer=70 | N=7680 | M=2560 | Q=3.00 | alpha=2.37 | D_ks=0.06 | alpha-hat=3.78 | num traps=0
Layer 71
Layer=71 | N=2560 | M=2560 | Q=1.00 | alpha=3.74 | D_ks=0.06 | alpha-hat=3.09 | num traps=0
Layer 73
Layer=73 | N=10240 | M=2560 | Q=4.00 | alpha=3.66 | D_ks=0.03 | alpha-hat=7.73 | num traps=0
Layer 74
Layer=74 | N=10240 | M=2560 | Q=4.00 | alpha=4.53 | D_ks=0.04 | alpha-hat=7.66 | num traps=0
Layer 76
Layer=76 | N=7680 | M=2560 | Q=3.00 | alpha=2.29 | D_ks=0.06 | alpha-hat=3.71 | num traps=0
Layer 77
Layer=77 | N=2560 | M=2560 | Q=1.00 | alpha=3.75 | D_ks=0.05 | alpha-hat=2.75 | num traps=0
Layer 79
Layer=79 | N=10240 | M=2560 | Q=4.00 | alpha=3.55 | D_ks=0.02 | alpha-hat=6.99 | num traps=0
Layer 80
Layer=80 | N=10240 | M=2560 | Q=4.00 | alpha=3.49 | D_ks=0.03 | alpha-hat=4.82 | num traps=0
Layer 82
Layer=82 | N=7680 | M=2560 | Q=3.00 | alpha=2.41 | D_ks=0.08 | alpha-hat=3.81 | num traps=0
Layer 83
Layer=83 | N=2560 | M=2560 | Q=1.00 | alpha=4.33 | D_ks=0.06 | alpha-hat=3.16 | num traps=0
Layer 85
Layer=85 | N=10240 | M=2560 | Q=4.00 | alpha=3.29 | D_ks=0.03 | alpha-hat=6.44 | num traps=0
Layer 86
Layer=86 | N=10240 | M=2560 | Q=4.00 | alpha=3.58 | D_ks=0.03 | alpha-hat=4.84 | num traps=0
Layer 88
Layer=88 | N=7680 | M=2560 | Q=3.00 | alpha=3.03 | D_ks=0.07 | alpha-hat=4.72 | num traps=0
Layer 89
Layer=89 | N=2560 | M=2560 | Q=1.00 | alpha=3.52 | D_ks=0.06 | alpha-hat=2.63 | num traps=0
Layer 91
Layer=91 | N=10240 | M=2560 | Q=4.00 | alpha=3.76 | D_ks=0.04 | alpha-hat=7.86 | num traps=0
Layer 92
Layer=92 | N=10240 | M=2560 | Q=4.00 | alpha=4.06 | D_ks=0.05 | alpha-hat=5.08 | num traps=0
Layer 94
Layer=94 | N=7680 | M=2560 | Q=3.00 | alpha=3.10 | D_ks=0.06 | alpha-hat=4.90 | num traps=0
Layer 95
Layer=95 | N=2560 | M=2560 | Q=1.00 | alpha=3.70 | D_ks=0.06 | alpha-hat=2.97 | num traps=0
Layer 97
Layer=97 | N=10240 | M=2560 | Q=4.00 | alpha=3.25 | D_ks=0.03 | alpha-hat=7.02 | num traps=0
Layer 98
Layer=98 | N=10240 | M=2560 | Q=4.00 | alpha=4.02 | D_ks=0.06 | alpha-hat=4.95 | num traps=0
Layer 100
Layer=100 | N=7680 | M=2560 | Q=3.00 | alpha=3.04 | D_ks=0.05 | alpha-hat=4.68 | num traps=0
Layer 101
Layer=101 | N=2560 | M=2560 | Q=1.00 | alpha=3.11 | D_ks=0.07 | alpha-hat=2.57 | num traps=0
Layer 103
Layer=103 | N=10240 | M=2560 | Q=4.00 | alpha=3.20 | D_ks=0.03 | alpha-hat=7.05 | num traps=0
Layer 104
Layer=104 | N=10240 | M=2560 | Q=4.00 | alpha=3.55 | D_ks=0.05 | alpha-hat=4.61 | num traps=0
Layer 106
Layer=106 | N=7680 | M=2560 | Q=3.00 | alpha=3.24 | D_ks=0.04 | alpha-hat=5.04 | num traps=0
Layer 107
Layer=107 | N=2560 | M=2560 | Q=1.00 | alpha=4.29 | D_ks=0.07 | alpha-hat=3.20 | num traps=0
Layer 109
Layer=109 | N=10240 | M=2560 | Q=4.00 | alpha=3.13 | D_ks=0.03 | alpha-hat=6.90 | num traps=0
Layer 110
Layer=110 | N=10240 | M=2560 | Q=4.00 | alpha=3.41 | D_ks=0.05 | alpha-hat=4.63 | num traps=0
Layer 112
Layer=112 | N=7680 | M=2560 | Q=3.00 | alpha=3.20 | D_ks=0.05 | alpha-hat=4.97 | num traps=0
Layer 113
Layer=113 | N=2560 | M=2560 | Q=1.00 | alpha=3.54 | D_ks=0.08 | alpha-hat=2.87 | num traps=0
Layer 115
Layer=115 | N=10240 | M=2560 | Q=4.00 | alpha=3.20 | D_ks=0.03 | alpha-hat=7.05 | num traps=0
Layer 116
Layer=116 | N=10240 | M=2560 | Q=4.00 | alpha=3.68 | D_ks=0.04 | alpha-hat=5.30 | num traps=0
Layer 118
Layer=118 | N=7680 | M=2560 | Q=3.00 | alpha=3.06 | D_ks=0.05 | alpha-hat=4.74 | num traps=0
Layer 119
Layer=119 | N=2560 | M=2560 | Q=1.00 | alpha=3.14 | D_ks=0.03 | alpha-hat=2.73 | num traps=0
Layer 121
Layer=121 | N=10240 | M=2560 | Q=4.00 | alpha=3.27 | D_ks=0.03 | alpha-hat=7.25 | num traps=0
Layer 122
Layer=122 | N=10240 | M=2560 | Q=4.00 | alpha=3.63 | D_ks=0.03 | alpha-hat=5.43 | num traps=0
Layer 124
Layer=124 | N=7680 | M=2560 | Q=3.00 | alpha=3.24 | D_ks=0.04 | alpha-hat=4.94 | num traps=0
Layer 125
Layer=125 | N=2560 | M=2560 | Q=1.00 | alpha=3.56 | D_ks=0.10 | alpha-hat=2.90 | num traps=0
Layer 127
Layer=127 | N=10240 | M=2560 | Q=4.00 | alpha=3.38 | D_ks=0.02 | alpha-hat=7.48 | num traps=0
Layer 128
Layer=128 | N=10240 | M=2560 | Q=4.00 | alpha=4.10 | D_ks=0.03 | alpha-hat=6.42 | num traps=0
Layer 130
Layer=130 | N=7680 | M=2560 | Q=3.00 | alpha=3.40 | D_ks=0.04 | alpha-hat=5.31 | num traps=0
Layer 131
Layer=131 | N=2560 | M=2560 | Q=1.00 | alpha=5.95 | D_ks=0.03 | alpha-hat=6.59 | num traps=0
Layer 133
Layer=133 | N=10240 | M=2560 | Q=4.00 | alpha=3.38 | D_ks=0.02 | alpha-hat=7.55 | num traps=0
Layer 134
Layer=134 | N=10240 | M=2560 | Q=4.00 | alpha=4.56 | D_ks=0.03 | alpha-hat=7.67 | num traps=0
Layer 136
Layer=136 | N=7680 | M=2560 | Q=3.00 | alpha=3.67 | D_ks=0.04 | alpha-hat=5.76 | num traps=0
Layer 137
Layer=137 | N=2560 | M=2560 | Q=1.00 | alpha=6.64 | D_ks=0.05 | alpha-hat=7.49 | num traps=0
Layer 139
Layer=139 | N=10240 | M=2560 | Q=4.00 | alpha=3.44 | D_ks=0.02 | alpha-hat=7.68 | num traps=0
Layer 140
Layer=140 | N=10240 | M=2560 | Q=4.00 | alpha=4.89 | D_ks=0.03 | alpha-hat=7.88 | num traps=0
Layer 142
Layer=142 | N=7680 | M=2560 | Q=3.00 | alpha=3.86 | D_ks=0.04 | alpha-hat=5.91 | num traps=0
Layer 143
Layer=143 | N=2560 | M=2560 | Q=1.00 | alpha=5.39 | D_ks=0.03 | alpha-hat=7.90 | num traps=0
Layer 145
Layer=145 | N=10240 | M=2560 | Q=4.00 | alpha=3.43 | D_ks=0.01 | alpha-hat=7.71 | num traps=0
Layer 146
Layer=146 | N=10240 | M=2560 | Q=4.00 | alpha=5.80 | D_ks=0.04 | alpha-hat=9.31 | num traps=1
Layer 148
Layer=148 | N=7680 | M=2560 | Q=3.00 | alpha=3.98 | D_ks=0.02 | alpha-hat=6.42 | num traps=0
Layer 149
Layer=149 | N=2560 | M=2560 | Q=1.00 | alpha=3.87 | D_ks=0.04 | alpha-hat=6.42 | num traps=0
Layer 151
Layer=151 | N=10240 | M=2560 | Q=4.00 | alpha=3.33 | D_ks=0.01 | alpha-hat=7.56 | num traps=0
Layer 152
Layer=152 | N=10240 | M=2560 | Q=4.00 | alpha=4.91 | D_ks=0.03 | alpha-hat=8.29 | num traps=1
Layer 154
Layer=154 | N=7680 | M=2560 | Q=3.00 | alpha=3.88 | D_ks=0.03 | alpha-hat=6.10 | num traps=0
Layer 155
Layer=155 | N=2560 | M=2560 | Q=1.00 | alpha=3.69 | D_ks=0.06 | alpha-hat=6.60 | num traps=4
Layer 157
Layer=157 | N=10240 | M=2560 | Q=4.00 | alpha=3.19 | D_ks=0.01 | alpha-hat=7.28 | num traps=0
Layer 158
Layer=158 | N=10240 | M=2560 | Q=4.00 | alpha=4.92 | D_ks=0.02 | alpha-hat=8.53 | num traps=0
Layer 160
Layer=160 | N=7680 | M=2560 | Q=3.00 | alpha=3.60 | D_ks=0.02 | alpha-hat=5.76 | num traps=0
Layer 161
Layer=161 | N=2560 | M=2560 | Q=1.00 | alpha=3.05 | D_ks=0.08 | alpha-hat=6.05 | num traps=22
Layer 163
Layer=163 | N=10240 | M=2560 | Q=4.00 | alpha=3.15 | D_ks=0.01 | alpha-hat=7.21 | num traps=0
Layer 164
Layer=164 | N=10240 | M=2560 | Q=4.00 | alpha=4.54 | D_ks=0.01 | alpha-hat=8.21 | num traps=3
Layer 166
Layer=166 | N=7680 | M=2560 | Q=3.00 | alpha=3.57 | D_ks=0.02 | alpha-hat=5.82 | num traps=0
Layer 167
Layer=167 | N=2560 | M=2560 | Q=1.00 | alpha=2.03 | D_ks=0.05 | alpha-hat=4.69 | num traps=39
Layer 169
Layer=169 | N=10240 | M=2560 | Q=4.00 | alpha=3.12 | D_ks=0.01 | alpha-hat=7.38 | num traps=0
Layer 170
Layer=170 | N=10240 | M=2560 | Q=4.00 | alpha=4.13 | D_ks=0.03 | alpha-hat=10.78 | num traps=4
Layer 172
Layer=172 | N=7680 | M=2560 | Q=3.00 | alpha=3.40 | D_ks=0.01 | alpha-hat=6.40 | num traps=0
Layer 173
Layer=173 | N=2560 | M=2560 | Q=1.00 | alpha=2.07 | D_ks=0.05 | alpha-hat=4.38 | num traps=36
Layer 175
Layer=175 | N=10240 | M=2560 | Q=4.00 | alpha=3.21 | D_ks=0.01 | alpha-hat=7.49 | num traps=0
Layer 176
Layer=176 | N=10240 | M=2560 | Q=4.00 | alpha=4.31 | D_ks=0.03 | alpha-hat=10.29 | num traps=1
Layer 178
Layer=178 | N=7680 | M=2560 | Q=3.00 | alpha=3.18 | D_ks=0.01 | alpha-hat=5.95 | num traps=0
Layer 179
Layer=179 | N=2560 | M=2560 | Q=1.00 | alpha=2.75 | D_ks=0.04 | alpha-hat=6.26 | num traps=35
Layer 181
Layer=181 | N=10240 | M=2560 | Q=4.00 | alpha=3.46 | D_ks=0.02 | alpha-hat=7.88 | num traps=0
Layer 182
Layer=182 | N=10240 | M=2560 | Q=4.00 | alpha=4.48 | D_ks=0.04 | alpha-hat=9.68 | num traps=4
bigscience-bloom-3b Model Summary Plots
bigscience-bloom-3b Model Selected Details
| layer_type | N | M | Q | alpha | D | alpha-hat | log_SN | % Rand | num_traps | num_fingers | rank_loss | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| layer_id | ||||||||||||
| 4 | DENSE | 7680 | 2560 | 3.00 | 3.40 | 0.02 | 5.85 | 1.72 | 83.81 | 0 | 0 | 0 |
| 5 | DENSE | 2560 | 2560 | 1.00 | 3.35 | 0.04 | 6.25 | 1.86 | 80.95 | 16 | 0 | 3 |
| 7 | DENSE | 10240 | 2560 | 4.00 | 5.30 | 0.05 | 11.63 | 2.19 | 88.42 | 1 | 0 | 0 |
| 8 | DENSE | 10240 | 2560 | 4.00 | 4.51 | 0.03 | 9.59 | 2.13 | 85.38 | 9 | 0 | 0 |
| 10 | DENSE | 7680 | 2560 | 3.00 | 3.71 | 0.02 | 6.44 | 1.74 | 86.05 | 0 | 0 | 0 |
| 11 | DENSE | 2560 | 2560 | 1.00 | 3.72 | 0.03 | 5.99 | 1.61 | 85.01 | 7 | 0 | 4 |
| 13 | DENSE | 10240 | 2560 | 4.00 | 2.78 | 0.03 | 5.81 | 2.09 | 84.07 | 1 | 0 | 0 |
| 14 | DENSE | 10240 | 2560 | 4.00 | 4.80 | 0.02 | 9.10 | 1.90 | 86.26 | 0 | 0 | 0 |
| 16 | DENSE | 7680 | 2560 | 3.00 | 3.43 | 0.02 | 5.72 | 1.67 | 86.14 | 0 | 0 | 0 |
| 17 | DENSE | 2560 | 2560 | 1.00 | 3.81 | 0.02 | 5.95 | 1.56 | 85.55 | 3 | 0 | 3 |
| 19 | DENSE | 10240 | 2560 | 4.00 | 2.76 | 0.03 | 5.62 | 2.04 | 82.88 | 0 | 0 | 0 |
| 20 | DENSE | 10240 | 2560 | 4.00 | 4.83 | 0.03 | 8.63 | 1.79 | 86.52 | 1 | 0 | 0 |
| 22 | DENSE | 7680 | 2560 | 3.00 | 3.38 | 0.02 | 5.67 | 1.68 | 86.17 | 0 | 0 | 0 |
| 23 | DENSE | 2560 | 2560 | 1.00 | 4.10 | 0.02 | 5.42 | 1.32 | 87.88 | 4 | 0 | 3 |
| 25 | DENSE | 10240 | 2560 | 4.00 | 2.95 | 0.03 | 5.89 | 2.00 | 82.71 | 0 | 0 | 0 |
| 26 | DENSE | 10240 | 2560 | 4.00 | 4.38 | 0.03 | 7.43 | 1.70 | 85.72 | 0 | 0 | 0 |
| 28 | DENSE | 7680 | 2560 | 3.00 | 3.38 | 0.04 | 5.52 | 1.63 | 85.62 | 0 | 0 | 0 |
| 29 | DENSE | 2560 | 2560 | 1.00 | 4.30 | 0.02 | 5.35 | 1.24 | 89.15 | 0 | 0 | 4 |
| 31 | DENSE | 10240 | 2560 | 4.00 | 3.36 | 0.01 | 6.45 | 1.92 | 82.58 | 0 | 0 | 0 |
| 32 | DENSE | 10240 | 2560 | 4.00 | 4.19 | 0.02 | 6.94 | 1.65 | 84.48 | 1 | 0 | 0 |
| 34 | DENSE | 7680 | 2560 | 3.00 | 3.52 | 0.03 | 6.03 | 1.71 | 85.92 | 0 | 0 | 0 |
| 35 | DENSE | 2560 | 2560 | 1.00 | 4.29 | 0.03 | 5.71 | 1.33 | 88.74 | 2 | 0 | 4 |
| 37 | DENSE | 10240 | 2560 | 4.00 | 3.41 | 0.01 | 6.34 | 1.86 | 82.55 | 0 | 0 | 0 |
| 38 | DENSE | 10240 | 2560 | 4.00 | 4.36 | 0.02 | 7.19 | 1.65 | 83.70 | 10 | 0 | 0 |
| 40 | DENSE | 7680 | 2560 | 3.00 | 3.56 | 0.03 | 6.02 | 1.69 | 85.99 | 0 | 0 | 0 |
| 41 | DENSE | 2560 | 2560 | 1.00 | 4.43 | 0.02 | 6.37 | 1.44 | 88.77 | 2 | 0 | 3 |
| 43 | DENSE | 10240 | 2560 | 4.00 | 3.27 | 0.02 | 7.80 | 2.39 | 78.55 | 0 | 0 | 0 |
| 44 | DENSE | 10240 | 2560 | 4.00 | 3.92 | 0.03 | 7.93 | 2.03 | 83.98 | 11 | 0 | 0 |
| 46 | DENSE | 7680 | 2560 | 3.00 | 3.65 | 0.03 | 6.07 | 1.67 | 85.91 | 0 | 0 | 0 |
| 47 | DENSE | 2560 | 2560 | 1.00 | 4.29 | 0.02 | 5.48 | 1.28 | 88.62 | 0 | 0 | 3 |
| 49 | DENSE | 10240 | 2560 | 4.00 | 3.28 | 0.01 | 7.51 | 2.29 | 79.32 | 0 | 0 | 0 |
| 50 | DENSE | 10240 | 2560 | 4.00 | 3.86 | 0.02 | 8.18 | 2.12 | 83.58 | 9 | 0 | 0 |
| 52 | DENSE | 7680 | 2560 | 3.00 | 3.50 | 0.03 | 5.70 | 1.63 | 85.70 | 0 | 0 | 0 |
| 53 | DENSE | 2560 | 2560 | 1.00 | 3.41 | 0.08 | 3.44 | 1.01 | 89.54 | 0 | 4 | 4 |
| 55 | DENSE | 10240 | 2560 | 4.00 | 3.32 | 0.01 | 6.87 | 2.07 | 82.84 | 1 | 0 | 0 |
| 56 | DENSE | 10240 | 2560 | 4.00 | 3.72 | 0.02 | 6.82 | 1.83 | 86.22 | 6 | 0 | 0 |
| 58 | DENSE | 7680 | 2560 | 3.00 | 3.41 | 0.04 | 5.56 | 1.63 | 85.44 | 0 | 0 | 0 |
| 59 | DENSE | 2560 | 2560 | 1.00 | 3.22 | 0.07 | 3.35 | 1.04 | 88.24 | 0 | 1 | 4 |
| 61 | DENSE | 10240 | 2560 | 4.00 | 3.28 | 0.01 | 6.55 | 2.00 | 85.18 | 0 | 0 | 0 |
| 62 | DENSE | 10240 | 2560 | 4.00 | 3.74 | 0.03 | 5.74 | 1.54 | 88.20 | 4 | 0 | 0 |
| 64 | DENSE | 7680 | 2560 | 3.00 | 2.64 | 0.05 | 4.31 | 1.63 | 82.04 | 0 | 0 | 0 |
| 65 | DENSE | 2560 | 2560 | 1.00 | 3.43 | 0.07 | 3.46 | 1.01 | 89.77 | 0 | 1 | 3 |
| 67 | DENSE | 10240 | 2560 | 4.00 | 3.39 | 0.01 | 7.13 | 2.10 | 86.71 | 0 | 0 | 0 |
| 68 | DENSE | 10240 | 2560 | 4.00 | 3.75 | 0.03 | 6.36 | 1.70 | 88.04 | 0 | 0 | 0 |
| 70 | DENSE | 7680 | 2560 | 3.00 | 2.37 | 0.06 | 3.78 | 1.59 | 79.43 | 0 | 0 | 0 |
| 71 | DENSE | 2560 | 2560 | 1.00 | 3.74 | 0.06 | 3.09 | 0.83 | 86.48 | 0 | 0 | 3 |
| 73 | DENSE | 10240 | 2560 | 4.00 | 3.66 | 0.03 | 7.73 | 2.11 | 87.03 | 0 | 0 | 0 |
| 74 | DENSE | 10240 | 2560 | 4.00 | 4.53 | 0.04 | 7.66 | 1.69 | 87.76 | 0 | 0 | 0 |
| 76 | DENSE | 7680 | 2560 | 3.00 | 2.29 | 0.06 | 3.71 | 1.62 | 76.55 | 0 | 0 | 0 |
| 77 | DENSE | 2560 | 2560 | 1.00 | 3.75 | 0.05 | 2.75 | 0.73 | 84.56 | 0 | 0 | 4 |
| 79 | DENSE | 10240 | 2560 | 4.00 | 3.55 | 0.02 | 6.99 | 1.97 | 85.75 | 0 | 0 | 0 |
| 80 | DENSE | 10240 | 2560 | 4.00 | 3.49 | 0.03 | 4.82 | 1.38 | 88.08 | 0 | 0 | 0 |
| 82 | DENSE | 7680 | 2560 | 3.00 | 2.41 | 0.08 | 3.81 | 1.58 | 80.25 | 0 | 7 | 0 |
| 83 | DENSE | 2560 | 2560 | 1.00 | 4.33 | 0.06 | 3.16 | 0.73 | 89.12 | 0 | 0 | 4 |
| 85 | DENSE | 10240 | 2560 | 4.00 | 3.29 | 0.03 | 6.44 | 1.96 | 84.38 | 0 | 0 | 0 |
| 86 | DENSE | 10240 | 2560 | 4.00 | 3.58 | 0.03 | 4.84 | 1.35 | 88.93 | 0 | 0 | 0 |
| 88 | DENSE | 7680 | 2560 | 3.00 | 3.03 | 0.07 | 4.72 | 1.56 | 79.01 | 0 | 0 | 0 |
| 89 | DENSE | 2560 | 2560 | 1.00 | 3.52 | 0.06 | 2.63 | 0.75 | 87.99 | 0 | 0 | 3 |
| 91 | DENSE | 10240 | 2560 | 4.00 | 3.76 | 0.04 | 7.86 | 2.09 | 84.37 | 0 | 0 | 0 |
| 92 | DENSE | 10240 | 2560 | 4.00 | 4.06 | 0.05 | 5.08 | 1.25 | 90.92 | 0 | 0 | 0 |
| 94 | DENSE | 7680 | 2560 | 3.00 | 3.10 | 0.06 | 4.90 | 1.58 | 80.46 | 0 | 0 | 0 |
| 95 | DENSE | 2560 | 2560 | 1.00 | 3.70 | 0.06 | 2.97 | 0.80 | 87.22 | 0 | 0 | 4 |
| 97 | DENSE | 10240 | 2560 | 4.00 | 3.25 | 0.03 | 7.02 | 2.16 | 83.80 | 0 | 0 | 0 |
| 98 | DENSE | 10240 | 2560 | 4.00 | 4.02 | 0.06 | 4.95 | 1.23 | 89.83 | 0 | 0 | 0 |
| 100 | DENSE | 7680 | 2560 | 3.00 | 3.04 | 0.05 | 4.68 | 1.54 | 82.26 | 0 | 0 | 0 |
| 101 | DENSE | 2560 | 2560 | 1.00 | 3.11 | 0.07 | 2.57 | 0.83 | 85.92 | 0 | 0 | 3 |
| 103 | DENSE | 10240 | 2560 | 4.00 | 3.20 | 0.03 | 7.05 | 2.20 | 82.93 | 0 | 0 | 0 |
| 104 | DENSE | 10240 | 2560 | 4.00 | 3.55 | 0.05 | 4.61 | 1.30 | 88.47 | 0 | 0 | 0 |
| 106 | DENSE | 7680 | 2560 | 3.00 | 3.24 | 0.04 | 5.04 | 1.55 | 84.34 | 0 | 0 | 0 |
| 107 | DENSE | 2560 | 2560 | 1.00 | 4.29 | 0.07 | 3.20 | 0.75 | 89.66 | 0 | 0 | 3 |
| 109 | DENSE | 10240 | 2560 | 4.00 | 3.13 | 0.03 | 6.90 | 2.20 | 82.56 | 0 | 0 | 0 |
| 110 | DENSE | 10240 | 2560 | 4.00 | 3.41 | 0.05 | 4.63 | 1.36 | 88.22 | 0 | 0 | 0 |
| 112 | DENSE | 7680 | 2560 | 3.00 | 3.20 | 0.05 | 4.97 | 1.55 | 84.08 | 0 | 0 | 0 |
| 113 | DENSE | 2560 | 2560 | 1.00 | 3.54 | 0.08 | 2.87 | 0.81 | 87.07 | 0 | 1 | 4 |
| 115 | DENSE | 10240 | 2560 | 4.00 | 3.20 | 0.03 | 7.05 | 2.20 | 83.39 | 0 | 0 | 0 |
| 116 | DENSE | 10240 | 2560 | 4.00 | 3.68 | 0.04 | 5.30 | 1.44 | 88.26 | 0 | 0 | 0 |
| 118 | DENSE | 7680 | 2560 | 3.00 | 3.06 | 0.05 | 4.74 | 1.55 | 82.78 | 0 | 0 | 0 |
| 119 | DENSE | 2560 | 2560 | 1.00 | 3.14 | 0.03 | 2.73 | 0.87 | 84.63 | 0 | 0 | 4 |
| 121 | DENSE | 10240 | 2560 | 4.00 | 3.27 | 0.03 | 7.25 | 2.22 | 84.38 | 0 | 0 | 0 |
| 122 | DENSE | 10240 | 2560 | 4.00 | 3.63 | 0.03 | 5.43 | 1.50 | 88.73 | 0 | 0 | 0 |
| 124 | DENSE | 7680 | 2560 | 3.00 | 3.24 | 0.04 | 4.94 | 1.52 | 86.66 | 0 | 0 | 0 |
| 125 | DENSE | 2560 | 2560 | 1.00 | 3.56 | 0.10 | 2.90 | 0.82 | 87.84 | 0 | 2 | 3 |
| 127 | DENSE | 10240 | 2560 | 4.00 | 3.38 | 0.02 | 7.48 | 2.21 | 85.05 | 0 | 0 | 0 |
| 128 | DENSE | 10240 | 2560 | 4.00 | 4.10 | 0.03 | 6.42 | 1.57 | 90.22 | 0 | 0 | 0 |
| 130 | DENSE | 7680 | 2560 | 3.00 | 3.40 | 0.04 | 5.31 | 1.56 | 86.70 | 0 | 0 | 0 |
| 131 | DENSE | 2560 | 2560 | 1.00 | 5.95 | 0.03 | 6.59 | 1.11 | 91.74 | 0 | 0 | 4 |
| 133 | DENSE | 10240 | 2560 | 4.00 | 3.38 | 0.02 | 7.55 | 2.23 | 85.12 | 0 | 0 | 0 |
| 134 | DENSE | 10240 | 2560 | 4.00 | 4.56 | 0.03 | 7.67 | 1.68 | 90.04 | 0 | 0 | 0 |
| 136 | DENSE | 7680 | 2560 | 3.00 | 3.67 | 0.04 | 5.76 | 1.57 | 87.10 | 0 | 0 | 0 |
| 137 | DENSE | 2560 | 2560 | 1.00 | 6.64 | 0.05 | 7.49 | 1.13 | 89.96 | 0 | 0 | 3 |
| 139 | DENSE | 10240 | 2560 | 4.00 | 3.44 | 0.02 | 7.68 | 2.23 | 85.40 | 0 | 0 | 0 |
| 140 | DENSE | 10240 | 2560 | 4.00 | 4.89 | 0.03 | 7.88 | 1.61 | 90.90 | 0 | 0 | 0 |
| 142 | DENSE | 7680 | 2560 | 3.00 | 3.86 | 0.04 | 5.91 | 1.53 | 88.76 | 0 | 0 | 0 |
| 143 | DENSE | 2560 | 2560 | 1.00 | 5.39 | 0.03 | 7.90 | 1.47 | 89.17 | 0 | 0 | 3 |
| 145 | DENSE | 10240 | 2560 | 4.00 | 3.43 | 0.01 | 7.71 | 2.24 | 85.50 | 0 | 0 | 0 |
| 146 | DENSE | 10240 | 2560 | 4.00 | 5.80 | 0.04 | 9.31 | 1.60 | 91.62 | 1 | 0 | 0 |
| 148 | DENSE | 7680 | 2560 | 3.00 | 3.98 | 0.02 | 6.42 | 1.61 | 89.85 | 0 | 0 | 0 |
| 149 | DENSE | 2560 | 2560 | 1.00 | 3.87 | 0.04 | 6.42 | 1.66 | 87.24 | 0 | 0 | 3 |
| 151 | DENSE | 10240 | 2560 | 4.00 | 3.33 | 0.01 | 7.56 | 2.27 | 84.92 | 0 | 0 | 0 |
| 152 | DENSE | 10240 | 2560 | 4.00 | 4.91 | 0.03 | 8.29 | 1.69 | 92.51 | 1 | 0 | 0 |
| 154 | DENSE | 7680 | 2560 | 3.00 | 3.88 | 0.03 | 6.10 | 1.57 | 88.58 | 0 | 0 | 0 |
| 155 | DENSE | 2560 | 2560 | 1.00 | 3.69 | 0.06 | 6.60 | 1.79 | 85.19 | 4 | 0 | 3 |
| 157 | DENSE | 10240 | 2560 | 4.00 | 3.19 | 0.01 | 7.28 | 2.28 | 84.32 | 0 | 0 | 0 |
| 158 | DENSE | 10240 | 2560 | 4.00 | 4.92 | 0.02 | 8.53 | 1.73 | 92.84 | 0 | 0 | 0 |
| 160 | DENSE | 7680 | 2560 | 3.00 | 3.60 | 0.02 | 5.76 | 1.60 | 87.77 | 0 | 0 | 0 |
| 161 | DENSE | 2560 | 2560 | 1.00 | 3.05 | 0.08 | 6.05 | 1.98 | 81.38 | 22 | 0 | 2 |
| 163 | DENSE | 10240 | 2560 | 4.00 | 3.15 | 0.01 | 7.21 | 2.29 | 84.13 | 0 | 0 | 0 |
| 164 | DENSE | 10240 | 2560 | 4.00 | 4.54 | 0.01 | 8.21 | 1.81 | 92.35 | 3 | 0 | 0 |
| 166 | DENSE | 7680 | 2560 | 3.00 | 3.57 | 0.02 | 5.82 | 1.63 | 87.13 | 0 | 0 | 0 |
| 167 | DENSE | 2560 | 2560 | 1.00 | 2.03 | 0.05 | 4.69 | 2.31 | 74.82 | 39 | 0 | 3 |
| 169 | DENSE | 10240 | 2560 | 4.00 | 3.12 | 0.01 | 7.38 | 2.36 | 83.68 | 0 | 0 | 0 |
| 170 | DENSE | 10240 | 2560 | 4.00 | 4.13 | 0.03 | 10.78 | 2.61 | 86.99 | 4 | 0 | 0 |
| 172 | DENSE | 7680 | 2560 | 3.00 | 3.40 | 0.01 | 6.40 | 1.88 | 86.50 | 0 | 0 | 0 |
| 173 | DENSE | 2560 | 2560 | 1.00 | 2.07 | 0.05 | 4.38 | 2.12 | 73.50 | 36 | 0 | 3 |
| 175 | DENSE | 10240 | 2560 | 4.00 | 3.21 | 0.01 | 7.49 | 2.33 | 84.57 | 0 | 0 | 0 |
| 176 | DENSE | 10240 | 2560 | 4.00 | 4.31 | 0.03 | 10.29 | 2.39 | 88.27 | 1 | 0 | 0 |
| 178 | DENSE | 7680 | 2560 | 3.00 | 3.18 | 0.01 | 5.95 | 1.87 | 85.24 | 0 | 0 | 0 |
| 179 | DENSE | 2560 | 2560 | 1.00 | 2.75 | 0.04 | 6.26 | 2.27 | 71.82 | 35 | 0 | 3 |
| 181 | DENSE | 10240 | 2560 | 4.00 | 3.46 | 0.02 | 7.88 | 2.28 | 85.31 | 0 | 0 | 0 |
| 182 | DENSE | 10240 | 2560 | 4.00 | 4.48 | 0.04 | 9.68 | 2.16 | 89.53 | 4 | 0 | 0 |
bigscience-bloom-3b Layer Plots
Layer 4
Layer=4 | N=7680 | M=2560 | Q=3.00 | alpha=3.40 | D_ks=0.02 | alpha-hat=5.85 | num traps=0
Layer 5
Layer=5 | N=2560 | M=2560 | Q=1.00 | alpha=3.35 | D_ks=0.04 | alpha-hat=6.25 | num traps=16
Layer 7
Layer=7 | N=10240 | M=2560 | Q=4.00 | alpha=5.30 | D_ks=0.05 | alpha-hat=11.63 | num traps=1
Layer 8
Layer=8 | N=10240 | M=2560 | Q=4.00 | alpha=4.51 | D_ks=0.03 | alpha-hat=9.59 | num traps=9
Layer 10
Layer=10 | N=7680 | M=2560 | Q=3.00 | alpha=3.71 | D_ks=0.02 | alpha-hat=6.44 | num traps=0
Layer 11
Layer=11 | N=2560 | M=2560 | Q=1.00 | alpha=3.72 | D_ks=0.03 | alpha-hat=5.99 | num traps=7
Layer 13
Layer=13 | N=10240 | M=2560 | Q=4.00 | alpha=2.78 | D_ks=0.03 | alpha-hat=5.81 | num traps=1
Layer 14
Layer=14 | N=10240 | M=2560 | Q=4.00 | alpha=4.80 | D_ks=0.02 | alpha-hat=9.10 | num traps=0
Layer 16
Layer=16 | N=7680 | M=2560 | Q=3.00 | alpha=3.43 | D_ks=0.02 | alpha-hat=5.72 | num traps=0
Layer 17
Layer=17 | N=2560 | M=2560 | Q=1.00 | alpha=3.81 | D_ks=0.02 | alpha-hat=5.95 | num traps=3
Layer 19
Layer=19 | N=10240 | M=2560 | Q=4.00 | alpha=2.76 | D_ks=0.03 | alpha-hat=5.62 | num traps=0
Layer 20
Layer=20 | N=10240 | M=2560 | Q=4.00 | alpha=4.83 | D_ks=0.03 | alpha-hat=8.63 | num traps=1
Layer 22
Layer=22 | N=7680 | M=2560 | Q=3.00 | alpha=3.38 | D_ks=0.02 | alpha-hat=5.67 | num traps=0
Layer 23
Layer=23 | N=2560 | M=2560 | Q=1.00 | alpha=4.10 | D_ks=0.02 | alpha-hat=5.42 | num traps=4
Layer 25
Layer=25 | N=10240 | M=2560 | Q=4.00 | alpha=2.95 | D_ks=0.03 | alpha-hat=5.89 | num traps=0
Layer 26
Layer=26 | N=10240 | M=2560 | Q=4.00 | alpha=4.38 | D_ks=0.03 | alpha-hat=7.43 | num traps=0
Layer 28
Layer=28 | N=7680 | M=2560 | Q=3.00 | alpha=3.38 | D_ks=0.04 | alpha-hat=5.52 | num traps=0
Layer 29
Layer=29 | N=2560 | M=2560 | Q=1.00 | alpha=4.30 | D_ks=0.02 | alpha-hat=5.35 | num traps=0
Layer 31
Layer=31 | N=10240 | M=2560 | Q=4.00 | alpha=3.36 | D_ks=0.01 | alpha-hat=6.45 | num traps=0
Layer 32
Layer=32 | N=10240 | M=2560 | Q=4.00 | alpha=4.19 | D_ks=0.02 | alpha-hat=6.94 | num traps=1
Layer 34
Layer=34 | N=7680 | M=2560 | Q=3.00 | alpha=3.52 | D_ks=0.03 | alpha-hat=6.03 | num traps=0
Layer 35
Layer=35 | N=2560 | M=2560 | Q=1.00 | alpha=4.29 | D_ks=0.03 | alpha-hat=5.71 | num traps=2
Layer 37
Layer=37 | N=10240 | M=2560 | Q=4.00 | alpha=3.41 | D_ks=0.01 | alpha-hat=6.34 | num traps=0
Layer 38
Layer=38 | N=10240 | M=2560 | Q=4.00 | alpha=4.36 | D_ks=0.02 | alpha-hat=7.19 | num traps=10
Layer 40
Layer=40 | N=7680 | M=2560 | Q=3.00 | alpha=3.56 | D_ks=0.03 | alpha-hat=6.02 | num traps=0
Layer 41
Layer=41 | N=2560 | M=2560 | Q=1.00 | alpha=4.43 | D_ks=0.02 | alpha-hat=6.37 | num traps=2
Layer 43
Layer=43 | N=10240 | M=2560 | Q=4.00 | alpha=3.27 | D_ks=0.02 | alpha-hat=7.80 | num traps=0
Layer 44
Layer=44 | N=10240 | M=2560 | Q=4.00 | alpha=3.92 | D_ks=0.03 | alpha-hat=7.93 | num traps=11
Layer 46
Layer=46 | N=7680 | M=2560 | Q=3.00 | alpha=3.65 | D_ks=0.03 | alpha-hat=6.07 | num traps=0
Layer 47
Layer=47 | N=2560 | M=2560 | Q=1.00 | alpha=4.29 | D_ks=0.02 | alpha-hat=5.48 | num traps=0
Layer 49
Layer=49 | N=10240 | M=2560 | Q=4.00 | alpha=3.28 | D_ks=0.01 | alpha-hat=7.51 | num traps=0
Layer 50
Layer=50 | N=10240 | M=2560 | Q=4.00 | alpha=3.86 | D_ks=0.02 | alpha-hat=8.18 | num traps=9
Layer 52
Layer=52 | N=7680 | M=2560 | Q=3.00 | alpha=3.50 | D_ks=0.03 | alpha-hat=5.70 | num traps=0
Layer 53
Layer=53 | N=2560 | M=2560 | Q=1.00 | alpha=3.41 | D_ks=0.08 | alpha-hat=3.44 | num traps=0
Layer 55
Layer=55 | N=10240 | M=2560 | Q=4.00 | alpha=3.32 | D_ks=0.01 | alpha-hat=6.87 | num traps=1
Layer 56
Layer=56 | N=10240 | M=2560 | Q=4.00 | alpha=3.72 | D_ks=0.02 | alpha-hat=6.82 | num traps=6
Layer 58
Layer=58 | N=7680 | M=2560 | Q=3.00 | alpha=3.41 | D_ks=0.04 | alpha-hat=5.56 | num traps=0
Layer 59
Layer=59 | N=2560 | M=2560 | Q=1.00 | alpha=3.22 | D_ks=0.07 | alpha-hat=3.35 | num traps=0
Layer 61
Layer=61 | N=10240 | M=2560 | Q=4.00 | alpha=3.28 | D_ks=0.01 | alpha-hat=6.55 | num traps=0
Layer 62
Layer=62 | N=10240 | M=2560 | Q=4.00 | alpha=3.74 | D_ks=0.03 | alpha-hat=5.74 | num traps=4
Layer 64
Layer=64 | N=7680 | M=2560 | Q=3.00 | alpha=2.64 | D_ks=0.05 | alpha-hat=4.31 | num traps=0
Layer 65
Layer=65 | N=2560 | M=2560 | Q=1.00 | alpha=3.43 | D_ks=0.07 | alpha-hat=3.46 | num traps=0
Layer 67
Layer=67 | N=10240 | M=2560 | Q=4.00 | alpha=3.39 | D_ks=0.01 | alpha-hat=7.13 | num traps=0
Layer 68
Layer=68 | N=10240 | M=2560 | Q=4.00 | alpha=3.75 | D_ks=0.03 | alpha-hat=6.36 | num traps=0
Layer 70
Layer=70 | N=7680 | M=2560 | Q=3.00 | alpha=2.37 | D_ks=0.06 | alpha-hat=3.78 | num traps=0
Layer 71
Layer=71 | N=2560 | M=2560 | Q=1.00 | alpha=3.74 | D_ks=0.06 | alpha-hat=3.09 | num traps=0
Layer 73
Layer=73 | N=10240 | M=2560 | Q=4.00 | alpha=3.66 | D_ks=0.03 | alpha-hat=7.73 | num traps=0
Layer 74
Layer=74 | N=10240 | M=2560 | Q=4.00 | alpha=4.53 | D_ks=0.04 | alpha-hat=7.66 | num traps=0
Layer 76
Layer=76 | N=7680 | M=2560 | Q=3.00 | alpha=2.29 | D_ks=0.06 | alpha-hat=3.71 | num traps=0
Layer 77
Layer=77 | N=2560 | M=2560 | Q=1.00 | alpha=3.75 | D_ks=0.05 | alpha-hat=2.75 | num traps=0
Layer 79
Layer=79 | N=10240 | M=2560 | Q=4.00 | alpha=3.55 | D_ks=0.02 | alpha-hat=6.99 | num traps=0
Layer 80
Layer=80 | N=10240 | M=2560 | Q=4.00 | alpha=3.49 | D_ks=0.03 | alpha-hat=4.82 | num traps=0
Layer 82
Layer=82 | N=7680 | M=2560 | Q=3.00 | alpha=2.41 | D_ks=0.08 | alpha-hat=3.81 | num traps=0
Layer 83
Layer=83 | N=2560 | M=2560 | Q=1.00 | alpha=4.33 | D_ks=0.06 | alpha-hat=3.16 | num traps=0
Layer 85
Layer=85 | N=10240 | M=2560 | Q=4.00 | alpha=3.29 | D_ks=0.03 | alpha-hat=6.44 | num traps=0
Layer 86
Layer=86 | N=10240 | M=2560 | Q=4.00 | alpha=3.58 | D_ks=0.03 | alpha-hat=4.84 | num traps=0
Layer 88
Layer=88 | N=7680 | M=2560 | Q=3.00 | alpha=3.03 | D_ks=0.07 | alpha-hat=4.72 | num traps=0
Layer 89
Layer=89 | N=2560 | M=2560 | Q=1.00 | alpha=3.52 | D_ks=0.06 | alpha-hat=2.63 | num traps=0
Layer 91
Layer=91 | N=10240 | M=2560 | Q=4.00 | alpha=3.76 | D_ks=0.04 | alpha-hat=7.86 | num traps=0
Layer 92
Layer=92 | N=10240 | M=2560 | Q=4.00 | alpha=4.06 | D_ks=0.05 | alpha-hat=5.08 | num traps=0
Layer 94
Layer=94 | N=7680 | M=2560 | Q=3.00 | alpha=3.10 | D_ks=0.06 | alpha-hat=4.90 | num traps=0
Layer 95
Layer=95 | N=2560 | M=2560 | Q=1.00 | alpha=3.70 | D_ks=0.06 | alpha-hat=2.97 | num traps=0
Layer 97
Layer=97 | N=10240 | M=2560 | Q=4.00 | alpha=3.25 | D_ks=0.03 | alpha-hat=7.02 | num traps=0
Layer 98
Layer=98 | N=10240 | M=2560 | Q=4.00 | alpha=4.02 | D_ks=0.06 | alpha-hat=4.95 | num traps=0
Layer 100
Layer=100 | N=7680 | M=2560 | Q=3.00 | alpha=3.04 | D_ks=0.05 | alpha-hat=4.68 | num traps=0
Layer 101
Layer=101 | N=2560 | M=2560 | Q=1.00 | alpha=3.11 | D_ks=0.07 | alpha-hat=2.57 | num traps=0
Layer 103
Layer=103 | N=10240 | M=2560 | Q=4.00 | alpha=3.20 | D_ks=0.03 | alpha-hat=7.05 | num traps=0
Layer 104
Layer=104 | N=10240 | M=2560 | Q=4.00 | alpha=3.55 | D_ks=0.05 | alpha-hat=4.61 | num traps=0
Layer 106
Layer=106 | N=7680 | M=2560 | Q=3.00 | alpha=3.24 | D_ks=0.04 | alpha-hat=5.04 | num traps=0
Layer 107
Layer=107 | N=2560 | M=2560 | Q=1.00 | alpha=4.29 | D_ks=0.07 | alpha-hat=3.20 | num traps=0
Layer 109
Layer=109 | N=10240 | M=2560 | Q=4.00 | alpha=3.13 | D_ks=0.03 | alpha-hat=6.90 | num traps=0
Layer 110
Layer=110 | N=10240 | M=2560 | Q=4.00 | alpha=3.41 | D_ks=0.05 | alpha-hat=4.63 | num traps=0
Layer 112
Layer=112 | N=7680 | M=2560 | Q=3.00 | alpha=3.20 | D_ks=0.05 | alpha-hat=4.97 | num traps=0
Layer 113
Layer=113 | N=2560 | M=2560 | Q=1.00 | alpha=3.54 | D_ks=0.08 | alpha-hat=2.87 | num traps=0
Layer 115
Layer=115 | N=10240 | M=2560 | Q=4.00 | alpha=3.20 | D_ks=0.03 | alpha-hat=7.05 | num traps=0
Layer 116
Layer=116 | N=10240 | M=2560 | Q=4.00 | alpha=3.68 | D_ks=0.04 | alpha-hat=5.30 | num traps=0
Layer 118
Layer=118 | N=7680 | M=2560 | Q=3.00 | alpha=3.06 | D_ks=0.05 | alpha-hat=4.74 | num traps=0
Layer 119
Layer=119 | N=2560 | M=2560 | Q=1.00 | alpha=3.14 | D_ks=0.03 | alpha-hat=2.73 | num traps=0
Layer 121
Layer=121 | N=10240 | M=2560 | Q=4.00 | alpha=3.27 | D_ks=0.03 | alpha-hat=7.25 | num traps=0
Layer 122
Layer=122 | N=10240 | M=2560 | Q=4.00 | alpha=3.63 | D_ks=0.03 | alpha-hat=5.43 | num traps=0
Layer 124
Layer=124 | N=7680 | M=2560 | Q=3.00 | alpha=3.24 | D_ks=0.04 | alpha-hat=4.94 | num traps=0
Layer 125
Layer=125 | N=2560 | M=2560 | Q=1.00 | alpha=3.56 | D_ks=0.10 | alpha-hat=2.90 | num traps=0
Layer 127
Layer=127 | N=10240 | M=2560 | Q=4.00 | alpha=3.38 | D_ks=0.02 | alpha-hat=7.48 | num traps=0
Layer 128
Layer=128 | N=10240 | M=2560 | Q=4.00 | alpha=4.10 | D_ks=0.03 | alpha-hat=6.42 | num traps=0
Layer 130
Layer=130 | N=7680 | M=2560 | Q=3.00 | alpha=3.40 | D_ks=0.04 | alpha-hat=5.31 | num traps=0
Layer 131
Layer=131 | N=2560 | M=2560 | Q=1.00 | alpha=5.95 | D_ks=0.03 | alpha-hat=6.59 | num traps=0
Layer 133
Layer=133 | N=10240 | M=2560 | Q=4.00 | alpha=3.38 | D_ks=0.02 | alpha-hat=7.55 | num traps=0
Layer 134
Layer=134 | N=10240 | M=2560 | Q=4.00 | alpha=4.56 | D_ks=0.03 | alpha-hat=7.67 | num traps=0
Layer 136
Layer=136 | N=7680 | M=2560 | Q=3.00 | alpha=3.67 | D_ks=0.04 | alpha-hat=5.76 | num traps=0
Layer 137
Layer=137 | N=2560 | M=2560 | Q=1.00 | alpha=6.64 | D_ks=0.05 | alpha-hat=7.49 | num traps=0
Layer 139
Layer=139 | N=10240 | M=2560 | Q=4.00 | alpha=3.44 | D_ks=0.02 | alpha-hat=7.68 | num traps=0
Layer 140
Layer=140 | N=10240 | M=2560 | Q=4.00 | alpha=4.89 | D_ks=0.03 | alpha-hat=7.88 | num traps=0
Layer 142
Layer=142 | N=7680 | M=2560 | Q=3.00 | alpha=3.86 | D_ks=0.04 | alpha-hat=5.91 | num traps=0
Layer 143
Layer=143 | N=2560 | M=2560 | Q=1.00 | alpha=5.39 | D_ks=0.03 | alpha-hat=7.90 | num traps=0
Layer 145
Layer=145 | N=10240 | M=2560 | Q=4.00 | alpha=3.43 | D_ks=0.01 | alpha-hat=7.71 | num traps=0
Layer 146
Layer=146 | N=10240 | M=2560 | Q=4.00 | alpha=5.80 | D_ks=0.04 | alpha-hat=9.31 | num traps=1
Layer 148
Layer=148 | N=7680 | M=2560 | Q=3.00 | alpha=3.98 | D_ks=0.02 | alpha-hat=6.42 | num traps=0
Layer 149
Layer=149 | N=2560 | M=2560 | Q=1.00 | alpha=3.87 | D_ks=0.04 | alpha-hat=6.42 | num traps=0
Layer 151
Layer=151 | N=10240 | M=2560 | Q=4.00 | alpha=3.33 | D_ks=0.01 | alpha-hat=7.56 | num traps=0
Layer 152
Layer=152 | N=10240 | M=2560 | Q=4.00 | alpha=4.91 | D_ks=0.03 | alpha-hat=8.29 | num traps=1
Layer 154
Layer=154 | N=7680 | M=2560 | Q=3.00 | alpha=3.88 | D_ks=0.03 | alpha-hat=6.10 | num traps=0
Layer 155
Layer=155 | N=2560 | M=2560 | Q=1.00 | alpha=3.69 | D_ks=0.06 | alpha-hat=6.60 | num traps=4
Layer 157
Layer=157 | N=10240 | M=2560 | Q=4.00 | alpha=3.19 | D_ks=0.01 | alpha-hat=7.28 | num traps=0
Layer 158
Layer=158 | N=10240 | M=2560 | Q=4.00 | alpha=4.92 | D_ks=0.02 | alpha-hat=8.53 | num traps=0
Layer 160
Layer=160 | N=7680 | M=2560 | Q=3.00 | alpha=3.60 | D_ks=0.02 | alpha-hat=5.76 | num traps=0
Layer 161
Layer=161 | N=2560 | M=2560 | Q=1.00 | alpha=3.05 | D_ks=0.08 | alpha-hat=6.05 | num traps=22
Layer 163
Layer=163 | N=10240 | M=2560 | Q=4.00 | alpha=3.15 | D_ks=0.01 | alpha-hat=7.21 | num traps=0
Layer 164
Layer=164 | N=10240 | M=2560 | Q=4.00 | alpha=4.54 | D_ks=0.01 | alpha-hat=8.21 | num traps=3
Layer 166
Layer=166 | N=7680 | M=2560 | Q=3.00 | alpha=3.57 | D_ks=0.02 | alpha-hat=5.82 | num traps=0
Layer 167
Layer=167 | N=2560 | M=2560 | Q=1.00 | alpha=2.03 | D_ks=0.05 | alpha-hat=4.69 | num traps=39
Layer 169
Layer=169 | N=10240 | M=2560 | Q=4.00 | alpha=3.12 | D_ks=0.01 | alpha-hat=7.38 | num traps=0
Layer 170
Layer=170 | N=10240 | M=2560 | Q=4.00 | alpha=4.13 | D_ks=0.03 | alpha-hat=10.78 | num traps=4
Layer 172
Layer=172 | N=7680 | M=2560 | Q=3.00 | alpha=3.40 | D_ks=0.01 | alpha-hat=6.40 | num traps=0
Layer 173
Layer=173 | N=2560 | M=2560 | Q=1.00 | alpha=2.07 | D_ks=0.05 | alpha-hat=4.38 | num traps=36
Layer 175
Layer=175 | N=10240 | M=2560 | Q=4.00 | alpha=3.21 | D_ks=0.01 | alpha-hat=7.49 | num traps=0
Layer 176
Layer=176 | N=10240 | M=2560 | Q=4.00 | alpha=4.31 | D_ks=0.03 | alpha-hat=10.29 | num traps=1
Layer 178
Layer=178 | N=7680 | M=2560 | Q=3.00 | alpha=3.18 | D_ks=0.01 | alpha-hat=5.95 | num traps=0
Layer 179
Layer=179 | N=2560 | M=2560 | Q=1.00 | alpha=2.75 | D_ks=0.04 | alpha-hat=6.26 | num traps=35
Layer 181
Layer=181 | N=10240 | M=2560 | Q=4.00 | alpha=3.46 | D_ks=0.02 | alpha-hat=7.88 | num traps=0
Layer 182
Layer=182 | N=10240 | M=2560 | Q=4.00 | alpha=4.48 | D_ks=0.04 | alpha-hat=9.68 | num traps=4