RedPajama-Instruct-3b-v1


Find this model in the RedPajama model summary

RedPajama-Instruct-3b-v1 Model Summary Plots

RedPajama-Instruct-3b-v1 Model Selected Details
  layer_type N M Q alpha D alpha-hat log_SN rank_loss
layer_id                  
4 DENSE 7680 2560 3.00 2.11 0.04 6.75 3.20 0
5 DENSE 2560 2560 1.00 3.21 0.05 3.93 1.22 6
6 DENSE 10240 2560 4.00 5.10 0.07 11.12 2.18 0
7 DENSE 10240 2560 4.00 7.90 0.06 14.66 1.85 0
10 DENSE 7680 2560 3.00 3.69 0.02 7.59 2.06 0
11 DENSE 2560 2560 1.00 4.85 0.02 7.03 1.45 3
12 DENSE 10240 2560 4.00 3.07 0.02 6.85 2.23 0
13 DENSE 10240 2560 4.00 4.05 0.04 6.20 1.53 0
16 DENSE 7680 2560 3.00 2.53 0.04 4.84 1.91 0
17 DENSE 2560 2560 1.00 4.03 0.03 6.63 1.64 3
18 DENSE 10240 2560 4.00 3.21 0.01 8.42 2.63 0
19 DENSE 10240 2560 4.00 3.81 0.01 7.58 1.99 0
22 DENSE 7680 2560 3.00 3.28 0.02 5.94 1.81 0
23 DENSE 2560 2560 1.00 3.90 0.04 6.67 1.71 3
24 DENSE 10240 2560 4.00 3.39 0.02 7.25 2.14 0
25 DENSE 10240 2560 4.00 4.07 0.01 7.03 1.72 0
28 DENSE 7680 2560 3.00 3.55 0.01 6.78 1.91 0
29 DENSE 2560 2560 1.00 3.58 0.04 5.64 1.57 2
30 DENSE 10240 2560 4.00 3.37 0.02 7.27 2.16 0
31 DENSE 10240 2560 4.00 3.96 0.02 6.83 1.72 0
34 DENSE 7680 2560 3.00 3.57 0.01 6.92 1.94 0
35 DENSE 2560 2560 1.00 3.70 0.04 5.83 1.58 3
36 DENSE 10240 2560 4.00 3.46 0.02 7.43 2.15 0
37 DENSE 10240 2560 4.00 3.95 0.02 6.31 1.60 0
40 DENSE 7680 2560 3.00 3.79 0.02 7.26 1.92 0
41 DENSE 2560 2560 1.00 5.01 0.04 7.89 1.57 2
42 DENSE 10240 2560 4.00 3.45 0.01 7.44 2.16 0
43 DENSE 10240 2560 4.00 3.79 0.04 6.00 1.58 0
46 DENSE 7680 2560 3.00 3.60 0.01 6.85 1.91 0
47 DENSE 2560 2560 1.00 4.47 0.04 6.95 1.55 4
48 DENSE 10240 2560 4.00 3.42 0.02 7.37 2.15 0
49 DENSE 10240 2560 4.00 3.83 0.04 6.27 1.64 0
52 DENSE 7680 2560 3.00 3.67 0.02 7.02 1.91 0
53 DENSE 2560 2560 1.00 4.56 0.04 7.60 1.67 3
54 DENSE 10240 2560 4.00 3.42 0.03 7.34 2.15 0
55 DENSE 10240 2560 4.00 3.80 0.04 6.19 1.63 0
58 DENSE 7680 2560 3.00 3.42 0.02 6.44 1.88 0
59 DENSE 2560 2560 1.00 3.79 0.04 5.90 1.55 3
60 DENSE 10240 2560 4.00 3.43 0.03 7.33 2.13 0
61 DENSE 10240 2560 4.00 3.96 0.04 6.70 1.69 0
64 DENSE 7680 2560 3.00 3.30 0.02 6.14 1.86 0
65 DENSE 2560 2560 1.00 4.05 0.03 6.52 1.61 3
66 DENSE 10240 2560 4.00 3.46 0.03 7.31 2.11 0
67 DENSE 10240 2560 4.00 5.74 0.03 9.26 1.62 0
70 DENSE 7680 2560 3.00 3.35 0.02 6.29 1.87 0
71 DENSE 2560 2560 1.00 4.29 0.05 6.90 1.61 2
72 DENSE 10240 2560 4.00 3.42 0.03 7.11 2.08 0
73 DENSE 10240 2560 4.00 3.73 0.04 6.07 1.62 0
76 DENSE 7680 2560 3.00 3.19 0.02 6.06 1.90 0
77 DENSE 2560 2560 1.00 4.53 0.05 6.87 1.52 3
78 DENSE 10240 2560 4.00 3.87 0.03 8.18 2.11 0
79 DENSE 10240 2560 4.00 3.67 0.04 6.14 1.68 0
82 DENSE 7680 2560 3.00 3.21 0.03 5.98 1.86 0
83 DENSE 2560 2560 1.00 4.41 0.04 7.25 1.64 3
84 DENSE 10240 2560 4.00 3.89 0.03 8.10 2.08 0
85 DENSE 10240 2560 4.00 3.74 0.03 6.22 1.66 0
88 DENSE 7680 2560 3.00 3.07 0.03 5.77 1.88 0
89 DENSE 2560 2560 1.00 4.96 0.06 8.06 1.63 4
90 DENSE 10240 2560 4.00 3.54 0.04 7.31 2.07 0
91 DENSE 10240 2560 4.00 3.95 0.03 6.53 1.65 0
94 DENSE 7680 2560 3.00 3.14 0.02 5.84 1.86 0
95 DENSE 2560 2560 1.00 4.66 0.06 7.31 1.57 3
96 DENSE 10240 2560 4.00 3.81 0.03 7.96 2.09 0
97 DENSE 10240 2560 4.00 4.13 0.04 6.90 1.67 0
100 DENSE 7680 2560 3.00 3.16 0.01 5.84 1.85 0
101 DENSE 2560 2560 1.00 3.36 0.04 5.62 1.67 3
102 DENSE 10240 2560 4.00 3.74 0.03 7.90 2.11 0
103 DENSE 10240 2560 4.00 4.35 0.03 6.94 1.60 0
106 DENSE 7680 2560 3.00 3.49 0.02 6.46 1.85 0
107 DENSE 2560 2560 1.00 3.83 0.06 6.33 1.65 3
108 DENSE 10240 2560 4.00 4.46 0.03 9.39 2.10 0
109 DENSE 10240 2560 4.00 4.62 0.02 7.77 1.68 0
112 DENSE 7680 2560 3.00 3.53 0.01 6.69 1.89 0
113 DENSE 2560 2560 1.00 5.27 0.04 9.10 1.73 2
114 DENSE 10240 2560 4.00 4.37 0.03 9.25 2.12 0
115 DENSE 10240 2560 4.00 4.90 0.01 8.35 1.70 0
118 DENSE 7680 2560 3.00 3.69 0.01 7.01 1.90 0
119 DENSE 2560 2560 1.00 4.86 0.05 8.04 1.65 2
120 DENSE 10240 2560 4.00 4.02 0.03 8.65 2.15 0
121 DENSE 10240 2560 4.00 5.28 0.01 8.83 1.67 0
124 DENSE 7680 2560 3.00 4.09 0.03 8.05 1.97 0
125 DENSE 2560 2560 1.00 6.36 0.07 10.99 1.73 2
126 DENSE 10240 2560 4.00 3.62 0.02 7.97 2.20 0
127 DENSE 10240 2560 4.00 6.56 0.02 10.53 1.61 0
130 DENSE 7680 2560 3.00 4.29 0.04 8.34 1.95 0
131 DENSE 2560 2560 1.00 7.09 0.07 11.24 1.59 2
132 DENSE 10240 2560 4.00 3.99 0.03 8.78 2.20 0
133 DENSE 10240 2560 4.00 6.79 0.02 10.12 1.49 0
136 DENSE 7680 2560 3.00 3.92 0.03 7.73 1.97 0
137 DENSE 2560 2560 1.00 6.85 0.06 10.94 1.60 2
138 DENSE 10240 2560 4.00 4.03 0.04 8.89 2.20 0
139 DENSE 10240 2560 4.00 7.03 0.01 9.65 1.37 0
142 DENSE 7680 2560 3.00 4.23 0.03 8.45 2.00 0
143 DENSE 2560 2560 1.00 7.25 0.06 10.49 1.45 3
144 DENSE 10240 2560 4.00 4.05 0.04 8.86 2.18 0
145 DENSE 10240 2560 4.00 6.96 0.03 10.22 1.47 0
148 DENSE 7680 2560 3.00 3.15 0.01 6.31 2.00 0
149 DENSE 2560 2560 1.00 7.60 0.08 10.84 1.43 1
150 DENSE 10240 2560 4.00 3.80 0.04 8.32 2.19 0
151 DENSE 10240 2560 4.00 6.59 0.04 9.35 1.42 0
154 DENSE 7680 2560 3.00 3.76 0.01 7.52 2.00 0
155 DENSE 2560 2560 1.00 6.33 0.04 9.32 1.47 2
156 DENSE 10240 2560 4.00 4.07 0.04 8.91 2.19 0
157 DENSE 10240 2560 4.00 7.80 0.02 10.14 1.30 0
160 DENSE 7680 2560 3.00 3.87 0.01 7.68 1.99 0
161 DENSE 2560 2560 1.00 7.43 0.03 9.39 1.26 2
162 DENSE 10240 2560 4.00 3.95 0.04 8.59 2.17 0
163 DENSE 10240 2560 4.00 7.69 0.02 10.32 1.34 0
166 DENSE 7680 2560 3.00 3.66 0.02 7.34 2.01 0
167 DENSE 2560 2560 1.00 6.66 0.02 9.31 1.40 2
168 DENSE 10240 2560 4.00 4.09 0.02 8.92 2.18 0
169 DENSE 10240 2560 4.00 8.40 0.02 12.02 1.43 0
172 DENSE 7680 2560 3.00 3.95 0.02 7.89 2.00 0
173 DENSE 2560 2560 1.00 5.78 0.02 7.39 1.28 2
174 DENSE 10240 2560 4.00 4.20 0.02 9.07 2.16 0
175 DENSE 10240 2560 4.00 6.98 0.03 12.20 1.75 0
178 DENSE 7680 2560 3.00 3.37 0.02 6.72 2.00 0
179 DENSE 2560 2560 1.00 7.36 0.05 9.31 1.26 2
180 DENSE 10240 2560 4.00 4.42 0.01 9.36 2.12 0
181 DENSE 10240 2560 4.00 5.97 0.03 12.79 2.14 0
184 DENSE 7680 2560 3.00 3.46 0.03 10.97 3.17 0
185 DENSE 2560 2560 1.00 5.01 0.03 8.33 1.66 2
186 DENSE 10240 2560 4.00 4.31 0.02 8.95 2.08 0
187 DENSE 10240 2560 4.00 4.92 0.02 10.63 2.16 0
190 DENSE 7680 2560 3.00 2.70 0.05 8.54 3.16 0
191 DENSE 2560 2560 1.00 3.94 0.06 8.62 2.19 2
192 DENSE 10240 2560 4.00 4.00 0.03 8.38 2.10 0
193 DENSE 10240 2560 4.00 4.03 0.04 8.69 2.16 0