gpt_train_6_768

This model is a fine-tuned version of openai-community/gpt2 on the gokuls/wiki_book_corpus_raw_dataset_tiny dataset. It achieves the following results on the evaluation set:

• Loss: 7.6211
• Accuracy: 0.1118

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 32
• eval_batch_size: 32
• seed: 10
• distributed_type: multi-GPU
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 100
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
10.9531	0.0000	1	10.9609	0.0000
10.9453	0.0001	2	10.9609	0.0000
10.9609	0.0001	3	10.9609	0.0000
10.9531	0.0002	4	10.9609	0.0000
10.9531	0.0002	5	10.9609	0.0000
10.9609	0.0003	6	10.9609	0.0000
10.9531	0.0003	7	10.9609	0.0000
10.9531	0.0004	8	10.9609	0.0000
10.9609	0.0004	9	10.9609	0.0000
10.9609	0.0005	10	10.9609	0.0000
10.9609	0.0005	11	10.9609	0.0000
10.9609	0.0006	12	10.9609	0.0000
10.9609	0.0006	13	10.9609	0.0000
10.9531	0.0007	14	10.9609	0.0000
10.9531	0.0007	15	10.9609	0.0000
10.9531	0.0008	16	10.9609	0.0000
10.9609	0.0008	17	10.9609	0.0000
10.9609	0.0009	18	10.7188	0.0158
10.7422	0.0009	19	10.5078	0.0392
10.5391	0.0010	20	10.3516	0.0403
10.4219	0.0010	21	10.2422	0.0403
10.2656	0.0011	22	10.1484	0.0401
10.2109	0.0011	23	10.0703	0.0403
10.125	0.0012	24	10.0078	0.0420
10.0312	0.0012	25	9.9531	0.0449
9.9766	0.0013	26	9.9062	0.0489
9.9844	0.0013	27	9.8672	0.0504
9.9062	0.0014	28	9.8359	0.0508
9.875	0.0014	29	9.8047	0.0506
9.8359	0.0015	30	9.7812	0.0511
9.8516	0.0015	31	9.75	0.0513
9.875	0.0015	32	9.7344	0.0511
9.7109	0.0016	33	9.7109	0.0514
9.7266	0.0016	34	9.6953	0.0506
9.7344	0.0017	35	9.6797	0.0508
9.7344	0.0017	36	9.6641	0.0524
9.7422	0.0018	37	9.6484	0.0543
9.6094	0.0018	38	9.6328	0.0555
9.7188	0.0019	39	9.625	0.0562
9.6484	0.0019	40	9.6094	0.0572
9.6641	0.0020	41	9.6016	0.0573
9.6562	0.0020	42	9.5859	0.0573
9.6406	0.0021	43	9.5781	0.0590
9.5234	0.0021	44	9.5625	0.0605
9.5938	0.0022	45	9.5547	0.0615
9.5859	0.0022	46	9.5391	0.0623
9.5703	0.0023	47	9.5312	0.0626
9.5078	0.0023	48	9.5156	0.0627
9.6484	0.0024	49	9.5078	0.0628
9.4922	0.0024	50	9.4922	0.0629
9.5391	0.0025	51	9.4844	0.0633
9.5859	0.0025	52	9.4688	0.0639
9.5234	0.0026	53	9.4609	0.0647
9.5	0.0026	54	9.4453	0.0657
9.4609	0.0027	55	9.4375	0.0666
9.4531	0.0027	56	9.4219	0.0677
9.4375	0.0028	57	9.4141	0.0681
9.4141	0.0028	58	9.3984	0.0683
9.4844	0.0029	59	9.3906	0.0683
9.4297	0.0029	60	9.3828	0.0691
9.375	0.0030	61	9.3672	0.0694
9.4219	0.0030	62	9.3594	0.0693
9.3672	0.0031	63	9.3438	0.0692
9.3906	0.0031	64	9.3359	0.0700
9.4375	0.0031	65	9.3203	0.0713
9.3203	0.0032	66	9.3125	0.0718
9.375	0.0032	67	9.3047	0.0723
9.3516	0.0033	68	9.2891	0.0726
9.3359	0.0033	69	9.2812	0.0725
9.2891	0.0034	70	9.2656	0.0722
9.3047	0.0034	71	9.2578	0.0721
9.3125	0.0035	72	9.2422	0.0721
9.2891	0.0035	73	9.2344	0.0728
9.2578	0.0036	74	9.2188	0.0741
9.2422	0.0036	75	9.2109	0.0751
9.2031	0.0037	76	9.1953	0.0760
9.1641	0.0037	77	9.1875	0.0765
9.1953	0.0038	78	9.1797	0.0767
9.1484	0.0038	79	9.1641	0.0767
9.1953	0.0039	80	9.1562	0.0768
9.1328	0.0039	81	9.1406	0.0772
9.2109	0.0040	82	9.1328	0.0773
9.0547	0.0040	83	9.125	0.0771
9.1094	0.0041	84	9.1094	0.0770
9.1797	0.0041	85	9.1016	0.0768
9.1484	0.0042	86	9.0859	0.0770
9.1016	0.0042	87	9.0781	0.0772
9.1172	0.0043	88	9.0703	0.0772
9.0078	0.0043	89	9.0625	0.0772
8.9688	0.0044	90	9.0469	0.0775
9.0938	0.0044	91	9.0391	0.0781
9.0547	0.0045	92	9.0312	0.0791
9.0703	0.0045	93	9.0156	0.0798
9.0234	0.0046	94	9.0078	0.0801
9.0547	0.0046	95	9.0	0.0803
9.0391	0.0046	96	8.9844	0.0804
9.0703	0.0047	97	8.9766	0.0805
9.0234	0.0047	98	8.9688	0.0806
8.9062	0.0048	99	8.9609	0.0810
8.9688	0.0048	100	8.9453	0.0815
8.9609	0.0049	101	8.9375	0.0818
9.0391	0.0049	102	8.9297	0.0820
8.8984	0.0050	103	8.9141	0.0821
8.9688	0.0050	104	8.9062	0.0820
8.9922	0.0051	105	8.8984	0.0819
8.9062	0.0051	106	8.8906	0.0819
8.9062	0.0052	107	8.875	0.0822
8.9609	0.0052	108	8.8672	0.0826
8.8672	0.0053	109	8.8594	0.0831
8.8828	0.0053	110	8.8438	0.0836
8.8516	0.0054	111	8.8359	0.0840
8.8828	0.0054	112	8.8281	0.0844
8.9297	0.0055	113	8.8203	0.0845
8.9062	0.0055	114	8.8125	0.0847
8.7969	0.0056	115	8.7969	0.0851
8.8203	0.0056	116	8.7891	0.0855
8.8047	0.0057	117	8.7812	0.0858
8.7422	0.0057	118	8.7734	0.0858
8.7266	0.0058	119	8.7656	0.0860
8.6953	0.0058	120	8.7578	0.0861
8.7422	0.0059	121	8.75	0.0861
8.75	0.0059	122	8.7344	0.0863
8.7422	0.0060	123	8.7266	0.0867
8.6953	0.0060	124	8.7188	0.0870
8.6328	0.0061	125	8.7109	0.0871
8.7188	0.0061	126	8.7031	0.0871
8.7891	0.0062	127	8.6953	0.0873
8.7344	0.0062	128	8.6875	0.0874
8.7578	0.0062	129	8.6797	0.0874
8.6953	0.0063	130	8.6641	0.0874
8.7266	0.0063	131	8.6562	0.0877
8.6562	0.0064	132	8.6484	0.0882
8.7188	0.0064	133	8.6406	0.0884
8.6797	0.0065	134	8.6328	0.0883
8.6562	0.0065	135	8.625	0.0883
8.6172	0.0066	136	8.6172	0.0885
8.6406	0.0066	137	8.6094	0.0889
8.6797	0.0067	138	8.6016	0.0895
8.6406	0.0067	139	8.5938	0.0901
8.6094	0.0068	140	8.5859	0.0905
8.7031	0.0068	141	8.5781	0.0906
8.6797	0.0069	142	8.5703	0.0906
8.6719	0.0069	143	8.5625	0.0905
8.5703	0.0070	144	8.5547	0.0905
8.6016	0.0070	145	8.5391	0.0907
8.5859	0.0071	146	8.5312	0.0909
8.5469	0.0071	147	8.5312	0.0911
8.5391	0.0072	148	8.5234	0.0913
8.5391	0.0072	149	8.5156	0.0915
8.5625	0.0073	150	8.5078	0.0918
8.5469	0.0073	151	8.5	0.0921
8.5234	0.0074	152	8.4922	0.0920
8.5469	0.0074	153	8.4844	0.0922
8.4766	0.0075	154	8.4766	0.0923
8.4453	0.0075	155	8.4688	0.0925
8.375	0.0076	156	8.4609	0.0929
8.5156	0.0076	157	8.4531	0.0932
8.5234	0.0077	158	8.4453	0.0934
8.4844	0.0077	159	8.4375	0.0936
8.5	0.0077	160	8.4297	0.0938
8.3984	0.0078	161	8.4219	0.0936
8.5156	0.0078	162	8.4219	0.0935
8.4453	0.0079	163	8.4141	0.0934
8.4375	0.0079	164	8.4062	0.0937
8.4297	0.0080	165	8.3984	0.0944
8.4453	0.0080	166	8.3906	0.0953
8.4453	0.0081	167	8.3828	0.0961
8.3828	0.0081	168	8.375	0.0963
8.3828	0.0082	169	8.3672	0.0964
8.4297	0.0082	170	8.3594	0.0963
8.3828	0.0083	171	8.3516	0.0963
8.3984	0.0083	172	8.3438	0.0965
8.4375	0.0084	173	8.3359	0.0967
8.3906	0.0084	174	8.3281	0.0970
8.2578	0.0085	175	8.3203	0.0973
8.2891	0.0085	176	8.3203	0.0976
8.3125	0.0086	177	8.3125	0.0978
8.3359	0.0086	178	8.3047	0.0981
8.375	0.0087	179	8.2969	0.0982
8.3125	0.0087	180	8.2891	0.0982
8.2656	0.0088	181	8.2812	0.0981
8.2812	0.0088	182	8.2734	0.0980
8.3203	0.0089	183	8.2656	0.0979
8.2344	0.0089	184	8.2656	0.0979
8.3203	0.0090	185	8.2578	0.0982
8.2422	0.0090	186	8.25	0.0986
8.2344	0.0091	187	8.2422	0.0990
8.2891	0.0091	188	8.2344	0.0995
8.1875	0.0092	189	8.2266	0.0997
8.2188	0.0092	190	8.2266	0.0997
8.1953	0.0093	191	8.2188	0.0994
8.2578	0.0093	192	8.2109	0.0991
8.2188	0.0093	193	8.2031	0.0991
8.2812	0.0094	194	8.1953	0.0991
8.1328	0.0094	195	8.1875	0.0992
8.2578	0.0095	196	8.1875	0.0992
8.1719	0.0095	197	8.1797	0.0996
8.1953	0.0096	198	8.1719	0.1000
8.1875	0.0096	199	8.1641	0.1002
8.1953	0.0097	200	8.1562	0.1006
8.1406	0.0097	201	8.1562	0.1008
8.1797	0.0098	202	8.1484	0.1008
8.1484	0.0098	203	8.1406	0.1006
8.1719	0.0099	204	8.1328	0.1004
8.1641	0.0099	205	8.125	0.1002
8.2422	0.0100	206	8.125	0.1002
8.0703	0.0100	207	8.1172	0.1005
8.1328	0.0101	208	8.1094	0.1011
8.1562	0.0101	209	8.1016	0.1016
8.1797	0.0102	210	8.1016	0.1020
8.1641	0.0102	211	8.0938	0.1022
8.1016	0.0103	212	8.0859	0.1022
8.1719	0.0103	213	8.0781	0.1020
8.1094	0.0104	214	8.0703	0.1017
8.0469	0.0104	215	8.0703	0.1016
8.0859	0.0105	216	8.0625	0.1019
8.0625	0.0105	217	8.0547	0.1023
8.1406	0.0106	218	8.0547	0.1025
8.0547	0.0106	219	8.0469	0.1027
8.0234	0.0107	220	8.0391	0.1029
8.0469	0.0107	221	8.0391	0.1029
8.0312	0.0108	222	8.0312	0.1029
8.0391	0.0108	223	8.0234	0.1028
8.0391	0.0108	224	8.0156	0.1029
8.0859	0.0109	225	8.0156	0.1029
8.0391	0.0109	226	8.0078	0.1028
7.9883	0.0110	227	8.0	0.1028
8.0625	0.0110	228	7.9961	0.1029
8.1094	0.0111	229	7.9883	0.1032
8.0391	0.0111	230	7.9844	0.1034
8.0078	0.0112	231	7.9805	0.1037
8.0859	0.0112	232	7.9727	0.1039
7.9961	0.0113	233	7.9688	0.1039
8.0312	0.0113	234	7.9648	0.1039
8.0391	0.0114	235	7.9570	0.1037
7.9609	0.0114	236	7.9531	0.1037
7.9336	0.0115	237	7.9492	0.1038
7.9258	0.0115	238	7.9453	0.1040
7.9531	0.0116	239	7.9375	0.1042
7.9805	0.0116	240	7.9336	0.1045
8.0078	0.0117	241	7.9297	0.1048
7.8906	0.0117	242	7.9258	0.1051
7.9727	0.0118	243	7.9180	0.1054
7.9336	0.0118	244	7.9141	0.1055
7.9375	0.0119	245	7.9062	0.1055
7.9922	0.0119	246	7.9023	0.1054
7.9609	0.0120	247	7.8984	0.1053
7.8945	0.0120	248	7.8906	0.1053
7.8203	0.0121	249	7.8867	0.1055
7.8984	0.0121	250	7.8828	0.1057
7.9023	0.0122	251	7.8789	0.1058
7.918	0.0122	252	7.875	0.1058
7.832	0.0123	253	7.8672	0.1058
7.9609	0.0123	254	7.8633	0.1056
7.9531	0.0124	255	7.8594	0.1057
7.8125	0.0124	256	7.8555	0.1059
7.9648	0.0124	257	7.8516	0.1066
7.832	0.0125	258	7.8438	0.1068
7.8008	0.0125	259	7.8398	0.1069
7.8281	0.0126	260	7.8359	0.1069
7.8477	0.0126	261	7.8320	0.1069
7.8086	0.0127	262	7.8281	0.1069
7.8281	0.0127	263	7.8203	0.1069
7.8906	0.0128	264	7.8164	0.1067
7.8477	0.0128	265	7.8125	0.1067
7.8867	0.0129	266	7.8086	0.1067
7.8359	0.0129	267	7.8047	0.1069
7.7969	0.0130	268	7.8008	0.1074
7.8711	0.0130	269	7.7969	0.1079
7.7656	0.0131	270	7.7930	0.1081
7.8008	0.0131	271	7.7852	0.1080
7.8594	0.0132	272	7.7812	0.1079
7.8125	0.0132	273	7.7773	0.1077
7.7617	0.0133	274	7.7734	0.1075
7.7227	0.0133	275	7.7695	0.1074
7.8164	0.0134	276	7.7656	0.1077
7.7383	0.0134	277	7.7617	0.1081
7.8984	0.0135	278	7.7578	0.1085
7.793	0.0135	279	7.7539	0.1088
7.707	0.0136	280	7.75	0.1088
7.8086	0.0136	281	7.7422	0.1088
7.7773	0.0137	282	7.7383	0.1088
7.6875	0.0137	283	7.7344	0.1087
7.7188	0.0138	284	7.7305	0.1088
7.7539	0.0138	285	7.7266	0.1091
7.8008	0.0139	286	7.7227	0.1094
7.7578	0.0139	287	7.7188	0.1097
7.7148	0.0139	288	7.7148	0.1099
7.7266	0.0140	289	7.7109	0.1100
7.7031	0.0140	290	7.7070	0.1099
7.7383	0.0141	291	7.7031	0.1098
7.7266	0.0141	292	7.6953	0.1098
7.75	0.0142	293	7.6914	0.1100
7.7031	0.0142	294	7.6914	0.1102
7.7305	0.0143	295	7.6875	0.1103
7.7188	0.0143	296	7.6836	0.1104
7.6719	0.0144	297	7.6797	0.1105
7.6289	0.0144	298	7.6758	0.1108
7.6719	0.0145	299	7.6719	0.1110
7.7695	0.0145	300	7.6641	0.1111
7.7812	0.0146	301	7.6602	0.1109
7.707	0.0146	302	7.6562	0.1110
7.7539	0.0147	303	7.6523	0.1111
7.5898	0.0147	304	7.6523	0.1114
7.668	0.0148	305	7.6484	0.1116
7.6602	0.0148	306	7.6445	0.1116
7.6953	0.0149	307	7.6406	0.1117
7.7031	0.0149	308	7.6367	0.1118
7.6914	0.0150	309	7.6328	0.1120
7.582	0.0150	310	7.6289	0.1119
7.6445	0.0151	311	7.625	0.1118
7.5234	0.0151	312	7.6211	0.1118

Framework versions

• Transformers 4.41.2
• Pytorch 2.1.0a0+32f93b1
• Datasets 2.20.0
• Tokenizers 0.19.1