nanochat CPU / Apple-Silicon end-to-end run
command: bash runs/runcpu.sh && python -m scripts.chat_cli -p "What is the capital of France?"

=== Tokenizer, base training, and base evaluation ===
Cloning into 'repo'...
Using CPython 3.10.19
Creating virtual environment at: .venv
Activate with: source .venv/bin/activate
Resolving despite existing lockfile due to removal of global exclude newer
Resolved 105 packages in 1.23s
Downloading setuptools (1.1MiB)
Downloading networkx (1.6MiB)
Downloading sympy (6.0MiB)
Downloading pyarrow (29.8MiB)
Downloading hf-xet (2.5MiB)
Downloading pydantic-core (1.8MiB)
Downloading wandb (17.5MiB)
Downloading numpy (13.3MiB)
Downloading torch (70.9MiB)
 Downloaded setuptools
 Downloaded networkx
 Downloaded pydantic-core
 Downloaded hf-xet
 Downloaded torch
 Downloaded sympy
 Downloaded numpy
 Downloaded wandb
 Downloaded pyarrow
Prepared 40 packages in 14.36s
Installed 40 packages in 266ms
 + annotated-types==0.7.0
 + certifi==2025.8.3
 + charset-normalizer==3.4.3
 + click==8.2.1
 + filelock==3.19.1
 + fsspec==2025.3.0
 + gitdb==4.0.12
 + gitpython==3.1.45
 + hf-xet==1.1.9
 + huggingface-hub==0.34.4
 + idna==3.10
 + jinja2==3.1.6
 + kernels==0.11.7
 + markupsafe==3.0.2
 + mpmath==1.3.0
 + networkx==3.4.2
 + numpy==1.26.4
 + packaging==25.0
 + platformdirs==4.4.0
 + protobuf==6.32.0
 + psutil==7.1.0
 + pyarrow==21.0.0
 + pydantic==2.11.7
 + pydantic-core==2.33.2
 + pyyaml==6.0.2
 + regex==2025.9.1
 + requests==2.32.5
 + rustbpe==0.1.0
 + sentry-sdk==2.35.2
 + setuptools==80.9.0
 + smmap==5.0.2
 + sympy==1.14.0
 + tiktoken==0.11.0
 + tomli==2.2.1
 + torch==2.9.1
 + tqdm==4.67.1
 + typing-extensions==4.15.0
 + typing-inspection==0.4.1
 + urllib3==2.5.0
 + wandb==0.21.3
Downloading 9 shards using 4 workers...
Target directory: $ORX_RUN_DIR/repo/.cache/nanochat/base_data_climbmix

Downloading shard_00000.parquet...
Downloading shard_00001.parquet...
Downloading shard_00002.parquet...
Downloading shard_00003.parquet...
Successfully downloaded shard_00000.parquet
Downloading shard_00004.parquet...
Successfully downloaded shard_00001.parquet
Downloading shard_00005.parquet...
Successfully downloaded shard_00003.parquet
Downloading shard_00006.parquet...
Successfully downloaded shard_00002.parquet
Downloading shard_00007.parquet...
Successfully downloaded shard_00005.parquet
Downloading shard_06542.parquet...
Successfully downloaded shard_00004.parquet
Successfully downloaded shard_00006.parquet
Successfully downloaded shard_00007.parquet
Successfully downloaded shard_06542.parquet
Done! Downloaded: 9/9 shards to $ORX_RUN_DIR/repo/.cache/nanochat/base_data_climbmix
max_chars: 2,000,000,000
doc_cap: 10,000
vocab_size: 32,768
2026-08-04 10:51:52,050 - rustbpe - INFO - Processing sequences from iterator (buffer_size: 8192)
2026-08-04 10:52:25,784 - rustbpe - INFO - Processed 677888 sequences total, 1952346 unique
2026-08-04 10:52:25,844 - rustbpe - INFO - Starting BPE training: 32503 merges to compute
2026-08-04 10:52:25,844 - rustbpe - INFO - Computing initial pair counts from 1952346 unique sequences
2026-08-04 10:52:27,731 - rustbpe - INFO - Building heap with 18999 unique pairs
2026-08-04 10:52:27,732 - rustbpe - INFO - Starting merge loop
2026-08-04 10:52:30,514 - rustbpe - INFO - Progress: 1% (326/32503 merges) - Last merge: (276, 115) -> 581 (frequency: 547375)
2026-08-04 10:52:30,959 - rustbpe - INFO - Progress: 2% (651/32503 merges) - Last merge: (316, 291) -> 906 (frequency: 223745)
2026-08-04 10:52:31,216 - rustbpe - INFO - Progress: 3% (976/32503 merges) - Last merge: (257, 477) -> 1231 (frequency: 139428)
2026-08-04 10:52:31,409 - rustbpe - INFO - Progress: 4% (1301/32503 merges) - Last merge: (1447, 497) -> 1556 (frequency: 98242)
2026-08-04 10:52:31,548 - rustbpe - INFO - Progress: 5% (1626/32503 merges) - Last merge: (274, 284) -> 1881 (frequency: 75518)
2026-08-04 10:52:31,673 - rustbpe - INFO - Progress: 6% (1951/32503 merges) - Last merge: (1451, 2021) -> 2206 (frequency: 60503)
2026-08-04 10:52:31,769 - rustbpe - INFO - Progress: 7% (2276/32503 merges) - Last merge: (1287, 268) -> 2531 (frequency: 49702)
2026-08-04 10:52:31,854 - rustbpe - INFO - Progress: 8% (2601/32503 merges) - Last merge: (287, 2518) -> 2856 (frequency: 41911)
2026-08-04 10:52:31,971 - rustbpe - INFO - Progress: 9% (2926/32503 merges) - Last merge: (1031, 2449) -> 3181 (frequency: 35604)
2026-08-04 10:52:32,065 - rustbpe - INFO - Progress: 10% (3251/32503 merges) - Last merge: (2375, 344) -> 3506 (frequency: 30493)
2026-08-04 10:52:32,147 - rustbpe - INFO - Progress: 11% (3576/32503 merges) - Last merge: (1231, 1464) -> 3831 (frequency: 26858)
2026-08-04 10:52:32,210 - rustbpe - INFO - Progress: 12% (3901/32503 merges) - Last merge: (404, 261) -> 4156 (frequency: 23616)
2026-08-04 10:52:32,293 - rustbpe - INFO - Progress: 13% (4226/32503 merges) - Last merge: (65, 114) -> 4481 (frequency: 21288)
2026-08-04 10:52:32,376 - rustbpe - INFO - Progress: 14% (4551/32503 merges) - Last merge: (973, 2724) -> 4806 (frequency: 19098)
2026-08-04 10:52:32,423 - rustbpe - INFO - Progress: 15% (4876/32503 merges) - Last merge: (4082, 273) -> 5131 (frequency: 17288)
2026-08-04 10:52:32,481 - rustbpe - INFO - Progress: 16% (5201/32503 merges) - Last merge: (2310, 284) -> 5456 (frequency: 15786)
2026-08-04 10:52:32,527 - rustbpe - INFO - Progress: 17% (5526/32503 merges) - Last merge: (319, 4229) -> 5781 (frequency: 14559)
2026-08-04 10:52:32,580 - rustbpe - INFO - Progress: 18% (5851/32503 merges) - Last merge: (118, 258) -> 6106 (frequency: 13363)
2026-08-04 10:52:32,640 - rustbpe - INFO - Progress: 19% (6176/32503 merges) - Last merge: (493, 451) -> 6431 (frequency: 12442)
2026-08-04 10:52:32,704 - rustbpe - INFO - Progress: 20% (6501/32503 merges) - Last merge: (3748, 280) -> 6756 (frequency: 11481)
2026-08-04 10:52:32,765 - rustbpe - INFO - Progress: 21% (6826/32503 merges) - Last merge: (358, 940) -> 7081 (frequency: 10681)
2026-08-04 10:52:32,799 - rustbpe - INFO - Progress: 22% (7151/32503 merges) - Last merge: (1223, 2974) -> 7406 (frequency: 9974)
2026-08-04 10:52:32,845 - rustbpe - INFO - Progress: 23% (7476/32503 merges) - Last merge: (2711, 390) -> 7731 (frequency: 9384)
2026-08-04 10:52:32,893 - rustbpe - INFO - Progress: 24% (7801/32503 merges) - Last merge: (408, 117) -> 8056 (frequency: 8817)
2026-08-04 10:52:32,921 - rustbpe - INFO - Progress: 25% (8126/32503 merges) - Last merge: (83, 375) -> 8381 (frequency: 8317)
2026-08-04 10:52:32,974 - rustbpe - INFO - Progress: 26% (8451/32503 merges) - Last merge: (4136, 914) -> 8706 (frequency: 7869)
2026-08-04 10:52:33,027 - rustbpe - INFO - Progress: 27% (8776/32503 merges) - Last merge: (282, 597) -> 9031 (frequency: 7410)
2026-08-04 10:52:33,048 - rustbpe - INFO - Progress: 28% (9101/32503 merges) - Last merge: (7179, 299) -> 9356 (frequency: 7024)
2026-08-04 10:52:33,090 - rustbpe - INFO - Progress: 29% (9426/32503 merges) - Last merge: (416, 1146) -> 9681 (frequency: 6654)
2026-08-04 10:52:33,122 - rustbpe - INFO - Progress: 30% (9751/32503 merges) - Last merge: (7196, 336) -> 10006 (frequency: 6358)
2026-08-04 10:52:33,144 - rustbpe - INFO - Progress: 31% (10076/32503 merges) - Last merge: (859, 439) -> 10331 (frequency: 6017)
2026-08-04 10:52:33,173 - rustbpe - INFO - Progress: 32% (10401/32503 merges) - Last merge: (45, 3988) -> 10656 (frequency: 5728)
2026-08-04 10:52:33,208 - rustbpe - INFO - Progress: 33% (10726/32503 merges) - Last merge: (7322, 290) -> 10981 (frequency: 5454)
2026-08-04 10:52:33,238 - rustbpe - INFO - Progress: 34% (11052/32503 merges) - Last merge: (373, 450) -> 11307 (frequency: 5205)
2026-08-04 10:52:33,278 - rustbpe - INFO - Progress: 35% (11377/32503 merges) - Last merge: (98, 551) -> 11632 (frequency: 4975)
2026-08-04 10:52:33,316 - rustbpe - INFO - Progress: 36% (11702/32503 merges) - Last merge: (1889, 1428) -> 11957 (frequency: 4745)
2026-08-04 10:52:33,342 - rustbpe - INFO - Progress: 37% (12027/32503 merges) - Last merge: (10729, 265) -> 12282 (frequency: 4532)
2026-08-04 10:52:33,376 - rustbpe - INFO - Progress: 38% (12352/32503 merges) - Last merge: (6591, 73) -> 12607 (frequency: 4355)
2026-08-04 10:52:33,398 - rustbpe - INFO - Progress: 39% (12677/32503 merges) - Last merge: (780, 98) -> 12932 (frequency: 4162)
2026-08-04 10:52:33,450 - rustbpe - INFO - Progress: 40% (13002/32503 merges) - Last merge: (1782, 499) -> 13257 (frequency: 3990)
2026-08-04 10:52:33,490 - rustbpe - INFO - Progress: 41% (13327/32503 merges) - Last merge: (346, 12090) -> 13582 (frequency: 3831)
2026-08-04 10:52:33,510 - rustbpe - INFO - Progress: 42% (13652/32503 merges) - Last merge: (468, 441) -> 13907 (frequency: 3691)
2026-08-04 10:52:33,538 - rustbpe - INFO - Progress: 43% (13977/32503 merges) - Last merge: (2629, 12137) -> 14232 (frequency: 3559)
2026-08-04 10:52:33,563 - rustbpe - INFO - Progress: 44% (14302/32503 merges) - Last merge: (372, 466) -> 14557 (frequency: 3425)
2026-08-04 10:52:33,599 - rustbpe - INFO - Progress: 45% (14627/32503 merges) - Last merge: (309, 763) -> 14882 (frequency: 3293)
2026-08-04 10:52:33,639 - rustbpe - INFO - Progress: 46% (14952/32503 merges) - Last merge: (83, 85) -> 15207 (frequency: 3172)
2026-08-04 10:52:33,662 - rustbpe - INFO - Progress: 47% (15277/32503 merges) - Last merge: (9845, 726) -> 15532 (frequency: 3063)
2026-08-04 10:52:33,681 - rustbpe - INFO - Progress: 48% (15602/32503 merges) - Last merge: (114, 445) -> 15857 (frequency: 2954)
2026-08-04 10:52:33,706 - rustbpe - INFO - Progress: 49% (15927/32503 merges) - Last merge: (2799, 6222) -> 16182 (frequency: 2850)
2026-08-04 10:52:33,725 - rustbpe - INFO - Progress: 50% (16252/32503 merges) - Last merge: (725, 499) -> 16507 (frequency: 2756)
2026-08-04 10:52:33,752 - rustbpe - INFO - Progress: 51% (16577/32503 merges) - Last merge: (256, 667) -> 16832 (frequency: 2667)
2026-08-04 10:52:33,783 - rustbpe - INFO - Progress: 52% (16902/32503 merges) - Last merge: (15104, 602) -> 17157 (frequency: 2581)
2026-08-04 10:52:33,807 - rustbpe - INFO - Progress: 53% (17227/32503 merges) - Last merge: (7455, 4247) -> 17482 (frequency: 2507)
2026-08-04 10:52:33,830 - rustbpe - INFO - Progress: 54% (17552/32503 merges) - Last merge: (15653, 86) -> 17807 (frequency: 2435)
2026-08-04 10:52:33,855 - rustbpe - INFO - Progress: 55% (17877/32503 merges) - Last merge: (8374, 15411) -> 18132 (frequency: 2370)
2026-08-04 10:52:33,872 - rustbpe - INFO - Progress: 56% (18202/32503 merges) - Last merge: (440, 1257) -> 18457 (frequency: 2300)
2026-08-04 10:52:33,886 - rustbpe - INFO - Progress: 57% (18527/32503 merges) - Last merge: (10001, 367) -> 18782 (frequency: 2232)
2026-08-04 10:52:33,912 - rustbpe - INFO - Progress: 58% (18852/32503 merges) - Last merge: (259, 696) -> 19107 (frequency: 2173)
2026-08-04 10:52:33,933 - rustbpe - INFO - Progress: 59% (19177/32503 merges) - Last merge: (431, 281) -> 19432 (frequency: 2111)
2026-08-04 10:52:33,946 - rustbpe - INFO - Progress: 60% (19502/32503 merges) - Last merge: (431, 3034) -> 19757 (frequency: 2051)
2026-08-04 10:52:33,965 - rustbpe - INFO - Progress: 61% (19827/32503 merges) - Last merge: (430, 3872) -> 20082 (frequency: 1994)
2026-08-04 10:52:33,992 - rustbpe - INFO - Progress: 62% (20152/32503 merges) - Last merge: (355, 17408) -> 20407 (frequency: 1945)
2026-08-04 10:52:34,009 - rustbpe - INFO - Progress: 63% (20477/32503 merges) - Last merge: (272, 4626) -> 20732 (frequency: 1894)
2026-08-04 10:52:34,032 - rustbpe - INFO - Progress: 64% (20802/32503 merges) - Last merge: (12034, 515) -> 21057 (frequency: 1841)
2026-08-04 10:52:34,047 - rustbpe - INFO - Progress: 65% (21127/32503 merges) - Last merge: (73, 69) -> 21382 (frequency: 1793)
2026-08-04 10:52:34,075 - rustbpe - INFO - Progress: 66% (21452/32503 merges) - Last merge: (542, 432) -> 21707 (frequency: 1747)
2026-08-04 10:52:34,094 - rustbpe - INFO - Progress: 67% (21778/32503 merges) - Last merge: (1306, 20583) -> 22033 (frequency: 1704)
2026-08-04 10:52:34,112 - rustbpe - INFO - Progress: 68% (22103/32503 merges) - Last merge: (780, 11506) -> 22358 (frequency: 1662)
2026-08-04 10:52:34,126 - rustbpe - INFO - Progress: 69% (22428/32503 merges) - Last merge: (1486, 590) -> 22683 (frequency: 1619)
2026-08-04 10:52:34,159 - rustbpe - INFO - Progress: 70% (22753/32503 merges) - Last merge: (1771, 2947) -> 23008 (frequency: 1577)
2026-08-04 10:52:34,171 - rustbpe - INFO - Progress: 71% (23078/32503 merges) - Last merge: (7078, 284) -> 23333 (frequency: 1539)
2026-08-04 10:52:34,187 - rustbpe - INFO - Progress: 72% (23403/32503 merges) - Last merge: (411, 792) -> 23658 (frequency: 1503)
2026-08-04 10:52:34,201 - rustbpe - INFO - Progress: 73% (23728/32503 merges) - Last merge: (12140, 497) -> 23983 (frequency: 1471)
2026-08-04 10:52:34,209 - rustbpe - INFO - Progress: 74% (24053/32503 merges) - Last merge: (17894, 2094) -> 24308 (frequency: 1435)
2026-08-04 10:52:34,222 - rustbpe - INFO - Progress: 75% (24378/32503 merges) - Last merge: (15140, 290) -> 24633 (frequency: 1402)
2026-08-04 10:52:34,239 - rustbpe - INFO - Progress: 76% (24703/32503 merges) - Last merge: (8147, 121) -> 24958 (frequency: 1371)
2026-08-04 10:52:34,259 - rustbpe - INFO - Progress: 77% (25028/32503 merges) - Last merge: (2418, 915) -> 25283 (frequency: 1339)
2026-08-04 10:52:34,272 - rustbpe - INFO - Progress: 78% (25353/32503 merges) - Last merge: (15042, 3344) -> 25608 (frequency: 1309)
2026-08-04 10:52:34,284 - rustbpe - INFO - Progress: 79% (25678/32503 merges) - Last merge: (15779, 3089) -> 25933 (frequency: 1282)
2026-08-04 10:52:34,295 - rustbpe - INFO - Progress: 80% (26003/32503 merges) - Last merge: (4153, 378) -> 26258 (frequency: 1256)
2026-08-04 10:52:34,311 - rustbpe - INFO - Progress: 81% (26328/32503 merges) - Last merge: (5908, 284) -> 26583 (frequency: 1228)
2026-08-04 10:52:34,325 - rustbpe - INFO - Progress: 82% (26653/32503 merges) - Last merge: (4582, 18636) -> 26908 (frequency: 1202)
2026-08-04 10:52:34,341 - rustbpe - INFO - Progress: 83% (26978/32503 merges) - Last merge: (864, 4147) -> 27233 (frequency: 1174)
2026-08-04 10:52:34,360 - rustbpe - INFO - Progress: 84% (27303/32503 merges) - Last merge: (365, 4208) -> 27558 (frequency: 1149)
2026-08-04 10:52:34,373 - rustbpe - INFO - Progress: 85% (27628/32503 merges) - Last merge: (5117, 545) -> 27883 (frequency: 1127)
2026-08-04 10:52:34,397 - rustbpe - INFO - Progress: 86% (27953/32503 merges) - Last merge: (18048, 323) -> 28208 (frequency: 1103)
2026-08-04 10:52:34,410 - rustbpe - INFO - Progress: 87% (28278/32503 merges) - Last merge: (4388, 25244) -> 28533 (frequency: 1081)
2026-08-04 10:52:34,422 - rustbpe - INFO - Progress: 88% (28603/32503 merges) - Last merge: (10440, 563) -> 28858 (frequency: 1058)
2026-08-04 10:52:34,434 - rustbpe - INFO - Progress: 89% (28928/32503 merges) - Last merge: (39, 58) -> 29183 (frequency: 1035)
2026-08-04 10:52:34,447 - rustbpe - INFO - Progress: 90% (29253/32503 merges) - Last merge: (23466, 3269) -> 29508 (frequency: 1015)
2026-08-04 10:52:34,462 - rustbpe - INFO - Progress: 91% (29578/32503 merges) - Last merge: (1121, 22012) -> 29833 (frequency: 995)
2026-08-04 10:52:34,473 - rustbpe - INFO - Progress: 92% (29903/32503 merges) - Last merge: (6693, 364) -> 30158 (frequency: 976)
2026-08-04 10:52:34,484 - rustbpe - INFO - Progress: 93% (30228/32503 merges) - Last merge: (4959, 18871) -> 30483 (frequency: 959)
2026-08-04 10:52:34,500 - rustbpe - INFO - Progress: 94% (30553/32503 merges) - Last merge: (431, 67) -> 30808 (frequency: 939)
2026-08-04 10:52:34,515 - rustbpe - INFO - Progress: 95% (30878/32503 merges) - Last merge: (359, 6655) -> 31133 (frequency: 920)
2026-08-04 10:52:34,526 - rustbpe - INFO - Progress: 96% (31203/32503 merges) - Last merge: (5480, 27614) -> 31458 (frequency: 904)
2026-08-04 10:52:34,534 - rustbpe - INFO - Progress: 97% (31528/32503 merges) - Last merge: (1497, 825) -> 31783 (frequency: 886)
2026-08-04 10:52:34,553 - rustbpe - INFO - Progress: 98% (31853/32503 merges) - Last merge: (10729, 1319) -> 32108 (frequency: 872)
2026-08-04 10:52:34,565 - rustbpe - INFO - Progress: 99% (32178/32503 merges) - Last merge: (440, 3832) -> 32433 (frequency: 857)
2026-08-04 10:52:34,574 - rustbpe - INFO - Progress: 100% (32503/32503 merges) - Last merge: (115, 4714) -> 32758 (frequency: 842)
2026-08-04 10:52:34,574 - rustbpe - INFO - Finished training: 32503 merges completed
Training time: 42.69s
Saved tokenizer encoding to $ORX_RUN_DIR/repo/.cache/nanochat/tokenizer/tokenizer.pkl
Saved token_bytes to $ORX_RUN_DIR/repo/.cache/nanochat/tokenizer/token_bytes.pt

Vocab sizes:
GPT-2: 50257
GPT-4: 100277
Ours: 32768

Comparison with GPT-2:
===============================================================================================
Text Type  Bytes    GPT-2           Ours            Relative     Better
                    Tokens  Ratio   Tokens  Ratio   Diff %
-----------------------------------------------------------------------------------------------
news       1819     404     4.50    405     4.49       -0.2%     GPT-2
korean     893      745     1.20    741     1.21       +0.5%     Ours
code       1259     576     2.19    396     3.18      +31.2%     Ours
math       1834     936     1.96    911     2.01       +2.7%     Ours
science    1112     260     4.28    247     4.50       +5.0%     Ours
climbmix-train 2948778  631304  4.67    622511  4.74       +1.4%     Ours
climbmix-val 3024593  653067  4.63    644939  4.69       +1.2%     Ours

Comparison with GPT-4:
===============================================================================================
Text Type  Bytes    GPT-4           Ours            Relative     Better
                    Tokens  Ratio   Tokens  Ratio   Diff %
-----------------------------------------------------------------------------------------------
news       1819     387     4.70    405     4.49       -4.7%     GPT-4
korean     893      364     2.45    741     1.21     -103.6%     GPT-4
code       1259     309     4.07    396     3.18      -28.2%     GPT-4
math       1834     832     2.20    911     2.01       -9.5%     GPT-4
science    1112     249     4.47    247     4.50       +0.8%     Ours
climbmix-train 2948778  611619  4.82    622511  4.74       -1.8%     GPT-4
climbmix-val 3024593  631183  4.79    644939  4.69       -2.2%     GPT-4

                                                       █████                █████
                                                      ░░███                ░░███
     ████████    ██████   ████████    ██████   ██████  ░███████    ██████  ███████
    ░░███░░███  ░░░░░███ ░░███░░███  ███░░███ ███░░███ ░███░░███  ░░░░░███░░░███░
     ░███ ░███   ███████  ░███ ░███ ░███ ░███░███ ░░░  ░███ ░███   ███████  ░███
     ░███ ░███  ███░░███  ░███ ░███ ░███ ░███░███  ███ ░███ ░███  ███░░███  ░███ ███
     ████ █████░░████████ ████ █████░░██████ ░░██████  ████ █████░░███████  ░░█████
    ░░░░ ░░░░░  ░░░░░░░░ ░░░░ ░░░░░  ░░░░░░   ░░░░░░  ░░░░ ░░░░░  ░░░░░░░░   ░░░░░

Autodetected device type: mps
2026-08-04 10:52:42,212 - nanochat.common - INFO - Distributed world size: 1
COMPUTE_DTYPE: torch.float32 (auto-detected: no CUDA (CPU/MPS))
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
WARNING: Flash Attention 3 not available, using PyTorch SDPA fallback
WARNING: Training will be less efficient without FA3
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Vocab size: 32,768
Model config:
{
  "sequence_len": 512,
  "vocab_size": 32768,
  "n_layer": 6,
  "n_head": 6,
  "n_kv_head": 6,
  "n_embd": 384,
  "window_pattern": "L"
}
Parameter counts:
wte                     : 12,582,912
value_embeds            : 37,748,736
lm_head                 : 12,582,912
transformer_matrices    : 10,617,048
scalars                 : 38
total                   : 73,531,646
Estimated FLOPs per token: 1.533557e+08
Scaling LRs by 0.1768 for batch size 16,384 (reference: 524,288)
Scaling weight decay from 0.280000 to 0.234903 for depth 6
Scaling the LR for the AdamW parameters ∝1/√(384/768) = 1.414214
Using user-provided number of iterations: 5,000
Total number of training tokens: 81,920,000
Tokens : Scaling params ratio: 3.53
Total training FLOPs estimate: 1.256290e+16
Tokens / micro-batch / rank: 32 x 512 = 16,384
Tokens / micro-batch: 16,384
Total batch size 16,384 => gradient accumulation steps: 1
Step 00000 | Validation bpb: 3.195800
step 00000/05000 (0.00%) | loss: 10.397551 | lrm: 0.03 | dt: 2698.07ms | tok/sec: 6,072 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 1 | total time: 0.00m
step 00001/05000 (0.02%) | loss: 10.396379 | lrm: 0.05 | dt: 1609.34ms | tok/sec: 10,180 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 1 | total time: 0.00m
step 00002/05000 (0.04%) | loss: 10.394470 | lrm: 0.07 | dt: 1602.23ms | tok/sec: 10,225 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 1 | total time: 0.00m
step 00003/05000 (0.06%) | loss: 10.391348 | lrm: 0.10 | dt: 1597.28ms | tok/sec: 10,257 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 1 | total time: 0.00m
step 00004/05000 (0.08%) | loss: 10.387522 | lrm: 0.12 | dt: 1591.35ms | tok/sec: 10,295 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 1 | total time: 0.00m
step 00005/05000 (0.10%) | loss: 10.382877 | lrm: 0.15 | dt: 1597.10ms | tok/sec: 10,258 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 1 | total time: 0.00m
step 00006/05000 (0.12%) | loss: 10.376017 | lrm: 0.17 | dt: 1560.48ms | tok/sec: 10,499 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 1 | total time: 0.00m
step 00007/05000 (0.14%) | loss: 10.368099 | lrm: 0.20 | dt: 1568.80ms | tok/sec: 10,443 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 1 | total time: 0.00m
step 00008/05000 (0.16%) | loss: 10.357552 | lrm: 0.23 | dt: 1625.06ms | tok/sec: 10,082 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 1 | total time: 0.00m
step 00009/05000 (0.18%) | loss: 10.344278 | lrm: 0.25 | dt: 1519.37ms | tok/sec: 10,783 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 1 | total time: 0.00m
step 00010/05000 (0.20%) | loss: 10.327322 | lrm: 0.28 | dt: 1592.08ms | tok/sec: 10,290 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 1 | total time: 0.00m
step 00011/05000 (0.22%) | loss: 10.305425 | lrm: 0.30 | dt: 1531.07ms | tok/sec: 10,701 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 1 | total time: 0.03m | eta: 127.3m
step 00012/05000 (0.24%) | loss: 10.277484 | lrm: 0.33 | dt: 1588.30ms | tok/sec: 10,315 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 1 | total time: 0.05m | eta: 129.7m
step 00013/05000 (0.26%) | loss: 10.243526 | lrm: 0.35 | dt: 1572.05ms | tok/sec: 10,422 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 1 | total time: 0.08m | eta: 130.0m
step 00014/05000 (0.28%) | loss: 10.202690 | lrm: 0.38 | dt: 1551.39ms | tok/sec: 10,560 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 2 | total time: 0.10m | eta: 129.7m
step 00015/05000 (0.30%) | loss: 10.143537 | lrm: 0.40 | dt: 1563.22ms | tok/sec: 10,480 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 2 | total time: 0.13m | eta: 129.7m
step 00016/05000 (0.32%) | loss: 10.063040 | lrm: 0.42 | dt: 1536.44ms | tok/sec: 10,663 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 2 | total time: 0.16m | eta: 129.3m
step 00017/05000 (0.34%) | loss: 9.944315 | lrm: 0.45 | dt: 1563.74ms | tok/sec: 10,477 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 2 | total time: 0.18m | eta: 129.4m
step 00018/05000 (0.36%) | loss: 9.839947 | lrm: 0.47 | dt: 1474.55ms | tok/sec: 11,111 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 2 | total time: 0.21m | eta: 128.5m
step 00019/05000 (0.38%) | loss: 9.682587 | lrm: 0.50 | dt: 1582.62ms | tok/sec: 10,352 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 2 | total time: 0.23m | eta: 128.8m
step 00020/05000 (0.40%) | loss: 9.535879 | lrm: 0.53 | dt: 1576.52ms | tok/sec: 10,392 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 2 | total time: 0.26m | eta: 129.0m
step 00021/05000 (0.42%) | loss: 9.390616 | lrm: 0.55 | dt: 1474.92ms | tok/sec: 11,108 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 2 | total time: 0.28m | eta: 128.4m
step 00022/05000 (0.44%) | loss: 9.242283 | lrm: 0.57 | dt: 1459.00ms | tok/sec: 11,229 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 2 | total time: 0.31m | eta: 127.7m
step 00023/05000 (0.46%) | loss: 9.089623 | lrm: 0.60 | dt: 1525.67ms | tok/sec: 10,738 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 2 | total time: 0.33m | eta: 127.6m
step 00024/05000 (0.48%) | loss: 8.955663 | lrm: 0.62 | dt: 1508.54ms | tok/sec: 10,860 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 2 | total time: 0.36m | eta: 127.4m
step 00025/05000 (0.50%) | loss: 8.814061 | lrm: 0.65 | dt: 1532.57ms | tok/sec: 10,690 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 2 | total time: 0.38m | eta: 127.4m
step 00026/05000 (0.52%) | loss: 8.687078 | lrm: 0.68 | dt: 1503.02ms | tok/sec: 10,900 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 2 | total time: 0.41m | eta: 127.2m
step 00027/05000 (0.54%) | loss: 8.568739 | lrm: 0.70 | dt: 1490.92ms | tok/sec: 10,989 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 2 | total time: 0.43m | eta: 126.9m
step 00028/05000 (0.56%) | loss: 8.468792 | lrm: 0.72 | dt: 1511.64ms | tok/sec: 10,838 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 2 | total time: 0.46m | eta: 126.8m
step 00029/05000 (0.58%) | loss: 8.378134 | lrm: 0.75 | dt: 1578.58ms | tok/sec: 10,378 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 2 | total time: 0.49m | eta: 127.0m
step 00030/05000 (0.60%) | loss: 8.289777 | lrm: 0.78 | dt: 1527.59ms | tok/sec: 10,725 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 2 | total time: 0.51m | eta: 127.0m
step 00031/05000 (0.62%) | loss: 8.202727 | lrm: 0.80 | dt: 1489.15ms | tok/sec: 11,002 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 2 | total time: 0.54m | eta: 126.8m
step 00032/05000 (0.64%) | loss: 8.117556 | lrm: 0.82 | dt: 1487.06ms | tok/sec: 11,017 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 2 | total time: 0.56m | eta: 126.6m
step 00033/05000 (0.66%) | loss: 8.038115 | lrm: 0.85 | dt: 1500.50ms | tok/sec: 10,919 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 2 | total time: 0.59m | eta: 126.4m
step 00034/05000 (0.68%) | loss: 7.958634 | lrm: 0.88 | dt: 1482.08ms | tok/sec: 11,054 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 2 | total time: 0.61m | eta: 126.3m
step 00035/05000 (0.70%) | loss: 7.889547 | lrm: 0.90 | dt: 1464.55ms | tok/sec: 11,187 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 2 | total time: 0.63m | eta: 126.0m
step 00036/05000 (0.72%) | loss: 7.834000 | lrm: 0.93 | dt: 1503.00ms | tok/sec: 10,900 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 2 | total time: 0.66m | eta: 125.9m
step 00037/05000 (0.74%) | loss: 7.772990 | lrm: 0.95 | dt: 1437.34ms | tok/sec: 11,398 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 3 | total time: 0.68m | eta: 125.7m
step 00038/05000 (0.76%) | loss: 7.704178 | lrm: 0.97 | dt: 1446.40ms | tok/sec: 11,327 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 3 | total time: 0.71m | eta: 125.4m
step 00039/05000 (0.78%) | loss: 7.635180 | lrm: 1.00 | dt: 1542.93ms | tok/sec: 10,618 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 3 | total time: 0.73m | eta: 125.5m
step 00040/05000 (0.80%) | loss: 7.570863 | lrm: 1.00 | dt: 1549.13ms | tok/sec: 10,576 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 3 | total time: 0.76m | eta: 125.5m
step 00041/05000 (0.82%) | loss: 7.506861 | lrm: 1.00 | dt: 1460.93ms | tok/sec: 11,214 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 3 | total time: 0.78m | eta: 125.3m
step 00042/05000 (0.84%) | loss: 7.463189 | lrm: 1.00 | dt: 1513.86ms | tok/sec: 10,822 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 3 | total time: 0.81m | eta: 125.3m
step 00043/05000 (0.86%) | loss: 7.400624 | lrm: 1.00 | dt: 1467.64ms | tok/sec: 11,163 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 3 | total time: 0.83m | eta: 125.2m
step 00044/05000 (0.88%) | loss: 7.350554 | lrm: 1.00 | dt: 1533.49ms | tok/sec: 10,684 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 3 | total time: 0.86m | eta: 125.2m
step 00045/05000 (0.90%) | loss: 7.304329 | lrm: 1.00 | dt: 1629.91ms | tok/sec: 10,052 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 3 | total time: 0.89m | eta: 125.4m
step 00046/05000 (0.92%) | loss: 7.262331 | lrm: 1.00 | dt: 1594.46ms | tok/sec: 10,275 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 3 | total time: 0.91m | eta: 125.6m
step 00047/05000 (0.94%) | loss: 7.219421 | lrm: 1.00 | dt: 1528.56ms | tok/sec: 10,718 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 3 | total time: 0.94m | eta: 125.6m
step 00048/05000 (0.96%) | loss: 7.165313 | lrm: 1.00 | dt: 1546.76ms | tok/sec: 10,592 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 3 | total time: 0.96m | eta: 125.6m
step 00049/05000 (0.98%) | loss: 7.133292 | lrm: 1.00 | dt: 1483.56ms | tok/sec: 11,043 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 3 | total time: 0.99m | eta: 125.5m
step 00050/05000 (1.00%) | loss: 7.101772 | lrm: 1.00 | dt: 1513.25ms | tok/sec: 10,827 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 3 | total time: 1.01m | eta: 125.5m
step 00051/05000 (1.02%) | loss: 7.083357 | lrm: 1.00 | dt: 1551.40ms | tok/sec: 10,560 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 3 | total time: 1.04m | eta: 125.5m
step 00052/05000 (1.04%) | loss: 7.053215 | lrm: 1.00 | dt: 1564.27ms | tok/sec: 10,473 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 3 | total time: 1.07m | eta: 125.6m
step 00053/05000 (1.06%) | loss: 7.035129 | lrm: 1.00 | dt: 1552.76ms | tok/sec: 10,551 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 3 | total time: 1.09m | eta: 125.6m
step 00054/05000 (1.08%) | loss: 6.989665 | lrm: 1.00 | dt: 1606.93ms | tok/sec: 10,195 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 3 | total time: 1.12m | eta: 125.7m
step 00055/05000 (1.10%) | loss: 6.964000 | lrm: 1.00 | dt: 1579.63ms | tok/sec: 10,372 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 3 | total time: 1.14m | eta: 125.8m
step 00056/05000 (1.12%) | loss: 6.947931 | lrm: 1.00 | dt: 1515.35ms | tok/sec: 10,811 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 3 | total time: 1.17m | eta: 125.7m
step 00057/05000 (1.14%) | loss: 6.912264 | lrm: 1.00 | dt: 1483.74ms | tok/sec: 11,042 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 3 | total time: 1.19m | eta: 125.6m
step 00058/05000 (1.16%) | loss: 6.882016 | lrm: 1.00 | dt: 1530.31ms | tok/sec: 10,706 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 3 | total time: 1.22m | eta: 125.6m
step 00059/05000 (1.18%) | loss: 6.852830 | lrm: 1.00 | dt: 1424.15ms | tok/sec: 11,504 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 3 | total time: 1.24m | eta: 125.4m
step 00060/05000 (1.20%) | loss: 6.823568 | lrm: 1.00 | dt: 1460.58ms | tok/sec: 11,217 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 4 | total time: 1.27m | eta: 125.3m
step 00061/05000 (1.22%) | loss: 6.804969 | lrm: 1.00 | dt: 1478.89ms | tok/sec: 11,078 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 4 | total time: 1.29m | eta: 125.2m
step 00062/05000 (1.24%) | loss: 6.788221 | lrm: 1.00 | dt: 1494.71ms | tok/sec: 10,961 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 4 | total time: 1.32m | eta: 125.1m
step 00063/05000 (1.26%) | loss: 6.749067 | lrm: 1.00 | dt: 1542.81ms | tok/sec: 10,619 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 4 | total time: 1.34m | eta: 125.2m
step 00064/05000 (1.28%) | loss: 6.738475 | lrm: 1.00 | dt: 1541.78ms | tok/sec: 10,626 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 4 | total time: 1.37m | eta: 125.2m
step 00065/05000 (1.30%) | loss: 6.728843 | lrm: 1.00 | dt: 1548.95ms | tok/sec: 10,577 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 4 | total time: 1.40m | eta: 125.2m
step 00066/05000 (1.32%) | loss: 6.714016 | lrm: 1.00 | dt: 1553.58ms | tok/sec: 10,545 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 4 | total time: 1.42m | eta: 125.2m
step 00067/05000 (1.34%) | loss: 6.694574 | lrm: 1.00 | dt: 1594.08ms | tok/sec: 10,278 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 4 | total time: 1.45m | eta: 125.3m
step 00068/05000 (1.36%) | loss: 6.692098 | lrm: 1.00 | dt: 1558.28ms | tok/sec: 10,514 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 4 | total time: 1.47m | eta: 125.3m
step 00069/05000 (1.38%) | loss: 6.678002 | lrm: 1.00 | dt: 1553.71ms | tok/sec: 10,545 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 4 | total time: 1.50m | eta: 125.3m
step 00070/05000 (1.40%) | loss: 6.671523 | lrm: 1.00 | dt: 1545.26ms | tok/sec: 10,602 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 4 | total time: 1.53m | eta: 125.3m
step 00071/05000 (1.42%) | loss: 6.661179 | lrm: 1.00 | dt: 1594.75ms | tok/sec: 10,273 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 4 | total time: 1.55m | eta: 125.4m
step 00072/05000 (1.44%) | loss: 6.651430 | lrm: 1.00 | dt: 1522.17ms | tok/sec: 10,763 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 4 | total time: 1.58m | eta: 125.4m
step 00073/05000 (1.46%) | loss: 6.630572 | lrm: 1.00 | dt: 1530.33ms | tok/sec: 10,706 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 4 | total time: 1.60m | eta: 125.3m
step 00074/05000 (1.48%) | loss: 6.603403 | lrm: 1.00 | dt: 1537.36ms | tok/sec: 10,657 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 4 | total time: 1.63m | eta: 125.3m
step 00075/05000 (1.50%) | loss: 6.588104 | lrm: 1.00 | dt: 1566.28ms | tok/sec: 10,460 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 4 | total time: 1.65m | eta: 125.3m
step 00076/05000 (1.52%) | loss: 6.587701 | lrm: 1.00 | dt: 1446.38ms | tok/sec: 11,327 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 4 | total time: 1.68m | eta: 125.2m
step 00077/05000 (1.54%) | loss: 6.565280 | lrm: 1.00 | dt: 1584.55ms | tok/sec: 10,339 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 4 | total time: 1.70m | eta: 125.3m
step 00078/05000 (1.56%) | loss: 6.555304 | lrm: 1.00 | dt: 1568.06ms | tok/sec: 10,448 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 4 | total time: 1.73m | eta: 125.3m
step 00079/05000 (1.58%) | loss: 6.548659 | lrm: 1.00 | dt: 1510.87ms | tok/sec: 10,844 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 4 | total time: 1.76m | eta: 125.2m
step 00080/05000 (1.60%) | loss: 6.539865 | lrm: 1.00 | dt: 1541.27ms | tok/sec: 10,630 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 4 | total time: 1.78m | eta: 125.2m
step 00081/05000 (1.62%) | loss: 6.538257 | lrm: 1.00 | dt: 1532.55ms | tok/sec: 10,690 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 4 | total time: 1.81m | eta: 125.2m
step 00082/05000 (1.64%) | loss: 6.536972 | lrm: 1.00 | dt: 1554.45ms | tok/sec: 10,540 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 4 | total time: 1.83m | eta: 125.2m
step 00083/05000 (1.66%) | loss: 6.524735 | lrm: 1.00 | dt: 1578.58ms | tok/sec: 10,378 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 5 | total time: 1.86m | eta: 125.3m
step 00084/05000 (1.68%) | loss: 6.505210 | lrm: 1.00 | dt: 1572.11ms | tok/sec: 10,421 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 5 | total time: 1.89m | eta: 125.3m
step 00085/05000 (1.70%) | loss: 6.492740 | lrm: 1.00 | dt: 1499.38ms | tok/sec: 10,927 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 5 | total time: 1.91m | eta: 125.2m
step 00086/05000 (1.72%) | loss: 6.475678 | lrm: 1.00 | dt: 1551.07ms | tok/sec: 10,562 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 5 | total time: 1.94m | eta: 125.2m
step 00087/05000 (1.74%) | loss: 6.464394 | lrm: 1.00 | dt: 1523.51ms | tok/sec: 10,754 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 5 | total time: 1.96m | eta: 125.2m
step 00088/05000 (1.76%) | loss: 6.448489 | lrm: 1.00 | dt: 1520.33ms | tok/sec: 10,776 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 5 | total time: 1.99m | eta: 125.2m
step 00089/05000 (1.78%) | loss: 6.434203 | lrm: 1.00 | dt: 1507.51ms | tok/sec: 10,868 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 5 | total time: 2.01m | eta: 125.1m
step 00090/05000 (1.80%) | loss: 6.441341 | lrm: 1.00 | dt: 1635.10ms | tok/sec: 10,020 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 5 | total time: 2.04m | eta: 125.2m
step 00091/05000 (1.82%) | loss: 6.437485 | lrm: 1.00 | dt: 1614.72ms | tok/sec: 10,146 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 5 | total time: 2.07m | eta: 125.2m
step 00092/05000 (1.84%) | loss: 6.429016 | lrm: 1.00 | dt: 1588.62ms | tok/sec: 10,313 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 5 | total time: 2.09m | eta: 125.3m
step 00093/05000 (1.86%) | loss: 6.419055 | lrm: 1.00 | dt: 1555.36ms | tok/sec: 10,533 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 5 | total time: 2.12m | eta: 125.3m
step 00094/05000 (1.88%) | loss: 6.408738 | lrm: 1.00 | dt: 1518.88ms | tok/sec: 10,786 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 5 | total time: 2.14m | eta: 125.2m
step 00095/05000 (1.90%) | loss: 6.407279 | lrm: 1.00 | dt: 1564.57ms | tok/sec: 10,471 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 5 | total time: 2.17m | eta: 125.2m
step 00096/05000 (1.92%) | loss: 6.396828 | lrm: 1.00 | dt: 1596.43ms | tok/sec: 10,262 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 5 | total time: 2.20m | eta: 125.3m
step 00097/05000 (1.94%) | loss: 6.395266 | lrm: 1.00 | dt: 1501.39ms | tok/sec: 10,912 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 5 | total time: 2.22m | eta: 125.2m
step 00098/05000 (1.96%) | loss: 6.395747 | lrm: 1.00 | dt: 1570.99ms | tok/sec: 10,429 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 5 | total time: 2.25m | eta: 125.2m
step 00099/05000 (1.98%) | loss: 6.381023 | lrm: 1.00 | dt: 1510.42ms | tok/sec: 10,847 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 5 | total time: 2.27m | eta: 125.2m
Step 00100 | Validation bpb: 1.940739
<|bos|>The capital of France is a very popular and the most common in the most common in the most common in
<|bos|>The chemical symbol of gold is the most common of the most common of the same time, and the most common
<|bos|>If yesterday was Friday, then tomorrow will be a good for the most common in the most common in the most common in the
<|bos|>The opposite of hot is the most common of the most common of the most common of the most common of
<|bos|>The planets of the solar system are: "The researchers found in the most common in the most common in the most common
<|bos|>My favorite color is a good way to the use of the same way to the use of the same
<|bos|>If 5*x + 3 = 13, then x is 2020, 2020, 2020, 2020,
step 00100/05000 (2.00%) | loss: 6.374492 | lrm: 1.00 | dt: 1558.35ms | tok/sec: 10,513 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 5 | total time: 2.30m | eta: 125.2m
step 00101/05000 (2.02%) | loss: 6.382511 | lrm: 1.00 | dt: 1453.27ms | tok/sec: 11,273 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 5 | total time: 2.32m | eta: 125.1m
step 00102/05000 (2.04%) | loss: 6.391588 | lrm: 1.00 | dt: 1530.26ms | tok/sec: 10,706 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 5 | total time: 2.35m | eta: 125.1m
step 00103/05000 (2.06%) | loss: 6.376159 | lrm: 1.00 | dt: 1407.29ms | tok/sec: 11,642 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 5 | total time: 2.37m | eta: 124.9m
step 00104/05000 (2.08%) | loss: 6.371054 | lrm: 1.00 | dt: 1538.32ms | tok/sec: 10,650 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 5 | total time: 2.40m | eta: 124.9m
step 00105/05000 (2.10%) | loss: 6.358719 | lrm: 1.00 | dt: 1499.59ms | tok/sec: 10,925 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 5 | total time: 2.42m | eta: 124.9m
step 00106/05000 (2.12%) | loss: 6.334715 | lrm: 1.00 | dt: 1517.60ms | tok/sec: 10,795 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 5 | total time: 2.45m | eta: 124.8m
step 00107/05000 (2.14%) | loss: 6.331013 | lrm: 1.00 | dt: 1549.93ms | tok/sec: 10,570 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 6 | total time: 2.47m | eta: 124.8m
step 00108/05000 (2.16%) | loss: 6.325513 | lrm: 1.00 | dt: 1495.00ms | tok/sec: 10,959 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 6 | total time: 2.50m | eta: 124.8m
step 00109/05000 (2.18%) | loss: 6.317564 | lrm: 1.00 | dt: 1493.84ms | tok/sec: 10,967 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 6 | total time: 2.52m | eta: 124.7m
step 00110/05000 (2.20%) | loss: 6.324177 | lrm: 1.00 | dt: 1452.00ms | tok/sec: 11,283 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 6 | total time: 2.55m | eta: 124.6m
step 00111/05000 (2.22%) | loss: 6.305012 | lrm: 1.00 | dt: 1532.91ms | tok/sec: 10,688 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 6 | total time: 2.57m | eta: 124.6m
step 00112/05000 (2.24%) | loss: 6.291364 | lrm: 1.00 | dt: 1443.35ms | tok/sec: 11,351 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 6 | total time: 2.60m | eta: 124.5m
step 00113/05000 (2.26%) | loss: 6.290430 | lrm: 1.00 | dt: 1557.83ms | tok/sec: 10,517 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 6 | total time: 2.62m | eta: 124.5m
step 00114/05000 (2.28%) | loss: 6.269876 | lrm: 1.00 | dt: 1506.68ms | tok/sec: 10,874 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 6 | total time: 2.65m | eta: 124.5m
step 00115/05000 (2.30%) | loss: 6.261721 | lrm: 1.00 | dt: 1469.54ms | tok/sec: 11,149 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 6 | total time: 2.67m | eta: 124.4m
step 00116/05000 (2.32%) | loss: 6.255419 | lrm: 1.00 | dt: 1545.10ms | tok/sec: 10,603 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 6 | total time: 2.70m | eta: 124.4m
step 00117/05000 (2.34%) | loss: 6.247732 | lrm: 1.00 | dt: 1540.94ms | tok/sec: 10,632 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 6 | total time: 2.72m | eta: 124.4m
step 00118/05000 (2.36%) | loss: 6.234220 | lrm: 1.00 | dt: 1441.69ms | tok/sec: 11,364 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 6 | total time: 2.75m | eta: 124.3m
step 00119/05000 (2.38%) | loss: 6.219800 | lrm: 1.00 | dt: 1480.10ms | tok/sec: 11,069 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 6 | total time: 2.77m | eta: 124.2m
step 00120/05000 (2.40%) | loss: 6.214485 | lrm: 1.00 | dt: 1552.90ms | tok/sec: 10,550 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 6 | total time: 2.80m | eta: 124.2m
step 00121/05000 (2.42%) | loss: 6.212358 | lrm: 1.00 | dt: 1436.59ms | tok/sec: 11,404 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 6 | total time: 2.82m | eta: 124.1m
step 00122/05000 (2.44%) | loss: 6.192365 | lrm: 1.00 | dt: 1589.93ms | tok/sec: 10,304 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 6 | total time: 2.85m | eta: 124.1m
step 00123/05000 (2.46%) | loss: 6.186865 | lrm: 1.00 | dt: 1550.94ms | tok/sec: 10,563 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 6 | total time: 2.88m | eta: 124.1m
step 00124/05000 (2.48%) | loss: 6.189053 | lrm: 1.00 | dt: 1507.63ms | tok/sec: 10,867 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 6 | total time: 2.90m | eta: 124.1m
step 00125/05000 (2.50%) | loss: 6.180514 | lrm: 1.00 | dt: 1567.08ms | tok/sec: 10,455 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 6 | total time: 2.93m | eta: 124.1m
step 00126/05000 (2.52%) | loss: 6.186722 | lrm: 1.00 | dt: 1516.77ms | tok/sec: 10,801 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 6 | total time: 2.95m | eta: 124.0m
step 00127/05000 (2.54%) | loss: 6.183503 | lrm: 1.00 | dt: 1519.50ms | tok/sec: 10,782 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 6 | total time: 2.98m | eta: 124.0m
step 00128/05000 (2.56%) | loss: 6.181052 | lrm: 1.00 | dt: 1508.18ms | tok/sec: 10,863 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 6 | total time: 3.00m | eta: 124.0m
step 00129/05000 (2.58%) | loss: 6.187194 | lrm: 1.00 | dt: 1476.24ms | tok/sec: 11,098 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 6 | total time: 3.03m | eta: 123.9m
step 00130/05000 (2.60%) | loss: 6.186514 | lrm: 1.00 | dt: 1571.34ms | tok/sec: 10,426 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 7 | total time: 3.05m | eta: 123.9m
step 00131/05000 (2.62%) | loss: 6.198681 | lrm: 1.00 | dt: 1527.96ms | tok/sec: 10,722 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 7 | total time: 3.08m | eta: 123.9m
step 00132/05000 (2.64%) | loss: 6.200714 | lrm: 1.00 | dt: 1555.03ms | tok/sec: 10,536 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 7 | total time: 3.10m | eta: 123.9m
step 00133/05000 (2.66%) | loss: 6.201287 | lrm: 1.00 | dt: 1495.90ms | tok/sec: 10,952 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 7 | total time: 3.13m | eta: 123.8m
step 00134/05000 (2.68%) | loss: 6.181327 | lrm: 1.00 | dt: 1477.64ms | tok/sec: 11,087 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 7 | total time: 3.15m | eta: 123.8m
step 00135/05000 (2.70%) | loss: 6.167849 | lrm: 1.00 | dt: 1564.39ms | tok/sec: 10,473 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 7 | total time: 3.18m | eta: 123.8m
step 00136/05000 (2.72%) | loss: 6.166543 | lrm: 1.00 | dt: 1522.92ms | tok/sec: 10,758 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 7 | total time: 3.21m | eta: 123.8m
step 00137/05000 (2.74%) | loss: 6.171512 | lrm: 1.00 | dt: 1561.02ms | tok/sec: 10,495 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 7 | total time: 3.23m | eta: 123.8m
step 00138/05000 (2.76%) | loss: 6.166059 | lrm: 1.00 | dt: 1555.95ms | tok/sec: 10,529 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 7 | total time: 3.26m | eta: 123.8m
step 00139/05000 (2.78%) | loss: 6.152405 | lrm: 1.00 | dt: 1501.40ms | tok/sec: 10,912 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 7 | total time: 3.28m | eta: 123.7m
step 00140/05000 (2.80%) | loss: 6.150906 | lrm: 1.00 | dt: 1574.23ms | tok/sec: 10,407 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 7 | total time: 3.31m | eta: 123.7m
step 00141/05000 (2.82%) | loss: 6.130319 | lrm: 1.00 | dt: 1472.27ms | tok/sec: 11,128 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 7 | total time: 3.33m | eta: 123.7m
step 00142/05000 (2.84%) | loss: 6.115728 | lrm: 1.00 | dt: 1518.14ms | tok/sec: 10,792 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 7 | total time: 3.36m | eta: 123.6m
step 00143/05000 (2.86%) | loss: 6.112214 | lrm: 1.00 | dt: 1480.68ms | tok/sec: 11,065 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 7 | total time: 3.38m | eta: 123.6m
step 00144/05000 (2.88%) | loss: 6.095303 | lrm: 1.00 | dt: 1566.88ms | tok/sec: 10,456 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 7 | total time: 3.41m | eta: 123.6m
step 00145/05000 (2.90%) | loss: 6.102110 | lrm: 1.00 | dt: 1532.88ms | tok/sec: 10,688 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 7 | total time: 3.44m | eta: 123.5m
step 00146/05000 (2.92%) | loss: 6.098682 | lrm: 1.00 | dt: 1535.27ms | tok/sec: 10,671 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 7 | total time: 3.46m | eta: 123.5m
step 00147/05000 (2.94%) | loss: 6.094050 | lrm: 1.00 | dt: 1576.77ms | tok/sec: 10,390 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 7 | total time: 3.49m | eta: 123.5m
step 00148/05000 (2.96%) | loss: 6.084392 | lrm: 1.00 | dt: 1503.61ms | tok/sec: 10,896 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 7 | total time: 3.51m | eta: 123.5m
step 00149/05000 (2.98%) | loss: 6.086993 | lrm: 1.00 | dt: 1509.32ms | tok/sec: 10,855 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 7 | total time: 3.54m | eta: 123.5m
step 00150/05000 (3.00%) | loss: 6.083181 | lrm: 1.00 | dt: 1576.42ms | tok/sec: 10,393 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 7 | total time: 3.56m | eta: 123.5m
step 00151/05000 (3.02%) | loss: 6.076826 | lrm: 1.00 | dt: 1526.59ms | tok/sec: 10,732 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 7 | total time: 3.59m | eta: 123.4m
step 00152/05000 (3.04%) | loss: 6.059331 | lrm: 1.00 | dt: 1544.01ms | tok/sec: 10,611 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 7 | total time: 3.61m | eta: 123.4m
step 00153/05000 (3.06%) | loss: 6.053442 | lrm: 1.00 | dt: 1570.98ms | tok/sec: 10,429 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 7 | total time: 3.64m | eta: 123.4m
step 00154/05000 (3.08%) | loss: 6.033732 | lrm: 1.00 | dt: 1586.05ms | tok/sec: 10,330 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 8 | total time: 3.67m | eta: 123.4m
step 00155/05000 (3.10%) | loss: 6.028668 | lrm: 1.00 | dt: 1551.61ms | tok/sec: 10,559 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 8 | total time: 3.69m | eta: 123.4m
step 00156/05000 (3.12%) | loss: 6.010239 | lrm: 1.00 | dt: 1497.65ms | tok/sec: 10,939 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 8 | total time: 3.72m | eta: 123.4m
step 00157/05000 (3.14%) | loss: 6.014835 | lrm: 1.00 | dt: 1493.38ms | tok/sec: 10,971 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 8 | total time: 3.74m | eta: 123.3m
step 00158/05000 (3.16%) | loss: 6.011605 | lrm: 1.00 | dt: 1502.89ms | tok/sec: 10,901 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 8 | total time: 3.77m | eta: 123.3m
step 00159/05000 (3.18%) | loss: 5.999794 | lrm: 1.00 | dt: 1557.58ms | tok/sec: 10,518 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 8 | total time: 3.79m | eta: 123.3m
step 00160/05000 (3.20%) | loss: 5.989582 | lrm: 1.00 | dt: 1541.44ms | tok/sec: 10,629 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 8 | total time: 3.82m | eta: 123.3m
step 00161/05000 (3.22%) | loss: 5.990903 | lrm: 1.00 | dt: 1548.28ms | tok/sec: 10,582 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 8 | total time: 3.85m | eta: 123.2m
step 00162/05000 (3.24%) | loss: 5.987783 | lrm: 1.00 | dt: 1533.41ms | tok/sec: 10,684 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 8 | total time: 3.87m | eta: 123.2m
step 00163/05000 (3.26%) | loss: 6.002036 | lrm: 1.00 | dt: 1569.84ms | tok/sec: 10,436 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 8 | total time: 3.90m | eta: 123.2m
step 00164/05000 (3.28%) | loss: 5.981726 | lrm: 1.00 | dt: 1484.86ms | tok/sec: 11,034 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 8 | total time: 3.92m | eta: 123.2m
step 00165/05000 (3.30%) | loss: 5.968579 | lrm: 1.00 | dt: 1524.99ms | tok/sec: 10,743 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 8 | total time: 3.95m | eta: 123.1m
step 00166/05000 (3.32%) | loss: 5.967583 | lrm: 1.00 | dt: 1514.37ms | tok/sec: 10,819 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 8 | total time: 3.97m | eta: 123.1m
step 00167/05000 (3.34%) | loss: 5.951714 | lrm: 1.00 | dt: 1453.03ms | tok/sec: 11,275 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 8 | total time: 4.00m | eta: 123.0m
step 00168/05000 (3.36%) | loss: 5.945783 | lrm: 1.00 | dt: 1464.68ms | tok/sec: 11,186 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 8 | total time: 4.02m | eta: 123.0m
step 00169/05000 (3.38%) | loss: 5.940500 | lrm: 1.00 | dt: 1534.34ms | tok/sec: 10,678 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 8 | total time: 4.05m | eta: 123.0m
step 00170/05000 (3.40%) | loss: 5.950678 | lrm: 1.00 | dt: 1544.14ms | tok/sec: 10,610 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 8 | total time: 4.07m | eta: 122.9m
step 00171/05000 (3.42%) | loss: 5.947413 | lrm: 1.00 | dt: 1574.40ms | tok/sec: 10,406 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 8 | total time: 4.10m | eta: 122.9m
step 00172/05000 (3.44%) | loss: 5.954139 | lrm: 1.00 | dt: 1541.96ms | tok/sec: 10,625 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 8 | total time: 4.12m | eta: 122.9m
step 00173/05000 (3.46%) | loss: 5.943631 | lrm: 1.00 | dt: 1538.90ms | tok/sec: 10,646 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 8 | total time: 4.15m | eta: 122.9m
step 00174/05000 (3.48%) | loss: 5.936837 | lrm: 1.00 | dt: 1543.25ms | tok/sec: 10,616 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 8 | total time: 4.18m | eta: 122.9m
step 00175/05000 (3.50%) | loss: 5.929783 | lrm: 1.00 | dt: 1569.99ms | tok/sec: 10,435 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 8 | total time: 4.20m | eta: 122.9m
step 00176/05000 (3.52%) | loss: 5.920176 | lrm: 1.00 | dt: 1562.42ms | tok/sec: 10,486 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 8 | total time: 4.23m | eta: 122.9m
step 00177/05000 (3.54%) | loss: 5.919326 | lrm: 1.00 | dt: 1539.73ms | tok/sec: 10,640 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 9 | total time: 4.25m | eta: 122.9m
step 00178/05000 (3.56%) | loss: 5.912743 | lrm: 1.00 | dt: 1530.51ms | tok/sec: 10,704 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 9 | total time: 4.28m | eta: 122.8m
step 00179/05000 (3.58%) | loss: 5.905216 | lrm: 1.00 | dt: 1588.91ms | tok/sec: 10,311 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 9 | total time: 4.31m | eta: 122.8m
step 00180/05000 (3.60%) | loss: 5.913456 | lrm: 1.00 | dt: 1573.08ms | tok/sec: 10,415 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 9 | total time: 4.33m | eta: 122.8m
step 00181/05000 (3.62%) | loss: 5.913663 | lrm: 1.00 | dt: 1557.52ms | tok/sec: 10,519 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 9 | total time: 4.36m | eta: 122.8m
step 00182/05000 (3.64%) | loss: 5.894422 | lrm: 1.00 | dt: 1583.78ms | tok/sec: 10,344 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 9 | total time: 4.38m | eta: 122.8m
step 00183/05000 (3.66%) | loss: 5.895891 | lrm: 1.00 | dt: 1540.03ms | tok/sec: 10,638 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 9 | total time: 4.41m | eta: 122.8m
step 00184/05000 (3.68%) | loss: 5.885034 | lrm: 1.00 | dt: 1470.83ms | tok/sec: 11,139 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 9 | total time: 4.43m | eta: 122.7m
step 00185/05000 (3.70%) | loss: 5.876505 | lrm: 1.00 | dt: 1579.26ms | tok/sec: 10,374 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 9 | total time: 4.46m | eta: 122.7m
step 00186/05000 (3.72%) | loss: 5.869328 | lrm: 1.00 | dt: 1480.76ms | tok/sec: 11,064 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 9 | total time: 4.49m | eta: 122.7m
step 00187/05000 (3.74%) | loss: 5.852204 | lrm: 1.00 | dt: 1498.72ms | tok/sec: 10,931 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 9 | total time: 4.51m | eta: 122.7m
step 00188/05000 (3.76%) | loss: 5.841432 | lrm: 1.00 | dt: 1529.28ms | tok/sec: 10,713 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 9 | total time: 4.54m | eta: 122.6m
step 00189/05000 (3.78%) | loss: 5.844644 | lrm: 1.00 | dt: 1522.35ms | tok/sec: 10,762 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 9 | total time: 4.56m | eta: 122.6m
step 00190/05000 (3.80%) | loss: 5.823247 | lrm: 1.00 | dt: 1540.29ms | tok/sec: 10,636 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 9 | total time: 4.59m | eta: 122.6m
step 00191/05000 (3.82%) | loss: 5.821768 | lrm: 1.00 | dt: 1473.97ms | tok/sec: 11,115 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 9 | total time: 4.61m | eta: 122.5m
step 00192/05000 (3.84%) | loss: 5.815419 | lrm: 1.00 | dt: 1461.68ms | tok/sec: 11,209 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 9 | total time: 4.64m | eta: 122.5m
step 00193/05000 (3.86%) | loss: 5.796020 | lrm: 1.00 | dt: 1736.73ms | tok/sec: 9,433 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 9 | total time: 4.67m | eta: 122.5m
step 00194/05000 (3.88%) | loss: 5.789423 | lrm: 1.00 | dt: 1536.20ms | tok/sec: 10,665 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 9 | total time: 4.69m | eta: 122.5m
step 00195/05000 (3.90%) | loss: 5.794910 | lrm: 1.00 | dt: 1559.32ms | tok/sec: 10,507 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 9 | total time: 4.72m | eta: 122.5m
step 00196/05000 (3.92%) | loss: 5.786773 | lrm: 1.00 | dt: 1644.83ms | tok/sec: 9,960 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 9 | total time: 4.74m | eta: 122.5m
step 00197/05000 (3.94%) | loss: 5.782012 | lrm: 1.00 | dt: 1623.52ms | tok/sec: 10,091 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 9 | total time: 4.77m | eta: 122.5m
step 00198/05000 (3.96%) | loss: 5.770803 | lrm: 1.00 | dt: 1502.08ms | tok/sec: 10,907 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 9 | total time: 4.80m | eta: 122.5m
step 00199/05000 (3.98%) | loss: 5.763850 | lrm: 1.00 | dt: 1579.31ms | tok/sec: 10,374 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 10 | total time: 4.82m | eta: 122.5m
Step 00200 | Validation bpb: 1.762539
<|bos|>The capital of France is a good news for the world's best for the world's best for the world
<|bos|>The chemical symbol of gold is a common type of metal, and is a common type of metal. The process
<|bos|>If yesterday was Friday, then tomorrow will be a good news for a new study, and the study of the study of the
<|bos|>The opposite of hot is a good way to make sure you are looking for a good news for your family
<|bos|>The planets of the solar system are: the system of the system, the system, and the system, and the system
<|bos|>My favorite color is a good way to make sure you are looking for a good choice for you.
<|bos|>If 5*x + 3 = 13, then x is 1.5. The 1.2 is a 1.2
step 00200/05000 (4.00%) | loss: 5.764401 | lrm: 1.00 | dt: 1669.74ms | tok/sec: 9,812 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 10 | total time: 4.85m | eta: 122.5m
step 00201/05000 (4.02%) | loss: 5.761644 | lrm: 1.00 | dt: 1622.33ms | tok/sec: 10,099 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 10 | total time: 4.88m | eta: 122.5m
step 00202/05000 (4.04%) | loss: 5.756395 | lrm: 1.00 | dt: 1660.26ms | tok/sec: 9,868 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 10 | total time: 4.91m | eta: 122.6m
step 00203/05000 (4.06%) | loss: 5.755178 | lrm: 1.00 | dt: 1736.59ms | tok/sec: 9,434 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 10 | total time: 4.93m | eta: 122.6m
step 00204/05000 (4.08%) | loss: 5.751133 | lrm: 1.00 | dt: 1579.98ms | tok/sec: 10,369 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 10 | total time: 4.96m | eta: 122.6m
step 00205/05000 (4.10%) | loss: 5.733379 | lrm: 1.00 | dt: 1636.63ms | tok/sec: 10,010 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 10 | total time: 4.99m | eta: 122.6m
step 00206/05000 (4.12%) | loss: 5.733035 | lrm: 1.00 | dt: 1665.98ms | tok/sec: 9,834 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 10 | total time: 5.02m | eta: 122.7m
step 00207/05000 (4.14%) | loss: 5.721256 | lrm: 1.00 | dt: 1594.04ms | tok/sec: 10,278 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 10 | total time: 5.04m | eta: 122.7m
step 00208/05000 (4.16%) | loss: 5.718374 | lrm: 1.00 | dt: 1673.06ms | tok/sec: 9,792 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 10 | total time: 5.07m | eta: 122.7m
step 00209/05000 (4.18%) | loss: 5.715814 | lrm: 1.00 | dt: 1596.52ms | tok/sec: 10,262 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 10 | total time: 5.10m | eta: 122.7m
step 00210/05000 (4.20%) | loss: 5.712728 | lrm: 1.00 | dt: 1591.38ms | tok/sec: 10,295 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 10 | total time: 5.12m | eta: 122.7m
step 00211/05000 (4.22%) | loss: 5.711733 | lrm: 1.00 | dt: 1653.83ms | tok/sec: 9,906 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 10 | total time: 5.15m | eta: 122.7m
step 00212/05000 (4.24%) | loss: 5.697161 | lrm: 1.00 | dt: 1589.84ms | tok/sec: 10,305 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 10 | total time: 5.18m | eta: 122.7m
step 00213/05000 (4.26%) | loss: 5.684103 | lrm: 1.00 | dt: 1672.97ms | tok/sec: 9,793 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 10 | total time: 5.20m | eta: 122.7m
step 00214/05000 (4.28%) | loss: 5.692168 | lrm: 1.00 | dt: 1636.14ms | tok/sec: 10,013 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 10 | total time: 5.23m | eta: 122.8m
step 00215/05000 (4.30%) | loss: 5.674148 | lrm: 1.00 | dt: 1551.12ms | tok/sec: 10,562 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 10 | total time: 5.26m | eta: 122.7m
step 00216/05000 (4.32%) | loss: 5.658702 | lrm: 1.00 | dt: 1626.50ms | tok/sec: 10,073 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 10 | total time: 5.29m | eta: 122.7m
step 00217/05000 (4.34%) | loss: 5.661307 | lrm: 1.00 | dt: 1570.99ms | tok/sec: 10,429 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 10 | total time: 5.31m | eta: 122.7m
step 00218/05000 (4.36%) | loss: 5.659220 | lrm: 1.00 | dt: 1642.84ms | tok/sec: 9,972 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 10 | total time: 5.34m | eta: 122.7m
step 00219/05000 (4.38%) | loss: 5.661297 | lrm: 1.00 | dt: 1617.56ms | tok/sec: 10,128 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 10 | total time: 5.37m | eta: 122.7m
step 00220/05000 (4.40%) | loss: 5.663289 | lrm: 1.00 | dt: 1612.94ms | tok/sec: 10,157 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 10 | total time: 5.39m | eta: 122.7m
step 00221/05000 (4.42%) | loss: 5.660094 | lrm: 1.00 | dt: 1604.40ms | tok/sec: 10,211 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 10 | total time: 5.42m | eta: 122.7m
step 00222/05000 (4.44%) | loss: 5.657360 | lrm: 1.00 | dt: 1595.76ms | tok/sec: 10,267 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 10 | total time: 5.45m | eta: 122.7m
step 00223/05000 (4.46%) | loss: 5.655507 | lrm: 1.00 | dt: 1625.15ms | tok/sec: 10,081 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 11 | total time: 5.47m | eta: 122.7m
step 00224/05000 (4.48%) | loss: 5.661879 | lrm: 1.00 | dt: 1629.03ms | tok/sec: 10,057 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 11 | total time: 5.50m | eta: 122.8m
step 00225/05000 (4.50%) | loss: 5.659769 | lrm: 1.00 | dt: 1809.92ms | tok/sec: 9,052 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 11 | total time: 5.53m | eta: 122.8m
step 00226/05000 (4.52%) | loss: 5.658267 | lrm: 1.00 | dt: 1672.71ms | tok/sec: 9,794 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 11 | total time: 5.56m | eta: 122.8m
step 00227/05000 (4.54%) | loss: 5.655967 | lrm: 1.00 | dt: 1655.61ms | tok/sec: 9,896 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 11 | total time: 5.59m | eta: 122.9m
step 00228/05000 (4.56%) | loss: 5.640964 | lrm: 1.00 | dt: 1628.18ms | tok/sec: 10,062 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 11 | total time: 5.61m | eta: 122.9m
step 00229/05000 (4.58%) | loss: 5.647281 | lrm: 1.00 | dt: 1716.81ms | tok/sec: 9,543 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 11 | total time: 5.64m | eta: 122.9m
step 00230/05000 (4.60%) | loss: 5.651452 | lrm: 1.00 | dt: 1715.90ms | tok/sec: 9,548 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 11 | total time: 5.67m | eta: 122.9m
step 00231/05000 (4.62%) | loss: 5.638759 | lrm: 1.00 | dt: 1682.46ms | tok/sec: 9,738 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 11 | total time: 5.70m | eta: 123.0m
step 00232/05000 (4.64%) | loss: 5.651184 | lrm: 1.00 | dt: 1672.03ms | tok/sec: 9,798 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 11 | total time: 5.73m | eta: 123.0m
step 00233/05000 (4.66%) | loss: 5.632380 | lrm: 1.00 | dt: 1675.63ms | tok/sec: 9,777 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 11 | total time: 5.75m | eta: 123.0m
step 00234/05000 (4.68%) | loss: 5.628256 | lrm: 1.00 | dt: 1683.82ms | tok/sec: 9,730 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 11 | total time: 5.78m | eta: 123.0m
step 00235/05000 (4.70%) | loss: 5.629410 | lrm: 1.00 | dt: 1699.60ms | tok/sec: 9,639 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 11 | total time: 5.81m | eta: 123.1m
step 00236/05000 (4.72%) | loss: 5.627994 | lrm: 1.00 | dt: 1765.24ms | tok/sec: 9,281 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 11 | total time: 5.84m | eta: 123.1m
step 00237/05000 (4.74%) | loss: 5.616953 | lrm: 1.00 | dt: 1602.52ms | tok/sec: 10,223 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 11 | total time: 5.87m | eta: 123.1m
step 00238/05000 (4.76%) | loss: 5.593605 | lrm: 1.00 | dt: 1576.83ms | tok/sec: 10,390 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 11 | total time: 5.89m | eta: 123.1m
step 00239/05000 (4.78%) | loss: 5.589511 | lrm: 1.00 | dt: 1568.44ms | tok/sec: 10,446 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 11 | total time: 5.92m | eta: 123.1m
step 00240/05000 (4.80%) | loss: 5.584205 | lrm: 1.00 | dt: 1646.35ms | tok/sec: 9,951 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 11 | total time: 5.95m | eta: 123.1m
step 00241/05000 (4.82%) | loss: 5.565292 | lrm: 1.00 | dt: 1565.36ms | tok/sec: 10,466 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 11 | total time: 5.97m | eta: 123.0m
step 00242/05000 (4.84%) | loss: 5.565374 | lrm: 1.00 | dt: 1656.23ms | tok/sec: 9,892 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 11 | total time: 6.00m | eta: 123.1m
step 00243/05000 (4.86%) | loss: 5.562125 | lrm: 1.00 | dt: 1587.08ms | tok/sec: 10,323 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 11 | total time: 6.03m | eta: 123.0m
step 00244/05000 (4.88%) | loss: 5.565357 | lrm: 1.00 | dt: 1566.76ms | tok/sec: 10,457 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 11 | total time: 6.05m | eta: 123.0m
step 00245/05000 (4.90%) | loss: 5.559179 | lrm: 1.00 | dt: 1777.50ms | tok/sec: 9,217 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 11 | total time: 6.08m | eta: 123.1m
step 00246/05000 (4.92%) | loss: 5.555552 | lrm: 1.00 | dt: 1815.29ms | tok/sec: 9,025 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 12 | total time: 6.11m | eta: 123.1m
step 00247/05000 (4.94%) | loss: 5.562621 | lrm: 1.00 | dt: 1658.67ms | tok/sec: 9,877 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 12 | total time: 6.14m | eta: 123.1m
step 00248/05000 (4.96%) | loss: 5.566139 | lrm: 1.00 | dt: 1670.96ms | tok/sec: 9,805 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 12 | total time: 6.17m | eta: 123.2m
step 00249/05000 (4.98%) | loss: 5.575532 | lrm: 1.00 | dt: 1697.06ms | tok/sec: 9,654 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 12 | total time: 6.20m | eta: 123.2m
step 00250/05000 (5.00%) | loss: 5.576837 | lrm: 1.00 | dt: 1667.52ms | tok/sec: 9,825 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 12 | total time: 6.22m | eta: 123.2m
step 00251/05000 (5.02%) | loss: 5.565696 | lrm: 1.00 | dt: 1796.70ms | tok/sec: 9,118 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 12 | total time: 6.25m | eta: 123.2m
step 00252/05000 (5.04%) | loss: 5.566778 | lrm: 1.00 | dt: 1698.21ms | tok/sec: 9,647 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 12 | total time: 6.28m | eta: 123.3m
step 00253/05000 (5.06%) | loss: 5.556329 | lrm: 1.00 | dt: 1670.21ms | tok/sec: 9,809 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 12 | total time: 6.31m | eta: 123.3m
step 00254/05000 (5.08%) | loss: 5.543026 | lrm: 1.00 | dt: 1675.66ms | tok/sec: 9,777 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 12 | total time: 6.34m | eta: 123.3m
step 00255/05000 (5.10%) | loss: 5.541201 | lrm: 1.00 | dt: 1722.98ms | tok/sec: 9,509 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 12 | total time: 6.37m | eta: 123.3m
step 00256/05000 (5.12%) | loss: 5.538798 | lrm: 1.00 | dt: 1700.94ms | tok/sec: 9,632 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 12 | total time: 6.40m | eta: 123.3m
step 00257/05000 (5.14%) | loss: 5.533085 | lrm: 1.00 | dt: 1687.71ms | tok/sec: 9,707 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 12 | total time: 6.42m | eta: 123.3m
step 00258/05000 (5.16%) | loss: 5.537031 | lrm: 1.00 | dt: 1697.94ms | tok/sec: 9,649 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 12 | total time: 6.45m | eta: 123.4m
step 00259/05000 (5.18%) | loss: 5.536816 | lrm: 1.00 | dt: 1793.14ms | tok/sec: 9,137 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 12 | total time: 6.48m | eta: 123.4m
step 00260/05000 (5.20%) | loss: 5.518691 | lrm: 1.00 | dt: 1703.47ms | tok/sec: 9,618 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 12 | total time: 6.51m | eta: 123.4m
step 00261/05000 (5.22%) | loss: 5.525145 | lrm: 1.00 | dt: 1639.69ms | tok/sec: 9,992 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 12 | total time: 6.54m | eta: 123.4m
step 00262/05000 (5.24%) | loss: 5.520065 | lrm: 1.00 | dt: 1634.60ms | tok/sec: 10,023 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 12 | total time: 6.56m | eta: 123.4m
step 00263/05000 (5.26%) | loss: 5.501037 | lrm: 1.00 | dt: 1679.67ms | tok/sec: 9,754 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 12 | total time: 6.59m | eta: 123.4m
step 00264/05000 (5.28%) | loss: 5.490790 | lrm: 1.00 | dt: 1616.60ms | tok/sec: 10,134 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 12 | total time: 6.62m | eta: 123.4m
step 00265/05000 (5.30%) | loss: 5.487932 | lrm: 1.00 | dt: 1674.61ms | tok/sec: 9,783 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 12 | total time: 6.65m | eta: 123.4m
step 00266/05000 (5.32%) | loss: 5.487969 | lrm: 1.00 | dt: 1616.17ms | tok/sec: 10,137 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 12 | total time: 6.67m | eta: 123.4m
step 00267/05000 (5.34%) | loss: 5.496381 | lrm: 1.00 | dt: 1606.12ms | tok/sec: 10,200 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 12 | total time: 6.70m | eta: 123.4m
step 00268/05000 (5.36%) | loss: 5.498656 | lrm: 1.00 | dt: 1592.85ms | tok/sec: 10,285 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 12 | total time: 6.73m | eta: 123.4m
step 00269/05000 (5.38%) | loss: 5.495988 | lrm: 1.00 | dt: 1650.00ms | tok/sec: 9,929 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 13 | total time: 6.76m | eta: 123.4m
step 00270/05000 (5.40%) | loss: 5.494039 | lrm: 1.00 | dt: 1634.59ms | tok/sec: 10,023 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 13 | total time: 6.78m | eta: 123.4m
step 00271/05000 (5.42%) | loss: 5.507184 | lrm: 1.00 | dt: 1639.80ms | tok/sec: 9,991 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 13 | total time: 6.81m | eta: 123.4m
step 00272/05000 (5.44%) | loss: 5.497855 | lrm: 1.00 | dt: 1644.38ms | tok/sec: 9,963 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 13 | total time: 6.84m | eta: 123.4m
step 00273/05000 (5.46%) | loss: 5.490445 | lrm: 1.00 | dt: 1667.99ms | tok/sec: 9,822 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 13 | total time: 6.86m | eta: 123.4m
step 00274/05000 (5.48%) | loss: 5.482510 | lrm: 1.00 | dt: 1617.09ms | tok/sec: 10,131 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 13 | total time: 6.89m | eta: 123.4m
step 00275/05000 (5.50%) | loss: 5.476741 | lrm: 1.00 | dt: 1539.65ms | tok/sec: 10,641 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 13 | total time: 6.92m | eta: 123.3m
step 00276/05000 (5.52%) | loss: 5.474291 | lrm: 1.00 | dt: 1674.63ms | tok/sec: 9,783 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 13 | total time: 6.95m | eta: 123.3m
step 00277/05000 (5.54%) | loss: 5.464191 | lrm: 1.00 | dt: 1681.61ms | tok/sec: 9,743 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 13 | total time: 6.97m | eta: 123.4m
step 00278/05000 (5.56%) | loss: 5.466116 | lrm: 1.00 | dt: 1584.14ms | tok/sec: 10,342 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 13 | total time: 7.00m | eta: 123.3m
step 00279/05000 (5.58%) | loss: 5.462848 | lrm: 1.00 | dt: 1644.70ms | tok/sec: 9,961 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 13 | total time: 7.03m | eta: 123.3m
step 00280/05000 (5.60%) | loss: 5.449016 | lrm: 1.00 | dt: 1596.29ms | tok/sec: 10,263 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 13 | total time: 7.05m | eta: 123.3m
step 00281/05000 (5.62%) | loss: 5.445362 | lrm: 1.00 | dt: 1657.38ms | tok/sec: 9,885 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 13 | total time: 7.08m | eta: 123.3m
step 00282/05000 (5.64%) | loss: 5.441971 | lrm: 1.00 | dt: 1647.90ms | tok/sec: 9,942 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 13 | total time: 7.11m | eta: 123.3m
step 00283/05000 (5.66%) | loss: 5.427221 | lrm: 1.00 | dt: 1604.99ms | tok/sec: 10,208 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 13 | total time: 7.14m | eta: 123.3m
step 00284/05000 (5.68%) | loss: 5.438101 | lrm: 1.00 | dt: 1623.14ms | tok/sec: 10,094 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 13 | total time: 7.16m | eta: 123.3m
step 00285/05000 (5.70%) | loss: 5.429003 | lrm: 1.00 | dt: 1822.54ms | tok/sec: 8,989 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 13 | total time: 7.19m | eta: 123.3m
step 00286/05000 (5.72%) | loss: 5.440138 | lrm: 1.00 | dt: 1716.47ms | tok/sec: 9,545 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 13 | total time: 7.22m | eta: 123.3m
step 00287/05000 (5.74%) | loss: 5.438318 | lrm: 1.00 | dt: 1707.49ms | tok/sec: 9,595 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 13 | total time: 7.25m | eta: 123.4m
step 00288/05000 (5.76%) | loss: 5.443456 | lrm: 1.00 | dt: 1660.00ms | tok/sec: 9,869 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 13 | total time: 7.28m | eta: 123.4m
step 00289/05000 (5.78%) | loss: 5.426256 | lrm: 1.00 | dt: 1691.04ms | tok/sec: 9,688 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 13 | total time: 7.31m | eta: 123.4m
step 00290/05000 (5.80%) | loss: 5.429140 | lrm: 1.00 | dt: 1726.77ms | tok/sec: 9,488 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 13 | total time: 7.33m | eta: 123.4m
step 00291/05000 (5.82%) | loss: 5.420780 | lrm: 1.00 | dt: 1588.75ms | tok/sec: 10,312 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 13 | total time: 7.36m | eta: 123.4m
step 00292/05000 (5.84%) | loss: 5.412205 | lrm: 1.00 | dt: 1683.00ms | tok/sec: 9,734 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 14 | total time: 7.39m | eta: 123.4m
step 00293/05000 (5.86%) | loss: 5.411893 | lrm: 1.00 | dt: 1662.39ms | tok/sec: 9,855 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 14 | total time: 7.42m | eta: 123.4m
step 00294/05000 (5.88%) | loss: 5.416116 | lrm: 1.00 | dt: 1686.15ms | tok/sec: 9,716 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 14 | total time: 7.45m | eta: 123.4m
step 00295/05000 (5.90%) | loss: 5.414695 | lrm: 1.00 | dt: 1662.51ms | tok/sec: 9,854 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 14 | total time: 7.47m | eta: 123.4m
step 00296/05000 (5.92%) | loss: 5.413544 | lrm: 1.00 | dt: 1644.18ms | tok/sec: 9,964 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 14 | total time: 7.50m | eta: 123.4m
step 00297/05000 (5.94%) | loss: 5.407690 | lrm: 1.00 | dt: 1682.12ms | tok/sec: 9,740 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 14 | total time: 7.53m | eta: 123.4m
step 00298/05000 (5.96%) | loss: 5.420718 | lrm: 1.00 | dt: 1687.41ms | tok/sec: 9,709 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 14 | total time: 7.56m | eta: 123.4m
step 00299/05000 (5.98%) | loss: 5.414350 | lrm: 1.00 | dt: 1668.83ms | tok/sec: 9,817 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 14 | total time: 7.58m | eta: 123.4m
Step 00300 | Validation bpb: 1.656100
<|bos|>The capital of France is a popular choice for the market. The market is not only a good choice for
<|bos|>The chemical symbol of gold is a popular choice for the product of the product. The product is a product of
<|bos|>If yesterday was Friday, then tomorrow will be able to be able to be able to be able to be able to be able
<|bos|>The opposite of hot is a good idea of a new study. The study is that the study of the
<|bos|>The planets of the solar system are: the system of the system is the system of the system. The system is the
<|bos|>My favorite color is a great way to make a new color. It is a good color. It
<|bos|>If 5*x + 3 = 13, then x is 1.2. The 2 is 2.2.2.
step 00300/05000 (6.00%) | loss: 5.404131 | lrm: 1.00 | dt: 1755.47ms | tok/sec: 9,333 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 14 | total time: 7.61m | eta: 123.4m
step 00301/05000 (6.02%) | loss: 5.397726 | lrm: 1.00 | dt: 1624.71ms | tok/sec: 10,084 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 14 | total time: 7.64m | eta: 123.4m
step 00302/05000 (6.04%) | loss: 5.398089 | lrm: 1.00 | dt: 1667.44ms | tok/sec: 9,825 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 14 | total time: 7.67m | eta: 123.4m
step 00303/05000 (6.06%) | loss: 5.393029 | lrm: 1.00 | dt: 1848.46ms | tok/sec: 8,863 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 14 | total time: 7.70m | eta: 123.4m
step 00304/05000 (6.08%) | loss: 5.371437 | lrm: 1.00 | dt: 1616.91ms | tok/sec: 10,132 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 14 | total time: 7.73m | eta: 123.4m
step 00305/05000 (6.10%) | loss: 5.373434 | lrm: 1.00 | dt: 1722.08ms | tok/sec: 9,514 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 14 | total time: 7.75m | eta: 123.4m
step 00306/05000 (6.12%) | loss: 5.372479 | lrm: 1.00 | dt: 1757.09ms | tok/sec: 9,324 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 14 | total time: 7.78m | eta: 123.4m
step 00307/05000 (6.14%) | loss: 5.377760 | lrm: 1.00 | dt: 1758.30ms | tok/sec: 9,318 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 14 | total time: 7.81m | eta: 123.5m
step 00308/05000 (6.16%) | loss: 5.372427 | lrm: 1.00 | dt: 1677.61ms | tok/sec: 9,766 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 14 | total time: 7.84m | eta: 123.5m
step 00309/05000 (6.18%) | loss: 5.363731 | lrm: 1.00 | dt: 1638.67ms | tok/sec: 9,998 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 14 | total time: 7.87m | eta: 123.4m
step 00310/05000 (6.20%) | loss: 5.364087 | lrm: 1.00 | dt: 1615.69ms | tok/sec: 10,140 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 14 | total time: 7.90m | eta: 123.4m
step 00311/05000 (6.22%) | loss: 5.344803 | lrm: 1.00 | dt: 1612.32ms | tok/sec: 10,161 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 14 | total time: 7.92m | eta: 123.4m
step 00312/05000 (6.24%) | loss: 5.331326 | lrm: 1.00 | dt: 1609.49ms | tok/sec: 10,179 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 14 | total time: 7.95m | eta: 123.4m
step 00313/05000 (6.26%) | loss: 5.338226 | lrm: 1.00 | dt: 1623.41ms | tok/sec: 10,092 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 14 | total time: 7.98m | eta: 123.4m
step 00314/05000 (6.28%) | loss: 5.322125 | lrm: 1.00 | dt: 1683.34ms | tok/sec: 9,733 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 14 | total time: 8.00m | eta: 123.4m
step 00315/05000 (6.30%) | loss: 5.320113 | lrm: 1.00 | dt: 1605.12ms | tok/sec: 10,207 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 15 | total time: 8.03m | eta: 123.4m
step 00316/05000 (6.32%) | loss: 5.320336 | lrm: 1.00 | dt: 1664.74ms | tok/sec: 9,841 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 15 | total time: 8.06m | eta: 123.4m
step 00317/05000 (6.34%) | loss: 5.314665 | lrm: 1.00 | dt: 1565.12ms | tok/sec: 10,468 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 15 | total time: 8.08m | eta: 123.3m
step 00318/05000 (6.36%) | loss: 5.303667 | lrm: 1.00 | dt: 1648.63ms | tok/sec: 9,937 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 15 | total time: 8.11m | eta: 123.3m
step 00319/05000 (6.38%) | loss: 5.315404 | lrm: 1.00 | dt: 1510.16ms | tok/sec: 10,849 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 15 | total time: 8.14m | eta: 123.3m
step 00320/05000 (6.40%) | loss: 5.300634 | lrm: 1.00 | dt: 1537.03ms | tok/sec: 10,659 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 15 | total time: 8.16m | eta: 123.2m
step 00321/05000 (6.42%) | loss: 5.313371 | lrm: 1.00 | dt: 1538.63ms | tok/sec: 10,648 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 15 | total time: 8.19m | eta: 123.2m
step 00322/05000 (6.44%) | loss: 5.331101 | lrm: 1.00 | dt: 1618.13ms | tok/sec: 10,125 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 15 | total time: 8.22m | eta: 123.2m
step 00323/05000 (6.46%) | loss: 5.321851 | lrm: 1.00 | dt: 1608.09ms | tok/sec: 10,188 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 15 | total time: 8.24m | eta: 123.2m
step 00324/05000 (6.48%) | loss: 5.340584 | lrm: 1.00 | dt: 1456.12ms | tok/sec: 11,251 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 15 | total time: 8.27m | eta: 123.1m
step 00325/05000 (6.50%) | loss: 5.330322 | lrm: 1.00 | dt: 1496.82ms | tok/sec: 10,945 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 15 | total time: 8.29m | eta: 123.1m
step 00326/05000 (6.52%) | loss: 5.324779 | lrm: 1.00 | dt: 1534.95ms | tok/sec: 10,673 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 15 | total time: 8.32m | eta: 123.0m
step 00327/05000 (6.54%) | loss: 5.338126 | lrm: 1.00 | dt: 1509.38ms | tok/sec: 10,854 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 15 | total time: 8.34m | eta: 123.0m
step 00328/05000 (6.56%) | loss: 5.349283 | lrm: 1.00 | dt: 1462.76ms | tok/sec: 11,200 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 15 | total time: 8.37m | eta: 122.9m
step 00329/05000 (6.58%) | loss: 5.352116 | lrm: 1.00 | dt: 1473.23ms | tok/sec: 11,121 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 15 | total time: 8.39m | eta: 122.9m
step 00330/05000 (6.60%) | loss: 5.353620 | lrm: 1.00 | dt: 1511.34ms | tok/sec: 10,840 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 15 | total time: 8.42m | eta: 122.8m
step 00331/05000 (6.62%) | loss: 5.339722 | lrm: 1.00 | dt: 1480.75ms | tok/sec: 11,064 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 15 | total time: 8.44m | eta: 122.8m
step 00332/05000 (6.64%) | loss: 5.336779 | lrm: 1.00 | dt: 1567.02ms | tok/sec: 10,455 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 15 | total time: 8.47m | eta: 122.8m
step 00333/05000 (6.66%) | loss: 5.322601 | lrm: 1.00 | dt: 1544.31ms | tok/sec: 10,609 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 15 | total time: 8.49m | eta: 122.7m
step 00334/05000 (6.68%) | loss: 5.316690 | lrm: 1.00 | dt: 1548.44ms | tok/sec: 10,580 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 15 | total time: 8.52m | eta: 122.7m
step 00335/05000 (6.70%) | loss: 5.313889 | lrm: 1.00 | dt: 1552.98ms | tok/sec: 10,550 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 15 | total time: 8.54m | eta: 122.7m
step 00336/05000 (6.72%) | loss: 5.315608 | lrm: 1.00 | dt: 1489.71ms | tok/sec: 10,998 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 15 | total time: 8.57m | eta: 122.6m
step 00337/05000 (6.74%) | loss: 5.311719 | lrm: 1.00 | dt: 1500.40ms | tok/sec: 10,919 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 15 | total time: 8.59m | eta: 122.6m
step 00338/05000 (6.76%) | loss: 5.302921 | lrm: 1.00 | dt: 1575.85ms | tok/sec: 10,396 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 16 | total time: 8.62m | eta: 122.5m
step 00339/05000 (6.78%) | loss: 5.302530 | lrm: 1.00 | dt: 1512.74ms | tok/sec: 10,830 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 16 | total time: 8.65m | eta: 122.5m
step 00340/05000 (6.80%) | loss: 5.292137 | lrm: 1.00 | dt: 1506.39ms | tok/sec: 10,876 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 16 | total time: 8.67m | eta: 122.4m
step 00341/05000 (6.82%) | loss: 5.291826 | lrm: 1.00 | dt: 1549.87ms | tok/sec: 10,571 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 16 | total time: 8.70m | eta: 122.4m
step 00342/05000 (6.84%) | loss: 5.283110 | lrm: 1.00 | dt: 1500.07ms | tok/sec: 10,922 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 16 | total time: 8.72m | eta: 122.4m
step 00343/05000 (6.86%) | loss: 5.285799 | lrm: 1.00 | dt: 1560.45ms | tok/sec: 10,499 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 16 | total time: 8.75m | eta: 122.3m
step 00344/05000 (6.88%) | loss: 5.279977 | lrm: 1.00 | dt: 1457.48ms | tok/sec: 11,241 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 16 | total time: 8.77m | eta: 122.3m
step 00345/05000 (6.90%) | loss: 5.277602 | lrm: 1.00 | dt: 1580.25ms | tok/sec: 10,367 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 16 | total time: 8.80m | eta: 122.3m
step 00346/05000 (6.92%) | loss: 5.293056 | lrm: 1.00 | dt: 1543.12ms | tok/sec: 10,617 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 16 | total time: 8.82m | eta: 122.2m
step 00347/05000 (6.94%) | loss: 5.299342 | lrm: 1.00 | dt: 1572.01ms | tok/sec: 10,422 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 16 | total time: 8.85m | eta: 122.2m
step 00348/05000 (6.96%) | loss: 5.293770 | lrm: 1.00 | dt: 1524.75ms | tok/sec: 10,745 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 16 | total time: 8.88m | eta: 122.2m
step 00349/05000 (6.98%) | loss: 5.291455 | lrm: 1.00 | dt: 1528.33ms | tok/sec: 10,720 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 16 | total time: 8.90m | eta: 122.1m
step 00350/05000 (7.00%) | loss: 5.292284 | lrm: 1.00 | dt: 1490.24ms | tok/sec: 10,994 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 16 | total time: 8.93m | eta: 122.1m
step 00351/05000 (7.02%) | loss: 5.290899 | lrm: 1.00 | dt: 1558.59ms | tok/sec: 10,512 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 16 | total time: 8.95m | eta: 122.1m
step 00352/05000 (7.04%) | loss: 5.281569 | lrm: 1.00 | dt: 1544.94ms | tok/sec: 10,604 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 16 | total time: 8.98m | eta: 122.0m
step 00353/05000 (7.06%) | loss: 5.281701 | lrm: 1.00 | dt: 1517.59ms | tok/sec: 10,796 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 16 | total time: 9.00m | eta: 122.0m
step 00354/05000 (7.08%) | loss: 5.286651 | lrm: 1.00 | dt: 1508.34ms | tok/sec: 10,862 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 16 | total time: 9.03m | eta: 121.9m
step 00355/05000 (7.10%) | loss: 5.285504 | lrm: 1.00 | dt: 1545.40ms | tok/sec: 10,601 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 16 | total time: 9.05m | eta: 121.9m
step 00356/05000 (7.12%) | loss: 5.289518 | lrm: 1.00 | dt: 1573.15ms | tok/sec: 10,414 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 16 | total time: 9.08m | eta: 121.9m
step 00357/05000 (7.14%) | loss: 5.288929 | lrm: 1.00 | dt: 1615.68ms | tok/sec: 10,140 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 16 | total time: 9.11m | eta: 121.9m
step 00358/05000 (7.16%) | loss: 5.284435 | lrm: 1.00 | dt: 1531.00ms | tok/sec: 10,701 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 16 | total time: 9.13m | eta: 121.8m
step 00359/05000 (7.18%) | loss: 5.274110 | lrm: 1.00 | dt: 1524.85ms | tok/sec: 10,744 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 16 | total time: 9.16m | eta: 121.8m
step 00360/05000 (7.20%) | loss: 5.281866 | lrm: 1.00 | dt: 1589.74ms | tok/sec: 10,306 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 16 | total time: 9.18m | eta: 121.8m
step 00361/05000 (7.22%) | loss: 5.283601 | lrm: 1.00 | dt: 1602.96ms | tok/sec: 10,221 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 17 | total time: 9.21m | eta: 121.7m
step 00362/05000 (7.24%) | loss: 5.281359 | lrm: 1.00 | dt: 1617.66ms | tok/sec: 10,128 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 17 | total time: 9.24m | eta: 121.7m
step 00363/05000 (7.26%) | loss: 5.278649 | lrm: 1.00 | dt: 1630.16ms | tok/sec: 10,050 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 17 | total time: 9.27m | eta: 121.7m
step 00364/05000 (7.28%) | loss: 5.265586 | lrm: 1.00 | dt: 1612.61ms | tok/sec: 10,159 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 17 | total time: 9.29m | eta: 121.7m
step 00365/05000 (7.30%) | loss: 5.258490 | lrm: 1.00 | dt: 1653.77ms | tok/sec: 9,907 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 17 | total time: 9.32m | eta: 121.7m
step 00366/05000 (7.32%) | loss: 5.257318 | lrm: 1.00 | dt: 1625.16ms | tok/sec: 10,081 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 17 | total time: 9.35m | eta: 121.7m
step 00367/05000 (7.34%) | loss: 5.251466 | lrm: 1.00 | dt: 1655.79ms | tok/sec: 9,894 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 17 | total time: 9.37m | eta: 121.7m
step 00368/05000 (7.36%) | loss: 5.248037 | lrm: 1.00 | dt: 1575.04ms | tok/sec: 10,402 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 17 | total time: 9.40m | eta: 121.6m
step 00369/05000 (7.38%) | loss: 5.254873 | lrm: 1.00 | dt: 1623.56ms | tok/sec: 10,091 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 17 | total time: 9.43m | eta: 121.6m
step 00370/05000 (7.40%) | loss: 5.244935 | lrm: 1.00 | dt: 1585.98ms | tok/sec: 10,330 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 17 | total time: 9.45m | eta: 121.6m
step 00371/05000 (7.42%) | loss: 5.242706 | lrm: 1.00 | dt: 1641.36ms | tok/sec: 9,981 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 17 | total time: 9.48m | eta: 121.6m
step 00372/05000 (7.44%) | loss: 5.235025 | lrm: 1.00 | dt: 1648.75ms | tok/sec: 9,937 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 17 | total time: 9.51m | eta: 121.6m
step 00373/05000 (7.46%) | loss: 5.227520 | lrm: 1.00 | dt: 1591.11ms | tok/sec: 10,297 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 17 | total time: 9.54m | eta: 121.6m
step 00374/05000 (7.48%) | loss: 5.224606 | lrm: 1.00 | dt: 1643.48ms | tok/sec: 9,969 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 17 | total time: 9.56m | eta: 121.5m
step 00375/05000 (7.50%) | loss: 5.207938 | lrm: 1.00 | dt: 1612.03ms | tok/sec: 10,163 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 17 | total time: 9.59m | eta: 121.5m
step 00376/05000 (7.52%) | loss: 5.215439 | lrm: 1.00 | dt: 1618.57ms | tok/sec: 10,122 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 17 | total time: 9.62m | eta: 121.5m
step 00377/05000 (7.54%) | loss: 5.210800 | lrm: 1.00 | dt: 1629.71ms | tok/sec: 10,053 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 17 | total time: 9.64m | eta: 121.5m
step 00378/05000 (7.56%) | loss: 5.205812 | lrm: 1.00 | dt: 1658.80ms | tok/sec: 9,877 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 17 | total time: 9.67m | eta: 121.5m
step 00379/05000 (7.58%) | loss: 5.191068 | lrm: 1.00 | dt: 1618.26ms | tok/sec: 10,124 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 17 | total time: 9.70m | eta: 121.5m
step 00380/05000 (7.60%) | loss: 5.192353 | lrm: 1.00 | dt: 1604.15ms | tok/sec: 10,213 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 17 | total time: 9.73m | eta: 121.4m
step 00381/05000 (7.62%) | loss: 5.178041 | lrm: 1.00 | dt: 1613.36ms | tok/sec: 10,155 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 17 | total time: 9.75m | eta: 121.4m
step 00382/05000 (7.64%) | loss: 5.177933 | lrm: 1.00 | dt: 1653.30ms | tok/sec: 9,909 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 17 | total time: 9.78m | eta: 121.4m
step 00383/05000 (7.66%) | loss: 5.178089 | lrm: 1.00 | dt: 1670.61ms | tok/sec: 9,807 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 17 | total time: 9.81m | eta: 121.4m
step 00384/05000 (7.68%) | loss: 5.163804 | lrm: 1.00 | dt: 1604.97ms | tok/sec: 10,208 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 17 | total time: 9.83m | eta: 121.4m
step 00385/05000 (7.70%) | loss: 5.171951 | lrm: 1.00 | dt: 1641.05ms | tok/sec: 9,983 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 18 | total time: 9.86m | eta: 121.4m
step 00386/05000 (7.72%) | loss: 5.179308 | lrm: 1.00 | dt: 1598.28ms | tok/sec: 10,251 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 18 | total time: 9.89m | eta: 121.3m
step 00387/05000 (7.74%) | loss: 5.175034 | lrm: 1.00 | dt: 1650.73ms | tok/sec: 9,925 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 18 | total time: 9.92m | eta: 121.3m
step 00388/05000 (7.76%) | loss: 5.175444 | lrm: 1.00 | dt: 1614.82ms | tok/sec: 10,146 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 18 | total time: 9.94m | eta: 121.3m
step 00389/05000 (7.78%) | loss: 5.163338 | lrm: 1.00 | dt: 1605.35ms | tok/sec: 10,205 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 18 | total time: 9.97m | eta: 121.3m
step 00390/05000 (7.80%) | loss: 5.167196 | lrm: 1.00 | dt: 1614.53ms | tok/sec: 10,147 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 18 | total time: 10.00m | eta: 121.3m
step 00391/05000 (7.82%) | loss: 5.165009 | lrm: 1.00 | dt: 1626.35ms | tok/sec: 10,074 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 18 | total time: 10.02m | eta: 121.3m
step 00392/05000 (7.84%) | loss: 5.170127 | lrm: 1.00 | dt: 1552.70ms | tok/sec: 10,551 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 18 | total time: 10.05m | eta: 121.2m
step 00393/05000 (7.86%) | loss: 5.171063 | lrm: 1.00 | dt: 1644.70ms | tok/sec: 9,961 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 18 | total time: 10.08m | eta: 121.2m
step 00394/05000 (7.88%) | loss: 5.158941 | lrm: 1.00 | dt: 1571.43ms | tok/sec: 10,426 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 18 | total time: 10.10m | eta: 121.2m
step 00395/05000 (7.90%) | loss: 5.158210 | lrm: 1.00 | dt: 1657.25ms | tok/sec: 9,886 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 18 | total time: 10.13m | eta: 121.2m
step 00396/05000 (7.92%) | loss: 5.150955 | lrm: 1.00 | dt: 1606.41ms | tok/sec: 10,199 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 18 | total time: 10.16m | eta: 121.2m
step 00397/05000 (7.94%) | loss: 5.148353 | lrm: 1.00 | dt: 1592.52ms | tok/sec: 10,288 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 18 | total time: 10.18m | eta: 121.1m
step 00398/05000 (7.96%) | loss: 5.146910 | lrm: 1.00 | dt: 1618.29ms | tok/sec: 10,124 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 18 | total time: 10.21m | eta: 121.1m
step 00399/05000 (7.98%) | loss: 5.146665 | lrm: 1.00 | dt: 1505.07ms | tok/sec: 10,885 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 18 | total time: 10.24m | eta: 121.1m
Step 00400 | Validation bpb: 1.589013
<|bos|>The capital of France is a popular choice for the American American Association. The American Association is a popular choice
<|bos|>The chemical symbol of gold is a significant difference between the gold and the gold. The difference between the gold and
<|bos|>If yesterday was Friday, then tomorrow will be a little bit of a little bit, and then the day will be the first
<|bos|>The opposite of hot is the most common cause of the cold weather. The cold weather is the most common
<|bos|>The planets of the solar system are: the Earth's Earth's Earth's Earth's Earth's Earth's Earth's Earth
<|bos|>My favorite color is a good thing to do. I'm not sure I don't have a lot
<|bos|>If 5*x + 3 = 13, then x is 2.5, 2.5, 2, 2.
step 00400/05000 (8.00%) | loss: 5.162750 | lrm: 1.00 | dt: 1595.21ms | tok/sec: 10,270 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 18 | total time: 10.26m | eta: 121.1m
step 00401/05000 (8.02%) | loss: 5.155845 | lrm: 1.00 | dt: 1690.49ms | tok/sec: 9,691 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 18 | total time: 10.29m | eta: 121.0m
step 00402/05000 (8.04%) | loss: 5.138155 | lrm: 1.00 | dt: 1525.52ms | tok/sec: 10,739 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 18 | total time: 10.32m | eta: 121.0m
step 00403/05000 (8.06%) | loss: 5.139855 | lrm: 1.00 | dt: 1640.16ms | tok/sec: 9,989 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 18 | total time: 10.34m | eta: 121.0m
step 00404/05000 (8.08%) | loss: 5.149782 | lrm: 1.00 | dt: 1483.22ms | tok/sec: 11,046 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 18 | total time: 10.37m | eta: 121.0m
step 00405/05000 (8.10%) | loss: 5.142798 | lrm: 1.00 | dt: 1548.40ms | tok/sec: 10,581 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 18 | total time: 10.39m | eta: 120.9m
step 00406/05000 (8.12%) | loss: 5.144747 | lrm: 1.00 | dt: 1625.38ms | tok/sec: 10,080 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 18 | total time: 10.42m | eta: 120.9m
step 00407/05000 (8.14%) | loss: 5.129845 | lrm: 1.00 | dt: 1613.39ms | tok/sec: 10,155 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 18 | total time: 10.45m | eta: 120.9m
step 00408/05000 (8.16%) | loss: 5.117839 | lrm: 1.00 | dt: 1607.51ms | tok/sec: 10,192 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 19 | total time: 10.48m | eta: 120.9m
step 00409/05000 (8.18%) | loss: 5.110619 | lrm: 1.00 | dt: 1624.00ms | tok/sec: 10,088 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 19 | total time: 10.50m | eta: 120.8m
step 00410/05000 (8.20%) | loss: 5.121897 | lrm: 1.00 | dt: 1613.93ms | tok/sec: 10,151 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 19 | total time: 10.53m | eta: 120.8m
step 00411/05000 (8.22%) | loss: 5.117535 | lrm: 1.00 | dt: 1554.52ms | tok/sec: 10,539 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 19 | total time: 10.56m | eta: 120.8m
step 00412/05000 (8.24%) | loss: 5.126947 | lrm: 1.00 | dt: 1599.62ms | tok/sec: 10,242 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 19 | total time: 10.58m | eta: 120.8m
step 00413/05000 (8.26%) | loss: 5.133386 | lrm: 1.00 | dt: 1523.28ms | tok/sec: 10,755 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 19 | total time: 10.61m | eta: 120.7m
step 00414/05000 (8.28%) | loss: 5.128783 | lrm: 1.00 | dt: 1479.23ms | tok/sec: 11,076 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 19 | total time: 10.63m | eta: 120.7m
step 00415/05000 (8.30%) | loss: 5.126750 | lrm: 1.00 | dt: 1511.08ms | tok/sec: 10,842 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 19 | total time: 10.66m | eta: 120.6m
step 00416/05000 (8.32%) | loss: 5.132171 | lrm: 1.00 | dt: 1498.80ms | tok/sec: 10,931 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 19 | total time: 10.68m | eta: 120.6m
step 00417/05000 (8.34%) | loss: 5.120701 | lrm: 1.00 | dt: 1540.95ms | tok/sec: 10,632 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 19 | total time: 10.71m | eta: 120.6m
step 00418/05000 (8.36%) | loss: 5.126056 | lrm: 1.00 | dt: 1483.01ms | tok/sec: 11,047 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 19 | total time: 10.73m | eta: 120.5m
step 00419/05000 (8.38%) | loss: 5.127744 | lrm: 1.00 | dt: 1461.59ms | tok/sec: 11,209 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 19 | total time: 10.76m | eta: 120.5m
step 00420/05000 (8.40%) | loss: 5.130331 | lrm: 1.00 | dt: 1558.73ms | tok/sec: 10,511 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 19 | total time: 10.78m | eta: 120.5m
step 00421/05000 (8.42%) | loss: 5.134757 | lrm: 1.00 | dt: 1643.78ms | tok/sec: 9,967 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 19 | total time: 10.81m | eta: 120.4m
step 00422/05000 (8.44%) | loss: 5.133029 | lrm: 1.00 | dt: 1445.21ms | tok/sec: 11,336 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 19 | total time: 10.83m | eta: 120.4m
step 00423/05000 (8.46%) | loss: 5.135137 | lrm: 1.00 | dt: 1631.00ms | tok/sec: 10,045 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 19 | total time: 10.86m | eta: 120.4m
step 00424/05000 (8.48%) | loss: 5.134164 | lrm: 1.00 | dt: 1631.10ms | tok/sec: 10,044 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 19 | total time: 10.89m | eta: 120.4m
step 00425/05000 (8.50%) | loss: 5.132793 | lrm: 1.00 | dt: 1506.86ms | tok/sec: 10,872 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 19 | total time: 10.91m | eta: 120.3m
step 00426/05000 (8.52%) | loss: 5.131282 | lrm: 1.00 | dt: 1594.93ms | tok/sec: 10,272 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 19 | total time: 10.94m | eta: 120.3m
step 00427/05000 (8.54%) | loss: 5.130493 | lrm: 1.00 | dt: 1483.50ms | tok/sec: 11,044 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 19 | total time: 10.96m | eta: 120.2m
step 00428/05000 (8.56%) | loss: 5.115971 | lrm: 1.00 | dt: 1499.66ms | tok/sec: 10,925 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 19 | total time: 10.99m | eta: 120.2m
step 00429/05000 (8.58%) | loss: 5.130174 | lrm: 1.00 | dt: 1473.63ms | tok/sec: 11,118 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 19 | total time: 11.01m | eta: 120.2m
step 00430/05000 (8.60%) | loss: 5.122094 | lrm: 1.00 | dt: 1494.57ms | tok/sec: 10,962 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 19 | total time: 11.04m | eta: 120.1m
step 00431/05000 (8.62%) | loss: 5.099120 | lrm: 1.00 | dt: 1475.21ms | tok/sec: 11,106 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 20 | total time: 11.06m | eta: 120.1m
step 00432/05000 (8.64%) | loss: 5.109051 | lrm: 1.00 | dt: 1502.49ms | tok/sec: 10,904 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 20 | total time: 11.09m | eta: 120.0m
step 00433/05000 (8.66%) | loss: 5.115782 | lrm: 1.00 | dt: 1566.03ms | tok/sec: 10,462 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 20 | total time: 11.12m | eta: 120.0m
step 00434/05000 (8.68%) | loss: 5.123273 | lrm: 1.00 | dt: 1489.94ms | tok/sec: 10,996 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 20 | total time: 11.14m | eta: 120.0m
step 00435/05000 (8.70%) | loss: 5.119873 | lrm: 1.00 | dt: 1577.33ms | tok/sec: 10,387 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 20 | total time: 11.17m | eta: 119.9m
step 00436/05000 (8.72%) | loss: 5.116569 | lrm: 1.00 | dt: 1469.69ms | tok/sec: 11,147 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 20 | total time: 11.19m | eta: 119.9m
step 00437/05000 (8.74%) | loss: 5.115448 | lrm: 1.00 | dt: 1475.73ms | tok/sec: 11,102 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 20 | total time: 11.22m | eta: 119.8m
step 00438/05000 (8.76%) | loss: 5.123473 | lrm: 1.00 | dt: 1504.19ms | tok/sec: 10,892 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 20 | total time: 11.24m | eta: 119.8m
step 00439/05000 (8.78%) | loss: 5.102357 | lrm: 1.00 | dt: 1487.16ms | tok/sec: 11,016 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 20 | total time: 11.27m | eta: 119.8m
step 00440/05000 (8.80%) | loss: 5.086470 | lrm: 1.00 | dt: 1532.07ms | tok/sec: 10,694 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 20 | total time: 11.29m | eta: 119.7m
step 00441/05000 (8.82%) | loss: 5.083984 | lrm: 1.00 | dt: 1559.64ms | tok/sec: 10,504 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 20 | total time: 11.32m | eta: 119.7m
step 00442/05000 (8.84%) | loss: 5.082578 | lrm: 1.00 | dt: 1625.07ms | tok/sec: 10,082 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 20 | total time: 11.34m | eta: 119.7m
step 00443/05000 (8.86%) | loss: 5.091238 | lrm: 1.00 | dt: 1777.78ms | tok/sec: 9,215 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 20 | total time: 11.37m | eta: 119.7m
step 00444/05000 (8.88%) | loss: 5.102613 | lrm: 1.00 | dt: 1697.48ms | tok/sec: 9,651 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 20 | total time: 11.40m | eta: 119.7m
step 00445/05000 (8.90%) | loss: 5.097041 | lrm: 1.00 | dt: 1735.03ms | tok/sec: 9,443 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 20 | total time: 11.43m | eta: 119.7m
step 00446/05000 (8.92%) | loss: 5.089694 | lrm: 1.00 | dt: 1710.05ms | tok/sec: 9,581 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 20 | total time: 11.46m | eta: 119.7m
step 00447/05000 (8.94%) | loss: 5.094113 | lrm: 1.00 | dt: 1729.48ms | tok/sec: 9,473 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 20 | total time: 11.49m | eta: 119.7m
step 00448/05000 (8.96%) | loss: 5.076484 | lrm: 1.00 | dt: 1723.61ms | tok/sec: 9,505 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 20 | total time: 11.52m | eta: 119.7m
step 00449/05000 (8.98%) | loss: 5.081294 | lrm: 1.00 | dt: 1748.63ms | tok/sec: 9,369 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 20 | total time: 11.55m | eta: 119.7m
step 00450/05000 (9.00%) | loss: 5.081637 | lrm: 1.00 | dt: 1739.13ms | tok/sec: 9,420 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 20 | total time: 11.57m | eta: 119.7m
step 00451/05000 (9.02%) | loss: 5.070259 | lrm: 1.00 | dt: 1736.98ms | tok/sec: 9,432 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 20 | total time: 11.60m | eta: 119.7m
step 00452/05000 (9.04%) | loss: 5.075004 | lrm: 1.00 | dt: 1674.85ms | tok/sec: 9,782 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 20 | total time: 11.63m | eta: 119.7m
step 00453/05000 (9.06%) | loss: 5.093108 | lrm: 1.00 | dt: 1723.03ms | tok/sec: 9,508 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 20 | total time: 11.66m | eta: 119.7m
step 00454/05000 (9.08%) | loss: 5.091906 | lrm: 1.00 | dt: 1710.00ms | tok/sec: 9,581 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 20 | total time: 11.69m | eta: 119.7m
step 00455/05000 (9.10%) | loss: 5.081114 | lrm: 1.00 | dt: 1655.05ms | tok/sec: 9,899 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 21 | total time: 11.72m | eta: 119.7m
step 00456/05000 (9.12%) | loss: 5.085234 | lrm: 1.00 | dt: 1754.81ms | tok/sec: 9,336 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 21 | total time: 11.75m | eta: 119.7m
step 00457/05000 (9.14%) | loss: 5.079938 | lrm: 1.00 | dt: 1733.95ms | tok/sec: 9,448 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 21 | total time: 11.77m | eta: 119.7m
step 00458/05000 (9.16%) | loss: 5.069873 | lrm: 1.00 | dt: 1756.17ms | tok/sec: 9,329 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 21 | total time: 11.80m | eta: 119.7m
step 00459/05000 (9.18%) | loss: 5.068410 | lrm: 1.00 | dt: 1694.91ms | tok/sec: 9,666 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 21 | total time: 11.83m | eta: 119.7m
step 00460/05000 (9.20%) | loss: 5.052966 | lrm: 1.00 | dt: 1681.17ms | tok/sec: 9,745 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 21 | total time: 11.86m | eta: 119.7m
step 00461/05000 (9.22%) | loss: 5.046207 | lrm: 1.00 | dt: 1695.17ms | tok/sec: 9,665 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 21 | total time: 11.89m | eta: 119.6m
step 00462/05000 (9.24%) | loss: 5.039496 | lrm: 1.00 | dt: 1708.00ms | tok/sec: 9,592 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 21 | total time: 11.92m | eta: 119.6m
step 00463/05000 (9.26%) | loss: 5.051601 | lrm: 1.00 | dt: 2010.79ms | tok/sec: 8,148 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 21 | total time: 11.95m | eta: 119.7m
step 00464/05000 (9.28%) | loss: 5.044780 | lrm: 1.00 | dt: 1718.13ms | tok/sec: 9,535 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 21 | total time: 11.98m | eta: 119.7m
step 00465/05000 (9.30%) | loss: 5.046777 | lrm: 1.00 | dt: 1730.32ms | tok/sec: 9,468 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 21 | total time: 12.01m | eta: 119.7m
step 00466/05000 (9.32%) | loss: 5.023136 | lrm: 1.00 | dt: 1701.43ms | tok/sec: 9,629 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 21 | total time: 12.04m | eta: 119.7m
step 00467/05000 (9.34%) | loss: 5.016066 | lrm: 1.00 | dt: 1662.53ms | tok/sec: 9,854 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 21 | total time: 12.06m | eta: 119.7m
step 00468/05000 (9.36%) | loss: 5.026448 | lrm: 1.00 | dt: 1756.88ms | tok/sec: 9,325 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 21 | total time: 12.09m | eta: 119.7m
step 00469/05000 (9.38%) | loss: 5.036847 | lrm: 1.00 | dt: 1783.94ms | tok/sec: 9,184 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 21 | total time: 12.12m | eta: 119.7m
step 00470/05000 (9.40%) | loss: 5.045063 | lrm: 1.00 | dt: 1703.35ms | tok/sec: 9,618 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 21 | total time: 12.15m | eta: 119.7m
step 00471/05000 (9.42%) | loss: 5.036939 | lrm: 1.00 | dt: 1777.59ms | tok/sec: 9,216 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 21 | total time: 12.18m | eta: 119.7m
step 00472/05000 (9.44%) | loss: 5.014714 | lrm: 1.00 | dt: 1733.04ms | tok/sec: 9,453 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 21 | total time: 12.21m | eta: 119.7m
step 00473/05000 (9.46%) | loss: 5.027264 | lrm: 1.00 | dt: 1724.92ms | tok/sec: 9,498 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 21 | total time: 12.24m | eta: 119.7m
step 00474/05000 (9.48%) | loss: 5.021803 | lrm: 1.00 | dt: 1686.49ms | tok/sec: 9,714 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 21 | total time: 12.27m | eta: 119.7m
step 00475/05000 (9.50%) | loss: 5.029893 | lrm: 1.00 | dt: 1749.66ms | tok/sec: 9,364 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 21 | total time: 12.30m | eta: 119.7m
step 00476/05000 (9.52%) | loss: 5.038981 | lrm: 1.00 | dt: 1689.49ms | tok/sec: 9,697 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 21 | total time: 12.32m | eta: 119.6m
step 00477/05000 (9.54%) | loss: 5.052397 | lrm: 1.00 | dt: 1742.67ms | tok/sec: 9,401 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 21 | total time: 12.35m | eta: 119.6m
step 00478/05000 (9.56%) | loss: 5.039770 | lrm: 1.00 | dt: 1749.29ms | tok/sec: 9,366 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 22 | total time: 12.38m | eta: 119.6m
step 00479/05000 (9.58%) | loss: 5.032304 | lrm: 1.00 | dt: 1739.04ms | tok/sec: 9,421 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 22 | total time: 12.41m | eta: 119.6m
step 00480/05000 (9.60%) | loss: 5.030068 | lrm: 1.00 | dt: 1712.10ms | tok/sec: 9,569 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 22 | total time: 12.44m | eta: 119.6m
step 00481/05000 (9.62%) | loss: 5.031350 | lrm: 1.00 | dt: 1713.00ms | tok/sec: 9,564 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 22 | total time: 12.47m | eta: 119.6m
step 00482/05000 (9.64%) | loss: 5.037476 | lrm: 1.00 | dt: 1667.74ms | tok/sec: 9,824 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 22 | total time: 12.50m | eta: 119.6m
step 00483/05000 (9.66%) | loss: 5.032343 | lrm: 1.00 | dt: 1692.68ms | tok/sec: 9,679 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 22 | total time: 12.52m | eta: 119.6m
step 00484/05000 (9.68%) | loss: 5.032431 | lrm: 1.00 | dt: 1766.94ms | tok/sec: 9,272 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 22 | total time: 12.55m | eta: 119.6m
step 00485/05000 (9.70%) | loss: 5.013612 | lrm: 1.00 | dt: 1644.15ms | tok/sec: 9,965 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 22 | total time: 12.58m | eta: 119.6m
step 00486/05000 (9.72%) | loss: 5.005050 | lrm: 1.00 | dt: 1690.20ms | tok/sec: 9,693 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 22 | total time: 12.61m | eta: 119.6m
step 00487/05000 (9.74%) | loss: 4.992436 | lrm: 1.00 | dt: 1671.80ms | tok/sec: 9,800 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 22 | total time: 12.64m | eta: 119.6m
step 00488/05000 (9.76%) | loss: 4.993002 | lrm: 1.00 | dt: 1670.44ms | tok/sec: 9,808 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 22 | total time: 12.67m | eta: 119.5m
step 00489/05000 (9.78%) | loss: 4.992734 | lrm: 1.00 | dt: 1632.33ms | tok/sec: 10,037 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 22 | total time: 12.69m | eta: 119.5m
step 00490/05000 (9.80%) | loss: 4.984846 | lrm: 1.00 | dt: 1700.47ms | tok/sec: 9,634 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 22 | total time: 12.72m | eta: 119.5m
step 00491/05000 (9.82%) | loss: 4.983747 | lrm: 1.00 | dt: 1644.73ms | tok/sec: 9,961 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 22 | total time: 12.75m | eta: 119.5m
step 00492/05000 (9.84%) | loss: 4.960795 | lrm: 1.00 | dt: 1699.44ms | tok/sec: 9,640 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 22 | total time: 12.78m | eta: 119.5m
step 00493/05000 (9.86%) | loss: 4.960509 | lrm: 1.00 | dt: 1664.44ms | tok/sec: 9,843 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 22 | total time: 12.80m | eta: 119.5m
step 00494/05000 (9.88%) | loss: 4.956774 | lrm: 1.00 | dt: 1661.21ms | tok/sec: 9,862 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 22 | total time: 12.83m | eta: 119.5m
step 00495/05000 (9.90%) | loss: 4.956656 | lrm: 1.00 | dt: 1703.33ms | tok/sec: 9,618 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 22 | total time: 12.86m | eta: 119.5m
step 00496/05000 (9.92%) | loss: 4.967337 | lrm: 1.00 | dt: 1688.53ms | tok/sec: 9,703 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 22 | total time: 12.89m | eta: 119.4m
step 00497/05000 (9.94%) | loss: 4.970128 | lrm: 1.00 | dt: 1701.41ms | tok/sec: 9,629 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 22 | total time: 12.92m | eta: 119.4m
step 00498/05000 (9.96%) | loss: 4.964654 | lrm: 1.00 | dt: 1642.03ms | tok/sec: 9,977 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 22 | total time: 12.94m | eta: 119.4m
step 00499/05000 (9.98%) | loss: 4.967073 | lrm: 1.00 | dt: 1667.66ms | tok/sec: 9,824 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 22 | total time: 12.97m | eta: 119.4m
Step 00500 | Validation bpb: 1.530241
<|bos|>The capital of France is a major part of the world's largest economy, and the world's largest economy
<|bos|>The chemical symbol of gold is a symbol of the role of the gold and the gold of the gold gold.
<|bos|>If yesterday was Friday, then tomorrow will be a little bit more than a day, and the first time will be the first
<|bos|>The opposite of hot is a hot water, and the water is a hot water, and the water is
<|bos|>The planets of the solar system are: the solar system, the solar system, the solar system, the solar system,
<|bos|>My favorite color is a great way to make a difference in the color of the color. The color
<|bos|>If 5*x + 3 = 13, then x is 2x2 = 2x2 = 3x2 =
step 00500/05000 (10.00%) | loss: 4.956850 | lrm: 1.00 | dt: 1733.17ms | tok/sec: 9,453 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 22 | total time: 13.00m | eta: 119.4m
step 00501/05000 (10.02%) | loss: 4.949587 | lrm: 1.00 | dt: 1671.38ms | tok/sec: 9,802 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 23 | total time: 13.03m | eta: 119.4m
step 00502/05000 (10.04%) | loss: 4.949116 | lrm: 1.00 | dt: 1628.08ms | tok/sec: 10,063 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 23 | total time: 13.06m | eta: 119.4m
step 00503/05000 (10.06%) | loss: 4.961250 | lrm: 1.00 | dt: 1656.04ms | tok/sec: 9,893 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 23 | total time: 13.08m | eta: 119.3m
step 00504/05000 (10.08%) | loss: 4.950739 | lrm: 1.00 | dt: 1655.68ms | tok/sec: 9,895 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 23 | total time: 13.11m | eta: 119.3m
step 00505/05000 (10.10%) | loss: 4.938183 | lrm: 1.00 | dt: 1654.92ms | tok/sec: 9,900 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 23 | total time: 13.14m | eta: 119.3m
step 00506/05000 (10.12%) | loss: 4.933513 | lrm: 1.00 | dt: 1667.08ms | tok/sec: 9,827 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 23 | total time: 13.17m | eta: 119.3m
step 00507/05000 (10.14%) | loss: 4.920662 | lrm: 1.00 | dt: 1666.67ms | tok/sec: 9,830 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 23 | total time: 13.19m | eta: 119.3m
step 00508/05000 (10.16%) | loss: 4.927239 | lrm: 1.00 | dt: 1628.71ms | tok/sec: 10,059 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 23 | total time: 13.22m | eta: 119.3m
step 00509/05000 (10.18%) | loss: 4.936050 | lrm: 1.00 | dt: 1676.64ms | tok/sec: 9,771 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 23 | total time: 13.25m | eta: 119.2m
step 00510/05000 (10.20%) | loss: 4.924076 | lrm: 1.00 | dt: 1703.78ms | tok/sec: 9,616 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 23 | total time: 13.28m | eta: 119.2m
step 00511/05000 (10.22%) | loss: 4.917610 | lrm: 1.00 | dt: 1683.24ms | tok/sec: 9,733 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 23 | total time: 13.31m | eta: 119.2m
step 00512/05000 (10.24%) | loss: 4.901268 | lrm: 1.00 | dt: 1685.40ms | tok/sec: 9,721 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 23 | total time: 13.33m | eta: 119.2m
step 00513/05000 (10.26%) | loss: 4.897140 | lrm: 1.00 | dt: 1667.40ms | tok/sec: 9,826 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 23 | total time: 13.36m | eta: 119.2m
step 00514/05000 (10.28%) | loss: 4.901928 | lrm: 1.00 | dt: 1649.82ms | tok/sec: 9,930 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 23 | total time: 13.39m | eta: 119.2m
step 00515/05000 (10.30%) | loss: 4.898617 | lrm: 1.00 | dt: 1636.68ms | tok/sec: 10,010 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 23 | total time: 13.42m | eta: 119.2m
step 00516/05000 (10.32%) | loss: 4.884616 | lrm: 1.00 | dt: 1660.00ms | tok/sec: 9,869 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 23 | total time: 13.44m | eta: 119.1m
step 00517/05000 (10.34%) | loss: 4.883983 | lrm: 1.00 | dt: 1669.42ms | tok/sec: 9,814 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 23 | total time: 13.47m | eta: 119.1m
step 00518/05000 (10.36%) | loss: 4.880980 | lrm: 1.00 | dt: 1676.04ms | tok/sec: 9,775 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 23 | total time: 13.50m | eta: 119.1m
step 00519/05000 (10.38%) | loss: 4.881885 | lrm: 1.00 | dt: 1687.17ms | tok/sec: 9,710 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 23 | total time: 13.53m | eta: 119.1m
step 00520/05000 (10.40%) | loss: 4.890756 | lrm: 1.00 | dt: 1664.31ms | tok/sec: 9,844 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 23 | total time: 13.56m | eta: 119.1m
step 00521/05000 (10.42%) | loss: 4.893724 | lrm: 1.00 | dt: 1668.92ms | tok/sec: 9,817 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 23 | total time: 13.58m | eta: 119.1m
step 00522/05000 (10.44%) | loss: 4.879406 | lrm: 1.00 | dt: 1651.55ms | tok/sec: 9,920 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 23 | total time: 13.61m | eta: 119.0m
step 00523/05000 (10.46%) | loss: 4.880446 | lrm: 1.00 | dt: 1668.93ms | tok/sec: 9,817 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 23 | total time: 13.64m | eta: 119.0m
step 00524/05000 (10.48%) | loss: 4.887979 | lrm: 1.00 | dt: 1658.75ms | tok/sec: 9,877 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 23 | total time: 13.67m | eta: 119.0m
step 00525/05000 (10.50%) | loss: 4.877457 | lrm: 1.00 | dt: 1652.56ms | tok/sec: 9,914 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 24 | total time: 13.69m | eta: 119.0m
step 00526/05000 (10.52%) | loss: 4.890098 | lrm: 1.00 | dt: 1633.44ms | tok/sec: 10,030 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 24 | total time: 13.72m | eta: 119.0m
step 00527/05000 (10.54%) | loss: 4.864105 | lrm: 1.00 | dt: 1703.94ms | tok/sec: 9,615 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 24 | total time: 13.75m | eta: 119.0m
step 00528/05000 (10.56%) | loss: 4.861552 | lrm: 1.00 | dt: 1619.21ms | tok/sec: 10,118 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 24 | total time: 13.78m | eta: 118.9m
step 00529/05000 (10.58%) | loss: 4.872519 | lrm: 1.00 | dt: 1662.68ms | tok/sec: 9,853 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 24 | total time: 13.80m | eta: 118.9m
step 00530/05000 (10.60%) | loss: 4.872726 | lrm: 1.00 | dt: 1661.22ms | tok/sec: 9,862 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 24 | total time: 13.83m | eta: 118.9m
step 00531/05000 (10.62%) | loss: 4.880007 | lrm: 1.00 | dt: 1674.48ms | tok/sec: 9,784 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 24 | total time: 13.86m | eta: 118.9m
step 00532/05000 (10.64%) | loss: 4.863036 | lrm: 1.00 | dt: 1660.76ms | tok/sec: 9,865 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 24 | total time: 13.89m | eta: 118.9m
step 00533/05000 (10.66%) | loss: 4.857250 | lrm: 1.00 | dt: 1656.88ms | tok/sec: 9,888 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 24 | total time: 13.92m | eta: 118.9m
step 00534/05000 (10.68%) | loss: 4.869319 | lrm: 1.00 | dt: 1646.23ms | tok/sec: 9,952 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 24 | total time: 13.94m | eta: 118.8m
step 00535/05000 (10.70%) | loss: 4.878481 | lrm: 1.00 | dt: 1630.62ms | tok/sec: 10,047 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 24 | total time: 13.97m | eta: 118.8m
step 00536/05000 (10.72%) | loss: 4.877163 | lrm: 1.00 | dt: 1673.78ms | tok/sec: 9,788 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 24 | total time: 14.00m | eta: 118.8m
step 00537/05000 (10.74%) | loss: 4.886117 | lrm: 1.00 | dt: 1682.35ms | tok/sec: 9,738 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 24 | total time: 14.03m | eta: 118.8m
step 00538/05000 (10.76%) | loss: 4.875892 | lrm: 1.00 | dt: 1666.03ms | tok/sec: 9,834 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 24 | total time: 14.05m | eta: 118.8m
step 00539/05000 (10.78%) | loss: 4.861840 | lrm: 1.00 | dt: 1634.95ms | tok/sec: 10,021 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 24 | total time: 14.08m | eta: 118.7m
step 00540/05000 (10.80%) | loss: 4.869928 | lrm: 1.00 | dt: 1670.03ms | tok/sec: 9,810 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 24 | total time: 14.11m | eta: 118.7m
step 00541/05000 (10.82%) | loss: 4.878575 | lrm: 1.00 | dt: 1637.79ms | tok/sec: 10,003 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 24 | total time: 14.14m | eta: 118.7m
step 00542/05000 (10.84%) | loss: 4.867279 | lrm: 1.00 | dt: 1655.50ms | tok/sec: 9,896 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 24 | total time: 14.16m | eta: 118.7m
step 00543/05000 (10.86%) | loss: 4.877878 | lrm: 1.00 | dt: 1633.03ms | tok/sec: 10,032 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 24 | total time: 14.19m | eta: 118.7m
step 00544/05000 (10.88%) | loss: 4.881561 | lrm: 1.00 | dt: 1662.37ms | tok/sec: 9,855 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 24 | total time: 14.22m | eta: 118.6m
step 00545/05000 (10.90%) | loss: 4.878375 | lrm: 1.00 | dt: 1689.17ms | tok/sec: 9,699 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 24 | total time: 14.25m | eta: 118.6m
step 00546/05000 (10.92%) | loss: 4.876958 | lrm: 1.00 | dt: 1635.58ms | tok/sec: 10,017 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 24 | total time: 14.27m | eta: 118.6m
step 00547/05000 (10.94%) | loss: 4.874567 | lrm: 1.00 | dt: 1670.83ms | tok/sec: 9,805 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 24 | total time: 14.30m | eta: 118.6m
step 00548/05000 (10.96%) | loss: 4.874901 | lrm: 1.00 | dt: 1653.73ms | tok/sec: 9,907 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 25 | total time: 14.33m | eta: 118.6m
step 00549/05000 (10.98%) | loss: 4.877691 | lrm: 1.00 | dt: 1650.00ms | tok/sec: 9,929 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 25 | total time: 14.36m | eta: 118.6m
step 00550/05000 (11.00%) | loss: 4.885866 | lrm: 1.00 | dt: 1646.12ms | tok/sec: 9,953 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 25 | total time: 14.38m | eta: 118.5m
step 00551/05000 (11.02%) | loss: 4.906774 | lrm: 1.00 | dt: 1666.15ms | tok/sec: 9,833 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 25 | total time: 14.41m | eta: 118.5m
step 00552/05000 (11.04%) | loss: 4.898321 | lrm: 1.00 | dt: 1637.59ms | tok/sec: 10,004 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 25 | total time: 14.44m | eta: 118.5m
step 00553/05000 (11.06%) | loss: 4.906621 | lrm: 1.00 | dt: 1678.61ms | tok/sec: 9,760 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 25 | total time: 14.47m | eta: 118.5m
step 00554/05000 (11.08%) | loss: 4.897605 | lrm: 1.00 | dt: 1663.66ms | tok/sec: 9,848 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 25 | total time: 14.49m | eta: 118.5m
step 00555/05000 (11.10%) | loss: 4.891800 | lrm: 1.00 | dt: 1634.77ms | tok/sec: 10,022 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 25 | total time: 14.52m | eta: 118.4m
step 00556/05000 (11.12%) | loss: 4.879922 | lrm: 1.00 | dt: 1657.61ms | tok/sec: 9,884 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 25 | total time: 14.55m | eta: 118.4m
step 00557/05000 (11.14%) | loss: 4.879313 | lrm: 1.00 | dt: 1626.44ms | tok/sec: 10,073 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 25 | total time: 14.58m | eta: 118.4m
step 00558/05000 (11.16%) | loss: 4.883795 | lrm: 1.00 | dt: 1658.05ms | tok/sec: 9,881 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 25 | total time: 14.60m | eta: 118.4m
step 00559/05000 (11.18%) | loss: 4.875627 | lrm: 1.00 | dt: 1688.98ms | tok/sec: 9,700 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 25 | total time: 14.63m | eta: 118.4m
step 00560/05000 (11.20%) | loss: 4.868364 | lrm: 1.00 | dt: 1649.79ms | tok/sec: 9,930 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 25 | total time: 14.66m | eta: 118.3m
step 00561/05000 (11.22%) | loss: 4.869431 | lrm: 1.00 | dt: 1680.08ms | tok/sec: 9,751 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 25 | total time: 14.69m | eta: 118.3m
step 00562/05000 (11.24%) | loss: 4.870519 | lrm: 1.00 | dt: 1684.70ms | tok/sec: 9,725 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 25 | total time: 14.72m | eta: 118.3m
step 00563/05000 (11.26%) | loss: 4.868865 | lrm: 1.00 | dt: 1689.05ms | tok/sec: 9,700 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 25 | total time: 14.74m | eta: 118.3m
step 00564/05000 (11.28%) | loss: 4.883295 | lrm: 1.00 | dt: 1694.39ms | tok/sec: 9,669 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 25 | total time: 14.77m | eta: 118.3m
step 00565/05000 (11.30%) | loss: 4.878025 | lrm: 1.00 | dt: 1677.12ms | tok/sec: 9,769 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 25 | total time: 14.80m | eta: 118.3m
step 00566/05000 (11.32%) | loss: 4.873573 | lrm: 1.00 | dt: 1694.19ms | tok/sec: 9,670 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 25 | total time: 14.83m | eta: 118.3m
step 00567/05000 (11.34%) | loss: 4.867673 | lrm: 1.00 | dt: 1765.18ms | tok/sec: 9,281 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 25 | total time: 14.86m | eta: 118.3m
step 00568/05000 (11.36%) | loss: 4.853822 | lrm: 1.00 | dt: 1723.88ms | tok/sec: 9,504 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 25 | total time: 14.89m | eta: 118.2m
step 00569/05000 (11.38%) | loss: 4.853931 | lrm: 1.00 | dt: 1749.83ms | tok/sec: 9,363 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 25 | total time: 14.92m | eta: 118.2m
step 00570/05000 (11.40%) | loss: 4.851223 | lrm: 1.00 | dt: 1728.60ms | tok/sec: 9,478 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 25 | total time: 14.94m | eta: 118.2m
step 00571/05000 (11.42%) | loss: 4.841429 | lrm: 1.00 | dt: 1760.23ms | tok/sec: 9,307 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 25 | total time: 14.97m | eta: 118.2m
step 00572/05000 (11.44%) | loss: 4.834403 | lrm: 1.00 | dt: 1686.66ms | tok/sec: 9,713 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 26 | total time: 15.00m | eta: 118.2m
step 00573/05000 (11.46%) | loss: 4.832754 | lrm: 1.00 | dt: 1695.93ms | tok/sec: 9,660 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 26 | total time: 15.03m | eta: 118.2m
step 00574/05000 (11.48%) | loss: 4.829320 | lrm: 1.00 | dt: 1723.62ms | tok/sec: 9,505 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 26 | total time: 15.06m | eta: 118.2m
step 00575/05000 (11.50%) | loss: 4.824542 | lrm: 1.00 | dt: 1761.42ms | tok/sec: 9,301 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 26 | total time: 15.09m | eta: 118.2m
step 00576/05000 (11.52%) | loss: 4.825003 | lrm: 1.00 | dt: 1697.41ms | tok/sec: 9,652 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 26 | total time: 15.12m | eta: 118.2m
step 00577/05000 (11.54%) | loss: 4.811555 | lrm: 1.00 | dt: 1697.55ms | tok/sec: 9,651 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 26 | total time: 15.15m | eta: 118.1m
step 00578/05000 (11.56%) | loss: 4.807698 | lrm: 1.00 | dt: 1631.03ms | tok/sec: 10,045 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 26 | total time: 15.17m | eta: 118.1m
step 00579/05000 (11.58%) | loss: 4.805619 | lrm: 1.00 | dt: 1666.87ms | tok/sec: 9,829 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 26 | total time: 15.20m | eta: 118.1m
step 00580/05000 (11.60%) | loss: 4.803307 | lrm: 1.00 | dt: 1729.29ms | tok/sec: 9,474 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 26 | total time: 15.23m | eta: 118.1m
step 00581/05000 (11.62%) | loss: 4.790114 | lrm: 1.00 | dt: 1748.29ms | tok/sec: 9,371 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 26 | total time: 15.26m | eta: 118.1m
step 00582/05000 (11.64%) | loss: 4.794996 | lrm: 1.00 | dt: 1737.82ms | tok/sec: 9,427 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 26 | total time: 15.29m | eta: 118.1m
step 00583/05000 (11.66%) | loss: 4.804350 | lrm: 1.00 | dt: 1761.57ms | tok/sec: 9,300 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 26 | total time: 15.32m | eta: 118.1m
step 00584/05000 (11.68%) | loss: 4.813474 | lrm: 1.00 | dt: 1734.49ms | tok/sec: 9,445 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 26 | total time: 15.35m | eta: 118.1m
step 00585/05000 (11.70%) | loss: 4.815761 | lrm: 1.00 | dt: 1678.04ms | tok/sec: 9,763 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 26 | total time: 15.37m | eta: 118.0m
step 00586/05000 (11.72%) | loss: 4.820310 | lrm: 1.00 | dt: 1668.20ms | tok/sec: 9,821 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 26 | total time: 15.40m | eta: 118.0m
step 00587/05000 (11.74%) | loss: 4.824628 | lrm: 1.00 | dt: 1656.74ms | tok/sec: 9,889 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 26 | total time: 15.43m | eta: 118.0m
step 00588/05000 (11.76%) | loss: 4.831829 | lrm: 1.00 | dt: 1651.39ms | tok/sec: 9,921 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 26 | total time: 15.46m | eta: 118.0m
step 00589/05000 (11.78%) | loss: 4.817143 | lrm: 1.00 | dt: 1673.54ms | tok/sec: 9,790 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 26 | total time: 15.48m | eta: 118.0m
step 00590/05000 (11.80%) | loss: 4.823867 | lrm: 1.00 | dt: 1642.96ms | tok/sec: 9,972 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 26 | total time: 15.51m | eta: 117.9m
step 00591/05000 (11.82%) | loss: 4.818290 | lrm: 1.00 | dt: 1665.84ms | tok/sec: 9,835 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 26 | total time: 15.54m | eta: 117.9m
step 00592/05000 (11.84%) | loss: 4.839511 | lrm: 1.00 | dt: 1698.89ms | tok/sec: 9,643 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 26 | total time: 15.57m | eta: 117.9m
step 00593/05000 (11.86%) | loss: 4.838007 | lrm: 1.00 | dt: 1776.61ms | tok/sec: 9,222 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 26 | total time: 15.60m | eta: 117.9m
step 00594/05000 (11.88%) | loss: 4.828703 | lrm: 1.00 | dt: 1714.71ms | tok/sec: 9,554 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 26 | total time: 15.63m | eta: 117.9m
step 00595/05000 (11.90%) | loss: 4.832279 | lrm: 1.00 | dt: 1768.01ms | tok/sec: 9,266 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 27 | total time: 15.66m | eta: 117.9m
step 00596/05000 (11.92%) | loss: 4.822536 | lrm: 1.00 | dt: 1700.77ms | tok/sec: 9,633 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 27 | total time: 15.68m | eta: 117.9m
step 00597/05000 (11.94%) | loss: 4.832865 | lrm: 1.00 | dt: 1742.36ms | tok/sec: 9,403 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 27 | total time: 15.71m | eta: 117.9m
step 00598/05000 (11.96%) | loss: 4.836873 | lrm: 1.00 | dt: 1747.86ms | tok/sec: 9,373 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 27 | total time: 15.74m | eta: 117.8m
step 00599/05000 (11.98%) | loss: 4.828687 | lrm: 1.00 | dt: 1632.99ms | tok/sec: 10,033 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 27 | total time: 15.77m | eta: 117.8m
Step 00600 | Validation bpb: 1.478824
<|bos|>The capital of France is the largest of the world's most popular countries. The country's country is the
<|bos|>The chemical symbol of gold is a symbol of the gold of gold, and the gold of gold is the symbol
<|bos|>If yesterday was Friday, then tomorrow will be a week in the next week, and will be a week in the next week
<|bos|>The opposite of hot is a very common cold. The cold is a cold, and the cold is cold
<|bos|>The planets of the solar system are: the solar system of the solar system, the solar system is the most abundant,
<|bos|>My favorite color is a bit more than a bit of a bit of a bit. I think it
<|bos|>If 5*x + 3 = 13, then x is 1 = 1 = 1 = 1 = 1 =
step 00600/05000 (12.00%) | loss: 4.823979 | lrm: 1.00 | dt: 1711.52ms | tok/sec: 9,572 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 27 | total time: 15.80m | eta: 117.8m
step 00601/05000 (12.02%) | loss: 4.804806 | lrm: 1.00 | dt: 1781.58ms | tok/sec: 9,196 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 27 | total time: 15.83m | eta: 117.8m
step 00602/05000 (12.04%) | loss: 4.803033 | lrm: 1.00 | dt: 1632.12ms | tok/sec: 10,038 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 27 | total time: 15.85m | eta: 117.8m
step 00603/05000 (12.06%) | loss: 4.791107 | lrm: 1.00 | dt: 1834.34ms | tok/sec: 8,931 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 27 | total time: 15.89m | eta: 117.8m
step 00604/05000 (12.08%) | loss: 4.803603 | lrm: 1.00 | dt: 1594.36ms | tok/sec: 10,276 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 27 | total time: 15.91m | eta: 117.8m
step 00605/05000 (12.10%) | loss: 4.805458 | lrm: 1.00 | dt: 1856.38ms | tok/sec: 8,825 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 27 | total time: 15.94m | eta: 117.8m
step 00606/05000 (12.12%) | loss: 4.802805 | lrm: 1.00 | dt: 1646.59ms | tok/sec: 9,950 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 27 | total time: 15.97m | eta: 117.7m
step 00607/05000 (12.14%) | loss: 4.799197 | lrm: 1.00 | dt: 1661.94ms | tok/sec: 9,858 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 27 | total time: 16.00m | eta: 117.7m
step 00608/05000 (12.16%) | loss: 4.796039 | lrm: 1.00 | dt: 1706.14ms | tok/sec: 9,602 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 27 | total time: 16.03m | eta: 117.7m
step 00609/05000 (12.18%) | loss: 4.797841 | lrm: 1.00 | dt: 1672.18ms | tok/sec: 9,797 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 27 | total time: 16.05m | eta: 117.7m
step 00610/05000 (12.20%) | loss: 4.803635 | lrm: 1.00 | dt: 1746.55ms | tok/sec: 9,380 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 27 | total time: 16.08m | eta: 117.7m
step 00611/05000 (12.22%) | loss: 4.808378 | lrm: 1.00 | dt: 1595.60ms | tok/sec: 10,268 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 27 | total time: 16.11m | eta: 117.6m
step 00612/05000 (12.24%) | loss: 4.802361 | lrm: 1.00 | dt: 1612.98ms | tok/sec: 10,157 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 27 | total time: 16.14m | eta: 117.6m
step 00613/05000 (12.26%) | loss: 4.807316 | lrm: 1.00 | dt: 1671.42ms | tok/sec: 9,802 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 27 | total time: 16.16m | eta: 117.6m
step 00614/05000 (12.28%) | loss: 4.801516 | lrm: 1.00 | dt: 1653.59ms | tok/sec: 9,908 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 27 | total time: 16.19m | eta: 117.6m
step 00615/05000 (12.30%) | loss: 4.781642 | lrm: 1.00 | dt: 1660.46ms | tok/sec: 9,867 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 27 | total time: 16.22m | eta: 117.6m
step 00616/05000 (12.32%) | loss: 4.779086 | lrm: 1.00 | dt: 1842.30ms | tok/sec: 8,893 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 27 | total time: 16.25m | eta: 117.6m
step 00617/05000 (12.34%) | loss: 4.787783 | lrm: 1.00 | dt: 1534.50ms | tok/sec: 10,677 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 27 | total time: 16.28m | eta: 117.5m
step 00618/05000 (12.36%) | loss: 4.761959 | lrm: 1.00 | dt: 1559.91ms | tok/sec: 10,503 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 28 | total time: 16.30m | eta: 117.5m
step 00619/05000 (12.38%) | loss: 4.761951 | lrm: 1.00 | dt: 1595.05ms | tok/sec: 10,271 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 28 | total time: 16.33m | eta: 117.5m
step 00620/05000 (12.40%) | loss: 4.761780 | lrm: 1.00 | dt: 1587.97ms | tok/sec: 10,317 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 28 | total time: 16.36m | eta: 117.4m
step 00621/05000 (12.42%) | loss: 4.755931 | lrm: 1.00 | dt: 1574.49ms | tok/sec: 10,405 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 28 | total time: 16.38m | eta: 117.4m
step 00622/05000 (12.44%) | loss: 4.752261 | lrm: 1.00 | dt: 1565.08ms | tok/sec: 10,468 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 28 | total time: 16.41m | eta: 117.4m
step 00623/05000 (12.46%) | loss: 4.753412 | lrm: 1.00 | dt: 1514.01ms | tok/sec: 10,821 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 28 | total time: 16.43m | eta: 117.3m
step 00624/05000 (12.48%) | loss: 4.754706 | lrm: 1.00 | dt: 1573.38ms | tok/sec: 10,413 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 28 | total time: 16.46m | eta: 117.3m
step 00625/05000 (12.50%) | loss: 4.749621 | lrm: 1.00 | dt: 1607.02ms | tok/sec: 10,195 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 28 | total time: 16.49m | eta: 117.3m
step 00626/05000 (12.52%) | loss: 4.750358 | lrm: 1.00 | dt: 1645.48ms | tok/sec: 9,956 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 28 | total time: 16.51m | eta: 117.3m
step 00627/05000 (12.54%) | loss: 4.753597 | lrm: 1.00 | dt: 1580.59ms | tok/sec: 10,365 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 28 | total time: 16.54m | eta: 117.2m
step 00628/05000 (12.56%) | loss: 4.754582 | lrm: 1.00 | dt: 1601.22ms | tok/sec: 10,232 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 28 | total time: 16.57m | eta: 117.2m
step 00629/05000 (12.58%) | loss: 4.760643 | lrm: 1.00 | dt: 1558.40ms | tok/sec: 10,513 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 28 | total time: 16.59m | eta: 117.2m
step 00630/05000 (12.60%) | loss: 4.765540 | lrm: 1.00 | dt: 1567.95ms | tok/sec: 10,449 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 28 | total time: 16.62m | eta: 117.1m
step 00631/05000 (12.62%) | loss: 4.772974 | lrm: 1.00 | dt: 1564.23ms | tok/sec: 10,474 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 28 | total time: 16.64m | eta: 117.1m
step 00632/05000 (12.64%) | loss: 4.764733 | lrm: 1.00 | dt: 1548.03ms | tok/sec: 10,583 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 28 | total time: 16.67m | eta: 117.1m
step 00633/05000 (12.66%) | loss: 4.780094 | lrm: 1.00 | dt: 1570.23ms | tok/sec: 10,434 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 28 | total time: 16.70m | eta: 117.0m
step 00634/05000 (12.68%) | loss: 4.777510 | lrm: 1.00 | dt: 1583.57ms | tok/sec: 10,346 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 28 | total time: 16.72m | eta: 117.0m
step 00635/05000 (12.70%) | loss: 4.770446 | lrm: 1.00 | dt: 1580.46ms | tok/sec: 10,366 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 28 | total time: 16.75m | eta: 117.0m
step 00636/05000 (12.72%) | loss: 4.755867 | lrm: 1.00 | dt: 1561.65ms | tok/sec: 10,491 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 28 | total time: 16.77m | eta: 116.9m
step 00637/05000 (12.74%) | loss: 4.746921 | lrm: 1.00 | dt: 1544.42ms | tok/sec: 10,608 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 28 | total time: 16.80m | eta: 116.9m
step 00638/05000 (12.76%) | loss: 4.735290 | lrm: 1.00 | dt: 1550.53ms | tok/sec: 10,566 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 28 | total time: 16.83m | eta: 116.9m
step 00639/05000 (12.78%) | loss: 4.734638 | lrm: 1.00 | dt: 1556.73ms | tok/sec: 10,524 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 28 | total time: 16.85m | eta: 116.8m
step 00640/05000 (12.80%) | loss: 4.748764 | lrm: 1.00 | dt: 1566.73ms | tok/sec: 10,457 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 28 | total time: 16.88m | eta: 116.8m
step 00641/05000 (12.82%) | loss: 4.738111 | lrm: 1.00 | dt: 1612.18ms | tok/sec: 10,162 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 29 | total time: 16.91m | eta: 116.8m
step 00642/05000 (12.84%) | loss: 4.723390 | lrm: 1.00 | dt: 1626.79ms | tok/sec: 10,071 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 29 | total time: 16.93m | eta: 116.8m
step 00643/05000 (12.86%) | loss: 4.724755 | lrm: 1.00 | dt: 1689.79ms | tok/sec: 9,695 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 29 | total time: 16.96m | eta: 116.7m
step 00644/05000 (12.88%) | loss: 4.728417 | lrm: 1.00 | dt: 1552.50ms | tok/sec: 10,553 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 29 | total time: 16.99m | eta: 116.7m
step 00645/05000 (12.90%) | loss: 4.733562 | lrm: 1.00 | dt: 1566.49ms | tok/sec: 10,459 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 29 | total time: 17.01m | eta: 116.7m
step 00646/05000 (12.92%) | loss: 4.742740 | lrm: 1.00 | dt: 1657.81ms | tok/sec: 9,882 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 29 | total time: 17.04m | eta: 116.7m
step 00647/05000 (12.94%) | loss: 4.733669 | lrm: 1.00 | dt: 1625.22ms | tok/sec: 10,081 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 29 | total time: 17.07m | eta: 116.6m
step 00648/05000 (12.96%) | loss: 4.743961 | lrm: 1.00 | dt: 1601.26ms | tok/sec: 10,231 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 29 | total time: 17.09m | eta: 116.6m
step 00649/05000 (12.98%) | loss: 4.734490 | lrm: 1.00 | dt: 1551.31ms | tok/sec: 10,561 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 29 | total time: 17.12m | eta: 116.6m
step 00650/05000 (13.00%) | loss: 4.733596 | lrm: 1.00 | dt: 1563.80ms | tok/sec: 10,477 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 29 | total time: 17.15m | eta: 116.5m
step 00651/05000 (13.02%) | loss: 4.716583 | lrm: 1.00 | dt: 1637.91ms | tok/sec: 10,003 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 29 | total time: 17.17m | eta: 116.5m
step 00652/05000 (13.04%) | loss: 4.718582 | lrm: 1.00 | dt: 1605.26ms | tok/sec: 10,206 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 29 | total time: 17.20m | eta: 116.5m
step 00653/05000 (13.06%) | loss: 4.712066 | lrm: 1.00 | dt: 1642.33ms | tok/sec: 9,976 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 29 | total time: 17.23m | eta: 116.5m
step 00654/05000 (13.08%) | loss: 4.719608 | lrm: 1.00 | dt: 1576.28ms | tok/sec: 10,394 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 29 | total time: 17.25m | eta: 116.4m
step 00655/05000 (13.10%) | loss: 4.716257 | lrm: 1.00 | dt: 1652.07ms | tok/sec: 9,917 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 29 | total time: 17.28m | eta: 116.4m
step 00656/05000 (13.12%) | loss: 4.713249 | lrm: 1.00 | dt: 1683.30ms | tok/sec: 9,733 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 29 | total time: 17.31m | eta: 116.4m
step 00657/05000 (13.14%) | loss: 4.708714 | lrm: 1.00 | dt: 1631.23ms | tok/sec: 10,043 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 29 | total time: 17.34m | eta: 116.4m
step 00658/05000 (13.16%) | loss: 4.704379 | lrm: 1.00 | dt: 1831.24ms | tok/sec: 8,946 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 29 | total time: 17.37m | eta: 116.4m
step 00659/05000 (13.18%) | loss: 4.683182 | lrm: 1.00 | dt: 1916.48ms | tok/sec: 8,549 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 29 | total time: 17.40m | eta: 116.4m
step 00660/05000 (13.20%) | loss: 4.678739 | lrm: 1.00 | dt: 1650.52ms | tok/sec: 9,926 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 29 | total time: 17.43m | eta: 116.4m
step 00661/05000 (13.22%) | loss: 4.680138 | lrm: 1.00 | dt: 1639.09ms | tok/sec: 9,995 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 29 | total time: 17.45m | eta: 116.3m
step 00662/05000 (13.24%) | loss: 4.681672 | lrm: 1.00 | dt: 1679.88ms | tok/sec: 9,753 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 29 | total time: 17.48m | eta: 116.3m
step 00663/05000 (13.26%) | loss: 4.671904 | lrm: 1.00 | dt: 1651.01ms | tok/sec: 9,923 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 29 | total time: 17.51m | eta: 116.3m
step 00664/05000 (13.28%) | loss: 4.683367 | lrm: 1.00 | dt: 1615.20ms | tok/sec: 10,143 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 29 | total time: 17.54m | eta: 116.3m
step 00665/05000 (13.30%) | loss: 4.679605 | lrm: 1.00 | dt: 1631.39ms | tok/sec: 10,042 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 30 | total time: 17.56m | eta: 116.2m
step 00666/05000 (13.32%) | loss: 4.697346 | lrm: 1.00 | dt: 1640.84ms | tok/sec: 9,985 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 30 | total time: 17.59m | eta: 116.2m
step 00667/05000 (13.34%) | loss: 4.688920 | lrm: 1.00 | dt: 1640.89ms | tok/sec: 9,984 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 30 | total time: 17.62m | eta: 116.2m
step 00668/05000 (13.36%) | loss: 4.688351 | lrm: 1.00 | dt: 1573.98ms | tok/sec: 10,409 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 30 | total time: 17.64m | eta: 116.2m
step 00669/05000 (13.38%) | loss: 4.686423 | lrm: 1.00 | dt: 1679.73ms | tok/sec: 9,753 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 30 | total time: 17.67m | eta: 116.1m
step 00670/05000 (13.40%) | loss: 4.683746 | lrm: 1.00 | dt: 1666.54ms | tok/sec: 9,831 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 30 | total time: 17.70m | eta: 116.1m
step 00671/05000 (13.42%) | loss: 4.682674 | lrm: 1.00 | dt: 1628.91ms | tok/sec: 10,058 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 30 | total time: 17.73m | eta: 116.1m
step 00672/05000 (13.44%) | loss: 4.691722 | lrm: 1.00 | dt: 1641.34ms | tok/sec: 9,982 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 30 | total time: 17.75m | eta: 116.1m
step 00673/05000 (13.46%) | loss: 4.704726 | lrm: 1.00 | dt: 1713.68ms | tok/sec: 9,560 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 30 | total time: 17.78m | eta: 116.1m
step 00674/05000 (13.48%) | loss: 4.705643 | lrm: 1.00 | dt: 1630.45ms | tok/sec: 10,048 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 30 | total time: 17.81m | eta: 116.0m
step 00675/05000 (13.50%) | loss: 4.698189 | lrm: 1.00 | dt: 1579.06ms | tok/sec: 10,375 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 30 | total time: 17.84m | eta: 116.0m
step 00676/05000 (13.52%) | loss: 4.694763 | lrm: 1.00 | dt: 1569.38ms | tok/sec: 10,439 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 30 | total time: 17.86m | eta: 116.0m
step 00677/05000 (13.54%) | loss: 4.703373 | lrm: 1.00 | dt: 1553.25ms | tok/sec: 10,548 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 30 | total time: 17.89m | eta: 115.9m
step 00678/05000 (13.56%) | loss: 4.713208 | lrm: 1.00 | dt: 1606.51ms | tok/sec: 10,198 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 30 | total time: 17.92m | eta: 115.9m
step 00679/05000 (13.58%) | loss: 4.689981 | lrm: 1.00 | dt: 1569.76ms | tok/sec: 10,437 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 30 | total time: 17.94m | eta: 115.9m
step 00680/05000 (13.60%) | loss: 4.682552 | lrm: 1.00 | dt: 1575.63ms | tok/sec: 10,398 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 30 | total time: 17.97m | eta: 115.9m
step 00681/05000 (13.62%) | loss: 4.684155 | lrm: 1.00 | dt: 1556.26ms | tok/sec: 10,527 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 30 | total time: 17.99m | eta: 115.8m
step 00682/05000 (13.64%) | loss: 4.663156 | lrm: 1.00 | dt: 1571.31ms | tok/sec: 10,426 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 30 | total time: 18.02m | eta: 115.8m
step 00683/05000 (13.66%) | loss: 4.679922 | lrm: 1.00 | dt: 1592.17ms | tok/sec: 10,290 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 30 | total time: 18.05m | eta: 115.8m
step 00684/05000 (13.68%) | loss: 4.677192 | lrm: 1.00 | dt: 1574.47ms | tok/sec: 10,406 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 30 | total time: 18.07m | eta: 115.7m
step 00685/05000 (13.70%) | loss: 4.668968 | lrm: 1.00 | dt: 1581.05ms | tok/sec: 10,362 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 30 | total time: 18.10m | eta: 115.7m
step 00686/05000 (13.72%) | loss: 4.665887 | lrm: 1.00 | dt: 1580.64ms | tok/sec: 10,365 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 30 | total time: 18.13m | eta: 115.7m
step 00687/05000 (13.74%) | loss: 4.670931 | lrm: 1.00 | dt: 1563.41ms | tok/sec: 10,479 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 30 | total time: 18.15m | eta: 115.6m
step 00688/05000 (13.76%) | loss: 4.672495 | lrm: 1.00 | dt: 1572.55ms | tok/sec: 10,418 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 31 | total time: 18.18m | eta: 115.6m
step 00689/05000 (13.78%) | loss: 4.668472 | lrm: 1.00 | dt: 1606.59ms | tok/sec: 10,197 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 31 | total time: 18.20m | eta: 115.6m
step 00690/05000 (13.80%) | loss: 4.664247 | lrm: 1.00 | dt: 1532.75ms | tok/sec: 10,689 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 31 | total time: 18.23m | eta: 115.5m
step 00691/05000 (13.82%) | loss: 4.680306 | lrm: 1.00 | dt: 1595.79ms | tok/sec: 10,266 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 31 | total time: 18.26m | eta: 115.5m
step 00692/05000 (13.84%) | loss: 4.684585 | lrm: 1.00 | dt: 1543.33ms | tok/sec: 10,615 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 31 | total time: 18.28m | eta: 115.5m
step 00693/05000 (13.86%) | loss: 4.675157 | lrm: 1.00 | dt: 1588.40ms | tok/sec: 10,314 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 31 | total time: 18.31m | eta: 115.5m
step 00694/05000 (13.88%) | loss: 4.675794 | lrm: 1.00 | dt: 1566.05ms | tok/sec: 10,461 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 31 | total time: 18.34m | eta: 115.4m
step 00695/05000 (13.90%) | loss: 4.678276 | lrm: 1.00 | dt: 1549.20ms | tok/sec: 10,575 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 31 | total time: 18.36m | eta: 115.4m
step 00696/05000 (13.92%) | loss: 4.676260 | lrm: 1.00 | dt: 1562.78ms | tok/sec: 10,483 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 31 | total time: 18.39m | eta: 115.4m
step 00697/05000 (13.94%) | loss: 4.675792 | lrm: 1.00 | dt: 1624.77ms | tok/sec: 10,083 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 31 | total time: 18.41m | eta: 115.3m
step 00698/05000 (13.96%) | loss: 4.682544 | lrm: 1.00 | dt: 1589.21ms | tok/sec: 10,309 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 31 | total time: 18.44m | eta: 115.3m
step 00699/05000 (13.98%) | loss: 4.674581 | lrm: 1.00 | dt: 1570.67ms | tok/sec: 10,431 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 31 | total time: 18.47m | eta: 115.3m
Step 00700 | Validation bpb: 1.438952
<|bos|>The capital of France is a very popular and widely used country. It is a popular country where the country
<|bos|>The chemical symbol of gold is a symbol of the value of gold and gold. The symbol of gold is a
<|bos|>If yesterday was Friday, then tomorrow will be a new year, and you will be able to get your new new new new
<|bos|>The opposite of hot is a hot and cold. The cold is cold and cold. The cold is cold
<|bos|>The planets of the solar system are: the solar system of the solar system. The solar system is a solar system that
<|bos|>My favorite color is the color of the color. It is a color, color, and color.
<|bos|>If 5*x + 3 = 13, then x is 3 = 3 = 4 = 3 = 3 =
step 00700/05000 (14.00%) | loss: 4.652661 | lrm: 1.00 | dt: 1650.87ms | tok/sec: 9,924 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 31 | total time: 18.49m | eta: 115.3m
step 00701/05000 (14.02%) | loss: 4.673694 | lrm: 1.00 | dt: 1532.77ms | tok/sec: 10,689 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 31 | total time: 18.52m | eta: 115.2m
step 00702/05000 (14.04%) | loss: 4.669472 | lrm: 1.00 | dt: 1573.47ms | tok/sec: 10,412 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 31 | total time: 18.55m | eta: 115.2m
step 00703/05000 (14.06%) | loss: 4.668616 | lrm: 1.00 | dt: 1584.71ms | tok/sec: 10,338 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 31 | total time: 18.57m | eta: 115.2m
step 00704/05000 (14.08%) | loss: 4.668388 | lrm: 1.00 | dt: 1590.20ms | tok/sec: 10,303 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 31 | total time: 18.60m | eta: 115.1m
step 00705/05000 (14.10%) | loss: 4.665897 | lrm: 1.00 | dt: 1568.32ms | tok/sec: 10,446 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 31 | total time: 18.62m | eta: 115.1m
step 00706/05000 (14.12%) | loss: 4.664133 | lrm: 1.00 | dt: 1554.07ms | tok/sec: 10,542 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 31 | total time: 18.65m | eta: 115.1m
step 00707/05000 (14.14%) | loss: 4.668390 | lrm: 1.00 | dt: 1565.61ms | tok/sec: 10,464 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 31 | total time: 18.68m | eta: 115.0m
step 00708/05000 (14.16%) | loss: 4.665971 | lrm: 1.00 | dt: 1541.50ms | tok/sec: 10,628 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 31 | total time: 18.70m | eta: 115.0m
step 00709/05000 (14.18%) | loss: 4.649499 | lrm: 1.00 | dt: 1598.22ms | tok/sec: 10,251 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 31 | total time: 18.73m | eta: 115.0m
step 00710/05000 (14.20%) | loss: 4.650498 | lrm: 1.00 | dt: 1560.43ms | tok/sec: 10,499 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 31 | total time: 18.76m | eta: 114.9m
step 00711/05000 (14.22%) | loss: 4.653602 | lrm: 1.00 | dt: 1604.50ms | tok/sec: 10,211 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 32 | total time: 18.78m | eta: 114.9m
step 00712/05000 (14.24%) | loss: 4.643780 | lrm: 1.00 | dt: 1572.79ms | tok/sec: 10,417 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 32 | total time: 18.81m | eta: 114.9m
step 00713/05000 (14.26%) | loss: 4.633733 | lrm: 1.00 | dt: 1570.61ms | tok/sec: 10,431 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 32 | total time: 18.83m | eta: 114.9m
step 00714/05000 (14.28%) | loss: 4.636067 | lrm: 1.00 | dt: 1558.14ms | tok/sec: 10,515 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 32 | total time: 18.86m | eta: 114.8m
step 00715/05000 (14.30%) | loss: 4.636061 | lrm: 1.00 | dt: 1559.27ms | tok/sec: 10,507 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 32 | total time: 18.89m | eta: 114.8m
step 00716/05000 (14.32%) | loss: 4.643840 | lrm: 1.00 | dt: 1554.53ms | tok/sec: 10,539 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 32 | total time: 18.91m | eta: 114.8m
step 00717/05000 (14.34%) | loss: 4.649990 | lrm: 1.00 | dt: 1558.56ms | tok/sec: 10,512 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 32 | total time: 18.94m | eta: 114.7m
step 00718/05000 (14.36%) | loss: 4.642071 | lrm: 1.00 | dt: 1606.56ms | tok/sec: 10,198 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 32 | total time: 18.97m | eta: 114.7m
step 00719/05000 (14.38%) | loss: 4.647815 | lrm: 1.00 | dt: 1568.61ms | tok/sec: 10,444 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 32 | total time: 18.99m | eta: 114.7m
step 00720/05000 (14.40%) | loss: 4.640717 | lrm: 1.00 | dt: 1578.23ms | tok/sec: 10,381 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 32 | total time: 19.02m | eta: 114.6m
step 00721/05000 (14.42%) | loss: 4.632852 | lrm: 1.00 | dt: 1585.60ms | tok/sec: 10,333 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 32 | total time: 19.04m | eta: 114.6m
step 00722/05000 (14.44%) | loss: 4.631716 | lrm: 1.00 | dt: 1588.48ms | tok/sec: 10,314 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 32 | total time: 19.07m | eta: 114.6m
step 00723/05000 (14.46%) | loss: 4.619285 | lrm: 1.00 | dt: 1554.03ms | tok/sec: 10,542 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 32 | total time: 19.10m | eta: 114.6m
step 00724/05000 (14.48%) | loss: 4.621219 | lrm: 1.00 | dt: 1582.19ms | tok/sec: 10,355 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 32 | total time: 19.12m | eta: 114.5m
step 00725/05000 (14.50%) | loss: 4.614802 | lrm: 1.00 | dt: 1600.13ms | tok/sec: 10,239 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 32 | total time: 19.15m | eta: 114.5m
step 00726/05000 (14.52%) | loss: 4.621295 | lrm: 1.00 | dt: 1551.44ms | tok/sec: 10,560 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 32 | total time: 19.18m | eta: 114.5m
step 00727/05000 (14.54%) | loss: 4.628454 | lrm: 1.00 | dt: 1576.80ms | tok/sec: 10,390 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 32 | total time: 19.20m | eta: 114.4m
step 00728/05000 (14.56%) | loss: 4.634373 | lrm: 1.00 | dt: 1549.47ms | tok/sec: 10,573 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 32 | total time: 19.23m | eta: 114.4m
step 00729/05000 (14.58%) | loss: 4.632843 | lrm: 1.00 | dt: 1555.56ms | tok/sec: 10,532 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 32 | total time: 19.25m | eta: 114.4m
step 00730/05000 (14.60%) | loss: 4.630248 | lrm: 1.00 | dt: 1561.69ms | tok/sec: 10,491 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 32 | total time: 19.28m | eta: 114.3m
step 00731/05000 (14.62%) | loss: 4.627596 | lrm: 1.00 | dt: 1566.77ms | tok/sec: 10,457 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 32 | total time: 19.31m | eta: 114.3m
step 00732/05000 (14.64%) | loss: 4.626248 | lrm: 1.00 | dt: 1571.72ms | tok/sec: 10,424 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 32 | total time: 19.33m | eta: 114.3m
step 00733/05000 (14.66%) | loss: 4.627018 | lrm: 1.00 | dt: 1591.76ms | tok/sec: 10,293 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 32 | total time: 19.36m | eta: 114.2m
step 00734/05000 (14.68%) | loss: 4.612450 | lrm: 1.00 | dt: 1580.38ms | tok/sec: 10,367 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 33 | total time: 19.38m | eta: 114.2m
step 00735/05000 (14.70%) | loss: 4.613582 | lrm: 1.00 | dt: 1579.10ms | tok/sec: 10,375 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 33 | total time: 19.41m | eta: 114.2m
step 00736/05000 (14.72%) | loss: 4.619878 | lrm: 1.00 | dt: 1550.63ms | tok/sec: 10,566 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 33 | total time: 19.44m | eta: 114.2m
step 00737/05000 (14.74%) | loss: 4.606440 | lrm: 1.00 | dt: 1565.35ms | tok/sec: 10,466 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 33 | total time: 19.46m | eta: 114.1m
step 00738/05000 (14.76%) | loss: 4.607200 | lrm: 1.00 | dt: 1589.79ms | tok/sec: 10,305 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 33 | total time: 19.49m | eta: 114.1m
step 00739/05000 (14.78%) | loss: 4.612818 | lrm: 1.00 | dt: 1581.30ms | tok/sec: 10,361 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 33 | total time: 19.52m | eta: 114.1m
step 00740/05000 (14.80%) | loss: 4.602916 | lrm: 1.00 | dt: 1591.33ms | tok/sec: 10,295 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 33 | total time: 19.54m | eta: 114.0m
step 00741/05000 (14.82%) | loss: 4.613979 | lrm: 1.00 | dt: 1550.64ms | tok/sec: 10,565 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 33 | total time: 19.57m | eta: 114.0m
step 00742/05000 (14.84%) | loss: 4.616226 | lrm: 1.00 | dt: 1565.76ms | tok/sec: 10,463 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 33 | total time: 19.59m | eta: 114.0m
step 00743/05000 (14.86%) | loss: 4.618436 | lrm: 1.00 | dt: 1580.24ms | tok/sec: 10,368 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 33 | total time: 19.62m | eta: 113.9m
step 00744/05000 (14.88%) | loss: 4.627637 | lrm: 1.00 | dt: 1541.13ms | tok/sec: 10,631 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 33 | total time: 19.65m | eta: 113.9m
step 00745/05000 (14.90%) | loss: 4.622462 | lrm: 1.00 | dt: 1575.80ms | tok/sec: 10,397 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 33 | total time: 19.67m | eta: 113.9m
step 00746/05000 (14.92%) | loss: 4.610821 | lrm: 1.00 | dt: 1589.40ms | tok/sec: 10,308 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 33 | total time: 19.70m | eta: 113.9m
step 00747/05000 (14.94%) | loss: 4.606279 | lrm: 1.00 | dt: 1573.31ms | tok/sec: 10,413 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 33 | total time: 19.72m | eta: 113.8m
step 00748/05000 (14.96%) | loss: 4.603318 | lrm: 1.00 | dt: 1572.19ms | tok/sec: 10,421 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 33 | total time: 19.75m | eta: 113.8m
step 00749/05000 (14.98%) | loss: 4.607471 | lrm: 1.00 | dt: 1578.15ms | tok/sec: 10,381 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 33 | total time: 19.78m | eta: 113.8m
step 00750/05000 (15.00%) | loss: 4.616681 | lrm: 1.00 | dt: 1571.58ms | tok/sec: 10,425 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 33 | total time: 19.80m | eta: 113.7m
step 00751/05000 (15.02%) | loss: 4.611172 | lrm: 1.00 | dt: 1611.35ms | tok/sec: 10,167 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 33 | total time: 19.83m | eta: 113.7m
step 00752/05000 (15.04%) | loss: 4.609200 | lrm: 1.00 | dt: 1628.18ms | tok/sec: 10,062 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 33 | total time: 19.86m | eta: 113.7m
step 00753/05000 (15.06%) | loss: 4.605717 | lrm: 1.00 | dt: 1594.04ms | tok/sec: 10,278 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 33 | total time: 19.88m | eta: 113.7m
step 00754/05000 (15.08%) | loss: 4.602000 | lrm: 1.00 | dt: 1562.53ms | tok/sec: 10,485 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 33 | total time: 19.91m | eta: 113.6m
step 00755/05000 (15.10%) | loss: 4.589204 | lrm: 1.00 | dt: 1551.98ms | tok/sec: 10,556 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 33 | total time: 19.94m | eta: 113.6m
step 00756/05000 (15.12%) | loss: 4.611078 | lrm: 1.00 | dt: 1574.72ms | tok/sec: 10,404 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 33 | total time: 19.96m | eta: 113.6m
step 00757/05000 (15.14%) | loss: 4.596667 | lrm: 1.00 | dt: 1584.98ms | tok/sec: 10,337 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 34 | total time: 19.99m | eta: 113.5m
step 00758/05000 (15.16%) | loss: 4.594963 | lrm: 1.00 | dt: 1591.11ms | tok/sec: 10,297 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 34 | total time: 20.02m | eta: 113.5m
step 00759/05000 (15.18%) | loss: 4.585065 | lrm: 1.00 | dt: 1598.50ms | tok/sec: 10,249 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 34 | total time: 20.04m | eta: 113.5m
step 00760/05000 (15.20%) | loss: 4.588870 | lrm: 1.00 | dt: 1622.73ms | tok/sec: 10,096 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 34 | total time: 20.07m | eta: 113.5m
step 00761/05000 (15.22%) | loss: 4.599227 | lrm: 1.00 | dt: 1635.79ms | tok/sec: 10,015 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 34 | total time: 20.10m | eta: 113.4m
step 00762/05000 (15.24%) | loss: 4.605298 | lrm: 1.00 | dt: 1593.05ms | tok/sec: 10,284 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 34 | total time: 20.12m | eta: 113.4m
step 00763/05000 (15.26%) | loss: 4.606098 | lrm: 1.00 | dt: 1671.67ms | tok/sec: 9,801 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 34 | total time: 20.15m | eta: 113.4m
step 00764/05000 (15.28%) | loss: 4.603905 | lrm: 1.00 | dt: 1575.69ms | tok/sec: 10,397 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 34 | total time: 20.18m | eta: 113.4m
step 00765/05000 (15.30%) | loss: 4.591936 | lrm: 1.00 | dt: 1622.82ms | tok/sec: 10,096 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 34 | total time: 20.20m | eta: 113.3m
step 00766/05000 (15.32%) | loss: 4.584822 | lrm: 1.00 | dt: 1679.71ms | tok/sec: 9,754 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 34 | total time: 20.23m | eta: 113.3m
step 00767/05000 (15.34%) | loss: 4.572929 | lrm: 1.00 | dt: 1653.21ms | tok/sec: 9,910 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 34 | total time: 20.26m | eta: 113.3m
step 00768/05000 (15.36%) | loss: 4.576004 | lrm: 1.00 | dt: 1685.36ms | tok/sec: 9,721 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 34 | total time: 20.29m | eta: 113.3m
step 00769/05000 (15.38%) | loss: 4.589318 | lrm: 1.00 | dt: 1630.44ms | tok/sec: 10,048 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 34 | total time: 20.31m | eta: 113.2m
step 00770/05000 (15.40%) | loss: 4.569685 | lrm: 1.00 | dt: 1594.36ms | tok/sec: 10,276 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 34 | total time: 20.34m | eta: 113.2m
step 00771/05000 (15.42%) | loss: 4.565818 | lrm: 1.00 | dt: 1634.79ms | tok/sec: 10,022 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 34 | total time: 20.37m | eta: 113.2m
step 00772/05000 (15.44%) | loss: 4.569205 | lrm: 1.00 | dt: 1532.93ms | tok/sec: 10,688 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 34 | total time: 20.39m | eta: 113.2m
step 00773/05000 (15.46%) | loss: 4.570749 | lrm: 1.00 | dt: 1574.89ms | tok/sec: 10,403 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 34 | total time: 20.42m | eta: 113.1m
step 00774/05000 (15.48%) | loss: 4.571912 | lrm: 1.00 | dt: 1570.05ms | tok/sec: 10,435 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 34 | total time: 20.45m | eta: 113.1m
step 00775/05000 (15.50%) | loss: 4.572664 | lrm: 1.00 | dt: 1538.70ms | tok/sec: 10,647 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 34 | total time: 20.47m | eta: 113.1m
step 00776/05000 (15.52%) | loss: 4.580788 | lrm: 1.00 | dt: 1650.05ms | tok/sec: 9,929 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 34 | total time: 20.50m | eta: 113.0m
step 00777/05000 (15.54%) | loss: 4.575356 | lrm: 1.00 | dt: 1641.68ms | tok/sec: 9,980 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 34 | total time: 20.53m | eta: 113.0m
step 00778/05000 (15.56%) | loss: 4.570386 | lrm: 1.00 | dt: 1614.75ms | tok/sec: 10,146 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 34 | total time: 20.55m | eta: 113.0m
step 00779/05000 (15.58%) | loss: 4.574854 | lrm: 1.00 | dt: 1629.68ms | tok/sec: 10,053 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 34 | total time: 20.58m | eta: 113.0m
step 00780/05000 (15.60%) | loss: 4.565452 | lrm: 1.00 | dt: 1630.30ms | tok/sec: 10,049 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 34 | total time: 20.61m | eta: 112.9m
step 00781/05000 (15.62%) | loss: 4.560358 | lrm: 1.00 | dt: 1600.46ms | tok/sec: 10,237 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 35 | total time: 20.64m | eta: 112.9m
step 00782/05000 (15.64%) | loss: 4.564773 | lrm: 1.00 | dt: 1611.65ms | tok/sec: 10,165 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 35 | total time: 20.66m | eta: 112.9m
step 00783/05000 (15.66%) | loss: 4.567073 | lrm: 1.00 | dt: 1618.34ms | tok/sec: 10,123 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 35 | total time: 20.69m | eta: 112.9m
step 00784/05000 (15.68%) | loss: 4.567803 | lrm: 1.00 | dt: 1607.24ms | tok/sec: 10,193 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 35 | total time: 20.72m | eta: 112.8m
step 00785/05000 (15.70%) | loss: 4.565484 | lrm: 1.00 | dt: 1598.22ms | tok/sec: 10,251 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 35 | total time: 20.74m | eta: 112.8m
step 00786/05000 (15.72%) | loss: 4.570294 | lrm: 1.00 | dt: 1564.71ms | tok/sec: 10,470 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 35 | total time: 20.77m | eta: 112.8m
step 00787/05000 (15.74%) | loss: 4.565549 | lrm: 1.00 | dt: 1578.08ms | tok/sec: 10,382 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 35 | total time: 20.79m | eta: 112.8m
step 00788/05000 (15.76%) | loss: 4.565256 | lrm: 1.00 | dt: 1559.46ms | tok/sec: 10,506 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 35 | total time: 20.82m | eta: 112.7m
step 00789/05000 (15.78%) | loss: 4.563891 | lrm: 1.00 | dt: 1565.26ms | tok/sec: 10,467 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 35 | total time: 20.85m | eta: 112.7m
step 00790/05000 (15.80%) | loss: 4.554214 | lrm: 1.00 | dt: 1610.45ms | tok/sec: 10,173 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 35 | total time: 20.87m | eta: 112.7m
step 00791/05000 (15.82%) | loss: 4.563933 | lrm: 1.00 | dt: 1608.45ms | tok/sec: 10,186 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 35 | total time: 20.90m | eta: 112.6m
step 00792/05000 (15.84%) | loss: 4.555909 | lrm: 1.00 | dt: 1607.05ms | tok/sec: 10,195 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 35 | total time: 20.93m | eta: 112.6m
step 00793/05000 (15.86%) | loss: 4.556955 | lrm: 1.00 | dt: 1578.82ms | tok/sec: 10,377 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 35 | total time: 20.95m | eta: 112.6m
step 00794/05000 (15.88%) | loss: 4.556871 | lrm: 1.00 | dt: 1587.29ms | tok/sec: 10,322 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 35 | total time: 20.98m | eta: 112.6m
step 00795/05000 (15.90%) | loss: 4.557344 | lrm: 1.00 | dt: 1611.06ms | tok/sec: 10,169 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 35 | total time: 21.01m | eta: 112.5m
step 00796/05000 (15.92%) | loss: 4.559523 | lrm: 1.00 | dt: 1587.00ms | tok/sec: 10,323 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 35 | total time: 21.03m | eta: 112.5m
step 00797/05000 (15.94%) | loss: 4.554977 | lrm: 1.00 | dt: 1617.40ms | tok/sec: 10,129 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 35 | total time: 21.06m | eta: 112.5m
step 00798/05000 (15.96%) | loss: 4.565494 | lrm: 1.00 | dt: 1624.59ms | tok/sec: 10,084 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 35 | total time: 21.09m | eta: 112.4m
step 00799/05000 (15.98%) | loss: 4.578864 | lrm: 1.00 | dt: 1587.39ms | tok/sec: 10,321 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 35 | total time: 21.11m | eta: 112.4m
Step 00800 | Validation bpb: 1.411252
<|bos|>The capital of France is a country of originality, which is a country of originality, which is
<|bos|>The chemical symbol of gold is a symbol of strength and strength. The chemical symbol of strength and strength is a
<|bos|>If yesterday was Friday, then tomorrow will be a new time to start the week, and then again, the time to start
<|bos|>The opposite of hot is true. The most common cold is cold, cold, cold, cold, cold
<|bos|>The planets of the solar system are: the solar system is the solar system. The solar system is the solar system,
<|bos|>My favorite color is a type of color that is a type of color that is a color. The
<|bos|>If 5*x + 3 = 13, then x is 1. 1. 2. 3. 3.
step 00800/05000 (16.00%) | loss: 4.580388 | lrm: 1.00 | dt: 2321.42ms | tok/sec: 7,057 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 35 | total time: 21.15m | eta: 112.5m
step 00801/05000 (16.02%) | loss: 4.574565 | lrm: 1.00 | dt: 1659.63ms | tok/sec: 9,872 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 35 | total time: 21.18m | eta: 112.4m
step 00802/05000 (16.04%) | loss: 4.580489 | lrm: 1.00 | dt: 1565.81ms | tok/sec: 10,463 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 35 | total time: 21.21m | eta: 112.4m
step 00803/05000 (16.06%) | loss: 4.573317 | lrm: 1.00 | dt: 1587.19ms | tok/sec: 10,322 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 35 | total time: 21.23m | eta: 112.4m
step 00804/05000 (16.08%) | loss: 4.581579 | lrm: 1.00 | dt: 1593.53ms | tok/sec: 10,281 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 36 | total time: 21.26m | eta: 112.3m
step 00805/05000 (16.10%) | loss: 4.568244 | lrm: 1.00 | dt: 1642.10ms | tok/sec: 9,977 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 36 | total time: 21.29m | eta: 112.3m
step 00806/05000 (16.12%) | loss: 4.571696 | lrm: 1.00 | dt: 1656.71ms | tok/sec: 9,889 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 36 | total time: 21.31m | eta: 112.3m
step 00807/05000 (16.14%) | loss: 4.571638 | lrm: 1.00 | dt: 1615.74ms | tok/sec: 10,140 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 36 | total time: 21.34m | eta: 112.3m
step 00808/05000 (16.16%) | loss: 4.573259 | lrm: 1.00 | dt: 1569.87ms | tok/sec: 10,436 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 36 | total time: 21.37m | eta: 112.2m
step 00809/05000 (16.18%) | loss: 4.568250 | lrm: 1.00 | dt: 1571.72ms | tok/sec: 10,424 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 36 | total time: 21.39m | eta: 112.2m
step 00810/05000 (16.20%) | loss: 4.561031 | lrm: 1.00 | dt: 1625.36ms | tok/sec: 10,080 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 36 | total time: 21.42m | eta: 112.2m
step 00811/05000 (16.22%) | loss: 4.572170 | lrm: 1.00 | dt: 1627.48ms | tok/sec: 10,067 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 36 | total time: 21.45m | eta: 112.2m
step 00812/05000 (16.24%) | loss: 4.568694 | lrm: 1.00 | dt: 1621.47ms | tok/sec: 10,104 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 36 | total time: 21.47m | eta: 112.1m
step 00813/05000 (16.26%) | loss: 4.569724 | lrm: 1.00 | dt: 1671.21ms | tok/sec: 9,803 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 36 | total time: 21.50m | eta: 112.1m
step 00814/05000 (16.28%) | loss: 4.573468 | lrm: 1.00 | dt: 1573.17ms | tok/sec: 10,414 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 36 | total time: 21.53m | eta: 112.1m
step 00815/05000 (16.30%) | loss: 4.587301 | lrm: 1.00 | dt: 1584.83ms | tok/sec: 10,338 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 36 | total time: 21.56m | eta: 112.1m
step 00816/05000 (16.32%) | loss: 4.571643 | lrm: 1.00 | dt: 1612.66ms | tok/sec: 10,159 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 36 | total time: 21.58m | eta: 112.0m
step 00817/05000 (16.34%) | loss: 4.568167 | lrm: 1.00 | dt: 1570.20ms | tok/sec: 10,434 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 36 | total time: 21.61m | eta: 112.0m
step 00818/05000 (16.36%) | loss: 4.576584 | lrm: 1.00 | dt: 1560.52ms | tok/sec: 10,499 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 36 | total time: 21.63m | eta: 112.0m
step 00819/05000 (16.38%) | loss: 4.582908 | lrm: 1.00 | dt: 1549.26ms | tok/sec: 10,575 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 36 | total time: 21.66m | eta: 111.9m
step 00820/05000 (16.40%) | loss: 4.583492 | lrm: 1.00 | dt: 1562.15ms | tok/sec: 10,488 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 36 | total time: 21.69m | eta: 111.9m
step 00821/05000 (16.42%) | loss: 4.568082 | lrm: 1.00 | dt: 1540.69ms | tok/sec: 10,634 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 36 | total time: 21.71m | eta: 111.9m
step 00822/05000 (16.44%) | loss: 4.558084 | lrm: 1.00 | dt: 1568.48ms | tok/sec: 10,445 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 36 | total time: 21.74m | eta: 111.8m
step 00823/05000 (16.46%) | loss: 4.559773 | lrm: 1.00 | dt: 1538.64ms | tok/sec: 10,648 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 36 | total time: 21.76m | eta: 111.8m
step 00824/05000 (16.48%) | loss: 4.563788 | lrm: 1.00 | dt: 1557.42ms | tok/sec: 10,519 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 36 | total time: 21.79m | eta: 111.8m
step 00825/05000 (16.50%) | loss: 4.571987 | lrm: 1.00 | dt: 1581.82ms | tok/sec: 10,357 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 36 | total time: 21.82m | eta: 111.8m
step 00826/05000 (16.52%) | loss: 4.572630 | lrm: 1.00 | dt: 1594.43ms | tok/sec: 10,275 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 36 | total time: 21.84m | eta: 111.7m
step 00827/05000 (16.54%) | loss: 4.559163 | lrm: 1.00 | dt: 1592.58ms | tok/sec: 10,287 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 37 | total time: 21.87m | eta: 111.7m
step 00828/05000 (16.56%) | loss: 4.552434 | lrm: 1.00 | dt: 1685.25ms | tok/sec: 9,721 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 37 | total time: 21.90m | eta: 111.7m
step 00829/05000 (16.58%) | loss: 4.546507 | lrm: 1.00 | dt: 1555.81ms | tok/sec: 10,530 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 37 | total time: 21.92m | eta: 111.6m
step 00830/05000 (16.60%) | loss: 4.545770 | lrm: 1.00 | dt: 1562.76ms | tok/sec: 10,484 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 37 | total time: 21.95m | eta: 111.6m
step 00831/05000 (16.62%) | loss: 4.544633 | lrm: 1.00 | dt: 1608.02ms | tok/sec: 10,188 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 37 | total time: 21.98m | eta: 111.6m
step 00832/05000 (16.64%) | loss: 4.523327 | lrm: 1.00 | dt: 1612.94ms | tok/sec: 10,157 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 37 | total time: 22.00m | eta: 111.6m
step 00833/05000 (16.66%) | loss: 4.531875 | lrm: 1.00 | dt: 1603.00ms | tok/sec: 10,220 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 37 | total time: 22.03m | eta: 111.5m
step 00834/05000 (16.68%) | loss: 4.519552 | lrm: 1.00 | dt: 1601.06ms | tok/sec: 10,233 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 37 | total time: 22.06m | eta: 111.5m
step 00835/05000 (16.70%) | loss: 4.521001 | lrm: 1.00 | dt: 1579.92ms | tok/sec: 10,370 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 37 | total time: 22.08m | eta: 111.5m
step 00836/05000 (16.72%) | loss: 4.508763 | lrm: 1.00 | dt: 1563.41ms | tok/sec: 10,479 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 37 | total time: 22.11m | eta: 111.5m
step 00837/05000 (16.74%) | loss: 4.493310 | lrm: 1.00 | dt: 1565.49ms | tok/sec: 10,465 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 37 | total time: 22.13m | eta: 111.4m
step 00838/05000 (16.76%) | loss: 4.484567 | lrm: 1.00 | dt: 1606.69ms | tok/sec: 10,197 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 37 | total time: 22.16m | eta: 111.4m
step 00839/05000 (16.78%) | loss: 4.485169 | lrm: 1.00 | dt: 1596.91ms | tok/sec: 10,259 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 37 | total time: 22.19m | eta: 111.4m
step 00840/05000 (16.80%) | loss: 4.466842 | lrm: 1.00 | dt: 1588.37ms | tok/sec: 10,314 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 37 | total time: 22.21m | eta: 111.3m
step 00841/05000 (16.82%) | loss: 4.488652 | lrm: 1.00 | dt: 1596.99ms | tok/sec: 10,259 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 37 | total time: 22.24m | eta: 111.3m
step 00842/05000 (16.84%) | loss: 4.481103 | lrm: 1.00 | dt: 1556.76ms | tok/sec: 10,524 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 37 | total time: 22.27m | eta: 111.3m
step 00843/05000 (16.86%) | loss: 4.498050 | lrm: 1.00 | dt: 1591.40ms | tok/sec: 10,295 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 37 | total time: 22.29m | eta: 111.3m
step 00844/05000 (16.88%) | loss: 4.497544 | lrm: 1.00 | dt: 1575.59ms | tok/sec: 10,398 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 37 | total time: 22.32m | eta: 111.2m
step 00845/05000 (16.90%) | loss: 4.502880 | lrm: 1.00 | dt: 1555.71ms | tok/sec: 10,531 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 37 | total time: 22.35m | eta: 111.2m
step 00846/05000 (16.92%) | loss: 4.519438 | lrm: 1.00 | dt: 1584.06ms | tok/sec: 10,343 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 37 | total time: 22.37m | eta: 111.2m
step 00847/05000 (16.94%) | loss: 4.512396 | lrm: 1.00 | dt: 1547.61ms | tok/sec: 10,586 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 37 | total time: 22.40m | eta: 111.1m
step 00848/05000 (16.96%) | loss: 4.524231 | lrm: 1.00 | dt: 1582.84ms | tok/sec: 10,351 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 37 | total time: 22.42m | eta: 111.1m
step 00849/05000 (16.98%) | loss: 4.523557 | lrm: 1.00 | dt: 1575.26ms | tok/sec: 10,400 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 37 | total time: 22.45m | eta: 111.1m
step 00850/05000 (17.00%) | loss: 4.529269 | lrm: 1.00 | dt: 1603.59ms | tok/sec: 10,217 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 37 | total time: 22.48m | eta: 111.0m
step 00851/05000 (17.02%) | loss: 4.527079 | lrm: 1.00 | dt: 1613.42ms | tok/sec: 10,154 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 38 | total time: 22.50m | eta: 111.0m
step 00852/05000 (17.04%) | loss: 4.524760 | lrm: 1.00 | dt: 1586.76ms | tok/sec: 10,325 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 38 | total time: 22.53m | eta: 111.0m
step 00853/05000 (17.06%) | loss: 4.524528 | lrm: 1.00 | dt: 1590.83ms | tok/sec: 10,299 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 38 | total time: 22.56m | eta: 111.0m
step 00854/05000 (17.08%) | loss: 4.523368 | lrm: 1.00 | dt: 1643.19ms | tok/sec: 9,970 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 38 | total time: 22.58m | eta: 110.9m
step 00855/05000 (17.10%) | loss: 4.518841 | lrm: 1.00 | dt: 1611.81ms | tok/sec: 10,164 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 38 | total time: 22.61m | eta: 110.9m
step 00856/05000 (17.12%) | loss: 4.505224 | lrm: 1.00 | dt: 1592.39ms | tok/sec: 10,288 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 38 | total time: 22.64m | eta: 110.9m
step 00857/05000 (17.14%) | loss: 4.501066 | lrm: 1.00 | dt: 1641.99ms | tok/sec: 9,978 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 38 | total time: 22.67m | eta: 110.9m
step 00858/05000 (17.16%) | loss: 4.497214 | lrm: 1.00 | dt: 1582.06ms | tok/sec: 10,356 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 38 | total time: 22.69m | eta: 110.8m
step 00859/05000 (17.18%) | loss: 4.490352 | lrm: 1.00 | dt: 1560.51ms | tok/sec: 10,499 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 38 | total time: 22.72m | eta: 110.8m
step 00860/05000 (17.20%) | loss: 4.495922 | lrm: 1.00 | dt: 1634.26ms | tok/sec: 10,025 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 38 | total time: 22.74m | eta: 110.8m
step 00861/05000 (17.22%) | loss: 4.515897 | lrm: 1.00 | dt: 1623.62ms | tok/sec: 10,091 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 38 | total time: 22.77m | eta: 110.8m
step 00862/05000 (17.24%) | loss: 4.508312 | lrm: 1.00 | dt: 1606.62ms | tok/sec: 10,197 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 38 | total time: 22.80m | eta: 110.7m
step 00863/05000 (17.26%) | loss: 4.494448 | lrm: 1.00 | dt: 1659.24ms | tok/sec: 9,874 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 38 | total time: 22.83m | eta: 110.7m
step 00864/05000 (17.28%) | loss: 4.505039 | lrm: 1.00 | dt: 1555.83ms | tok/sec: 10,530 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 38 | total time: 22.85m | eta: 110.7m
step 00865/05000 (17.30%) | loss: 4.505402 | lrm: 1.00 | dt: 1549.71ms | tok/sec: 10,572 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 38 | total time: 22.88m | eta: 110.6m
step 00866/05000 (17.32%) | loss: 4.496367 | lrm: 1.00 | dt: 1539.29ms | tok/sec: 10,643 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 38 | total time: 22.90m | eta: 110.6m
step 00867/05000 (17.34%) | loss: 4.511222 | lrm: 1.00 | dt: 1557.56ms | tok/sec: 10,519 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 38 | total time: 22.93m | eta: 110.6m
step 00868/05000 (17.36%) | loss: 4.513799 | lrm: 1.00 | dt: 1608.29ms | tok/sec: 10,187 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 38 | total time: 22.96m | eta: 110.6m
step 00869/05000 (17.38%) | loss: 4.501539 | lrm: 1.00 | dt: 1646.05ms | tok/sec: 9,953 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 38 | total time: 22.98m | eta: 110.5m
step 00870/05000 (17.40%) | loss: 4.499187 | lrm: 1.00 | dt: 1761.20ms | tok/sec: 9,302 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 38 | total time: 23.01m | eta: 110.5m
step 00871/05000 (17.42%) | loss: 4.512411 | lrm: 1.00 | dt: 1603.94ms | tok/sec: 10,214 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 38 | total time: 23.04m | eta: 110.5m
step 00872/05000 (17.44%) | loss: 4.512756 | lrm: 1.00 | dt: 1592.49ms | tok/sec: 10,288 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 38 | total time: 23.07m | eta: 110.5m
step 00873/05000 (17.46%) | loss: 4.512226 | lrm: 1.00 | dt: 1853.68ms | tok/sec: 8,838 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 38 | total time: 23.10m | eta: 110.5m
step 00874/05000 (17.48%) | loss: 4.498362 | lrm: 1.00 | dt: 1642.35ms | tok/sec: 9,975 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 39 | total time: 23.12m | eta: 110.4m
step 00875/05000 (17.50%) | loss: 4.509969 | lrm: 1.00 | dt: 1680.47ms | tok/sec: 9,749 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 39 | total time: 23.15m | eta: 110.4m
step 00876/05000 (17.52%) | loss: 4.506426 | lrm: 1.00 | dt: 1592.33ms | tok/sec: 10,289 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 39 | total time: 23.18m | eta: 110.4m
step 00877/05000 (17.54%) | loss: 4.506857 | lrm: 1.00 | dt: 1582.60ms | tok/sec: 10,352 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 39 | total time: 23.21m | eta: 110.4m
step 00878/05000 (17.56%) | loss: 4.496222 | lrm: 1.00 | dt: 1711.17ms | tok/sec: 9,574 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 39 | total time: 23.23m | eta: 110.3m
step 00879/05000 (17.58%) | loss: 4.498181 | lrm: 1.00 | dt: 1562.72ms | tok/sec: 10,484 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 39 | total time: 23.26m | eta: 110.3m
step 00880/05000 (17.60%) | loss: 4.501121 | lrm: 1.00 | dt: 1599.94ms | tok/sec: 10,240 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 39 | total time: 23.29m | eta: 110.3m
step 00881/05000 (17.62%) | loss: 4.516012 | lrm: 1.00 | dt: 1632.87ms | tok/sec: 10,033 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 39 | total time: 23.31m | eta: 110.3m
step 00882/05000 (17.64%) | loss: 4.519471 | lrm: 1.00 | dt: 1581.10ms | tok/sec: 10,362 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 39 | total time: 23.34m | eta: 110.2m
step 00883/05000 (17.66%) | loss: 4.511841 | lrm: 1.00 | dt: 1616.86ms | tok/sec: 10,133 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 39 | total time: 23.37m | eta: 110.2m
step 00884/05000 (17.68%) | loss: 4.502351 | lrm: 1.00 | dt: 1615.71ms | tok/sec: 10,140 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 39 | total time: 23.39m | eta: 110.2m
step 00885/05000 (17.70%) | loss: 4.496134 | lrm: 1.00 | dt: 1633.68ms | tok/sec: 10,028 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 39 | total time: 23.42m | eta: 110.1m
step 00886/05000 (17.72%) | loss: 4.500853 | lrm: 1.00 | dt: 1605.46ms | tok/sec: 10,205 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 39 | total time: 23.45m | eta: 110.1m
step 00887/05000 (17.74%) | loss: 4.499943 | lrm: 1.00 | dt: 1588.30ms | tok/sec: 10,315 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 39 | total time: 23.47m | eta: 110.1m
step 00888/05000 (17.76%) | loss: 4.491069 | lrm: 1.00 | dt: 1567.21ms | tok/sec: 10,454 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 39 | total time: 23.50m | eta: 110.1m
step 00889/05000 (17.78%) | loss: 4.500552 | lrm: 1.00 | dt: 1588.38ms | tok/sec: 10,314 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 39 | total time: 23.53m | eta: 110.0m
step 00890/05000 (17.80%) | loss: 4.509080 | lrm: 1.00 | dt: 1596.79ms | tok/sec: 10,260 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 39 | total time: 23.55m | eta: 110.0m
step 00891/05000 (17.82%) | loss: 4.505759 | lrm: 1.00 | dt: 1641.05ms | tok/sec: 9,983 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 39 | total time: 23.58m | eta: 110.0m
step 00892/05000 (17.84%) | loss: 4.510357 | lrm: 1.00 | dt: 1683.39ms | tok/sec: 9,732 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 39 | total time: 23.61m | eta: 110.0m
step 00893/05000 (17.86%) | loss: 4.509765 | lrm: 1.00 | dt: 1670.28ms | tok/sec: 9,809 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 39 | total time: 23.64m | eta: 109.9m
step 00894/05000 (17.88%) | loss: 4.515476 | lrm: 1.00 | dt: 1610.24ms | tok/sec: 10,174 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 39 | total time: 23.66m | eta: 109.9m
step 00895/05000 (17.90%) | loss: 4.505910 | lrm: 1.00 | dt: 1588.31ms | tok/sec: 10,315 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 39 | total time: 23.69m | eta: 109.9m
step 00896/05000 (17.92%) | loss: 4.515543 | lrm: 1.00 | dt: 1602.91ms | tok/sec: 10,221 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 39 | total time: 23.72m | eta: 109.9m
step 00897/05000 (17.94%) | loss: 4.513475 | lrm: 1.00 | dt: 1581.74ms | tok/sec: 10,358 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 40 | total time: 23.74m | eta: 109.8m
step 00898/05000 (17.96%) | loss: 4.500135 | lrm: 1.00 | dt: 1657.79ms | tok/sec: 9,883 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 40 | total time: 23.77m | eta: 109.8m
step 00899/05000 (17.98%) | loss: 4.492823 | lrm: 1.00 | dt: 1574.53ms | tok/sec: 10,405 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 40 | total time: 23.80m | eta: 109.8m
Step 00900 | Validation bpb: 1.389210
<|bos|>The capital of France is the largest city in the world. The city is the largest city in the world
<|bos|>The chemical symbol of gold is a chemical symbol of gold and gold. It is a chemical symbol of gold and
<|bos|>If yesterday was Friday, then tomorrow will be the first time to start the day. The first time, the first time,
<|bos|>The opposite of hot is true. The same thing is true. The same thing is true. The same
<|bos|>The planets of the solar system are: the planets, planets, planets, planets, planets, planets, planets, planets
<|bos|>My favorite color is the color of the color of the color of the color of the color of the
<|bos|>If 5*x + 3 = 13, then x is the same as the 5*x + 3 = 3 =
step 00900/05000 (18.00%) | loss: 4.493711 | lrm: 1.00 | dt: 1642.32ms | tok/sec: 9,976 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 40 | total time: 23.83m | eta: 109.8m
step 00901/05000 (18.02%) | loss: 4.495232 | lrm: 1.00 | dt: 1591.26ms | tok/sec: 10,296 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 40 | total time: 23.85m | eta: 109.7m
step 00902/05000 (18.04%) | loss: 4.494038 | lrm: 1.00 | dt: 1613.10ms | tok/sec: 10,156 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 40 | total time: 23.88m | eta: 109.7m
step 00903/05000 (18.06%) | loss: 4.505275 | lrm: 1.00 | dt: 1554.70ms | tok/sec: 10,538 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 40 | total time: 23.90m | eta: 109.7m
step 00904/05000 (18.08%) | loss: 4.499391 | lrm: 1.00 | dt: 1577.58ms | tok/sec: 10,385 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 40 | total time: 23.93m | eta: 109.6m
step 00905/05000 (18.10%) | loss: 4.493774 | lrm: 1.00 | dt: 1578.28ms | tok/sec: 10,380 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 40 | total time: 23.96m | eta: 109.6m
step 00906/05000 (18.12%) | loss: 4.476993 | lrm: 1.00 | dt: 1621.19ms | tok/sec: 10,106 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 40 | total time: 23.98m | eta: 109.6m
step 00907/05000 (18.14%) | loss: 4.480449 | lrm: 1.00 | dt: 1618.82ms | tok/sec: 10,120 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 40 | total time: 24.01m | eta: 109.6m
step 00908/05000 (18.16%) | loss: 4.481853 | lrm: 1.00 | dt: 1609.68ms | tok/sec: 10,178 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 40 | total time: 24.04m | eta: 109.5m
step 00909/05000 (18.18%) | loss: 4.467052 | lrm: 1.00 | dt: 1565.02ms | tok/sec: 10,468 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 40 | total time: 24.06m | eta: 109.5m
step 00910/05000 (18.20%) | loss: 4.473825 | lrm: 1.00 | dt: 1586.98ms | tok/sec: 10,324 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 40 | total time: 24.09m | eta: 109.5m
step 00911/05000 (18.22%) | loss: 4.472921 | lrm: 1.00 | dt: 1614.99ms | tok/sec: 10,144 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 40 | total time: 24.12m | eta: 109.5m
step 00912/05000 (18.24%) | loss: 4.470727 | lrm: 1.00 | dt: 1591.75ms | tok/sec: 10,293 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 40 | total time: 24.14m | eta: 109.4m
step 00913/05000 (18.26%) | loss: 4.460674 | lrm: 1.00 | dt: 1615.59ms | tok/sec: 10,141 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 40 | total time: 24.17m | eta: 109.4m
step 00914/05000 (18.28%) | loss: 4.455225 | lrm: 1.00 | dt: 1606.91ms | tok/sec: 10,195 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 40 | total time: 24.20m | eta: 109.4m
step 00915/05000 (18.30%) | loss: 4.436796 | lrm: 1.00 | dt: 1574.98ms | tok/sec: 10,402 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 40 | total time: 24.22m | eta: 109.3m
step 00916/05000 (18.32%) | loss: 4.455251 | lrm: 1.00 | dt: 1612.53ms | tok/sec: 10,160 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 40 | total time: 24.25m | eta: 109.3m
step 00917/05000 (18.34%) | loss: 4.458975 | lrm: 1.00 | dt: 1774.15ms | tok/sec: 9,234 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 40 | total time: 24.28m | eta: 109.3m
step 00918/05000 (18.36%) | loss: 4.468806 | lrm: 1.00 | dt: 1861.12ms | tok/sec: 8,803 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 40 | total time: 24.31m | eta: 109.3m
step 00919/05000 (18.38%) | loss: 4.474115 | lrm: 1.00 | dt: 1713.91ms | tok/sec: 9,559 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 40 | total time: 24.34m | eta: 109.3m
step 00920/05000 (18.40%) | loss: 4.474206 | lrm: 1.00 | dt: 1681.73ms | tok/sec: 9,742 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 41 | total time: 24.37m | eta: 109.3m
step 00921/05000 (18.42%) | loss: 4.471156 | lrm: 1.00 | dt: 1836.55ms | tok/sec: 8,921 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 41 | total time: 24.40m | eta: 109.2m
step 00922/05000 (18.44%) | loss: 4.469614 | lrm: 1.00 | dt: 1790.60ms | tok/sec: 9,150 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 41 | total time: 24.43m | eta: 109.2m
step 00923/05000 (18.46%) | loss: 4.477626 | lrm: 1.00 | dt: 1582.15ms | tok/sec: 10,355 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 41 | total time: 24.45m | eta: 109.2m
step 00924/05000 (18.48%) | loss: 4.470897 | lrm: 1.00 | dt: 1607.89ms | tok/sec: 10,189 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 41 | total time: 24.48m | eta: 109.2m
step 00925/05000 (18.50%) | loss: 4.459707 | lrm: 1.00 | dt: 1627.41ms | tok/sec: 10,067 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 41 | total time: 24.51m | eta: 109.2m
step 00926/05000 (18.52%) | loss: 4.466975 | lrm: 1.00 | dt: 1614.86ms | tok/sec: 10,145 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 41 | total time: 24.54m | eta: 109.1m
step 00927/05000 (18.54%) | loss: 4.463994 | lrm: 1.00 | dt: 1605.81ms | tok/sec: 10,202 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 41 | total time: 24.56m | eta: 109.1m
step 00928/05000 (18.56%) | loss: 4.456122 | lrm: 1.00 | dt: 1618.09ms | tok/sec: 10,125 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 41 | total time: 24.59m | eta: 109.1m
step 00929/05000 (18.58%) | loss: 4.457654 | lrm: 1.00 | dt: 1569.78ms | tok/sec: 10,437 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 41 | total time: 24.62m | eta: 109.0m
step 00930/05000 (18.60%) | loss: 4.460998 | lrm: 1.00 | dt: 1694.81ms | tok/sec: 9,667 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 41 | total time: 24.64m | eta: 109.0m
step 00931/05000 (18.62%) | loss: 4.458432 | lrm: 1.00 | dt: 1646.15ms | tok/sec: 9,952 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 41 | total time: 24.67m | eta: 109.0m
step 00932/05000 (18.64%) | loss: 4.452750 | lrm: 1.00 | dt: 1660.28ms | tok/sec: 9,868 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 41 | total time: 24.70m | eta: 109.0m
step 00933/05000 (18.66%) | loss: 4.442058 | lrm: 1.00 | dt: 1619.10ms | tok/sec: 10,119 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 41 | total time: 24.73m | eta: 108.9m
step 00934/05000 (18.68%) | loss: 4.436472 | lrm: 1.00 | dt: 1599.93ms | tok/sec: 10,240 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 41 | total time: 24.75m | eta: 108.9m
step 00935/05000 (18.70%) | loss: 4.456891 | lrm: 1.00 | dt: 1637.07ms | tok/sec: 10,008 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 41 | total time: 24.78m | eta: 108.9m
step 00936/05000 (18.72%) | loss: 4.462962 | lrm: 1.00 | dt: 1589.13ms | tok/sec: 10,310 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 41 | total time: 24.81m | eta: 108.9m
step 00937/05000 (18.74%) | loss: 4.463555 | lrm: 1.00 | dt: 1586.16ms | tok/sec: 10,329 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 41 | total time: 24.83m | eta: 108.8m
step 00938/05000 (18.76%) | loss: 4.463154 | lrm: 1.00 | dt: 1558.51ms | tok/sec: 10,512 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 41 | total time: 24.86m | eta: 108.8m
step 00939/05000 (18.78%) | loss: 4.477996 | lrm: 1.00 | dt: 1580.94ms | tok/sec: 10,363 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 41 | total time: 24.88m | eta: 108.8m
step 00940/05000 (18.80%) | loss: 4.466556 | lrm: 1.00 | dt: 1600.74ms | tok/sec: 10,235 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 41 | total time: 24.91m | eta: 108.8m
step 00941/05000 (18.82%) | loss: 4.454611 | lrm: 1.00 | dt: 1603.37ms | tok/sec: 10,218 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 41 | total time: 24.94m | eta: 108.7m
step 00942/05000 (18.84%) | loss: 4.453937 | lrm: 1.00 | dt: 1573.19ms | tok/sec: 10,414 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 41 | total time: 24.96m | eta: 108.7m
step 00943/05000 (18.86%) | loss: 4.452801 | lrm: 1.00 | dt: 1572.44ms | tok/sec: 10,419 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 42 | total time: 24.99m | eta: 108.7m
step 00944/05000 (18.88%) | loss: 4.447253 | lrm: 1.00 | dt: 1565.56ms | tok/sec: 10,465 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 42 | total time: 25.02m | eta: 108.6m
step 00945/05000 (18.90%) | loss: 4.466573 | lrm: 1.00 | dt: 1549.05ms | tok/sec: 10,576 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 42 | total time: 25.04m | eta: 108.6m
step 00946/05000 (18.92%) | loss: 4.454063 | lrm: 1.00 | dt: 1580.29ms | tok/sec: 10,367 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 42 | total time: 25.07m | eta: 108.6m
step 00947/05000 (18.94%) | loss: 4.459854 | lrm: 1.00 | dt: 1592.24ms | tok/sec: 10,289 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 42 | total time: 25.10m | eta: 108.6m
step 00948/05000 (18.96%) | loss: 4.467468 | lrm: 1.00 | dt: 1576.01ms | tok/sec: 10,395 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 42 | total time: 25.12m | eta: 108.5m
step 00949/05000 (18.98%) | loss: 4.471551 | lrm: 1.00 | dt: 1570.71ms | tok/sec: 10,430 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 42 | total time: 25.15m | eta: 108.5m
step 00950/05000 (19.00%) | loss: 4.465269 | lrm: 1.00 | dt: 1553.94ms | tok/sec: 10,543 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 42 | total time: 25.17m | eta: 108.5m
step 00951/05000 (19.02%) | loss: 4.465691 | lrm: 1.00 | dt: 1576.59ms | tok/sec: 10,392 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 42 | total time: 25.20m | eta: 108.4m
step 00952/05000 (19.04%) | loss: 4.448087 | lrm: 1.00 | dt: 1542.54ms | tok/sec: 10,621 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 42 | total time: 25.23m | eta: 108.4m
step 00953/05000 (19.06%) | loss: 4.445159 | lrm: 1.00 | dt: 1571.50ms | tok/sec: 10,425 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 42 | total time: 25.25m | eta: 108.4m
step 00954/05000 (19.08%) | loss: 4.449361 | lrm: 1.00 | dt: 1590.91ms | tok/sec: 10,298 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 42 | total time: 25.28m | eta: 108.3m
step 00955/05000 (19.10%) | loss: 4.451447 | lrm: 1.00 | dt: 1535.17ms | tok/sec: 10,672 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 42 | total time: 25.30m | eta: 108.3m
step 00956/05000 (19.12%) | loss: 4.459403 | lrm: 1.00 | dt: 1556.58ms | tok/sec: 10,525 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 42 | total time: 25.33m | eta: 108.3m
step 00957/05000 (19.14%) | loss: 4.457230 | lrm: 1.00 | dt: 1592.67ms | tok/sec: 10,287 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 42 | total time: 25.36m | eta: 108.3m
step 00958/05000 (19.16%) | loss: 4.459277 | lrm: 1.00 | dt: 1561.37ms | tok/sec: 10,493 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 42 | total time: 25.38m | eta: 108.2m
step 00959/05000 (19.18%) | loss: 4.452698 | lrm: 1.00 | dt: 1554.14ms | tok/sec: 10,542 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 42 | total time: 25.41m | eta: 108.2m
step 00960/05000 (19.20%) | loss: 4.446957 | lrm: 1.00 | dt: 1564.79ms | tok/sec: 10,470 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 42 | total time: 25.43m | eta: 108.2m
step 00961/05000 (19.22%) | loss: 4.452467 | lrm: 1.00 | dt: 1559.67ms | tok/sec: 10,504 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 42 | total time: 25.46m | eta: 108.1m
step 00962/05000 (19.24%) | loss: 4.456186 | lrm: 1.00 | dt: 1575.32ms | tok/sec: 10,400 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 42 | total time: 25.49m | eta: 108.1m
step 00963/05000 (19.26%) | loss: 4.437612 | lrm: 1.00 | dt: 1542.43ms | tok/sec: 10,622 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 42 | total time: 25.51m | eta: 108.1m
step 00964/05000 (19.28%) | loss: 4.446750 | lrm: 1.00 | dt: 1561.43ms | tok/sec: 10,492 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 42 | total time: 25.54m | eta: 108.0m
step 00965/05000 (19.30%) | loss: 4.462723 | lrm: 1.00 | dt: 1547.15ms | tok/sec: 10,589 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 42 | total time: 25.56m | eta: 108.0m
step 00966/05000 (19.32%) | loss: 4.455844 | lrm: 1.00 | dt: 1581.51ms | tok/sec: 10,359 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 42 | total time: 25.59m | eta: 108.0m
step 00967/05000 (19.34%) | loss: 4.447490 | lrm: 1.00 | dt: 1561.01ms | tok/sec: 10,495 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 43 | total time: 25.62m | eta: 108.0m
step 00968/05000 (19.36%) | loss: 4.447828 | lrm: 1.00 | dt: 1581.61ms | tok/sec: 10,359 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 43 | total time: 25.64m | eta: 107.9m
step 00969/05000 (19.38%) | loss: 4.438618 | lrm: 1.00 | dt: 1538.05ms | tok/sec: 10,652 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 43 | total time: 25.67m | eta: 107.9m
step 00970/05000 (19.40%) | loss: 4.419770 | lrm: 1.00 | dt: 1587.04ms | tok/sec: 10,323 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 43 | total time: 25.70m | eta: 107.9m
step 00971/05000 (19.42%) | loss: 4.422287 | lrm: 1.00 | dt: 1625.06ms | tok/sec: 10,082 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 43 | total time: 25.72m | eta: 107.8m
step 00972/05000 (19.44%) | loss: 4.416319 | lrm: 1.00 | dt: 1585.13ms | tok/sec: 10,336 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 43 | total time: 25.75m | eta: 107.8m
step 00973/05000 (19.46%) | loss: 4.409622 | lrm: 1.00 | dt: 1612.67ms | tok/sec: 10,159 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 43 | total time: 25.78m | eta: 107.8m
step 00974/05000 (19.48%) | loss: 4.415893 | lrm: 1.00 | dt: 1633.54ms | tok/sec: 10,029 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 43 | total time: 25.80m | eta: 107.8m
step 00975/05000 (19.50%) | loss: 4.413749 | lrm: 1.00 | dt: 1590.77ms | tok/sec: 10,299 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 43 | total time: 25.83m | eta: 107.7m
step 00976/05000 (19.52%) | loss: 4.414477 | lrm: 1.00 | dt: 1675.92ms | tok/sec: 9,776 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 43 | total time: 25.86m | eta: 107.7m
step 00977/05000 (19.54%) | loss: 4.405266 | lrm: 1.00 | dt: 1596.27ms | tok/sec: 10,263 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 43 | total time: 25.88m | eta: 107.7m
step 00978/05000 (19.56%) | loss: 4.383851 | lrm: 1.00 | dt: 1760.53ms | tok/sec: 9,306 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 43 | total time: 25.91m | eta: 107.7m
step 00979/05000 (19.58%) | loss: 4.402540 | lrm: 1.00 | dt: 1832.57ms | tok/sec: 8,940 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 43 | total time: 25.94m | eta: 107.7m
step 00980/05000 (19.60%) | loss: 4.397469 | lrm: 1.00 | dt: 1672.78ms | tok/sec: 9,794 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 43 | total time: 25.97m | eta: 107.6m
step 00981/05000 (19.62%) | loss: 4.388618 | lrm: 1.00 | dt: 1694.40ms | tok/sec: 9,669 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 43 | total time: 26.00m | eta: 107.6m
step 00982/05000 (19.64%) | loss: 4.387367 | lrm: 1.00 | dt: 1672.61ms | tok/sec: 9,795 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 43 | total time: 26.03m | eta: 107.6m
step 00983/05000 (19.66%) | loss: 4.386861 | lrm: 1.00 | dt: 1717.90ms | tok/sec: 9,537 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 43 | total time: 26.06m | eta: 107.6m
step 00984/05000 (19.68%) | loss: 4.380794 | lrm: 1.00 | dt: 1681.91ms | tok/sec: 9,741 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 43 | total time: 26.08m | eta: 107.6m
step 00985/05000 (19.70%) | loss: 4.376461 | lrm: 1.00 | dt: 1644.21ms | tok/sec: 9,964 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 43 | total time: 26.11m | eta: 107.5m
step 00986/05000 (19.72%) | loss: 4.386272 | lrm: 1.00 | dt: 1717.49ms | tok/sec: 9,539 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 43 | total time: 26.14m | eta: 107.5m
step 00987/05000 (19.74%) | loss: 4.372638 | lrm: 1.00 | dt: 1700.09ms | tok/sec: 9,637 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 43 | total time: 26.17m | eta: 107.5m
step 00988/05000 (19.76%) | loss: 4.368430 | lrm: 1.00 | dt: 1711.76ms | tok/sec: 9,571 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 43 | total time: 26.20m | eta: 107.5m
step 00989/05000 (19.78%) | loss: 4.375752 | lrm: 1.00 | dt: 1729.44ms | tok/sec: 9,473 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 43 | total time: 26.23m | eta: 107.4m
step 00990/05000 (19.80%) | loss: 4.375578 | lrm: 1.00 | dt: 1763.54ms | tok/sec: 9,290 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 44 | total time: 26.26m | eta: 107.4m
step 00991/05000 (19.82%) | loss: 4.378647 | lrm: 1.00 | dt: 1729.68ms | tok/sec: 9,472 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 44 | total time: 26.28m | eta: 107.4m
step 00992/05000 (19.84%) | loss: 4.388274 | lrm: 1.00 | dt: 1685.30ms | tok/sec: 9,721 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 44 | total time: 26.31m | eta: 107.4m
step 00993/05000 (19.86%) | loss: 4.384134 | lrm: 1.00 | dt: 1723.21ms | tok/sec: 9,507 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 44 | total time: 26.34m | eta: 107.4m
step 00994/05000 (19.88%) | loss: 4.389445 | lrm: 1.00 | dt: 1708.38ms | tok/sec: 9,590 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 44 | total time: 26.37m | eta: 107.4m
step 00995/05000 (19.90%) | loss: 4.397208 | lrm: 1.00 | dt: 1734.34ms | tok/sec: 9,446 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 44 | total time: 26.40m | eta: 107.3m
step 00996/05000 (19.92%) | loss: 4.383793 | lrm: 1.00 | dt: 1743.65ms | tok/sec: 9,396 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 44 | total time: 26.43m | eta: 107.3m
step 00997/05000 (19.94%) | loss: 4.360792 | lrm: 1.00 | dt: 1728.76ms | tok/sec: 9,477 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 44 | total time: 26.46m | eta: 107.3m
step 00998/05000 (19.96%) | loss: 4.360088 | lrm: 1.00 | dt: 1683.92ms | tok/sec: 9,729 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 44 | total time: 26.48m | eta: 107.3m
step 00999/05000 (19.98%) | loss: 4.374241 | lrm: 1.00 | dt: 1658.52ms | tok/sec: 9,878 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 44 | total time: 26.51m | eta: 107.3m
Step 01000 | Validation bpb: 1.371813
<|bos|>The capital of France is a large, large, and large, large, and large, large, and
<|bos|>The chemical symbol of gold is a symbol of gold and gold. It is a symbol of gold and gold.
<|bos|>If yesterday was Friday, then tomorrow will be the first to be the first to be the first to be the first to be
<|bos|>The opposite of hot is true. The opposite is true. The opposite is true. The opposite is true
<|bos|>The planets of the solar system are: the solar system is a very small, and the solar system is a very small
<|bos|>My favorite color is a color that is a bit different from the color. It is a bit different
<|bos|>If 5*x + 3 = 13, then x is 5*x + 3 = 5, then x is 5
step 01000/05000 (20.00%) | loss: 4.380859 | lrm: 1.00 | dt: 2395.70ms | tok/sec: 6,838 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 44 | total time: 26.55m | eta: 107.3m
step 01001/05000 (20.02%) | loss: 4.381770 | lrm: 1.00 | dt: 1622.31ms | tok/sec: 10,099 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 44 | total time: 26.58m | eta: 107.3m
step 01002/05000 (20.04%) | loss: 4.390199 | lrm: 1.00 | dt: 1550.84ms | tok/sec: 10,564 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 44 | total time: 26.60m | eta: 107.2m
step 01003/05000 (20.06%) | loss: 4.409095 | lrm: 1.00 | dt: 1585.62ms | tok/sec: 10,332 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 44 | total time: 26.63m | eta: 107.2m
step 01004/05000 (20.08%) | loss: 4.413947 | lrm: 1.00 | dt: 1590.09ms | tok/sec: 10,303 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 44 | total time: 26.66m | eta: 107.2m
step 01005/05000 (20.10%) | loss: 4.417300 | lrm: 1.00 | dt: 1561.44ms | tok/sec: 10,492 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 44 | total time: 26.68m | eta: 107.1m
step 01006/05000 (20.12%) | loss: 4.411008 | lrm: 1.00 | dt: 1582.56ms | tok/sec: 10,352 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 44 | total time: 26.71m | eta: 107.1m
step 01007/05000 (20.14%) | loss: 4.435591 | lrm: 1.00 | dt: 1564.49ms | tok/sec: 10,472 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 44 | total time: 26.74m | eta: 107.1m
step 01008/05000 (20.16%) | loss: 4.441953 | lrm: 1.00 | dt: 1599.62ms | tok/sec: 10,242 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 44 | total time: 26.76m | eta: 107.1m
step 01009/05000 (20.18%) | loss: 4.442867 | lrm: 1.00 | dt: 1550.15ms | tok/sec: 10,569 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 44 | total time: 26.79m | eta: 107.0m
step 01010/05000 (20.20%) | loss: 4.462763 | lrm: 1.00 | dt: 1660.85ms | tok/sec: 9,864 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 44 | total time: 26.82m | eta: 107.0m
step 01011/05000 (20.22%) | loss: 4.453047 | lrm: 1.00 | dt: 1616.47ms | tok/sec: 10,135 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 44 | total time: 26.84m | eta: 107.0m
step 01012/05000 (20.24%) | loss: 4.458602 | lrm: 1.00 | dt: 1565.12ms | tok/sec: 10,468 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 44 | total time: 26.87m | eta: 106.9m
step 01013/05000 (20.26%) | loss: 4.475096 | lrm: 1.00 | dt: 1585.83ms | tok/sec: 10,331 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 44 | total time: 26.90m | eta: 106.9m
step 01014/05000 (20.28%) | loss: 4.460769 | lrm: 1.00 | dt: 1567.53ms | tok/sec: 10,452 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 45 | total time: 26.92m | eta: 106.9m
step 01015/05000 (20.30%) | loss: 4.444605 | lrm: 1.00 | dt: 1586.95ms | tok/sec: 10,324 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 45 | total time: 26.95m | eta: 106.9m
step 01016/05000 (20.32%) | loss: 4.448264 | lrm: 1.00 | dt: 1541.20ms | tok/sec: 10,630 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 45 | total time: 26.97m | eta: 106.8m
step 01017/05000 (20.34%) | loss: 4.446704 | lrm: 1.00 | dt: 1567.58ms | tok/sec: 10,451 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 45 | total time: 27.00m | eta: 106.8m
step 01018/05000 (20.36%) | loss: 4.441256 | lrm: 1.00 | dt: 1550.09ms | tok/sec: 10,569 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 45 | total time: 27.03m | eta: 106.8m
step 01019/05000 (20.38%) | loss: 4.434862 | lrm: 1.00 | dt: 1603.83ms | tok/sec: 10,215 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 45 | total time: 27.05m | eta: 106.7m
step 01020/05000 (20.40%) | loss: 4.451183 | lrm: 1.00 | dt: 1576.34ms | tok/sec: 10,393 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 45 | total time: 27.08m | eta: 106.7m
step 01021/05000 (20.42%) | loss: 4.451797 | lrm: 1.00 | dt: 1576.27ms | tok/sec: 10,394 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 45 | total time: 27.11m | eta: 106.7m
step 01022/05000 (20.44%) | loss: 4.455675 | lrm: 1.00 | dt: 1552.63ms | tok/sec: 10,552 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 45 | total time: 27.13m | eta: 106.6m
step 01023/05000 (20.46%) | loss: 4.447015 | lrm: 1.00 | dt: 1581.45ms | tok/sec: 10,360 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 45 | total time: 27.16m | eta: 106.6m
step 01024/05000 (20.48%) | loss: 4.450057 | lrm: 1.00 | dt: 1566.14ms | tok/sec: 10,461 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 45 | total time: 27.18m | eta: 106.6m
step 01025/05000 (20.50%) | loss: 4.454498 | lrm: 1.00 | dt: 1558.32ms | tok/sec: 10,513 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 45 | total time: 27.21m | eta: 106.6m
step 01026/05000 (20.52%) | loss: 4.426223 | lrm: 1.00 | dt: 1585.14ms | tok/sec: 10,335 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 45 | total time: 27.24m | eta: 106.5m
step 01027/05000 (20.54%) | loss: 4.413258 | lrm: 1.00 | dt: 1569.49ms | tok/sec: 10,439 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 45 | total time: 27.26m | eta: 106.5m
step 01028/05000 (20.56%) | loss: 4.408422 | lrm: 1.00 | dt: 1599.95ms | tok/sec: 10,240 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 45 | total time: 27.29m | eta: 106.5m
step 01029/05000 (20.58%) | loss: 4.403433 | lrm: 1.00 | dt: 1568.02ms | tok/sec: 10,448 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 45 | total time: 27.32m | eta: 106.4m
step 01030/05000 (20.60%) | loss: 4.412383 | lrm: 1.00 | dt: 1567.75ms | tok/sec: 10,450 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 45 | total time: 27.34m | eta: 106.4m
step 01031/05000 (20.62%) | loss: 4.409126 | lrm: 1.00 | dt: 1537.13ms | tok/sec: 10,658 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 45 | total time: 27.37m | eta: 106.4m
step 01032/05000 (20.64%) | loss: 4.402938 | lrm: 1.00 | dt: 1562.79ms | tok/sec: 10,483 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 45 | total time: 27.39m | eta: 106.4m
step 01033/05000 (20.66%) | loss: 4.416016 | lrm: 1.00 | dt: 1569.69ms | tok/sec: 10,437 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 45 | total time: 27.42m | eta: 106.3m
step 01034/05000 (20.68%) | loss: 4.414661 | lrm: 1.00 | dt: 1563.59ms | tok/sec: 10,478 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 45 | total time: 27.45m | eta: 106.3m
step 01035/05000 (20.70%) | loss: 4.400647 | lrm: 1.00 | dt: 1579.24ms | tok/sec: 10,374 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 45 | total time: 27.47m | eta: 106.3m
step 01036/05000 (20.72%) | loss: 4.406834 | lrm: 1.00 | dt: 1571.20ms | tok/sec: 10,427 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 45 | total time: 27.50m | eta: 106.2m
step 01037/05000 (20.74%) | loss: 4.390281 | lrm: 1.00 | dt: 1552.47ms | tok/sec: 10,553 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 46 | total time: 27.52m | eta: 106.2m
step 01038/05000 (20.76%) | loss: 4.390515 | lrm: 1.00 | dt: 1562.99ms | tok/sec: 10,482 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 46 | total time: 27.55m | eta: 106.2m
step 01039/05000 (20.78%) | loss: 4.394342 | lrm: 1.00 | dt: 1552.11ms | tok/sec: 10,555 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 46 | total time: 27.58m | eta: 106.1m
step 01040/05000 (20.80%) | loss: 4.393337 | lrm: 1.00 | dt: 1561.49ms | tok/sec: 10,492 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 46 | total time: 27.60m | eta: 106.1m
step 01041/05000 (20.82%) | loss: 4.392282 | lrm: 1.00 | dt: 1572.77ms | tok/sec: 10,417 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 46 | total time: 27.63m | eta: 106.1m
step 01042/05000 (20.84%) | loss: 4.396212 | lrm: 1.00 | dt: 1596.25ms | tok/sec: 10,264 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 46 | total time: 27.65m | eta: 106.1m
step 01043/05000 (20.86%) | loss: 4.386475 | lrm: 1.00 | dt: 1559.38ms | tok/sec: 10,506 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 46 | total time: 27.68m | eta: 106.0m
step 01044/05000 (20.88%) | loss: 4.393965 | lrm: 1.00 | dt: 1579.03ms | tok/sec: 10,376 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 46 | total time: 27.71m | eta: 106.0m
step 01045/05000 (20.90%) | loss: 4.402042 | lrm: 1.00 | dt: 1558.84ms | tok/sec: 10,510 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 46 | total time: 27.73m | eta: 106.0m
step 01046/05000 (20.92%) | loss: 4.406793 | lrm: 1.00 | dt: 1593.90ms | tok/sec: 10,279 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 46 | total time: 27.76m | eta: 105.9m
step 01047/05000 (20.94%) | loss: 4.392435 | lrm: 1.00 | dt: 1578.57ms | tok/sec: 10,378 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 46 | total time: 27.79m | eta: 105.9m
step 01048/05000 (20.96%) | loss: 4.392604 | lrm: 1.00 | dt: 1561.79ms | tok/sec: 10,490 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 46 | total time: 27.81m | eta: 105.9m
step 01049/05000 (20.98%) | loss: 4.392579 | lrm: 1.00 | dt: 1606.19ms | tok/sec: 10,200 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 46 | total time: 27.84m | eta: 105.9m
step 01050/05000 (21.00%) | loss: 4.385737 | lrm: 1.00 | dt: 1568.33ms | tok/sec: 10,446 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 46 | total time: 27.86m | eta: 105.8m
step 01051/05000 (21.02%) | loss: 4.387178 | lrm: 1.00 | dt: 1550.99ms | tok/sec: 10,563 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 46 | total time: 27.89m | eta: 105.8m
step 01052/05000 (21.04%) | loss: 4.399935 | lrm: 1.00 | dt: 1562.99ms | tok/sec: 10,482 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 46 | total time: 27.92m | eta: 105.8m
step 01053/05000 (21.06%) | loss: 4.395574 | lrm: 1.00 | dt: 1550.20ms | tok/sec: 10,568 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 46 | total time: 27.94m | eta: 105.7m
step 01054/05000 (21.08%) | loss: 4.398187 | lrm: 1.00 | dt: 1571.52ms | tok/sec: 10,425 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 46 | total time: 27.97m | eta: 105.7m
step 01055/05000 (21.10%) | loss: 4.405870 | lrm: 1.00 | dt: 1541.52ms | tok/sec: 10,628 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 46 | total time: 27.99m | eta: 105.7m
step 01056/05000 (21.12%) | loss: 4.402936 | lrm: 1.00 | dt: 1592.23ms | tok/sec: 10,289 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 46 | total time: 28.02m | eta: 105.7m
step 01057/05000 (21.14%) | loss: 4.400779 | lrm: 1.00 | dt: 1566.03ms | tok/sec: 10,462 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 46 | total time: 28.05m | eta: 105.6m
step 01058/05000 (21.16%) | loss: 4.387458 | lrm: 1.00 | dt: 1555.96ms | tok/sec: 10,529 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 46 | total time: 28.07m | eta: 105.6m
step 01059/05000 (21.18%) | loss: 4.386832 | lrm: 1.00 | dt: 1573.95ms | tok/sec: 10,409 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 46 | total time: 28.10m | eta: 105.6m
step 01060/05000 (21.20%) | loss: 4.382617 | lrm: 1.00 | dt: 1542.03ms | tok/sec: 10,624 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 46 | total time: 28.12m | eta: 105.5m
step 01061/05000 (21.22%) | loss: 4.380274 | lrm: 1.00 | dt: 1556.21ms | tok/sec: 10,528 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 47 | total time: 28.15m | eta: 105.5m
step 01062/05000 (21.24%) | loss: 4.386055 | lrm: 1.00 | dt: 1557.85ms | tok/sec: 10,517 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 47 | total time: 28.18m | eta: 105.5m
step 01063/05000 (21.26%) | loss: 4.382154 | lrm: 1.00 | dt: 1543.34ms | tok/sec: 10,615 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 47 | total time: 28.20m | eta: 105.4m
step 01064/05000 (21.28%) | loss: 4.380044 | lrm: 1.00 | dt: 1582.11ms | tok/sec: 10,355 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 47 | total time: 28.23m | eta: 105.4m
step 01065/05000 (21.30%) | loss: 4.404158 | lrm: 1.00 | dt: 1613.48ms | tok/sec: 10,154 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 47 | total time: 28.26m | eta: 105.4m
step 01066/05000 (21.32%) | loss: 4.405214 | lrm: 1.00 | dt: 1592.35ms | tok/sec: 10,289 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 47 | total time: 28.28m | eta: 105.4m
step 01067/05000 (21.34%) | loss: 4.417140 | lrm: 1.00 | dt: 1567.91ms | tok/sec: 10,449 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 47 | total time: 28.31m | eta: 105.3m
step 01068/05000 (21.36%) | loss: 4.413396 | lrm: 1.00 | dt: 1599.55ms | tok/sec: 10,242 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 47 | total time: 28.33m | eta: 105.3m
step 01069/05000 (21.38%) | loss: 4.412624 | lrm: 1.00 | dt: 1571.41ms | tok/sec: 10,426 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 47 | total time: 28.36m | eta: 105.3m
step 01070/05000 (21.40%) | loss: 4.400049 | lrm: 1.00 | dt: 1621.62ms | tok/sec: 10,103 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 47 | total time: 28.39m | eta: 105.2m
step 01071/05000 (21.42%) | loss: 4.408682 | lrm: 1.00 | dt: 1597.40ms | tok/sec: 10,256 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 47 | total time: 28.41m | eta: 105.2m
step 01072/05000 (21.44%) | loss: 4.415588 | lrm: 1.00 | dt: 1585.73ms | tok/sec: 10,332 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 47 | total time: 28.44m | eta: 105.2m
step 01073/05000 (21.46%) | loss: 4.406372 | lrm: 1.00 | dt: 1593.72ms | tok/sec: 10,280 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 47 | total time: 28.47m | eta: 105.2m
step 01074/05000 (21.48%) | loss: 4.406429 | lrm: 1.00 | dt: 1609.83ms | tok/sec: 10,177 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 47 | total time: 28.49m | eta: 105.1m
step 01075/05000 (21.50%) | loss: 4.407084 | lrm: 1.00 | dt: 1620.67ms | tok/sec: 10,109 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 47 | total time: 28.52m | eta: 105.1m
step 01076/05000 (21.52%) | loss: 4.389762 | lrm: 1.00 | dt: 1570.52ms | tok/sec: 10,432 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 47 | total time: 28.55m | eta: 105.1m
step 01077/05000 (21.54%) | loss: 4.395692 | lrm: 1.00 | dt: 1581.72ms | tok/sec: 10,358 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 47 | total time: 28.57m | eta: 105.1m
step 01078/05000 (21.56%) | loss: 4.396843 | lrm: 1.00 | dt: 1588.83ms | tok/sec: 10,311 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 47 | total time: 28.60m | eta: 105.0m
step 01079/05000 (21.58%) | loss: 4.381091 | lrm: 1.00 | dt: 1576.88ms | tok/sec: 10,390 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 47 | total time: 28.63m | eta: 105.0m
step 01080/05000 (21.60%) | loss: 4.388162 | lrm: 1.00 | dt: 1571.11ms | tok/sec: 10,428 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 47 | total time: 28.65m | eta: 105.0m
step 01081/05000 (21.62%) | loss: 4.408515 | lrm: 1.00 | dt: 1604.00ms | tok/sec: 10,214 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 47 | total time: 28.68m | eta: 104.9m
step 01082/05000 (21.64%) | loss: 4.415806 | lrm: 1.00 | dt: 1582.38ms | tok/sec: 10,354 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 47 | total time: 28.71m | eta: 104.9m
step 01083/05000 (21.66%) | loss: 4.410440 | lrm: 1.00 | dt: 1587.36ms | tok/sec: 10,321 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 47 | total time: 28.73m | eta: 104.9m
step 01084/05000 (21.68%) | loss: 4.409497 | lrm: 1.00 | dt: 1591.42ms | tok/sec: 10,295 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 47 | total time: 28.76m | eta: 104.9m
step 01085/05000 (21.70%) | loss: 4.405321 | lrm: 1.00 | dt: 1570.73ms | tok/sec: 10,430 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 48 | total time: 28.79m | eta: 104.8m
step 01086/05000 (21.72%) | loss: 4.398885 | lrm: 1.00 | dt: 1572.52ms | tok/sec: 10,418 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 48 | total time: 28.81m | eta: 104.8m
step 01087/05000 (21.74%) | loss: 4.387005 | lrm: 1.00 | dt: 1591.37ms | tok/sec: 10,295 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 48 | total time: 28.84m | eta: 104.8m
step 01088/05000 (21.76%) | loss: 4.388525 | lrm: 1.00 | dt: 1570.36ms | tok/sec: 10,433 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 48 | total time: 28.86m | eta: 104.7m
step 01089/05000 (21.78%) | loss: 4.392627 | lrm: 1.00 | dt: 1552.15ms | tok/sec: 10,555 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 48 | total time: 28.89m | eta: 104.7m
step 01090/05000 (21.80%) | loss: 4.387825 | lrm: 1.00 | dt: 1552.64ms | tok/sec: 10,552 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 48 | total time: 28.92m | eta: 104.7m
step 01091/05000 (21.82%) | loss: 4.387727 | lrm: 1.00 | dt: 1566.38ms | tok/sec: 10,459 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 48 | total time: 28.94m | eta: 104.7m
step 01092/05000 (21.84%) | loss: 4.388582 | lrm: 1.00 | dt: 1579.40ms | tok/sec: 10,373 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 48 | total time: 28.97m | eta: 104.6m
step 01093/05000 (21.86%) | loss: 4.393973 | lrm: 1.00 | dt: 1569.26ms | tok/sec: 10,440 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 48 | total time: 28.99m | eta: 104.6m
step 01094/05000 (21.88%) | loss: 4.384049 | lrm: 1.00 | dt: 1584.67ms | tok/sec: 10,339 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 48 | total time: 29.02m | eta: 104.6m
step 01095/05000 (21.90%) | loss: 4.384612 | lrm: 1.00 | dt: 1573.89ms | tok/sec: 10,409 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 48 | total time: 29.05m | eta: 104.5m
step 01096/05000 (21.92%) | loss: 4.391219 | lrm: 1.00 | dt: 1552.85ms | tok/sec: 10,550 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 48 | total time: 29.07m | eta: 104.5m
step 01097/05000 (21.94%) | loss: 4.399458 | lrm: 1.00 | dt: 1569.26ms | tok/sec: 10,440 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 48 | total time: 29.10m | eta: 104.5m
step 01098/05000 (21.96%) | loss: 4.401729 | lrm: 1.00 | dt: 1550.74ms | tok/sec: 10,565 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 48 | total time: 29.12m | eta: 104.5m
step 01099/05000 (21.98%) | loss: 4.392903 | lrm: 1.00 | dt: 1566.82ms | tok/sec: 10,456 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 48 | total time: 29.15m | eta: 104.4m
Step 01100 | Validation bpb: 1.357232
<|bos|>The capital of France is the largest of the world's largest city of France. The capital of France is
<|bos|>The chemical symbol of gold is a symbol of the beauty and beauty of gold. It is a symbol of the
<|bos|>If yesterday was Friday, then tomorrow will be a great opportunity to learn more about the world around us. We'll be a
<|bos|>The opposite of hot is true. It is a very common type of hot, hot, and hot.
<|bos|>The planets of the solar system are: the sun, the sun, the sun, the sun, the sun, the
<|bos|>My favorite color is the white, white, and white. The white is a white, white,
<|bos|>If 5*x + 3 = 13, then x is 5*x + 3 + 3 + 3 + 3
step 01100/05000 (22.00%) | loss: 4.386683 | lrm: 1.00 | dt: 1579.72ms | tok/sec: 10,371 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 48 | total time: 29.18m | eta: 104.4m
step 01101/05000 (22.02%) | loss: 4.375814 | lrm: 1.00 | dt: 1565.64ms | tok/sec: 10,464 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 48 | total time: 29.20m | eta: 104.4m
step 01102/05000 (22.04%) | loss: 4.373438 | lrm: 1.00 | dt: 1544.32ms | tok/sec: 10,609 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 48 | total time: 29.23m | eta: 104.3m
step 01103/05000 (22.06%) | loss: 4.381326 | lrm: 1.00 | dt: 1549.53ms | tok/sec: 10,573 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 48 | total time: 29.26m | eta: 104.3m
step 01104/05000 (22.08%) | loss: 4.374831 | lrm: 1.00 | dt: 1567.94ms | tok/sec: 10,449 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 48 | total time: 29.28m | eta: 104.3m
step 01105/05000 (22.10%) | loss: 4.378762 | lrm: 1.00 | dt: 1578.21ms | tok/sec: 10,381 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 48 | total time: 29.31m | eta: 104.2m
step 01106/05000 (22.12%) | loss: 4.382688 | lrm: 1.00 | dt: 1550.34ms | tok/sec: 10,568 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 48 | total time: 29.33m | eta: 104.2m
step 01107/05000 (22.14%) | loss: 4.369735 | lrm: 1.00 | dt: 1543.20ms | tok/sec: 10,616 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 48 | total time: 29.36m | eta: 104.2m
step 01108/05000 (22.16%) | loss: 4.373907 | lrm: 1.00 | dt: 1573.67ms | tok/sec: 10,411 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 49 | total time: 29.39m | eta: 104.2m
step 01109/05000 (22.18%) | loss: 4.376369 | lrm: 1.00 | dt: 1569.95ms | tok/sec: 10,436 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 49 | total time: 29.41m | eta: 104.1m
step 01110/05000 (22.20%) | loss: 4.372037 | lrm: 1.00 | dt: 1598.97ms | tok/sec: 10,246 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 49 | total time: 29.44m | eta: 104.1m
step 01111/05000 (22.22%) | loss: 4.361625 | lrm: 1.00 | dt: 1550.61ms | tok/sec: 10,566 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 49 | total time: 29.46m | eta: 104.1m
step 01112/05000 (22.24%) | loss: 4.378236 | lrm: 1.00 | dt: 1568.41ms | tok/sec: 10,446 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 49 | total time: 29.49m | eta: 104.0m
step 01113/05000 (22.26%) | loss: 4.369525 | lrm: 1.00 | dt: 1543.26ms | tok/sec: 10,616 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 49 | total time: 29.52m | eta: 104.0m
step 01114/05000 (22.28%) | loss: 4.368812 | lrm: 1.00 | dt: 1573.16ms | tok/sec: 10,414 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 49 | total time: 29.54m | eta: 104.0m
step 01115/05000 (22.30%) | loss: 4.377952 | lrm: 1.00 | dt: 1556.07ms | tok/sec: 10,529 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 49 | total time: 29.57m | eta: 104.0m
step 01116/05000 (22.32%) | loss: 4.385275 | lrm: 1.00 | dt: 1558.72ms | tok/sec: 10,511 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 49 | total time: 29.59m | eta: 103.9m
step 01117/05000 (22.34%) | loss: 4.381763 | lrm: 1.00 | dt: 1556.82ms | tok/sec: 10,523 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 49 | total time: 29.62m | eta: 103.9m
step 01118/05000 (22.36%) | loss: 4.382570 | lrm: 1.00 | dt: 1579.15ms | tok/sec: 10,375 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 49 | total time: 29.65m | eta: 103.9m
step 01119/05000 (22.38%) | loss: 4.369265 | lrm: 1.00 | dt: 1575.40ms | tok/sec: 10,399 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 49 | total time: 29.67m | eta: 103.8m
step 01120/05000 (22.40%) | loss: 4.385428 | lrm: 1.00 | dt: 1590.30ms | tok/sec: 10,302 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 49 | total time: 29.70m | eta: 103.8m
step 01121/05000 (22.42%) | loss: 4.392829 | lrm: 1.00 | dt: 1549.25ms | tok/sec: 10,575 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 49 | total time: 29.72m | eta: 103.8m
step 01122/05000 (22.44%) | loss: 4.384444 | lrm: 1.00 | dt: 1572.21ms | tok/sec: 10,420 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 49 | total time: 29.75m | eta: 103.8m
step 01123/05000 (22.46%) | loss: 4.374830 | lrm: 1.00 | dt: 1584.63ms | tok/sec: 10,339 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 49 | total time: 29.78m | eta: 103.7m
step 01124/05000 (22.48%) | loss: 4.385688 | lrm: 1.00 | dt: 1574.93ms | tok/sec: 10,403 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 49 | total time: 29.80m | eta: 103.7m
step 01125/05000 (22.50%) | loss: 4.369135 | lrm: 1.00 | dt: 1559.23ms | tok/sec: 10,507 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 49 | total time: 29.83m | eta: 103.7m
step 01126/05000 (22.52%) | loss: 4.376024 | lrm: 1.00 | dt: 1573.36ms | tok/sec: 10,413 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 49 | total time: 29.86m | eta: 103.6m
step 01127/05000 (22.54%) | loss: 4.374096 | lrm: 1.00 | dt: 1536.05ms | tok/sec: 10,666 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 49 | total time: 29.88m | eta: 103.6m
step 01128/05000 (22.56%) | loss: 4.368066 | lrm: 1.00 | dt: 1582.28ms | tok/sec: 10,354 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 49 | total time: 29.91m | eta: 103.6m
step 01129/05000 (22.58%) | loss: 4.373696 | lrm: 1.00 | dt: 1562.66ms | tok/sec: 10,484 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 49 | total time: 29.93m | eta: 103.6m
step 01130/05000 (22.60%) | loss: 4.373567 | lrm: 1.00 | dt: 1583.22ms | tok/sec: 10,348 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 49 | total time: 29.96m | eta: 103.5m
step 01131/05000 (22.62%) | loss: 4.386454 | lrm: 1.00 | dt: 1617.46ms | tok/sec: 10,129 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 50 | total time: 29.99m | eta: 103.5m
step 01132/05000 (22.64%) | loss: 4.384658 | lrm: 1.00 | dt: 1568.30ms | tok/sec: 10,446 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 50 | total time: 30.01m | eta: 103.5m
step 01133/05000 (22.66%) | loss: 4.385754 | lrm: 1.00 | dt: 1632.28ms | tok/sec: 10,037 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 50 | total time: 30.04m | eta: 103.4m
step 01134/05000 (22.68%) | loss: 4.383657 | lrm: 1.00 | dt: 1617.09ms | tok/sec: 10,131 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 50 | total time: 30.07m | eta: 103.4m
step 01135/05000 (22.70%) | loss: 4.378349 | lrm: 1.00 | dt: 1605.76ms | tok/sec: 10,203 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 50 | total time: 30.09m | eta: 103.4m
step 01136/05000 (22.72%) | loss: 4.372000 | lrm: 1.00 | dt: 1633.06ms | tok/sec: 10,032 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 50 | total time: 30.12m | eta: 103.4m
step 01137/05000 (22.74%) | loss: 4.372423 | lrm: 1.00 | dt: 1557.82ms | tok/sec: 10,517 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 50 | total time: 30.15m | eta: 103.3m
step 01138/05000 (22.76%) | loss: 4.373431 | lrm: 1.00 | dt: 1575.58ms | tok/sec: 10,398 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 50 | total time: 30.17m | eta: 103.3m
step 01139/05000 (22.78%) | loss: 4.373464 | lrm: 1.00 | dt: 1572.22ms | tok/sec: 10,420 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 50 | total time: 30.20m | eta: 103.3m
step 01140/05000 (22.80%) | loss: 4.371500 | lrm: 1.00 | dt: 1552.22ms | tok/sec: 10,555 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 50 | total time: 30.23m | eta: 103.2m
step 01141/05000 (22.82%) | loss: 4.363863 | lrm: 1.00 | dt: 1562.09ms | tok/sec: 10,488 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 50 | total time: 30.25m | eta: 103.2m
step 01142/05000 (22.84%) | loss: 4.361249 | lrm: 1.00 | dt: 1578.45ms | tok/sec: 10,379 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 50 | total time: 30.28m | eta: 103.2m
step 01143/05000 (22.86%) | loss: 4.360078 | lrm: 1.00 | dt: 1590.34ms | tok/sec: 10,302 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 50 | total time: 30.30m | eta: 103.2m
step 01144/05000 (22.88%) | loss: 4.359949 | lrm: 1.00 | dt: 1563.69ms | tok/sec: 10,477 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 50 | total time: 30.33m | eta: 103.1m
step 01145/05000 (22.90%) | loss: 4.356594 | lrm: 1.00 | dt: 1573.58ms | tok/sec: 10,411 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 50 | total time: 30.36m | eta: 103.1m
step 01146/05000 (22.92%) | loss: 4.357180 | lrm: 1.00 | dt: 1579.26ms | tok/sec: 10,374 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 50 | total time: 30.38m | eta: 103.1m
step 01147/05000 (22.94%) | loss: 4.367440 | lrm: 1.00 | dt: 1570.46ms | tok/sec: 10,432 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 50 | total time: 30.41m | eta: 103.0m
step 01148/05000 (22.96%) | loss: 4.375528 | lrm: 1.00 | dt: 1579.81ms | tok/sec: 10,370 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 50 | total time: 30.44m | eta: 103.0m
step 01149/05000 (22.98%) | loss: 4.376288 | lrm: 1.00 | dt: 1575.86ms | tok/sec: 10,396 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 50 | total time: 30.46m | eta: 103.0m
step 01150/05000 (23.00%) | loss: 4.362891 | lrm: 1.00 | dt: 1555.76ms | tok/sec: 10,531 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 50 | total time: 30.49m | eta: 103.0m
step 01151/05000 (23.02%) | loss: 4.356952 | lrm: 1.00 | dt: 1575.41ms | tok/sec: 10,399 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 50 | total time: 30.51m | eta: 102.9m
step 01152/05000 (23.04%) | loss: 4.347577 | lrm: 1.00 | dt: 1578.70ms | tok/sec: 10,378 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 50 | total time: 30.54m | eta: 102.9m
step 01153/05000 (23.06%) | loss: 4.364860 | lrm: 1.00 | dt: 1544.61ms | tok/sec: 10,607 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 50 | total time: 30.57m | eta: 102.9m
step 01154/05000 (23.08%) | loss: 4.374861 | lrm: 1.00 | dt: 1558.74ms | tok/sec: 10,511 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 51 | total time: 30.59m | eta: 102.8m
step 01155/05000 (23.10%) | loss: 4.367287 | lrm: 1.00 | dt: 1578.81ms | tok/sec: 10,377 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 51 | total time: 30.62m | eta: 102.8m
step 01156/05000 (23.12%) | loss: 4.371655 | lrm: 1.00 | dt: 1580.28ms | tok/sec: 10,367 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 51 | total time: 30.64m | eta: 102.8m
step 01157/05000 (23.14%) | loss: 4.368242 | lrm: 1.00 | dt: 1631.37ms | tok/sec: 10,043 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 51 | total time: 30.67m | eta: 102.8m
step 01158/05000 (23.16%) | loss: 4.359324 | lrm: 1.00 | dt: 1577.86ms | tok/sec: 10,383 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 51 | total time: 30.70m | eta: 102.7m
step 01159/05000 (23.18%) | loss: 4.378030 | lrm: 1.00 | dt: 1597.27ms | tok/sec: 10,257 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 51 | total time: 30.72m | eta: 102.7m
step 01160/05000 (23.20%) | loss: 4.380412 | lrm: 1.00 | dt: 1638.20ms | tok/sec: 10,001 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 51 | total time: 30.75m | eta: 102.7m
step 01161/05000 (23.22%) | loss: 4.374456 | lrm: 1.00 | dt: 1691.23ms | tok/sec: 9,687 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 51 | total time: 30.78m | eta: 102.7m
step 01162/05000 (23.24%) | loss: 4.373256 | lrm: 1.00 | dt: 1619.47ms | tok/sec: 10,116 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 51 | total time: 30.81m | eta: 102.6m
step 01163/05000 (23.26%) | loss: 4.354328 | lrm: 1.00 | dt: 1591.52ms | tok/sec: 10,294 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 51 | total time: 30.83m | eta: 102.6m
step 01164/05000 (23.28%) | loss: 4.360107 | lrm: 1.00 | dt: 1616.63ms | tok/sec: 10,134 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 51 | total time: 30.86m | eta: 102.6m
step 01165/05000 (23.30%) | loss: 4.360427 | lrm: 1.00 | dt: 1589.69ms | tok/sec: 10,306 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 51 | total time: 30.89m | eta: 102.6m
step 01166/05000 (23.32%) | loss: 4.362786 | lrm: 1.00 | dt: 1594.09ms | tok/sec: 10,277 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 51 | total time: 30.91m | eta: 102.5m
step 01167/05000 (23.34%) | loss: 4.376206 | lrm: 1.00 | dt: 1673.20ms | tok/sec: 9,792 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 51 | total time: 30.94m | eta: 102.5m
step 01168/05000 (23.36%) | loss: 4.367072 | lrm: 1.00 | dt: 1584.72ms | tok/sec: 10,338 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 51 | total time: 30.97m | eta: 102.5m
step 01169/05000 (23.38%) | loss: 4.351979 | lrm: 1.00 | dt: 1569.19ms | tok/sec: 10,441 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 51 | total time: 30.99m | eta: 102.5m
step 01170/05000 (23.40%) | loss: 4.370602 | lrm: 1.00 | dt: 1571.75ms | tok/sec: 10,424 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 51 | total time: 31.02m | eta: 102.4m
step 01171/05000 (23.42%) | loss: 4.375048 | lrm: 1.00 | dt: 1586.29ms | tok/sec: 10,328 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 51 | total time: 31.05m | eta: 102.4m
step 01172/05000 (23.44%) | loss: 4.361165 | lrm: 1.00 | dt: 1559.13ms | tok/sec: 10,508 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 51 | total time: 31.07m | eta: 102.4m
step 01173/05000 (23.46%) | loss: 4.365960 | lrm: 1.00 | dt: 1580.41ms | tok/sec: 10,366 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 51 | total time: 31.10m | eta: 102.3m
step 01174/05000 (23.48%) | loss: 4.351006 | lrm: 1.00 | dt: 1583.84ms | tok/sec: 10,344 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 51 | total time: 31.13m | eta: 102.3m
step 01175/05000 (23.50%) | loss: 4.336079 | lrm: 1.00 | dt: 1555.90ms | tok/sec: 10,530 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 51 | total time: 31.15m | eta: 102.3m
step 01176/05000 (23.52%) | loss: 4.332778 | lrm: 1.00 | dt: 1548.43ms | tok/sec: 10,581 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 51 | total time: 31.18m | eta: 102.2m
step 01177/05000 (23.54%) | loss: 4.335038 | lrm: 1.00 | dt: 1603.12ms | tok/sec: 10,220 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 52 | total time: 31.20m | eta: 102.2m
step 01178/05000 (23.56%) | loss: 4.348300 | lrm: 1.00 | dt: 1622.68ms | tok/sec: 10,096 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 52 | total time: 31.23m | eta: 102.2m
step 01179/05000 (23.58%) | loss: 4.358819 | lrm: 1.00 | dt: 1575.70ms | tok/sec: 10,397 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 52 | total time: 31.26m | eta: 102.2m
step 01180/05000 (23.60%) | loss: 4.356914 | lrm: 1.00 | dt: 1558.94ms | tok/sec: 10,509 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 52 | total time: 31.28m | eta: 102.1m
step 01181/05000 (23.62%) | loss: 4.351151 | lrm: 1.00 | dt: 1576.44ms | tok/sec: 10,393 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 52 | total time: 31.31m | eta: 102.1m
step 01182/05000 (23.64%) | loss: 4.347877 | lrm: 1.00 | dt: 1559.41ms | tok/sec: 10,506 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 52 | total time: 31.34m | eta: 102.1m
step 01183/05000 (23.66%) | loss: 4.346471 | lrm: 1.00 | dt: 1551.71ms | tok/sec: 10,558 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 52 | total time: 31.36m | eta: 102.1m
step 01184/05000 (23.68%) | loss: 4.362426 | lrm: 1.00 | dt: 1565.93ms | tok/sec: 10,462 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 52 | total time: 31.39m | eta: 102.0m
step 01185/05000 (23.70%) | loss: 4.370588 | lrm: 1.00 | dt: 1584.28ms | tok/sec: 10,341 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 52 | total time: 31.41m | eta: 102.0m
step 01186/05000 (23.72%) | loss: 4.377062 | lrm: 1.00 | dt: 1599.76ms | tok/sec: 10,241 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 52 | total time: 31.44m | eta: 102.0m
step 01187/05000 (23.74%) | loss: 4.386978 | lrm: 1.00 | dt: 1563.65ms | tok/sec: 10,478 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 52 | total time: 31.47m | eta: 101.9m
step 01188/05000 (23.76%) | loss: 4.369971 | lrm: 1.00 | dt: 1566.25ms | tok/sec: 10,460 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 52 | total time: 31.49m | eta: 101.9m
step 01189/05000 (23.78%) | loss: 4.376274 | lrm: 1.00 | dt: 1636.24ms | tok/sec: 10,013 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 52 | total time: 31.52m | eta: 101.9m
step 01190/05000 (23.80%) | loss: 4.378859 | lrm: 1.00 | dt: 1594.65ms | tok/sec: 10,274 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 52 | total time: 31.55m | eta: 101.9m
step 01191/05000 (23.82%) | loss: 4.363739 | lrm: 1.00 | dt: 1648.02ms | tok/sec: 9,941 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 52 | total time: 31.57m | eta: 101.8m
step 01192/05000 (23.84%) | loss: 4.358563 | lrm: 1.00 | dt: 1878.85ms | tok/sec: 8,720 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 52 | total time: 31.61m | eta: 101.8m
step 01193/05000 (23.86%) | loss: 4.363447 | lrm: 1.00 | dt: 1815.18ms | tok/sec: 9,026 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 52 | total time: 31.64m | eta: 101.8m
step 01194/05000 (23.88%) | loss: 4.362398 | lrm: 1.00 | dt: 1802.58ms | tok/sec: 9,089 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 52 | total time: 31.67m | eta: 101.8m
step 01195/05000 (23.90%) | loss: 4.345164 | lrm: 1.00 | dt: 1752.17ms | tok/sec: 9,350 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 52 | total time: 31.70m | eta: 101.8m
step 01196/05000 (23.92%) | loss: 4.354272 | lrm: 1.00 | dt: 1790.64ms | tok/sec: 9,149 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 52 | total time: 31.72m | eta: 101.8m
step 01197/05000 (23.94%) | loss: 4.340601 | lrm: 1.00 | dt: 1780.60ms | tok/sec: 9,201 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 52 | total time: 31.75m | eta: 101.7m
step 01198/05000 (23.96%) | loss: 4.338346 | lrm: 1.00 | dt: 1782.35ms | tok/sec: 9,192 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 52 | total time: 31.78m | eta: 101.7m
step 01199/05000 (23.98%) | loss: 4.351600 | lrm: 1.00 | dt: 1800.53ms | tok/sec: 9,099 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 52 | total time: 31.81m | eta: 101.7m
Step 01200 | Validation bpb: 1.345296
<|bos|>The capital of France is the capital of the world's largest city. The capital of France is the capital
<|bos|>The chemical symbol of gold is a symbol of the beauty of the gold. The gold is a symbol of the
<|bos|>If yesterday was Friday, then tomorrow will be the first time you've been in the country. You've been in the country
<|bos|>The opposite of hot is true. It is true that hot water is not the same as hot water.
<|bos|>The planets of the solar system are: the sun, the sun, the sun, the sun, the sun, the
<|bos|>My favorite color is the color of the color. I have a color of color that is not a
<|bos|>If 5*x + 3 = 13, then x is 5*x + 3.5*x + 3.5
step 01200/05000 (24.00%) | loss: 4.349391 | lrm: 1.00 | dt: 1820.11ms | tok/sec: 9,001 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 53 | total time: 31.84m | eta: 101.7m
step 01201/05000 (24.02%) | loss: 4.343875 | lrm: 1.00 | dt: 1800.33ms | tok/sec: 9,100 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 53 | total time: 31.87m | eta: 101.7m
step 01202/05000 (24.04%) | loss: 4.340484 | lrm: 1.00 | dt: 1804.33ms | tok/sec: 9,080 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 53 | total time: 31.90m | eta: 101.7m
step 01203/05000 (24.06%) | loss: 4.335483 | lrm: 1.00 | dt: 1801.68ms | tok/sec: 9,093 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 53 | total time: 31.93m | eta: 101.6m
step 01204/05000 (24.08%) | loss: 4.334934 | lrm: 1.00 | dt: 1773.04ms | tok/sec: 9,240 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 53 | total time: 31.96m | eta: 101.6m
step 01205/05000 (24.10%) | loss: 4.348706 | lrm: 1.00 | dt: 1800.94ms | tok/sec: 9,097 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 53 | total time: 31.99m | eta: 101.6m
step 01206/05000 (24.12%) | loss: 4.337696 | lrm: 1.00 | dt: 1787.18ms | tok/sec: 9,167 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 53 | total time: 32.02m | eta: 101.6m
step 01207/05000 (24.14%) | loss: 4.327702 | lrm: 1.00 | dt: 1774.44ms | tok/sec: 9,233 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 53 | total time: 32.05m | eta: 101.6m
step 01208/05000 (24.16%) | loss: 4.331804 | lrm: 1.00 | dt: 1769.27ms | tok/sec: 9,260 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 53 | total time: 32.08m | eta: 101.6m
step 01209/05000 (24.18%) | loss: 4.318107 | lrm: 1.00 | dt: 1781.64ms | tok/sec: 9,196 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 53 | total time: 32.11m | eta: 101.5m
step 01210/05000 (24.20%) | loss: 4.329831 | lrm: 1.00 | dt: 1779.63ms | tok/sec: 9,206 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 53 | total time: 32.14m | eta: 101.5m
step 01211/05000 (24.22%) | loss: 4.336820 | lrm: 1.00 | dt: 1768.18ms | tok/sec: 9,266 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 53 | total time: 32.17m | eta: 101.5m
step 01212/05000 (24.24%) | loss: 4.325962 | lrm: 1.00 | dt: 1792.66ms | tok/sec: 9,139 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 53 | total time: 32.20m | eta: 101.5m
step 01213/05000 (24.26%) | loss: 4.343142 | lrm: 1.00 | dt: 1755.96ms | tok/sec: 9,330 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 53 | total time: 32.23m | eta: 101.5m
step 01214/05000 (24.28%) | loss: 4.331011 | lrm: 1.00 | dt: 1795.60ms | tok/sec: 9,124 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 53 | total time: 32.26m | eta: 101.4m
step 01215/05000 (24.30%) | loss: 4.328246 | lrm: 1.00 | dt: 1795.03ms | tok/sec: 9,127 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 53 | total time: 32.29m | eta: 101.4m
step 01216/05000 (24.32%) | loss: 4.330728 | lrm: 1.00 | dt: 1794.84ms | tok/sec: 9,128 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 53 | total time: 32.32m | eta: 101.4m
step 01217/05000 (24.34%) | loss: 4.341340 | lrm: 1.00 | dt: 1787.60ms | tok/sec: 9,165 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 53 | total time: 32.35m | eta: 101.4m
step 01218/05000 (24.36%) | loss: 4.338616 | lrm: 1.00 | dt: 1790.29ms | tok/sec: 9,151 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 53 | total time: 32.38m | eta: 101.4m
step 01219/05000 (24.38%) | loss: 4.347155 | lrm: 1.00 | dt: 1801.05ms | tok/sec: 9,096 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 53 | total time: 32.41m | eta: 101.4m
step 01220/05000 (24.40%) | loss: 4.336559 | lrm: 1.00 | dt: 1740.83ms | tok/sec: 9,411 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 53 | total time: 32.44m | eta: 101.3m
step 01221/05000 (24.42%) | loss: 4.338281 | lrm: 1.00 | dt: 1784.96ms | tok/sec: 9,178 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 53 | total time: 32.47m | eta: 101.3m
step 01222/05000 (24.44%) | loss: 4.341933 | lrm: 1.00 | dt: 1771.64ms | tok/sec: 9,247 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 53 | total time: 32.50m | eta: 101.3m
step 01223/05000 (24.46%) | loss: 4.334276 | lrm: 1.00 | dt: 1791.71ms | tok/sec: 9,144 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 54 | total time: 32.53m | eta: 101.3m
step 01224/05000 (24.48%) | loss: 4.331582 | lrm: 1.00 | dt: 1754.91ms | tok/sec: 9,336 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 54 | total time: 32.56m | eta: 101.3m
step 01225/05000 (24.50%) | loss: 4.345626 | lrm: 1.00 | dt: 1785.02ms | tok/sec: 9,178 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 54 | total time: 32.59m | eta: 101.2m
step 01226/05000 (24.52%) | loss: 4.353255 | lrm: 1.00 | dt: 1796.19ms | tok/sec: 9,121 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 54 | total time: 32.62m | eta: 101.2m
step 01227/05000 (24.54%) | loss: 4.364013 | lrm: 1.00 | dt: 1786.53ms | tok/sec: 9,170 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 54 | total time: 32.65m | eta: 101.2m
step 01228/05000 (24.56%) | loss: 4.355681 | lrm: 1.00 | dt: 1781.05ms | tok/sec: 9,199 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 54 | total time: 32.68m | eta: 101.2m
step 01229/05000 (24.58%) | loss: 4.340360 | lrm: 1.00 | dt: 1782.60ms | tok/sec: 9,191 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 54 | total time: 32.71m | eta: 101.2m
step 01230/05000 (24.60%) | loss: 4.317975 | lrm: 1.00 | dt: 1769.16ms | tok/sec: 9,260 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 54 | total time: 32.74m | eta: 101.2m
step 01231/05000 (24.62%) | loss: 4.327649 | lrm: 1.00 | dt: 1797.66ms | tok/sec: 9,114 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 54 | total time: 32.77m | eta: 101.1m
step 01232/05000 (24.64%) | loss: 4.325729 | lrm: 1.00 | dt: 1767.95ms | tok/sec: 9,267 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 54 | total time: 32.80m | eta: 101.1m
step 01233/05000 (24.66%) | loss: 4.328745 | lrm: 1.00 | dt: 1774.37ms | tok/sec: 9,233 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 54 | total time: 32.83m | eta: 101.1m
step 01234/05000 (24.68%) | loss: 4.340411 | lrm: 1.00 | dt: 1779.42ms | tok/sec: 9,207 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 54 | total time: 32.85m | eta: 101.1m
step 01235/05000 (24.70%) | loss: 4.336536 | lrm: 1.00 | dt: 1794.21ms | tok/sec: 9,131 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 54 | total time: 32.88m | eta: 101.1m
step 01236/05000 (24.72%) | loss: 4.340037 | lrm: 1.00 | dt: 1770.63ms | tok/sec: 9,253 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 54 | total time: 32.91m | eta: 101.1m
step 01237/05000 (24.74%) | loss: 4.331620 | lrm: 1.00 | dt: 1831.17ms | tok/sec: 8,947 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 54 | total time: 32.94m | eta: 101.0m
step 01238/05000 (24.76%) | loss: 4.355006 | lrm: 1.00 | dt: 1752.13ms | tok/sec: 9,350 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 54 | total time: 32.97m | eta: 101.0m
step 01239/05000 (24.78%) | loss: 4.352965 | lrm: 1.00 | dt: 1779.97ms | tok/sec: 9,204 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 54 | total time: 33.00m | eta: 101.0m
step 01240/05000 (24.80%) | loss: 4.352548 | lrm: 1.00 | dt: 1765.95ms | tok/sec: 9,277 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 54 | total time: 33.03m | eta: 101.0m
step 01241/05000 (24.82%) | loss: 4.344462 | lrm: 1.00 | dt: 1753.55ms | tok/sec: 9,343 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 54 | total time: 33.06m | eta: 101.0m
step 01242/05000 (24.84%) | loss: 4.338988 | lrm: 1.00 | dt: 1780.34ms | tok/sec: 9,202 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 54 | total time: 33.09m | eta: 100.9m
step 01243/05000 (24.86%) | loss: 4.346895 | lrm: 1.00 | dt: 1761.38ms | tok/sec: 9,301 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 54 | total time: 33.12m | eta: 100.9m
step 01244/05000 (24.88%) | loss: 4.359669 | lrm: 1.00 | dt: 1797.70ms | tok/sec: 9,113 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 54 | total time: 33.15m | eta: 100.9m
step 01245/05000 (24.90%) | loss: 4.336861 | lrm: 1.00 | dt: 1761.33ms | tok/sec: 9,302 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 54 | total time: 33.18m | eta: 100.9m
step 01246/05000 (24.92%) | loss: 4.325605 | lrm: 1.00 | dt: 1787.64ms | tok/sec: 9,165 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 54 | total time: 33.21m | eta: 100.9m
step 01247/05000 (24.94%) | loss: 4.310837 | lrm: 1.00 | dt: 1762.65ms | tok/sec: 9,295 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 55 | total time: 33.24m | eta: 100.8m
step 01248/05000 (24.96%) | loss: 4.307501 | lrm: 1.00 | dt: 1822.18ms | tok/sec: 8,991 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 55 | total time: 33.27m | eta: 100.8m
step 01249/05000 (24.98%) | loss: 4.313655 | lrm: 1.00 | dt: 1779.10ms | tok/sec: 9,209 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 55 | total time: 33.30m | eta: 100.8m
step 01250/05000 (25.00%) | loss: 4.316629 | lrm: 1.00 | dt: 1781.47ms | tok/sec: 9,196 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 55 | total time: 33.33m | eta: 100.8m
step 01251/05000 (25.02%) | loss: 4.319860 | lrm: 1.00 | dt: 1770.50ms | tok/sec: 9,253 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 55 | total time: 33.36m | eta: 100.8m
step 01252/05000 (25.04%) | loss: 4.309985 | lrm: 1.00 | dt: 1793.99ms | tok/sec: 9,132 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 55 | total time: 33.39m | eta: 100.8m
step 01253/05000 (25.06%) | loss: 4.312297 | lrm: 1.00 | dt: 1779.32ms | tok/sec: 9,208 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 55 | total time: 33.42m | eta: 100.7m
step 01254/05000 (25.08%) | loss: 4.317872 | lrm: 1.00 | dt: 1783.94ms | tok/sec: 9,184 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 55 | total time: 33.45m | eta: 100.7m
step 01255/05000 (25.10%) | loss: 4.322844 | lrm: 1.00 | dt: 1783.88ms | tok/sec: 9,184 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 55 | total time: 33.48m | eta: 100.7m
step 01256/05000 (25.12%) | loss: 4.314359 | lrm: 1.00 | dt: 1800.76ms | tok/sec: 9,098 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 55 | total time: 33.51m | eta: 100.7m
step 01257/05000 (25.14%) | loss: 4.310870 | lrm: 1.00 | dt: 1791.45ms | tok/sec: 9,145 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 55 | total time: 33.54m | eta: 100.7m
step 01258/05000 (25.16%) | loss: 4.324153 | lrm: 1.00 | dt: 1779.90ms | tok/sec: 9,205 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 55 | total time: 33.57m | eta: 100.6m
step 01259/05000 (25.18%) | loss: 4.308726 | lrm: 1.00 | dt: 1777.52ms | tok/sec: 9,217 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 55 | total time: 33.60m | eta: 100.6m
step 01260/05000 (25.20%) | loss: 4.310389 | lrm: 1.00 | dt: 1791.10ms | tok/sec: 9,147 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 55 | total time: 33.63m | eta: 100.6m
step 01261/05000 (25.22%) | loss: 4.318628 | lrm: 1.00 | dt: 1786.73ms | tok/sec: 9,169 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 55 | total time: 33.66m | eta: 100.6m
step 01262/05000 (25.24%) | loss: 4.313829 | lrm: 1.00 | dt: 1778.12ms | tok/sec: 9,214 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 55 | total time: 33.69m | eta: 100.6m
step 01263/05000 (25.26%) | loss: 4.325716 | lrm: 1.00 | dt: 1803.05ms | tok/sec: 9,086 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 55 | total time: 33.72m | eta: 100.6m
step 01264/05000 (25.28%) | loss: 4.309129 | lrm: 1.00 | dt: 1791.36ms | tok/sec: 9,146 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 55 | total time: 33.75m | eta: 100.5m
step 01265/05000 (25.30%) | loss: 4.318512 | lrm: 1.00 | dt: 1780.24ms | tok/sec: 9,203 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 55 | total time: 33.78m | eta: 100.5m
step 01266/05000 (25.32%) | loss: 4.321066 | lrm: 1.00 | dt: 1771.41ms | tok/sec: 9,249 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 55 | total time: 33.81m | eta: 100.5m
step 01267/05000 (25.34%) | loss: 4.317265 | lrm: 1.00 | dt: 1774.52ms | tok/sec: 9,232 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 55 | total time: 33.84m | eta: 100.5m
step 01268/05000 (25.36%) | loss: 4.312391 | lrm: 1.00 | dt: 1767.49ms | tok/sec: 9,269 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 55 | total time: 33.86m | eta: 100.5m
step 01269/05000 (25.38%) | loss: 4.318059 | lrm: 1.00 | dt: 1773.36ms | tok/sec: 9,238 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 55 | total time: 33.89m | eta: 100.4m
step 01270/05000 (25.40%) | loss: 4.314687 | lrm: 1.00 | dt: 1757.40ms | tok/sec: 9,322 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 56 | total time: 33.92m | eta: 100.4m
step 01271/05000 (25.42%) | loss: 4.322987 | lrm: 1.00 | dt: 1792.44ms | tok/sec: 9,140 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 56 | total time: 33.95m | eta: 100.4m
step 01272/05000 (25.44%) | loss: 4.331443 | lrm: 1.00 | dt: 1786.20ms | tok/sec: 9,172 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 56 | total time: 33.98m | eta: 100.4m
step 01273/05000 (25.46%) | loss: 4.316192 | lrm: 1.00 | dt: 1781.83ms | tok/sec: 9,195 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 56 | total time: 34.01m | eta: 100.4m
step 01274/05000 (25.48%) | loss: 4.323672 | lrm: 1.00 | dt: 1770.80ms | tok/sec: 9,252 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 56 | total time: 34.04m | eta: 100.3m
step 01275/05000 (25.50%) | loss: 4.322337 | lrm: 1.00 | dt: 1796.23ms | tok/sec: 9,121 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 56 | total time: 34.07m | eta: 100.3m
step 01276/05000 (25.52%) | loss: 4.324337 | lrm: 1.00 | dt: 1801.78ms | tok/sec: 9,093 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 56 | total time: 34.10m | eta: 100.3m
step 01277/05000 (25.54%) | loss: 4.316876 | lrm: 1.00 | dt: 1794.86ms | tok/sec: 9,128 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 56 | total time: 34.13m | eta: 100.3m
step 01278/05000 (25.56%) | loss: 4.317959 | lrm: 1.00 | dt: 1788.54ms | tok/sec: 9,160 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 56 | total time: 34.16m | eta: 100.3m
step 01279/05000 (25.58%) | loss: 4.315957 | lrm: 1.00 | dt: 1766.28ms | tok/sec: 9,275 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 56 | total time: 34.19m | eta: 100.3m
step 01280/05000 (25.60%) | loss: 4.311764 | lrm: 1.00 | dt: 1764.93ms | tok/sec: 9,283 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 56 | total time: 34.22m | eta: 100.2m
step 01281/05000 (25.62%) | loss: 4.310216 | lrm: 1.00 | dt: 1778.46ms | tok/sec: 9,212 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 56 | total time: 34.25m | eta: 100.2m
step 01282/05000 (25.64%) | loss: 4.309534 | lrm: 1.00 | dt: 1793.59ms | tok/sec: 9,134 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 56 | total time: 34.28m | eta: 100.2m
step 01283/05000 (25.66%) | loss: 4.318074 | lrm: 1.00 | dt: 1795.97ms | tok/sec: 9,122 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 56 | total time: 34.31m | eta: 100.2m
step 01284/05000 (25.68%) | loss: 4.321093 | lrm: 1.00 | dt: 1787.83ms | tok/sec: 9,164 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 56 | total time: 34.34m | eta: 100.2m
step 01285/05000 (25.70%) | loss: 4.317577 | lrm: 1.00 | dt: 1780.48ms | tok/sec: 9,201 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 56 | total time: 34.37m | eta: 100.1m
step 01286/05000 (25.72%) | loss: 4.301286 | lrm: 1.00 | dt: 1798.76ms | tok/sec: 9,108 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 56 | total time: 34.40m | eta: 100.1m
step 01287/05000 (25.74%) | loss: 4.317613 | lrm: 1.00 | dt: 1733.59ms | tok/sec: 9,450 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 56 | total time: 34.43m | eta: 100.1m
step 01288/05000 (25.76%) | loss: 4.319363 | lrm: 1.00 | dt: 1800.42ms | tok/sec: 9,100 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 56 | total time: 34.46m | eta: 100.1m
step 01289/05000 (25.78%) | loss: 4.302217 | lrm: 1.00 | dt: 1785.74ms | tok/sec: 9,174 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 56 | total time: 34.49m | eta: 100.1m
step 01290/05000 (25.80%) | loss: 4.284711 | lrm: 1.00 | dt: 1794.64ms | tok/sec: 9,129 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 56 | total time: 34.52m | eta: 100.0m
step 01291/05000 (25.82%) | loss: 4.284813 | lrm: 1.00 | dt: 1777.47ms | tok/sec: 9,217 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 56 | total time: 34.55m | eta: 100.0m
step 01292/05000 (25.84%) | loss: 4.282931 | lrm: 1.00 | dt: 1789.29ms | tok/sec: 9,156 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 56 | total time: 34.58m | eta: 100.0m
step 01293/05000 (25.86%) | loss: 4.287718 | lrm: 1.00 | dt: 1800.45ms | tok/sec: 9,099 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 57 | total time: 34.61m | eta: 100.0m
step 01294/05000 (25.88%) | loss: 4.298191 | lrm: 1.00 | dt: 1767.12ms | tok/sec: 9,271 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 57 | total time: 34.64m | eta: 100.0m
step 01295/05000 (25.90%) | loss: 4.304669 | lrm: 1.00 | dt: 1794.57ms | tok/sec: 9,129 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 57 | total time: 34.67m | eta: 100.0m
step 01296/05000 (25.92%) | loss: 4.294258 | lrm: 1.00 | dt: 1771.20ms | tok/sec: 9,250 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 57 | total time: 34.70m | eta: 99.9m
step 01297/05000 (25.94%) | loss: 4.302398 | lrm: 1.00 | dt: 1789.17ms | tok/sec: 9,157 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 57 | total time: 34.73m | eta: 99.9m
step 01298/05000 (25.96%) | loss: 4.295694 | lrm: 1.00 | dt: 1806.04ms | tok/sec: 9,071 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 57 | total time: 34.76m | eta: 99.9m
step 01299/05000 (25.98%) | loss: 4.287201 | lrm: 1.00 | dt: 1788.42ms | tok/sec: 9,161 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 57 | total time: 34.79m | eta: 99.9m
Step 01300 | Validation bpb: 1.332653
<|bos|>The capital of France is the largest, most important part of the world. The capital of France is the
<|bos|>The chemical symbol of gold is a symbol of gold, a symbol of gold, and a symbol of gold.
<|bos|>If yesterday was Friday, then tomorrow will be the first time in the world. The first time in the world is the first
<|bos|>The opposite of hot is true. The heat is a very high heat, which is a very high heat
<|bos|>The planets of the solar system are: the planets of the solar system are the same as the planets of the solar system
<|bos|>My favorite color is the color of the color. The color of the color is the color of the
<|bos|>If 5*x + 3 = 13, then x is 5*x + 5*x + 5*x +
step 01300/05000 (26.00%) | loss: 4.297047 | lrm: 1.00 | dt: 1891.41ms | tok/sec: 8,662 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 57 | total time: 34.82m | eta: 99.9m
step 01301/05000 (26.02%) | loss: 4.299280 | lrm: 1.00 | dt: 1753.24ms | tok/sec: 9,344 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 57 | total time: 34.85m | eta: 99.8m
step 01302/05000 (26.04%) | loss: 4.294815 | lrm: 1.00 | dt: 1767.65ms | tok/sec: 9,268 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 57 | total time: 34.88m | eta: 99.8m
step 01303/05000 (26.06%) | loss: 4.306054 | lrm: 1.00 | dt: 1799.51ms | tok/sec: 9,104 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 57 | total time: 34.91m | eta: 99.8m
step 01304/05000 (26.08%) | loss: 4.298307 | lrm: 1.00 | dt: 1806.03ms | tok/sec: 9,071 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 57 | total time: 34.94m | eta: 99.8m
step 01305/05000 (26.10%) | loss: 4.300929 | lrm: 1.00 | dt: 1772.71ms | tok/sec: 9,242 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 57 | total time: 34.97m | eta: 99.8m
step 01306/05000 (26.12%) | loss: 4.315427 | lrm: 1.00 | dt: 1765.32ms | tok/sec: 9,281 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 57 | total time: 35.00m | eta: 99.7m
step 01307/05000 (26.14%) | loss: 4.324407 | lrm: 1.00 | dt: 1756.60ms | tok/sec: 9,327 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 57 | total time: 35.03m | eta: 99.7m
step 01308/05000 (26.16%) | loss: 4.321816 | lrm: 1.00 | dt: 1791.55ms | tok/sec: 9,145 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 57 | total time: 35.05m | eta: 99.7m
step 01309/05000 (26.18%) | loss: 4.322707 | lrm: 1.00 | dt: 1783.99ms | tok/sec: 9,183 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 57 | total time: 35.08m | eta: 99.7m
step 01310/05000 (26.20%) | loss: 4.312563 | lrm: 1.00 | dt: 1777.56ms | tok/sec: 9,217 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 57 | total time: 35.11m | eta: 99.7m
step 01311/05000 (26.22%) | loss: 4.322012 | lrm: 1.00 | dt: 1777.73ms | tok/sec: 9,216 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 57 | total time: 35.14m | eta: 99.7m
step 01312/05000 (26.24%) | loss: 4.309090 | lrm: 1.00 | dt: 1782.79ms | tok/sec: 9,190 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 57 | total time: 35.17m | eta: 99.6m
step 01313/05000 (26.26%) | loss: 4.299032 | lrm: 1.00 | dt: 1763.33ms | tok/sec: 9,291 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 57 | total time: 35.20m | eta: 99.6m
step 01314/05000 (26.28%) | loss: 4.305215 | lrm: 1.00 | dt: 1759.96ms | tok/sec: 9,309 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 57 | total time: 35.23m | eta: 99.6m
step 01315/05000 (26.30%) | loss: 4.298964 | lrm: 1.00 | dt: 1784.03ms | tok/sec: 9,183 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 57 | total time: 35.26m | eta: 99.6m
step 01316/05000 (26.32%) | loss: 4.297839 | lrm: 1.00 | dt: 1770.32ms | tok/sec: 9,254 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 58 | total time: 35.29m | eta: 99.6m
step 01317/05000 (26.34%) | loss: 4.296632 | lrm: 1.00 | dt: 1773.99ms | tok/sec: 9,235 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 58 | total time: 35.32m | eta: 99.5m
step 01318/05000 (26.36%) | loss: 4.290214 | lrm: 1.00 | dt: 1801.17ms | tok/sec: 9,096 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 58 | total time: 35.35m | eta: 99.5m
step 01319/05000 (26.38%) | loss: 4.280398 | lrm: 1.00 | dt: 1812.20ms | tok/sec: 9,040 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 58 | total time: 35.38m | eta: 99.5m
step 01320/05000 (26.40%) | loss: 4.279416 | lrm: 1.00 | dt: 1750.77ms | tok/sec: 9,358 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 58 | total time: 35.41m | eta: 99.5m
step 01321/05000 (26.42%) | loss: 4.267018 | lrm: 1.00 | dt: 1778.34ms | tok/sec: 9,213 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 58 | total time: 35.44m | eta: 99.5m
step 01322/05000 (26.44%) | loss: 4.268850 | lrm: 1.00 | dt: 1784.88ms | tok/sec: 9,179 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 58 | total time: 35.47m | eta: 99.4m
step 01323/05000 (26.46%) | loss: 4.284419 | lrm: 1.00 | dt: 1777.54ms | tok/sec: 9,217 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 58 | total time: 35.50m | eta: 99.4m
step 01324/05000 (26.48%) | loss: 4.279309 | lrm: 1.00 | dt: 1747.11ms | tok/sec: 9,377 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 58 | total time: 35.53m | eta: 99.4m
step 01325/05000 (26.50%) | loss: 4.290834 | lrm: 1.00 | dt: 1814.14ms | tok/sec: 9,031 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 58 | total time: 35.56m | eta: 99.4m
step 01326/05000 (26.52%) | loss: 4.305425 | lrm: 1.00 | dt: 1844.43ms | tok/sec: 8,882 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 58 | total time: 35.59m | eta: 99.4m
step 01327/05000 (26.54%) | loss: 4.293362 | lrm: 1.00 | dt: 1797.17ms | tok/sec: 9,116 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 58 | total time: 35.62m | eta: 99.3m
step 01328/05000 (26.56%) | loss: 4.302071 | lrm: 1.00 | dt: 1769.50ms | tok/sec: 9,259 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 58 | total time: 35.65m | eta: 99.3m
step 01329/05000 (26.58%) | loss: 4.305182 | lrm: 1.00 | dt: 1802.60ms | tok/sec: 9,089 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 58 | total time: 35.68m | eta: 99.3m
step 01330/05000 (26.60%) | loss: 4.304088 | lrm: 1.00 | dt: 1742.30ms | tok/sec: 9,403 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 58 | total time: 35.71m | eta: 99.3m
step 01331/05000 (26.62%) | loss: 4.285828 | lrm: 1.00 | dt: 1793.00ms | tok/sec: 9,137 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 58 | total time: 35.74m | eta: 99.3m
step 01332/05000 (26.64%) | loss: 4.293785 | lrm: 1.00 | dt: 1787.56ms | tok/sec: 9,165 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 58 | total time: 35.77m | eta: 99.2m
step 01333/05000 (26.66%) | loss: 4.277508 | lrm: 1.00 | dt: 1797.72ms | tok/sec: 9,113 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 58 | total time: 35.80m | eta: 99.2m
step 01334/05000 (26.68%) | loss: 4.267855 | lrm: 1.00 | dt: 1762.99ms | tok/sec: 9,293 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 58 | total time: 35.83m | eta: 99.2m
step 01335/05000 (26.70%) | loss: 4.271924 | lrm: 1.00 | dt: 1795.15ms | tok/sec: 9,126 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 58 | total time: 35.86m | eta: 99.2m
step 01336/05000 (26.72%) | loss: 4.275311 | lrm: 1.00 | dt: 1791.26ms | tok/sec: 9,146 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 58 | total time: 35.89m | eta: 99.2m
step 01337/05000 (26.74%) | loss: 4.282497 | lrm: 1.00 | dt: 1819.71ms | tok/sec: 9,003 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 58 | total time: 35.92m | eta: 99.1m
step 01338/05000 (26.76%) | loss: 4.279818 | lrm: 1.00 | dt: 1798.34ms | tok/sec: 9,110 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 58 | total time: 35.95m | eta: 99.1m
step 01339/05000 (26.78%) | loss: 4.283826 | lrm: 1.00 | dt: 1796.72ms | tok/sec: 9,118 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 59 | total time: 35.98m | eta: 99.1m
step 01340/05000 (26.80%) | loss: 4.275927 | lrm: 1.00 | dt: 1793.95ms | tok/sec: 9,132 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 59 | total time: 36.01m | eta: 99.1m
step 01341/05000 (26.82%) | loss: 4.270532 | lrm: 1.00 | dt: 1789.34ms | tok/sec: 9,156 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 59 | total time: 36.04m | eta: 99.1m
step 01342/05000 (26.84%) | loss: 4.260562 | lrm: 1.00 | dt: 1756.73ms | tok/sec: 9,326 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 59 | total time: 36.07m | eta: 99.0m
step 01343/05000 (26.86%) | loss: 4.260003 | lrm: 1.00 | dt: 1779.50ms | tok/sec: 9,207 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 59 | total time: 36.10m | eta: 99.0m
step 01344/05000 (26.88%) | loss: 4.250827 | lrm: 1.00 | dt: 1778.52ms | tok/sec: 9,212 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 59 | total time: 36.13m | eta: 99.0m
step 01345/05000 (26.90%) | loss: 4.254918 | lrm: 1.00 | dt: 1771.32ms | tok/sec: 9,249 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 59 | total time: 36.16m | eta: 99.0m
step 01346/05000 (26.92%) | loss: 4.263372 | lrm: 1.00 | dt: 1782.32ms | tok/sec: 9,192 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 59 | total time: 36.18m | eta: 99.0m
step 01347/05000 (26.94%) | loss: 4.252798 | lrm: 1.00 | dt: 1743.56ms | tok/sec: 9,396 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 59 | total time: 36.21m | eta: 98.9m
step 01348/05000 (26.96%) | loss: 4.239524 | lrm: 1.00 | dt: 1601.38ms | tok/sec: 10,231 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 59 | total time: 36.24m | eta: 98.9m
step 01349/05000 (26.98%) | loss: 4.252756 | lrm: 1.00 | dt: 1569.16ms | tok/sec: 10,441 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 59 | total time: 36.27m | eta: 98.9m
step 01350/05000 (27.00%) | loss: 4.272419 | lrm: 1.00 | dt: 1553.32ms | tok/sec: 10,547 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 59 | total time: 36.29m | eta: 98.9m
step 01351/05000 (27.02%) | loss: 4.273136 | lrm: 1.00 | dt: 1620.18ms | tok/sec: 10,112 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 59 | total time: 36.32m | eta: 98.8m
step 01352/05000 (27.04%) | loss: 4.270419 | lrm: 1.00 | dt: 1548.89ms | tok/sec: 10,577 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 59 | total time: 36.35m | eta: 98.8m
step 01353/05000 (27.06%) | loss: 4.283051 | lrm: 1.00 | dt: 1546.36ms | tok/sec: 10,595 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 59 | total time: 36.37m | eta: 98.8m
step 01354/05000 (27.08%) | loss: 4.281912 | lrm: 1.00 | dt: 1550.82ms | tok/sec: 10,564 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 59 | total time: 36.40m | eta: 98.7m
step 01355/05000 (27.10%) | loss: 4.295769 | lrm: 1.00 | dt: 1557.21ms | tok/sec: 10,521 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 59 | total time: 36.42m | eta: 98.7m
step 01356/05000 (27.12%) | loss: 4.306394 | lrm: 1.00 | dt: 1567.91ms | tok/sec: 10,449 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 59 | total time: 36.45m | eta: 98.7m
step 01357/05000 (27.14%) | loss: 4.291228 | lrm: 1.00 | dt: 1516.18ms | tok/sec: 10,806 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 59 | total time: 36.47m | eta: 98.6m
step 01358/05000 (27.16%) | loss: 4.305954 | lrm: 1.00 | dt: 1552.41ms | tok/sec: 10,553 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 59 | total time: 36.50m | eta: 98.6m
step 01359/05000 (27.18%) | loss: 4.301331 | lrm: 1.00 | dt: 1532.66ms | tok/sec: 10,689 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 59 | total time: 36.53m | eta: 98.6m
step 01360/05000 (27.20%) | loss: 4.296763 | lrm: 1.00 | dt: 1551.14ms | tok/sec: 10,562 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 59 | total time: 36.55m | eta: 98.6m
step 01361/05000 (27.22%) | loss: 4.293615 | lrm: 1.00 | dt: 1513.81ms | tok/sec: 10,823 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 59 | total time: 36.58m | eta: 98.5m
step 01362/05000 (27.24%) | loss: 4.294816 | lrm: 1.00 | dt: 1589.09ms | tok/sec: 10,310 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 60 | total time: 36.60m | eta: 98.5m
step 01363/05000 (27.26%) | loss: 4.276887 | lrm: 1.00 | dt: 1562.29ms | tok/sec: 10,487 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 60 | total time: 36.63m | eta: 98.5m
step 01364/05000 (27.28%) | loss: 4.294857 | lrm: 1.00 | dt: 1534.19ms | tok/sec: 10,679 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 60 | total time: 36.65m | eta: 98.4m
step 01365/05000 (27.30%) | loss: 4.288781 | lrm: 1.00 | dt: 1537.27ms | tok/sec: 10,657 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 60 | total time: 36.68m | eta: 98.4m
step 01366/05000 (27.32%) | loss: 4.306993 | lrm: 1.00 | dt: 1559.02ms | tok/sec: 10,509 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 60 | total time: 36.71m | eta: 98.4m
step 01367/05000 (27.34%) | loss: 4.310947 | lrm: 1.00 | dt: 1524.16ms | tok/sec: 10,749 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 60 | total time: 36.73m | eta: 98.3m
step 01368/05000 (27.36%) | loss: 4.302053 | lrm: 1.00 | dt: 1565.25ms | tok/sec: 10,467 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 60 | total time: 36.76m | eta: 98.3m
step 01369/05000 (27.38%) | loss: 4.311622 | lrm: 1.00 | dt: 1571.41ms | tok/sec: 10,426 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 60 | total time: 36.78m | eta: 98.3m
step 01370/05000 (27.40%) | loss: 4.306308 | lrm: 1.00 | dt: 1545.20ms | tok/sec: 10,603 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 60 | total time: 36.81m | eta: 98.3m
step 01371/05000 (27.42%) | loss: 4.282222 | lrm: 1.00 | dt: 1544.73ms | tok/sec: 10,606 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 60 | total time: 36.84m | eta: 98.2m
step 01372/05000 (27.44%) | loss: 4.282707 | lrm: 1.00 | dt: 1561.54ms | tok/sec: 10,492 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 60 | total time: 36.86m | eta: 98.2m
step 01373/05000 (27.46%) | loss: 4.278231 | lrm: 1.00 | dt: 1529.47ms | tok/sec: 10,712 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 60 | total time: 36.89m | eta: 98.2m
step 01374/05000 (27.48%) | loss: 4.289395 | lrm: 1.00 | dt: 1546.29ms | tok/sec: 10,595 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 60 | total time: 36.91m | eta: 98.1m
step 01375/05000 (27.50%) | loss: 4.295307 | lrm: 1.00 | dt: 1576.17ms | tok/sec: 10,394 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 60 | total time: 36.94m | eta: 98.1m
step 01376/05000 (27.52%) | loss: 4.292979 | lrm: 1.00 | dt: 1533.40ms | tok/sec: 10,684 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 60 | total time: 36.96m | eta: 98.1m
step 01377/05000 (27.54%) | loss: 4.294007 | lrm: 1.00 | dt: 1540.26ms | tok/sec: 10,637 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 60 | total time: 36.99m | eta: 98.0m
step 01378/05000 (27.56%) | loss: 4.286361 | lrm: 1.00 | dt: 1544.02ms | tok/sec: 10,611 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 60 | total time: 37.02m | eta: 98.0m
step 01379/05000 (27.58%) | loss: 4.280206 | lrm: 1.00 | dt: 1557.87ms | tok/sec: 10,516 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 60 | total time: 37.04m | eta: 98.0m
step 01380/05000 (27.60%) | loss: 4.292577 | lrm: 1.00 | dt: 1542.07ms | tok/sec: 10,624 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 60 | total time: 37.07m | eta: 97.9m
step 01381/05000 (27.62%) | loss: 4.295577 | lrm: 1.00 | dt: 1519.98ms | tok/sec: 10,779 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 60 | total time: 37.09m | eta: 97.9m
step 01382/05000 (27.64%) | loss: 4.290614 | lrm: 1.00 | dt: 1515.45ms | tok/sec: 10,811 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 60 | total time: 37.12m | eta: 97.9m
step 01383/05000 (27.66%) | loss: 4.305427 | lrm: 1.00 | dt: 1536.69ms | tok/sec: 10,661 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 60 | total time: 37.14m | eta: 97.9m
step 01384/05000 (27.68%) | loss: 4.313749 | lrm: 1.00 | dt: 1554.70ms | tok/sec: 10,538 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 60 | total time: 37.17m | eta: 97.8m
step 01385/05000 (27.70%) | loss: 4.297447 | lrm: 1.00 | dt: 1578.88ms | tok/sec: 10,376 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 61 | total time: 37.20m | eta: 97.8m
step 01386/05000 (27.72%) | loss: 4.297695 | lrm: 1.00 | dt: 1559.86ms | tok/sec: 10,503 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 61 | total time: 37.22m | eta: 97.8m
step 01387/05000 (27.74%) | loss: 4.303947 | lrm: 1.00 | dt: 1588.32ms | tok/sec: 10,315 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 61 | total time: 37.25m | eta: 97.7m
step 01388/05000 (27.76%) | loss: 4.297170 | lrm: 1.00 | dt: 1551.12ms | tok/sec: 10,562 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 61 | total time: 37.27m | eta: 97.7m
step 01389/05000 (27.78%) | loss: 4.296125 | lrm: 1.00 | dt: 1571.95ms | tok/sec: 10,422 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 61 | total time: 37.30m | eta: 97.7m
step 01390/05000 (27.80%) | loss: 4.298205 | lrm: 1.00 | dt: 1546.17ms | tok/sec: 10,596 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 61 | total time: 37.33m | eta: 97.6m
step 01391/05000 (27.82%) | loss: 4.313230 | lrm: 1.00 | dt: 1542.44ms | tok/sec: 10,622 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 61 | total time: 37.35m | eta: 97.6m
step 01392/05000 (27.84%) | loss: 4.302201 | lrm: 1.00 | dt: 1561.73ms | tok/sec: 10,490 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 61 | total time: 37.38m | eta: 97.6m
step 01393/05000 (27.86%) | loss: 4.305194 | lrm: 1.00 | dt: 1540.65ms | tok/sec: 10,634 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 61 | total time: 37.40m | eta: 97.6m
step 01394/05000 (27.88%) | loss: 4.299330 | lrm: 1.00 | dt: 1532.60ms | tok/sec: 10,690 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 61 | total time: 37.43m | eta: 97.5m
step 01395/05000 (27.90%) | loss: 4.298747 | lrm: 1.00 | dt: 1569.99ms | tok/sec: 10,435 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 61 | total time: 37.46m | eta: 97.5m
step 01396/05000 (27.92%) | loss: 4.308686 | lrm: 1.00 | dt: 1551.42ms | tok/sec: 10,560 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 61 | total time: 37.48m | eta: 97.5m
step 01397/05000 (27.94%) | loss: 4.307898 | lrm: 1.00 | dt: 1543.35ms | tok/sec: 10,615 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 61 | total time: 37.51m | eta: 97.4m
step 01398/05000 (27.96%) | loss: 4.310230 | lrm: 1.00 | dt: 1536.50ms | tok/sec: 10,663 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 61 | total time: 37.53m | eta: 97.4m
step 01399/05000 (27.98%) | loss: 4.308737 | lrm: 1.00 | dt: 1547.29ms | tok/sec: 10,588 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 61 | total time: 37.56m | eta: 97.4m
Step 01400 | Validation bpb: 1.323681
<|bos|>The capital of France is the most important part of the world's most important part of the world. The
<|bos|>The chemical symbol of gold is a symbol of the symbol of the symbol of the symbol of the symbol of the
<|bos|>If yesterday was Friday, then tomorrow will be the first day of the year. The first day of the year, the first
<|bos|>The opposite of hot is true. The hot weather is a good thing. The hot weather is a good
<|bos|>The planets of the solar system are: the solar system is the most important part of the solar system. The solar system
<|bos|>My favorite color is the color of the color of the color. The color of the color is the
<|bos|>If 5*x + 3 = 13, then x is 1. The 3. The 3. The 3. The
step 01400/05000 (28.00%) | loss: 4.296358 | lrm: 1.00 | dt: 1595.51ms | tok/sec: 10,268 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 61 | total time: 37.59m | eta: 97.3m
step 01401/05000 (28.02%) | loss: 4.299813 | lrm: 1.00 | dt: 1560.99ms | tok/sec: 10,495 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 61 | total time: 37.61m | eta: 97.3m
step 01402/05000 (28.04%) | loss: 4.291694 | lrm: 1.00 | dt: 1521.74ms | tok/sec: 10,766 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 61 | total time: 37.64m | eta: 97.3m
step 01403/05000 (28.06%) | loss: 4.298055 | lrm: 1.00 | dt: 1561.58ms | tok/sec: 10,491 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 61 | total time: 37.66m | eta: 97.3m
step 01404/05000 (28.08%) | loss: 4.290308 | lrm: 1.00 | dt: 1564.86ms | tok/sec: 10,469 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 61 | total time: 37.69m | eta: 97.2m
step 01405/05000 (28.10%) | loss: 4.280370 | lrm: 1.00 | dt: 1556.72ms | tok/sec: 10,524 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 61 | total time: 37.71m | eta: 97.2m
step 01406/05000 (28.12%) | loss: 4.290250 | lrm: 1.00 | dt: 1528.93ms | tok/sec: 10,715 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 61 | total time: 37.74m | eta: 97.2m
step 01407/05000 (28.14%) | loss: 4.296154 | lrm: 1.00 | dt: 1556.61ms | tok/sec: 10,525 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 61 | total time: 37.77m | eta: 97.1m
step 01408/05000 (28.16%) | loss: 4.291761 | lrm: 1.00 | dt: 1566.16ms | tok/sec: 10,461 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 62 | total time: 37.79m | eta: 97.1m
step 01409/05000 (28.18%) | loss: 4.300620 | lrm: 1.00 | dt: 1550.86ms | tok/sec: 10,564 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 62 | total time: 37.82m | eta: 97.1m
step 01410/05000 (28.20%) | loss: 4.308403 | lrm: 1.00 | dt: 1763.67ms | tok/sec: 9,289 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 62 | total time: 37.85m | eta: 97.1m
step 01411/05000 (28.22%) | loss: 4.295833 | lrm: 1.00 | dt: 1753.56ms | tok/sec: 9,343 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 62 | total time: 37.88m | eta: 97.0m
step 01412/05000 (28.24%) | loss: 4.302884 | lrm: 1.00 | dt: 1816.61ms | tok/sec: 9,019 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 62 | total time: 37.91m | eta: 97.0m
step 01413/05000 (28.26%) | loss: 4.304703 | lrm: 1.00 | dt: 1878.07ms | tok/sec: 8,723 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 62 | total time: 37.94m | eta: 97.0m
step 01414/05000 (28.28%) | loss: 4.293716 | lrm: 1.00 | dt: 1997.37ms | tok/sec: 8,202 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 62 | total time: 37.97m | eta: 97.0m
step 01415/05000 (28.30%) | loss: 4.301053 | lrm: 1.00 | dt: 1742.49ms | tok/sec: 9,402 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 62 | total time: 38.00m | eta: 97.0m
step 01416/05000 (28.32%) | loss: 4.293479 | lrm: 1.00 | dt: 1660.02ms | tok/sec: 9,869 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 62 | total time: 38.03m | eta: 96.9m
step 01417/05000 (28.34%) | loss: 4.266694 | lrm: 1.00 | dt: 1646.24ms | tok/sec: 9,952 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 62 | total time: 38.06m | eta: 96.9m
step 01418/05000 (28.36%) | loss: 4.266734 | lrm: 1.00 | dt: 1621.32ms | tok/sec: 10,105 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 62 | total time: 38.08m | eta: 96.9m
step 01419/05000 (28.38%) | loss: 4.268179 | lrm: 1.00 | dt: 1600.56ms | tok/sec: 10,236 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 62 | total time: 38.11m | eta: 96.9m
step 01420/05000 (28.40%) | loss: 4.269618 | lrm: 1.00 | dt: 1596.79ms | tok/sec: 10,260 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 62 | total time: 38.14m | eta: 96.8m
step 01421/05000 (28.42%) | loss: 4.274519 | lrm: 1.00 | dt: 1600.28ms | tok/sec: 10,238 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 62 | total time: 38.16m | eta: 96.8m
step 01422/05000 (28.44%) | loss: 4.277629 | lrm: 1.00 | dt: 1629.86ms | tok/sec: 10,052 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 62 | total time: 38.19m | eta: 96.8m
step 01423/05000 (28.46%) | loss: 4.290884 | lrm: 1.00 | dt: 1773.41ms | tok/sec: 9,238 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 62 | total time: 38.22m | eta: 96.8m
step 01424/05000 (28.48%) | loss: 4.290403 | lrm: 1.00 | dt: 1887.45ms | tok/sec: 8,680 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 62 | total time: 38.25m | eta: 96.7m
step 01425/05000 (28.50%) | loss: 4.293064 | lrm: 1.00 | dt: 1786.25ms | tok/sec: 9,172 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 62 | total time: 38.28m | eta: 96.7m
step 01426/05000 (28.52%) | loss: 4.287501 | lrm: 1.00 | dt: 1655.61ms | tok/sec: 9,896 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 62 | total time: 38.31m | eta: 96.7m
step 01427/05000 (28.54%) | loss: 4.290157 | lrm: 1.00 | dt: 1726.73ms | tok/sec: 9,488 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 62 | total time: 38.34m | eta: 96.7m
step 01428/05000 (28.56%) | loss: 4.297905 | lrm: 1.00 | dt: 1722.66ms | tok/sec: 9,510 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 62 | total time: 38.37m | eta: 96.6m
step 01429/05000 (28.58%) | loss: 4.293073 | lrm: 1.00 | dt: 1608.99ms | tok/sec: 10,182 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 62 | total time: 38.39m | eta: 96.6m
step 01430/05000 (28.60%) | loss: 4.307938 | lrm: 1.00 | dt: 1676.49ms | tok/sec: 9,772 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 62 | total time: 38.42m | eta: 96.6m
step 01431/05000 (28.62%) | loss: 4.313562 | lrm: 1.00 | dt: 1616.15ms | tok/sec: 10,137 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 62 | total time: 38.45m | eta: 96.6m
step 01432/05000 (28.64%) | loss: 4.308754 | lrm: 1.00 | dt: 1703.05ms | tok/sec: 9,620 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 63 | total time: 38.48m | eta: 96.5m
step 01433/05000 (28.66%) | loss: 4.311173 | lrm: 1.00 | dt: 1610.55ms | tok/sec: 10,172 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 63 | total time: 38.50m | eta: 96.5m
step 01434/05000 (28.68%) | loss: 4.292291 | lrm: 1.00 | dt: 1563.61ms | tok/sec: 10,478 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 63 | total time: 38.53m | eta: 96.5m
step 01435/05000 (28.70%) | loss: 4.263844 | lrm: 1.00 | dt: 1543.77ms | tok/sec: 10,612 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 63 | total time: 38.55m | eta: 96.5m
step 01436/05000 (28.72%) | loss: 4.269383 | lrm: 1.00 | dt: 1559.54ms | tok/sec: 10,505 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 63 | total time: 38.58m | eta: 96.4m
step 01437/05000 (28.74%) | loss: 4.255203 | lrm: 1.00 | dt: 1596.74ms | tok/sec: 10,260 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 63 | total time: 38.61m | eta: 96.4m
step 01438/05000 (28.76%) | loss: 4.250534 | lrm: 1.00 | dt: 1596.59ms | tok/sec: 10,261 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 63 | total time: 38.63m | eta: 96.4m
step 01439/05000 (28.78%) | loss: 4.256065 | lrm: 1.00 | dt: 1541.37ms | tok/sec: 10,629 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 63 | total time: 38.66m | eta: 96.3m
step 01440/05000 (28.80%) | loss: 4.245725 | lrm: 1.00 | dt: 1596.09ms | tok/sec: 10,265 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 63 | total time: 38.69m | eta: 96.3m
step 01441/05000 (28.82%) | loss: 4.249653 | lrm: 1.00 | dt: 1668.10ms | tok/sec: 9,821 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 63 | total time: 38.71m | eta: 96.3m
step 01442/05000 (28.84%) | loss: 4.250921 | lrm: 1.00 | dt: 1555.33ms | tok/sec: 10,534 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 63 | total time: 38.74m | eta: 96.3m
step 01443/05000 (28.86%) | loss: 4.259844 | lrm: 1.00 | dt: 1577.22ms | tok/sec: 10,387 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 63 | total time: 38.77m | eta: 96.2m
step 01444/05000 (28.88%) | loss: 4.261380 | lrm: 1.00 | dt: 1572.83ms | tok/sec: 10,416 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 63 | total time: 38.79m | eta: 96.2m
step 01445/05000 (28.90%) | loss: 4.253027 | lrm: 1.00 | dt: 1604.40ms | tok/sec: 10,211 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 63 | total time: 38.82m | eta: 96.2m
step 01446/05000 (28.92%) | loss: 4.256201 | lrm: 1.00 | dt: 1624.60ms | tok/sec: 10,084 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 63 | total time: 38.85m | eta: 96.1m
step 01447/05000 (28.94%) | loss: 4.260563 | lrm: 1.00 | dt: 1622.31ms | tok/sec: 10,099 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 63 | total time: 38.87m | eta: 96.1m
step 01448/05000 (28.96%) | loss: 4.258489 | lrm: 1.00 | dt: 1623.98ms | tok/sec: 10,088 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 63 | total time: 38.90m | eta: 96.1m
step 01449/05000 (28.98%) | loss: 4.252995 | lrm: 1.00 | dt: 1645.23ms | tok/sec: 9,958 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 63 | total time: 38.93m | eta: 96.1m
step 01450/05000 (29.00%) | loss: 4.253913 | lrm: 1.00 | dt: 1681.03ms | tok/sec: 9,746 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 63 | total time: 38.96m | eta: 96.0m
step 01451/05000 (29.02%) | loss: 4.281522 | lrm: 1.00 | dt: 1681.32ms | tok/sec: 9,744 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 63 | total time: 38.98m | eta: 96.0m
step 01452/05000 (29.04%) | loss: 4.283034 | lrm: 1.00 | dt: 1623.37ms | tok/sec: 10,092 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 63 | total time: 39.01m | eta: 96.0m
step 01453/05000 (29.06%) | loss: 4.276030 | lrm: 1.00 | dt: 1584.94ms | tok/sec: 10,337 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 63 | total time: 39.04m | eta: 96.0m
step 01454/05000 (29.08%) | loss: 4.277705 | lrm: 1.00 | dt: 1552.23ms | tok/sec: 10,555 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 64 | total time: 39.06m | eta: 95.9m
step 01455/05000 (29.10%) | loss: 4.272210 | lrm: 1.00 | dt: 1570.07ms | tok/sec: 10,435 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 64 | total time: 39.09m | eta: 95.9m
step 01456/05000 (29.12%) | loss: 4.287269 | lrm: 1.00 | dt: 1676.80ms | tok/sec: 9,770 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 64 | total time: 39.12m | eta: 95.9m
step 01457/05000 (29.14%) | loss: 4.280705 | lrm: 1.00 | dt: 1580.08ms | tok/sec: 10,369 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 64 | total time: 39.14m | eta: 95.8m
step 01458/05000 (29.16%) | loss: 4.302430 | lrm: 1.00 | dt: 1567.57ms | tok/sec: 10,451 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 64 | total time: 39.17m | eta: 95.8m
step 01459/05000 (29.18%) | loss: 4.302404 | lrm: 1.00 | dt: 1536.02ms | tok/sec: 10,666 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 64 | total time: 39.20m | eta: 95.8m
step 01460/05000 (29.20%) | loss: 4.298995 | lrm: 1.00 | dt: 1625.62ms | tok/sec: 10,078 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 64 | total time: 39.22m | eta: 95.8m
step 01461/05000 (29.22%) | loss: 4.283122 | lrm: 1.00 | dt: 1584.45ms | tok/sec: 10,340 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 64 | total time: 39.25m | eta: 95.7m
step 01462/05000 (29.24%) | loss: 4.286132 | lrm: 1.00 | dt: 1563.47ms | tok/sec: 10,479 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 64 | total time: 39.27m | eta: 95.7m
step 01463/05000 (29.26%) | loss: 4.300904 | lrm: 1.00 | dt: 1664.82ms | tok/sec: 9,841 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 64 | total time: 39.30m | eta: 95.7m
step 01464/05000 (29.28%) | loss: 4.310691 | lrm: 1.00 | dt: 1642.62ms | tok/sec: 9,974 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 64 | total time: 39.33m | eta: 95.6m
step 01465/05000 (29.30%) | loss: 4.311552 | lrm: 1.00 | dt: 1603.66ms | tok/sec: 10,216 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 64 | total time: 39.36m | eta: 95.6m
step 01466/05000 (29.32%) | loss: 4.294355 | lrm: 1.00 | dt: 1564.76ms | tok/sec: 10,470 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 64 | total time: 39.38m | eta: 95.6m
step 01467/05000 (29.34%) | loss: 4.288631 | lrm: 1.00 | dt: 1565.40ms | tok/sec: 10,466 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 64 | total time: 39.41m | eta: 95.6m
step 01468/05000 (29.36%) | loss: 4.274317 | lrm: 1.00 | dt: 1603.01ms | tok/sec: 10,220 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 64 | total time: 39.44m | eta: 95.5m
step 01469/05000 (29.38%) | loss: 4.257323 | lrm: 1.00 | dt: 1628.87ms | tok/sec: 10,058 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 64 | total time: 39.46m | eta: 95.5m
step 01470/05000 (29.40%) | loss: 4.261013 | lrm: 1.00 | dt: 1625.18ms | tok/sec: 10,081 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 64 | total time: 39.49m | eta: 95.5m
step 01471/05000 (29.42%) | loss: 4.244318 | lrm: 1.00 | dt: 1681.13ms | tok/sec: 9,745 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 64 | total time: 39.52m | eta: 95.5m
step 01472/05000 (29.44%) | loss: 4.258105 | lrm: 1.00 | dt: 1640.85ms | tok/sec: 9,985 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 64 | total time: 39.55m | eta: 95.4m
step 01473/05000 (29.46%) | loss: 4.241333 | lrm: 1.00 | dt: 1778.85ms | tok/sec: 9,210 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 64 | total time: 39.57m | eta: 95.4m
step 01474/05000 (29.48%) | loss: 4.240992 | lrm: 1.00 | dt: 1580.92ms | tok/sec: 10,363 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 64 | total time: 39.60m | eta: 95.4m
step 01475/05000 (29.50%) | loss: 4.244466 | lrm: 1.00 | dt: 1601.21ms | tok/sec: 10,232 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 64 | total time: 39.63m | eta: 95.4m
step 01476/05000 (29.52%) | loss: 4.255204 | lrm: 1.00 | dt: 1573.87ms | tok/sec: 10,410 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 64 | total time: 39.65m | eta: 95.3m
step 01477/05000 (29.54%) | loss: 4.249050 | lrm: 1.00 | dt: 1587.64ms | tok/sec: 10,319 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 64 | total time: 39.68m | eta: 95.3m
step 01478/05000 (29.56%) | loss: 4.240910 | lrm: 1.00 | dt: 1561.78ms | tok/sec: 10,490 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 65 | total time: 39.71m | eta: 95.3m
step 01479/05000 (29.58%) | loss: 4.233708 | lrm: 1.00 | dt: 1562.71ms | tok/sec: 10,484 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 65 | total time: 39.73m | eta: 95.2m
step 01480/05000 (29.60%) | loss: 4.244191 | lrm: 1.00 | dt: 1584.57ms | tok/sec: 10,339 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 65 | total time: 39.76m | eta: 95.2m
step 01481/05000 (29.62%) | loss: 4.243746 | lrm: 1.00 | dt: 1566.90ms | tok/sec: 10,456 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 65 | total time: 39.79m | eta: 95.2m
step 01482/05000 (29.64%) | loss: 4.238932 | lrm: 1.00 | dt: 1533.31ms | tok/sec: 10,685 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 65 | total time: 39.81m | eta: 95.1m
step 01483/05000 (29.66%) | loss: 4.237959 | lrm: 1.00 | dt: 1562.88ms | tok/sec: 10,483 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 65 | total time: 39.84m | eta: 95.1m
step 01484/05000 (29.68%) | loss: 4.236547 | lrm: 1.00 | dt: 1570.77ms | tok/sec: 10,430 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 65 | total time: 39.86m | eta: 95.1m
step 01485/05000 (29.70%) | loss: 4.248406 | lrm: 1.00 | dt: 1598.41ms | tok/sec: 10,250 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 65 | total time: 39.89m | eta: 95.1m
step 01486/05000 (29.72%) | loss: 4.252108 | lrm: 1.00 | dt: 1583.61ms | tok/sec: 10,345 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 65 | total time: 39.92m | eta: 95.0m
step 01487/05000 (29.74%) | loss: 4.254874 | lrm: 1.00 | dt: 1576.99ms | tok/sec: 10,389 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 65 | total time: 39.94m | eta: 95.0m
step 01488/05000 (29.76%) | loss: 4.265918 | lrm: 1.00 | dt: 1580.69ms | tok/sec: 10,365 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 65 | total time: 39.97m | eta: 95.0m
step 01489/05000 (29.78%) | loss: 4.279489 | lrm: 1.00 | dt: 1568.04ms | tok/sec: 10,448 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 65 | total time: 39.99m | eta: 94.9m
step 01490/05000 (29.80%) | loss: 4.286699 | lrm: 1.00 | dt: 1594.40ms | tok/sec: 10,275 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 65 | total time: 40.02m | eta: 94.9m
step 01491/05000 (29.82%) | loss: 4.293586 | lrm: 1.00 | dt: 1653.08ms | tok/sec: 9,911 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 65 | total time: 40.05m | eta: 94.9m
step 01492/05000 (29.84%) | loss: 4.290980 | lrm: 1.00 | dt: 1612.07ms | tok/sec: 10,163 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 65 | total time: 40.08m | eta: 94.9m
step 01493/05000 (29.86%) | loss: 4.284518 | lrm: 1.00 | dt: 1611.64ms | tok/sec: 10,166 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 65 | total time: 40.10m | eta: 94.8m
step 01494/05000 (29.88%) | loss: 4.281207 | lrm: 1.00 | dt: 1613.09ms | tok/sec: 10,156 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 65 | total time: 40.13m | eta: 94.8m
step 01495/05000 (29.90%) | loss: 4.266703 | lrm: 1.00 | dt: 1617.98ms | tok/sec: 10,126 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 65 | total time: 40.16m | eta: 94.8m
step 01496/05000 (29.92%) | loss: 4.266133 | lrm: 1.00 | dt: 1648.22ms | tok/sec: 9,940 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 65 | total time: 40.18m | eta: 94.8m
step 01497/05000 (29.94%) | loss: 4.269356 | lrm: 1.00 | dt: 1586.03ms | tok/sec: 10,330 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 65 | total time: 40.21m | eta: 94.7m
step 01498/05000 (29.96%) | loss: 4.266168 | lrm: 1.00 | dt: 1658.16ms | tok/sec: 9,880 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 65 | total time: 40.24m | eta: 94.7m
step 01499/05000 (29.98%) | loss: 4.237392 | lrm: 1.00 | dt: 1699.12ms | tok/sec: 9,642 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 65 | total time: 40.27m | eta: 94.7m
Step 01500 | Validation bpb: 1.315164
<|bos|>The capital of France is the largest city in France. It is the largest city in France, and it
<|bos|>The chemical symbol of gold is a symbol of the symbol of gold. It is a symbol of gold, and
<|bos|>If yesterday was Friday, then tomorrow will be the first time to take the first time to take the first time to take the
<|bos|>The opposite of hot is true. The hot is true, and the hot is true. The hot is
<|bos|>The planets of the solar system are: the solar system is a system that is a system that is system that is system
<|bos|>My favorite color is the color of the color of the color of the color of the color of the
<|bos|>If 5*x + 3 = 13, then x is 5*x + 3*x + 3*x +
step 01500/05000 (30.00%) | loss: 4.242125 | lrm: 1.00 | dt: 2583.73ms | tok/sec: 6,341 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 65 | total time: 40.31m | eta: 94.7m
step 01501/05000 (30.02%) | loss: 4.232749 | lrm: 1.00 | dt: 1853.11ms | tok/sec: 8,841 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 66 | total time: 40.34m | eta: 94.7m
step 01502/05000 (30.04%) | loss: 4.228912 | lrm: 1.00 | dt: 1752.69ms | tok/sec: 9,347 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 66 | total time: 40.37m | eta: 94.6m
step 01503/05000 (30.06%) | loss: 4.247189 | lrm: 1.00 | dt: 1770.57ms | tok/sec: 9,253 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 66 | total time: 40.40m | eta: 94.6m
step 01504/05000 (30.08%) | loss: 4.260605 | lrm: 1.00 | dt: 1745.73ms | tok/sec: 9,385 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 66 | total time: 40.43m | eta: 94.6m
step 01505/05000 (30.10%) | loss: 4.252007 | lrm: 1.00 | dt: 1677.58ms | tok/sec: 9,766 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 66 | total time: 40.46m | eta: 94.6m
step 01506/05000 (30.12%) | loss: 4.249716 | lrm: 1.00 | dt: 1723.27ms | tok/sec: 9,507 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 66 | total time: 40.48m | eta: 94.6m
step 01507/05000 (30.14%) | loss: 4.238743 | lrm: 1.00 | dt: 1725.73ms | tok/sec: 9,493 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 66 | total time: 40.51m | eta: 94.5m
step 01508/05000 (30.16%) | loss: 4.234626 | lrm: 1.00 | dt: 1699.78ms | tok/sec: 9,638 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 66 | total time: 40.54m | eta: 94.5m
step 01509/05000 (30.18%) | loss: 4.232888 | lrm: 1.00 | dt: 1684.02ms | tok/sec: 9,729 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 66 | total time: 40.57m | eta: 94.5m
step 01510/05000 (30.20%) | loss: 4.231359 | lrm: 1.00 | dt: 1773.85ms | tok/sec: 9,236 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 66 | total time: 40.60m | eta: 94.5m
step 01511/05000 (30.22%) | loss: 4.228514 | lrm: 1.00 | dt: 1739.28ms | tok/sec: 9,419 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 66 | total time: 40.63m | eta: 94.4m
step 01512/05000 (30.24%) | loss: 4.244919 | lrm: 1.00 | dt: 1768.59ms | tok/sec: 9,263 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 66 | total time: 40.66m | eta: 94.4m
step 01513/05000 (30.26%) | loss: 4.245862 | lrm: 1.00 | dt: 1770.29ms | tok/sec: 9,255 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 66 | total time: 40.69m | eta: 94.4m
step 01514/05000 (30.28%) | loss: 4.245415 | lrm: 1.00 | dt: 1698.56ms | tok/sec: 9,645 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 66 | total time: 40.72m | eta: 94.4m
step 01515/05000 (30.30%) | loss: 4.241336 | lrm: 1.00 | dt: 1702.47ms | tok/sec: 9,623 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 66 | total time: 40.74m | eta: 94.3m
step 01516/05000 (30.32%) | loss: 4.232645 | lrm: 1.00 | dt: 1658.97ms | tok/sec: 9,876 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 66 | total time: 40.77m | eta: 94.3m
step 01517/05000 (30.34%) | loss: 4.236773 | lrm: 1.00 | dt: 1686.70ms | tok/sec: 9,713 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 66 | total time: 40.80m | eta: 94.3m
step 01518/05000 (30.36%) | loss: 4.232014 | lrm: 1.00 | dt: 1732.65ms | tok/sec: 9,456 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 66 | total time: 40.83m | eta: 94.3m
step 01519/05000 (30.38%) | loss: 4.240422 | lrm: 1.00 | dt: 1706.96ms | tok/sec: 9,598 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 66 | total time: 40.86m | eta: 94.3m
step 01520/05000 (30.40%) | loss: 4.247132 | lrm: 1.00 | dt: 1736.65ms | tok/sec: 9,434 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 66 | total time: 40.89m | eta: 94.2m
step 01521/05000 (30.42%) | loss: 4.250017 | lrm: 1.00 | dt: 1689.20ms | tok/sec: 9,699 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 66 | total time: 40.91m | eta: 94.2m
step 01522/05000 (30.44%) | loss: 4.233971 | lrm: 1.00 | dt: 1721.88ms | tok/sec: 9,515 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 66 | total time: 40.94m | eta: 94.2m
step 01523/05000 (30.46%) | loss: 4.241082 | lrm: 1.00 | dt: 1696.99ms | tok/sec: 9,654 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 66 | total time: 40.97m | eta: 94.2m
step 01524/05000 (30.48%) | loss: 4.246266 | lrm: 1.00 | dt: 1689.17ms | tok/sec: 9,699 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 66 | total time: 41.00m | eta: 94.1m
step 01525/05000 (30.50%) | loss: 4.244662 | lrm: 1.00 | dt: 1650.77ms | tok/sec: 9,925 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 67 | total time: 41.03m | eta: 94.1m
step 01526/05000 (30.52%) | loss: 4.261294 | lrm: 1.00 | dt: 1630.05ms | tok/sec: 10,051 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 67 | total time: 41.05m | eta: 94.1m
step 01527/05000 (30.54%) | loss: 4.270030 | lrm: 1.00 | dt: 1665.84ms | tok/sec: 9,835 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 67 | total time: 41.08m | eta: 94.1m
step 01528/05000 (30.56%) | loss: 4.250660 | lrm: 1.00 | dt: 1671.45ms | tok/sec: 9,802 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 67 | total time: 41.11m | eta: 94.0m
step 01529/05000 (30.58%) | loss: 4.258801 | lrm: 1.00 | dt: 1658.42ms | tok/sec: 9,879 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 67 | total time: 41.14m | eta: 94.0m
step 01530/05000 (30.60%) | loss: 4.254322 | lrm: 1.00 | dt: 1655.16ms | tok/sec: 9,898 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 67 | total time: 41.16m | eta: 94.0m
step 01531/05000 (30.62%) | loss: 4.237857 | lrm: 1.00 | dt: 1657.43ms | tok/sec: 9,885 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 67 | total time: 41.19m | eta: 93.9m
step 01532/05000 (30.64%) | loss: 4.234045 | lrm: 1.00 | dt: 1646.57ms | tok/sec: 9,950 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 67 | total time: 41.22m | eta: 93.9m
step 01533/05000 (30.66%) | loss: 4.244264 | lrm: 1.00 | dt: 1651.24ms | tok/sec: 9,922 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 67 | total time: 41.25m | eta: 93.9m
step 01534/05000 (30.68%) | loss: 4.235042 | lrm: 1.00 | dt: 1694.18ms | tok/sec: 9,670 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 67 | total time: 41.28m | eta: 93.9m
step 01535/05000 (30.70%) | loss: 4.247912 | lrm: 1.00 | dt: 1650.30ms | tok/sec: 9,927 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 67 | total time: 41.30m | eta: 93.8m
step 01536/05000 (30.72%) | loss: 4.239801 | lrm: 1.00 | dt: 1645.39ms | tok/sec: 9,957 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 67 | total time: 41.33m | eta: 93.8m
step 01537/05000 (30.74%) | loss: 4.247592 | lrm: 1.00 | dt: 1642.41ms | tok/sec: 9,975 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 67 | total time: 41.36m | eta: 93.8m
step 01538/05000 (30.76%) | loss: 4.250254 | lrm: 1.00 | dt: 1672.80ms | tok/sec: 9,794 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 67 | total time: 41.39m | eta: 93.8m
step 01539/05000 (30.78%) | loss: 4.256841 | lrm: 1.00 | dt: 1653.62ms | tok/sec: 9,907 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 67 | total time: 41.41m | eta: 93.7m
step 01540/05000 (30.80%) | loss: 4.246712 | lrm: 1.00 | dt: 1647.45ms | tok/sec: 9,945 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 67 | total time: 41.44m | eta: 93.7m
step 01541/05000 (30.82%) | loss: 4.257193 | lrm: 1.00 | dt: 1654.87ms | tok/sec: 9,900 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 67 | total time: 41.47m | eta: 93.7m
step 01542/05000 (30.84%) | loss: 4.247794 | lrm: 1.00 | dt: 1695.12ms | tok/sec: 9,665 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 67 | total time: 41.50m | eta: 93.7m
step 01543/05000 (30.86%) | loss: 4.253268 | lrm: 1.00 | dt: 1674.76ms | tok/sec: 9,782 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 67 | total time: 41.52m | eta: 93.6m
step 01544/05000 (30.88%) | loss: 4.249470 | lrm: 1.00 | dt: 1668.92ms | tok/sec: 9,817 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 67 | total time: 41.55m | eta: 93.6m
step 01545/05000 (30.90%) | loss: 4.248750 | lrm: 1.00 | dt: 1642.24ms | tok/sec: 9,976 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 67 | total time: 41.58m | eta: 93.6m
step 01546/05000 (30.92%) | loss: 4.256112 | lrm: 1.00 | dt: 1658.14ms | tok/sec: 9,880 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 67 | total time: 41.61m | eta: 93.6m
step 01547/05000 (30.94%) | loss: 4.248002 | lrm: 1.00 | dt: 1631.46ms | tok/sec: 10,042 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 67 | total time: 41.63m | eta: 93.5m
step 01548/05000 (30.96%) | loss: 4.249476 | lrm: 1.00 | dt: 1655.23ms | tok/sec: 9,898 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 68 | total time: 41.66m | eta: 93.5m
step 01549/05000 (30.98%) | loss: 4.247342 | lrm: 1.00 | dt: 1636.50ms | tok/sec: 10,011 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 68 | total time: 41.69m | eta: 93.5m
step 01550/05000 (31.00%) | loss: 4.243510 | lrm: 1.00 | dt: 1690.20ms | tok/sec: 9,693 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 68 | total time: 41.72m | eta: 93.5m
step 01551/05000 (31.02%) | loss: 4.241273 | lrm: 1.00 | dt: 1699.35ms | tok/sec: 9,641 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 68 | total time: 41.75m | eta: 93.4m
step 01552/05000 (31.04%) | loss: 4.237371 | lrm: 1.00 | dt: 1729.89ms | tok/sec: 9,471 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 68 | total time: 41.77m | eta: 93.4m
step 01553/05000 (31.06%) | loss: 4.230978 | lrm: 1.00 | dt: 1809.41ms | tok/sec: 9,054 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 68 | total time: 41.80m | eta: 93.4m
step 01554/05000 (31.08%) | loss: 4.228370 | lrm: 1.00 | dt: 1653.75ms | tok/sec: 9,907 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 68 | total time: 41.83m | eta: 93.4m
step 01555/05000 (31.10%) | loss: 4.222989 | lrm: 1.00 | dt: 1706.99ms | tok/sec: 9,598 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 68 | total time: 41.86m | eta: 93.3m
step 01556/05000 (31.12%) | loss: 4.231167 | lrm: 1.00 | dt: 1723.27ms | tok/sec: 9,507 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 68 | total time: 41.89m | eta: 93.3m
step 01557/05000 (31.14%) | loss: 4.246032 | lrm: 1.00 | dt: 1763.21ms | tok/sec: 9,292 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 68 | total time: 41.92m | eta: 93.3m
step 01558/05000 (31.16%) | loss: 4.242911 | lrm: 1.00 | dt: 1719.56ms | tok/sec: 9,528 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 68 | total time: 41.95m | eta: 93.3m
step 01559/05000 (31.18%) | loss: 4.236795 | lrm: 1.00 | dt: 1689.33ms | tok/sec: 9,698 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 68 | total time: 41.98m | eta: 93.2m
step 01560/05000 (31.20%) | loss: 4.239084 | lrm: 1.00 | dt: 1631.76ms | tok/sec: 10,040 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 68 | total time: 42.00m | eta: 93.2m
step 01561/05000 (31.22%) | loss: 4.250590 | lrm: 1.00 | dt: 1667.38ms | tok/sec: 9,826 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 68 | total time: 42.03m | eta: 93.2m
step 01562/05000 (31.24%) | loss: 4.243579 | lrm: 1.00 | dt: 1660.78ms | tok/sec: 9,865 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 68 | total time: 42.06m | eta: 93.2m
step 01563/05000 (31.26%) | loss: 4.233067 | lrm: 1.00 | dt: 1690.84ms | tok/sec: 9,689 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 68 | total time: 42.09m | eta: 93.1m
step 01564/05000 (31.28%) | loss: 4.245327 | lrm: 1.00 | dt: 1642.21ms | tok/sec: 9,976 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 68 | total time: 42.11m | eta: 93.1m
step 01565/05000 (31.30%) | loss: 4.242472 | lrm: 1.00 | dt: 1652.25ms | tok/sec: 9,916 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 68 | total time: 42.14m | eta: 93.1m
step 01566/05000 (31.32%) | loss: 4.248826 | lrm: 1.00 | dt: 1592.52ms | tok/sec: 10,288 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 68 | total time: 42.17m | eta: 93.1m
step 01567/05000 (31.34%) | loss: 4.254885 | lrm: 1.00 | dt: 1624.63ms | tok/sec: 10,084 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 68 | total time: 42.20m | eta: 93.0m
step 01568/05000 (31.36%) | loss: 4.250841 | lrm: 1.00 | dt: 1731.42ms | tok/sec: 9,462 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 68 | total time: 42.22m | eta: 93.0m
step 01569/05000 (31.38%) | loss: 4.253376 | lrm: 1.00 | dt: 1594.15ms | tok/sec: 10,277 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 68 | total time: 42.25m | eta: 93.0m
step 01570/05000 (31.40%) | loss: 4.260371 | lrm: 1.00 | dt: 1577.07ms | tok/sec: 10,388 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 68 | total time: 42.28m | eta: 93.0m
step 01571/05000 (31.42%) | loss: 4.246246 | lrm: 1.00 | dt: 1530.89ms | tok/sec: 10,702 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 69 | total time: 42.30m | eta: 92.9m
step 01572/05000 (31.44%) | loss: 4.235137 | lrm: 1.00 | dt: 1453.30ms | tok/sec: 11,273 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 69 | total time: 42.33m | eta: 92.9m
step 01573/05000 (31.46%) | loss: 4.253076 | lrm: 1.00 | dt: 1458.56ms | tok/sec: 11,232 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 69 | total time: 42.35m | eta: 92.9m
step 01574/05000 (31.48%) | loss: 4.245859 | lrm: 1.00 | dt: 1558.13ms | tok/sec: 10,515 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 69 | total time: 42.38m | eta: 92.8m
step 01575/05000 (31.50%) | loss: 4.257556 | lrm: 1.00 | dt: 1496.66ms | tok/sec: 10,947 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 69 | total time: 42.40m | eta: 92.8m
step 01576/05000 (31.52%) | loss: 4.246305 | lrm: 1.00 | dt: 1533.00ms | tok/sec: 10,687 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 69 | total time: 42.43m | eta: 92.8m
step 01577/05000 (31.54%) | loss: 4.263048 | lrm: 1.00 | dt: 1557.81ms | tok/sec: 10,517 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 69 | total time: 42.45m | eta: 92.7m
step 01578/05000 (31.56%) | loss: 4.260896 | lrm: 1.00 | dt: 1477.70ms | tok/sec: 11,087 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 69 | total time: 42.48m | eta: 92.7m
step 01579/05000 (31.58%) | loss: 4.256134 | lrm: 1.00 | dt: 1505.34ms | tok/sec: 10,883 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 69 | total time: 42.50m | eta: 92.7m
step 01580/05000 (31.60%) | loss: 4.248105 | lrm: 1.00 | dt: 1565.24ms | tok/sec: 10,467 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 69 | total time: 42.53m | eta: 92.6m
step 01581/05000 (31.62%) | loss: 4.244285 | lrm: 1.00 | dt: 1521.87ms | tok/sec: 10,765 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 69 | total time: 42.55m | eta: 92.6m
step 01582/05000 (31.64%) | loss: 4.242545 | lrm: 1.00 | dt: 1478.16ms | tok/sec: 11,084 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 69 | total time: 42.58m | eta: 92.6m
step 01583/05000 (31.66%) | loss: 4.250254 | lrm: 1.00 | dt: 1516.44ms | tok/sec: 10,804 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 69 | total time: 42.60m | eta: 92.5m
step 01584/05000 (31.68%) | loss: 4.245309 | lrm: 1.00 | dt: 1506.44ms | tok/sec: 10,875 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 69 | total time: 42.63m | eta: 92.5m
step 01585/05000 (31.70%) | loss: 4.236941 | lrm: 1.00 | dt: 1481.41ms | tok/sec: 11,059 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 69 | total time: 42.65m | eta: 92.5m
step 01586/05000 (31.72%) | loss: 4.238057 | lrm: 1.00 | dt: 1583.59ms | tok/sec: 10,346 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 69 | total time: 42.68m | eta: 92.5m
step 01587/05000 (31.74%) | loss: 4.239302 | lrm: 1.00 | dt: 1568.74ms | tok/sec: 10,444 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 69 | total time: 42.71m | eta: 92.4m
step 01588/05000 (31.76%) | loss: 4.234537 | lrm: 1.00 | dt: 1490.80ms | tok/sec: 10,990 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 69 | total time: 42.73m | eta: 92.4m
step 01589/05000 (31.78%) | loss: 4.242569 | lrm: 1.00 | dt: 1471.46ms | tok/sec: 11,134 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 69 | total time: 42.76m | eta: 92.4m
step 01590/05000 (31.80%) | loss: 4.231451 | lrm: 1.00 | dt: 1591.53ms | tok/sec: 10,294 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 69 | total time: 42.78m | eta: 92.3m
step 01591/05000 (31.82%) | loss: 4.234188 | lrm: 1.00 | dt: 1555.41ms | tok/sec: 10,533 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 69 | total time: 42.81m | eta: 92.3m
step 01592/05000 (31.84%) | loss: 4.237542 | lrm: 1.00 | dt: 1488.92ms | tok/sec: 11,003 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 69 | total time: 42.83m | eta: 92.3m
step 01593/05000 (31.86%) | loss: 4.243943 | lrm: 1.00 | dt: 1531.54ms | tok/sec: 10,697 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 69 | total time: 42.86m | eta: 92.2m
step 01594/05000 (31.88%) | loss: 4.222907 | lrm: 1.00 | dt: 1502.07ms | tok/sec: 10,907 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 70 | total time: 42.88m | eta: 92.2m
step 01595/05000 (31.90%) | loss: 4.235104 | lrm: 1.00 | dt: 1532.31ms | tok/sec: 10,692 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 70 | total time: 42.91m | eta: 92.2m
step 01596/05000 (31.92%) | loss: 4.226969 | lrm: 1.00 | dt: 1492.14ms | tok/sec: 10,980 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 70 | total time: 42.93m | eta: 92.1m
step 01597/05000 (31.94%) | loss: 4.214115 | lrm: 1.00 | dt: 1555.13ms | tok/sec: 10,535 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 70 | total time: 42.96m | eta: 92.1m
step 01598/05000 (31.96%) | loss: 4.206035 | lrm: 1.00 | dt: 1538.86ms | tok/sec: 10,646 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 70 | total time: 42.99m | eta: 92.1m
step 01599/05000 (31.98%) | loss: 4.213901 | lrm: 1.00 | dt: 1548.20ms | tok/sec: 10,582 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 70 | total time: 43.01m | eta: 92.1m
Step 01600 | Validation bpb: 1.307505
<|bos|>The capital of France is the capital of France. The capital of France is the capital of France. The
<|bos|>The chemical symbol of gold is a symbol of the symbol of the symbol of the symbol of the symbol of the
<|bos|>If yesterday was Friday, then tomorrow will be the first day of the day. The day was the first day of the day
<|bos|>The opposite of hot is true. The opposite of hot is true. The opposite of hot is true.
<|bos|>The planets of the solar system are: the sun's sun's sun's sun's sun's sun's sun's sun
<|bos|>My favorite color is a white, white, white, white, white, white, white, white
<|bos|>If 5*x + 3 = 13, then x is 3 = 3 = 3 = 3 = 3 =
step 01600/05000 (32.00%) | loss: 4.218252 | lrm: 1.00 | dt: 1542.40ms | tok/sec: 10,622 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 70 | total time: 43.04m | eta: 92.0m
step 01601/05000 (32.02%) | loss: 4.226084 | lrm: 1.00 | dt: 1589.16ms | tok/sec: 10,309 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 70 | total time: 43.06m | eta: 92.0m
step 01602/05000 (32.04%) | loss: 4.208844 | lrm: 1.00 | dt: 1500.54ms | tok/sec: 10,918 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 70 | total time: 43.09m | eta: 92.0m
step 01603/05000 (32.06%) | loss: 4.199096 | lrm: 1.00 | dt: 1517.35ms | tok/sec: 10,797 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 70 | total time: 43.11m | eta: 91.9m
step 01604/05000 (32.08%) | loss: 4.211788 | lrm: 1.00 | dt: 1503.57ms | tok/sec: 10,896 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 70 | total time: 43.14m | eta: 91.9m
step 01605/05000 (32.10%) | loss: 4.201692 | lrm: 1.00 | dt: 1496.33ms | tok/sec: 10,949 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 70 | total time: 43.16m | eta: 91.9m
step 01606/05000 (32.12%) | loss: 4.185422 | lrm: 1.00 | dt: 1506.08ms | tok/sec: 10,878 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 70 | total time: 43.19m | eta: 91.8m
step 01607/05000 (32.14%) | loss: 4.192149 | lrm: 1.00 | dt: 1513.28ms | tok/sec: 10,826 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 70 | total time: 43.21m | eta: 91.8m
step 01608/05000 (32.16%) | loss: 4.193080 | lrm: 1.00 | dt: 1511.21ms | tok/sec: 10,841 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 70 | total time: 43.24m | eta: 91.8m
step 01609/05000 (32.18%) | loss: 4.190567 | lrm: 1.00 | dt: 1516.46ms | tok/sec: 10,804 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 70 | total time: 43.27m | eta: 91.8m
step 01610/05000 (32.20%) | loss: 4.205018 | lrm: 1.00 | dt: 1468.34ms | tok/sec: 11,158 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 70 | total time: 43.29m | eta: 91.7m
step 01611/05000 (32.22%) | loss: 4.202827 | lrm: 1.00 | dt: 1487.55ms | tok/sec: 11,014 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 70 | total time: 43.31m | eta: 91.7m
step 01612/05000 (32.24%) | loss: 4.198287 | lrm: 1.00 | dt: 1559.92ms | tok/sec: 10,503 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 70 | total time: 43.34m | eta: 91.7m
step 01613/05000 (32.26%) | loss: 4.195346 | lrm: 1.00 | dt: 1542.35ms | tok/sec: 10,622 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 70 | total time: 43.37m | eta: 91.6m
step 01614/05000 (32.28%) | loss: 4.186228 | lrm: 1.00 | dt: 1552.05ms | tok/sec: 10,556 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 70 | total time: 43.39m | eta: 91.6m
step 01615/05000 (32.30%) | loss: 4.165446 | lrm: 1.00 | dt: 1528.88ms | tok/sec: 10,716 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 70 | total time: 43.42m | eta: 91.6m
step 01616/05000 (32.32%) | loss: 4.166266 | lrm: 1.00 | dt: 1588.48ms | tok/sec: 10,314 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 70 | total time: 43.44m | eta: 91.5m
step 01617/05000 (32.34%) | loss: 4.173538 | lrm: 1.00 | dt: 1706.32ms | tok/sec: 9,601 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 71 | total time: 43.47m | eta: 91.5m
step 01618/05000 (32.36%) | loss: 4.165154 | lrm: 1.00 | dt: 1703.22ms | tok/sec: 9,619 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 71 | total time: 43.50m | eta: 91.5m
step 01619/05000 (32.38%) | loss: 4.170406 | lrm: 1.00 | dt: 1690.83ms | tok/sec: 9,689 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 71 | total time: 43.53m | eta: 91.5m
step 01620/05000 (32.40%) | loss: 4.171237 | lrm: 1.00 | dt: 1716.50ms | tok/sec: 9,544 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 71 | total time: 43.56m | eta: 91.4m
step 01621/05000 (32.42%) | loss: 4.179365 | lrm: 1.00 | dt: 1662.30ms | tok/sec: 9,856 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 71 | total time: 43.59m | eta: 91.4m
step 01622/05000 (32.44%) | loss: 4.185859 | lrm: 1.00 | dt: 1621.26ms | tok/sec: 10,105 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 71 | total time: 43.61m | eta: 91.4m
step 01623/05000 (32.46%) | loss: 4.195010 | lrm: 1.00 | dt: 1661.18ms | tok/sec: 9,862 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 71 | total time: 43.64m | eta: 91.4m
step 01624/05000 (32.48%) | loss: 4.213566 | lrm: 1.00 | dt: 1681.40ms | tok/sec: 9,744 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 71 | total time: 43.67m | eta: 91.3m
step 01625/05000 (32.50%) | loss: 4.214602 | lrm: 1.00 | dt: 1677.96ms | tok/sec: 9,764 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 71 | total time: 43.70m | eta: 91.3m
step 01626/05000 (32.52%) | loss: 4.218676 | lrm: 1.00 | dt: 1776.73ms | tok/sec: 9,221 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 71 | total time: 43.73m | eta: 91.3m
step 01627/05000 (32.54%) | loss: 4.219952 | lrm: 1.00 | dt: 1783.15ms | tok/sec: 9,188 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 71 | total time: 43.76m | eta: 91.3m
step 01628/05000 (32.56%) | loss: 4.219047 | lrm: 1.00 | dt: 1568.15ms | tok/sec: 10,447 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 71 | total time: 43.78m | eta: 91.2m
step 01629/05000 (32.58%) | loss: 4.211750 | lrm: 1.00 | dt: 1725.35ms | tok/sec: 9,496 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 71 | total time: 43.81m | eta: 91.2m
step 01630/05000 (32.60%) | loss: 4.206012 | lrm: 1.00 | dt: 1946.51ms | tok/sec: 8,417 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 71 | total time: 43.84m | eta: 91.2m
step 01631/05000 (32.62%) | loss: 4.193282 | lrm: 1.00 | dt: 1637.42ms | tok/sec: 10,005 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 71 | total time: 43.87m | eta: 91.2m
step 01632/05000 (32.64%) | loss: 4.193954 | lrm: 1.00 | dt: 1566.22ms | tok/sec: 10,460 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 71 | total time: 43.90m | eta: 91.1m
step 01633/05000 (32.66%) | loss: 4.210481 | lrm: 1.00 | dt: 1637.67ms | tok/sec: 10,004 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 71 | total time: 43.92m | eta: 91.1m
step 01634/05000 (32.68%) | loss: 4.200205 | lrm: 1.00 | dt: 1635.42ms | tok/sec: 10,018 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 71 | total time: 43.95m | eta: 91.1m
step 01635/05000 (32.70%) | loss: 4.184927 | lrm: 1.00 | dt: 1512.34ms | tok/sec: 10,833 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 71 | total time: 43.98m | eta: 91.1m
step 01636/05000 (32.72%) | loss: 4.191003 | lrm: 1.00 | dt: 1594.52ms | tok/sec: 10,275 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 71 | total time: 44.00m | eta: 91.0m
step 01637/05000 (32.74%) | loss: 4.194668 | lrm: 1.00 | dt: 1541.39ms | tok/sec: 10,629 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 71 | total time: 44.03m | eta: 91.0m
step 01638/05000 (32.76%) | loss: 4.185971 | lrm: 1.00 | dt: 1571.86ms | tok/sec: 10,423 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 71 | total time: 44.05m | eta: 91.0m
step 01639/05000 (32.78%) | loss: 4.213540 | lrm: 1.00 | dt: 1595.29ms | tok/sec: 10,270 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 71 | total time: 44.08m | eta: 90.9m
step 01640/05000 (32.80%) | loss: 4.215768 | lrm: 1.00 | dt: 1566.84ms | tok/sec: 10,456 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 71 | total time: 44.11m | eta: 90.9m
step 01641/05000 (32.82%) | loss: 4.215751 | lrm: 1.00 | dt: 1569.98ms | tok/sec: 10,435 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 72 | total time: 44.13m | eta: 90.9m
step 01642/05000 (32.84%) | loss: 4.218140 | lrm: 1.00 | dt: 1623.43ms | tok/sec: 10,092 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 72 | total time: 44.16m | eta: 90.9m
step 01643/05000 (32.86%) | loss: 4.217975 | lrm: 1.00 | dt: 1561.06ms | tok/sec: 10,495 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 72 | total time: 44.19m | eta: 90.8m
step 01644/05000 (32.88%) | loss: 4.202238 | lrm: 1.00 | dt: 1591.98ms | tok/sec: 10,291 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 72 | total time: 44.21m | eta: 90.8m
step 01645/05000 (32.90%) | loss: 4.204456 | lrm: 1.00 | dt: 1483.24ms | tok/sec: 11,046 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 72 | total time: 44.24m | eta: 90.8m
step 01646/05000 (32.92%) | loss: 4.204027 | lrm: 1.00 | dt: 1607.93ms | tok/sec: 10,189 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 72 | total time: 44.26m | eta: 90.7m
step 01647/05000 (32.94%) | loss: 4.207338 | lrm: 1.00 | dt: 1512.77ms | tok/sec: 10,830 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 72 | total time: 44.29m | eta: 90.7m
step 01648/05000 (32.96%) | loss: 4.219482 | lrm: 1.00 | dt: 1523.22ms | tok/sec: 10,756 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 72 | total time: 44.31m | eta: 90.7m
step 01649/05000 (32.98%) | loss: 4.213503 | lrm: 1.00 | dt: 1515.10ms | tok/sec: 10,813 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 72 | total time: 44.34m | eta: 90.7m
step 01650/05000 (33.00%) | loss: 4.208144 | lrm: 1.00 | dt: 1551.62ms | tok/sec: 10,559 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 72 | total time: 44.37m | eta: 90.6m
step 01651/05000 (33.02%) | loss: 4.210460 | lrm: 1.00 | dt: 1534.02ms | tok/sec: 10,680 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 72 | total time: 44.39m | eta: 90.6m
step 01652/05000 (33.04%) | loss: 4.199755 | lrm: 1.00 | dt: 1515.90ms | tok/sec: 10,808 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 72 | total time: 44.42m | eta: 90.6m
step 01653/05000 (33.06%) | loss: 4.204958 | lrm: 1.00 | dt: 1546.84ms | tok/sec: 10,591 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 72 | total time: 44.44m | eta: 90.5m
step 01654/05000 (33.08%) | loss: 4.209603 | lrm: 1.00 | dt: 1528.31ms | tok/sec: 10,720 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 72 | total time: 44.47m | eta: 90.5m
step 01655/05000 (33.10%) | loss: 4.197699 | lrm: 1.00 | dt: 1444.44ms | tok/sec: 11,342 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 72 | total time: 44.49m | eta: 90.5m
step 01656/05000 (33.12%) | loss: 4.205910 | lrm: 1.00 | dt: 1492.74ms | tok/sec: 10,975 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 72 | total time: 44.52m | eta: 90.4m
step 01657/05000 (33.14%) | loss: 4.219445 | lrm: 1.00 | dt: 1565.08ms | tok/sec: 10,468 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 72 | total time: 44.54m | eta: 90.4m
step 01658/05000 (33.16%) | loss: 4.218445 | lrm: 1.00 | dt: 1463.38ms | tok/sec: 11,196 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 72 | total time: 44.57m | eta: 90.4m
step 01659/05000 (33.18%) | loss: 4.203072 | lrm: 1.00 | dt: 1515.00ms | tok/sec: 10,814 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 72 | total time: 44.59m | eta: 90.3m
step 01660/05000 (33.20%) | loss: 4.203181 | lrm: 1.00 | dt: 1521.61ms | tok/sec: 10,767 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 72 | total time: 44.62m | eta: 90.3m
step 01661/05000 (33.22%) | loss: 4.194599 | lrm: 1.00 | dt: 1450.55ms | tok/sec: 11,295 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 72 | total time: 44.64m | eta: 90.3m
step 01662/05000 (33.24%) | loss: 4.181901 | lrm: 1.00 | dt: 1490.13ms | tok/sec: 10,995 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 72 | total time: 44.67m | eta: 90.3m
step 01663/05000 (33.26%) | loss: 4.198439 | lrm: 1.00 | dt: 1510.29ms | tok/sec: 10,848 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 72 | total time: 44.69m | eta: 90.2m
step 01664/05000 (33.28%) | loss: 4.200289 | lrm: 1.00 | dt: 1540.50ms | tok/sec: 10,635 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 73 | total time: 44.72m | eta: 90.2m
step 01665/05000 (33.30%) | loss: 4.207703 | lrm: 1.00 | dt: 1467.09ms | tok/sec: 11,167 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 73 | total time: 44.74m | eta: 90.2m
step 01666/05000 (33.32%) | loss: 4.210178 | lrm: 1.00 | dt: 1451.03ms | tok/sec: 11,291 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 73 | total time: 44.77m | eta: 90.1m
step 01667/05000 (33.34%) | loss: 4.207865 | lrm: 1.00 | dt: 1562.39ms | tok/sec: 10,486 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 73 | total time: 44.79m | eta: 90.1m
step 01668/05000 (33.36%) | loss: 4.216889 | lrm: 1.00 | dt: 1458.48ms | tok/sec: 11,233 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 73 | total time: 44.82m | eta: 90.1m
step 01669/05000 (33.38%) | loss: 4.212315 | lrm: 1.00 | dt: 1527.33ms | tok/sec: 10,727 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 73 | total time: 44.84m | eta: 90.0m
step 01670/05000 (33.40%) | loss: 4.224112 | lrm: 1.00 | dt: 1552.67ms | tok/sec: 10,552 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 73 | total time: 44.87m | eta: 90.0m
step 01671/05000 (33.42%) | loss: 4.220932 | lrm: 1.00 | dt: 1503.72ms | tok/sec: 10,895 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 73 | total time: 44.89m | eta: 90.0m
step 01672/05000 (33.44%) | loss: 4.221983 | lrm: 1.00 | dt: 1460.76ms | tok/sec: 11,216 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 73 | total time: 44.92m | eta: 89.9m
step 01673/05000 (33.46%) | loss: 4.220863 | lrm: 1.00 | dt: 1448.13ms | tok/sec: 11,313 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 73 | total time: 44.94m | eta: 89.9m
step 01674/05000 (33.48%) | loss: 4.217306 | lrm: 1.00 | dt: 1534.38ms | tok/sec: 10,677 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 73 | total time: 44.97m | eta: 89.9m
step 01675/05000 (33.50%) | loss: 4.216430 | lrm: 1.00 | dt: 1513.79ms | tok/sec: 10,823 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 73 | total time: 44.99m | eta: 89.8m
step 01676/05000 (33.52%) | loss: 4.220449 | lrm: 1.00 | dt: 1535.25ms | tok/sec: 10,671 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 73 | total time: 45.02m | eta: 89.8m
step 01677/05000 (33.54%) | loss: 4.226135 | lrm: 1.00 | dt: 1507.44ms | tok/sec: 10,868 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 73 | total time: 45.04m | eta: 89.8m
step 01678/05000 (33.56%) | loss: 4.213321 | lrm: 1.00 | dt: 1520.37ms | tok/sec: 10,776 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 73 | total time: 45.07m | eta: 89.8m
step 01679/05000 (33.58%) | loss: 4.193648 | lrm: 1.00 | dt: 1474.17ms | tok/sec: 11,114 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 73 | total time: 45.09m | eta: 89.7m
step 01680/05000 (33.60%) | loss: 4.198267 | lrm: 1.00 | dt: 1521.03ms | tok/sec: 10,771 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 73 | total time: 45.12m | eta: 89.7m
step 01681/05000 (33.62%) | loss: 4.198172 | lrm: 1.00 | dt: 1485.12ms | tok/sec: 11,032 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 73 | total time: 45.14m | eta: 89.7m
step 01682/05000 (33.64%) | loss: 4.204841 | lrm: 1.00 | dt: 1498.42ms | tok/sec: 10,934 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 73 | total time: 45.17m | eta: 89.6m
step 01683/05000 (33.66%) | loss: 4.205976 | lrm: 1.00 | dt: 1564.05ms | tok/sec: 10,475 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 73 | total time: 45.19m | eta: 89.6m
step 01684/05000 (33.68%) | loss: 4.207632 | lrm: 1.00 | dt: 1479.66ms | tok/sec: 11,072 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 73 | total time: 45.22m | eta: 89.6m
step 01685/05000 (33.70%) | loss: 4.211818 | lrm: 1.00 | dt: 1524.78ms | tok/sec: 10,745 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 73 | total time: 45.24m | eta: 89.5m
step 01686/05000 (33.72%) | loss: 4.212444 | lrm: 1.00 | dt: 1522.37ms | tok/sec: 10,762 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 73 | total time: 45.27m | eta: 89.5m
step 01687/05000 (33.74%) | loss: 4.225059 | lrm: 1.00 | dt: 1511.45ms | tok/sec: 10,839 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 73 | total time: 45.29m | eta: 89.5m
step 01688/05000 (33.76%) | loss: 4.218102 | lrm: 1.00 | dt: 1526.03ms | tok/sec: 10,736 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 74 | total time: 45.32m | eta: 89.5m
step 01689/05000 (33.78%) | loss: 4.211031 | lrm: 1.00 | dt: 1551.87ms | tok/sec: 10,557 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 74 | total time: 45.35m | eta: 89.4m
step 01690/05000 (33.80%) | loss: 4.194070 | lrm: 1.00 | dt: 1459.76ms | tok/sec: 11,223 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 74 | total time: 45.37m | eta: 89.4m
step 01691/05000 (33.82%) | loss: 4.201094 | lrm: 1.00 | dt: 1480.43ms | tok/sec: 11,067 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 74 | total time: 45.40m | eta: 89.4m
step 01692/05000 (33.84%) | loss: 4.196804 | lrm: 1.00 | dt: 1535.99ms | tok/sec: 10,666 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 74 | total time: 45.42m | eta: 89.3m
step 01693/05000 (33.86%) | loss: 4.196803 | lrm: 1.00 | dt: 1476.06ms | tok/sec: 11,099 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 74 | total time: 45.45m | eta: 89.3m
step 01694/05000 (33.88%) | loss: 4.196310 | lrm: 1.00 | dt: 1526.87ms | tok/sec: 10,730 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 74 | total time: 45.47m | eta: 89.3m
step 01695/05000 (33.90%) | loss: 4.200899 | lrm: 1.00 | dt: 1522.66ms | tok/sec: 10,760 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 74 | total time: 45.50m | eta: 89.2m
step 01696/05000 (33.92%) | loss: 4.198712 | lrm: 1.00 | dt: 1473.14ms | tok/sec: 11,121 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 74 | total time: 45.52m | eta: 89.2m
step 01697/05000 (33.94%) | loss: 4.196647 | lrm: 1.00 | dt: 1417.23ms | tok/sec: 11,560 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 74 | total time: 45.54m | eta: 89.2m
step 01698/05000 (33.96%) | loss: 4.194484 | lrm: 1.00 | dt: 1546.22ms | tok/sec: 10,596 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 74 | total time: 45.57m | eta: 89.1m
step 01699/05000 (33.98%) | loss: 4.182007 | lrm: 1.00 | dt: 1505.55ms | tok/sec: 10,882 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 74 | total time: 45.60m | eta: 89.1m
Step 01700 | Validation bpb: 1.302495
<|bos|>The capital of France is the 19th century, which was the first to be the first to the
<|bos|>The chemical symbol of gold is a symbol of gold, and is a symbol of gold. The symbol of gold
<|bos|>If yesterday was Friday, then tomorrow will be the first day of the year. The day was 1,000 years
<|bos|>The opposite of hot is true. The hot is true. The hot is true. The hot is true
<|bos|>The planets of the solar system are: the sun's sun's sun's sun's sun's sun's sun's sun
<|bos|>My favorite color is a combination of colors, and I love to see the colors. I love to
<|bos|>If 5*x + 3 = 13, then x is 3*x + 3*x + 3*x +
step 01700/05000 (34.00%) | loss: 4.178021 | lrm: 1.00 | dt: 1642.46ms | tok/sec: 9,975 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 74 | total time: 45.62m | eta: 89.1m
step 01701/05000 (34.02%) | loss: 4.176126 | lrm: 1.00 | dt: 1769.31ms | tok/sec: 9,260 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 74 | total time: 45.65m | eta: 89.1m
step 01702/05000 (34.04%) | loss: 4.176255 | lrm: 1.00 | dt: 1645.33ms | tok/sec: 9,957 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 74 | total time: 45.68m | eta: 89.0m
step 01703/05000 (34.06%) | loss: 4.183454 | lrm: 1.00 | dt: 1648.08ms | tok/sec: 9,941 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 74 | total time: 45.71m | eta: 89.0m
step 01704/05000 (34.08%) | loss: 4.184188 | lrm: 1.00 | dt: 1683.91ms | tok/sec: 9,729 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 74 | total time: 45.74m | eta: 89.0m
step 01705/05000 (34.10%) | loss: 4.180935 | lrm: 1.00 | dt: 1652.04ms | tok/sec: 9,917 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 74 | total time: 45.76m | eta: 89.0m
step 01706/05000 (34.12%) | loss: 4.165827 | lrm: 1.00 | dt: 1595.04ms | tok/sec: 10,271 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 74 | total time: 45.79m | eta: 88.9m
step 01707/05000 (34.14%) | loss: 4.164658 | lrm: 1.00 | dt: 1618.52ms | tok/sec: 10,122 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 74 | total time: 45.82m | eta: 88.9m
step 01708/05000 (34.16%) | loss: 4.169866 | lrm: 1.00 | dt: 1549.28ms | tok/sec: 10,575 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 74 | total time: 45.84m | eta: 88.9m
step 01709/05000 (34.18%) | loss: 4.164318 | lrm: 1.00 | dt: 1588.25ms | tok/sec: 10,315 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 74 | total time: 45.87m | eta: 88.8m
step 01710/05000 (34.20%) | loss: 4.163657 | lrm: 1.00 | dt: 1571.99ms | tok/sec: 10,422 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 74 | total time: 45.89m | eta: 88.8m
step 01711/05000 (34.22%) | loss: 4.176309 | lrm: 1.00 | dt: 1586.10ms | tok/sec: 10,329 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 75 | total time: 45.92m | eta: 88.8m
step 01712/05000 (34.24%) | loss: 4.168867 | lrm: 1.00 | dt: 1685.99ms | tok/sec: 9,717 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 75 | total time: 45.95m | eta: 88.8m
step 01713/05000 (34.26%) | loss: 4.173749 | lrm: 1.00 | dt: 1581.48ms | tok/sec: 10,359 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 75 | total time: 45.98m | eta: 88.7m
step 01714/05000 (34.28%) | loss: 4.201847 | lrm: 1.00 | dt: 1643.11ms | tok/sec: 9,971 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 75 | total time: 46.00m | eta: 88.7m
step 01715/05000 (34.30%) | loss: 4.204136 | lrm: 1.00 | dt: 1712.78ms | tok/sec: 9,565 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 75 | total time: 46.03m | eta: 88.7m
step 01716/05000 (34.32%) | loss: 4.199863 | lrm: 1.00 | dt: 1723.89ms | tok/sec: 9,504 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 75 | total time: 46.06m | eta: 88.7m
step 01717/05000 (34.34%) | loss: 4.182775 | lrm: 1.00 | dt: 1668.91ms | tok/sec: 9,817 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 75 | total time: 46.09m | eta: 88.6m
step 01718/05000 (34.36%) | loss: 4.190969 | lrm: 1.00 | dt: 1647.79ms | tok/sec: 9,942 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 75 | total time: 46.12m | eta: 88.6m
step 01719/05000 (34.38%) | loss: 4.198071 | lrm: 1.00 | dt: 1549.90ms | tok/sec: 10,571 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 75 | total time: 46.14m | eta: 88.6m
step 01720/05000 (34.40%) | loss: 4.191657 | lrm: 1.00 | dt: 1522.63ms | tok/sec: 10,760 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 75 | total time: 46.17m | eta: 88.6m
step 01721/05000 (34.42%) | loss: 4.197955 | lrm: 1.00 | dt: 1488.33ms | tok/sec: 11,008 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 75 | total time: 46.19m | eta: 88.5m
step 01722/05000 (34.44%) | loss: 4.197850 | lrm: 1.00 | dt: 1636.55ms | tok/sec: 10,011 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 75 | total time: 46.22m | eta: 88.5m
step 01723/05000 (34.46%) | loss: 4.188652 | lrm: 1.00 | dt: 1628.62ms | tok/sec: 10,060 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 75 | total time: 46.25m | eta: 88.5m
step 01724/05000 (34.48%) | loss: 4.201471 | lrm: 1.00 | dt: 1648.15ms | tok/sec: 9,940 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 75 | total time: 46.27m | eta: 88.4m
step 01725/05000 (34.50%) | loss: 4.183981 | lrm: 1.00 | dt: 1493.44ms | tok/sec: 10,970 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 75 | total time: 46.30m | eta: 88.4m
step 01726/05000 (34.52%) | loss: 4.182629 | lrm: 1.00 | dt: 1490.66ms | tok/sec: 10,991 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 75 | total time: 46.32m | eta: 88.4m
step 01727/05000 (34.54%) | loss: 4.175178 | lrm: 1.00 | dt: 1537.16ms | tok/sec: 10,658 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 75 | total time: 46.35m | eta: 88.4m
step 01728/05000 (34.56%) | loss: 4.166427 | lrm: 1.00 | dt: 1548.00ms | tok/sec: 10,583 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 75 | total time: 46.37m | eta: 88.3m
step 01729/05000 (34.58%) | loss: 4.172736 | lrm: 1.00 | dt: 1492.05ms | tok/sec: 10,980 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 75 | total time: 46.40m | eta: 88.3m
step 01730/05000 (34.60%) | loss: 4.174518 | lrm: 1.00 | dt: 1529.69ms | tok/sec: 10,710 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 75 | total time: 46.42m | eta: 88.3m
step 01731/05000 (34.62%) | loss: 4.181954 | lrm: 1.00 | dt: 1417.22ms | tok/sec: 11,560 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 75 | total time: 46.45m | eta: 88.2m
step 01732/05000 (34.64%) | loss: 4.181383 | lrm: 1.00 | dt: 1456.08ms | tok/sec: 11,252 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 75 | total time: 46.47m | eta: 88.2m
step 01733/05000 (34.66%) | loss: 4.184154 | lrm: 1.00 | dt: 1654.01ms | tok/sec: 9,905 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 75 | total time: 46.50m | eta: 88.2m
step 01734/05000 (34.68%) | loss: 4.192818 | lrm: 1.00 | dt: 1579.47ms | tok/sec: 10,373 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 76 | total time: 46.53m | eta: 88.1m
step 01735/05000 (34.70%) | loss: 4.192494 | lrm: 1.00 | dt: 1550.44ms | tok/sec: 10,567 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 76 | total time: 46.55m | eta: 88.1m
step 01736/05000 (34.72%) | loss: 4.207701 | lrm: 1.00 | dt: 1541.12ms | tok/sec: 10,631 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 76 | total time: 46.58m | eta: 88.1m
step 01737/05000 (34.74%) | loss: 4.201584 | lrm: 1.00 | dt: 1590.03ms | tok/sec: 10,304 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 76 | total time: 46.60m | eta: 88.1m
step 01738/05000 (34.76%) | loss: 4.222827 | lrm: 1.00 | dt: 1569.83ms | tok/sec: 10,436 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 76 | total time: 46.63m | eta: 88.0m
step 01739/05000 (34.78%) | loss: 4.231282 | lrm: 1.00 | dt: 1597.13ms | tok/sec: 10,258 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 76 | total time: 46.66m | eta: 88.0m
step 01740/05000 (34.80%) | loss: 4.236680 | lrm: 1.00 | dt: 1656.80ms | tok/sec: 9,888 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 76 | total time: 46.69m | eta: 88.0m
step 01741/05000 (34.82%) | loss: 4.236220 | lrm: 1.00 | dt: 1638.60ms | tok/sec: 9,998 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 76 | total time: 46.71m | eta: 87.9m
step 01742/05000 (34.84%) | loss: 4.226357 | lrm: 1.00 | dt: 1672.84ms | tok/sec: 9,794 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 76 | total time: 46.74m | eta: 87.9m
step 01743/05000 (34.86%) | loss: 4.224004 | lrm: 1.00 | dt: 1517.73ms | tok/sec: 10,795 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 76 | total time: 46.77m | eta: 87.9m
step 01744/05000 (34.88%) | loss: 4.223255 | lrm: 1.00 | dt: 1510.54ms | tok/sec: 10,846 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 76 | total time: 46.79m | eta: 87.9m
step 01745/05000 (34.90%) | loss: 4.230433 | lrm: 1.00 | dt: 1472.68ms | tok/sec: 11,125 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 76 | total time: 46.82m | eta: 87.8m
step 01746/05000 (34.92%) | loss: 4.233248 | lrm: 1.00 | dt: 1457.25ms | tok/sec: 11,243 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 76 | total time: 46.84m | eta: 87.8m
step 01747/05000 (34.94%) | loss: 4.231931 | lrm: 1.00 | dt: 1461.96ms | tok/sec: 11,206 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 76 | total time: 46.86m | eta: 87.8m
step 01748/05000 (34.96%) | loss: 4.241536 | lrm: 1.00 | dt: 1661.48ms | tok/sec: 9,861 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 76 | total time: 46.89m | eta: 87.7m
step 01749/05000 (34.98%) | loss: 4.239018 | lrm: 1.00 | dt: 1553.12ms | tok/sec: 10,549 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 76 | total time: 46.92m | eta: 87.7m
step 01750/05000 (35.00%) | loss: 4.218527 | lrm: 1.00 | dt: 1580.91ms | tok/sec: 10,363 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 76 | total time: 46.94m | eta: 87.7m
step 01751/05000 (35.02%) | loss: 4.202189 | lrm: 1.00 | dt: 1651.84ms | tok/sec: 9,918 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 76 | total time: 46.97m | eta: 87.7m
step 01752/05000 (35.04%) | loss: 4.192079 | lrm: 1.00 | dt: 1825.64ms | tok/sec: 8,974 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 76 | total time: 47.00m | eta: 87.6m
step 01753/05000 (35.06%) | loss: 4.193295 | lrm: 1.00 | dt: 1814.94ms | tok/sec: 9,027 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 76 | total time: 47.03m | eta: 87.6m
step 01754/05000 (35.08%) | loss: 4.206594 | lrm: 1.00 | dt: 1844.97ms | tok/sec: 8,880 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 76 | total time: 47.06m | eta: 87.6m
step 01755/05000 (35.10%) | loss: 4.182746 | lrm: 1.00 | dt: 1809.83ms | tok/sec: 9,052 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 76 | total time: 47.09m | eta: 87.6m
step 01756/05000 (35.12%) | loss: 4.193624 | lrm: 1.00 | dt: 1827.54ms | tok/sec: 8,965 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 76 | total time: 47.12m | eta: 87.6m
step 01757/05000 (35.14%) | loss: 4.197702 | lrm: 1.00 | dt: 1826.47ms | tok/sec: 8,970 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 76 | total time: 47.15m | eta: 87.5m
step 01758/05000 (35.16%) | loss: 4.208765 | lrm: 1.00 | dt: 1858.05ms | tok/sec: 8,817 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 77 | total time: 47.18m | eta: 87.5m
step 01759/05000 (35.18%) | loss: 4.209334 | lrm: 1.00 | dt: 1832.83ms | tok/sec: 8,939 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 77 | total time: 47.22m | eta: 87.5m
step 01760/05000 (35.20%) | loss: 4.197935 | lrm: 1.00 | dt: 1828.89ms | tok/sec: 8,958 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 77 | total time: 47.25m | eta: 87.5m
step 01761/05000 (35.22%) | loss: 4.192683 | lrm: 1.00 | dt: 1820.83ms | tok/sec: 8,998 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 77 | total time: 47.28m | eta: 87.5m
step 01762/05000 (35.24%) | loss: 4.203966 | lrm: 1.00 | dt: 1804.39ms | tok/sec: 9,080 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 77 | total time: 47.31m | eta: 87.4m
step 01763/05000 (35.26%) | loss: 4.194569 | lrm: 1.00 | dt: 1848.50ms | tok/sec: 8,863 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 77 | total time: 47.34m | eta: 87.4m
step 01764/05000 (35.28%) | loss: 4.207874 | lrm: 1.00 | dt: 1796.07ms | tok/sec: 9,122 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 77 | total time: 47.37m | eta: 87.4m
step 01765/05000 (35.30%) | loss: 4.214835 | lrm: 1.00 | dt: 1841.82ms | tok/sec: 8,895 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 77 | total time: 47.40m | eta: 87.4m
step 01766/05000 (35.32%) | loss: 4.210049 | lrm: 1.00 | dt: 1844.96ms | tok/sec: 8,880 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 77 | total time: 47.43m | eta: 87.3m
step 01767/05000 (35.34%) | loss: 4.208609 | lrm: 1.00 | dt: 1833.33ms | tok/sec: 8,936 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 77 | total time: 47.46m | eta: 87.3m
step 01768/05000 (35.36%) | loss: 4.200990 | lrm: 0.99 | dt: 1820.01ms | tok/sec: 9,002 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 77 | total time: 47.49m | eta: 87.3m
step 01769/05000 (35.38%) | loss: 4.192064 | lrm: 0.99 | dt: 1761.70ms | tok/sec: 9,300 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 77 | total time: 47.52m | eta: 87.3m
step 01770/05000 (35.40%) | loss: 4.186136 | lrm: 0.99 | dt: 1554.46ms | tok/sec: 10,539 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 77 | total time: 47.54m | eta: 87.3m
step 01771/05000 (35.42%) | loss: 4.190983 | lrm: 0.99 | dt: 1700.36ms | tok/sec: 9,635 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 77 | total time: 47.57m | eta: 87.2m
step 01772/05000 (35.44%) | loss: 4.186168 | lrm: 0.99 | dt: 1638.90ms | tok/sec: 9,996 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 77 | total time: 47.60m | eta: 87.2m
step 01773/05000 (35.46%) | loss: 4.189930 | lrm: 0.99 | dt: 1575.53ms | tok/sec: 10,399 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 77 | total time: 47.63m | eta: 87.2m
step 01774/05000 (35.48%) | loss: 4.183641 | lrm: 0.99 | dt: 1512.79ms | tok/sec: 10,830 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 77 | total time: 47.65m | eta: 87.1m
step 01775/05000 (35.50%) | loss: 4.169523 | lrm: 0.99 | dt: 1475.59ms | tok/sec: 11,103 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 77 | total time: 47.68m | eta: 87.1m
step 01776/05000 (35.52%) | loss: 4.164252 | lrm: 0.99 | dt: 1485.71ms | tok/sec: 11,027 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 77 | total time: 47.70m | eta: 87.1m
step 01777/05000 (35.54%) | loss: 4.171779 | lrm: 0.99 | dt: 1534.93ms | tok/sec: 10,674 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 77 | total time: 47.73m | eta: 87.1m
step 01778/05000 (35.56%) | loss: 4.184276 | lrm: 0.99 | dt: 1619.61ms | tok/sec: 10,115 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 77 | total time: 47.75m | eta: 87.0m
step 01779/05000 (35.58%) | loss: 4.192501 | lrm: 0.99 | dt: 1533.71ms | tok/sec: 10,682 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 77 | total time: 47.78m | eta: 87.0m
step 01780/05000 (35.60%) | loss: 4.186662 | lrm: 0.99 | dt: 1567.81ms | tok/sec: 10,450 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 77 | total time: 47.81m | eta: 87.0m
step 01781/05000 (35.62%) | loss: 4.189212 | lrm: 0.99 | dt: 1515.70ms | tok/sec: 10,809 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 78 | total time: 47.83m | eta: 86.9m
step 01782/05000 (35.64%) | loss: 4.200790 | lrm: 0.99 | dt: 1530.97ms | tok/sec: 10,701 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 78 | total time: 47.86m | eta: 86.9m
step 01783/05000 (35.66%) | loss: 4.205455 | lrm: 0.99 | dt: 1513.94ms | tok/sec: 10,822 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 78 | total time: 47.88m | eta: 86.9m
step 01784/05000 (35.68%) | loss: 4.206125 | lrm: 0.99 | dt: 1511.85ms | tok/sec: 10,837 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 78 | total time: 47.91m | eta: 86.8m
step 01785/05000 (35.70%) | loss: 4.206668 | lrm: 0.99 | dt: 1546.52ms | tok/sec: 10,594 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 78 | total time: 47.93m | eta: 86.8m
step 01786/05000 (35.72%) | loss: 4.215929 | lrm: 0.99 | dt: 1481.84ms | tok/sec: 11,056 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 78 | total time: 47.96m | eta: 86.8m
step 01787/05000 (35.74%) | loss: 4.209820 | lrm: 0.99 | dt: 1457.40ms | tok/sec: 11,241 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 78 | total time: 47.98m | eta: 86.8m
step 01788/05000 (35.76%) | loss: 4.219263 | lrm: 0.99 | dt: 1508.67ms | tok/sec: 10,859 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 78 | total time: 48.01m | eta: 86.7m
step 01789/05000 (35.78%) | loss: 4.202881 | lrm: 0.99 | dt: 1536.46ms | tok/sec: 10,663 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 78 | total time: 48.03m | eta: 86.7m
step 01790/05000 (35.80%) | loss: 4.178730 | lrm: 0.99 | dt: 1476.43ms | tok/sec: 11,097 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 78 | total time: 48.06m | eta: 86.7m
step 01791/05000 (35.82%) | loss: 4.187166 | lrm: 0.99 | dt: 1491.60ms | tok/sec: 10,984 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 78 | total time: 48.08m | eta: 86.6m
step 01792/05000 (35.84%) | loss: 4.183325 | lrm: 0.99 | dt: 1488.99ms | tok/sec: 11,003 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 78 | total time: 48.11m | eta: 86.6m
step 01793/05000 (35.86%) | loss: 4.177482 | lrm: 0.99 | dt: 1556.12ms | tok/sec: 10,528 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 78 | total time: 48.13m | eta: 86.6m
step 01794/05000 (35.88%) | loss: 4.170930 | lrm: 0.99 | dt: 1533.94ms | tok/sec: 10,680 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 78 | total time: 48.16m | eta: 86.5m
step 01795/05000 (35.90%) | loss: 4.165818 | lrm: 0.99 | dt: 1532.93ms | tok/sec: 10,688 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 78 | total time: 48.18m | eta: 86.5m
step 01796/05000 (35.92%) | loss: 4.175745 | lrm: 0.99 | dt: 1477.81ms | tok/sec: 11,086 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 78 | total time: 48.21m | eta: 86.5m
step 01797/05000 (35.94%) | loss: 4.175720 | lrm: 0.99 | dt: 1453.27ms | tok/sec: 11,273 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 78 | total time: 48.23m | eta: 86.5m
step 01798/05000 (35.96%) | loss: 4.171553 | lrm: 0.99 | dt: 1508.52ms | tok/sec: 10,860 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 78 | total time: 48.26m | eta: 86.4m
step 01799/05000 (35.98%) | loss: 4.184945 | lrm: 0.99 | dt: 1453.46ms | tok/sec: 11,272 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 78 | total time: 48.28m | eta: 86.4m
Step 01800 | Validation bpb: 1.294347
<|bos|>The capital of France is the most important part of the world. The capital of France is the most important
<|bos|>The chemical symbol of gold is a symbol of the symbol of the symbol of the symbol of the symbol of the
<|bos|>If yesterday was Friday, then tomorrow will be the first time to take a look at the 20th anniversary of the
<|bos|>The opposite of hot is true. The heat is a constant pressure. The heat is a constant pressure.
<|bos|>The planets of the solar system are: the solar system, the solar system, the solar system, the solar system,
<|bos|>My favorite color is the red, yellow, and yellow. I have a few colors that are not
<|bos|>If 5*x + 3 = 13, then x is 5*x + 3. If x is 5*x +
step 01800/05000 (36.00%) | loss: 4.191450 | lrm: 0.99 | dt: 1606.32ms | tok/sec: 10,199 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 78 | total time: 48.31m | eta: 86.4m
step 01801/05000 (36.02%) | loss: 4.189114 | lrm: 0.99 | dt: 1517.64ms | tok/sec: 10,795 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 78 | total time: 48.33m | eta: 86.3m
step 01802/05000 (36.04%) | loss: 4.194916 | lrm: 0.98 | dt: 1511.58ms | tok/sec: 10,838 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 78 | total time: 48.36m | eta: 86.3m
step 01803/05000 (36.06%) | loss: 4.204041 | lrm: 0.98 | dt: 1537.18ms | tok/sec: 10,658 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 78 | total time: 48.38m | eta: 86.3m
step 01804/05000 (36.08%) | loss: 4.186121 | lrm: 0.98 | dt: 1497.41ms | tok/sec: 10,941 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 79 | total time: 48.41m | eta: 86.2m
step 01805/05000 (36.10%) | loss: 4.179489 | lrm: 0.98 | dt: 1564.78ms | tok/sec: 10,470 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 79 | total time: 48.44m | eta: 86.2m
step 01806/05000 (36.12%) | loss: 4.192445 | lrm: 0.98 | dt: 1605.66ms | tok/sec: 10,203 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 79 | total time: 48.46m | eta: 86.2m
step 01807/05000 (36.14%) | loss: 4.201721 | lrm: 0.98 | dt: 1583.37ms | tok/sec: 10,347 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 79 | total time: 48.49m | eta: 86.2m
step 01808/05000 (36.16%) | loss: 4.196316 | lrm: 0.98 | dt: 1643.28ms | tok/sec: 9,970 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 79 | total time: 48.52m | eta: 86.1m
step 01809/05000 (36.18%) | loss: 4.191091 | lrm: 0.98 | dt: 1553.70ms | tok/sec: 10,545 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 79 | total time: 48.54m | eta: 86.1m
step 01810/05000 (36.20%) | loss: 4.205746 | lrm: 0.98 | dt: 1546.81ms | tok/sec: 10,592 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 79 | total time: 48.57m | eta: 86.1m
step 01811/05000 (36.22%) | loss: 4.194045 | lrm: 0.98 | dt: 1480.81ms | tok/sec: 11,064 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 79 | total time: 48.59m | eta: 86.0m
step 01812/05000 (36.24%) | loss: 4.197437 | lrm: 0.98 | dt: 1573.03ms | tok/sec: 10,415 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 79 | total time: 48.62m | eta: 86.0m
step 01813/05000 (36.26%) | loss: 4.178256 | lrm: 0.98 | dt: 1549.88ms | tok/sec: 10,571 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 79 | total time: 48.64m | eta: 86.0m
step 01814/05000 (36.28%) | loss: 4.181686 | lrm: 0.98 | dt: 1548.05ms | tok/sec: 10,583 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 79 | total time: 48.67m | eta: 86.0m
step 01815/05000 (36.30%) | loss: 4.170079 | lrm: 0.98 | dt: 1477.31ms | tok/sec: 11,090 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 79 | total time: 48.70m | eta: 85.9m
step 01816/05000 (36.32%) | loss: 4.157857 | lrm: 0.98 | dt: 1463.43ms | tok/sec: 11,195 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 79 | total time: 48.72m | eta: 85.9m
step 01817/05000 (36.34%) | loss: 4.160595 | lrm: 0.98 | dt: 1579.78ms | tok/sec: 10,371 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 79 | total time: 48.75m | eta: 85.9m
step 01818/05000 (36.36%) | loss: 4.158941 | lrm: 0.98 | dt: 1496.10ms | tok/sec: 10,951 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 79 | total time: 48.77m | eta: 85.8m
step 01819/05000 (36.38%) | loss: 4.150015 | lrm: 0.98 | dt: 1567.87ms | tok/sec: 10,449 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 79 | total time: 48.80m | eta: 85.8m
step 01820/05000 (36.40%) | loss: 4.157891 | lrm: 0.98 | dt: 1435.98ms | tok/sec: 11,409 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 79 | total time: 48.82m | eta: 85.8m
step 01821/05000 (36.42%) | loss: 4.143654 | lrm: 0.98 | dt: 1485.50ms | tok/sec: 11,029 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 79 | total time: 48.85m | eta: 85.7m
step 01822/05000 (36.44%) | loss: 4.142243 | lrm: 0.98 | dt: 1506.50ms | tok/sec: 10,875 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 79 | total time: 48.87m | eta: 85.7m
step 01823/05000 (36.46%) | loss: 4.150442 | lrm: 0.98 | dt: 1586.85ms | tok/sec: 10,324 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 79 | total time: 48.90m | eta: 85.7m
step 01824/05000 (36.48%) | loss: 4.155547 | lrm: 0.98 | dt: 1616.17ms | tok/sec: 10,137 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 79 | total time: 48.92m | eta: 85.7m
step 01825/05000 (36.50%) | loss: 4.149389 | lrm: 0.98 | dt: 1539.33ms | tok/sec: 10,643 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 79 | total time: 48.95m | eta: 85.6m
step 01826/05000 (36.52%) | loss: 4.157594 | lrm: 0.98 | dt: 1473.87ms | tok/sec: 11,116 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 79 | total time: 48.97m | eta: 85.6m
step 01827/05000 (36.54%) | loss: 4.152669 | lrm: 0.98 | dt: 1401.24ms | tok/sec: 11,692 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 80 | total time: 49.00m | eta: 85.6m
step 01828/05000 (36.56%) | loss: 4.149276 | lrm: 0.98 | dt: 1470.83ms | tok/sec: 11,139 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 80 | total time: 49.02m | eta: 85.5m
step 01829/05000 (36.58%) | loss: 4.146883 | lrm: 0.98 | dt: 1488.52ms | tok/sec: 11,006 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 80 | total time: 49.05m | eta: 85.5m
step 01830/05000 (36.60%) | loss: 4.149102 | lrm: 0.98 | dt: 1490.05ms | tok/sec: 10,995 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 80 | total time: 49.07m | eta: 85.5m
step 01831/05000 (36.62%) | loss: 4.142779 | lrm: 0.98 | dt: 1461.42ms | tok/sec: 11,211 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 80 | total time: 49.10m | eta: 85.4m
step 01832/05000 (36.64%) | loss: 4.143436 | lrm: 0.98 | dt: 1490.70ms | tok/sec: 10,990 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 80 | total time: 49.12m | eta: 85.4m
step 01833/05000 (36.66%) | loss: 4.140459 | lrm: 0.98 | dt: 1487.11ms | tok/sec: 11,017 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 80 | total time: 49.15m | eta: 85.4m
step 01834/05000 (36.68%) | loss: 4.144269 | lrm: 0.98 | dt: 1475.55ms | tok/sec: 11,103 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 80 | total time: 49.17m | eta: 85.3m
step 01835/05000 (36.70%) | loss: 4.152806 | lrm: 0.98 | dt: 1467.56ms | tok/sec: 11,164 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 80 | total time: 49.19m | eta: 85.3m
step 01836/05000 (36.72%) | loss: 4.160751 | lrm: 0.97 | dt: 1510.65ms | tok/sec: 10,845 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 80 | total time: 49.22m | eta: 85.3m
step 01837/05000 (36.74%) | loss: 4.176324 | lrm: 0.97 | dt: 1479.05ms | tok/sec: 11,077 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 80 | total time: 49.24m | eta: 85.3m
step 01838/05000 (36.76%) | loss: 4.171766 | lrm: 0.97 | dt: 1487.01ms | tok/sec: 11,018 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 80 | total time: 49.27m | eta: 85.2m
step 01839/05000 (36.78%) | loss: 4.169161 | lrm: 0.97 | dt: 1519.07ms | tok/sec: 10,785 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 80 | total time: 49.29m | eta: 85.2m
step 01840/05000 (36.80%) | loss: 4.171031 | lrm: 0.97 | dt: 1545.30ms | tok/sec: 10,602 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 80 | total time: 49.32m | eta: 85.2m
step 01841/05000 (36.82%) | loss: 4.178524 | lrm: 0.97 | dt: 1443.47ms | tok/sec: 11,350 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 80 | total time: 49.34m | eta: 85.1m
step 01842/05000 (36.84%) | loss: 4.186367 | lrm: 0.97 | dt: 1489.82ms | tok/sec: 10,997 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 80 | total time: 49.37m | eta: 85.1m
step 01843/05000 (36.86%) | loss: 4.186433 | lrm: 0.97 | dt: 1543.23ms | tok/sec: 10,616 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 80 | total time: 49.40m | eta: 85.1m
step 01844/05000 (36.88%) | loss: 4.194453 | lrm: 0.97 | dt: 1505.02ms | tok/sec: 10,886 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 80 | total time: 49.42m | eta: 85.0m
step 01845/05000 (36.90%) | loss: 4.187542 | lrm: 0.97 | dt: 1494.27ms | tok/sec: 10,964 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 80 | total time: 49.45m | eta: 85.0m
step 01846/05000 (36.92%) | loss: 4.185715 | lrm: 0.97 | dt: 1555.71ms | tok/sec: 10,531 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 80 | total time: 49.47m | eta: 85.0m
step 01847/05000 (36.94%) | loss: 4.178412 | lrm: 0.97 | dt: 1460.75ms | tok/sec: 11,216 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 80 | total time: 49.50m | eta: 85.0m
step 01848/05000 (36.96%) | loss: 4.179043 | lrm: 0.97 | dt: 1469.91ms | tok/sec: 11,146 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 80 | total time: 49.52m | eta: 84.9m
step 01849/05000 (36.98%) | loss: 4.170305 | lrm: 0.97 | dt: 1474.23ms | tok/sec: 11,113 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 80 | total time: 49.54m | eta: 84.9m
step 01850/05000 (37.00%) | loss: 4.182376 | lrm: 0.97 | dt: 1537.62ms | tok/sec: 10,655 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 81 | total time: 49.57m | eta: 84.9m
step 01851/05000 (37.02%) | loss: 4.178977 | lrm: 0.97 | dt: 1554.92ms | tok/sec: 10,536 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 81 | total time: 49.60m | eta: 84.8m
step 01852/05000 (37.04%) | loss: 4.188937 | lrm: 0.97 | dt: 1503.42ms | tok/sec: 10,897 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 81 | total time: 49.62m | eta: 84.8m
step 01853/05000 (37.06%) | loss: 4.176999 | lrm: 0.97 | dt: 1665.42ms | tok/sec: 9,837 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 81 | total time: 49.65m | eta: 84.8m
step 01854/05000 (37.08%) | loss: 4.169875 | lrm: 0.97 | dt: 1541.41ms | tok/sec: 10,629 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 81 | total time: 49.67m | eta: 84.7m
step 01855/05000 (37.10%) | loss: 4.170286 | lrm: 0.97 | dt: 1511.01ms | tok/sec: 10,843 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 81 | total time: 49.70m | eta: 84.7m
step 01856/05000 (37.12%) | loss: 4.159272 | lrm: 0.97 | dt: 1536.33ms | tok/sec: 10,664 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 81 | total time: 49.73m | eta: 84.7m
step 01857/05000 (37.14%) | loss: 4.155950 | lrm: 0.97 | dt: 1562.36ms | tok/sec: 10,486 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 81 | total time: 49.75m | eta: 84.7m
step 01858/05000 (37.16%) | loss: 4.166552 | lrm: 0.97 | dt: 1569.19ms | tok/sec: 10,441 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 81 | total time: 49.78m | eta: 84.6m
step 01859/05000 (37.18%) | loss: 4.166144 | lrm: 0.97 | dt: 1472.04ms | tok/sec: 11,130 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 81 | total time: 49.80m | eta: 84.6m
step 01860/05000 (37.20%) | loss: 4.170571 | lrm: 0.97 | dt: 1526.95ms | tok/sec: 10,729 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 81 | total time: 49.83m | eta: 84.6m
step 01861/05000 (37.22%) | loss: 4.158960 | lrm: 0.97 | dt: 1579.99ms | tok/sec: 10,369 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 81 | total time: 49.85m | eta: 84.5m
step 01862/05000 (37.24%) | loss: 4.172613 | lrm: 0.97 | dt: 1530.66ms | tok/sec: 10,703 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 81 | total time: 49.88m | eta: 84.5m
step 01863/05000 (37.26%) | loss: 4.166399 | lrm: 0.97 | dt: 1547.10ms | tok/sec: 10,590 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 81 | total time: 49.91m | eta: 84.5m
step 01864/05000 (37.28%) | loss: 4.167729 | lrm: 0.97 | dt: 1480.05ms | tok/sec: 11,069 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 81 | total time: 49.93m | eta: 84.5m
step 01865/05000 (37.30%) | loss: 4.169821 | lrm: 0.97 | dt: 1562.43ms | tok/sec: 10,486 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 81 | total time: 49.96m | eta: 84.4m
step 01866/05000 (37.32%) | loss: 4.181543 | lrm: 0.97 | dt: 1461.69ms | tok/sec: 11,208 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 81 | total time: 49.98m | eta: 84.4m
step 01867/05000 (37.34%) | loss: 4.159024 | lrm: 0.97 | dt: 1497.28ms | tok/sec: 10,942 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 81 | total time: 50.01m | eta: 84.4m
step 01868/05000 (37.36%) | loss: 4.154856 | lrm: 0.97 | dt: 1559.94ms | tok/sec: 10,502 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 81 | total time: 50.03m | eta: 84.3m
step 01869/05000 (37.38%) | loss: 4.155169 | lrm: 0.97 | dt: 1557.53ms | tok/sec: 10,519 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 81 | total time: 50.06m | eta: 84.3m
step 01870/05000 (37.40%) | loss: 4.155353 | lrm: 0.96 | dt: 1481.40ms | tok/sec: 11,059 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 81 | total time: 50.08m | eta: 84.3m
step 01871/05000 (37.42%) | loss: 4.150565 | lrm: 0.96 | dt: 1647.28ms | tok/sec: 9,946 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 81 | total time: 50.11m | eta: 84.3m
step 01872/05000 (37.44%) | loss: 4.161815 | lrm: 0.96 | dt: 1643.15ms | tok/sec: 9,971 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 81 | total time: 50.14m | eta: 84.2m
step 01873/05000 (37.46%) | loss: 4.156453 | lrm: 0.96 | dt: 1649.32ms | tok/sec: 9,933 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 81 | total time: 50.16m | eta: 84.2m
step 01874/05000 (37.48%) | loss: 4.153475 | lrm: 0.96 | dt: 1646.08ms | tok/sec: 9,953 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 82 | total time: 50.19m | eta: 84.2m
step 01875/05000 (37.50%) | loss: 4.157906 | lrm: 0.96 | dt: 1628.21ms | tok/sec: 10,062 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 82 | total time: 50.22m | eta: 84.1m
step 01876/05000 (37.52%) | loss: 4.141537 | lrm: 0.96 | dt: 1703.02ms | tok/sec: 9,620 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 82 | total time: 50.25m | eta: 84.1m
step 01877/05000 (37.54%) | loss: 4.127739 | lrm: 0.96 | dt: 1706.92ms | tok/sec: 9,598 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 82 | total time: 50.28m | eta: 84.1m
step 01878/05000 (37.56%) | loss: 4.130966 | lrm: 0.96 | dt: 1708.82ms | tok/sec: 9,587 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 82 | total time: 50.30m | eta: 84.1m
step 01879/05000 (37.58%) | loss: 4.141065 | lrm: 0.96 | dt: 1727.03ms | tok/sec: 9,486 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 82 | total time: 50.33m | eta: 84.0m
step 01880/05000 (37.60%) | loss: 4.148486 | lrm: 0.96 | dt: 1717.96ms | tok/sec: 9,536 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 82 | total time: 50.36m | eta: 84.0m
step 01881/05000 (37.62%) | loss: 4.155564 | lrm: 0.96 | dt: 1717.27ms | tok/sec: 9,540 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 82 | total time: 50.39m | eta: 84.0m
step 01882/05000 (37.64%) | loss: 4.148093 | lrm: 0.96 | dt: 1693.16ms | tok/sec: 9,676 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 82 | total time: 50.42m | eta: 84.0m
step 01883/05000 (37.66%) | loss: 4.143392 | lrm: 0.96 | dt: 1672.49ms | tok/sec: 9,796 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 82 | total time: 50.45m | eta: 84.0m
step 01884/05000 (37.68%) | loss: 4.153867 | lrm: 0.96 | dt: 1739.57ms | tok/sec: 9,418 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 82 | total time: 50.48m | eta: 83.9m
step 01885/05000 (37.70%) | loss: 4.148457 | lrm: 0.96 | dt: 1706.02ms | tok/sec: 9,603 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 82 | total time: 50.50m | eta: 83.9m
step 01886/05000 (37.72%) | loss: 4.146593 | lrm: 0.96 | dt: 1726.91ms | tok/sec: 9,487 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 82 | total time: 50.53m | eta: 83.9m
step 01887/05000 (37.74%) | loss: 4.158440 | lrm: 0.96 | dt: 1675.71ms | tok/sec: 9,777 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 82 | total time: 50.56m | eta: 83.9m
step 01888/05000 (37.76%) | loss: 4.155281 | lrm: 0.96 | dt: 1692.64ms | tok/sec: 9,679 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 82 | total time: 50.59m | eta: 83.8m
step 01889/05000 (37.78%) | loss: 4.159295 | lrm: 0.96 | dt: 1723.31ms | tok/sec: 9,507 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 82 | total time: 50.62m | eta: 83.8m
step 01890/05000 (37.80%) | loss: 4.167693 | lrm: 0.96 | dt: 1719.25ms | tok/sec: 9,529 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 82 | total time: 50.65m | eta: 83.8m
step 01891/05000 (37.82%) | loss: 4.160866 | lrm: 0.96 | dt: 1690.56ms | tok/sec: 9,691 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 82 | total time: 50.67m | eta: 83.8m
step 01892/05000 (37.84%) | loss: 4.153592 | lrm: 0.96 | dt: 1672.76ms | tok/sec: 9,794 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 82 | total time: 50.70m | eta: 83.7m
step 01893/05000 (37.86%) | loss: 4.146229 | lrm: 0.96 | dt: 1681.63ms | tok/sec: 9,742 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 82 | total time: 50.73m | eta: 83.7m
step 01894/05000 (37.88%) | loss: 4.127641 | lrm: 0.96 | dt: 1671.69ms | tok/sec: 9,800 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 82 | total time: 50.76m | eta: 83.7m
step 01895/05000 (37.90%) | loss: 4.134474 | lrm: 0.96 | dt: 1686.37ms | tok/sec: 9,715 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 82 | total time: 50.79m | eta: 83.7m
step 01896/05000 (37.92%) | loss: 4.144238 | lrm: 0.96 | dt: 1720.68ms | tok/sec: 9,521 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 82 | total time: 50.81m | eta: 83.6m
step 01897/05000 (37.94%) | loss: 4.147108 | lrm: 0.96 | dt: 1586.00ms | tok/sec: 10,330 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 83 | total time: 50.84m | eta: 83.6m
step 01898/05000 (37.96%) | loss: 4.146615 | lrm: 0.96 | dt: 1591.43ms | tok/sec: 10,295 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 83 | total time: 50.87m | eta: 83.6m
step 01899/05000 (37.98%) | loss: 4.161533 | lrm: 0.96 | dt: 1621.73ms | tok/sec: 10,102 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 83 | total time: 50.89m | eta: 83.5m
Step 01900 | Validation bpb: 1.287435
<|bos|>The capital of France is the largest producer of the world's largest producer of the world. The capital of
<|bos|>The chemical symbol of gold is the gold, gold, gold, gold, gold, gold, gold, gold
<|bos|>If yesterday was Friday, then tomorrow will be the first time to get to the next level. The first time was the first
<|bos|>The opposite of hot is the heat of the heat. The heat of the heat is the heat of the
<|bos|>The planets of the solar system are: the solar system is the most efficient solar system. The solar system is the most
<|bos|>My favorite color is the white, white, and yellow. I have a white, white, and
<|bos|>If 5*x + 3 = 13, then x is the most common 5*x + 3.5. The most common
step 01900/05000 (38.00%) | loss: 4.178792 | lrm: 0.96 | dt: 1852.08ms | tok/sec: 8,846 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 83 | total time: 50.93m | eta: 83.5m
step 01901/05000 (38.02%) | loss: 4.174801 | lrm: 0.96 | dt: 1493.41ms | tok/sec: 10,970 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 83 | total time: 50.95m | eta: 83.5m
step 01902/05000 (38.04%) | loss: 4.163948 | lrm: 0.96 | dt: 1496.89ms | tok/sec: 10,945 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 83 | total time: 50.98m | eta: 83.5m
step 01903/05000 (38.06%) | loss: 4.142833 | lrm: 0.96 | dt: 1502.06ms | tok/sec: 10,907 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 83 | total time: 51.00m | eta: 83.4m
step 01904/05000 (38.08%) | loss: 4.157401 | lrm: 0.95 | dt: 1505.05ms | tok/sec: 10,886 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 83 | total time: 51.03m | eta: 83.4m
step 01905/05000 (38.10%) | loss: 4.160084 | lrm: 0.95 | dt: 1519.18ms | tok/sec: 10,784 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 83 | total time: 51.05m | eta: 83.4m
step 01906/05000 (38.12%) | loss: 4.166875 | lrm: 0.95 | dt: 1489.31ms | tok/sec: 11,001 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 83 | total time: 51.08m | eta: 83.3m
step 01907/05000 (38.14%) | loss: 4.175943 | lrm: 0.95 | dt: 1502.29ms | tok/sec: 10,905 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 83 | total time: 51.10m | eta: 83.3m
step 01908/05000 (38.16%) | loss: 4.155388 | lrm: 0.95 | dt: 1506.89ms | tok/sec: 10,872 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 83 | total time: 51.13m | eta: 83.3m
step 01909/05000 (38.18%) | loss: 4.147561 | lrm: 0.95 | dt: 1476.23ms | tok/sec: 11,098 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 83 | total time: 51.15m | eta: 83.3m
step 01910/05000 (38.20%) | loss: 4.148418 | lrm: 0.95 | dt: 1474.85ms | tok/sec: 11,108 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 83 | total time: 51.17m | eta: 83.2m
step 01911/05000 (38.22%) | loss: 4.150230 | lrm: 0.95 | dt: 1587.82ms | tok/sec: 10,318 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 83 | total time: 51.20m | eta: 83.2m
step 01912/05000 (38.24%) | loss: 4.147626 | lrm: 0.95 | dt: 1524.24ms | tok/sec: 10,748 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 83 | total time: 51.23m | eta: 83.2m
step 01913/05000 (38.26%) | loss: 4.158438 | lrm: 0.95 | dt: 1482.18ms | tok/sec: 11,053 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 83 | total time: 51.25m | eta: 83.1m
step 01914/05000 (38.28%) | loss: 4.152022 | lrm: 0.95 | dt: 1530.60ms | tok/sec: 10,704 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 83 | total time: 51.28m | eta: 83.1m
step 01915/05000 (38.30%) | loss: 4.141096 | lrm: 0.95 | dt: 1526.75ms | tok/sec: 10,731 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 83 | total time: 51.30m | eta: 83.1m
step 01916/05000 (38.32%) | loss: 4.151792 | lrm: 0.95 | dt: 1465.36ms | tok/sec: 11,180 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 83 | total time: 51.33m | eta: 83.0m
step 01917/05000 (38.34%) | loss: 4.154774 | lrm: 0.95 | dt: 1486.74ms | tok/sec: 11,020 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 83 | total time: 51.35m | eta: 83.0m
step 01918/05000 (38.36%) | loss: 4.144466 | lrm: 0.95 | dt: 1501.91ms | tok/sec: 10,908 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 83 | total time: 51.38m | eta: 83.0m
step 01919/05000 (38.38%) | loss: 4.137300 | lrm: 0.95 | dt: 1511.35ms | tok/sec: 10,840 | bf16_mfu: 0.00 | epoch: 1 pq: 0 rg: 83 | total time: 51.40m | eta: 83.0m
step 01920/05000 (38.40%) | loss: 4.138108 | lrm: 0.95 | dt: 1497.29ms | tok/sec: 10,942 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 0 | total time: 51.43m | eta: 82.9m
step 01921/05000 (38.42%) | loss: 4.136258 | lrm: 0.95 | dt: 1513.95ms | tok/sec: 10,822 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 0 | total time: 51.45m | eta: 82.9m
step 01922/05000 (38.44%) | loss: 4.143231 | lrm: 0.95 | dt: 1457.01ms | tok/sec: 11,244 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 0 | total time: 51.48m | eta: 82.9m
step 01923/05000 (38.46%) | loss: 4.142170 | lrm: 0.95 | dt: 1589.01ms | tok/sec: 10,310 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 0 | total time: 51.50m | eta: 82.8m
step 01924/05000 (38.48%) | loss: 4.140824 | lrm: 0.95 | dt: 1486.59ms | tok/sec: 11,021 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 0 | total time: 51.53m | eta: 82.8m
step 01925/05000 (38.50%) | loss: 4.151864 | lrm: 0.95 | dt: 1565.15ms | tok/sec: 10,467 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 0 | total time: 51.55m | eta: 82.8m
step 01926/05000 (38.52%) | loss: 4.151452 | lrm: 0.95 | dt: 1486.20ms | tok/sec: 11,024 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 0 | total time: 51.58m | eta: 82.8m
step 01927/05000 (38.54%) | loss: 4.160161 | lrm: 0.95 | dt: 1491.77ms | tok/sec: 10,982 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 0 | total time: 51.60m | eta: 82.7m
step 01928/05000 (38.56%) | loss: 4.165730 | lrm: 0.95 | dt: 1472.01ms | tok/sec: 11,130 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 0 | total time: 51.63m | eta: 82.7m
step 01929/05000 (38.58%) | loss: 4.168494 | lrm: 0.95 | dt: 1465.54ms | tok/sec: 11,179 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 0 | total time: 51.65m | eta: 82.7m
step 01930/05000 (38.60%) | loss: 4.165345 | lrm: 0.95 | dt: 1503.53ms | tok/sec: 10,897 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 0 | total time: 51.68m | eta: 82.6m
step 01931/05000 (38.62%) | loss: 4.160117 | lrm: 0.95 | dt: 1498.01ms | tok/sec: 10,937 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 0 | total time: 51.70m | eta: 82.6m
step 01932/05000 (38.64%) | loss: 4.150128 | lrm: 0.95 | dt: 1434.31ms | tok/sec: 11,422 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 0 | total time: 51.73m | eta: 82.6m
step 01933/05000 (38.66%) | loss: 4.148916 | lrm: 0.95 | dt: 1467.05ms | tok/sec: 11,167 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 0 | total time: 51.75m | eta: 82.5m
step 01934/05000 (38.68%) | loss: 4.147369 | lrm: 0.95 | dt: 1502.88ms | tok/sec: 10,901 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 0 | total time: 51.78m | eta: 82.5m
step 01935/05000 (38.70%) | loss: 4.149946 | lrm: 0.95 | dt: 1561.05ms | tok/sec: 10,495 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 0 | total time: 51.80m | eta: 82.5m
step 01936/05000 (38.72%) | loss: 4.153436 | lrm: 0.95 | dt: 1706.26ms | tok/sec: 9,602 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 0 | total time: 51.83m | eta: 82.5m
step 01937/05000 (38.74%) | loss: 4.153895 | lrm: 0.95 | dt: 1593.38ms | tok/sec: 10,282 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 0 | total time: 51.86m | eta: 82.4m
step 01938/05000 (38.76%) | loss: 4.155788 | lrm: 0.95 | dt: 1619.53ms | tok/sec: 10,116 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 0 | total time: 51.88m | eta: 82.4m
step 01939/05000 (38.78%) | loss: 4.162402 | lrm: 0.94 | dt: 1634.43ms | tok/sec: 10,024 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 0 | total time: 51.91m | eta: 82.4m
step 01940/05000 (38.80%) | loss: 4.160663 | lrm: 0.94 | dt: 1701.07ms | tok/sec: 9,631 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 0 | total time: 51.94m | eta: 82.3m
step 01941/05000 (38.82%) | loss: 4.163008 | lrm: 0.94 | dt: 1587.33ms | tok/sec: 10,321 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 0 | total time: 51.97m | eta: 82.3m
step 01942/05000 (38.84%) | loss: 4.167439 | lrm: 0.94 | dt: 1500.17ms | tok/sec: 10,921 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 0 | total time: 51.99m | eta: 82.3m
step 01943/05000 (38.86%) | loss: 4.166922 | lrm: 0.94 | dt: 1556.97ms | tok/sec: 10,523 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 1 | total time: 52.02m | eta: 82.3m
step 01944/05000 (38.88%) | loss: 4.166183 | lrm: 0.94 | dt: 1495.33ms | tok/sec: 10,956 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 1 | total time: 52.04m | eta: 82.2m
step 01945/05000 (38.90%) | loss: 4.179127 | lrm: 0.94 | dt: 1569.00ms | tok/sec: 10,442 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 1 | total time: 52.07m | eta: 82.2m
step 01946/05000 (38.92%) | loss: 4.174892 | lrm: 0.94 | dt: 1487.34ms | tok/sec: 11,015 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 1 | total time: 52.09m | eta: 82.2m
step 01947/05000 (38.94%) | loss: 4.182971 | lrm: 0.94 | dt: 1547.43ms | tok/sec: 10,587 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 1 | total time: 52.12m | eta: 82.1m
step 01948/05000 (38.96%) | loss: 4.185415 | lrm: 0.94 | dt: 1537.30ms | tok/sec: 10,657 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 1 | total time: 52.14m | eta: 82.1m
step 01949/05000 (38.98%) | loss: 4.201294 | lrm: 0.94 | dt: 1448.42ms | tok/sec: 11,311 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 1 | total time: 52.17m | eta: 82.1m
step 01950/05000 (39.00%) | loss: 4.184675 | lrm: 0.94 | dt: 1530.18ms | tok/sec: 10,707 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 1 | total time: 52.19m | eta: 82.1m
step 01951/05000 (39.02%) | loss: 4.186942 | lrm: 0.94 | dt: 1484.91ms | tok/sec: 11,033 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 1 | total time: 52.22m | eta: 82.0m
step 01952/05000 (39.04%) | loss: 4.190602 | lrm: 0.94 | dt: 1540.81ms | tok/sec: 10,633 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 1 | total time: 52.24m | eta: 82.0m
step 01953/05000 (39.06%) | loss: 4.180248 | lrm: 0.94 | dt: 1481.17ms | tok/sec: 11,061 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 1 | total time: 52.27m | eta: 82.0m
step 01954/05000 (39.08%) | loss: 4.174236 | lrm: 0.94 | dt: 1568.50ms | tok/sec: 10,445 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 1 | total time: 52.29m | eta: 81.9m
step 01955/05000 (39.10%) | loss: 4.175146 | lrm: 0.94 | dt: 1454.73ms | tok/sec: 11,262 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 1 | total time: 52.32m | eta: 81.9m
step 01956/05000 (39.12%) | loss: 4.172093 | lrm: 0.94 | dt: 1528.81ms | tok/sec: 10,716 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 1 | total time: 52.34m | eta: 81.9m
step 01957/05000 (39.14%) | loss: 4.176412 | lrm: 0.94 | dt: 1504.22ms | tok/sec: 10,891 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 1 | total time: 52.37m | eta: 81.8m
step 01958/05000 (39.16%) | loss: 4.175688 | lrm: 0.94 | dt: 1552.14ms | tok/sec: 10,555 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 1 | total time: 52.40m | eta: 81.8m
step 01959/05000 (39.18%) | loss: 4.181948 | lrm: 0.94 | dt: 1505.00ms | tok/sec: 10,886 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 1 | total time: 52.42m | eta: 81.8m
step 01960/05000 (39.20%) | loss: 4.177166 | lrm: 0.94 | dt: 1415.30ms | tok/sec: 11,576 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 1 | total time: 52.44m | eta: 81.8m
step 01961/05000 (39.22%) | loss: 4.174237 | lrm: 0.94 | dt: 1511.06ms | tok/sec: 10,842 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 1 | total time: 52.47m | eta: 81.7m
step 01962/05000 (39.24%) | loss: 4.170782 | lrm: 0.94 | dt: 1530.05ms | tok/sec: 10,708 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 1 | total time: 52.49m | eta: 81.7m
step 01963/05000 (39.26%) | loss: 4.161994 | lrm: 0.94 | dt: 1486.87ms | tok/sec: 11,019 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 1 | total time: 52.52m | eta: 81.7m
step 01964/05000 (39.28%) | loss: 4.151121 | lrm: 0.94 | dt: 1525.09ms | tok/sec: 10,742 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 1 | total time: 52.54m | eta: 81.6m
step 01965/05000 (39.30%) | loss: 4.156833 | lrm: 0.94 | dt: 1555.09ms | tok/sec: 10,535 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 1 | total time: 52.57m | eta: 81.6m
step 01966/05000 (39.32%) | loss: 4.145785 | lrm: 0.94 | dt: 1535.87ms | tok/sec: 10,667 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 2 | total time: 52.60m | eta: 81.6m
step 01967/05000 (39.34%) | loss: 4.135003 | lrm: 0.94 | dt: 1549.61ms | tok/sec: 10,572 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 2 | total time: 52.62m | eta: 81.6m
step 01968/05000 (39.36%) | loss: 4.136606 | lrm: 0.94 | dt: 1455.14ms | tok/sec: 11,259 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 2 | total time: 52.65m | eta: 81.5m
step 01969/05000 (39.38%) | loss: 4.160287 | lrm: 0.94 | dt: 1504.36ms | tok/sec: 10,890 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 2 | total time: 52.67m | eta: 81.5m
step 01970/05000 (39.40%) | loss: 4.150330 | lrm: 0.94 | dt: 1486.87ms | tok/sec: 11,019 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 2 | total time: 52.70m | eta: 81.5m
step 01971/05000 (39.42%) | loss: 4.153712 | lrm: 0.94 | dt: 1608.01ms | tok/sec: 10,189 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 2 | total time: 52.72m | eta: 81.4m
step 01972/05000 (39.44%) | loss: 4.150571 | lrm: 0.94 | dt: 1468.99ms | tok/sec: 11,153 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 2 | total time: 52.75m | eta: 81.4m
step 01973/05000 (39.46%) | loss: 4.151434 | lrm: 0.93 | dt: 1577.12ms | tok/sec: 10,388 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 2 | total time: 52.77m | eta: 81.4m
step 01974/05000 (39.48%) | loss: 4.146141 | lrm: 0.93 | dt: 1445.70ms | tok/sec: 11,332 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 2 | total time: 52.80m | eta: 81.3m
step 01975/05000 (39.50%) | loss: 4.148183 | lrm: 0.93 | dt: 1505.89ms | tok/sec: 10,879 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 2 | total time: 52.82m | eta: 81.3m
step 01976/05000 (39.52%) | loss: 4.147314 | lrm: 0.93 | dt: 1472.19ms | tok/sec: 11,128 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 2 | total time: 52.85m | eta: 81.3m
step 01977/05000 (39.54%) | loss: 4.144958 | lrm: 0.93 | dt: 1506.74ms | tok/sec: 10,873 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 2 | total time: 52.87m | eta: 81.3m
step 01978/05000 (39.56%) | loss: 4.148803 | lrm: 0.93 | dt: 1574.95ms | tok/sec: 10,402 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 2 | total time: 52.90m | eta: 81.2m
step 01979/05000 (39.58%) | loss: 4.147574 | lrm: 0.93 | dt: 1524.59ms | tok/sec: 10,746 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 2 | total time: 52.92m | eta: 81.2m
step 01980/05000 (39.60%) | loss: 4.143442 | lrm: 0.93 | dt: 1490.51ms | tok/sec: 10,992 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 2 | total time: 52.95m | eta: 81.2m
step 01981/05000 (39.62%) | loss: 4.128761 | lrm: 0.93 | dt: 1578.45ms | tok/sec: 10,379 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 2 | total time: 52.98m | eta: 81.1m
step 01982/05000 (39.64%) | loss: 4.136325 | lrm: 0.93 | dt: 1480.21ms | tok/sec: 11,068 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 2 | total time: 53.00m | eta: 81.1m
step 01983/05000 (39.66%) | loss: 4.144330 | lrm: 0.93 | dt: 1563.00ms | tok/sec: 10,482 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 2 | total time: 53.03m | eta: 81.1m
step 01984/05000 (39.68%) | loss: 4.150953 | lrm: 0.93 | dt: 1510.96ms | tok/sec: 10,843 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 2 | total time: 53.05m | eta: 81.1m
step 01985/05000 (39.70%) | loss: 4.143237 | lrm: 0.93 | dt: 1574.48ms | tok/sec: 10,405 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 2 | total time: 53.08m | eta: 81.0m
step 01986/05000 (39.72%) | loss: 4.153989 | lrm: 0.93 | dt: 1511.88ms | tok/sec: 10,836 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 2 | total time: 53.10m | eta: 81.0m
step 01987/05000 (39.74%) | loss: 4.160667 | lrm: 0.93 | dt: 1469.49ms | tok/sec: 11,149 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 2 | total time: 53.13m | eta: 81.0m
step 01988/05000 (39.76%) | loss: 4.161022 | lrm: 0.93 | dt: 1586.11ms | tok/sec: 10,329 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 2 | total time: 53.15m | eta: 80.9m
step 01989/05000 (39.78%) | loss: 4.158626 | lrm: 0.93 | dt: 1481.82ms | tok/sec: 11,056 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 3 | total time: 53.18m | eta: 80.9m
step 01990/05000 (39.80%) | loss: 4.128100 | lrm: 0.93 | dt: 1491.32ms | tok/sec: 10,986 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 3 | total time: 53.20m | eta: 80.9m
step 01991/05000 (39.82%) | loss: 4.120605 | lrm: 0.93 | dt: 1529.42ms | tok/sec: 10,712 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 3 | total time: 53.23m | eta: 80.9m
step 01992/05000 (39.84%) | loss: 4.124595 | lrm: 0.93 | dt: 1539.60ms | tok/sec: 10,641 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 3 | total time: 53.25m | eta: 80.8m
step 01993/05000 (39.86%) | loss: 4.123150 | lrm: 0.93 | dt: 1510.60ms | tok/sec: 10,846 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 3 | total time: 53.28m | eta: 80.8m
step 01994/05000 (39.88%) | loss: 4.119194 | lrm: 0.93 | dt: 1474.28ms | tok/sec: 11,113 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 3 | total time: 53.30m | eta: 80.8m
step 01995/05000 (39.90%) | loss: 4.124558 | lrm: 0.93 | dt: 1444.93ms | tok/sec: 11,338 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 3 | total time: 53.33m | eta: 80.7m
step 01996/05000 (39.92%) | loss: 4.136748 | lrm: 0.93 | dt: 1433.82ms | tok/sec: 11,426 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 3 | total time: 53.35m | eta: 80.7m
step 01997/05000 (39.94%) | loss: 4.138933 | lrm: 0.93 | dt: 1547.62ms | tok/sec: 10,586 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 3 | total time: 53.38m | eta: 80.7m
step 01998/05000 (39.96%) | loss: 4.130966 | lrm: 0.93 | dt: 1552.92ms | tok/sec: 10,550 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 3 | total time: 53.40m | eta: 80.6m
step 01999/05000 (39.98%) | loss: 4.130789 | lrm: 0.93 | dt: 1554.65ms | tok/sec: 10,538 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 3 | total time: 53.43m | eta: 80.6m
Step 02000 | Validation bpb: 1.279230
<|bos|>The capital of France is the first of the 20th century, and the first of the 20
<|bos|>The chemical symbol of gold is the symbol of the symbol of the symbol of the symbol of the symbol of the
<|bos|>If yesterday was Friday, then tomorrow will be a chance to see the world's population of the world's population. The world
<|bos|>The opposite of hot is the heat of the air. The heat of the air is the heat of the
<|bos|>The planets of the solar system are: the solar system, the solar system, the solar system, the solar system,
<|bos|>My favorite color is the red, green, and yellow. I've been using the color to create
<|bos|>If 5*x + 3 = 13, then x is 1.5.5.5.5.5.5.5
step 02000/05000 (40.00%) | loss: 4.141383 | lrm: 0.93 | dt: 1564.19ms | tok/sec: 10,474 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 3 | total time: 53.46m | eta: 80.6m
step 02001/05000 (40.02%) | loss: 4.130139 | lrm: 0.93 | dt: 1535.40ms | tok/sec: 10,670 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 3 | total time: 53.48m | eta: 80.6m
step 02002/05000 (40.04%) | loss: 4.141198 | lrm: 0.93 | dt: 1511.54ms | tok/sec: 10,839 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 3 | total time: 53.51m | eta: 80.5m
step 02003/05000 (40.06%) | loss: 4.165167 | lrm: 0.93 | dt: 1531.58ms | tok/sec: 10,697 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 3 | total time: 53.53m | eta: 80.5m
step 02004/05000 (40.08%) | loss: 4.152363 | lrm: 0.93 | dt: 1555.46ms | tok/sec: 10,533 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 3 | total time: 53.56m | eta: 80.5m
step 02005/05000 (40.10%) | loss: 4.142356 | lrm: 0.93 | dt: 1545.80ms | tok/sec: 10,599 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 3 | total time: 53.58m | eta: 80.4m
step 02006/05000 (40.12%) | loss: 4.155129 | lrm: 0.93 | dt: 1453.45ms | tok/sec: 11,272 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 3 | total time: 53.61m | eta: 80.4m
step 02007/05000 (40.14%) | loss: 4.146686 | lrm: 0.92 | dt: 1472.13ms | tok/sec: 11,129 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 3 | total time: 53.63m | eta: 80.4m
step 02008/05000 (40.16%) | loss: 4.148149 | lrm: 0.92 | dt: 1505.74ms | tok/sec: 10,881 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 3 | total time: 53.66m | eta: 80.4m
step 02009/05000 (40.18%) | loss: 4.146667 | lrm: 0.92 | dt: 1538.27ms | tok/sec: 10,650 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 3 | total time: 53.68m | eta: 80.3m
step 02010/05000 (40.20%) | loss: 4.145867 | lrm: 0.92 | dt: 1507.13ms | tok/sec: 10,870 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 3 | total time: 53.71m | eta: 80.3m
step 02011/05000 (40.22%) | loss: 4.130341 | lrm: 0.92 | dt: 1535.17ms | tok/sec: 10,672 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 3 | total time: 53.73m | eta: 80.3m
step 02012/05000 (40.24%) | loss: 4.138189 | lrm: 0.92 | dt: 1718.23ms | tok/sec: 9,535 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 3 | total time: 53.76m | eta: 80.2m
step 02013/05000 (40.26%) | loss: 4.133431 | lrm: 0.92 | dt: 1778.58ms | tok/sec: 9,211 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 4 | total time: 53.79m | eta: 80.2m
step 02014/05000 (40.28%) | loss: 4.135354 | lrm: 0.92 | dt: 1676.74ms | tok/sec: 9,771 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 4 | total time: 53.82m | eta: 80.2m
step 02015/05000 (40.30%) | loss: 4.131654 | lrm: 0.92 | dt: 1672.13ms | tok/sec: 9,798 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 4 | total time: 53.85m | eta: 80.2m
step 02016/05000 (40.32%) | loss: 4.141785 | lrm: 0.92 | dt: 1610.98ms | tok/sec: 10,170 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 4 | total time: 53.88m | eta: 80.1m
step 02017/05000 (40.34%) | loss: 4.132666 | lrm: 0.92 | dt: 1631.21ms | tok/sec: 10,044 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 4 | total time: 53.90m | eta: 80.1m
step 02018/05000 (40.36%) | loss: 4.133260 | lrm: 0.92 | dt: 1577.57ms | tok/sec: 10,385 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 4 | total time: 53.93m | eta: 80.1m
step 02019/05000 (40.38%) | loss: 4.131455 | lrm: 0.92 | dt: 1618.94ms | tok/sec: 10,120 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 4 | total time: 53.96m | eta: 80.1m
step 02020/05000 (40.40%) | loss: 4.133163 | lrm: 0.92 | dt: 1604.04ms | tok/sec: 10,214 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 4 | total time: 53.98m | eta: 80.0m
step 02021/05000 (40.42%) | loss: 4.124138 | lrm: 0.92 | dt: 1796.03ms | tok/sec: 9,122 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 4 | total time: 54.01m | eta: 80.0m
step 02022/05000 (40.44%) | loss: 4.136732 | lrm: 0.92 | dt: 1600.16ms | tok/sec: 10,238 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 4 | total time: 54.04m | eta: 80.0m
step 02023/05000 (40.46%) | loss: 4.145719 | lrm: 0.92 | dt: 1676.11ms | tok/sec: 9,774 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 4 | total time: 54.07m | eta: 80.0m
step 02024/05000 (40.48%) | loss: 4.156994 | lrm: 0.92 | dt: 1577.79ms | tok/sec: 10,384 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 4 | total time: 54.09m | eta: 79.9m
step 02025/05000 (40.50%) | loss: 4.151285 | lrm: 0.92 | dt: 1685.40ms | tok/sec: 9,721 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 4 | total time: 54.12m | eta: 79.9m
step 02026/05000 (40.52%) | loss: 4.137348 | lrm: 0.92 | dt: 1572.60ms | tok/sec: 10,418 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 4 | total time: 54.15m | eta: 79.9m
step 02027/05000 (40.54%) | loss: 4.128885 | lrm: 0.92 | dt: 1634.38ms | tok/sec: 10,024 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 4 | total time: 54.17m | eta: 79.9m
step 02028/05000 (40.56%) | loss: 4.118783 | lrm: 0.92 | dt: 1647.66ms | tok/sec: 9,943 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 4 | total time: 54.20m | eta: 79.8m
step 02029/05000 (40.58%) | loss: 4.117896 | lrm: 0.92 | dt: 1699.78ms | tok/sec: 9,638 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 4 | total time: 54.23m | eta: 79.8m
step 02030/05000 (40.60%) | loss: 4.119959 | lrm: 0.92 | dt: 1657.62ms | tok/sec: 9,884 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 4 | total time: 54.26m | eta: 79.8m
step 02031/05000 (40.62%) | loss: 4.124179 | lrm: 0.92 | dt: 1692.18ms | tok/sec: 9,682 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 4 | total time: 54.29m | eta: 79.8m
step 02032/05000 (40.64%) | loss: 4.123807 | lrm: 0.92 | dt: 1571.51ms | tok/sec: 10,425 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 4 | total time: 54.31m | eta: 79.7m
step 02033/05000 (40.66%) | loss: 4.124326 | lrm: 0.92 | dt: 1589.85ms | tok/sec: 10,305 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 4 | total time: 54.34m | eta: 79.7m
step 02034/05000 (40.68%) | loss: 4.128609 | lrm: 0.92 | dt: 1525.25ms | tok/sec: 10,741 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 4 | total time: 54.36m | eta: 79.7m
step 02035/05000 (40.70%) | loss: 4.117361 | lrm: 0.92 | dt: 1562.97ms | tok/sec: 10,482 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 4 | total time: 54.39m | eta: 79.6m
step 02036/05000 (40.72%) | loss: 4.106144 | lrm: 0.92 | dt: 1601.51ms | tok/sec: 10,230 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 4 | total time: 54.42m | eta: 79.6m
step 02037/05000 (40.74%) | loss: 4.107046 | lrm: 0.92 | dt: 1635.17ms | tok/sec: 10,019 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 5 | total time: 54.44m | eta: 79.6m
step 02038/05000 (40.76%) | loss: 4.103692 | lrm: 0.92 | dt: 1595.07ms | tok/sec: 10,271 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 5 | total time: 54.47m | eta: 79.6m
step 02039/05000 (40.78%) | loss: 4.101982 | lrm: 0.92 | dt: 1592.27ms | tok/sec: 10,289 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 5 | total time: 54.50m | eta: 79.5m
step 02040/05000 (40.80%) | loss: 4.120399 | lrm: 0.92 | dt: 1602.60ms | tok/sec: 10,223 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 5 | total time: 54.52m | eta: 79.5m
step 02041/05000 (40.82%) | loss: 4.119705 | lrm: 0.91 | dt: 1598.82ms | tok/sec: 10,247 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 5 | total time: 54.55m | eta: 79.5m
step 02042/05000 (40.84%) | loss: 4.115309 | lrm: 0.91 | dt: 1566.00ms | tok/sec: 10,462 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 5 | total time: 54.58m | eta: 79.4m
step 02043/05000 (40.86%) | loss: 4.118149 | lrm: 0.91 | dt: 1550.94ms | tok/sec: 10,563 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 5 | total time: 54.60m | eta: 79.4m
step 02044/05000 (40.88%) | loss: 4.125391 | lrm: 0.91 | dt: 1576.40ms | tok/sec: 10,393 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 5 | total time: 54.63m | eta: 79.4m
step 02045/05000 (40.90%) | loss: 4.133238 | lrm: 0.91 | dt: 1598.02ms | tok/sec: 10,252 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 5 | total time: 54.66m | eta: 79.4m
step 02046/05000 (40.92%) | loss: 4.142349 | lrm: 0.91 | dt: 1541.10ms | tok/sec: 10,631 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 5 | total time: 54.68m | eta: 79.3m
step 02047/05000 (40.94%) | loss: 4.158475 | lrm: 0.91 | dt: 1585.51ms | tok/sec: 10,333 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 5 | total time: 54.71m | eta: 79.3m
step 02048/05000 (40.96%) | loss: 4.142666 | lrm: 0.91 | dt: 1562.97ms | tok/sec: 10,482 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 5 | total time: 54.73m | eta: 79.3m
step 02049/05000 (40.98%) | loss: 4.145631 | lrm: 0.91 | dt: 1554.49ms | tok/sec: 10,539 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 5 | total time: 54.76m | eta: 79.3m
step 02050/05000 (41.00%) | loss: 4.142644 | lrm: 0.91 | dt: 1518.03ms | tok/sec: 10,792 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 5 | total time: 54.79m | eta: 79.2m
step 02051/05000 (41.02%) | loss: 4.147114 | lrm: 0.91 | dt: 1609.60ms | tok/sec: 10,178 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 5 | total time: 54.81m | eta: 79.2m
step 02052/05000 (41.04%) | loss: 4.129386 | lrm: 0.91 | dt: 1600.51ms | tok/sec: 10,236 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 5 | total time: 54.84m | eta: 79.2m
step 02053/05000 (41.06%) | loss: 4.118466 | lrm: 0.91 | dt: 1557.66ms | tok/sec: 10,518 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 5 | total time: 54.86m | eta: 79.1m
step 02054/05000 (41.08%) | loss: 4.114503 | lrm: 0.91 | dt: 1567.54ms | tok/sec: 10,452 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 5 | total time: 54.89m | eta: 79.1m
step 02055/05000 (41.10%) | loss: 4.108536 | lrm: 0.91 | dt: 1691.11ms | tok/sec: 9,688 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 5 | total time: 54.92m | eta: 79.1m
step 02056/05000 (41.12%) | loss: 4.109921 | lrm: 0.91 | dt: 1577.26ms | tok/sec: 10,387 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 5 | total time: 54.95m | eta: 79.1m
step 02057/05000 (41.14%) | loss: 4.094553 | lrm: 0.91 | dt: 1550.75ms | tok/sec: 10,565 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 5 | total time: 54.97m | eta: 79.0m
step 02058/05000 (41.16%) | loss: 4.099258 | lrm: 0.91 | dt: 1606.00ms | tok/sec: 10,201 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 5 | total time: 55.00m | eta: 79.0m
step 02059/05000 (41.18%) | loss: 4.108445 | lrm: 0.91 | dt: 1709.22ms | tok/sec: 9,585 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 5 | total time: 55.03m | eta: 79.0m
step 02060/05000 (41.20%) | loss: 4.107078 | lrm: 0.91 | dt: 1675.74ms | tok/sec: 9,777 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 6 | total time: 55.05m | eta: 79.0m
step 02061/05000 (41.22%) | loss: 4.096143 | lrm: 0.91 | dt: 1598.70ms | tok/sec: 10,248 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 6 | total time: 55.08m | eta: 78.9m
step 02062/05000 (41.24%) | loss: 4.104498 | lrm: 0.91 | dt: 1624.80ms | tok/sec: 10,083 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 6 | total time: 55.11m | eta: 78.9m
step 02063/05000 (41.26%) | loss: 4.103228 | lrm: 0.91 | dt: 1657.40ms | tok/sec: 9,885 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 6 | total time: 55.14m | eta: 78.9m
step 02064/05000 (41.28%) | loss: 4.115172 | lrm: 0.91 | dt: 1525.99ms | tok/sec: 10,736 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 6 | total time: 55.16m | eta: 78.8m
step 02065/05000 (41.30%) | loss: 4.114642 | lrm: 0.91 | dt: 1608.79ms | tok/sec: 10,184 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 6 | total time: 55.19m | eta: 78.8m
step 02066/05000 (41.32%) | loss: 4.106708 | lrm: 0.91 | dt: 1581.13ms | tok/sec: 10,362 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 6 | total time: 55.21m | eta: 78.8m
step 02067/05000 (41.34%) | loss: 4.105143 | lrm: 0.91 | dt: 1598.13ms | tok/sec: 10,251 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 6 | total time: 55.24m | eta: 78.8m
step 02068/05000 (41.36%) | loss: 4.105891 | lrm: 0.91 | dt: 1601.65ms | tok/sec: 10,229 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 6 | total time: 55.27m | eta: 78.7m
step 02069/05000 (41.38%) | loss: 4.116155 | lrm: 0.91 | dt: 1606.54ms | tok/sec: 10,198 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 6 | total time: 55.29m | eta: 78.7m
step 02070/05000 (41.40%) | loss: 4.109868 | lrm: 0.91 | dt: 1568.41ms | tok/sec: 10,446 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 6 | total time: 55.32m | eta: 78.7m
step 02071/05000 (41.42%) | loss: 4.113945 | lrm: 0.91 | dt: 1543.21ms | tok/sec: 10,616 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 6 | total time: 55.35m | eta: 78.7m
step 02072/05000 (41.44%) | loss: 4.107801 | lrm: 0.91 | dt: 1642.10ms | tok/sec: 9,977 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 6 | total time: 55.37m | eta: 78.6m
step 02073/05000 (41.46%) | loss: 4.121058 | lrm: 0.91 | dt: 1577.50ms | tok/sec: 10,386 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 6 | total time: 55.40m | eta: 78.6m
step 02074/05000 (41.48%) | loss: 4.130859 | lrm: 0.91 | dt: 1555.56ms | tok/sec: 10,532 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 6 | total time: 55.43m | eta: 78.6m
step 02075/05000 (41.50%) | loss: 4.118174 | lrm: 0.91 | dt: 1586.37ms | tok/sec: 10,328 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 6 | total time: 55.45m | eta: 78.5m
step 02076/05000 (41.52%) | loss: 4.131333 | lrm: 0.90 | dt: 1622.03ms | tok/sec: 10,100 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 6 | total time: 55.48m | eta: 78.5m
step 02077/05000 (41.54%) | loss: 4.118870 | lrm: 0.90 | dt: 1594.71ms | tok/sec: 10,273 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 6 | total time: 55.51m | eta: 78.5m
step 02078/05000 (41.56%) | loss: 4.116034 | lrm: 0.90 | dt: 1549.91ms | tok/sec: 10,570 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 6 | total time: 55.53m | eta: 78.5m
step 02079/05000 (41.58%) | loss: 4.101819 | lrm: 0.90 | dt: 1531.49ms | tok/sec: 10,698 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 6 | total time: 55.56m | eta: 78.4m
step 02080/05000 (41.60%) | loss: 4.095542 | lrm: 0.90 | dt: 1560.70ms | tok/sec: 10,497 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 6 | total time: 55.58m | eta: 78.4m
step 02081/05000 (41.62%) | loss: 4.102214 | lrm: 0.90 | dt: 1642.14ms | tok/sec: 9,977 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 6 | total time: 55.61m | eta: 78.4m
step 02082/05000 (41.64%) | loss: 4.116606 | lrm: 0.90 | dt: 1697.96ms | tok/sec: 9,649 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 6 | total time: 55.64m | eta: 78.4m
step 02083/05000 (41.66%) | loss: 4.116551 | lrm: 0.90 | dt: 1618.46ms | tok/sec: 10,123 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 7 | total time: 55.67m | eta: 78.3m
step 02084/05000 (41.68%) | loss: 4.111579 | lrm: 0.90 | dt: 1579.76ms | tok/sec: 10,371 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 7 | total time: 55.69m | eta: 78.3m
step 02085/05000 (41.70%) | loss: 4.123969 | lrm: 0.90 | dt: 1606.50ms | tok/sec: 10,198 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 7 | total time: 55.72m | eta: 78.3m
step 02086/05000 (41.72%) | loss: 4.117846 | lrm: 0.90 | dt: 1579.80ms | tok/sec: 10,370 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 7 | total time: 55.75m | eta: 78.2m
step 02087/05000 (41.74%) | loss: 4.102858 | lrm: 0.90 | dt: 1562.31ms | tok/sec: 10,487 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 7 | total time: 55.77m | eta: 78.2m
step 02088/05000 (41.76%) | loss: 4.106764 | lrm: 0.90 | dt: 1574.33ms | tok/sec: 10,406 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 7 | total time: 55.80m | eta: 78.2m
step 02089/05000 (41.78%) | loss: 4.089001 | lrm: 0.90 | dt: 1587.29ms | tok/sec: 10,321 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 7 | total time: 55.82m | eta: 78.2m
step 02090/05000 (41.80%) | loss: 4.109112 | lrm: 0.90 | dt: 1596.64ms | tok/sec: 10,261 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 7 | total time: 55.85m | eta: 78.1m
step 02091/05000 (41.82%) | loss: 4.115709 | lrm: 0.90 | dt: 1582.31ms | tok/sec: 10,354 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 7 | total time: 55.88m | eta: 78.1m
step 02092/05000 (41.84%) | loss: 4.101357 | lrm: 0.90 | dt: 1541.25ms | tok/sec: 10,630 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 7 | total time: 55.90m | eta: 78.1m
step 02093/05000 (41.86%) | loss: 4.107096 | lrm: 0.90 | dt: 1564.79ms | tok/sec: 10,470 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 7 | total time: 55.93m | eta: 78.1m
step 02094/05000 (41.88%) | loss: 4.098729 | lrm: 0.90 | dt: 1576.08ms | tok/sec: 10,395 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 7 | total time: 55.95m | eta: 78.0m
step 02095/05000 (41.90%) | loss: 4.092075 | lrm: 0.90 | dt: 1593.44ms | tok/sec: 10,282 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 7 | total time: 55.98m | eta: 78.0m
step 02096/05000 (41.92%) | loss: 4.078463 | lrm: 0.90 | dt: 1542.51ms | tok/sec: 10,621 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 7 | total time: 56.01m | eta: 78.0m
step 02097/05000 (41.94%) | loss: 4.095160 | lrm: 0.90 | dt: 1538.28ms | tok/sec: 10,650 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 7 | total time: 56.03m | eta: 77.9m
step 02098/05000 (41.96%) | loss: 4.099541 | lrm: 0.90 | dt: 1543.38ms | tok/sec: 10,615 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 7 | total time: 56.06m | eta: 77.9m
step 02099/05000 (41.98%) | loss: 4.100938 | lrm: 0.90 | dt: 1578.62ms | tok/sec: 10,378 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 7 | total time: 56.08m | eta: 77.9m
Step 02100 | Validation bpb: 1.272744
<|bos|>The capital of France is a city of city, and the city is a city of city, and the
<|bos|>The chemical symbol of gold is the symbol of the symbol of the symbol of the symbol of the symbol of the
<|bos|>If yesterday was Friday, then tomorrow will be the first time you have been to the world. The first time you have been
<|bos|>The opposite of hot is true. The hot water is a very good source of water. The hot water
<|bos|>The planets of the solar system are: the sun's rays are the sun's rays. The sun's rays are the
<|bos|>My favorite color is the red, orange, and orange. I have a few colors that I have
<|bos|>If 5*x + 3 = 13, then x is 3*x + 3*x + 3*x +
step 02100/05000 (42.00%) | loss: 4.112083 | lrm: 0.90 | dt: 2591.47ms | tok/sec: 6,322 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 7 | total time: 56.13m | eta: 77.9m
step 02101/05000 (42.02%) | loss: 4.116706 | lrm: 0.90 | dt: 1674.21ms | tok/sec: 9,786 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 7 | total time: 56.16m | eta: 77.9m
step 02102/05000 (42.04%) | loss: 4.111635 | lrm: 0.90 | dt: 1697.64ms | tok/sec: 9,651 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 7 | total time: 56.18m | eta: 77.8m
step 02103/05000 (42.06%) | loss: 4.111992 | lrm: 0.90 | dt: 1713.91ms | tok/sec: 9,559 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 7 | total time: 56.21m | eta: 77.8m
step 02104/05000 (42.08%) | loss: 4.112875 | lrm: 0.90 | dt: 1652.10ms | tok/sec: 9,917 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 7 | total time: 56.24m | eta: 77.8m
step 02105/05000 (42.10%) | loss: 4.099514 | lrm: 0.90 | dt: 1799.86ms | tok/sec: 9,102 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 7 | total time: 56.27m | eta: 77.8m
step 02106/05000 (42.12%) | loss: 4.110537 | lrm: 0.90 | dt: 1711.83ms | tok/sec: 9,571 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 7 | total time: 56.30m | eta: 77.7m
step 02107/05000 (42.14%) | loss: 4.108723 | lrm: 0.90 | dt: 1630.06ms | tok/sec: 10,051 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 8 | total time: 56.33m | eta: 77.7m
step 02108/05000 (42.16%) | loss: 4.093715 | lrm: 0.90 | dt: 1684.64ms | tok/sec: 9,725 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 8 | total time: 56.35m | eta: 77.7m
step 02109/05000 (42.18%) | loss: 4.099502 | lrm: 0.90 | dt: 1758.90ms | tok/sec: 9,314 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 8 | total time: 56.38m | eta: 77.7m
step 02110/05000 (42.20%) | loss: 4.090485 | lrm: 0.89 | dt: 1739.28ms | tok/sec: 9,420 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 8 | total time: 56.41m | eta: 77.6m
step 02111/05000 (42.22%) | loss: 4.092674 | lrm: 0.89 | dt: 1587.39ms | tok/sec: 10,321 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 8 | total time: 56.44m | eta: 77.6m
step 02112/05000 (42.24%) | loss: 4.100617 | lrm: 0.89 | dt: 1545.33ms | tok/sec: 10,602 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 8 | total time: 56.46m | eta: 77.6m
step 02113/05000 (42.26%) | loss: 4.077516 | lrm: 0.89 | dt: 1631.95ms | tok/sec: 10,039 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 8 | total time: 56.49m | eta: 77.6m
step 02114/05000 (42.28%) | loss: 4.070573 | lrm: 0.89 | dt: 1588.72ms | tok/sec: 10,312 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 8 | total time: 56.52m | eta: 77.5m
step 02115/05000 (42.30%) | loss: 4.080673 | lrm: 0.89 | dt: 1625.68ms | tok/sec: 10,078 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 8 | total time: 56.55m | eta: 77.5m
step 02116/05000 (42.32%) | loss: 4.069548 | lrm: 0.89 | dt: 1577.36ms | tok/sec: 10,386 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 8 | total time: 56.57m | eta: 77.5m
step 02117/05000 (42.34%) | loss: 4.072569 | lrm: 0.89 | dt: 1551.90ms | tok/sec: 10,557 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 8 | total time: 56.60m | eta: 77.4m
step 02118/05000 (42.36%) | loss: 4.076475 | lrm: 0.89 | dt: 1571.96ms | tok/sec: 10,422 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 8 | total time: 56.62m | eta: 77.4m
step 02119/05000 (42.38%) | loss: 4.090329 | lrm: 0.89 | dt: 1627.24ms | tok/sec: 10,068 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 8 | total time: 56.65m | eta: 77.4m
step 02120/05000 (42.40%) | loss: 4.074108 | lrm: 0.89 | dt: 1599.12ms | tok/sec: 10,245 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 8 | total time: 56.68m | eta: 77.4m
step 02121/05000 (42.42%) | loss: 4.069973 | lrm: 0.89 | dt: 1802.59ms | tok/sec: 9,089 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 8 | total time: 56.71m | eta: 77.3m
step 02122/05000 (42.44%) | loss: 4.069261 | lrm: 0.89 | dt: 1630.93ms | tok/sec: 10,045 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 8 | total time: 56.73m | eta: 77.3m
step 02123/05000 (42.46%) | loss: 4.071705 | lrm: 0.89 | dt: 1568.78ms | tok/sec: 10,443 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 8 | total time: 56.76m | eta: 77.3m
step 02124/05000 (42.48%) | loss: 4.059593 | lrm: 0.89 | dt: 1602.87ms | tok/sec: 10,221 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 8 | total time: 56.79m | eta: 77.3m
step 02125/05000 (42.50%) | loss: 4.055579 | lrm: 0.89 | dt: 1581.69ms | tok/sec: 10,358 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 8 | total time: 56.81m | eta: 77.2m
step 02126/05000 (42.52%) | loss: 4.068488 | lrm: 0.89 | dt: 1503.66ms | tok/sec: 10,896 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 8 | total time: 56.84m | eta: 77.2m
step 02127/05000 (42.54%) | loss: 4.077533 | lrm: 0.89 | dt: 1533.42ms | tok/sec: 10,684 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 8 | total time: 56.86m | eta: 77.2m
step 02128/05000 (42.56%) | loss: 4.086180 | lrm: 0.89 | dt: 1560.26ms | tok/sec: 10,500 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 8 | total time: 56.89m | eta: 77.1m
step 02129/05000 (42.58%) | loss: 4.070876 | lrm: 0.89 | dt: 1557.96ms | tok/sec: 10,516 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 8 | total time: 56.92m | eta: 77.1m
step 02130/05000 (42.60%) | loss: 4.086670 | lrm: 0.89 | dt: 1635.97ms | tok/sec: 10,014 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 8 | total time: 56.94m | eta: 77.1m
step 02131/05000 (42.62%) | loss: 4.079311 | lrm: 0.89 | dt: 1534.32ms | tok/sec: 10,678 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 9 | total time: 56.97m | eta: 77.1m
step 02132/05000 (42.64%) | loss: 4.086840 | lrm: 0.89 | dt: 1488.21ms | tok/sec: 11,009 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 9 | total time: 56.99m | eta: 77.0m
step 02133/05000 (42.66%) | loss: 4.084827 | lrm: 0.89 | dt: 1502.25ms | tok/sec: 10,906 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 9 | total time: 57.02m | eta: 77.0m
step 02134/05000 (42.68%) | loss: 4.099202 | lrm: 0.89 | dt: 1514.89ms | tok/sec: 10,815 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 9 | total time: 57.04m | eta: 77.0m
step 02135/05000 (42.70%) | loss: 4.087113 | lrm: 0.89 | dt: 1566.24ms | tok/sec: 10,460 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 9 | total time: 57.07m | eta: 76.9m
step 02136/05000 (42.72%) | loss: 4.082759 | lrm: 0.89 | dt: 1439.52ms | tok/sec: 11,381 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 9 | total time: 57.09m | eta: 76.9m
step 02137/05000 (42.74%) | loss: 4.081792 | lrm: 0.89 | dt: 1607.32ms | tok/sec: 10,193 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 9 | total time: 57.12m | eta: 76.9m
step 02138/05000 (42.76%) | loss: 4.099369 | lrm: 0.89 | dt: 1754.99ms | tok/sec: 9,335 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 9 | total time: 57.15m | eta: 76.9m
step 02139/05000 (42.78%) | loss: 4.108021 | lrm: 0.89 | dt: 1529.33ms | tok/sec: 10,713 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 9 | total time: 57.18m | eta: 76.8m
step 02140/05000 (42.80%) | loss: 4.105313 | lrm: 0.89 | dt: 1446.96ms | tok/sec: 11,323 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 9 | total time: 57.20m | eta: 76.8m
step 02141/05000 (42.82%) | loss: 4.132737 | lrm: 0.89 | dt: 1453.46ms | tok/sec: 11,272 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 9 | total time: 57.22m | eta: 76.8m
step 02142/05000 (42.84%) | loss: 4.137407 | lrm: 0.89 | dt: 1644.98ms | tok/sec: 9,960 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 9 | total time: 57.25m | eta: 76.7m
step 02143/05000 (42.86%) | loss: 4.138196 | lrm: 0.89 | dt: 1451.97ms | tok/sec: 11,283 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 9 | total time: 57.28m | eta: 76.7m
step 02144/05000 (42.88%) | loss: 4.138530 | lrm: 0.88 | dt: 1604.38ms | tok/sec: 10,212 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 9 | total time: 57.30m | eta: 76.7m
step 02145/05000 (42.90%) | loss: 4.133797 | lrm: 0.88 | dt: 1480.76ms | tok/sec: 11,064 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 9 | total time: 57.33m | eta: 76.7m
step 02146/05000 (42.92%) | loss: 4.115086 | lrm: 0.88 | dt: 1524.76ms | tok/sec: 10,745 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 9 | total time: 57.35m | eta: 76.6m
step 02147/05000 (42.94%) | loss: 4.124147 | lrm: 0.88 | dt: 1441.62ms | tok/sec: 11,364 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 9 | total time: 57.38m | eta: 76.6m
step 02148/05000 (42.96%) | loss: 4.128772 | lrm: 0.88 | dt: 1449.25ms | tok/sec: 11,305 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 9 | total time: 57.40m | eta: 76.6m
step 02149/05000 (42.98%) | loss: 4.115266 | lrm: 0.88 | dt: 1595.88ms | tok/sec: 10,266 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 9 | total time: 57.43m | eta: 76.5m
step 02150/05000 (43.00%) | loss: 4.113409 | lrm: 0.88 | dt: 1528.83ms | tok/sec: 10,716 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 9 | total time: 57.45m | eta: 76.5m
step 02151/05000 (43.02%) | loss: 4.111347 | lrm: 0.88 | dt: 1472.90ms | tok/sec: 11,123 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 9 | total time: 57.48m | eta: 76.5m
step 02152/05000 (43.04%) | loss: 4.117324 | lrm: 0.88 | dt: 1527.25ms | tok/sec: 10,727 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 9 | total time: 57.50m | eta: 76.5m
step 02153/05000 (43.06%) | loss: 4.119584 | lrm: 0.88 | dt: 1490.94ms | tok/sec: 10,989 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 9 | total time: 57.53m | eta: 76.4m
step 02154/05000 (43.08%) | loss: 4.112967 | lrm: 0.88 | dt: 1534.78ms | tok/sec: 10,675 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 10 | total time: 57.55m | eta: 76.4m
step 02155/05000 (43.10%) | loss: 4.116324 | lrm: 0.88 | dt: 1483.91ms | tok/sec: 11,041 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 10 | total time: 57.58m | eta: 76.4m
step 02156/05000 (43.12%) | loss: 4.104314 | lrm: 0.88 | dt: 1565.16ms | tok/sec: 10,467 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 10 | total time: 57.60m | eta: 76.3m
step 02157/05000 (43.14%) | loss: 4.103459 | lrm: 0.88 | dt: 1462.46ms | tok/sec: 11,203 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 10 | total time: 57.63m | eta: 76.3m
step 02158/05000 (43.16%) | loss: 4.109575 | lrm: 0.88 | dt: 1549.44ms | tok/sec: 10,574 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 10 | total time: 57.65m | eta: 76.3m
step 02159/05000 (43.18%) | loss: 4.100800 | lrm: 0.88 | dt: 1429.90ms | tok/sec: 11,458 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 10 | total time: 57.68m | eta: 76.3m
step 02160/05000 (43.20%) | loss: 4.094998 | lrm: 0.88 | dt: 1601.15ms | tok/sec: 10,232 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 10 | total time: 57.71m | eta: 76.2m
step 02161/05000 (43.22%) | loss: 4.095628 | lrm: 0.88 | dt: 1570.62ms | tok/sec: 10,431 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 10 | total time: 57.73m | eta: 76.2m
step 02162/05000 (43.24%) | loss: 4.105815 | lrm: 0.88 | dt: 1551.72ms | tok/sec: 10,558 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 10 | total time: 57.76m | eta: 76.2m
step 02163/05000 (43.26%) | loss: 4.118174 | lrm: 0.88 | dt: 1541.64ms | tok/sec: 10,627 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 10 | total time: 57.78m | eta: 76.1m
step 02164/05000 (43.28%) | loss: 4.131028 | lrm: 0.88 | dt: 1578.60ms | tok/sec: 10,378 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 10 | total time: 57.81m | eta: 76.1m
step 02165/05000 (43.30%) | loss: 4.123195 | lrm: 0.88 | dt: 1496.81ms | tok/sec: 10,945 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 10 | total time: 57.83m | eta: 76.1m
step 02166/05000 (43.32%) | loss: 4.123263 | lrm: 0.88 | dt: 1580.54ms | tok/sec: 10,366 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 10 | total time: 57.86m | eta: 76.1m
step 02167/05000 (43.34%) | loss: 4.123644 | lrm: 0.88 | dt: 1583.01ms | tok/sec: 10,349 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 10 | total time: 57.89m | eta: 76.0m
step 02168/05000 (43.36%) | loss: 4.126863 | lrm: 0.88 | dt: 1609.99ms | tok/sec: 10,176 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 10 | total time: 57.91m | eta: 76.0m
step 02169/05000 (43.38%) | loss: 4.124897 | lrm: 0.88 | dt: 1516.56ms | tok/sec: 10,803 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 10 | total time: 57.94m | eta: 76.0m
step 02170/05000 (43.40%) | loss: 4.124979 | lrm: 0.88 | dt: 1599.49ms | tok/sec: 10,243 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 10 | total time: 57.97m | eta: 75.9m
step 02171/05000 (43.42%) | loss: 4.100174 | lrm: 0.88 | dt: 1581.29ms | tok/sec: 10,361 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 10 | total time: 57.99m | eta: 75.9m
step 02172/05000 (43.44%) | loss: 4.102750 | lrm: 0.88 | dt: 1588.58ms | tok/sec: 10,313 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 10 | total time: 58.02m | eta: 75.9m
step 02173/05000 (43.46%) | loss: 4.113692 | lrm: 0.88 | dt: 1662.44ms | tok/sec: 9,855 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 10 | total time: 58.05m | eta: 75.9m
step 02174/05000 (43.48%) | loss: 4.092159 | lrm: 0.88 | dt: 1574.72ms | tok/sec: 10,404 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 10 | total time: 58.07m | eta: 75.8m
step 02175/05000 (43.50%) | loss: 4.089404 | lrm: 0.88 | dt: 1639.89ms | tok/sec: 9,990 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 10 | total time: 58.10m | eta: 75.8m
step 02176/05000 (43.52%) | loss: 4.099407 | lrm: 0.88 | dt: 1599.10ms | tok/sec: 10,245 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 10 | total time: 58.13m | eta: 75.8m
step 02177/05000 (43.54%) | loss: 4.099474 | lrm: 0.88 | dt: 1483.16ms | tok/sec: 11,046 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 11 | total time: 58.15m | eta: 75.8m
step 02178/05000 (43.56%) | loss: 4.094699 | lrm: 0.87 | dt: 1600.79ms | tok/sec: 10,234 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 11 | total time: 58.18m | eta: 75.7m
step 02179/05000 (43.58%) | loss: 4.090335 | lrm: 0.87 | dt: 1619.01ms | tok/sec: 10,119 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 11 | total time: 58.20m | eta: 75.7m
step 02180/05000 (43.60%) | loss: 4.087773 | lrm: 0.87 | dt: 1586.63ms | tok/sec: 10,326 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 11 | total time: 58.23m | eta: 75.7m
step 02181/05000 (43.62%) | loss: 4.075822 | lrm: 0.87 | dt: 1666.01ms | tok/sec: 9,834 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 11 | total time: 58.26m | eta: 75.6m
step 02182/05000 (43.64%) | loss: 4.080842 | lrm: 0.87 | dt: 1739.94ms | tok/sec: 9,416 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 11 | total time: 58.29m | eta: 75.6m
step 02183/05000 (43.66%) | loss: 4.094842 | lrm: 0.87 | dt: 1466.03ms | tok/sec: 11,175 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 11 | total time: 58.31m | eta: 75.6m
step 02184/05000 (43.68%) | loss: 4.101000 | lrm: 0.87 | dt: 1552.78ms | tok/sec: 10,551 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 11 | total time: 58.34m | eta: 75.6m
step 02185/05000 (43.70%) | loss: 4.113300 | lrm: 0.87 | dt: 1455.14ms | tok/sec: 11,259 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 11 | total time: 58.36m | eta: 75.5m
step 02186/05000 (43.72%) | loss: 4.095524 | lrm: 0.87 | dt: 1542.45ms | tok/sec: 10,622 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 11 | total time: 58.39m | eta: 75.5m
step 02187/05000 (43.74%) | loss: 4.091062 | lrm: 0.87 | dt: 1508.39ms | tok/sec: 10,861 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 11 | total time: 58.41m | eta: 75.5m
step 02188/05000 (43.76%) | loss: 4.083969 | lrm: 0.87 | dt: 1464.05ms | tok/sec: 11,190 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 11 | total time: 58.44m | eta: 75.4m
step 02189/05000 (43.78%) | loss: 4.064650 | lrm: 0.87 | dt: 1540.12ms | tok/sec: 10,638 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 11 | total time: 58.46m | eta: 75.4m
step 02190/05000 (43.80%) | loss: 4.075362 | lrm: 0.87 | dt: 1496.08ms | tok/sec: 10,951 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 11 | total time: 58.49m | eta: 75.4m
step 02191/05000 (43.82%) | loss: 4.074844 | lrm: 0.87 | dt: 1622.93ms | tok/sec: 10,095 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 11 | total time: 58.52m | eta: 75.4m
step 02192/05000 (43.84%) | loss: 4.073761 | lrm: 0.87 | dt: 1451.20ms | tok/sec: 11,289 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 11 | total time: 58.54m | eta: 75.3m
step 02193/05000 (43.86%) | loss: 4.071952 | lrm: 0.87 | dt: 1562.20ms | tok/sec: 10,487 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 11 | total time: 58.57m | eta: 75.3m
step 02194/05000 (43.88%) | loss: 4.059184 | lrm: 0.87 | dt: 1580.42ms | tok/sec: 10,366 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 11 | total time: 58.59m | eta: 75.3m
step 02195/05000 (43.90%) | loss: 4.077272 | lrm: 0.87 | dt: 1605.98ms | tok/sec: 10,201 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 11 | total time: 58.62m | eta: 75.3m
step 02196/05000 (43.92%) | loss: 4.067897 | lrm: 0.87 | dt: 1499.01ms | tok/sec: 10,929 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 11 | total time: 58.64m | eta: 75.2m
step 02197/05000 (43.94%) | loss: 4.070889 | lrm: 0.87 | dt: 1561.76ms | tok/sec: 10,490 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 11 | total time: 58.67m | eta: 75.2m
step 02198/05000 (43.96%) | loss: 4.082832 | lrm: 0.87 | dt: 1512.93ms | tok/sec: 10,829 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 11 | total time: 58.70m | eta: 75.2m
step 02199/05000 (43.98%) | loss: 4.092110 | lrm: 0.87 | dt: 1696.39ms | tok/sec: 9,658 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 11 | total time: 58.72m | eta: 75.1m
Step 02200 | Validation bpb: 1.266337
<|bos|>The capital of France is the largest exporter of the world's largest exporter. The capital of France
<|bos|>The chemical symbol of gold is the symbol of the gold, which is the symbol of the gold. The symbol
<|bos|>If yesterday was Friday, then tomorrow will be the first time to get back to the next year. The first time, we
<|bos|>The opposite of hot is true. The reason for this is that the hot is true. The reason for
<|bos|>The planets of the solar system are: the sun's rays, the sun's rays, the sun's rays, the
<|bos|>My favorite color is the red, yellow, and red. I'm not sure if you want to
<|bos|>If 5*x + 3 = 13, then x is 2.0 2.0 2.0 2.0
step 02200/05000 (44.00%) | loss: 4.073012 | lrm: 0.87 | dt: 2106.16ms | tok/sec: 7,779 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 11 | total time: 58.76m | eta: 75.1m
step 02201/05000 (44.02%) | loss: 4.067944 | lrm: 0.87 | dt: 1664.39ms | tok/sec: 9,843 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 12 | total time: 58.79m | eta: 75.1m
step 02202/05000 (44.04%) | loss: 4.079659 | lrm: 0.87 | dt: 1650.46ms | tok/sec: 9,926 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 12 | total time: 58.81m | eta: 75.1m
step 02203/05000 (44.06%) | loss: 4.077263 | lrm: 0.87 | dt: 1623.59ms | tok/sec: 10,091 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 12 | total time: 58.84m | eta: 75.0m
step 02204/05000 (44.08%) | loss: 4.082731 | lrm: 0.87 | dt: 1614.40ms | tok/sec: 10,148 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 12 | total time: 58.87m | eta: 75.0m
step 02205/05000 (44.10%) | loss: 4.093665 | lrm: 0.87 | dt: 1558.62ms | tok/sec: 10,511 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 12 | total time: 58.89m | eta: 75.0m
step 02206/05000 (44.12%) | loss: 4.092559 | lrm: 0.87 | dt: 1663.30ms | tok/sec: 9,850 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 12 | total time: 58.92m | eta: 75.0m
step 02207/05000 (44.14%) | loss: 4.093624 | lrm: 0.87 | dt: 1646.43ms | tok/sec: 9,951 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 12 | total time: 58.95m | eta: 74.9m
step 02208/05000 (44.16%) | loss: 4.099753 | lrm: 0.87 | dt: 1662.20ms | tok/sec: 9,856 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 12 | total time: 58.98m | eta: 74.9m
step 02209/05000 (44.18%) | loss: 4.114815 | lrm: 0.87 | dt: 1649.47ms | tok/sec: 9,932 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 12 | total time: 59.00m | eta: 74.9m
step 02210/05000 (44.20%) | loss: 4.103630 | lrm: 0.87 | dt: 1649.09ms | tok/sec: 9,935 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 12 | total time: 59.03m | eta: 74.9m
step 02211/05000 (44.22%) | loss: 4.105951 | lrm: 0.87 | dt: 1616.79ms | tok/sec: 10,133 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 12 | total time: 59.06m | eta: 74.8m
step 02212/05000 (44.24%) | loss: 4.096750 | lrm: 0.86 | dt: 1614.96ms | tok/sec: 10,145 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 12 | total time: 59.09m | eta: 74.8m
step 02213/05000 (44.26%) | loss: 4.085396 | lrm: 0.86 | dt: 1650.80ms | tok/sec: 9,924 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 12 | total time: 59.11m | eta: 74.8m
step 02214/05000 (44.28%) | loss: 4.079860 | lrm: 0.86 | dt: 1598.22ms | tok/sec: 10,251 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 12 | total time: 59.14m | eta: 74.8m
step 02215/05000 (44.30%) | loss: 4.069358 | lrm: 0.86 | dt: 1669.58ms | tok/sec: 9,813 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 12 | total time: 59.17m | eta: 74.7m
step 02216/05000 (44.32%) | loss: 4.071224 | lrm: 0.86 | dt: 1655.63ms | tok/sec: 9,895 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 12 | total time: 59.19m | eta: 74.7m
step 02217/05000 (44.34%) | loss: 4.075989 | lrm: 0.86 | dt: 1668.22ms | tok/sec: 9,821 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 12 | total time: 59.22m | eta: 74.7m
step 02218/05000 (44.36%) | loss: 4.076894 | lrm: 0.86 | dt: 1610.90ms | tok/sec: 10,170 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 12 | total time: 59.25m | eta: 74.7m
step 02219/05000 (44.38%) | loss: 4.074179 | lrm: 0.86 | dt: 1486.71ms | tok/sec: 11,020 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 12 | total time: 59.27m | eta: 74.6m
step 02220/05000 (44.40%) | loss: 4.073093 | lrm: 0.86 | dt: 1657.21ms | tok/sec: 9,886 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 12 | total time: 59.30m | eta: 74.6m
step 02221/05000 (44.42%) | loss: 4.072910 | lrm: 0.86 | dt: 1725.98ms | tok/sec: 9,492 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 12 | total time: 59.33m | eta: 74.6m
step 02222/05000 (44.44%) | loss: 4.074780 | lrm: 0.86 | dt: 1694.11ms | tok/sec: 9,671 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 12 | total time: 59.36m | eta: 74.5m
step 02223/05000 (44.46%) | loss: 4.079961 | lrm: 0.86 | dt: 1567.78ms | tok/sec: 10,450 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 12 | total time: 59.39m | eta: 74.5m
step 02224/05000 (44.48%) | loss: 4.075411 | lrm: 0.86 | dt: 1559.83ms | tok/sec: 10,503 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 13 | total time: 59.41m | eta: 74.5m
step 02225/05000 (44.50%) | loss: 4.061158 | lrm: 0.86 | dt: 1535.49ms | tok/sec: 10,670 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 13 | total time: 59.44m | eta: 74.5m
step 02226/05000 (44.52%) | loss: 4.062471 | lrm: 0.86 | dt: 1605.07ms | tok/sec: 10,207 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 13 | total time: 59.46m | eta: 74.4m
step 02227/05000 (44.54%) | loss: 4.067272 | lrm: 0.86 | dt: 1500.68ms | tok/sec: 10,917 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 13 | total time: 59.49m | eta: 74.4m
step 02228/05000 (44.56%) | loss: 4.067105 | lrm: 0.86 | dt: 1640.79ms | tok/sec: 9,985 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 13 | total time: 59.52m | eta: 74.4m
step 02229/05000 (44.58%) | loss: 4.068230 | lrm: 0.86 | dt: 1604.30ms | tok/sec: 10,212 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 13 | total time: 59.54m | eta: 74.4m
step 02230/05000 (44.60%) | loss: 4.076397 | lrm: 0.86 | dt: 1597.11ms | tok/sec: 10,258 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 13 | total time: 59.57m | eta: 74.3m
step 02231/05000 (44.62%) | loss: 4.083254 | lrm: 0.86 | dt: 1594.02ms | tok/sec: 10,278 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 13 | total time: 59.60m | eta: 74.3m
step 02232/05000 (44.64%) | loss: 4.085032 | lrm: 0.86 | dt: 1625.68ms | tok/sec: 10,078 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 13 | total time: 59.62m | eta: 74.3m
step 02233/05000 (44.66%) | loss: 4.079090 | lrm: 0.86 | dt: 1604.25ms | tok/sec: 10,212 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 13 | total time: 59.65m | eta: 74.2m
step 02234/05000 (44.68%) | loss: 4.076740 | lrm: 0.86 | dt: 1574.77ms | tok/sec: 10,404 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 13 | total time: 59.68m | eta: 74.2m
step 02235/05000 (44.70%) | loss: 4.072216 | lrm: 0.86 | dt: 1571.80ms | tok/sec: 10,423 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 13 | total time: 59.70m | eta: 74.2m
step 02236/05000 (44.72%) | loss: 4.073539 | lrm: 0.86 | dt: 1550.00ms | tok/sec: 10,570 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 13 | total time: 59.73m | eta: 74.2m
step 02237/05000 (44.74%) | loss: 4.067842 | lrm: 0.86 | dt: 1628.12ms | tok/sec: 10,063 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 13 | total time: 59.75m | eta: 74.1m
step 02238/05000 (44.76%) | loss: 4.061323 | lrm: 0.86 | dt: 1636.42ms | tok/sec: 10,012 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 13 | total time: 59.78m | eta: 74.1m
step 02239/05000 (44.78%) | loss: 4.077189 | lrm: 0.86 | dt: 1560.88ms | tok/sec: 10,496 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 13 | total time: 59.81m | eta: 74.1m
step 02240/05000 (44.80%) | loss: 4.078338 | lrm: 0.86 | dt: 1600.91ms | tok/sec: 10,234 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 13 | total time: 59.83m | eta: 74.1m
step 02241/05000 (44.82%) | loss: 4.078184 | lrm: 0.86 | dt: 1523.23ms | tok/sec: 10,756 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 13 | total time: 59.86m | eta: 74.0m
step 02242/05000 (44.84%) | loss: 4.093686 | lrm: 0.86 | dt: 1628.62ms | tok/sec: 10,060 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 13 | total time: 59.89m | eta: 74.0m
step 02243/05000 (44.86%) | loss: 4.084492 | lrm: 0.86 | dt: 1480.10ms | tok/sec: 11,069 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 13 | total time: 59.91m | eta: 74.0m
step 02244/05000 (44.88%) | loss: 4.080675 | lrm: 0.86 | dt: 1667.36ms | tok/sec: 9,826 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 13 | total time: 59.94m | eta: 73.9m
step 02245/05000 (44.90%) | loss: 4.069847 | lrm: 0.86 | dt: 1583.70ms | tok/sec: 10,345 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 13 | total time: 59.97m | eta: 73.9m
step 02246/05000 (44.92%) | loss: 4.062148 | lrm: 0.86 | dt: 1555.40ms | tok/sec: 10,533 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 13 | total time: 59.99m | eta: 73.9m
step 02247/05000 (44.94%) | loss: 4.071589 | lrm: 0.85 | dt: 1527.04ms | tok/sec: 10,729 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 13 | total time: 60.02m | eta: 73.9m
step 02248/05000 (44.96%) | loss: 4.084127 | lrm: 0.85 | dt: 1483.93ms | tok/sec: 11,040 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 14 | total time: 60.04m | eta: 73.8m
step 02249/05000 (44.98%) | loss: 4.084486 | lrm: 0.85 | dt: 1454.67ms | tok/sec: 11,263 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 14 | total time: 60.07m | eta: 73.8m
step 02250/05000 (45.00%) | loss: 4.067736 | lrm: 0.85 | dt: 1560.96ms | tok/sec: 10,496 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 14 | total time: 60.09m | eta: 73.8m
step 02251/05000 (45.02%) | loss: 4.061442 | lrm: 0.85 | dt: 1499.44ms | tok/sec: 10,926 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 14 | total time: 60.12m | eta: 73.7m
step 02252/05000 (45.04%) | loss: 4.051628 | lrm: 0.85 | dt: 1468.48ms | tok/sec: 11,157 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 14 | total time: 60.14m | eta: 73.7m
step 02253/05000 (45.06%) | loss: 4.052078 | lrm: 0.85 | dt: 1680.43ms | tok/sec: 9,749 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 14 | total time: 60.17m | eta: 73.7m
step 02254/05000 (45.08%) | loss: 4.052686 | lrm: 0.85 | dt: 1497.11ms | tok/sec: 10,943 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 14 | total time: 60.20m | eta: 73.7m
step 02255/05000 (45.10%) | loss: 4.044926 | lrm: 0.85 | dt: 1552.04ms | tok/sec: 10,556 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 14 | total time: 60.22m | eta: 73.6m
step 02256/05000 (45.12%) | loss: 4.063414 | lrm: 0.85 | dt: 1650.43ms | tok/sec: 9,927 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 14 | total time: 60.25m | eta: 73.6m
step 02257/05000 (45.14%) | loss: 4.084729 | lrm: 0.85 | dt: 1490.49ms | tok/sec: 10,992 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 14 | total time: 60.27m | eta: 73.6m
step 02258/05000 (45.16%) | loss: 4.088121 | lrm: 0.85 | dt: 1477.37ms | tok/sec: 11,089 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 14 | total time: 60.30m | eta: 73.5m
step 02259/05000 (45.18%) | loss: 4.086456 | lrm: 0.85 | dt: 1577.63ms | tok/sec: 10,385 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 14 | total time: 60.32m | eta: 73.5m
step 02260/05000 (45.20%) | loss: 4.084504 | lrm: 0.85 | dt: 1561.18ms | tok/sec: 10,494 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 14 | total time: 60.35m | eta: 73.5m
step 02261/05000 (45.22%) | loss: 4.084294 | lrm: 0.85 | dt: 1499.52ms | tok/sec: 10,926 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 14 | total time: 60.38m | eta: 73.5m
step 02262/05000 (45.24%) | loss: 4.089657 | lrm: 0.85 | dt: 1499.22ms | tok/sec: 10,928 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 14 | total time: 60.40m | eta: 73.4m
step 02263/05000 (45.26%) | loss: 4.103429 | lrm: 0.85 | dt: 1471.94ms | tok/sec: 11,130 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 14 | total time: 60.42m | eta: 73.4m
step 02264/05000 (45.28%) | loss: 4.100706 | lrm: 0.85 | dt: 1495.94ms | tok/sec: 10,952 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 14 | total time: 60.45m | eta: 73.4m
step 02265/05000 (45.30%) | loss: 4.096027 | lrm: 0.85 | dt: 1650.95ms | tok/sec: 9,923 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 14 | total time: 60.48m | eta: 73.4m
step 02266/05000 (45.32%) | loss: 4.093310 | lrm: 0.85 | dt: 1658.20ms | tok/sec: 9,880 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 14 | total time: 60.50m | eta: 73.3m
step 02267/05000 (45.34%) | loss: 4.096296 | lrm: 0.85 | dt: 1686.55ms | tok/sec: 9,714 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 14 | total time: 60.53m | eta: 73.3m
step 02268/05000 (45.36%) | loss: 4.110217 | lrm: 0.85 | dt: 1514.61ms | tok/sec: 10,817 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 14 | total time: 60.56m | eta: 73.3m
step 02269/05000 (45.38%) | loss: 4.118674 | lrm: 0.85 | dt: 1605.95ms | tok/sec: 10,202 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 14 | total time: 60.58m | eta: 73.2m
step 02270/05000 (45.40%) | loss: 4.109771 | lrm: 0.85 | dt: 1586.94ms | tok/sec: 10,324 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 14 | total time: 60.61m | eta: 73.2m
step 02271/05000 (45.42%) | loss: 4.102879 | lrm: 0.85 | dt: 1635.36ms | tok/sec: 10,018 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 15 | total time: 60.64m | eta: 73.2m
step 02272/05000 (45.44%) | loss: 4.107375 | lrm: 0.85 | dt: 1500.06ms | tok/sec: 10,922 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 15 | total time: 60.66m | eta: 73.2m
step 02273/05000 (45.46%) | loss: 4.095550 | lrm: 0.85 | dt: 1489.99ms | tok/sec: 10,996 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 15 | total time: 60.69m | eta: 73.1m
step 02274/05000 (45.48%) | loss: 4.082145 | lrm: 0.85 | dt: 1712.43ms | tok/sec: 9,567 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 15 | total time: 60.72m | eta: 73.1m
step 02275/05000 (45.50%) | loss: 4.080257 | lrm: 0.85 | dt: 1715.74ms | tok/sec: 9,549 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 15 | total time: 60.75m | eta: 73.1m
step 02276/05000 (45.52%) | loss: 4.083164 | lrm: 0.85 | dt: 1664.96ms | tok/sec: 9,840 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 15 | total time: 60.77m | eta: 73.1m
step 02277/05000 (45.54%) | loss: 4.088083 | lrm: 0.85 | dt: 1614.36ms | tok/sec: 10,148 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 15 | total time: 60.80m | eta: 73.0m
step 02278/05000 (45.56%) | loss: 4.107171 | lrm: 0.85 | dt: 1637.33ms | tok/sec: 10,006 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 15 | total time: 60.83m | eta: 73.0m
step 02279/05000 (45.58%) | loss: 4.109337 | lrm: 0.85 | dt: 1640.10ms | tok/sec: 9,989 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 15 | total time: 60.85m | eta: 73.0m
step 02280/05000 (45.60%) | loss: 4.109128 | lrm: 0.85 | dt: 1776.24ms | tok/sec: 9,223 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 15 | total time: 60.88m | eta: 73.0m
step 02281/05000 (45.62%) | loss: 4.112002 | lrm: 0.84 | dt: 1589.68ms | tok/sec: 10,306 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 15 | total time: 60.91m | eta: 72.9m
step 02282/05000 (45.64%) | loss: 4.103046 | lrm: 0.84 | dt: 1511.73ms | tok/sec: 10,837 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 15 | total time: 60.94m | eta: 72.9m
step 02283/05000 (45.66%) | loss: 4.098282 | lrm: 0.84 | dt: 1526.52ms | tok/sec: 10,732 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 15 | total time: 60.96m | eta: 72.9m
step 02284/05000 (45.68%) | loss: 4.087606 | lrm: 0.84 | dt: 1645.65ms | tok/sec: 9,955 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 15 | total time: 60.99m | eta: 72.8m
step 02285/05000 (45.70%) | loss: 4.087023 | lrm: 0.84 | dt: 1499.25ms | tok/sec: 10,928 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 15 | total time: 61.01m | eta: 72.8m
step 02286/05000 (45.72%) | loss: 4.081382 | lrm: 0.84 | dt: 1585.87ms | tok/sec: 10,331 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 15 | total time: 61.04m | eta: 72.8m
step 02287/05000 (45.74%) | loss: 4.083440 | lrm: 0.84 | dt: 1568.89ms | tok/sec: 10,443 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 15 | total time: 61.07m | eta: 72.8m
step 02288/05000 (45.76%) | loss: 4.092569 | lrm: 0.84 | dt: 1460.88ms | tok/sec: 11,215 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 15 | total time: 61.09m | eta: 72.7m
step 02289/05000 (45.78%) | loss: 4.084880 | lrm: 0.84 | dt: 1455.08ms | tok/sec: 11,259 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 15 | total time: 61.12m | eta: 72.7m
step 02290/05000 (45.80%) | loss: 4.071826 | lrm: 0.84 | dt: 1457.34ms | tok/sec: 11,242 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 15 | total time: 61.14m | eta: 72.7m
step 02291/05000 (45.82%) | loss: 4.075546 | lrm: 0.84 | dt: 1478.12ms | tok/sec: 11,084 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 15 | total time: 61.16m | eta: 72.6m
step 02292/05000 (45.84%) | loss: 4.078444 | lrm: 0.84 | dt: 1575.99ms | tok/sec: 10,395 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 15 | total time: 61.19m | eta: 72.6m
step 02293/05000 (45.86%) | loss: 4.079175 | lrm: 0.84 | dt: 1459.39ms | tok/sec: 11,226 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 15 | total time: 61.21m | eta: 72.6m
step 02294/05000 (45.88%) | loss: 4.091492 | lrm: 0.84 | dt: 1508.01ms | tok/sec: 10,864 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 16 | total time: 61.24m | eta: 72.6m
step 02295/05000 (45.90%) | loss: 4.093827 | lrm: 0.84 | dt: 1461.99ms | tok/sec: 11,206 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 16 | total time: 61.26m | eta: 72.5m
step 02296/05000 (45.92%) | loss: 4.086887 | lrm: 0.84 | dt: 1527.43ms | tok/sec: 10,726 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 16 | total time: 61.29m | eta: 72.5m
step 02297/05000 (45.94%) | loss: 4.060515 | lrm: 0.84 | dt: 1514.23ms | tok/sec: 10,820 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 16 | total time: 61.31m | eta: 72.5m
step 02298/05000 (45.96%) | loss: 4.067883 | lrm: 0.84 | dt: 1494.06ms | tok/sec: 10,966 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 16 | total time: 61.34m | eta: 72.4m
step 02299/05000 (45.98%) | loss: 4.070992 | lrm: 0.84 | dt: 1504.19ms | tok/sec: 10,892 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 16 | total time: 61.36m | eta: 72.4m
Step 02300 | Validation bpb: 1.259622
<|bos|>The capital of France is the capital of France, the capital of France, the capital of France, the
<|bos|>The chemical symbol of gold is the gold, and gold is the gold, and gold is the gold, and
<|bos|>If yesterday was Friday, then tomorrow will be the first time to get a new one. The first time, we have a
<|bos|>The opposite of hot is true. The hot is true. The hot is true. The hot is true
<|bos|>The planets of the solar system are: the sun, the sun, the sun, the sun, the sun, the
<|bos|>My favorite color is the red, red, and red. I have a lot of color and I
<|bos|>If 5*x + 3 = 13, then x is 13*x + 3*x + 3*x +
step 02300/05000 (46.00%) | loss: 4.077547 | lrm: 0.84 | dt: 1543.68ms | tok/sec: 10,613 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 16 | total time: 61.39m | eta: 72.4m
step 02301/05000 (46.02%) | loss: 4.084598 | lrm: 0.84 | dt: 1523.02ms | tok/sec: 10,757 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 16 | total time: 61.42m | eta: 72.4m
step 02302/05000 (46.04%) | loss: 4.082928 | lrm: 0.84 | dt: 1541.22ms | tok/sec: 10,630 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 16 | total time: 61.44m | eta: 72.3m
step 02303/05000 (46.06%) | loss: 4.085042 | lrm: 0.84 | dt: 1500.77ms | tok/sec: 10,917 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 16 | total time: 61.47m | eta: 72.3m
step 02304/05000 (46.08%) | loss: 4.095913 | lrm: 0.84 | dt: 1515.72ms | tok/sec: 10,809 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 16 | total time: 61.49m | eta: 72.3m
step 02305/05000 (46.10%) | loss: 4.107622 | lrm: 0.84 | dt: 1476.93ms | tok/sec: 11,093 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 16 | total time: 61.52m | eta: 72.2m
step 02306/05000 (46.12%) | loss: 4.110967 | lrm: 0.84 | dt: 1448.91ms | tok/sec: 11,307 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 16 | total time: 61.54m | eta: 72.2m
step 02307/05000 (46.14%) | loss: 4.110180 | lrm: 0.84 | dt: 1571.53ms | tok/sec: 10,425 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 16 | total time: 61.57m | eta: 72.2m
step 02308/05000 (46.16%) | loss: 4.094039 | lrm: 0.84 | dt: 1820.09ms | tok/sec: 9,001 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 16 | total time: 61.60m | eta: 72.2m
step 02309/05000 (46.18%) | loss: 4.093788 | lrm: 0.84 | dt: 1611.78ms | tok/sec: 10,165 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 16 | total time: 61.62m | eta: 72.1m
step 02310/05000 (46.20%) | loss: 4.095088 | lrm: 0.84 | dt: 1568.17ms | tok/sec: 10,447 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 16 | total time: 61.65m | eta: 72.1m
step 02311/05000 (46.22%) | loss: 4.101552 | lrm: 0.84 | dt: 1555.42ms | tok/sec: 10,533 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 16 | total time: 61.68m | eta: 72.1m
step 02312/05000 (46.24%) | loss: 4.087099 | lrm: 0.84 | dt: 1605.98ms | tok/sec: 10,201 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 16 | total time: 61.70m | eta: 72.0m
step 02313/05000 (46.26%) | loss: 4.085729 | lrm: 0.84 | dt: 1622.99ms | tok/sec: 10,094 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 16 | total time: 61.73m | eta: 72.0m
step 02314/05000 (46.28%) | loss: 4.078294 | lrm: 0.84 | dt: 1620.24ms | tok/sec: 10,112 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 16 | total time: 61.76m | eta: 72.0m
step 02315/05000 (46.30%) | loss: 4.076366 | lrm: 0.83 | dt: 1601.47ms | tok/sec: 10,230 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 16 | total time: 61.78m | eta: 72.0m
step 02316/05000 (46.32%) | loss: 4.080160 | lrm: 0.83 | dt: 1626.87ms | tok/sec: 10,070 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 16 | total time: 61.81m | eta: 71.9m
step 02317/05000 (46.34%) | loss: 4.072331 | lrm: 0.83 | dt: 1634.34ms | tok/sec: 10,024 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 17 | total time: 61.84m | eta: 71.9m
step 02318/05000 (46.36%) | loss: 4.084018 | lrm: 0.83 | dt: 1718.41ms | tok/sec: 9,534 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 17 | total time: 61.87m | eta: 71.9m
step 02319/05000 (46.38%) | loss: 4.082000 | lrm: 0.83 | dt: 1665.13ms | tok/sec: 9,839 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 17 | total time: 61.89m | eta: 71.9m
step 02320/05000 (46.40%) | loss: 4.064637 | lrm: 0.83 | dt: 1535.74ms | tok/sec: 10,668 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 17 | total time: 61.92m | eta: 71.8m
step 02321/05000 (46.42%) | loss: 4.067611 | lrm: 0.83 | dt: 1479.18ms | tok/sec: 11,076 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 17 | total time: 61.94m | eta: 71.8m
step 02322/05000 (46.44%) | loss: 4.055286 | lrm: 0.83 | dt: 1607.73ms | tok/sec: 10,190 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 17 | total time: 61.97m | eta: 71.8m
step 02323/05000 (46.46%) | loss: 4.053127 | lrm: 0.83 | dt: 1470.10ms | tok/sec: 11,144 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 17 | total time: 62.00m | eta: 71.8m
step 02324/05000 (46.48%) | loss: 4.048008 | lrm: 0.83 | dt: 1569.71ms | tok/sec: 10,437 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 17 | total time: 62.02m | eta: 71.7m
step 02325/05000 (46.50%) | loss: 4.044480 | lrm: 0.83 | dt: 1470.78ms | tok/sec: 11,139 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 17 | total time: 62.05m | eta: 71.7m
step 02326/05000 (46.52%) | loss: 4.062456 | lrm: 0.83 | dt: 1436.35ms | tok/sec: 11,406 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 17 | total time: 62.07m | eta: 71.7m
step 02327/05000 (46.54%) | loss: 4.058664 | lrm: 0.83 | dt: 1466.41ms | tok/sec: 11,172 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 17 | total time: 62.10m | eta: 71.6m
step 02328/05000 (46.56%) | loss: 4.072848 | lrm: 0.83 | dt: 1453.12ms | tok/sec: 11,275 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 17 | total time: 62.12m | eta: 71.6m
step 02329/05000 (46.58%) | loss: 4.067330 | lrm: 0.83 | dt: 1499.07ms | tok/sec: 10,929 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 17 | total time: 62.14m | eta: 71.6m
step 02330/05000 (46.60%) | loss: 4.085954 | lrm: 0.83 | dt: 1549.09ms | tok/sec: 10,576 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 17 | total time: 62.17m | eta: 71.5m
step 02331/05000 (46.62%) | loss: 4.084124 | lrm: 0.83 | dt: 1504.33ms | tok/sec: 10,891 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 17 | total time: 62.20m | eta: 71.5m
step 02332/05000 (46.64%) | loss: 4.081940 | lrm: 0.83 | dt: 1470.36ms | tok/sec: 11,142 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 17 | total time: 62.22m | eta: 71.5m
step 02333/05000 (46.66%) | loss: 4.098099 | lrm: 0.83 | dt: 1502.91ms | tok/sec: 10,901 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 17 | total time: 62.24m | eta: 71.5m
step 02334/05000 (46.68%) | loss: 4.095191 | lrm: 0.83 | dt: 1575.01ms | tok/sec: 10,402 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 17 | total time: 62.27m | eta: 71.4m
step 02335/05000 (46.70%) | loss: 4.086520 | lrm: 0.83 | dt: 1527.47ms | tok/sec: 10,726 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 17 | total time: 62.30m | eta: 71.4m
step 02336/05000 (46.72%) | loss: 4.081401 | lrm: 0.83 | dt: 1487.61ms | tok/sec: 11,013 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 17 | total time: 62.32m | eta: 71.4m
step 02337/05000 (46.74%) | loss: 4.083367 | lrm: 0.83 | dt: 1484.25ms | tok/sec: 11,038 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 17 | total time: 62.35m | eta: 71.3m
step 02338/05000 (46.76%) | loss: 4.094396 | lrm: 0.83 | dt: 1492.06ms | tok/sec: 10,980 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 17 | total time: 62.37m | eta: 71.3m
step 02339/05000 (46.78%) | loss: 4.101185 | lrm: 0.83 | dt: 1497.41ms | tok/sec: 10,941 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 17 | total time: 62.40m | eta: 71.3m
step 02340/05000 (46.80%) | loss: 4.082657 | lrm: 0.83 | dt: 1482.45ms | tok/sec: 11,051 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 17 | total time: 62.42m | eta: 71.3m
step 02341/05000 (46.82%) | loss: 4.075178 | lrm: 0.83 | dt: 1471.41ms | tok/sec: 11,134 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 18 | total time: 62.45m | eta: 71.2m
step 02342/05000 (46.84%) | loss: 4.079940 | lrm: 0.83 | dt: 1501.98ms | tok/sec: 10,908 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 18 | total time: 62.47m | eta: 71.2m
step 02343/05000 (46.86%) | loss: 4.068629 | lrm: 0.83 | dt: 1492.05ms | tok/sec: 10,980 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 18 | total time: 62.49m | eta: 71.2m
step 02344/05000 (46.88%) | loss: 4.087759 | lrm: 0.83 | dt: 1480.81ms | tok/sec: 11,064 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 18 | total time: 62.52m | eta: 71.1m
step 02345/05000 (46.90%) | loss: 4.102453 | lrm: 0.83 | dt: 1498.63ms | tok/sec: 10,932 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 18 | total time: 62.54m | eta: 71.1m
step 02346/05000 (46.92%) | loss: 4.093450 | lrm: 0.83 | dt: 1557.72ms | tok/sec: 10,517 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 18 | total time: 62.57m | eta: 71.1m
step 02347/05000 (46.94%) | loss: 4.104115 | lrm: 0.83 | dt: 1486.75ms | tok/sec: 11,019 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 18 | total time: 62.60m | eta: 71.1m
step 02348/05000 (46.96%) | loss: 4.104574 | lrm: 0.83 | dt: 1498.26ms | tok/sec: 10,935 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 18 | total time: 62.62m | eta: 71.0m
step 02349/05000 (46.98%) | loss: 4.106827 | lrm: 0.82 | dt: 1495.27ms | tok/sec: 10,957 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 18 | total time: 62.65m | eta: 71.0m
step 02350/05000 (47.00%) | loss: 4.103755 | lrm: 0.82 | dt: 1475.30ms | tok/sec: 11,105 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 18 | total time: 62.67m | eta: 71.0m
step 02351/05000 (47.02%) | loss: 4.099425 | lrm: 0.82 | dt: 1460.94ms | tok/sec: 11,214 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 18 | total time: 62.69m | eta: 70.9m
step 02352/05000 (47.04%) | loss: 4.093685 | lrm: 0.82 | dt: 1556.30ms | tok/sec: 10,527 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 18 | total time: 62.72m | eta: 70.9m
step 02353/05000 (47.06%) | loss: 4.089369 | lrm: 0.82 | dt: 1495.60ms | tok/sec: 10,954 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 18 | total time: 62.75m | eta: 70.9m
step 02354/05000 (47.08%) | loss: 4.093112 | lrm: 0.82 | dt: 1491.86ms | tok/sec: 10,982 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 18 | total time: 62.77m | eta: 70.9m
step 02355/05000 (47.10%) | loss: 4.083271 | lrm: 0.82 | dt: 1478.45ms | tok/sec: 11,081 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 18 | total time: 62.79m | eta: 70.8m
step 02356/05000 (47.12%) | loss: 4.069522 | lrm: 0.82 | dt: 1475.66ms | tok/sec: 11,102 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 18 | total time: 62.82m | eta: 70.8m
step 02357/05000 (47.14%) | loss: 4.091351 | lrm: 0.82 | dt: 1517.02ms | tok/sec: 10,800 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 18 | total time: 62.84m | eta: 70.8m
step 02358/05000 (47.16%) | loss: 4.098850 | lrm: 0.82 | dt: 1555.05ms | tok/sec: 10,536 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 18 | total time: 62.87m | eta: 70.7m
step 02359/05000 (47.18%) | loss: 4.094112 | lrm: 0.82 | dt: 1484.50ms | tok/sec: 11,036 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 18 | total time: 62.90m | eta: 70.7m
step 02360/05000 (47.20%) | loss: 4.099264 | lrm: 0.82 | dt: 1480.92ms | tok/sec: 11,063 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 18 | total time: 62.92m | eta: 70.7m
step 02361/05000 (47.22%) | loss: 4.084681 | lrm: 0.82 | dt: 1476.88ms | tok/sec: 11,093 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 18 | total time: 62.94m | eta: 70.7m
step 02362/05000 (47.24%) | loss: 4.070320 | lrm: 0.82 | dt: 1535.21ms | tok/sec: 10,672 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 18 | total time: 62.97m | eta: 70.6m
step 02363/05000 (47.26%) | loss: 4.064966 | lrm: 0.82 | dt: 1431.47ms | tok/sec: 11,445 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 18 | total time: 62.99m | eta: 70.6m
step 02364/05000 (47.28%) | loss: 4.065091 | lrm: 0.82 | dt: 1566.85ms | tok/sec: 10,456 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 19 | total time: 63.02m | eta: 70.6m
step 02365/05000 (47.30%) | loss: 4.065122 | lrm: 0.82 | dt: 1451.66ms | tok/sec: 11,286 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 19 | total time: 63.04m | eta: 70.5m
step 02366/05000 (47.32%) | loss: 4.065396 | lrm: 0.82 | dt: 1544.92ms | tok/sec: 10,605 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 19 | total time: 63.07m | eta: 70.5m
step 02367/05000 (47.34%) | loss: 4.055415 | lrm: 0.82 | dt: 1477.16ms | tok/sec: 11,091 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 19 | total time: 63.09m | eta: 70.5m
step 02368/05000 (47.36%) | loss: 4.065951 | lrm: 0.82 | dt: 1571.09ms | tok/sec: 10,428 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 19 | total time: 63.12m | eta: 70.5m
step 02369/05000 (47.38%) | loss: 4.075408 | lrm: 0.82 | dt: 1662.76ms | tok/sec: 9,853 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 19 | total time: 63.15m | eta: 70.4m
step 02370/05000 (47.40%) | loss: 4.059387 | lrm: 0.82 | dt: 1718.78ms | tok/sec: 9,532 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 19 | total time: 63.18m | eta: 70.4m
step 02371/05000 (47.42%) | loss: 4.068956 | lrm: 0.82 | dt: 1657.57ms | tok/sec: 9,884 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 19 | total time: 63.20m | eta: 70.4m
step 02372/05000 (47.44%) | loss: 4.085891 | lrm: 0.82 | dt: 1468.46ms | tok/sec: 11,157 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 19 | total time: 63.23m | eta: 70.3m
step 02373/05000 (47.46%) | loss: 4.092846 | lrm: 0.82 | dt: 1605.16ms | tok/sec: 10,207 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 19 | total time: 63.26m | eta: 70.3m
step 02374/05000 (47.48%) | loss: 4.079635 | lrm: 0.82 | dt: 1444.76ms | tok/sec: 11,340 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 19 | total time: 63.28m | eta: 70.3m
step 02375/05000 (47.50%) | loss: 4.091559 | lrm: 0.82 | dt: 1650.85ms | tok/sec: 9,924 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 19 | total time: 63.31m | eta: 70.3m
step 02376/05000 (47.52%) | loss: 4.084768 | lrm: 0.82 | dt: 1589.05ms | tok/sec: 10,310 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 19 | total time: 63.33m | eta: 70.2m
step 02377/05000 (47.54%) | loss: 4.068726 | lrm: 0.82 | dt: 1523.82ms | tok/sec: 10,751 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 19 | total time: 63.36m | eta: 70.2m
step 02378/05000 (47.56%) | loss: 4.062816 | lrm: 0.82 | dt: 1566.52ms | tok/sec: 10,458 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 19 | total time: 63.39m | eta: 70.2m
step 02379/05000 (47.58%) | loss: 4.051947 | lrm: 0.82 | dt: 1531.67ms | tok/sec: 10,696 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 19 | total time: 63.41m | eta: 70.2m
step 02380/05000 (47.60%) | loss: 4.046873 | lrm: 0.82 | dt: 1584.65ms | tok/sec: 10,339 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 19 | total time: 63.44m | eta: 70.1m
step 02381/05000 (47.62%) | loss: 4.053096 | lrm: 0.82 | dt: 1719.38ms | tok/sec: 9,528 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 19 | total time: 63.47m | eta: 70.1m
step 02382/05000 (47.64%) | loss: 4.064886 | lrm: 0.82 | dt: 1595.79ms | tok/sec: 10,267 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 19 | total time: 63.49m | eta: 70.1m
step 02383/05000 (47.66%) | loss: 4.079015 | lrm: 0.81 | dt: 1477.11ms | tok/sec: 11,091 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 19 | total time: 63.52m | eta: 70.0m
step 02384/05000 (47.68%) | loss: 4.070956 | lrm: 0.81 | dt: 1521.22ms | tok/sec: 10,770 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 19 | total time: 63.54m | eta: 70.0m
step 02385/05000 (47.70%) | loss: 4.079744 | lrm: 0.81 | dt: 1530.65ms | tok/sec: 10,703 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 19 | total time: 63.57m | eta: 70.0m
step 02386/05000 (47.72%) | loss: 4.097398 | lrm: 0.81 | dt: 1569.19ms | tok/sec: 10,441 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 19 | total time: 63.59m | eta: 70.0m
step 02387/05000 (47.74%) | loss: 4.096902 | lrm: 0.81 | dt: 1579.43ms | tok/sec: 10,373 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 20 | total time: 63.62m | eta: 69.9m
step 02388/05000 (47.76%) | loss: 4.095096 | lrm: 0.81 | dt: 1551.74ms | tok/sec: 10,558 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 20 | total time: 63.65m | eta: 69.9m
step 02389/05000 (47.78%) | loss: 4.103378 | lrm: 0.81 | dt: 1524.81ms | tok/sec: 10,744 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 20 | total time: 63.67m | eta: 69.9m
step 02390/05000 (47.80%) | loss: 4.105926 | lrm: 0.81 | dt: 1550.26ms | tok/sec: 10,568 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 20 | total time: 63.70m | eta: 69.9m
step 02391/05000 (47.82%) | loss: 4.104216 | lrm: 0.81 | dt: 1529.57ms | tok/sec: 10,711 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 20 | total time: 63.72m | eta: 69.8m
step 02392/05000 (47.84%) | loss: 4.114374 | lrm: 0.81 | dt: 1583.80ms | tok/sec: 10,344 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 20 | total time: 63.75m | eta: 69.8m
step 02393/05000 (47.86%) | loss: 4.107030 | lrm: 0.81 | dt: 1472.13ms | tok/sec: 11,129 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 20 | total time: 63.77m | eta: 69.8m
step 02394/05000 (47.88%) | loss: 4.115571 | lrm: 0.81 | dt: 1466.62ms | tok/sec: 11,171 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 20 | total time: 63.80m | eta: 69.7m
step 02395/05000 (47.90%) | loss: 4.098042 | lrm: 0.81 | dt: 1586.26ms | tok/sec: 10,328 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 20 | total time: 63.83m | eta: 69.7m
step 02396/05000 (47.92%) | loss: 4.108029 | lrm: 0.81 | dt: 1649.57ms | tok/sec: 9,932 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 20 | total time: 63.85m | eta: 69.7m
step 02397/05000 (47.94%) | loss: 4.100975 | lrm: 0.81 | dt: 1666.31ms | tok/sec: 9,832 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 20 | total time: 63.88m | eta: 69.7m
step 02398/05000 (47.96%) | loss: 4.101455 | lrm: 0.81 | dt: 1636.52ms | tok/sec: 10,011 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 20 | total time: 63.91m | eta: 69.6m
step 02399/05000 (47.98%) | loss: 4.096968 | lrm: 0.81 | dt: 1659.98ms | tok/sec: 9,869 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 20 | total time: 63.94m | eta: 69.6m
Step 02400 | Validation bpb: 1.253774
<|bos|>The capital of France is the capital of France. The capital of France is the capital of France. The
<|bos|>The chemical symbol of gold is the gold, which is the gold, and the gold, which is the gold
<|bos|>If yesterday was Friday, then tomorrow will be the first time the first time the first time the first time the first time the
<|bos|>The opposite of hot is the heat of the sun. The heat of the sun is the heat of the
<|bos|>The planets of the solar system are: the sun's rays, the sun's rays, the sun's rays, the
<|bos|>My favorite color is red, and I have a few colors of red. I have a red color
<|bos|>If 5*x + 3 = 13, then x is 2*x + 3*x + 3*x +
step 02400/05000 (48.00%) | loss: 4.095301 | lrm: 0.81 | dt: 2599.82ms | tok/sec: 6,301 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 20 | total time: 63.98m | eta: 69.6m
step 02401/05000 (48.02%) | loss: 4.084920 | lrm: 0.81 | dt: 1673.31ms | tok/sec: 9,791 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 20 | total time: 64.01m | eta: 69.6m
step 02402/05000 (48.04%) | loss: 4.087617 | lrm: 0.81 | dt: 1633.37ms | tok/sec: 10,030 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 20 | total time: 64.03m | eta: 69.5m
step 02403/05000 (48.06%) | loss: 4.071501 | lrm: 0.81 | dt: 1702.10ms | tok/sec: 9,625 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 20 | total time: 64.06m | eta: 69.5m
step 02404/05000 (48.08%) | loss: 4.063305 | lrm: 0.81 | dt: 1616.09ms | tok/sec: 10,138 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 20 | total time: 64.09m | eta: 69.5m
step 02405/05000 (48.10%) | loss: 4.062932 | lrm: 0.81 | dt: 1635.85ms | tok/sec: 10,015 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 20 | total time: 64.12m | eta: 69.5m
step 02406/05000 (48.12%) | loss: 4.069627 | lrm: 0.81 | dt: 1598.74ms | tok/sec: 10,248 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 20 | total time: 64.14m | eta: 69.4m
step 02407/05000 (48.14%) | loss: 4.048337 | lrm: 0.81 | dt: 1600.32ms | tok/sec: 10,237 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 20 | total time: 64.17m | eta: 69.4m
step 02408/05000 (48.16%) | loss: 4.048455 | lrm: 0.81 | dt: 1618.98ms | tok/sec: 10,119 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 20 | total time: 64.20m | eta: 69.4m
step 02409/05000 (48.18%) | loss: 4.044975 | lrm: 0.81 | dt: 1604.29ms | tok/sec: 10,212 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 20 | total time: 64.22m | eta: 69.4m
step 02410/05000 (48.20%) | loss: 4.045533 | lrm: 0.81 | dt: 1599.61ms | tok/sec: 10,242 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 21 | total time: 64.25m | eta: 69.3m
step 02411/05000 (48.22%) | loss: 4.053229 | lrm: 0.81 | dt: 1587.25ms | tok/sec: 10,322 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 21 | total time: 64.28m | eta: 69.3m
step 02412/05000 (48.24%) | loss: 4.075565 | lrm: 0.81 | dt: 1608.95ms | tok/sec: 10,183 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 21 | total time: 64.30m | eta: 69.3m
step 02413/05000 (48.26%) | loss: 4.078226 | lrm: 0.81 | dt: 1708.15ms | tok/sec: 9,591 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 21 | total time: 64.33m | eta: 69.3m
step 02414/05000 (48.28%) | loss: 4.076379 | lrm: 0.81 | dt: 1642.61ms | tok/sec: 9,974 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 21 | total time: 64.36m | eta: 69.2m
step 02415/05000 (48.30%) | loss: 4.075098 | lrm: 0.81 | dt: 1627.25ms | tok/sec: 10,068 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 21 | total time: 64.39m | eta: 69.2m
step 02416/05000 (48.32%) | loss: 4.067870 | lrm: 0.81 | dt: 1583.18ms | tok/sec: 10,348 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 21 | total time: 64.41m | eta: 69.2m
step 02417/05000 (48.34%) | loss: 4.060737 | lrm: 0.81 | dt: 1576.19ms | tok/sec: 10,394 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 21 | total time: 64.44m | eta: 69.2m
step 02418/05000 (48.36%) | loss: 4.050466 | lrm: 0.80 | dt: 1574.56ms | tok/sec: 10,405 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 21 | total time: 64.47m | eta: 69.1m
step 02419/05000 (48.38%) | loss: 4.051985 | lrm: 0.80 | dt: 1564.10ms | tok/sec: 10,475 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 21 | total time: 64.49m | eta: 69.1m
step 02420/05000 (48.40%) | loss: 4.054791 | lrm: 0.80 | dt: 1566.09ms | tok/sec: 10,461 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 21 | total time: 64.52m | eta: 69.1m
step 02421/05000 (48.42%) | loss: 4.070336 | lrm: 0.80 | dt: 1511.60ms | tok/sec: 10,838 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 21 | total time: 64.54m | eta: 69.0m
step 02422/05000 (48.44%) | loss: 4.059477 | lrm: 0.80 | dt: 1579.13ms | tok/sec: 10,375 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 21 | total time: 64.57m | eta: 69.0m
step 02423/05000 (48.46%) | loss: 4.062747 | lrm: 0.80 | dt: 1525.75ms | tok/sec: 10,738 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 21 | total time: 64.59m | eta: 69.0m
step 02424/05000 (48.48%) | loss: 4.051105 | lrm: 0.80 | dt: 1562.61ms | tok/sec: 10,484 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 21 | total time: 64.62m | eta: 69.0m
step 02425/05000 (48.50%) | loss: 4.060855 | lrm: 0.80 | dt: 1563.82ms | tok/sec: 10,476 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 21 | total time: 64.65m | eta: 68.9m
step 02426/05000 (48.52%) | loss: 4.054371 | lrm: 0.80 | dt: 1568.03ms | tok/sec: 10,448 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 21 | total time: 64.67m | eta: 68.9m
step 02427/05000 (48.54%) | loss: 4.031207 | lrm: 0.80 | dt: 1544.39ms | tok/sec: 10,608 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 21 | total time: 64.70m | eta: 68.9m
step 02428/05000 (48.56%) | loss: 4.057155 | lrm: 0.80 | dt: 1574.42ms | tok/sec: 10,406 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 21 | total time: 64.72m | eta: 68.8m
step 02429/05000 (48.58%) | loss: 4.067562 | lrm: 0.80 | dt: 1553.84ms | tok/sec: 10,544 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 21 | total time: 64.75m | eta: 68.8m
step 02430/05000 (48.60%) | loss: 4.065333 | lrm: 0.80 | dt: 1587.74ms | tok/sec: 10,319 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 21 | total time: 64.78m | eta: 68.8m
step 02431/05000 (48.62%) | loss: 4.065678 | lrm: 0.80 | dt: 1604.56ms | tok/sec: 10,210 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 21 | total time: 64.80m | eta: 68.8m
step 02432/05000 (48.64%) | loss: 4.068470 | lrm: 0.80 | dt: 1521.21ms | tok/sec: 10,770 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 21 | total time: 64.83m | eta: 68.7m
step 02433/05000 (48.66%) | loss: 4.048498 | lrm: 0.80 | dt: 1642.49ms | tok/sec: 9,975 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 22 | total time: 64.86m | eta: 68.7m
step 02434/05000 (48.68%) | loss: 4.043806 | lrm: 0.80 | dt: 1684.10ms | tok/sec: 9,728 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 22 | total time: 64.88m | eta: 68.7m
step 02435/05000 (48.70%) | loss: 4.027129 | lrm: 0.80 | dt: 1683.39ms | tok/sec: 9,732 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 22 | total time: 64.91m | eta: 68.7m
step 02436/05000 (48.72%) | loss: 4.028581 | lrm: 0.80 | dt: 1468.42ms | tok/sec: 11,157 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 22 | total time: 64.94m | eta: 68.6m
step 02437/05000 (48.74%) | loss: 4.043904 | lrm: 0.80 | dt: 1518.68ms | tok/sec: 10,788 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 22 | total time: 64.96m | eta: 68.6m
step 02438/05000 (48.76%) | loss: 4.054845 | lrm: 0.80 | dt: 1488.09ms | tok/sec: 11,010 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 22 | total time: 64.99m | eta: 68.6m
step 02439/05000 (48.78%) | loss: 4.048926 | lrm: 0.80 | dt: 1469.57ms | tok/sec: 11,148 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 22 | total time: 65.01m | eta: 68.5m
step 02440/05000 (48.80%) | loss: 4.052290 | lrm: 0.80 | dt: 1487.86ms | tok/sec: 11,011 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 22 | total time: 65.04m | eta: 68.5m
step 02441/05000 (48.82%) | loss: 4.062241 | lrm: 0.80 | dt: 1515.55ms | tok/sec: 10,810 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 22 | total time: 65.06m | eta: 68.5m
step 02442/05000 (48.84%) | loss: 4.062436 | lrm: 0.80 | dt: 1449.88ms | tok/sec: 11,300 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 22 | total time: 65.09m | eta: 68.5m
step 02443/05000 (48.86%) | loss: 4.051328 | lrm: 0.80 | dt: 1488.83ms | tok/sec: 11,004 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 22 | total time: 65.11m | eta: 68.4m
step 02444/05000 (48.88%) | loss: 4.060036 | lrm: 0.80 | dt: 1473.84ms | tok/sec: 11,116 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 22 | total time: 65.14m | eta: 68.4m
step 02445/05000 (48.90%) | loss: 4.067114 | lrm: 0.80 | dt: 1465.65ms | tok/sec: 11,178 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 22 | total time: 65.16m | eta: 68.4m
step 02446/05000 (48.92%) | loss: 4.084316 | lrm: 0.80 | dt: 1539.63ms | tok/sec: 10,641 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 22 | total time: 65.19m | eta: 68.3m
step 02447/05000 (48.94%) | loss: 4.086176 | lrm: 0.80 | dt: 1503.38ms | tok/sec: 10,898 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 22 | total time: 65.21m | eta: 68.3m
step 02448/05000 (48.96%) | loss: 4.087879 | lrm: 0.80 | dt: 1434.82ms | tok/sec: 11,418 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 22 | total time: 65.23m | eta: 68.3m
step 02449/05000 (48.98%) | loss: 4.077946 | lrm: 0.80 | dt: 1535.02ms | tok/sec: 10,673 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 22 | total time: 65.26m | eta: 68.3m
step 02450/05000 (49.00%) | loss: 4.079450 | lrm: 0.80 | dt: 1500.37ms | tok/sec: 10,919 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 22 | total time: 65.28m | eta: 68.2m
step 02451/05000 (49.02%) | loss: 4.071591 | lrm: 0.80 | dt: 1486.29ms | tok/sec: 11,023 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 22 | total time: 65.31m | eta: 68.2m
step 02452/05000 (49.04%) | loss: 4.056064 | lrm: 0.79 | dt: 1520.93ms | tok/sec: 10,772 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 22 | total time: 65.33m | eta: 68.2m
step 02453/05000 (49.06%) | loss: 4.055466 | lrm: 0.79 | dt: 1506.65ms | tok/sec: 10,874 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 22 | total time: 65.36m | eta: 68.1m
step 02454/05000 (49.08%) | loss: 4.052459 | lrm: 0.79 | dt: 1501.18ms | tok/sec: 10,914 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 22 | total time: 65.38m | eta: 68.1m
step 02455/05000 (49.10%) | loss: 4.053262 | lrm: 0.79 | dt: 1587.13ms | tok/sec: 10,323 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 22 | total time: 65.41m | eta: 68.1m
step 02456/05000 (49.12%) | loss: 4.037280 | lrm: 0.79 | dt: 1533.96ms | tok/sec: 10,680 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 22 | total time: 65.44m | eta: 68.1m
step 02457/05000 (49.14%) | loss: 4.043010 | lrm: 0.79 | dt: 1564.34ms | tok/sec: 10,473 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 23 | total time: 65.46m | eta: 68.0m
step 02458/05000 (49.16%) | loss: 4.031411 | lrm: 0.79 | dt: 1559.91ms | tok/sec: 10,503 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 23 | total time: 65.49m | eta: 68.0m
step 02459/05000 (49.18%) | loss: 4.030075 | lrm: 0.79 | dt: 1444.96ms | tok/sec: 11,338 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 23 | total time: 65.51m | eta: 68.0m
step 02460/05000 (49.20%) | loss: 4.036459 | lrm: 0.79 | dt: 1541.77ms | tok/sec: 10,626 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 23 | total time: 65.54m | eta: 67.9m
step 02461/05000 (49.22%) | loss: 4.041661 | lrm: 0.79 | dt: 1483.33ms | tok/sec: 11,045 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 23 | total time: 65.56m | eta: 67.9m
step 02462/05000 (49.24%) | loss: 4.046787 | lrm: 0.79 | dt: 1544.18ms | tok/sec: 10,610 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 23 | total time: 65.59m | eta: 67.9m
step 02463/05000 (49.26%) | loss: 4.034835 | lrm: 0.79 | dt: 1399.25ms | tok/sec: 11,709 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 23 | total time: 65.61m | eta: 67.9m
step 02464/05000 (49.28%) | loss: 4.025098 | lrm: 0.79 | dt: 1461.77ms | tok/sec: 11,208 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 23 | total time: 65.64m | eta: 67.8m
step 02465/05000 (49.30%) | loss: 4.030183 | lrm: 0.79 | dt: 1469.84ms | tok/sec: 11,146 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 23 | total time: 65.66m | eta: 67.8m
step 02466/05000 (49.32%) | loss: 4.036486 | lrm: 0.79 | dt: 1463.03ms | tok/sec: 11,198 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 23 | total time: 65.69m | eta: 67.8m
step 02467/05000 (49.34%) | loss: 4.042342 | lrm: 0.79 | dt: 1497.82ms | tok/sec: 10,938 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 23 | total time: 65.71m | eta: 67.7m
step 02468/05000 (49.36%) | loss: 4.030950 | lrm: 0.79 | dt: 1503.67ms | tok/sec: 10,895 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 23 | total time: 65.74m | eta: 67.7m
step 02469/05000 (49.38%) | loss: 4.031170 | lrm: 0.79 | dt: 1496.05ms | tok/sec: 10,951 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 23 | total time: 65.76m | eta: 67.7m
step 02470/05000 (49.40%) | loss: 4.017336 | lrm: 0.79 | dt: 1493.79ms | tok/sec: 10,968 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 23 | total time: 65.79m | eta: 67.7m
step 02471/05000 (49.42%) | loss: 4.023071 | lrm: 0.79 | dt: 1465.27ms | tok/sec: 11,181 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 23 | total time: 65.81m | eta: 67.6m
step 02472/05000 (49.44%) | loss: 3.999341 | lrm: 0.79 | dt: 1445.24ms | tok/sec: 11,336 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 23 | total time: 65.83m | eta: 67.6m
step 02473/05000 (49.46%) | loss: 3.981178 | lrm: 0.79 | dt: 1522.12ms | tok/sec: 10,763 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 23 | total time: 65.86m | eta: 67.6m
step 02474/05000 (49.48%) | loss: 3.975388 | lrm: 0.79 | dt: 1541.54ms | tok/sec: 10,628 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 23 | total time: 65.89m | eta: 67.5m
step 02475/05000 (49.50%) | loss: 3.962970 | lrm: 0.79 | dt: 1473.04ms | tok/sec: 11,122 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 23 | total time: 65.91m | eta: 67.5m
step 02476/05000 (49.52%) | loss: 3.972870 | lrm: 0.79 | dt: 1479.25ms | tok/sec: 11,075 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 23 | total time: 65.93m | eta: 67.5m
step 02477/05000 (49.54%) | loss: 3.976965 | lrm: 0.79 | dt: 1507.31ms | tok/sec: 10,869 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 23 | total time: 65.96m | eta: 67.5m
step 02478/05000 (49.56%) | loss: 3.970518 | lrm: 0.79 | dt: 1455.38ms | tok/sec: 11,257 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 23 | total time: 65.98m | eta: 67.4m
step 02479/05000 (49.58%) | loss: 3.980788 | lrm: 0.79 | dt: 1505.65ms | tok/sec: 10,881 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 23 | total time: 66.01m | eta: 67.4m
step 02480/05000 (49.60%) | loss: 3.989459 | lrm: 0.79 | dt: 1541.21ms | tok/sec: 10,630 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 23 | total time: 66.03m | eta: 67.4m
step 02481/05000 (49.62%) | loss: 3.990839 | lrm: 0.79 | dt: 1435.82ms | tok/sec: 11,410 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 24 | total time: 66.06m | eta: 67.3m
step 02482/05000 (49.64%) | loss: 4.007945 | lrm: 0.79 | dt: 1521.65ms | tok/sec: 10,767 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 24 | total time: 66.08m | eta: 67.3m
step 02483/05000 (49.66%) | loss: 4.000280 | lrm: 0.79 | dt: 1531.53ms | tok/sec: 10,697 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 24 | total time: 66.11m | eta: 67.3m
step 02484/05000 (49.68%) | loss: 4.002070 | lrm: 0.79 | dt: 1535.86ms | tok/sec: 10,667 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 24 | total time: 66.14m | eta: 67.3m
step 02485/05000 (49.70%) | loss: 3.991703 | lrm: 0.79 | dt: 1521.82ms | tok/sec: 10,766 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 24 | total time: 66.16m | eta: 67.2m
step 02486/05000 (49.72%) | loss: 3.996778 | lrm: 0.78 | dt: 1514.26ms | tok/sec: 10,819 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 24 | total time: 66.19m | eta: 67.2m
step 02487/05000 (49.74%) | loss: 4.003618 | lrm: 0.78 | dt: 1482.76ms | tok/sec: 11,049 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 24 | total time: 66.21m | eta: 67.2m
step 02488/05000 (49.76%) | loss: 4.005928 | lrm: 0.78 | dt: 1485.02ms | tok/sec: 11,032 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 24 | total time: 66.24m | eta: 67.1m
step 02489/05000 (49.78%) | loss: 3.996925 | lrm: 0.78 | dt: 1507.76ms | tok/sec: 10,866 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 24 | total time: 66.26m | eta: 67.1m
step 02490/05000 (49.80%) | loss: 4.004488 | lrm: 0.78 | dt: 1578.21ms | tok/sec: 10,381 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 24 | total time: 66.29m | eta: 67.1m
step 02491/05000 (49.82%) | loss: 4.014632 | lrm: 0.78 | dt: 1553.19ms | tok/sec: 10,548 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 24 | total time: 66.31m | eta: 67.1m
step 02492/05000 (49.84%) | loss: 4.009565 | lrm: 0.78 | dt: 1582.45ms | tok/sec: 10,353 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 24 | total time: 66.34m | eta: 67.0m
step 02493/05000 (49.86%) | loss: 4.013975 | lrm: 0.78 | dt: 1771.12ms | tok/sec: 9,250 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 24 | total time: 66.37m | eta: 67.0m
step 02494/05000 (49.88%) | loss: 4.028438 | lrm: 0.78 | dt: 1769.18ms | tok/sec: 9,260 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 24 | total time: 66.40m | eta: 67.0m
step 02495/05000 (49.90%) | loss: 4.019891 | lrm: 0.78 | dt: 1611.28ms | tok/sec: 10,168 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 24 | total time: 66.42m | eta: 67.0m
step 02496/05000 (49.92%) | loss: 4.011234 | lrm: 0.78 | dt: 1699.75ms | tok/sec: 9,639 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 24 | total time: 66.45m | eta: 66.9m
step 02497/05000 (49.94%) | loss: 4.014286 | lrm: 0.78 | dt: 1666.72ms | tok/sec: 9,830 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 24 | total time: 66.48m | eta: 66.9m
step 02498/05000 (49.96%) | loss: 4.010304 | lrm: 0.78 | dt: 1667.57ms | tok/sec: 9,825 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 24 | total time: 66.51m | eta: 66.9m
step 02499/05000 (49.98%) | loss: 4.003899 | lrm: 0.78 | dt: 1590.58ms | tok/sec: 10,300 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 24 | total time: 66.54m | eta: 66.9m
Step 02500 | Validation bpb: 1.248040
<|bos|>The capital of France is a country of country that is a country of country. It is a country of
<|bos|>The chemical symbol of gold is the symbol of the symbol of the symbol of the symbol of the symbol of the
<|bos|>If yesterday was Friday, then tomorrow will be the first time. The first time we got to the next level was in the
<|bos|>The opposite of hot is the fact that the heat of the sun is not the same as the sun is
<|bos|>The planets of the solar system are: the solar system is the solar system. The solar system is the solar system.
<|bos|>My favorite color is the red, green, and blue. I have a lot of color, but
<|bos|>If 5*x + 3 = 13, then x is 3.5*x + 3.5*x + 3
step 02500/05000 (50.00%) | loss: 4.008123 | lrm: 0.78 | dt: 2363.94ms | tok/sec: 6,930 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 24 | total time: 66.57m | eta: 66.8m
step 02501/05000 (50.02%) | loss: 4.017455 | lrm: 0.78 | dt: 1574.35ms | tok/sec: 10,406 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 24 | total time: 66.60m | eta: 66.8m
step 02502/05000 (50.04%) | loss: 4.011346 | lrm: 0.78 | dt: 1553.83ms | tok/sec: 10,544 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 24 | total time: 66.63m | eta: 66.8m
step 02503/05000 (50.06%) | loss: 4.017480 | lrm: 0.78 | dt: 1560.06ms | tok/sec: 10,502 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 24 | total time: 66.65m | eta: 66.8m
step 02504/05000 (50.08%) | loss: 4.037014 | lrm: 0.78 | dt: 1598.05ms | tok/sec: 10,252 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 24 | total time: 66.68m | eta: 66.7m
step 02505/05000 (50.10%) | loss: 4.030397 | lrm: 0.78 | dt: 1569.39ms | tok/sec: 10,439 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 25 | total time: 66.71m | eta: 66.7m
step 02506/05000 (50.12%) | loss: 4.047843 | lrm: 0.78 | dt: 1562.19ms | tok/sec: 10,487 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 25 | total time: 66.73m | eta: 66.7m
step 02507/05000 (50.14%) | loss: 4.033374 | lrm: 0.78 | dt: 1632.03ms | tok/sec: 10,039 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 25 | total time: 66.76m | eta: 66.7m
step 02508/05000 (50.16%) | loss: 4.022998 | lrm: 0.78 | dt: 1625.45ms | tok/sec: 10,079 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 25 | total time: 66.79m | eta: 66.6m
step 02509/05000 (50.18%) | loss: 4.035793 | lrm: 0.78 | dt: 1572.69ms | tok/sec: 10,417 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 25 | total time: 66.81m | eta: 66.6m
step 02510/05000 (50.20%) | loss: 4.027736 | lrm: 0.78 | dt: 1593.52ms | tok/sec: 10,281 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 25 | total time: 66.84m | eta: 66.6m
step 02511/05000 (50.22%) | loss: 4.030376 | lrm: 0.78 | dt: 1614.92ms | tok/sec: 10,145 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 25 | total time: 66.87m | eta: 66.5m
step 02512/05000 (50.24%) | loss: 4.026629 | lrm: 0.78 | dt: 1653.09ms | tok/sec: 9,911 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 25 | total time: 66.89m | eta: 66.5m
step 02513/05000 (50.26%) | loss: 4.025248 | lrm: 0.78 | dt: 1579.40ms | tok/sec: 10,373 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 25 | total time: 66.92m | eta: 66.5m
step 02514/05000 (50.28%) | loss: 4.019036 | lrm: 0.78 | dt: 1694.12ms | tok/sec: 9,671 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 25 | total time: 66.95m | eta: 66.5m
step 02515/05000 (50.30%) | loss: 4.031155 | lrm: 0.78 | dt: 1535.05ms | tok/sec: 10,673 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 25 | total time: 66.97m | eta: 66.4m
step 02516/05000 (50.32%) | loss: 4.058132 | lrm: 0.78 | dt: 1519.21ms | tok/sec: 10,784 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 25 | total time: 67.00m | eta: 66.4m
step 02517/05000 (50.34%) | loss: 4.065607 | lrm: 0.78 | dt: 1578.92ms | tok/sec: 10,376 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 25 | total time: 67.02m | eta: 66.4m
step 02518/05000 (50.36%) | loss: 4.057904 | lrm: 0.78 | dt: 1518.28ms | tok/sec: 10,791 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 25 | total time: 67.05m | eta: 66.4m
step 02519/05000 (50.38%) | loss: 4.044512 | lrm: 0.78 | dt: 1499.69ms | tok/sec: 10,924 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 25 | total time: 67.08m | eta: 66.3m
step 02520/05000 (50.40%) | loss: 4.054696 | lrm: 0.77 | dt: 1480.54ms | tok/sec: 11,066 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 25 | total time: 67.10m | eta: 66.3m
step 02521/05000 (50.42%) | loss: 4.063020 | lrm: 0.77 | dt: 1502.51ms | tok/sec: 10,904 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 25 | total time: 67.12m | eta: 66.3m
step 02522/05000 (50.44%) | loss: 4.046747 | lrm: 0.77 | dt: 1529.93ms | tok/sec: 10,708 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 25 | total time: 67.15m | eta: 66.2m
step 02523/05000 (50.46%) | loss: 4.051224 | lrm: 0.77 | dt: 1469.72ms | tok/sec: 11,147 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 25 | total time: 67.17m | eta: 66.2m
step 02524/05000 (50.48%) | loss: 4.062795 | lrm: 0.77 | dt: 1486.47ms | tok/sec: 11,022 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 25 | total time: 67.20m | eta: 66.2m
step 02525/05000 (50.50%) | loss: 4.057630 | lrm: 0.77 | dt: 1427.94ms | tok/sec: 11,473 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 25 | total time: 67.22m | eta: 66.2m
step 02526/05000 (50.52%) | loss: 4.061154 | lrm: 0.77 | dt: 1542.78ms | tok/sec: 10,619 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 25 | total time: 67.25m | eta: 66.1m
step 02527/05000 (50.54%) | loss: 4.051100 | lrm: 0.77 | dt: 1522.03ms | tok/sec: 10,764 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 25 | total time: 67.27m | eta: 66.1m
step 02528/05000 (50.56%) | loss: 4.045420 | lrm: 0.77 | dt: 1535.87ms | tok/sec: 10,667 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 26 | total time: 67.30m | eta: 66.1m
step 02529/05000 (50.58%) | loss: 4.040941 | lrm: 0.77 | dt: 1518.92ms | tok/sec: 10,786 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 26 | total time: 67.33m | eta: 66.0m
step 02530/05000 (50.60%) | loss: 4.038511 | lrm: 0.77 | dt: 1562.53ms | tok/sec: 10,485 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 26 | total time: 67.35m | eta: 66.0m
step 02531/05000 (50.62%) | loss: 4.044449 | lrm: 0.77 | dt: 1482.38ms | tok/sec: 11,052 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 26 | total time: 67.38m | eta: 66.0m
step 02532/05000 (50.64%) | loss: 4.036524 | lrm: 0.77 | dt: 1527.57ms | tok/sec: 10,725 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 26 | total time: 67.40m | eta: 66.0m
step 02533/05000 (50.66%) | loss: 4.047932 | lrm: 0.77 | dt: 1512.73ms | tok/sec: 10,830 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 26 | total time: 67.43m | eta: 65.9m
step 02534/05000 (50.68%) | loss: 4.042727 | lrm: 0.77 | dt: 1507.32ms | tok/sec: 10,869 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 26 | total time: 67.45m | eta: 65.9m
step 02535/05000 (50.70%) | loss: 4.044760 | lrm: 0.77 | dt: 1484.19ms | tok/sec: 11,039 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 26 | total time: 67.48m | eta: 65.9m
step 02536/05000 (50.72%) | loss: 4.025530 | lrm: 0.77 | dt: 1532.26ms | tok/sec: 10,692 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 26 | total time: 67.50m | eta: 65.8m
step 02537/05000 (50.74%) | loss: 4.026953 | lrm: 0.77 | dt: 1532.66ms | tok/sec: 10,689 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 26 | total time: 67.53m | eta: 65.8m
step 02538/05000 (50.76%) | loss: 4.036548 | lrm: 0.77 | dt: 1470.48ms | tok/sec: 11,141 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 26 | total time: 67.55m | eta: 65.8m
step 02539/05000 (50.78%) | loss: 4.038498 | lrm: 0.77 | dt: 1602.00ms | tok/sec: 10,227 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 26 | total time: 67.58m | eta: 65.8m
step 02540/05000 (50.80%) | loss: 4.014073 | lrm: 0.77 | dt: 1532.25ms | tok/sec: 10,692 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 26 | total time: 67.60m | eta: 65.7m
step 02541/05000 (50.82%) | loss: 4.021807 | lrm: 0.77 | dt: 1572.49ms | tok/sec: 10,419 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 26 | total time: 67.63m | eta: 65.7m
step 02542/05000 (50.84%) | loss: 4.006549 | lrm: 0.77 | dt: 1847.18ms | tok/sec: 8,869 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 26 | total time: 67.66m | eta: 65.7m
step 02543/05000 (50.86%) | loss: 4.021398 | lrm: 0.77 | dt: 1829.54ms | tok/sec: 8,955 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 26 | total time: 67.69m | eta: 65.7m
step 02544/05000 (50.88%) | loss: 4.022424 | lrm: 0.77 | dt: 1839.90ms | tok/sec: 8,904 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 26 | total time: 67.72m | eta: 65.6m
step 02545/05000 (50.90%) | loss: 4.012019 | lrm: 0.77 | dt: 1655.38ms | tok/sec: 9,897 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 26 | total time: 67.75m | eta: 65.6m
step 02546/05000 (50.92%) | loss: 4.013158 | lrm: 0.77 | dt: 1697.64ms | tok/sec: 9,651 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 26 | total time: 67.78m | eta: 65.6m
step 02547/05000 (50.94%) | loss: 4.031742 | lrm: 0.77 | dt: 1649.10ms | tok/sec: 9,935 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 26 | total time: 67.81m | eta: 65.6m
step 02548/05000 (50.96%) | loss: 4.029454 | lrm: 0.77 | dt: 1633.15ms | tok/sec: 10,032 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 26 | total time: 67.83m | eta: 65.5m
step 02549/05000 (50.98%) | loss: 4.014942 | lrm: 0.77 | dt: 1650.45ms | tok/sec: 9,926 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 26 | total time: 67.86m | eta: 65.5m
step 02550/05000 (51.00%) | loss: 4.009381 | lrm: 0.77 | dt: 1616.84ms | tok/sec: 10,133 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 26 | total time: 67.89m | eta: 65.5m
step 02551/05000 (51.02%) | loss: 4.009432 | lrm: 0.77 | dt: 1673.29ms | tok/sec: 9,791 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 27 | total time: 67.92m | eta: 65.5m
step 02552/05000 (51.04%) | loss: 4.022744 | lrm: 0.77 | dt: 1646.93ms | tok/sec: 9,948 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 27 | total time: 67.94m | eta: 65.4m
step 02553/05000 (51.06%) | loss: 4.028468 | lrm: 0.77 | dt: 1495.23ms | tok/sec: 10,957 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 27 | total time: 67.97m | eta: 65.4m
step 02554/05000 (51.08%) | loss: 4.032752 | lrm: 0.76 | dt: 1581.25ms | tok/sec: 10,361 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 27 | total time: 67.99m | eta: 65.4m
step 02555/05000 (51.10%) | loss: 4.035092 | lrm: 0.76 | dt: 1509.55ms | tok/sec: 10,853 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 27 | total time: 68.02m | eta: 65.3m
step 02556/05000 (51.12%) | loss: 4.037509 | lrm: 0.76 | dt: 1500.76ms | tok/sec: 10,917 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 27 | total time: 68.04m | eta: 65.3m
step 02557/05000 (51.14%) | loss: 4.043875 | lrm: 0.76 | dt: 1451.54ms | tok/sec: 11,287 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 27 | total time: 68.07m | eta: 65.3m
step 02558/05000 (51.16%) | loss: 4.034408 | lrm: 0.76 | dt: 1478.05ms | tok/sec: 11,084 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 27 | total time: 68.09m | eta: 65.3m
step 02559/05000 (51.18%) | loss: 4.023724 | lrm: 0.76 | dt: 1481.65ms | tok/sec: 11,057 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 27 | total time: 68.12m | eta: 65.2m
step 02560/05000 (51.20%) | loss: 4.022067 | lrm: 0.76 | dt: 1482.71ms | tok/sec: 11,050 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 27 | total time: 68.14m | eta: 65.2m
step 02561/05000 (51.22%) | loss: 4.031055 | lrm: 0.76 | dt: 1503.49ms | tok/sec: 10,897 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 27 | total time: 68.17m | eta: 65.2m
step 02562/05000 (51.24%) | loss: 4.031777 | lrm: 0.76 | dt: 1521.60ms | tok/sec: 10,767 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 27 | total time: 68.19m | eta: 65.1m
step 02563/05000 (51.26%) | loss: 4.027368 | lrm: 0.76 | dt: 1457.82ms | tok/sec: 11,238 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 27 | total time: 68.22m | eta: 65.1m
step 02564/05000 (51.28%) | loss: 4.036781 | lrm: 0.76 | dt: 1432.48ms | tok/sec: 11,437 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 27 | total time: 68.24m | eta: 65.1m
step 02565/05000 (51.30%) | loss: 4.030714 | lrm: 0.76 | dt: 1426.69ms | tok/sec: 11,483 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 27 | total time: 68.26m | eta: 65.1m
step 02566/05000 (51.32%) | loss: 4.022742 | lrm: 0.76 | dt: 1534.77ms | tok/sec: 10,675 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 27 | total time: 68.29m | eta: 65.0m
step 02567/05000 (51.34%) | loss: 4.034774 | lrm: 0.76 | dt: 1440.91ms | tok/sec: 11,370 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 27 | total time: 68.31m | eta: 65.0m
step 02568/05000 (51.36%) | loss: 4.039374 | lrm: 0.76 | dt: 1641.14ms | tok/sec: 9,983 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 27 | total time: 68.34m | eta: 65.0m
step 02569/05000 (51.38%) | loss: 4.031931 | lrm: 0.76 | dt: 1643.92ms | tok/sec: 9,966 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 27 | total time: 68.37m | eta: 64.9m
step 02570/05000 (51.40%) | loss: 4.033528 | lrm: 0.76 | dt: 1754.94ms | tok/sec: 9,335 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 27 | total time: 68.40m | eta: 64.9m
step 02571/05000 (51.42%) | loss: 4.029833 | lrm: 0.76 | dt: 1653.24ms | tok/sec: 9,910 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 27 | total time: 68.43m | eta: 64.9m
step 02572/05000 (51.44%) | loss: 4.024841 | lrm: 0.76 | dt: 1764.10ms | tok/sec: 9,287 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 27 | total time: 68.46m | eta: 64.9m
step 02573/05000 (51.46%) | loss: 4.033227 | lrm: 0.76 | dt: 1678.29ms | tok/sec: 9,762 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 27 | total time: 68.48m | eta: 64.8m
step 02574/05000 (51.48%) | loss: 4.042949 | lrm: 0.76 | dt: 1480.38ms | tok/sec: 11,067 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 28 | total time: 68.51m | eta: 64.8m
step 02575/05000 (51.50%) | loss: 4.034511 | lrm: 0.76 | dt: 1504.34ms | tok/sec: 10,891 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 28 | total time: 68.53m | eta: 64.8m
step 02576/05000 (51.52%) | loss: 4.052902 | lrm: 0.76 | dt: 1508.90ms | tok/sec: 10,858 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 28 | total time: 68.56m | eta: 64.8m
step 02577/05000 (51.54%) | loss: 4.045924 | lrm: 0.76 | dt: 1444.29ms | tok/sec: 11,343 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 28 | total time: 68.58m | eta: 64.7m
step 02578/05000 (51.56%) | loss: 4.046682 | lrm: 0.76 | dt: 1721.56ms | tok/sec: 9,516 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 28 | total time: 68.61m | eta: 64.7m
step 02579/05000 (51.58%) | loss: 4.040542 | lrm: 0.76 | dt: 1649.72ms | tok/sec: 9,931 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 28 | total time: 68.64m | eta: 64.7m
step 02580/05000 (51.60%) | loss: 4.040881 | lrm: 0.76 | dt: 1710.69ms | tok/sec: 9,577 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 28 | total time: 68.67m | eta: 64.7m
step 02581/05000 (51.62%) | loss: 4.046909 | lrm: 0.76 | dt: 1517.49ms | tok/sec: 10,796 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 28 | total time: 68.69m | eta: 64.6m
step 02582/05000 (51.64%) | loss: 4.056223 | lrm: 0.76 | dt: 1495.00ms | tok/sec: 10,959 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 28 | total time: 68.72m | eta: 64.6m
step 02583/05000 (51.66%) | loss: 4.040398 | lrm: 0.76 | dt: 1605.93ms | tok/sec: 10,202 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 28 | total time: 68.74m | eta: 64.6m
step 02584/05000 (51.68%) | loss: 4.054340 | lrm: 0.76 | dt: 1537.40ms | tok/sec: 10,656 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 28 | total time: 68.77m | eta: 64.5m
step 02585/05000 (51.70%) | loss: 4.057355 | lrm: 0.76 | dt: 1585.55ms | tok/sec: 10,333 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 28 | total time: 68.80m | eta: 64.5m
step 02586/05000 (51.72%) | loss: 4.043452 | lrm: 0.76 | dt: 1496.43ms | tok/sec: 10,948 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 28 | total time: 68.82m | eta: 64.5m
step 02587/05000 (51.74%) | loss: 4.039669 | lrm: 0.76 | dt: 1523.58ms | tok/sec: 10,753 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 28 | total time: 68.85m | eta: 64.5m
step 02588/05000 (51.76%) | loss: 4.046796 | lrm: 0.76 | dt: 1555.34ms | tok/sec: 10,534 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 28 | total time: 68.87m | eta: 64.4m
step 02589/05000 (51.78%) | loss: 4.031487 | lrm: 0.75 | dt: 1524.81ms | tok/sec: 10,744 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 28 | total time: 68.90m | eta: 64.4m
step 02590/05000 (51.80%) | loss: 4.023339 | lrm: 0.75 | dt: 1576.31ms | tok/sec: 10,393 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 28 | total time: 68.92m | eta: 64.4m
step 02591/05000 (51.82%) | loss: 4.026652 | lrm: 0.75 | dt: 1630.12ms | tok/sec: 10,050 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 28 | total time: 68.95m | eta: 64.4m
step 02592/05000 (51.84%) | loss: 4.000110 | lrm: 0.75 | dt: 1582.49ms | tok/sec: 10,353 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 28 | total time: 68.98m | eta: 64.3m
step 02593/05000 (51.86%) | loss: 3.986843 | lrm: 0.75 | dt: 1518.17ms | tok/sec: 10,791 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 28 | total time: 69.00m | eta: 64.3m
step 02594/05000 (51.88%) | loss: 4.001293 | lrm: 0.75 | dt: 1634.90ms | tok/sec: 10,021 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 28 | total time: 69.03m | eta: 64.3m
step 02595/05000 (51.90%) | loss: 4.003794 | lrm: 0.75 | dt: 1568.09ms | tok/sec: 10,448 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 28 | total time: 69.06m | eta: 64.2m
step 02596/05000 (51.92%) | loss: 3.999942 | lrm: 0.75 | dt: 1680.46ms | tok/sec: 9,749 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 28 | total time: 69.08m | eta: 64.2m
step 02597/05000 (51.94%) | loss: 3.996921 | lrm: 0.75 | dt: 1623.16ms | tok/sec: 10,093 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 28 | total time: 69.11m | eta: 64.2m
step 02598/05000 (51.96%) | loss: 3.991375 | lrm: 0.75 | dt: 1704.77ms | tok/sec: 9,610 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 29 | total time: 69.14m | eta: 64.2m
step 02599/05000 (51.98%) | loss: 3.992915 | lrm: 0.75 | dt: 1477.04ms | tok/sec: 11,092 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 29 | total time: 69.16m | eta: 64.1m
Step 02600 | Validation bpb: 1.243302
<|bos|>The capital of France is the capital of France. The capital of France is the capital of France. The
<|bos|>The chemical symbol of gold is the gold, which is a symbol of gold. It is a symbol of gold
<|bos|>If yesterday was Friday, then tomorrow will be the first time of year. The first time of year was Friday, then the
<|bos|>The opposite of hot is the heat of the sun. The sun is the same as the sun is the
<|bos|>The planets of the solar system are: the solar system, the solar system, the solar system, the solar system,
<|bos|>My favorite color is the blue, green, and blue. I have a lot of colors and I
<|bos|>If 5*x + 3 = 13, then x is 3.2.2.2.2.2.2.2
step 02600/05000 (52.00%) | loss: 3.989062 | lrm: 0.75 | dt: 1665.55ms | tok/sec: 9,836 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 29 | total time: 69.19m | eta: 64.1m
step 02601/05000 (52.02%) | loss: 3.981579 | lrm: 0.75 | dt: 1627.08ms | tok/sec: 10,069 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 29 | total time: 69.22m | eta: 64.1m
step 02602/05000 (52.04%) | loss: 3.990627 | lrm: 0.75 | dt: 1572.32ms | tok/sec: 10,420 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 29 | total time: 69.25m | eta: 64.1m
step 02603/05000 (52.06%) | loss: 4.005072 | lrm: 0.75 | dt: 1581.33ms | tok/sec: 10,360 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 29 | total time: 69.27m | eta: 64.0m
step 02604/05000 (52.08%) | loss: 4.011602 | lrm: 0.75 | dt: 1622.63ms | tok/sec: 10,097 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 29 | total time: 69.30m | eta: 64.0m
step 02605/05000 (52.10%) | loss: 4.017005 | lrm: 0.75 | dt: 1467.42ms | tok/sec: 11,165 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 29 | total time: 69.32m | eta: 64.0m
step 02606/05000 (52.12%) | loss: 4.025214 | lrm: 0.75 | dt: 1726.98ms | tok/sec: 9,487 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 29 | total time: 69.35m | eta: 64.0m
step 02607/05000 (52.14%) | loss: 4.022822 | lrm: 0.75 | dt: 1652.83ms | tok/sec: 9,912 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 29 | total time: 69.38m | eta: 63.9m
step 02608/05000 (52.16%) | loss: 4.016284 | lrm: 0.75 | dt: 1608.97ms | tok/sec: 10,182 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 29 | total time: 69.41m | eta: 63.9m
step 02609/05000 (52.18%) | loss: 4.013225 | lrm: 0.75 | dt: 1533.57ms | tok/sec: 10,683 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 29 | total time: 69.43m | eta: 63.9m
step 02610/05000 (52.20%) | loss: 4.018381 | lrm: 0.75 | dt: 1518.80ms | tok/sec: 10,787 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 29 | total time: 69.46m | eta: 63.8m
step 02611/05000 (52.22%) | loss: 4.022417 | lrm: 0.75 | dt: 1543.89ms | tok/sec: 10,612 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 29 | total time: 69.48m | eta: 63.8m
step 02612/05000 (52.24%) | loss: 4.022460 | lrm: 0.75 | dt: 1544.69ms | tok/sec: 10,606 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 29 | total time: 69.51m | eta: 63.8m
step 02613/05000 (52.26%) | loss: 4.024848 | lrm: 0.75 | dt: 1590.86ms | tok/sec: 10,298 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 29 | total time: 69.54m | eta: 63.8m
step 02614/05000 (52.28%) | loss: 4.021027 | lrm: 0.75 | dt: 1604.00ms | tok/sec: 10,214 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 29 | total time: 69.56m | eta: 63.7m
step 02615/05000 (52.30%) | loss: 4.020702 | lrm: 0.75 | dt: 1528.60ms | tok/sec: 10,718 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 29 | total time: 69.59m | eta: 63.7m
step 02616/05000 (52.32%) | loss: 4.013741 | lrm: 0.75 | dt: 1496.95ms | tok/sec: 10,944 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 29 | total time: 69.61m | eta: 63.7m
step 02617/05000 (52.34%) | loss: 4.029203 | lrm: 0.75 | dt: 1534.20ms | tok/sec: 10,679 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 29 | total time: 69.64m | eta: 63.7m
step 02618/05000 (52.36%) | loss: 4.033740 | lrm: 0.75 | dt: 1486.74ms | tok/sec: 11,020 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 29 | total time: 69.66m | eta: 63.6m
step 02619/05000 (52.38%) | loss: 4.018882 | lrm: 0.75 | dt: 1623.08ms | tok/sec: 10,094 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 29 | total time: 69.69m | eta: 63.6m
step 02620/05000 (52.40%) | loss: 4.020507 | lrm: 0.75 | dt: 1611.97ms | tok/sec: 10,163 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 29 | total time: 69.72m | eta: 63.6m
step 02621/05000 (52.42%) | loss: 4.018954 | lrm: 0.75 | dt: 1482.47ms | tok/sec: 11,051 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 30 | total time: 69.74m | eta: 63.5m
step 02622/05000 (52.44%) | loss: 4.019183 | lrm: 0.75 | dt: 1690.39ms | tok/sec: 9,692 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 30 | total time: 69.77m | eta: 63.5m
step 02623/05000 (52.46%) | loss: 4.021460 | lrm: 0.74 | dt: 1609.74ms | tok/sec: 10,178 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 30 | total time: 69.80m | eta: 63.5m
step 02624/05000 (52.48%) | loss: 4.018856 | lrm: 0.74 | dt: 1516.32ms | tok/sec: 10,805 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 30 | total time: 69.82m | eta: 63.5m
step 02625/05000 (52.50%) | loss: 4.007597 | lrm: 0.74 | dt: 1553.49ms | tok/sec: 10,546 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 30 | total time: 69.85m | eta: 63.4m
step 02626/05000 (52.52%) | loss: 4.005677 | lrm: 0.74 | dt: 1618.09ms | tok/sec: 10,125 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 30 | total time: 69.87m | eta: 63.4m
step 02627/05000 (52.54%) | loss: 4.001795 | lrm: 0.74 | dt: 1620.70ms | tok/sec: 10,109 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 30 | total time: 69.90m | eta: 63.4m
step 02628/05000 (52.56%) | loss: 4.006754 | lrm: 0.74 | dt: 1621.64ms | tok/sec: 10,103 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 30 | total time: 69.93m | eta: 63.4m
step 02629/05000 (52.58%) | loss: 3.998847 | lrm: 0.74 | dt: 1452.04ms | tok/sec: 11,283 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 30 | total time: 69.95m | eta: 63.3m
step 02630/05000 (52.60%) | loss: 4.004866 | lrm: 0.74 | dt: 1628.13ms | tok/sec: 10,063 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 30 | total time: 69.98m | eta: 63.3m
step 02631/05000 (52.62%) | loss: 3.989622 | lrm: 0.74 | dt: 1613.23ms | tok/sec: 10,156 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 30 | total time: 70.01m | eta: 63.3m
step 02632/05000 (52.64%) | loss: 3.996819 | lrm: 0.74 | dt: 1645.56ms | tok/sec: 9,956 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 30 | total time: 70.03m | eta: 63.2m
step 02633/05000 (52.66%) | loss: 3.992717 | lrm: 0.74 | dt: 1657.27ms | tok/sec: 9,886 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 30 | total time: 70.06m | eta: 63.2m
step 02634/05000 (52.68%) | loss: 3.996420 | lrm: 0.74 | dt: 1599.37ms | tok/sec: 10,244 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 30 | total time: 70.09m | eta: 63.2m
step 02635/05000 (52.70%) | loss: 3.992548 | lrm: 0.74 | dt: 1611.82ms | tok/sec: 10,164 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 30 | total time: 70.12m | eta: 63.2m
step 02636/05000 (52.72%) | loss: 4.000243 | lrm: 0.74 | dt: 1616.42ms | tok/sec: 10,135 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 30 | total time: 70.14m | eta: 63.1m
step 02637/05000 (52.74%) | loss: 4.005875 | lrm: 0.74 | dt: 1553.31ms | tok/sec: 10,547 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 30 | total time: 70.17m | eta: 63.1m
step 02638/05000 (52.76%) | loss: 4.010674 | lrm: 0.74 | dt: 1564.29ms | tok/sec: 10,473 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 30 | total time: 70.19m | eta: 63.1m
step 02639/05000 (52.78%) | loss: 4.017403 | lrm: 0.74 | dt: 1580.40ms | tok/sec: 10,366 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 30 | total time: 70.22m | eta: 63.1m
step 02640/05000 (52.80%) | loss: 4.018766 | lrm: 0.74 | dt: 1555.56ms | tok/sec: 10,532 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 30 | total time: 70.25m | eta: 63.0m
step 02641/05000 (52.82%) | loss: 4.000083 | lrm: 0.74 | dt: 1565.71ms | tok/sec: 10,464 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 30 | total time: 70.27m | eta: 63.0m
step 02642/05000 (52.84%) | loss: 3.990719 | lrm: 0.74 | dt: 1626.36ms | tok/sec: 10,073 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 30 | total time: 70.30m | eta: 63.0m
step 02643/05000 (52.86%) | loss: 3.999786 | lrm: 0.74 | dt: 1523.40ms | tok/sec: 10,754 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 30 | total time: 70.33m | eta: 63.0m
step 02644/05000 (52.88%) | loss: 3.998385 | lrm: 0.74 | dt: 1496.23ms | tok/sec: 10,950 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 31 | total time: 70.35m | eta: 62.9m
step 02645/05000 (52.90%) | loss: 4.006335 | lrm: 0.74 | dt: 1572.78ms | tok/sec: 10,417 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 31 | total time: 70.38m | eta: 62.9m
step 02646/05000 (52.92%) | loss: 3.997015 | lrm: 0.74 | dt: 1443.70ms | tok/sec: 11,348 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 31 | total time: 70.40m | eta: 62.9m
step 02647/05000 (52.94%) | loss: 3.999478 | lrm: 0.74 | dt: 1481.39ms | tok/sec: 11,059 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 31 | total time: 70.43m | eta: 62.8m
step 02648/05000 (52.96%) | loss: 4.011333 | lrm: 0.74 | dt: 1591.70ms | tok/sec: 10,293 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 31 | total time: 70.45m | eta: 62.8m
step 02649/05000 (52.98%) | loss: 3.998899 | lrm: 0.74 | dt: 1667.27ms | tok/sec: 9,826 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 31 | total time: 70.48m | eta: 62.8m
step 02650/05000 (53.00%) | loss: 3.991889 | lrm: 0.74 | dt: 1655.33ms | tok/sec: 9,897 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 31 | total time: 70.51m | eta: 62.8m
step 02651/05000 (53.02%) | loss: 3.990495 | lrm: 0.74 | dt: 1605.49ms | tok/sec: 10,204 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 31 | total time: 70.53m | eta: 62.7m
step 02652/05000 (53.04%) | loss: 3.993728 | lrm: 0.74 | dt: 1597.94ms | tok/sec: 10,253 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 31 | total time: 70.56m | eta: 62.7m
step 02653/05000 (53.06%) | loss: 3.990351 | lrm: 0.74 | dt: 1626.14ms | tok/sec: 10,075 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 31 | total time: 70.59m | eta: 62.7m
step 02654/05000 (53.08%) | loss: 3.977505 | lrm: 0.74 | dt: 1707.69ms | tok/sec: 9,594 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 31 | total time: 70.62m | eta: 62.7m
step 02655/05000 (53.10%) | loss: 3.979185 | lrm: 0.74 | dt: 1671.55ms | tok/sec: 9,801 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 31 | total time: 70.64m | eta: 62.6m
step 02656/05000 (53.12%) | loss: 3.966548 | lrm: 0.74 | dt: 1563.71ms | tok/sec: 10,477 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 31 | total time: 70.67m | eta: 62.6m
step 02657/05000 (53.14%) | loss: 3.969342 | lrm: 0.73 | dt: 1789.76ms | tok/sec: 9,154 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 31 | total time: 70.70m | eta: 62.6m
step 02658/05000 (53.16%) | loss: 3.974801 | lrm: 0.73 | dt: 1626.60ms | tok/sec: 10,072 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 31 | total time: 70.73m | eta: 62.6m
step 02659/05000 (53.18%) | loss: 3.966482 | lrm: 0.73 | dt: 1655.48ms | tok/sec: 9,896 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 31 | total time: 70.75m | eta: 62.5m
step 02660/05000 (53.20%) | loss: 3.972098 | lrm: 0.73 | dt: 1594.27ms | tok/sec: 10,276 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 31 | total time: 70.78m | eta: 62.5m
step 02661/05000 (53.22%) | loss: 3.980536 | lrm: 0.73 | dt: 1451.08ms | tok/sec: 11,290 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 31 | total time: 70.81m | eta: 62.5m
step 02662/05000 (53.24%) | loss: 3.978745 | lrm: 0.73 | dt: 1628.88ms | tok/sec: 10,058 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 31 | total time: 70.83m | eta: 62.4m
step 02663/05000 (53.26%) | loss: 3.979426 | lrm: 0.73 | dt: 1584.76ms | tok/sec: 10,338 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 31 | total time: 70.86m | eta: 62.4m
step 02664/05000 (53.28%) | loss: 3.974742 | lrm: 0.73 | dt: 1549.21ms | tok/sec: 10,575 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 31 | total time: 70.88m | eta: 62.4m
step 02665/05000 (53.30%) | loss: 3.964326 | lrm: 0.73 | dt: 1821.95ms | tok/sec: 8,992 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 31 | total time: 70.91m | eta: 62.4m
step 02666/05000 (53.32%) | loss: 3.969036 | lrm: 0.73 | dt: 1908.24ms | tok/sec: 8,585 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 31 | total time: 70.95m | eta: 62.3m
step 02667/05000 (53.34%) | loss: 3.975984 | lrm: 0.73 | dt: 1605.69ms | tok/sec: 10,203 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 32 | total time: 70.97m | eta: 62.3m
step 02668/05000 (53.36%) | loss: 3.981917 | lrm: 0.73 | dt: 1659.57ms | tok/sec: 9,872 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 32 | total time: 71.00m | eta: 62.3m
step 02669/05000 (53.38%) | loss: 3.994791 | lrm: 0.73 | dt: 1461.11ms | tok/sec: 11,213 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 32 | total time: 71.03m | eta: 62.3m
step 02670/05000 (53.40%) | loss: 3.996922 | lrm: 0.73 | dt: 1726.51ms | tok/sec: 9,489 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 32 | total time: 71.05m | eta: 62.2m
step 02671/05000 (53.42%) | loss: 3.995240 | lrm: 0.73 | dt: 1554.13ms | tok/sec: 10,542 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 32 | total time: 71.08m | eta: 62.2m
step 02672/05000 (53.44%) | loss: 3.996622 | lrm: 0.73 | dt: 1628.01ms | tok/sec: 10,063 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 32 | total time: 71.11m | eta: 62.2m
step 02673/05000 (53.46%) | loss: 4.010904 | lrm: 0.73 | dt: 1519.37ms | tok/sec: 10,783 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 32 | total time: 71.13m | eta: 62.2m
step 02674/05000 (53.48%) | loss: 4.006623 | lrm: 0.73 | dt: 1458.67ms | tok/sec: 11,232 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 32 | total time: 71.16m | eta: 62.1m
step 02675/05000 (53.50%) | loss: 3.988891 | lrm: 0.73 | dt: 1560.24ms | tok/sec: 10,500 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 32 | total time: 71.18m | eta: 62.1m
step 02676/05000 (53.52%) | loss: 3.980182 | lrm: 0.73 | dt: 1465.13ms | tok/sec: 11,182 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 32 | total time: 71.21m | eta: 62.1m
step 02677/05000 (53.54%) | loss: 3.992730 | lrm: 0.73 | dt: 1537.87ms | tok/sec: 10,653 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 32 | total time: 71.23m | eta: 62.0m
step 02678/05000 (53.56%) | loss: 3.994915 | lrm: 0.73 | dt: 1501.79ms | tok/sec: 10,909 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 32 | total time: 71.26m | eta: 62.0m
step 02679/05000 (53.58%) | loss: 4.001708 | lrm: 0.73 | dt: 1500.08ms | tok/sec: 10,922 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 32 | total time: 71.28m | eta: 62.0m
step 02680/05000 (53.60%) | loss: 3.998296 | lrm: 0.73 | dt: 1482.13ms | tok/sec: 11,054 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 32 | total time: 71.31m | eta: 62.0m
step 02681/05000 (53.62%) | loss: 4.000832 | lrm: 0.73 | dt: 1548.02ms | tok/sec: 10,583 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 32 | total time: 71.33m | eta: 61.9m
step 02682/05000 (53.64%) | loss: 3.986703 | lrm: 0.73 | dt: 1617.09ms | tok/sec: 10,131 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 32 | total time: 71.36m | eta: 61.9m
step 02683/05000 (53.66%) | loss: 3.994995 | lrm: 0.73 | dt: 1770.75ms | tok/sec: 9,252 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 32 | total time: 71.39m | eta: 61.9m
step 02684/05000 (53.68%) | loss: 3.997279 | lrm: 0.73 | dt: 1557.08ms | tok/sec: 10,522 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 32 | total time: 71.42m | eta: 61.9m
step 02685/05000 (53.70%) | loss: 3.996041 | lrm: 0.73 | dt: 1562.22ms | tok/sec: 10,487 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 32 | total time: 71.44m | eta: 61.8m
step 02686/05000 (53.72%) | loss: 3.997896 | lrm: 0.73 | dt: 1583.82ms | tok/sec: 10,344 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 32 | total time: 71.47m | eta: 61.8m
step 02687/05000 (53.74%) | loss: 4.002096 | lrm: 0.73 | dt: 1703.21ms | tok/sec: 9,619 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 32 | total time: 71.50m | eta: 61.8m
step 02688/05000 (53.76%) | loss: 4.021025 | lrm: 0.73 | dt: 1600.32ms | tok/sec: 10,237 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 32 | total time: 71.52m | eta: 61.7m
step 02689/05000 (53.78%) | loss: 4.020253 | lrm: 0.73 | dt: 1629.97ms | tok/sec: 10,051 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 32 | total time: 71.55m | eta: 61.7m
step 02690/05000 (53.80%) | loss: 4.011977 | lrm: 0.73 | dt: 1628.98ms | tok/sec: 10,057 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 32 | total time: 71.58m | eta: 61.7m
step 02691/05000 (53.82%) | loss: 4.015500 | lrm: 0.72 | dt: 1662.58ms | tok/sec: 9,854 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 33 | total time: 71.61m | eta: 61.7m
step 02692/05000 (53.84%) | loss: 4.020899 | lrm: 0.72 | dt: 1554.72ms | tok/sec: 10,538 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 33 | total time: 71.63m | eta: 61.6m
step 02693/05000 (53.86%) | loss: 4.013334 | lrm: 0.72 | dt: 1626.41ms | tok/sec: 10,073 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 33 | total time: 71.66m | eta: 61.6m
step 02694/05000 (53.88%) | loss: 4.010875 | lrm: 0.72 | dt: 1615.94ms | tok/sec: 10,138 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 33 | total time: 71.69m | eta: 61.6m
step 02695/05000 (53.90%) | loss: 4.017449 | lrm: 0.72 | dt: 1585.59ms | tok/sec: 10,333 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 33 | total time: 71.71m | eta: 61.6m
step 02696/05000 (53.92%) | loss: 4.017468 | lrm: 0.72 | dt: 1504.55ms | tok/sec: 10,889 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 33 | total time: 71.74m | eta: 61.5m
step 02697/05000 (53.94%) | loss: 4.021247 | lrm: 0.72 | dt: 1895.60ms | tok/sec: 8,643 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 33 | total time: 71.77m | eta: 61.5m
step 02698/05000 (53.96%) | loss: 4.034964 | lrm: 0.72 | dt: 1610.73ms | tok/sec: 10,171 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 33 | total time: 71.80m | eta: 61.5m
step 02699/05000 (53.98%) | loss: 4.015348 | lrm: 0.72 | dt: 1630.85ms | tok/sec: 10,046 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 33 | total time: 71.82m | eta: 61.5m
Step 02700 | Validation bpb: 1.237713
<|bos|>The capital of France is the capital of France. The capital of France is the capital of France. The
<|bos|>The chemical symbol of gold is the symbol of the gold. The symbol of gold is the symbol of the gold
<|bos|>If yesterday was Friday, then tomorrow will be the first time in the world to be the first of the 20th century
<|bos|>The opposite of hot is the same as the hot is the same as the hot is the same as the
<|bos|>The planets of the solar system are: the solar system, the solar system, the solar system, the solar system,
<|bos|>My favorite color is the red, yellow, and red. I have a red, yellow, and
<|bos|>If 5*x + 3 = 13, then x is 3.2.2.2.2.2.2.2
step 02700/05000 (54.00%) | loss: 4.001722 | lrm: 0.72 | dt: 1546.18ms | tok/sec: 10,596 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 33 | total time: 71.85m | eta: 61.4m
step 02701/05000 (54.02%) | loss: 3.997237 | lrm: 0.72 | dt: 1623.76ms | tok/sec: 10,090 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 33 | total time: 71.88m | eta: 61.4m
step 02702/05000 (54.04%) | loss: 3.985828 | lrm: 0.72 | dt: 1563.82ms | tok/sec: 10,476 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 33 | total time: 71.90m | eta: 61.4m
step 02703/05000 (54.06%) | loss: 3.988258 | lrm: 0.72 | dt: 1488.25ms | tok/sec: 11,008 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 33 | total time: 71.93m | eta: 61.3m
step 02704/05000 (54.08%) | loss: 3.992152 | lrm: 0.72 | dt: 1560.66ms | tok/sec: 10,498 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 33 | total time: 71.95m | eta: 61.3m
step 02705/05000 (54.10%) | loss: 3.989765 | lrm: 0.72 | dt: 1538.05ms | tok/sec: 10,652 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 33 | total time: 71.98m | eta: 61.3m
step 02706/05000 (54.12%) | loss: 3.998866 | lrm: 0.72 | dt: 1517.02ms | tok/sec: 10,800 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 33 | total time: 72.00m | eta: 61.3m
step 02707/05000 (54.14%) | loss: 3.993533 | lrm: 0.72 | dt: 1586.18ms | tok/sec: 10,329 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 33 | total time: 72.03m | eta: 61.2m
step 02708/05000 (54.16%) | loss: 3.982410 | lrm: 0.72 | dt: 1543.40ms | tok/sec: 10,615 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 33 | total time: 72.06m | eta: 61.2m
step 02709/05000 (54.18%) | loss: 3.980995 | lrm: 0.72 | dt: 1551.30ms | tok/sec: 10,561 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 33 | total time: 72.08m | eta: 61.2m
step 02710/05000 (54.20%) | loss: 3.989393 | lrm: 0.72 | dt: 1521.52ms | tok/sec: 10,768 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 33 | total time: 72.11m | eta: 61.2m
step 02711/05000 (54.22%) | loss: 3.995914 | lrm: 0.72 | dt: 1564.72ms | tok/sec: 10,470 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 33 | total time: 72.13m | eta: 61.1m
step 02712/05000 (54.24%) | loss: 4.004854 | lrm: 0.72 | dt: 1517.50ms | tok/sec: 10,796 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 33 | total time: 72.16m | eta: 61.1m
step 02713/05000 (54.26%) | loss: 4.000020 | lrm: 0.72 | dt: 1614.13ms | tok/sec: 10,150 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 33 | total time: 72.18m | eta: 61.1m
step 02714/05000 (54.28%) | loss: 4.007439 | lrm: 0.72 | dt: 1570.52ms | tok/sec: 10,432 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 34 | total time: 72.21m | eta: 61.0m
step 02715/05000 (54.30%) | loss: 4.002192 | lrm: 0.72 | dt: 1550.47ms | tok/sec: 10,567 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 34 | total time: 72.24m | eta: 61.0m
step 02716/05000 (54.32%) | loss: 3.990966 | lrm: 0.72 | dt: 1493.30ms | tok/sec: 10,971 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 34 | total time: 72.26m | eta: 61.0m
step 02717/05000 (54.34%) | loss: 3.993094 | lrm: 0.72 | dt: 1486.60ms | tok/sec: 11,021 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 34 | total time: 72.29m | eta: 61.0m
step 02718/05000 (54.36%) | loss: 4.004673 | lrm: 0.72 | dt: 1500.72ms | tok/sec: 10,917 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 34 | total time: 72.31m | eta: 60.9m
step 02719/05000 (54.38%) | loss: 3.997270 | lrm: 0.72 | dt: 1622.17ms | tok/sec: 10,100 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 34 | total time: 72.34m | eta: 60.9m
step 02720/05000 (54.40%) | loss: 4.009859 | lrm: 0.72 | dt: 1522.98ms | tok/sec: 10,757 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 34 | total time: 72.36m | eta: 60.9m
step 02721/05000 (54.42%) | loss: 4.001604 | lrm: 0.72 | dt: 1553.14ms | tok/sec: 10,548 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 34 | total time: 72.39m | eta: 60.9m
step 02722/05000 (54.44%) | loss: 3.999515 | lrm: 0.72 | dt: 1440.62ms | tok/sec: 11,372 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 34 | total time: 72.41m | eta: 60.8m
step 02723/05000 (54.46%) | loss: 3.970948 | lrm: 0.72 | dt: 1598.33ms | tok/sec: 10,250 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 34 | total time: 72.44m | eta: 60.8m
step 02724/05000 (54.48%) | loss: 3.968043 | lrm: 0.72 | dt: 1667.12ms | tok/sec: 9,827 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 34 | total time: 72.47m | eta: 60.8m
step 02725/05000 (54.50%) | loss: 3.961046 | lrm: 0.71 | dt: 1673.09ms | tok/sec: 9,792 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 34 | total time: 72.50m | eta: 60.7m
step 02726/05000 (54.52%) | loss: 3.956590 | lrm: 0.71 | dt: 1683.99ms | tok/sec: 9,729 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 34 | total time: 72.52m | eta: 60.7m
step 02727/05000 (54.54%) | loss: 3.958564 | lrm: 0.71 | dt: 1606.38ms | tok/sec: 10,199 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 34 | total time: 72.55m | eta: 60.7m
step 02728/05000 (54.56%) | loss: 3.967431 | lrm: 0.71 | dt: 1632.69ms | tok/sec: 10,034 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 34 | total time: 72.58m | eta: 60.7m
step 02729/05000 (54.58%) | loss: 3.962553 | lrm: 0.71 | dt: 1575.90ms | tok/sec: 10,396 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 34 | total time: 72.60m | eta: 60.6m
step 02730/05000 (54.60%) | loss: 3.963594 | lrm: 0.71 | dt: 1662.08ms | tok/sec: 9,857 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 34 | total time: 72.63m | eta: 60.6m
step 02731/05000 (54.62%) | loss: 3.965303 | lrm: 0.71 | dt: 1537.24ms | tok/sec: 10,658 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 34 | total time: 72.66m | eta: 60.6m
step 02732/05000 (54.64%) | loss: 3.978882 | lrm: 0.71 | dt: 1463.14ms | tok/sec: 11,197 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 34 | total time: 72.68m | eta: 60.6m
step 02733/05000 (54.66%) | loss: 3.968839 | lrm: 0.71 | dt: 1529.35ms | tok/sec: 10,713 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 34 | total time: 72.71m | eta: 60.5m
step 02734/05000 (54.68%) | loss: 3.970509 | lrm: 0.71 | dt: 1585.67ms | tok/sec: 10,332 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 34 | total time: 72.73m | eta: 60.5m
step 02735/05000 (54.70%) | loss: 3.970586 | lrm: 0.71 | dt: 1531.92ms | tok/sec: 10,695 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 34 | total time: 72.76m | eta: 60.5m
step 02736/05000 (54.72%) | loss: 3.976345 | lrm: 0.71 | dt: 1576.85ms | tok/sec: 10,390 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 34 | total time: 72.79m | eta: 60.5m
step 02737/05000 (54.74%) | loss: 3.973110 | lrm: 0.71 | dt: 1470.49ms | tok/sec: 11,141 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 35 | total time: 72.81m | eta: 60.4m
step 02738/05000 (54.76%) | loss: 3.987410 | lrm: 0.71 | dt: 1466.69ms | tok/sec: 11,170 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 35 | total time: 72.83m | eta: 60.4m
step 02739/05000 (54.78%) | loss: 3.973688 | lrm: 0.71 | dt: 1436.65ms | tok/sec: 11,404 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 35 | total time: 72.86m | eta: 60.4m
step 02740/05000 (54.80%) | loss: 3.978751 | lrm: 0.71 | dt: 1463.15ms | tok/sec: 11,197 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 35 | total time: 72.88m | eta: 60.3m
step 02741/05000 (54.82%) | loss: 3.979040 | lrm: 0.71 | dt: 1512.18ms | tok/sec: 10,834 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 35 | total time: 72.91m | eta: 60.3m
step 02742/05000 (54.84%) | loss: 3.976371 | lrm: 0.71 | dt: 1521.99ms | tok/sec: 10,764 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 35 | total time: 72.93m | eta: 60.3m
step 02743/05000 (54.86%) | loss: 3.951997 | lrm: 0.71 | dt: 1537.09ms | tok/sec: 10,659 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 35 | total time: 72.96m | eta: 60.3m
step 02744/05000 (54.88%) | loss: 3.960224 | lrm: 0.71 | dt: 1551.29ms | tok/sec: 10,561 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 35 | total time: 72.99m | eta: 60.2m
step 02745/05000 (54.90%) | loss: 3.952051 | lrm: 0.71 | dt: 1517.46ms | tok/sec: 10,796 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 35 | total time: 73.01m | eta: 60.2m
step 02746/05000 (54.92%) | loss: 3.947301 | lrm: 0.71 | dt: 1522.97ms | tok/sec: 10,757 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 35 | total time: 73.04m | eta: 60.2m
step 02747/05000 (54.94%) | loss: 3.940009 | lrm: 0.71 | dt: 1547.20ms | tok/sec: 10,589 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 35 | total time: 73.06m | eta: 60.1m
step 02748/05000 (54.96%) | loss: 3.953648 | lrm: 0.71 | dt: 1619.49ms | tok/sec: 10,116 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 35 | total time: 73.09m | eta: 60.1m
step 02749/05000 (54.98%) | loss: 3.962093 | lrm: 0.71 | dt: 1684.28ms | tok/sec: 9,727 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 35 | total time: 73.12m | eta: 60.1m
step 02750/05000 (55.00%) | loss: 3.964475 | lrm: 0.71 | dt: 1657.46ms | tok/sec: 9,884 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 35 | total time: 73.14m | eta: 60.1m
step 02751/05000 (55.02%) | loss: 3.968024 | lrm: 0.71 | dt: 1535.80ms | tok/sec: 10,668 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 35 | total time: 73.17m | eta: 60.0m
step 02752/05000 (55.04%) | loss: 3.981093 | lrm: 0.71 | dt: 1487.21ms | tok/sec: 11,016 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 35 | total time: 73.19m | eta: 60.0m
step 02753/05000 (55.06%) | loss: 3.974372 | lrm: 0.71 | dt: 1547.85ms | tok/sec: 10,585 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 35 | total time: 73.22m | eta: 60.0m
step 02754/05000 (55.08%) | loss: 3.979074 | lrm: 0.71 | dt: 1463.94ms | tok/sec: 11,191 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 35 | total time: 73.25m | eta: 60.0m
step 02755/05000 (55.10%) | loss: 3.987200 | lrm: 0.71 | dt: 1620.39ms | tok/sec: 10,111 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 35 | total time: 73.27m | eta: 59.9m
step 02756/05000 (55.12%) | loss: 3.983187 | lrm: 0.71 | dt: 1700.80ms | tok/sec: 9,633 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 35 | total time: 73.30m | eta: 59.9m
step 02757/05000 (55.14%) | loss: 3.989257 | lrm: 0.71 | dt: 1550.45ms | tok/sec: 10,567 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 35 | total time: 73.33m | eta: 59.9m
step 02758/05000 (55.16%) | loss: 3.984415 | lrm: 0.71 | dt: 1502.57ms | tok/sec: 10,904 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 35 | total time: 73.35m | eta: 59.8m
step 02759/05000 (55.18%) | loss: 3.976917 | lrm: 0.71 | dt: 1507.63ms | tok/sec: 10,867 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 36 | total time: 73.38m | eta: 59.8m
step 02760/05000 (55.20%) | loss: 3.985034 | lrm: 0.70 | dt: 1552.36ms | tok/sec: 10,554 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 36 | total time: 73.40m | eta: 59.8m
step 02761/05000 (55.22%) | loss: 3.988753 | lrm: 0.70 | dt: 1590.04ms | tok/sec: 10,304 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 36 | total time: 73.43m | eta: 59.8m
step 02762/05000 (55.24%) | loss: 3.973695 | lrm: 0.70 | dt: 1578.85ms | tok/sec: 10,377 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 36 | total time: 73.46m | eta: 59.7m
step 02763/05000 (55.26%) | loss: 3.963752 | lrm: 0.70 | dt: 1569.41ms | tok/sec: 10,439 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 36 | total time: 73.48m | eta: 59.7m
step 02764/05000 (55.28%) | loss: 3.966551 | lrm: 0.70 | dt: 1511.96ms | tok/sec: 10,836 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 36 | total time: 73.51m | eta: 59.7m
step 02765/05000 (55.30%) | loss: 3.955835 | lrm: 0.70 | dt: 1562.23ms | tok/sec: 10,487 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 36 | total time: 73.53m | eta: 59.7m
step 02766/05000 (55.32%) | loss: 3.953552 | lrm: 0.70 | dt: 1574.78ms | tok/sec: 10,404 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 36 | total time: 73.56m | eta: 59.6m
step 02767/05000 (55.34%) | loss: 3.946399 | lrm: 0.70 | dt: 1554.20ms | tok/sec: 10,541 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 36 | total time: 73.58m | eta: 59.6m
step 02768/05000 (55.36%) | loss: 3.933677 | lrm: 0.70 | dt: 1479.69ms | tok/sec: 11,072 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 36 | total time: 73.61m | eta: 59.6m
step 02769/05000 (55.38%) | loss: 3.937410 | lrm: 0.70 | dt: 1547.01ms | tok/sec: 10,590 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 36 | total time: 73.64m | eta: 59.5m
step 02770/05000 (55.40%) | loss: 3.944909 | lrm: 0.70 | dt: 1543.51ms | tok/sec: 10,614 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 36 | total time: 73.66m | eta: 59.5m
step 02771/05000 (55.42%) | loss: 3.936326 | lrm: 0.70 | dt: 1644.04ms | tok/sec: 9,965 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 36 | total time: 73.69m | eta: 59.5m
step 02772/05000 (55.44%) | loss: 3.930397 | lrm: 0.70 | dt: 1662.82ms | tok/sec: 9,853 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 36 | total time: 73.72m | eta: 59.5m
step 02773/05000 (55.46%) | loss: 3.954691 | lrm: 0.70 | dt: 1642.01ms | tok/sec: 9,978 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 36 | total time: 73.74m | eta: 59.4m
step 02774/05000 (55.48%) | loss: 3.961411 | lrm: 0.70 | dt: 1637.59ms | tok/sec: 10,004 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 36 | total time: 73.77m | eta: 59.4m
step 02775/05000 (55.50%) | loss: 3.951585 | lrm: 0.70 | dt: 1629.41ms | tok/sec: 10,055 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 36 | total time: 73.80m | eta: 59.4m
step 02776/05000 (55.52%) | loss: 3.962949 | lrm: 0.70 | dt: 1660.47ms | tok/sec: 9,867 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 36 | total time: 73.83m | eta: 59.4m
step 02777/05000 (55.54%) | loss: 3.962510 | lrm: 0.70 | dt: 1630.42ms | tok/sec: 10,048 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 36 | total time: 73.85m | eta: 59.3m
step 02778/05000 (55.56%) | loss: 3.974020 | lrm: 0.70 | dt: 1693.65ms | tok/sec: 9,673 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 36 | total time: 73.88m | eta: 59.3m
step 02779/05000 (55.58%) | loss: 3.981538 | lrm: 0.70 | dt: 1575.96ms | tok/sec: 10,396 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 36 | total time: 73.91m | eta: 59.3m
step 02780/05000 (55.60%) | loss: 3.988884 | lrm: 0.70 | dt: 1456.69ms | tok/sec: 11,247 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 36 | total time: 73.93m | eta: 59.3m
step 02781/05000 (55.62%) | loss: 3.983891 | lrm: 0.70 | dt: 1498.30ms | tok/sec: 10,935 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 36 | total time: 73.96m | eta: 59.2m
step 02782/05000 (55.64%) | loss: 3.985009 | lrm: 0.70 | dt: 1475.10ms | tok/sec: 11,107 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 36 | total time: 73.98m | eta: 59.2m
step 02783/05000 (55.66%) | loss: 3.971435 | lrm: 0.70 | dt: 1462.00ms | tok/sec: 11,206 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 37 | total time: 74.01m | eta: 59.2m
step 02784/05000 (55.68%) | loss: 3.978092 | lrm: 0.70 | dt: 1485.11ms | tok/sec: 11,032 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 37 | total time: 74.03m | eta: 59.1m
step 02785/05000 (55.70%) | loss: 3.970874 | lrm: 0.70 | dt: 1463.30ms | tok/sec: 11,196 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 37 | total time: 74.05m | eta: 59.1m
step 02786/05000 (55.72%) | loss: 3.971881 | lrm: 0.70 | dt: 1437.57ms | tok/sec: 11,397 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 37 | total time: 74.08m | eta: 59.1m
step 02787/05000 (55.74%) | loss: 3.980867 | lrm: 0.70 | dt: 1478.20ms | tok/sec: 11,083 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 37 | total time: 74.10m | eta: 59.1m
step 02788/05000 (55.76%) | loss: 3.974607 | lrm: 0.70 | dt: 1486.13ms | tok/sec: 11,024 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 37 | total time: 74.13m | eta: 59.0m
step 02789/05000 (55.78%) | loss: 3.983146 | lrm: 0.70 | dt: 1470.65ms | tok/sec: 11,140 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 37 | total time: 74.15m | eta: 59.0m
step 02790/05000 (55.80%) | loss: 3.974962 | lrm: 0.70 | dt: 1486.39ms | tok/sec: 11,022 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 37 | total time: 74.18m | eta: 59.0m
step 02791/05000 (55.82%) | loss: 3.952473 | lrm: 0.70 | dt: 1479.86ms | tok/sec: 11,071 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 37 | total time: 74.20m | eta: 58.9m
step 02792/05000 (55.84%) | loss: 3.945095 | lrm: 0.70 | dt: 1461.57ms | tok/sec: 11,209 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 37 | total time: 74.23m | eta: 58.9m
step 02793/05000 (55.86%) | loss: 3.945264 | lrm: 0.70 | dt: 1458.18ms | tok/sec: 11,235 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 37 | total time: 74.25m | eta: 58.9m
step 02794/05000 (55.88%) | loss: 3.934609 | lrm: 0.69 | dt: 1533.14ms | tok/sec: 10,686 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 37 | total time: 74.28m | eta: 58.9m
step 02795/05000 (55.90%) | loss: 3.918793 | lrm: 0.69 | dt: 1463.90ms | tok/sec: 11,192 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 37 | total time: 74.30m | eta: 58.8m
step 02796/05000 (55.92%) | loss: 3.928554 | lrm: 0.69 | dt: 1542.78ms | tok/sec: 10,619 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 37 | total time: 74.33m | eta: 58.8m
step 02797/05000 (55.94%) | loss: 3.940956 | lrm: 0.69 | dt: 1482.51ms | tok/sec: 11,051 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 37 | total time: 74.35m | eta: 58.8m
step 02798/05000 (55.96%) | loss: 3.938227 | lrm: 0.69 | dt: 1564.24ms | tok/sec: 10,474 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 37 | total time: 74.38m | eta: 58.7m
step 02799/05000 (55.98%) | loss: 3.946449 | lrm: 0.69 | dt: 1472.94ms | tok/sec: 11,123 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 37 | total time: 74.40m | eta: 58.7m
Step 02800 | Validation bpb: 1.233837
<|bos|>The capital of France is a very small, very small, very small, very small, very small,
<|bos|>The chemical symbol of gold is a symbol of the presence of a chemical symbol of the presence of a chemical symbol
<|bos|>If yesterday was Friday, then tomorrow will be the time to get the right amount of money. The first time we had to
<|bos|>The opposite of hot is the same as the hot is the same as the hot is the same as the
<|bos|>The planets of the solar system are: the solar system, the solar system, the solar system, the solar system,
<|bos|>My favorite color is the red, red, and green. I have a red, red, and
<|bos|>If 5*x + 3 = 13, then x is 3.5. If 3 = 13, then x is
step 02800/05000 (56.00%) | loss: 3.947943 | lrm: 0.69 | dt: 1498.32ms | tok/sec: 10,934 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 37 | total time: 74.43m | eta: 58.7m
step 02801/05000 (56.02%) | loss: 3.956522 | lrm: 0.69 | dt: 1449.84ms | tok/sec: 11,300 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 37 | total time: 74.45m | eta: 58.7m
step 02802/05000 (56.04%) | loss: 3.977282 | lrm: 0.69 | dt: 1559.54ms | tok/sec: 10,505 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 37 | total time: 74.48m | eta: 58.6m
step 02803/05000 (56.06%) | loss: 3.981480 | lrm: 0.69 | dt: 1500.08ms | tok/sec: 10,922 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 37 | total time: 74.50m | eta: 58.6m
step 02804/05000 (56.08%) | loss: 3.971648 | lrm: 0.69 | dt: 1609.49ms | tok/sec: 10,179 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 37 | total time: 74.53m | eta: 58.6m
step 02805/05000 (56.10%) | loss: 3.973589 | lrm: 0.69 | dt: 1570.20ms | tok/sec: 10,434 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 37 | total time: 74.55m | eta: 58.5m
step 02806/05000 (56.12%) | loss: 3.957018 | lrm: 0.69 | dt: 1601.34ms | tok/sec: 10,231 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 38 | total time: 74.58m | eta: 58.5m
step 02807/05000 (56.14%) | loss: 3.968263 | lrm: 0.69 | dt: 1567.21ms | tok/sec: 10,454 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 38 | total time: 74.61m | eta: 58.5m
step 02808/05000 (56.16%) | loss: 3.978531 | lrm: 0.69 | dt: 1550.96ms | tok/sec: 10,563 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 38 | total time: 74.63m | eta: 58.5m
step 02809/05000 (56.18%) | loss: 3.991931 | lrm: 0.69 | dt: 1765.08ms | tok/sec: 9,282 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 38 | total time: 74.66m | eta: 58.4m
step 02810/05000 (56.20%) | loss: 3.997664 | lrm: 0.69 | dt: 1665.14ms | tok/sec: 9,839 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 38 | total time: 74.69m | eta: 58.4m
step 02811/05000 (56.22%) | loss: 3.993662 | lrm: 0.69 | dt: 1653.81ms | tok/sec: 9,906 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 38 | total time: 74.72m | eta: 58.4m
step 02812/05000 (56.24%) | loss: 3.979315 | lrm: 0.69 | dt: 1654.08ms | tok/sec: 9,905 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 38 | total time: 74.75m | eta: 58.4m
step 02813/05000 (56.26%) | loss: 3.983316 | lrm: 0.69 | dt: 1718.01ms | tok/sec: 9,536 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 38 | total time: 74.77m | eta: 58.3m
step 02814/05000 (56.28%) | loss: 3.970814 | lrm: 0.69 | dt: 1578.79ms | tok/sec: 10,377 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 38 | total time: 74.80m | eta: 58.3m
step 02815/05000 (56.30%) | loss: 3.962286 | lrm: 0.69 | dt: 1586.10ms | tok/sec: 10,329 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 38 | total time: 74.83m | eta: 58.3m
step 02816/05000 (56.32%) | loss: 3.957858 | lrm: 0.69 | dt: 1554.99ms | tok/sec: 10,536 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 38 | total time: 74.85m | eta: 58.3m
step 02817/05000 (56.34%) | loss: 3.964732 | lrm: 0.69 | dt: 1552.65ms | tok/sec: 10,552 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 38 | total time: 74.88m | eta: 58.2m
step 02818/05000 (56.36%) | loss: 3.968520 | lrm: 0.69 | dt: 1633.42ms | tok/sec: 10,030 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 38 | total time: 74.91m | eta: 58.2m
step 02819/05000 (56.38%) | loss: 3.970046 | lrm: 0.69 | dt: 1446.29ms | tok/sec: 11,328 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 38 | total time: 74.93m | eta: 58.2m
step 02820/05000 (56.40%) | loss: 3.981906 | lrm: 0.69 | dt: 1803.33ms | tok/sec: 9,085 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 38 | total time: 74.96m | eta: 58.2m
step 02821/05000 (56.42%) | loss: 3.960086 | lrm: 0.69 | dt: 1673.08ms | tok/sec: 9,792 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 38 | total time: 74.99m | eta: 58.1m
step 02822/05000 (56.44%) | loss: 3.972423 | lrm: 0.69 | dt: 1668.30ms | tok/sec: 9,820 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 38 | total time: 75.02m | eta: 58.1m
step 02823/05000 (56.46%) | loss: 3.951378 | lrm: 0.69 | dt: 1634.32ms | tok/sec: 10,024 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 38 | total time: 75.04m | eta: 58.1m
step 02824/05000 (56.48%) | loss: 3.933507 | lrm: 0.69 | dt: 1575.11ms | tok/sec: 10,401 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 38 | total time: 75.07m | eta: 58.0m
step 02825/05000 (56.50%) | loss: 3.937672 | lrm: 0.69 | dt: 1491.32ms | tok/sec: 10,986 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 38 | total time: 75.09m | eta: 58.0m
step 02826/05000 (56.52%) | loss: 3.926188 | lrm: 0.69 | dt: 1498.94ms | tok/sec: 10,930 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 38 | total time: 75.12m | eta: 58.0m
step 02827/05000 (56.54%) | loss: 3.902239 | lrm: 0.69 | dt: 1485.58ms | tok/sec: 11,028 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 38 | total time: 75.14m | eta: 58.0m
step 02828/05000 (56.56%) | loss: 3.926403 | lrm: 0.68 | dt: 1565.55ms | tok/sec: 10,465 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 38 | total time: 75.17m | eta: 57.9m
step 02829/05000 (56.58%) | loss: 3.928978 | lrm: 0.68 | dt: 1513.34ms | tok/sec: 10,826 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 38 | total time: 75.20m | eta: 57.9m
step 02830/05000 (56.60%) | loss: 3.927735 | lrm: 0.68 | dt: 1610.82ms | tok/sec: 10,171 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 39 | total time: 75.22m | eta: 57.9m
step 02831/05000 (56.62%) | loss: 3.924153 | lrm: 0.68 | dt: 1532.28ms | tok/sec: 10,692 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 39 | total time: 75.25m | eta: 57.9m
step 02832/05000 (56.64%) | loss: 3.913005 | lrm: 0.68 | dt: 1560.73ms | tok/sec: 10,497 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 39 | total time: 75.27m | eta: 57.8m
step 02833/05000 (56.66%) | loss: 3.912383 | lrm: 0.68 | dt: 1477.68ms | tok/sec: 11,087 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 39 | total time: 75.30m | eta: 57.8m
step 02834/05000 (56.68%) | loss: 3.925168 | lrm: 0.68 | dt: 1581.73ms | tok/sec: 10,358 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 39 | total time: 75.32m | eta: 57.8m
step 02835/05000 (56.70%) | loss: 3.934010 | lrm: 0.68 | dt: 1495.86ms | tok/sec: 10,952 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 39 | total time: 75.35m | eta: 57.7m
step 02836/05000 (56.72%) | loss: 3.930945 | lrm: 0.68 | dt: 1582.49ms | tok/sec: 10,353 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 39 | total time: 75.38m | eta: 57.7m
step 02837/05000 (56.74%) | loss: 3.930491 | lrm: 0.68 | dt: 1589.41ms | tok/sec: 10,308 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 39 | total time: 75.40m | eta: 57.7m
step 02838/05000 (56.76%) | loss: 3.932927 | lrm: 0.68 | dt: 1447.19ms | tok/sec: 11,321 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 39 | total time: 75.43m | eta: 57.7m
step 02839/05000 (56.78%) | loss: 3.946100 | lrm: 0.68 | dt: 1587.44ms | tok/sec: 10,321 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 39 | total time: 75.45m | eta: 57.6m
step 02840/05000 (56.80%) | loss: 3.954481 | lrm: 0.68 | dt: 1466.51ms | tok/sec: 11,172 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 39 | total time: 75.48m | eta: 57.6m
step 02841/05000 (56.82%) | loss: 3.959680 | lrm: 0.68 | dt: 1519.12ms | tok/sec: 10,785 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 39 | total time: 75.50m | eta: 57.6m
step 02842/05000 (56.84%) | loss: 3.954854 | lrm: 0.68 | dt: 1587.98ms | tok/sec: 10,317 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 39 | total time: 75.53m | eta: 57.6m
step 02843/05000 (56.86%) | loss: 3.972116 | lrm: 0.68 | dt: 1482.18ms | tok/sec: 11,053 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 39 | total time: 75.55m | eta: 57.5m
step 02844/05000 (56.88%) | loss: 3.963485 | lrm: 0.68 | dt: 1464.93ms | tok/sec: 11,184 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 39 | total time: 75.58m | eta: 57.5m
step 02845/05000 (56.90%) | loss: 3.961805 | lrm: 0.68 | dt: 1502.63ms | tok/sec: 10,903 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 39 | total time: 75.60m | eta: 57.5m
step 02846/05000 (56.92%) | loss: 3.971805 | lrm: 0.68 | dt: 1468.10ms | tok/sec: 11,159 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 39 | total time: 75.63m | eta: 57.4m
step 02847/05000 (56.94%) | loss: 3.965950 | lrm: 0.68 | dt: 1486.21ms | tok/sec: 11,024 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 39 | total time: 75.65m | eta: 57.4m
step 02848/05000 (56.96%) | loss: 3.969895 | lrm: 0.68 | dt: 1437.12ms | tok/sec: 11,400 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 39 | total time: 75.68m | eta: 57.4m
step 02849/05000 (56.98%) | loss: 3.948116 | lrm: 0.68 | dt: 1472.57ms | tok/sec: 11,126 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 39 | total time: 75.70m | eta: 57.4m
step 02850/05000 (57.00%) | loss: 3.945591 | lrm: 0.68 | dt: 1526.60ms | tok/sec: 10,732 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 39 | total time: 75.73m | eta: 57.3m
step 02851/05000 (57.02%) | loss: 3.943156 | lrm: 0.68 | dt: 1453.24ms | tok/sec: 11,274 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 39 | total time: 75.75m | eta: 57.3m
step 02852/05000 (57.04%) | loss: 3.948850 | lrm: 0.68 | dt: 1487.55ms | tok/sec: 11,014 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 40 | total time: 75.78m | eta: 57.3m
step 02853/05000 (57.06%) | loss: 3.945176 | lrm: 0.68 | dt: 1435.42ms | tok/sec: 11,414 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 40 | total time: 75.80m | eta: 57.2m
step 02854/05000 (57.08%) | loss: 3.955236 | lrm: 0.68 | dt: 1495.70ms | tok/sec: 10,954 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 40 | total time: 75.82m | eta: 57.2m
step 02855/05000 (57.10%) | loss: 3.957106 | lrm: 0.68 | dt: 1451.02ms | tok/sec: 11,291 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 40 | total time: 75.85m | eta: 57.2m
step 02856/05000 (57.12%) | loss: 3.952143 | lrm: 0.68 | dt: 1462.07ms | tok/sec: 11,206 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 40 | total time: 75.87m | eta: 57.2m
step 02857/05000 (57.14%) | loss: 3.954322 | lrm: 0.68 | dt: 1457.75ms | tok/sec: 11,239 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 40 | total time: 75.90m | eta: 57.1m
step 02858/05000 (57.16%) | loss: 3.953044 | lrm: 0.68 | dt: 1469.76ms | tok/sec: 11,147 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 40 | total time: 75.92m | eta: 57.1m
step 02859/05000 (57.18%) | loss: 3.955232 | lrm: 0.68 | dt: 1456.03ms | tok/sec: 11,252 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 40 | total time: 75.95m | eta: 57.1m
step 02860/05000 (57.20%) | loss: 3.939462 | lrm: 0.68 | dt: 1460.10ms | tok/sec: 11,221 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 40 | total time: 75.97m | eta: 57.0m
step 02861/05000 (57.22%) | loss: 3.939563 | lrm: 0.68 | dt: 1497.70ms | tok/sec: 10,939 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 40 | total time: 76.00m | eta: 57.0m
step 02862/05000 (57.24%) | loss: 3.954547 | lrm: 0.67 | dt: 1464.64ms | tok/sec: 11,186 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 40 | total time: 76.02m | eta: 57.0m
step 02863/05000 (57.26%) | loss: 3.953478 | lrm: 0.67 | dt: 1552.60ms | tok/sec: 10,552 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 40 | total time: 76.05m | eta: 57.0m
step 02864/05000 (57.28%) | loss: 3.947507 | lrm: 0.67 | dt: 1515.67ms | tok/sec: 10,809 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 40 | total time: 76.07m | eta: 56.9m
step 02865/05000 (57.30%) | loss: 3.953803 | lrm: 0.67 | dt: 1472.44ms | tok/sec: 11,127 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 40 | total time: 76.10m | eta: 56.9m
step 02866/05000 (57.32%) | loss: 3.970147 | lrm: 0.67 | dt: 1525.50ms | tok/sec: 10,740 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 40 | total time: 76.12m | eta: 56.9m
step 02867/05000 (57.34%) | loss: 3.953216 | lrm: 0.67 | dt: 1531.54ms | tok/sec: 10,697 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 40 | total time: 76.15m | eta: 56.8m
step 02868/05000 (57.36%) | loss: 3.943073 | lrm: 0.67 | dt: 1561.52ms | tok/sec: 10,492 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 40 | total time: 76.17m | eta: 56.8m
step 02869/05000 (57.38%) | loss: 3.937066 | lrm: 0.67 | dt: 1482.94ms | tok/sec: 11,048 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 40 | total time: 76.20m | eta: 56.8m
step 02870/05000 (57.40%) | loss: 3.944260 | lrm: 0.67 | dt: 1628.31ms | tok/sec: 10,061 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 40 | total time: 76.22m | eta: 56.8m
step 02871/05000 (57.42%) | loss: 3.951234 | lrm: 0.67 | dt: 1410.54ms | tok/sec: 11,615 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 40 | total time: 76.25m | eta: 56.7m
step 02872/05000 (57.44%) | loss: 3.959838 | lrm: 0.67 | dt: 1545.91ms | tok/sec: 10,598 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 40 | total time: 76.27m | eta: 56.7m
step 02873/05000 (57.46%) | loss: 3.966764 | lrm: 0.67 | dt: 1449.98ms | tok/sec: 11,299 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 40 | total time: 76.30m | eta: 56.7m
step 02874/05000 (57.48%) | loss: 3.974845 | lrm: 0.67 | dt: 1514.03ms | tok/sec: 10,821 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 40 | total time: 76.32m | eta: 56.7m
step 02875/05000 (57.50%) | loss: 3.966294 | lrm: 0.67 | dt: 1562.97ms | tok/sec: 10,482 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 41 | total time: 76.35m | eta: 56.6m
step 02876/05000 (57.52%) | loss: 3.962244 | lrm: 0.67 | dt: 1499.76ms | tok/sec: 10,924 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 41 | total time: 76.37m | eta: 56.6m
step 02877/05000 (57.54%) | loss: 3.969130 | lrm: 0.67 | dt: 1508.64ms | tok/sec: 10,860 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 41 | total time: 76.40m | eta: 56.6m
step 02878/05000 (57.56%) | loss: 3.960244 | lrm: 0.67 | dt: 1523.81ms | tok/sec: 10,751 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 41 | total time: 76.42m | eta: 56.5m
step 02879/05000 (57.58%) | loss: 3.961980 | lrm: 0.67 | dt: 1456.72ms | tok/sec: 11,247 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 41 | total time: 76.45m | eta: 56.5m
step 02880/05000 (57.60%) | loss: 3.971118 | lrm: 0.67 | dt: 1508.37ms | tok/sec: 10,862 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 41 | total time: 76.47m | eta: 56.5m
step 02881/05000 (57.62%) | loss: 3.968376 | lrm: 0.67 | dt: 1450.90ms | tok/sec: 11,292 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 41 | total time: 76.50m | eta: 56.5m
step 02882/05000 (57.64%) | loss: 3.990801 | lrm: 0.67 | dt: 1469.36ms | tok/sec: 11,150 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 41 | total time: 76.52m | eta: 56.4m
step 02883/05000 (57.66%) | loss: 3.987678 | lrm: 0.67 | dt: 1474.21ms | tok/sec: 11,113 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 41 | total time: 76.55m | eta: 56.4m
step 02884/05000 (57.68%) | loss: 3.981653 | lrm: 0.67 | dt: 1426.65ms | tok/sec: 11,484 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 41 | total time: 76.57m | eta: 56.4m
step 02885/05000 (57.70%) | loss: 3.973781 | lrm: 0.67 | dt: 1476.40ms | tok/sec: 11,097 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 41 | total time: 76.60m | eta: 56.3m
step 02886/05000 (57.72%) | loss: 3.965957 | lrm: 0.67 | dt: 1485.51ms | tok/sec: 11,029 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 41 | total time: 76.62m | eta: 56.3m
step 02887/05000 (57.74%) | loss: 3.958147 | lrm: 0.67 | dt: 1500.28ms | tok/sec: 10,920 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 41 | total time: 76.65m | eta: 56.3m
step 02888/05000 (57.76%) | loss: 3.964777 | lrm: 0.67 | dt: 1457.40ms | tok/sec: 11,241 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 41 | total time: 76.67m | eta: 56.3m
step 02889/05000 (57.78%) | loss: 3.950648 | lrm: 0.67 | dt: 1468.32ms | tok/sec: 11,158 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 41 | total time: 76.69m | eta: 56.2m
step 02890/05000 (57.80%) | loss: 3.945655 | lrm: 0.67 | dt: 1447.12ms | tok/sec: 11,321 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 41 | total time: 76.72m | eta: 56.2m
step 02891/05000 (57.82%) | loss: 3.945485 | lrm: 0.67 | dt: 1488.03ms | tok/sec: 11,010 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 41 | total time: 76.74m | eta: 56.2m
step 02892/05000 (57.84%) | loss: 3.946313 | lrm: 0.67 | dt: 1652.10ms | tok/sec: 9,917 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 41 | total time: 76.77m | eta: 56.2m
step 02893/05000 (57.86%) | loss: 3.953562 | lrm: 0.67 | dt: 1615.55ms | tok/sec: 10,141 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 41 | total time: 76.80m | eta: 56.1m
step 02894/05000 (57.88%) | loss: 3.951969 | lrm: 0.67 | dt: 1521.48ms | tok/sec: 10,768 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 41 | total time: 76.82m | eta: 56.1m
step 02895/05000 (57.90%) | loss: 3.945953 | lrm: 0.67 | dt: 1482.94ms | tok/sec: 11,048 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 41 | total time: 76.85m | eta: 56.1m
step 02896/05000 (57.92%) | loss: 3.952815 | lrm: 0.67 | dt: 1557.77ms | tok/sec: 10,517 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 41 | total time: 76.87m | eta: 56.0m
step 02897/05000 (57.94%) | loss: 3.965280 | lrm: 0.66 | dt: 1530.76ms | tok/sec: 10,703 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 41 | total time: 76.90m | eta: 56.0m
step 02898/05000 (57.96%) | loss: 3.986229 | lrm: 0.66 | dt: 1625.06ms | tok/sec: 10,082 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 42 | total time: 76.93m | eta: 56.0m
step 02899/05000 (57.98%) | loss: 3.977681 | lrm: 0.66 | dt: 1690.15ms | tok/sec: 9,693 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 42 | total time: 76.95m | eta: 56.0m
Step 02900 | Validation bpb: 1.228945
<|bos|>The capital of France is a vast, diverse, and diverse group of people, each with its own unique
<|bos|>The chemical symbol of gold is the gold, which is a gold, and is a gold, which is a
<|bos|>If yesterday was Friday, then tomorrow will be the first time we've been in the midst of the pandemic. The first time
<|bos|>The opposite of hot is the same as the hot is the same as the hot is the same as the
<|bos|>The planets of the solar system are: the solar system, the solar system, the solar system, the solar system,
<|bos|>My favorite color is the red, blue, and blue. I have a lot of blue, blue
<|bos|>If 5*x + 3 = 13, then x is 3.5.5.5.5.5.5.5
step 02900/05000 (58.00%) | loss: 3.969992 | lrm: 0.66 | dt: 1525.54ms | tok/sec: 10,739 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 42 | total time: 76.98m | eta: 55.9m
step 02901/05000 (58.02%) | loss: 3.956534 | lrm: 0.66 | dt: 1577.17ms | tok/sec: 10,388 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 42 | total time: 77.01m | eta: 55.9m
step 02902/05000 (58.04%) | loss: 3.967204 | lrm: 0.66 | dt: 1446.58ms | tok/sec: 11,326 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 42 | total time: 77.03m | eta: 55.9m
step 02903/05000 (58.06%) | loss: 3.935808 | lrm: 0.66 | dt: 1585.91ms | tok/sec: 10,330 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 42 | total time: 77.06m | eta: 55.9m
step 02904/05000 (58.08%) | loss: 3.930955 | lrm: 0.66 | dt: 1572.26ms | tok/sec: 10,420 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 42 | total time: 77.08m | eta: 55.8m
step 02905/05000 (58.10%) | loss: 3.947737 | lrm: 0.66 | dt: 1653.26ms | tok/sec: 9,910 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 42 | total time: 77.11m | eta: 55.8m
step 02906/05000 (58.12%) | loss: 3.937901 | lrm: 0.66 | dt: 1623.25ms | tok/sec: 10,093 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 42 | total time: 77.14m | eta: 55.8m
step 02907/05000 (58.14%) | loss: 3.946751 | lrm: 0.66 | dt: 1541.73ms | tok/sec: 10,627 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 42 | total time: 77.16m | eta: 55.7m
step 02908/05000 (58.16%) | loss: 3.944171 | lrm: 0.66 | dt: 1549.30ms | tok/sec: 10,575 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 42 | total time: 77.19m | eta: 55.7m
step 02909/05000 (58.18%) | loss: 3.926888 | lrm: 0.66 | dt: 1480.63ms | tok/sec: 11,065 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 42 | total time: 77.21m | eta: 55.7m
step 02910/05000 (58.20%) | loss: 3.925292 | lrm: 0.66 | dt: 1487.45ms | tok/sec: 11,014 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 42 | total time: 77.24m | eta: 55.7m
step 02911/05000 (58.22%) | loss: 3.945233 | lrm: 0.66 | dt: 1460.13ms | tok/sec: 11,220 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 42 | total time: 77.26m | eta: 55.6m
step 02912/05000 (58.24%) | loss: 3.938948 | lrm: 0.66 | dt: 1497.38ms | tok/sec: 10,941 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 42 | total time: 77.29m | eta: 55.6m
step 02913/05000 (58.26%) | loss: 3.939510 | lrm: 0.66 | dt: 1439.82ms | tok/sec: 11,379 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 42 | total time: 77.31m | eta: 55.6m
step 02914/05000 (58.28%) | loss: 3.956779 | lrm: 0.66 | dt: 1492.52ms | tok/sec: 10,977 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 42 | total time: 77.34m | eta: 55.6m
step 02915/05000 (58.30%) | loss: 3.961398 | lrm: 0.66 | dt: 1441.25ms | tok/sec: 11,367 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 42 | total time: 77.36m | eta: 55.5m
step 02916/05000 (58.32%) | loss: 3.957119 | lrm: 0.66 | dt: 1513.10ms | tok/sec: 10,828 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 42 | total time: 77.39m | eta: 55.5m
step 02917/05000 (58.34%) | loss: 3.966529 | lrm: 0.66 | dt: 1456.02ms | tok/sec: 11,252 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 42 | total time: 77.41m | eta: 55.5m
step 02918/05000 (58.36%) | loss: 3.957096 | lrm: 0.66 | dt: 1470.59ms | tok/sec: 11,141 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 42 | total time: 77.43m | eta: 55.4m
step 02919/05000 (58.38%) | loss: 3.957211 | lrm: 0.66 | dt: 1444.06ms | tok/sec: 11,345 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 42 | total time: 77.46m | eta: 55.4m
step 02920/05000 (58.40%) | loss: 3.955500 | lrm: 0.66 | dt: 1474.61ms | tok/sec: 11,110 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 42 | total time: 77.48m | eta: 55.4m
step 02921/05000 (58.42%) | loss: 3.951091 | lrm: 0.66 | dt: 1468.49ms | tok/sec: 11,157 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 43 | total time: 77.51m | eta: 55.4m
step 02922/05000 (58.44%) | loss: 3.949681 | lrm: 0.66 | dt: 1451.54ms | tok/sec: 11,287 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 43 | total time: 77.53m | eta: 55.3m
step 02923/05000 (58.46%) | loss: 3.958107 | lrm: 0.66 | dt: 1454.94ms | tok/sec: 11,260 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 43 | total time: 77.56m | eta: 55.3m
step 02924/05000 (58.48%) | loss: 3.957694 | lrm: 0.66 | dt: 1442.72ms | tok/sec: 11,356 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 43 | total time: 77.58m | eta: 55.3m
step 02925/05000 (58.50%) | loss: 3.973706 | lrm: 0.66 | dt: 1442.38ms | tok/sec: 11,359 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 43 | total time: 77.60m | eta: 55.2m
step 02926/05000 (58.52%) | loss: 3.981262 | lrm: 0.66 | dt: 1477.43ms | tok/sec: 11,089 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 43 | total time: 77.63m | eta: 55.2m
step 02927/05000 (58.54%) | loss: 3.967955 | lrm: 0.66 | dt: 1524.65ms | tok/sec: 10,746 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 43 | total time: 77.65m | eta: 55.2m
step 02928/05000 (58.56%) | loss: 3.971406 | lrm: 0.66 | dt: 1484.65ms | tok/sec: 11,035 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 43 | total time: 77.68m | eta: 55.2m
step 02929/05000 (58.58%) | loss: 3.970232 | lrm: 0.66 | dt: 1571.77ms | tok/sec: 10,423 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 43 | total time: 77.70m | eta: 55.1m
step 02930/05000 (58.60%) | loss: 3.967527 | lrm: 0.66 | dt: 1477.50ms | tok/sec: 11,089 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 43 | total time: 77.73m | eta: 55.1m
step 02931/05000 (58.62%) | loss: 3.962825 | lrm: 0.65 | dt: 1475.40ms | tok/sec: 11,104 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 43 | total time: 77.75m | eta: 55.1m
step 02932/05000 (58.64%) | loss: 3.963119 | lrm: 0.65 | dt: 1681.62ms | tok/sec: 9,742 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 43 | total time: 77.78m | eta: 55.0m
step 02933/05000 (58.66%) | loss: 3.956032 | lrm: 0.65 | dt: 1594.07ms | tok/sec: 10,278 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 43 | total time: 77.81m | eta: 55.0m
step 02934/05000 (58.68%) | loss: 3.949337 | lrm: 0.65 | dt: 1616.04ms | tok/sec: 10,138 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 43 | total time: 77.84m | eta: 55.0m
step 02935/05000 (58.70%) | loss: 3.960301 | lrm: 0.65 | dt: 1586.15ms | tok/sec: 10,329 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 43 | total time: 77.86m | eta: 55.0m
step 02936/05000 (58.72%) | loss: 3.959628 | lrm: 0.65 | dt: 1602.43ms | tok/sec: 10,224 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 43 | total time: 77.89m | eta: 54.9m
step 02937/05000 (58.74%) | loss: 3.967577 | lrm: 0.65 | dt: 1634.19ms | tok/sec: 10,025 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 43 | total time: 77.92m | eta: 54.9m
step 02938/05000 (58.76%) | loss: 3.967762 | lrm: 0.65 | dt: 1480.15ms | tok/sec: 11,069 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 43 | total time: 77.94m | eta: 54.9m
step 02939/05000 (58.78%) | loss: 3.972818 | lrm: 0.65 | dt: 1690.63ms | tok/sec: 9,691 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 43 | total time: 77.97m | eta: 54.9m
step 02940/05000 (58.80%) | loss: 3.977317 | lrm: 0.65 | dt: 1662.93ms | tok/sec: 9,852 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 43 | total time: 78.00m | eta: 54.8m
step 02941/05000 (58.82%) | loss: 3.968327 | lrm: 0.65 | dt: 1627.53ms | tok/sec: 10,066 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 43 | total time: 78.02m | eta: 54.8m
step 02942/05000 (58.84%) | loss: 3.963583 | lrm: 0.65 | dt: 1701.52ms | tok/sec: 9,629 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 43 | total time: 78.05m | eta: 54.8m
step 02943/05000 (58.86%) | loss: 3.971109 | lrm: 0.65 | dt: 1725.21ms | tok/sec: 9,496 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 43 | total time: 78.08m | eta: 54.8m
step 02944/05000 (58.88%) | loss: 3.972455 | lrm: 0.65 | dt: 1647.34ms | tok/sec: 9,945 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 43 | total time: 78.11m | eta: 54.7m
step 02945/05000 (58.90%) | loss: 3.949537 | lrm: 0.65 | dt: 1664.99ms | tok/sec: 9,840 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 44 | total time: 78.14m | eta: 54.7m
step 02946/05000 (58.92%) | loss: 3.955738 | lrm: 0.65 | dt: 1736.36ms | tok/sec: 9,435 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 44 | total time: 78.17m | eta: 54.7m
step 02947/05000 (58.94%) | loss: 3.950379 | lrm: 0.65 | dt: 1711.59ms | tok/sec: 9,572 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 44 | total time: 78.19m | eta: 54.7m
step 02948/05000 (58.96%) | loss: 3.931877 | lrm: 0.65 | dt: 1538.57ms | tok/sec: 10,648 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 44 | total time: 78.22m | eta: 54.6m
step 02949/05000 (58.98%) | loss: 3.940337 | lrm: 0.65 | dt: 1593.68ms | tok/sec: 10,280 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 44 | total time: 78.25m | eta: 54.6m
step 02950/05000 (59.00%) | loss: 3.937197 | lrm: 0.65 | dt: 1631.88ms | tok/sec: 10,039 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 44 | total time: 78.27m | eta: 54.6m
step 02951/05000 (59.02%) | loss: 3.928960 | lrm: 0.65 | dt: 1830.63ms | tok/sec: 8,949 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 44 | total time: 78.30m | eta: 54.6m
step 02952/05000 (59.04%) | loss: 3.932574 | lrm: 0.65 | dt: 1764.42ms | tok/sec: 9,285 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 44 | total time: 78.33m | eta: 54.5m
step 02953/05000 (59.06%) | loss: 3.943307 | lrm: 0.65 | dt: 1836.33ms | tok/sec: 8,922 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 44 | total time: 78.36m | eta: 54.5m
step 02954/05000 (59.08%) | loss: 3.952984 | lrm: 0.65 | dt: 1658.51ms | tok/sec: 9,878 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 44 | total time: 78.39m | eta: 54.5m
step 02955/05000 (59.10%) | loss: 3.954305 | lrm: 0.65 | dt: 1622.58ms | tok/sec: 10,097 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 44 | total time: 78.42m | eta: 54.5m
step 02956/05000 (59.12%) | loss: 3.952191 | lrm: 0.65 | dt: 1648.21ms | tok/sec: 9,940 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 44 | total time: 78.45m | eta: 54.4m
step 02957/05000 (59.14%) | loss: 3.952157 | lrm: 0.65 | dt: 1641.06ms | tok/sec: 9,983 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 44 | total time: 78.47m | eta: 54.4m
step 02958/05000 (59.16%) | loss: 3.955659 | lrm: 0.65 | dt: 1564.20ms | tok/sec: 10,474 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 44 | total time: 78.50m | eta: 54.4m
step 02959/05000 (59.18%) | loss: 3.944157 | lrm: 0.65 | dt: 1561.37ms | tok/sec: 10,493 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 44 | total time: 78.53m | eta: 54.3m
step 02960/05000 (59.20%) | loss: 3.951352 | lrm: 0.65 | dt: 1595.23ms | tok/sec: 10,270 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 44 | total time: 78.55m | eta: 54.3m
step 02961/05000 (59.22%) | loss: 3.966233 | lrm: 0.65 | dt: 1579.02ms | tok/sec: 10,376 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 44 | total time: 78.58m | eta: 54.3m
step 02962/05000 (59.24%) | loss: 3.965164 | lrm: 0.65 | dt: 1557.49ms | tok/sec: 10,519 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 44 | total time: 78.60m | eta: 54.3m
step 02963/05000 (59.26%) | loss: 3.979923 | lrm: 0.65 | dt: 1564.89ms | tok/sec: 10,469 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 44 | total time: 78.63m | eta: 54.2m
step 02964/05000 (59.28%) | loss: 3.980483 | lrm: 0.65 | dt: 1586.97ms | tok/sec: 10,324 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 44 | total time: 78.66m | eta: 54.2m
step 02965/05000 (59.30%) | loss: 3.963476 | lrm: 0.64 | dt: 1412.23ms | tok/sec: 11,601 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 44 | total time: 78.68m | eta: 54.2m
step 02966/05000 (59.32%) | loss: 3.953925 | lrm: 0.64 | dt: 1592.42ms | tok/sec: 10,288 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 44 | total time: 78.71m | eta: 54.2m
step 02967/05000 (59.34%) | loss: 3.952627 | lrm: 0.64 | dt: 1499.99ms | tok/sec: 10,922 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 44 | total time: 78.73m | eta: 54.1m
step 02968/05000 (59.36%) | loss: 3.946560 | lrm: 0.64 | dt: 1448.91ms | tok/sec: 11,307 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 45 | total time: 78.76m | eta: 54.1m
step 02969/05000 (59.38%) | loss: 3.944561 | lrm: 0.64 | dt: 1480.14ms | tok/sec: 11,069 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 45 | total time: 78.78m | eta: 54.1m
step 02970/05000 (59.40%) | loss: 3.950140 | lrm: 0.64 | dt: 1455.80ms | tok/sec: 11,254 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 45 | total time: 78.80m | eta: 54.0m
step 02971/05000 (59.42%) | loss: 3.950256 | lrm: 0.64 | dt: 1426.19ms | tok/sec: 11,487 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 45 | total time: 78.83m | eta: 54.0m
step 02972/05000 (59.44%) | loss: 3.957624 | lrm: 0.64 | dt: 1512.11ms | tok/sec: 10,835 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 45 | total time: 78.85m | eta: 54.0m
step 02973/05000 (59.46%) | loss: 3.947425 | lrm: 0.64 | dt: 1436.76ms | tok/sec: 11,403 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 45 | total time: 78.88m | eta: 54.0m
step 02974/05000 (59.48%) | loss: 3.934921 | lrm: 0.64 | dt: 1482.52ms | tok/sec: 11,051 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 45 | total time: 78.90m | eta: 53.9m
step 02975/05000 (59.50%) | loss: 3.941854 | lrm: 0.64 | dt: 1465.82ms | tok/sec: 11,177 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 45 | total time: 78.93m | eta: 53.9m
step 02976/05000 (59.52%) | loss: 3.951670 | lrm: 0.64 | dt: 1491.67ms | tok/sec: 10,983 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 45 | total time: 78.95m | eta: 53.9m
step 02977/05000 (59.54%) | loss: 3.957523 | lrm: 0.64 | dt: 1448.77ms | tok/sec: 11,308 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 45 | total time: 78.98m | eta: 53.8m
step 02978/05000 (59.56%) | loss: 3.961275 | lrm: 0.64 | dt: 1458.11ms | tok/sec: 11,236 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 45 | total time: 79.00m | eta: 53.8m
step 02979/05000 (59.58%) | loss: 3.951172 | lrm: 0.64 | dt: 1558.12ms | tok/sec: 10,515 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 45 | total time: 79.03m | eta: 53.8m
step 02980/05000 (59.60%) | loss: 3.941388 | lrm: 0.64 | dt: 1458.99ms | tok/sec: 11,229 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 45 | total time: 79.05m | eta: 53.8m
step 02981/05000 (59.62%) | loss: 3.948568 | lrm: 0.64 | dt: 1465.77ms | tok/sec: 11,177 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 45 | total time: 79.07m | eta: 53.7m
step 02982/05000 (59.64%) | loss: 3.949246 | lrm: 0.64 | dt: 1439.73ms | tok/sec: 11,379 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 45 | total time: 79.10m | eta: 53.7m
step 02983/05000 (59.66%) | loss: 3.955031 | lrm: 0.64 | dt: 1470.04ms | tok/sec: 11,145 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 45 | total time: 79.12m | eta: 53.7m
step 02984/05000 (59.68%) | loss: 3.962112 | lrm: 0.64 | dt: 1484.64ms | tok/sec: 11,035 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 45 | total time: 79.15m | eta: 53.7m
step 02985/05000 (59.70%) | loss: 3.951762 | lrm: 0.64 | dt: 1545.41ms | tok/sec: 10,601 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 45 | total time: 79.17m | eta: 53.6m
step 02986/05000 (59.72%) | loss: 3.938999 | lrm: 0.64 | dt: 1666.77ms | tok/sec: 9,829 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 45 | total time: 79.20m | eta: 53.6m
step 02987/05000 (59.74%) | loss: 3.946434 | lrm: 0.64 | dt: 1548.36ms | tok/sec: 10,581 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 45 | total time: 79.23m | eta: 53.6m
step 02988/05000 (59.76%) | loss: 3.951503 | lrm: 0.64 | dt: 1581.58ms | tok/sec: 10,359 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 45 | total time: 79.25m | eta: 53.5m
step 02989/05000 (59.78%) | loss: 3.956549 | lrm: 0.64 | dt: 1542.62ms | tok/sec: 10,620 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 45 | total time: 79.28m | eta: 53.5m
step 02990/05000 (59.80%) | loss: 3.958944 | lrm: 0.64 | dt: 1562.56ms | tok/sec: 10,485 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 45 | total time: 79.31m | eta: 53.5m
step 02991/05000 (59.82%) | loss: 3.947117 | lrm: 0.64 | dt: 1542.16ms | tok/sec: 10,624 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 46 | total time: 79.33m | eta: 53.5m
step 02992/05000 (59.84%) | loss: 3.944035 | lrm: 0.64 | dt: 1565.92ms | tok/sec: 10,462 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 46 | total time: 79.36m | eta: 53.4m
step 02993/05000 (59.86%) | loss: 3.958255 | lrm: 0.64 | dt: 1608.66ms | tok/sec: 10,184 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 46 | total time: 79.38m | eta: 53.4m
step 02994/05000 (59.88%) | loss: 3.940805 | lrm: 0.64 | dt: 1570.10ms | tok/sec: 10,434 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 46 | total time: 79.41m | eta: 53.4m
step 02995/05000 (59.90%) | loss: 3.954777 | lrm: 0.64 | dt: 1651.15ms | tok/sec: 9,922 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 46 | total time: 79.44m | eta: 53.4m
step 02996/05000 (59.92%) | loss: 3.956853 | lrm: 0.64 | dt: 1609.40ms | tok/sec: 10,180 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 46 | total time: 79.46m | eta: 53.3m
step 02997/05000 (59.94%) | loss: 3.959821 | lrm: 0.64 | dt: 1543.72ms | tok/sec: 10,613 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 46 | total time: 79.49m | eta: 53.3m
step 02998/05000 (59.96%) | loss: 3.967983 | lrm: 0.64 | dt: 1553.12ms | tok/sec: 10,549 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 46 | total time: 79.52m | eta: 53.3m
step 02999/05000 (59.98%) | loss: 3.974882 | lrm: 0.63 | dt: 1587.24ms | tok/sec: 10,322 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 46 | total time: 79.54m | eta: 53.3m
Step 03000 | Validation bpb: 1.224042
<|bos|>The capital of France is the capital of France. The capital of France is the capital of France. The
<|bos|>The chemical symbol of gold is 1.5. 1. 1. 1. 2
<|bos|>If yesterday was Friday, then tomorrow will be Friday, Friday, Friday, Friday, Friday, Friday, Friday, Friday,
<|bos|>The opposite of hot is that the heat of the sun is the same as the sun. The heat of
<|bos|>The planets of the solar system are: 1. The solar system is 2. The solar system is 3
<|bos|>My favorite color is the red, blue, and blue. I have a lot of blue and I
<|bos|>If 5*x + 3 = 13, then x is 1.5.5.5.5.5.5.5
step 03000/05000 (60.00%) | loss: 3.972503 | lrm: 0.63 | dt: 1525.32ms | tok/sec: 10,741 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 46 | total time: 79.57m | eta: 53.2m
step 03001/05000 (60.02%) | loss: 3.969963 | lrm: 0.63 | dt: 1442.49ms | tok/sec: 11,358 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 46 | total time: 79.59m | eta: 53.2m
step 03002/05000 (60.04%) | loss: 3.956407 | lrm: 0.63 | dt: 1467.91ms | tok/sec: 11,161 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 46 | total time: 79.62m | eta: 53.2m
step 03003/05000 (60.06%) | loss: 3.939168 | lrm: 0.63 | dt: 1548.23ms | tok/sec: 10,582 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 46 | total time: 79.64m | eta: 53.1m
step 03004/05000 (60.08%) | loss: 3.942573 | lrm: 0.63 | dt: 1609.64ms | tok/sec: 10,178 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 46 | total time: 79.67m | eta: 53.1m
step 03005/05000 (60.10%) | loss: 3.944291 | lrm: 0.63 | dt: 1588.68ms | tok/sec: 10,312 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 46 | total time: 79.70m | eta: 53.1m
step 03006/05000 (60.12%) | loss: 3.940583 | lrm: 0.63 | dt: 1520.65ms | tok/sec: 10,774 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 46 | total time: 79.72m | eta: 53.1m
step 03007/05000 (60.14%) | loss: 3.941559 | lrm: 0.63 | dt: 1569.15ms | tok/sec: 10,441 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 46 | total time: 79.75m | eta: 53.0m
step 03008/05000 (60.16%) | loss: 3.953608 | lrm: 0.63 | dt: 1565.96ms | tok/sec: 10,462 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 46 | total time: 79.77m | eta: 53.0m
step 03009/05000 (60.18%) | loss: 3.944999 | lrm: 0.63 | dt: 1547.28ms | tok/sec: 10,588 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 46 | total time: 79.80m | eta: 53.0m
step 03010/05000 (60.20%) | loss: 3.950729 | lrm: 0.63 | dt: 1587.08ms | tok/sec: 10,323 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 46 | total time: 79.83m | eta: 53.0m
step 03011/05000 (60.22%) | loss: 3.958053 | lrm: 0.63 | dt: 1552.34ms | tok/sec: 10,554 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 46 | total time: 79.85m | eta: 52.9m
step 03012/05000 (60.24%) | loss: 3.945931 | lrm: 0.63 | dt: 1543.39ms | tok/sec: 10,615 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 46 | total time: 79.88m | eta: 52.9m
step 03013/05000 (60.26%) | loss: 3.948687 | lrm: 0.63 | dt: 1548.96ms | tok/sec: 10,577 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 46 | total time: 79.90m | eta: 52.9m
step 03014/05000 (60.28%) | loss: 3.937138 | lrm: 0.63 | dt: 1540.83ms | tok/sec: 10,633 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 46 | total time: 79.93m | eta: 52.8m
step 03015/05000 (60.30%) | loss: 3.938021 | lrm: 0.63 | dt: 1543.19ms | tok/sec: 10,616 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 47 | total time: 79.95m | eta: 52.8m
step 03016/05000 (60.32%) | loss: 3.932327 | lrm: 0.63 | dt: 1537.73ms | tok/sec: 10,654 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 47 | total time: 79.98m | eta: 52.8m
step 03017/05000 (60.34%) | loss: 3.934821 | lrm: 0.63 | dt: 1538.12ms | tok/sec: 10,651 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 47 | total time: 80.01m | eta: 52.8m
step 03018/05000 (60.36%) | loss: 3.934358 | lrm: 0.63 | dt: 1522.44ms | tok/sec: 10,761 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 47 | total time: 80.03m | eta: 52.7m
step 03019/05000 (60.38%) | loss: 3.946481 | lrm: 0.63 | dt: 1551.91ms | tok/sec: 10,557 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 47 | total time: 80.06m | eta: 52.7m
step 03020/05000 (60.40%) | loss: 3.947847 | lrm: 0.63 | dt: 1570.85ms | tok/sec: 10,430 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 47 | total time: 80.08m | eta: 52.7m
step 03021/05000 (60.42%) | loss: 3.943101 | lrm: 0.63 | dt: 1528.47ms | tok/sec: 10,719 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 47 | total time: 80.11m | eta: 52.7m
step 03022/05000 (60.44%) | loss: 3.937071 | lrm: 0.63 | dt: 1547.28ms | tok/sec: 10,588 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 47 | total time: 80.13m | eta: 52.6m
step 03023/05000 (60.46%) | loss: 3.929849 | lrm: 0.63 | dt: 1571.27ms | tok/sec: 10,427 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 47 | total time: 80.16m | eta: 52.6m
step 03024/05000 (60.48%) | loss: 3.917212 | lrm: 0.63 | dt: 1521.04ms | tok/sec: 10,771 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 47 | total time: 80.19m | eta: 52.6m
step 03025/05000 (60.50%) | loss: 3.923686 | lrm: 0.63 | dt: 1565.91ms | tok/sec: 10,462 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 47 | total time: 80.21m | eta: 52.5m
step 03026/05000 (60.52%) | loss: 3.914507 | lrm: 0.63 | dt: 1562.88ms | tok/sec: 10,483 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 47 | total time: 80.24m | eta: 52.5m
step 03027/05000 (60.54%) | loss: 3.896705 | lrm: 0.63 | dt: 1555.31ms | tok/sec: 10,534 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 47 | total time: 80.26m | eta: 52.5m
step 03028/05000 (60.56%) | loss: 3.895035 | lrm: 0.63 | dt: 1550.06ms | tok/sec: 10,569 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 47 | total time: 80.29m | eta: 52.5m
step 03029/05000 (60.58%) | loss: 3.905717 | lrm: 0.63 | dt: 1470.30ms | tok/sec: 11,143 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 47 | total time: 80.31m | eta: 52.4m
step 03030/05000 (60.60%) | loss: 3.915744 | lrm: 0.63 | dt: 1545.52ms | tok/sec: 10,600 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 47 | total time: 80.34m | eta: 52.4m
step 03031/05000 (60.62%) | loss: 3.932782 | lrm: 0.63 | dt: 1545.33ms | tok/sec: 10,602 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 47 | total time: 80.37m | eta: 52.4m
step 03032/05000 (60.64%) | loss: 3.926828 | lrm: 0.63 | dt: 1507.85ms | tok/sec: 10,865 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 47 | total time: 80.39m | eta: 52.4m
step 03033/05000 (60.66%) | loss: 3.927275 | lrm: 0.62 | dt: 1575.03ms | tok/sec: 10,402 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 47 | total time: 80.42m | eta: 52.3m
step 03034/05000 (60.68%) | loss: 3.920856 | lrm: 0.62 | dt: 1532.67ms | tok/sec: 10,689 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 47 | total time: 80.44m | eta: 52.3m
step 03035/05000 (60.70%) | loss: 3.914914 | lrm: 0.62 | dt: 1526.91ms | tok/sec: 10,730 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 47 | total time: 80.47m | eta: 52.3m
step 03036/05000 (60.72%) | loss: 3.912130 | lrm: 0.62 | dt: 1497.18ms | tok/sec: 10,943 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 47 | total time: 80.49m | eta: 52.2m
step 03037/05000 (60.74%) | loss: 3.916580 | lrm: 0.62 | dt: 1541.03ms | tok/sec: 10,631 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 47 | total time: 80.52m | eta: 52.2m
step 03038/05000 (60.76%) | loss: 3.915334 | lrm: 0.62 | dt: 1583.20ms | tok/sec: 10,348 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 48 | total time: 80.55m | eta: 52.2m
step 03039/05000 (60.78%) | loss: 3.913277 | lrm: 0.62 | dt: 1616.73ms | tok/sec: 10,134 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 48 | total time: 80.57m | eta: 52.2m
step 03040/05000 (60.80%) | loss: 3.910312 | lrm: 0.62 | dt: 1549.59ms | tok/sec: 10,573 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 48 | total time: 80.60m | eta: 52.1m
step 03041/05000 (60.82%) | loss: 3.895256 | lrm: 0.62 | dt: 1558.67ms | tok/sec: 10,511 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 48 | total time: 80.62m | eta: 52.1m
step 03042/05000 (60.84%) | loss: 3.890925 | lrm: 0.62 | dt: 1598.02ms | tok/sec: 10,252 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 48 | total time: 80.65m | eta: 52.1m
step 03043/05000 (60.86%) | loss: 3.893821 | lrm: 0.62 | dt: 1516.76ms | tok/sec: 10,801 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 48 | total time: 80.68m | eta: 52.1m
step 03044/05000 (60.88%) | loss: 3.886304 | lrm: 0.62 | dt: 1508.70ms | tok/sec: 10,859 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 48 | total time: 80.70m | eta: 52.0m
step 03045/05000 (60.90%) | loss: 3.901388 | lrm: 0.62 | dt: 1556.68ms | tok/sec: 10,524 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 48 | total time: 80.73m | eta: 52.0m
step 03046/05000 (60.92%) | loss: 3.905799 | lrm: 0.62 | dt: 1545.76ms | tok/sec: 10,599 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 48 | total time: 80.75m | eta: 52.0m
step 03047/05000 (60.94%) | loss: 3.920121 | lrm: 0.62 | dt: 1537.97ms | tok/sec: 10,652 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 48 | total time: 80.78m | eta: 51.9m
step 03048/05000 (60.96%) | loss: 3.918757 | lrm: 0.62 | dt: 1546.73ms | tok/sec: 10,592 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 48 | total time: 80.80m | eta: 51.9m
step 03049/05000 (60.98%) | loss: 3.919630 | lrm: 0.62 | dt: 1537.52ms | tok/sec: 10,656 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 48 | total time: 80.83m | eta: 51.9m
step 03050/05000 (61.00%) | loss: 3.924971 | lrm: 0.62 | dt: 1547.40ms | tok/sec: 10,588 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 48 | total time: 80.86m | eta: 51.9m
step 03051/05000 (61.02%) | loss: 3.944460 | lrm: 0.62 | dt: 1513.21ms | tok/sec: 10,827 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 48 | total time: 80.88m | eta: 51.8m
step 03052/05000 (61.04%) | loss: 3.924161 | lrm: 0.62 | dt: 1558.09ms | tok/sec: 10,515 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 48 | total time: 80.91m | eta: 51.8m
step 03053/05000 (61.06%) | loss: 3.928907 | lrm: 0.62 | dt: 1563.10ms | tok/sec: 10,481 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 48 | total time: 80.93m | eta: 51.8m
step 03054/05000 (61.08%) | loss: 3.938137 | lrm: 0.62 | dt: 1512.39ms | tok/sec: 10,833 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 48 | total time: 80.96m | eta: 51.8m
step 03055/05000 (61.10%) | loss: 3.939501 | lrm: 0.62 | dt: 1546.87ms | tok/sec: 10,591 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 48 | total time: 80.98m | eta: 51.7m
step 03056/05000 (61.12%) | loss: 3.936209 | lrm: 0.62 | dt: 1569.03ms | tok/sec: 10,442 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 48 | total time: 81.01m | eta: 51.7m
step 03057/05000 (61.14%) | loss: 3.942749 | lrm: 0.62 | dt: 1558.28ms | tok/sec: 10,514 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 48 | total time: 81.04m | eta: 51.7m
step 03058/05000 (61.16%) | loss: 3.944416 | lrm: 0.62 | dt: 1518.04ms | tok/sec: 10,792 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 48 | total time: 81.06m | eta: 51.6m
step 03059/05000 (61.18%) | loss: 3.937943 | lrm: 0.62 | dt: 1547.78ms | tok/sec: 10,585 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 48 | total time: 81.09m | eta: 51.6m
step 03060/05000 (61.20%) | loss: 3.945066 | lrm: 0.62 | dt: 1498.50ms | tok/sec: 10,933 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 48 | total time: 81.11m | eta: 51.6m
step 03061/05000 (61.22%) | loss: 3.943282 | lrm: 0.62 | dt: 1498.33ms | tok/sec: 10,934 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 49 | total time: 81.14m | eta: 51.6m
step 03062/05000 (61.24%) | loss: 3.946048 | lrm: 0.62 | dt: 1515.51ms | tok/sec: 10,810 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 49 | total time: 81.16m | eta: 51.5m
step 03063/05000 (61.26%) | loss: 3.953751 | lrm: 0.62 | dt: 1595.99ms | tok/sec: 10,265 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 49 | total time: 81.19m | eta: 51.5m
step 03064/05000 (61.28%) | loss: 3.956335 | lrm: 0.62 | dt: 1582.96ms | tok/sec: 10,350 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 49 | total time: 81.22m | eta: 51.5m
step 03065/05000 (61.30%) | loss: 3.967680 | lrm: 0.62 | dt: 1561.06ms | tok/sec: 10,495 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 49 | total time: 81.24m | eta: 51.5m
step 03066/05000 (61.32%) | loss: 3.961739 | lrm: 0.62 | dt: 1496.08ms | tok/sec: 10,951 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 49 | total time: 81.27m | eta: 51.4m
step 03067/05000 (61.34%) | loss: 3.969442 | lrm: 0.62 | dt: 1473.95ms | tok/sec: 11,115 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 49 | total time: 81.29m | eta: 51.4m
step 03068/05000 (61.36%) | loss: 3.967803 | lrm: 0.61 | dt: 1511.09ms | tok/sec: 10,842 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 49 | total time: 81.32m | eta: 51.4m
step 03069/05000 (61.38%) | loss: 3.970693 | lrm: 0.61 | dt: 1470.56ms | tok/sec: 11,141 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 49 | total time: 81.34m | eta: 51.3m
step 03070/05000 (61.40%) | loss: 3.967620 | lrm: 0.61 | dt: 1509.16ms | tok/sec: 10,856 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 49 | total time: 81.37m | eta: 51.3m
step 03071/05000 (61.42%) | loss: 3.952971 | lrm: 0.61 | dt: 1465.47ms | tok/sec: 11,180 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 49 | total time: 81.39m | eta: 51.3m
step 03072/05000 (61.44%) | loss: 3.959054 | lrm: 0.61 | dt: 1514.34ms | tok/sec: 10,819 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 49 | total time: 81.42m | eta: 51.3m
step 03073/05000 (61.46%) | loss: 3.939175 | lrm: 0.61 | dt: 1474.53ms | tok/sec: 11,111 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 49 | total time: 81.44m | eta: 51.2m
step 03074/05000 (61.48%) | loss: 3.952023 | lrm: 0.61 | dt: 1499.79ms | tok/sec: 10,924 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 49 | total time: 81.46m | eta: 51.2m
step 03075/05000 (61.50%) | loss: 3.954932 | lrm: 0.61 | dt: 1458.00ms | tok/sec: 11,237 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 49 | total time: 81.49m | eta: 51.2m
step 03076/05000 (61.52%) | loss: 3.947465 | lrm: 0.61 | dt: 1475.79ms | tok/sec: 11,101 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 49 | total time: 81.51m | eta: 51.2m
step 03077/05000 (61.54%) | loss: 3.962768 | lrm: 0.61 | dt: 1479.47ms | tok/sec: 11,074 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 49 | total time: 81.54m | eta: 51.1m
step 03078/05000 (61.56%) | loss: 3.958902 | lrm: 0.61 | dt: 1497.14ms | tok/sec: 10,943 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 49 | total time: 81.56m | eta: 51.1m
step 03079/05000 (61.58%) | loss: 3.960146 | lrm: 0.61 | dt: 1454.63ms | tok/sec: 11,263 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 49 | total time: 81.59m | eta: 51.1m
step 03080/05000 (61.60%) | loss: 3.963862 | lrm: 0.61 | dt: 1486.36ms | tok/sec: 11,022 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 49 | total time: 81.61m | eta: 51.0m
step 03081/05000 (61.62%) | loss: 3.954127 | lrm: 0.61 | dt: 1464.24ms | tok/sec: 11,189 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 49 | total time: 81.64m | eta: 51.0m
step 03082/05000 (61.64%) | loss: 3.935910 | lrm: 0.61 | dt: 1499.28ms | tok/sec: 10,927 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 49 | total time: 81.66m | eta: 51.0m
step 03083/05000 (61.66%) | loss: 3.940526 | lrm: 0.61 | dt: 1451.30ms | tok/sec: 11,289 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 49 | total time: 81.69m | eta: 51.0m
step 03084/05000 (61.68%) | loss: 3.940169 | lrm: 0.61 | dt: 1466.66ms | tok/sec: 11,170 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 50 | total time: 81.71m | eta: 50.9m
step 03085/05000 (61.70%) | loss: 3.942279 | lrm: 0.61 | dt: 1463.49ms | tok/sec: 11,195 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 50 | total time: 81.73m | eta: 50.9m
step 03086/05000 (61.72%) | loss: 3.941145 | lrm: 0.61 | dt: 1432.09ms | tok/sec: 11,440 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 50 | total time: 81.76m | eta: 50.9m
step 03087/05000 (61.74%) | loss: 3.937412 | lrm: 0.61 | dt: 1441.09ms | tok/sec: 11,369 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 50 | total time: 81.78m | eta: 50.8m
step 03088/05000 (61.76%) | loss: 3.935548 | lrm: 0.61 | dt: 1434.35ms | tok/sec: 11,422 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 50 | total time: 81.81m | eta: 50.8m
step 03089/05000 (61.78%) | loss: 3.933684 | lrm: 0.61 | dt: 1441.54ms | tok/sec: 11,365 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 50 | total time: 81.83m | eta: 50.8m
step 03090/05000 (61.80%) | loss: 3.933270 | lrm: 0.61 | dt: 1488.90ms | tok/sec: 11,004 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 50 | total time: 81.86m | eta: 50.8m
step 03091/05000 (61.82%) | loss: 3.921978 | lrm: 0.61 | dt: 1471.49ms | tok/sec: 11,134 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 50 | total time: 81.88m | eta: 50.7m
step 03092/05000 (61.84%) | loss: 3.921451 | lrm: 0.61 | dt: 1552.26ms | tok/sec: 10,554 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 50 | total time: 81.91m | eta: 50.7m
step 03093/05000 (61.86%) | loss: 3.912900 | lrm: 0.61 | dt: 1466.18ms | tok/sec: 11,174 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 50 | total time: 81.93m | eta: 50.7m
step 03094/05000 (61.88%) | loss: 3.931604 | lrm: 0.61 | dt: 1562.63ms | tok/sec: 10,484 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 50 | total time: 81.96m | eta: 50.7m
step 03095/05000 (61.90%) | loss: 3.942017 | lrm: 0.61 | dt: 1483.58ms | tok/sec: 11,043 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 50 | total time: 81.98m | eta: 50.6m
step 03096/05000 (61.92%) | loss: 3.957083 | lrm: 0.61 | dt: 1448.01ms | tok/sec: 11,314 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 50 | total time: 82.00m | eta: 50.6m
step 03097/05000 (61.94%) | loss: 3.950845 | lrm: 0.61 | dt: 1495.39ms | tok/sec: 10,956 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 50 | total time: 82.03m | eta: 50.6m
step 03098/05000 (61.96%) | loss: 3.942848 | lrm: 0.61 | dt: 1434.22ms | tok/sec: 11,423 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 50 | total time: 82.05m | eta: 50.5m
step 03099/05000 (61.98%) | loss: 3.942093 | lrm: 0.61 | dt: 1489.05ms | tok/sec: 11,002 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 50 | total time: 82.08m | eta: 50.5m
Step 03100 | Validation bpb: 1.220255
<|bos|>The capital of France is a small town in France, and it is a small town in France. It
<|bos|>The chemical symbol of gold is the symbol of the gold's ability to produce a gold. The symbol of gold
<|bos|>If yesterday was Friday, then tomorrow will be Friday. Friday, Friday, Friday, Friday, Friday, Friday, Friday,
<|bos|>The opposite of hot is the same as the above. The same thing is true. The same thing is
<|bos|>The planets of the solar system are: the solar system, the solar system, the solar system, the solar system,
<|bos|>My favorite color is the black, but I'm not sure how to use it. I'm not
<|bos|>If 5*x + 3 = 13, then x is 1.5.5.5.5.5.5.5
step 03100/05000 (62.00%) | loss: 3.956109 | lrm: 0.61 | dt: 1590.15ms | tok/sec: 10,303 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 50 | total time: 82.11m | eta: 50.5m
step 03101/05000 (62.02%) | loss: 3.955793 | lrm: 0.61 | dt: 1468.17ms | tok/sec: 11,159 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 50 | total time: 82.13m | eta: 50.5m
step 03102/05000 (62.04%) | loss: 3.967449 | lrm: 0.60 | dt: 1420.59ms | tok/sec: 11,533 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 50 | total time: 82.15m | eta: 50.4m
step 03103/05000 (62.06%) | loss: 3.970681 | lrm: 0.60 | dt: 1475.44ms | tok/sec: 11,104 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 50 | total time: 82.18m | eta: 50.4m
step 03104/05000 (62.08%) | loss: 3.970139 | lrm: 0.60 | dt: 1487.07ms | tok/sec: 11,017 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 50 | total time: 82.20m | eta: 50.4m
step 03105/05000 (62.10%) | loss: 3.964916 | lrm: 0.60 | dt: 1460.88ms | tok/sec: 11,215 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 50 | total time: 82.23m | eta: 50.3m
step 03106/05000 (62.12%) | loss: 3.971607 | lrm: 0.60 | dt: 1415.96ms | tok/sec: 11,570 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 50 | total time: 82.25m | eta: 50.3m
step 03107/05000 (62.14%) | loss: 3.968957 | lrm: 0.60 | dt: 1479.47ms | tok/sec: 11,074 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 51 | total time: 82.28m | eta: 50.3m
step 03108/05000 (62.16%) | loss: 3.959313 | lrm: 0.60 | dt: 1463.20ms | tok/sec: 11,197 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 51 | total time: 82.30m | eta: 50.3m
step 03109/05000 (62.18%) | loss: 3.961797 | lrm: 0.60 | dt: 1453.58ms | tok/sec: 11,271 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 51 | total time: 82.32m | eta: 50.2m
step 03110/05000 (62.20%) | loss: 3.960529 | lrm: 0.60 | dt: 1502.94ms | tok/sec: 10,901 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 51 | total time: 82.35m | eta: 50.2m
step 03111/05000 (62.22%) | loss: 3.971563 | lrm: 0.60 | dt: 1454.60ms | tok/sec: 11,263 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 51 | total time: 82.37m | eta: 50.2m
step 03112/05000 (62.24%) | loss: 3.963671 | lrm: 0.60 | dt: 1514.96ms | tok/sec: 10,814 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 51 | total time: 82.40m | eta: 50.2m
step 03113/05000 (62.26%) | loss: 3.956184 | lrm: 0.60 | dt: 1454.03ms | tok/sec: 11,267 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 51 | total time: 82.42m | eta: 50.1m
step 03114/05000 (62.28%) | loss: 3.952254 | lrm: 0.60 | dt: 1514.92ms | tok/sec: 10,815 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 51 | total time: 82.45m | eta: 50.1m
step 03115/05000 (62.30%) | loss: 3.941006 | lrm: 0.60 | dt: 1421.64ms | tok/sec: 11,524 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 51 | total time: 82.47m | eta: 50.1m
step 03116/05000 (62.32%) | loss: 3.943829 | lrm: 0.60 | dt: 1471.32ms | tok/sec: 11,135 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 51 | total time: 82.50m | eta: 50.0m
step 03117/05000 (62.34%) | loss: 3.954066 | lrm: 0.60 | dt: 1465.66ms | tok/sec: 11,178 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 51 | total time: 82.52m | eta: 50.0m
step 03118/05000 (62.36%) | loss: 3.947852 | lrm: 0.60 | dt: 1471.63ms | tok/sec: 11,133 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 51 | total time: 82.55m | eta: 50.0m
step 03119/05000 (62.38%) | loss: 3.947339 | lrm: 0.60 | dt: 1473.06ms | tok/sec: 11,122 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 51 | total time: 82.57m | eta: 50.0m
step 03120/05000 (62.40%) | loss: 3.948573 | lrm: 0.60 | dt: 1468.29ms | tok/sec: 11,158 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 51 | total time: 82.59m | eta: 49.9m
step 03121/05000 (62.42%) | loss: 3.950961 | lrm: 0.60 | dt: 1483.14ms | tok/sec: 11,046 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 51 | total time: 82.62m | eta: 49.9m
step 03122/05000 (62.44%) | loss: 3.960397 | lrm: 0.60 | dt: 1474.24ms | tok/sec: 11,113 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 51 | total time: 82.64m | eta: 49.9m
step 03123/05000 (62.46%) | loss: 3.957635 | lrm: 0.60 | dt: 1480.28ms | tok/sec: 11,068 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 51 | total time: 82.67m | eta: 49.8m
step 03124/05000 (62.48%) | loss: 3.939702 | lrm: 0.60 | dt: 1578.07ms | tok/sec: 10,382 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 51 | total time: 82.69m | eta: 49.8m
step 03125/05000 (62.50%) | loss: 3.940187 | lrm: 0.60 | dt: 1584.15ms | tok/sec: 10,342 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 51 | total time: 82.72m | eta: 49.8m
step 03126/05000 (62.52%) | loss: 3.937651 | lrm: 0.60 | dt: 1608.47ms | tok/sec: 10,186 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 51 | total time: 82.75m | eta: 49.8m
step 03127/05000 (62.54%) | loss: 3.945621 | lrm: 0.60 | dt: 1560.68ms | tok/sec: 10,497 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 51 | total time: 82.77m | eta: 49.7m
step 03128/05000 (62.56%) | loss: 3.958602 | lrm: 0.60 | dt: 1578.05ms | tok/sec: 10,382 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 51 | total time: 82.80m | eta: 49.7m
step 03129/05000 (62.58%) | loss: 3.956548 | lrm: 0.60 | dt: 1532.53ms | tok/sec: 10,690 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 51 | total time: 82.83m | eta: 49.7m
step 03130/05000 (62.60%) | loss: 3.965904 | lrm: 0.60 | dt: 1499.82ms | tok/sec: 10,924 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 52 | total time: 82.85m | eta: 49.7m
step 03131/05000 (62.62%) | loss: 3.958326 | lrm: 0.60 | dt: 1518.38ms | tok/sec: 10,790 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 52 | total time: 82.88m | eta: 49.6m
step 03132/05000 (62.64%) | loss: 3.949915 | lrm: 0.60 | dt: 1585.61ms | tok/sec: 10,332 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 52 | total time: 82.90m | eta: 49.6m
step 03133/05000 (62.66%) | loss: 3.936304 | lrm: 0.60 | dt: 1528.84ms | tok/sec: 10,716 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 52 | total time: 82.93m | eta: 49.6m
step 03134/05000 (62.68%) | loss: 3.946171 | lrm: 0.60 | dt: 1567.66ms | tok/sec: 10,451 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 52 | total time: 82.95m | eta: 49.5m
step 03135/05000 (62.70%) | loss: 3.954159 | lrm: 0.60 | dt: 1550.23ms | tok/sec: 10,568 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 52 | total time: 82.98m | eta: 49.5m
step 03136/05000 (62.72%) | loss: 3.970657 | lrm: 0.59 | dt: 1574.54ms | tok/sec: 10,405 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 52 | total time: 83.01m | eta: 49.5m
step 03137/05000 (62.74%) | loss: 3.964653 | lrm: 0.59 | dt: 1555.15ms | tok/sec: 10,535 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 52 | total time: 83.03m | eta: 49.5m
step 03138/05000 (62.76%) | loss: 3.958500 | lrm: 0.59 | dt: 1499.96ms | tok/sec: 10,922 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 52 | total time: 83.06m | eta: 49.4m
step 03139/05000 (62.78%) | loss: 3.928245 | lrm: 0.59 | dt: 1588.58ms | tok/sec: 10,313 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 52 | total time: 83.08m | eta: 49.4m
step 03140/05000 (62.80%) | loss: 3.928852 | lrm: 0.59 | dt: 1573.95ms | tok/sec: 10,409 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 52 | total time: 83.11m | eta: 49.4m
step 03141/05000 (62.82%) | loss: 3.912368 | lrm: 0.59 | dt: 1519.71ms | tok/sec: 10,780 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 52 | total time: 83.13m | eta: 49.4m
step 03142/05000 (62.84%) | loss: 3.908769 | lrm: 0.59 | dt: 1529.67ms | tok/sec: 10,710 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 52 | total time: 83.16m | eta: 49.3m
step 03143/05000 (62.86%) | loss: 3.915144 | lrm: 0.59 | dt: 1565.41ms | tok/sec: 10,466 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 52 | total time: 83.19m | eta: 49.3m
step 03144/05000 (62.88%) | loss: 3.913307 | lrm: 0.59 | dt: 1568.68ms | tok/sec: 10,444 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 52 | total time: 83.21m | eta: 49.3m
step 03145/05000 (62.90%) | loss: 3.917324 | lrm: 0.59 | dt: 1563.43ms | tok/sec: 10,479 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 52 | total time: 83.24m | eta: 49.3m
step 03146/05000 (62.92%) | loss: 3.910351 | lrm: 0.59 | dt: 1552.25ms | tok/sec: 10,555 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 52 | total time: 83.26m | eta: 49.2m
step 03147/05000 (62.94%) | loss: 3.905710 | lrm: 0.59 | dt: 1565.29ms | tok/sec: 10,467 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 52 | total time: 83.29m | eta: 49.2m
step 03148/05000 (62.96%) | loss: 3.906703 | lrm: 0.59 | dt: 1546.25ms | tok/sec: 10,595 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 52 | total time: 83.32m | eta: 49.2m
step 03149/05000 (62.98%) | loss: 3.922127 | lrm: 0.59 | dt: 1575.12ms | tok/sec: 10,401 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 52 | total time: 83.34m | eta: 49.1m
step 03150/05000 (63.00%) | loss: 3.920234 | lrm: 0.59 | dt: 1540.94ms | tok/sec: 10,632 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 52 | total time: 83.37m | eta: 49.1m
step 03151/05000 (63.02%) | loss: 3.904193 | lrm: 0.59 | dt: 1530.86ms | tok/sec: 10,702 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 52 | total time: 83.39m | eta: 49.1m
step 03152/05000 (63.04%) | loss: 3.915833 | lrm: 0.59 | dt: 1539.05ms | tok/sec: 10,645 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 52 | total time: 83.42m | eta: 49.1m
step 03153/05000 (63.06%) | loss: 3.929166 | lrm: 0.59 | dt: 1532.23ms | tok/sec: 10,692 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 53 | total time: 83.44m | eta: 49.0m
step 03154/05000 (63.08%) | loss: 3.935535 | lrm: 0.59 | dt: 1545.04ms | tok/sec: 10,604 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 53 | total time: 83.47m | eta: 49.0m
step 03155/05000 (63.10%) | loss: 3.927416 | lrm: 0.59 | dt: 1541.39ms | tok/sec: 10,629 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 53 | total time: 83.50m | eta: 49.0m
step 03156/05000 (63.12%) | loss: 3.933240 | lrm: 0.59 | dt: 1566.81ms | tok/sec: 10,456 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 53 | total time: 83.52m | eta: 49.0m
step 03157/05000 (63.14%) | loss: 3.936658 | lrm: 0.59 | dt: 1556.44ms | tok/sec: 10,526 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 53 | total time: 83.55m | eta: 48.9m
step 03158/05000 (63.16%) | loss: 3.925769 | lrm: 0.59 | dt: 1565.19ms | tok/sec: 10,467 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 53 | total time: 83.57m | eta: 48.9m
step 03159/05000 (63.18%) | loss: 3.921626 | lrm: 0.59 | dt: 1578.12ms | tok/sec: 10,381 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 53 | total time: 83.60m | eta: 48.9m
step 03160/05000 (63.20%) | loss: 3.923225 | lrm: 0.59 | dt: 1563.42ms | tok/sec: 10,479 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 53 | total time: 83.63m | eta: 48.8m
step 03161/05000 (63.22%) | loss: 3.923268 | lrm: 0.59 | dt: 1528.47ms | tok/sec: 10,719 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 53 | total time: 83.65m | eta: 48.8m
step 03162/05000 (63.24%) | loss: 3.912081 | lrm: 0.59 | dt: 1541.07ms | tok/sec: 10,631 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 53 | total time: 83.68m | eta: 48.8m
step 03163/05000 (63.26%) | loss: 3.914535 | lrm: 0.59 | dt: 1534.93ms | tok/sec: 10,674 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 53 | total time: 83.70m | eta: 48.8m
step 03164/05000 (63.28%) | loss: 3.902953 | lrm: 0.59 | dt: 1565.81ms | tok/sec: 10,463 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 53 | total time: 83.73m | eta: 48.7m
step 03165/05000 (63.30%) | loss: 3.908128 | lrm: 0.59 | dt: 1552.99ms | tok/sec: 10,549 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 53 | total time: 83.76m | eta: 48.7m
step 03166/05000 (63.32%) | loss: 3.925108 | lrm: 0.59 | dt: 1556.67ms | tok/sec: 10,525 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 53 | total time: 83.78m | eta: 48.7m
step 03167/05000 (63.34%) | loss: 3.932113 | lrm: 0.59 | dt: 1557.70ms | tok/sec: 10,518 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 53 | total time: 83.81m | eta: 48.7m
step 03168/05000 (63.36%) | loss: 3.906061 | lrm: 0.59 | dt: 1557.71ms | tok/sec: 10,518 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 53 | total time: 83.83m | eta: 48.6m
step 03169/05000 (63.38%) | loss: 3.915583 | lrm: 0.59 | dt: 1555.36ms | tok/sec: 10,533 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 53 | total time: 83.86m | eta: 48.6m
step 03170/05000 (63.40%) | loss: 3.925521 | lrm: 0.58 | dt: 1556.01ms | tok/sec: 10,529 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 53 | total time: 83.89m | eta: 48.6m
step 03171/05000 (63.42%) | loss: 3.933046 | lrm: 0.58 | dt: 1548.90ms | tok/sec: 10,577 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 53 | total time: 83.91m | eta: 48.6m
step 03172/05000 (63.44%) | loss: 3.934700 | lrm: 0.58 | dt: 1547.11ms | tok/sec: 10,590 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 53 | total time: 83.94m | eta: 48.5m
step 03173/05000 (63.46%) | loss: 3.928914 | lrm: 0.58 | dt: 1556.43ms | tok/sec: 10,526 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 53 | total time: 83.96m | eta: 48.5m
step 03174/05000 (63.48%) | loss: 3.934608 | lrm: 0.58 | dt: 1548.27ms | tok/sec: 10,582 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 53 | total time: 83.99m | eta: 48.5m
step 03175/05000 (63.50%) | loss: 3.932580 | lrm: 0.58 | dt: 1557.25ms | tok/sec: 10,521 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 53 | total time: 84.01m | eta: 48.4m
step 03176/05000 (63.52%) | loss: 3.922333 | lrm: 0.58 | dt: 1540.16ms | tok/sec: 10,637 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 54 | total time: 84.04m | eta: 48.4m
step 03177/05000 (63.54%) | loss: 3.927201 | lrm: 0.58 | dt: 1568.31ms | tok/sec: 10,446 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 54 | total time: 84.07m | eta: 48.4m
step 03178/05000 (63.56%) | loss: 3.932324 | lrm: 0.58 | dt: 1534.14ms | tok/sec: 10,679 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 54 | total time: 84.09m | eta: 48.4m
step 03179/05000 (63.58%) | loss: 3.928824 | lrm: 0.58 | dt: 1536.37ms | tok/sec: 10,664 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 54 | total time: 84.12m | eta: 48.3m
step 03180/05000 (63.60%) | loss: 3.931311 | lrm: 0.58 | dt: 1557.00ms | tok/sec: 10,522 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 54 | total time: 84.14m | eta: 48.3m
step 03181/05000 (63.62%) | loss: 3.925219 | lrm: 0.58 | dt: 1538.06ms | tok/sec: 10,652 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 54 | total time: 84.17m | eta: 48.3m
step 03182/05000 (63.64%) | loss: 3.908848 | lrm: 0.58 | dt: 1540.94ms | tok/sec: 10,632 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 54 | total time: 84.19m | eta: 48.3m
step 03183/05000 (63.66%) | loss: 3.904373 | lrm: 0.58 | dt: 1533.15ms | tok/sec: 10,686 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 54 | total time: 84.22m | eta: 48.2m
step 03184/05000 (63.68%) | loss: 3.915493 | lrm: 0.58 | dt: 1557.15ms | tok/sec: 10,521 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 54 | total time: 84.25m | eta: 48.2m
step 03185/05000 (63.70%) | loss: 3.924395 | lrm: 0.58 | dt: 1565.03ms | tok/sec: 10,468 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 54 | total time: 84.27m | eta: 48.2m
step 03186/05000 (63.72%) | loss: 3.925515 | lrm: 0.58 | dt: 1546.28ms | tok/sec: 10,595 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 54 | total time: 84.30m | eta: 48.1m
step 03187/05000 (63.74%) | loss: 3.929608 | lrm: 0.58 | dt: 1547.14ms | tok/sec: 10,589 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 54 | total time: 84.32m | eta: 48.1m
step 03188/05000 (63.76%) | loss: 3.914654 | lrm: 0.58 | dt: 1530.23ms | tok/sec: 10,706 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 54 | total time: 84.35m | eta: 48.1m
step 03189/05000 (63.78%) | loss: 3.903222 | lrm: 0.58 | dt: 1498.32ms | tok/sec: 10,934 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 54 | total time: 84.37m | eta: 48.1m
step 03190/05000 (63.80%) | loss: 3.909829 | lrm: 0.58 | dt: 1578.30ms | tok/sec: 10,380 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 54 | total time: 84.40m | eta: 48.0m
step 03191/05000 (63.82%) | loss: 3.913349 | lrm: 0.58 | dt: 1542.99ms | tok/sec: 10,618 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 54 | total time: 84.43m | eta: 48.0m
step 03192/05000 (63.84%) | loss: 3.915835 | lrm: 0.58 | dt: 1535.97ms | tok/sec: 10,666 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 54 | total time: 84.45m | eta: 48.0m
step 03193/05000 (63.86%) | loss: 3.921882 | lrm: 0.58 | dt: 1545.67ms | tok/sec: 10,599 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 54 | total time: 84.48m | eta: 48.0m
step 03194/05000 (63.88%) | loss: 3.923113 | lrm: 0.58 | dt: 1529.96ms | tok/sec: 10,708 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 54 | total time: 84.50m | eta: 47.9m
step 03195/05000 (63.90%) | loss: 3.904859 | lrm: 0.58 | dt: 1538.64ms | tok/sec: 10,648 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 54 | total time: 84.53m | eta: 47.9m
step 03196/05000 (63.92%) | loss: 3.911337 | lrm: 0.58 | dt: 1548.72ms | tok/sec: 10,579 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 54 | total time: 84.55m | eta: 47.9m
step 03197/05000 (63.94%) | loss: 3.899326 | lrm: 0.58 | dt: 1550.40ms | tok/sec: 10,567 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 54 | total time: 84.58m | eta: 47.9m
step 03198/05000 (63.96%) | loss: 3.907609 | lrm: 0.58 | dt: 1559.07ms | tok/sec: 10,508 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 54 | total time: 84.61m | eta: 47.8m
step 03199/05000 (63.98%) | loss: 3.915856 | lrm: 0.58 | dt: 1587.98ms | tok/sec: 10,317 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 54 | total time: 84.63m | eta: 47.8m
Step 03200 | Validation bpb: 1.216421
<|bos|>The capital of France is the capital of France. The capital of France is the capital of France. The
<|bos|>The chemical symbol of gold is the symbol of the gold. It is the symbol of the gold. It is
<|bos|>If yesterday was Friday, then tomorrow will be the first time we had to go to the hospital. The first time we had
<|bos|>The opposite of hot is the heat. The heat is the heat that is the heat that is the heat
<|bos|>The planets of the solar system are: the solar system, the solar system, the solar system, the solar system,
<|bos|>My favorite color is the white, white, and yellow. I have a lot of colors that I
<|bos|>If 5*x + 3 = 13, then x is 3*x + 3*x + 3*x +
step 03200/05000 (64.00%) | loss: 3.905745 | lrm: 0.58 | dt: 1921.31ms | tok/sec: 8,527 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 55 | total time: 84.67m | eta: 47.8m
step 03201/05000 (64.02%) | loss: 3.904011 | lrm: 0.58 | dt: 1555.67ms | tok/sec: 10,531 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 55 | total time: 84.69m | eta: 47.7m
step 03202/05000 (64.04%) | loss: 3.905006 | lrm: 0.58 | dt: 1569.64ms | tok/sec: 10,438 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 55 | total time: 84.72m | eta: 47.7m
step 03203/05000 (64.06%) | loss: 3.899500 | lrm: 0.58 | dt: 1550.25ms | tok/sec: 10,568 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 55 | total time: 84.74m | eta: 47.7m
step 03204/05000 (64.08%) | loss: 3.911054 | lrm: 0.57 | dt: 1570.51ms | tok/sec: 10,432 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 55 | total time: 84.77m | eta: 47.7m
step 03205/05000 (64.10%) | loss: 3.924964 | lrm: 0.57 | dt: 1537.68ms | tok/sec: 10,655 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 55 | total time: 84.79m | eta: 47.6m
step 03206/05000 (64.12%) | loss: 3.923586 | lrm: 0.57 | dt: 1536.21ms | tok/sec: 10,665 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 55 | total time: 84.82m | eta: 47.6m
step 03207/05000 (64.14%) | loss: 3.926461 | lrm: 0.57 | dt: 1521.12ms | tok/sec: 10,771 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 55 | total time: 84.85m | eta: 47.6m
step 03208/05000 (64.16%) | loss: 3.912649 | lrm: 0.57 | dt: 1548.50ms | tok/sec: 10,580 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 55 | total time: 84.87m | eta: 47.6m
step 03209/05000 (64.18%) | loss: 3.914360 | lrm: 0.57 | dt: 1558.93ms | tok/sec: 10,509 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 55 | total time: 84.90m | eta: 47.5m
step 03210/05000 (64.20%) | loss: 3.918162 | lrm: 0.57 | dt: 1546.27ms | tok/sec: 10,595 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 55 | total time: 84.92m | eta: 47.5m
step 03211/05000 (64.22%) | loss: 3.922041 | lrm: 0.57 | dt: 1603.05ms | tok/sec: 10,220 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 55 | total time: 84.95m | eta: 47.5m
step 03212/05000 (64.24%) | loss: 3.930241 | lrm: 0.57 | dt: 1605.22ms | tok/sec: 10,206 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 55 | total time: 84.98m | eta: 47.5m
step 03213/05000 (64.26%) | loss: 3.936679 | lrm: 0.57 | dt: 1572.65ms | tok/sec: 10,418 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 55 | total time: 85.00m | eta: 47.4m
step 03214/05000 (64.28%) | loss: 3.915220 | lrm: 0.57 | dt: 1539.71ms | tok/sec: 10,640 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 55 | total time: 85.03m | eta: 47.4m
step 03215/05000 (64.30%) | loss: 3.919307 | lrm: 0.57 | dt: 1577.91ms | tok/sec: 10,383 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 55 | total time: 85.06m | eta: 47.4m
step 03216/05000 (64.32%) | loss: 3.922997 | lrm: 0.57 | dt: 1522.08ms | tok/sec: 10,764 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 55 | total time: 85.08m | eta: 47.3m
step 03217/05000 (64.34%) | loss: 3.917944 | lrm: 0.57 | dt: 1587.85ms | tok/sec: 10,318 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 55 | total time: 85.11m | eta: 47.3m
step 03218/05000 (64.36%) | loss: 3.910190 | lrm: 0.57 | dt: 1657.39ms | tok/sec: 9,885 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 55 | total time: 85.13m | eta: 47.3m
step 03219/05000 (64.38%) | loss: 3.907948 | lrm: 0.57 | dt: 1533.37ms | tok/sec: 10,684 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 55 | total time: 85.16m | eta: 47.3m
step 03220/05000 (64.40%) | loss: 3.915691 | lrm: 0.57 | dt: 1519.66ms | tok/sec: 10,781 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 55 | total time: 85.19m | eta: 47.2m
step 03221/05000 (64.42%) | loss: 3.918349 | lrm: 0.57 | dt: 1569.93ms | tok/sec: 10,436 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 55 | total time: 85.21m | eta: 47.2m
step 03222/05000 (64.44%) | loss: 3.918456 | lrm: 0.57 | dt: 1538.83ms | tok/sec: 10,647 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 55 | total time: 85.24m | eta: 47.2m
step 03223/05000 (64.46%) | loss: 3.920662 | lrm: 0.57 | dt: 1546.85ms | tok/sec: 10,591 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 56 | total time: 85.26m | eta: 47.2m
step 03224/05000 (64.48%) | loss: 3.914150 | lrm: 0.57 | dt: 1563.65ms | tok/sec: 10,478 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 56 | total time: 85.29m | eta: 47.1m
step 03225/05000 (64.50%) | loss: 3.920198 | lrm: 0.57 | dt: 1548.62ms | tok/sec: 10,579 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 56 | total time: 85.31m | eta: 47.1m
step 03226/05000 (64.52%) | loss: 3.916146 | lrm: 0.57 | dt: 1560.39ms | tok/sec: 10,499 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 56 | total time: 85.34m | eta: 47.1m
step 03227/05000 (64.54%) | loss: 3.916015 | lrm: 0.57 | dt: 1551.59ms | tok/sec: 10,559 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 56 | total time: 85.37m | eta: 47.0m
step 03228/05000 (64.56%) | loss: 3.926446 | lrm: 0.57 | dt: 1565.99ms | tok/sec: 10,462 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 56 | total time: 85.39m | eta: 47.0m
step 03229/05000 (64.58%) | loss: 3.926096 | lrm: 0.57 | dt: 1499.94ms | tok/sec: 10,923 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 56 | total time: 85.42m | eta: 47.0m
step 03230/05000 (64.60%) | loss: 3.908751 | lrm: 0.57 | dt: 1561.49ms | tok/sec: 10,492 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 56 | total time: 85.44m | eta: 47.0m
step 03231/05000 (64.62%) | loss: 3.918008 | lrm: 0.57 | dt: 1531.98ms | tok/sec: 10,694 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 56 | total time: 85.47m | eta: 46.9m
step 03232/05000 (64.64%) | loss: 3.892722 | lrm: 0.57 | dt: 1549.40ms | tok/sec: 10,574 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 56 | total time: 85.50m | eta: 46.9m
step 03233/05000 (64.66%) | loss: 3.894797 | lrm: 0.57 | dt: 1506.83ms | tok/sec: 10,873 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 56 | total time: 85.52m | eta: 46.9m
step 03234/05000 (64.68%) | loss: 3.900633 | lrm: 0.57 | dt: 1528.27ms | tok/sec: 10,720 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 56 | total time: 85.55m | eta: 46.9m
step 03235/05000 (64.70%) | loss: 3.905079 | lrm: 0.57 | dt: 1532.90ms | tok/sec: 10,688 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 56 | total time: 85.57m | eta: 46.8m
step 03236/05000 (64.72%) | loss: 3.914176 | lrm: 0.57 | dt: 1504.85ms | tok/sec: 10,887 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 56 | total time: 85.60m | eta: 46.8m
step 03237/05000 (64.74%) | loss: 3.924537 | lrm: 0.57 | dt: 1537.22ms | tok/sec: 10,658 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 56 | total time: 85.62m | eta: 46.8m
step 03238/05000 (64.76%) | loss: 3.914304 | lrm: 0.57 | dt: 1538.76ms | tok/sec: 10,647 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 56 | total time: 85.65m | eta: 46.8m
step 03239/05000 (64.78%) | loss: 3.920667 | lrm: 0.56 | dt: 1571.72ms | tok/sec: 10,424 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 56 | total time: 85.67m | eta: 46.7m
step 03240/05000 (64.80%) | loss: 3.918080 | lrm: 0.56 | dt: 1511.20ms | tok/sec: 10,841 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 56 | total time: 85.70m | eta: 46.7m
step 03241/05000 (64.82%) | loss: 3.925319 | lrm: 0.56 | dt: 1575.71ms | tok/sec: 10,397 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 56 | total time: 85.73m | eta: 46.7m
step 03242/05000 (64.84%) | loss: 3.924786 | lrm: 0.56 | dt: 1576.18ms | tok/sec: 10,394 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 56 | total time: 85.75m | eta: 46.6m
step 03243/05000 (64.86%) | loss: 3.939225 | lrm: 0.56 | dt: 1557.23ms | tok/sec: 10,521 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 56 | total time: 85.78m | eta: 46.6m
step 03244/05000 (64.88%) | loss: 3.931908 | lrm: 0.56 | dt: 1546.92ms | tok/sec: 10,591 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 56 | total time: 85.80m | eta: 46.6m
step 03245/05000 (64.90%) | loss: 3.931030 | lrm: 0.56 | dt: 1528.02ms | tok/sec: 10,722 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 56 | total time: 85.83m | eta: 46.6m
step 03246/05000 (64.92%) | loss: 3.922250 | lrm: 0.56 | dt: 1517.81ms | tok/sec: 10,794 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 56 | total time: 85.85m | eta: 46.5m
step 03247/05000 (64.94%) | loss: 3.938427 | lrm: 0.56 | dt: 1501.31ms | tok/sec: 10,913 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 57 | total time: 85.88m | eta: 46.5m
step 03248/05000 (64.96%) | loss: 3.931425 | lrm: 0.56 | dt: 1553.19ms | tok/sec: 10,548 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 57 | total time: 85.90m | eta: 46.5m
step 03249/05000 (64.98%) | loss: 3.942012 | lrm: 0.56 | dt: 1531.93ms | tok/sec: 10,694 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 57 | total time: 85.93m | eta: 46.5m
step 03250/05000 (65.00%) | loss: 3.937918 | lrm: 0.56 | dt: 1535.62ms | tok/sec: 10,669 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 57 | total time: 85.96m | eta: 46.4m
step 03251/05000 (65.02%) | loss: 3.940978 | lrm: 0.56 | dt: 1554.10ms | tok/sec: 10,542 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 57 | total time: 85.98m | eta: 46.4m
step 03252/05000 (65.04%) | loss: 3.948857 | lrm: 0.56 | dt: 1595.04ms | tok/sec: 10,271 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 57 | total time: 86.01m | eta: 46.4m
step 03253/05000 (65.06%) | loss: 3.916751 | lrm: 0.56 | dt: 1520.14ms | tok/sec: 10,777 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 57 | total time: 86.03m | eta: 46.3m
step 03254/05000 (65.08%) | loss: 3.927734 | lrm: 0.56 | dt: 1557.65ms | tok/sec: 10,518 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 57 | total time: 86.06m | eta: 46.3m
step 03255/05000 (65.10%) | loss: 3.928598 | lrm: 0.56 | dt: 1584.00ms | tok/sec: 10,343 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 57 | total time: 86.09m | eta: 46.3m
step 03256/05000 (65.12%) | loss: 3.906923 | lrm: 0.56 | dt: 1547.11ms | tok/sec: 10,590 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 57 | total time: 86.11m | eta: 46.3m
step 03257/05000 (65.14%) | loss: 3.900203 | lrm: 0.56 | dt: 1501.06ms | tok/sec: 10,914 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 57 | total time: 86.14m | eta: 46.2m
step 03258/05000 (65.16%) | loss: 3.896094 | lrm: 0.56 | dt: 1499.55ms | tok/sec: 10,925 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 57 | total time: 86.16m | eta: 46.2m
step 03259/05000 (65.18%) | loss: 3.895658 | lrm: 0.56 | dt: 1604.56ms | tok/sec: 10,210 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 57 | total time: 86.19m | eta: 46.2m
step 03260/05000 (65.20%) | loss: 3.894253 | lrm: 0.56 | dt: 1550.73ms | tok/sec: 10,565 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 57 | total time: 86.21m | eta: 46.2m
step 03261/05000 (65.22%) | loss: 3.898523 | lrm: 0.56 | dt: 1562.08ms | tok/sec: 10,488 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 57 | total time: 86.24m | eta: 46.1m
step 03262/05000 (65.24%) | loss: 3.899364 | lrm: 0.56 | dt: 1567.92ms | tok/sec: 10,449 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 57 | total time: 86.27m | eta: 46.1m
step 03263/05000 (65.26%) | loss: 3.910280 | lrm: 0.56 | dt: 1534.70ms | tok/sec: 10,675 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 57 | total time: 86.29m | eta: 46.1m
step 03264/05000 (65.28%) | loss: 3.904495 | lrm: 0.56 | dt: 1572.86ms | tok/sec: 10,416 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 57 | total time: 86.32m | eta: 46.1m
step 03265/05000 (65.30%) | loss: 3.908224 | lrm: 0.56 | dt: 1540.62ms | tok/sec: 10,634 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 57 | total time: 86.34m | eta: 46.0m
step 03266/05000 (65.32%) | loss: 3.918853 | lrm: 0.56 | dt: 1528.10ms | tok/sec: 10,721 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 57 | total time: 86.37m | eta: 46.0m
step 03267/05000 (65.34%) | loss: 3.923511 | lrm: 0.56 | dt: 1533.21ms | tok/sec: 10,686 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 57 | total time: 86.40m | eta: 46.0m
step 03268/05000 (65.36%) | loss: 3.900621 | lrm: 0.56 | dt: 1513.30ms | tok/sec: 10,826 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 57 | total time: 86.42m | eta: 45.9m
step 03269/05000 (65.38%) | loss: 3.891071 | lrm: 0.56 | dt: 1552.67ms | tok/sec: 10,552 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 57 | total time: 86.45m | eta: 45.9m
step 03270/05000 (65.40%) | loss: 3.891072 | lrm: 0.56 | dt: 1561.86ms | tok/sec: 10,490 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 58 | total time: 86.47m | eta: 45.9m
step 03271/05000 (65.42%) | loss: 3.903066 | lrm: 0.56 | dt: 1574.39ms | tok/sec: 10,406 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 58 | total time: 86.50m | eta: 45.9m
step 03272/05000 (65.44%) | loss: 3.902175 | lrm: 0.56 | dt: 1550.54ms | tok/sec: 10,566 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 58 | total time: 86.52m | eta: 45.8m
step 03273/05000 (65.46%) | loss: 3.891004 | lrm: 0.55 | dt: 1547.20ms | tok/sec: 10,589 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 58 | total time: 86.55m | eta: 45.8m
step 03274/05000 (65.48%) | loss: 3.880958 | lrm: 0.55 | dt: 1558.56ms | tok/sec: 10,512 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 58 | total time: 86.58m | eta: 45.8m
step 03275/05000 (65.50%) | loss: 3.889330 | lrm: 0.55 | dt: 1600.22ms | tok/sec: 10,238 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 58 | total time: 86.60m | eta: 45.8m
step 03276/05000 (65.52%) | loss: 3.896557 | lrm: 0.55 | dt: 1568.23ms | tok/sec: 10,447 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 58 | total time: 86.63m | eta: 45.7m
step 03277/05000 (65.54%) | loss: 3.903970 | lrm: 0.55 | dt: 1545.94ms | tok/sec: 10,598 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 58 | total time: 86.65m | eta: 45.7m
step 03278/05000 (65.56%) | loss: 3.912830 | lrm: 0.55 | dt: 1568.48ms | tok/sec: 10,445 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 58 | total time: 86.68m | eta: 45.7m
step 03279/05000 (65.58%) | loss: 3.908826 | lrm: 0.55 | dt: 1520.77ms | tok/sec: 10,773 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 58 | total time: 86.71m | eta: 45.6m
step 03280/05000 (65.60%) | loss: 3.914298 | lrm: 0.55 | dt: 1495.10ms | tok/sec: 10,958 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 58 | total time: 86.73m | eta: 45.6m
step 03281/05000 (65.62%) | loss: 3.910140 | lrm: 0.55 | dt: 1565.81ms | tok/sec: 10,463 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 58 | total time: 86.76m | eta: 45.6m
step 03282/05000 (65.64%) | loss: 3.897853 | lrm: 0.55 | dt: 1554.02ms | tok/sec: 10,542 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 58 | total time: 86.78m | eta: 45.6m
step 03283/05000 (65.66%) | loss: 3.897160 | lrm: 0.55 | dt: 1518.53ms | tok/sec: 10,789 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 58 | total time: 86.81m | eta: 45.5m
step 03284/05000 (65.68%) | loss: 3.899648 | lrm: 0.55 | dt: 1547.14ms | tok/sec: 10,589 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 58 | total time: 86.83m | eta: 45.5m
step 03285/05000 (65.70%) | loss: 3.912545 | lrm: 0.55 | dt: 1554.73ms | tok/sec: 10,538 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 58 | total time: 86.86m | eta: 45.5m
step 03286/05000 (65.72%) | loss: 3.911556 | lrm: 0.55 | dt: 1489.55ms | tok/sec: 10,999 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 58 | total time: 86.89m | eta: 45.5m
step 03287/05000 (65.74%) | loss: 3.905759 | lrm: 0.55 | dt: 1559.58ms | tok/sec: 10,505 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 58 | total time: 86.91m | eta: 45.4m
step 03288/05000 (65.76%) | loss: 3.902795 | lrm: 0.55 | dt: 1561.17ms | tok/sec: 10,494 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 58 | total time: 86.94m | eta: 45.4m
step 03289/05000 (65.78%) | loss: 3.914428 | lrm: 0.55 | dt: 1526.14ms | tok/sec: 10,735 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 58 | total time: 86.96m | eta: 45.4m
step 03290/05000 (65.80%) | loss: 3.905506 | lrm: 0.55 | dt: 1530.09ms | tok/sec: 10,707 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 58 | total time: 86.99m | eta: 45.4m
step 03291/05000 (65.82%) | loss: 3.897411 | lrm: 0.55 | dt: 1552.13ms | tok/sec: 10,555 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 58 | total time: 87.01m | eta: 45.3m
step 03292/05000 (65.84%) | loss: 3.913185 | lrm: 0.55 | dt: 1549.17ms | tok/sec: 10,576 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 58 | total time: 87.04m | eta: 45.3m
step 03293/05000 (65.86%) | loss: 3.906683 | lrm: 0.55 | dt: 1546.63ms | tok/sec: 10,593 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 58 | total time: 87.07m | eta: 45.3m
step 03294/05000 (65.88%) | loss: 3.905539 | lrm: 0.55 | dt: 1569.10ms | tok/sec: 10,441 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 59 | total time: 87.09m | eta: 45.2m
step 03295/05000 (65.90%) | loss: 3.901427 | lrm: 0.55 | dt: 1564.10ms | tok/sec: 10,475 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 59 | total time: 87.12m | eta: 45.2m
step 03296/05000 (65.92%) | loss: 3.870790 | lrm: 0.55 | dt: 1510.47ms | tok/sec: 10,846 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 59 | total time: 87.14m | eta: 45.2m
step 03297/05000 (65.94%) | loss: 3.886573 | lrm: 0.55 | dt: 1558.63ms | tok/sec: 10,511 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 59 | total time: 87.17m | eta: 45.2m
step 03298/05000 (65.96%) | loss: 3.891144 | lrm: 0.55 | dt: 1571.85ms | tok/sec: 10,423 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 59 | total time: 87.20m | eta: 45.1m
step 03299/05000 (65.98%) | loss: 3.872892 | lrm: 0.55 | dt: 1568.92ms | tok/sec: 10,442 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 59 | total time: 87.22m | eta: 45.1m
Step 03300 | Validation bpb: 1.211869
<|bos|>The capital of France is a very small city in the world. The city is a very small city in
<|bos|>The chemical symbol of gold is the symbol of the gold. The symbol of gold is the symbol of the gold
<|bos|>If yesterday was Friday, then tomorrow will be Friday, then Friday, then Friday, then Friday, then Friday, then Friday
<|bos|>The opposite of hot is the heat. The heat is the heat that is produced by the heat. The
<|bos|>The planets of the solar system are: the solar system, the solar system, the solar system, the solar system,
<|bos|>My favorite color is the red, green, and blue. I have a lot of my favorite colors
<|bos|>If 5*x + 3 = 13, then x is 1.5. If 3 = 13, then x is
step 03300/05000 (66.00%) | loss: 3.862840 | lrm: 0.55 | dt: 1660.01ms | tok/sec: 9,869 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 59 | total time: 87.25m | eta: 45.1m
step 03301/05000 (66.02%) | loss: 3.871123 | lrm: 0.55 | dt: 1553.51ms | tok/sec: 10,546 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 59 | total time: 87.27m | eta: 45.1m
step 03302/05000 (66.04%) | loss: 3.867512 | lrm: 0.55 | dt: 1542.69ms | tok/sec: 10,620 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 59 | total time: 87.30m | eta: 45.0m
step 03303/05000 (66.06%) | loss: 3.883005 | lrm: 0.55 | dt: 1572.93ms | tok/sec: 10,416 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 59 | total time: 87.33m | eta: 45.0m
step 03304/05000 (66.08%) | loss: 3.899413 | lrm: 0.55 | dt: 1558.93ms | tok/sec: 10,509 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 59 | total time: 87.35m | eta: 45.0m
step 03305/05000 (66.10%) | loss: 3.914514 | lrm: 0.55 | dt: 1545.52ms | tok/sec: 10,600 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 59 | total time: 87.38m | eta: 44.9m
step 03306/05000 (66.12%) | loss: 3.902962 | lrm: 0.55 | dt: 1547.07ms | tok/sec: 10,590 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 59 | total time: 87.40m | eta: 44.9m
step 03307/05000 (66.14%) | loss: 3.902316 | lrm: 0.54 | dt: 1565.45ms | tok/sec: 10,465 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 59 | total time: 87.43m | eta: 44.9m
step 03308/05000 (66.16%) | loss: 3.897545 | lrm: 0.54 | dt: 1557.86ms | tok/sec: 10,516 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 59 | total time: 87.46m | eta: 44.9m
step 03309/05000 (66.18%) | loss: 3.909155 | lrm: 0.54 | dt: 1560.81ms | tok/sec: 10,497 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 59 | total time: 87.48m | eta: 44.8m
step 03310/05000 (66.20%) | loss: 3.920960 | lrm: 0.54 | dt: 1545.64ms | tok/sec: 10,600 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 59 | total time: 87.51m | eta: 44.8m
step 03311/05000 (66.22%) | loss: 3.934299 | lrm: 0.54 | dt: 1543.44ms | tok/sec: 10,615 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 59 | total time: 87.53m | eta: 44.8m
step 03312/05000 (66.24%) | loss: 3.938016 | lrm: 0.54 | dt: 1549.79ms | tok/sec: 10,571 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 59 | total time: 87.56m | eta: 44.8m
step 03313/05000 (66.26%) | loss: 3.937208 | lrm: 0.54 | dt: 1545.31ms | tok/sec: 10,602 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 59 | total time: 87.59m | eta: 44.7m
step 03314/05000 (66.28%) | loss: 3.938660 | lrm: 0.54 | dt: 1522.28ms | tok/sec: 10,762 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 59 | total time: 87.61m | eta: 44.7m
step 03315/05000 (66.30%) | loss: 3.927682 | lrm: 0.54 | dt: 1537.40ms | tok/sec: 10,656 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 59 | total time: 87.64m | eta: 44.7m
step 03316/05000 (66.32%) | loss: 3.941351 | lrm: 0.54 | dt: 1587.90ms | tok/sec: 10,318 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 59 | total time: 87.66m | eta: 44.7m
step 03317/05000 (66.34%) | loss: 3.923726 | lrm: 0.54 | dt: 1554.91ms | tok/sec: 10,536 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 60 | total time: 87.69m | eta: 44.6m
step 03318/05000 (66.36%) | loss: 3.940261 | lrm: 0.54 | dt: 1518.01ms | tok/sec: 10,793 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 60 | total time: 87.71m | eta: 44.6m
step 03319/05000 (66.38%) | loss: 3.946349 | lrm: 0.54 | dt: 1540.27ms | tok/sec: 10,637 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 60 | total time: 87.74m | eta: 44.6m
step 03320/05000 (66.40%) | loss: 3.951637 | lrm: 0.54 | dt: 1538.95ms | tok/sec: 10,646 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 60 | total time: 87.77m | eta: 44.5m
step 03321/05000 (66.42%) | loss: 3.957615 | lrm: 0.54 | dt: 1513.84ms | tok/sec: 10,822 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 60 | total time: 87.79m | eta: 44.5m
step 03322/05000 (66.44%) | loss: 3.943182 | lrm: 0.54 | dt: 1554.84ms | tok/sec: 10,537 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 60 | total time: 87.82m | eta: 44.5m
step 03323/05000 (66.46%) | loss: 3.940151 | lrm: 0.54 | dt: 1508.28ms | tok/sec: 10,862 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 60 | total time: 87.84m | eta: 44.5m
step 03324/05000 (66.48%) | loss: 3.946505 | lrm: 0.54 | dt: 1545.10ms | tok/sec: 10,603 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 60 | total time: 87.87m | eta: 44.4m
step 03325/05000 (66.50%) | loss: 3.941071 | lrm: 0.54 | dt: 1555.71ms | tok/sec: 10,531 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 60 | total time: 87.89m | eta: 44.4m
step 03326/05000 (66.52%) | loss: 3.945822 | lrm: 0.54 | dt: 1523.31ms | tok/sec: 10,755 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 60 | total time: 87.92m | eta: 44.4m
step 03327/05000 (66.54%) | loss: 3.941858 | lrm: 0.54 | dt: 1637.71ms | tok/sec: 10,004 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 60 | total time: 87.95m | eta: 44.4m
step 03328/05000 (66.56%) | loss: 3.926003 | lrm: 0.54 | dt: 1614.29ms | tok/sec: 10,149 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 60 | total time: 87.97m | eta: 44.3m
step 03329/05000 (66.58%) | loss: 3.919192 | lrm: 0.54 | dt: 1641.06ms | tok/sec: 9,983 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 60 | total time: 88.00m | eta: 44.3m
step 03330/05000 (66.60%) | loss: 3.926757 | lrm: 0.54 | dt: 1616.86ms | tok/sec: 10,133 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 60 | total time: 88.03m | eta: 44.3m
step 03331/05000 (66.62%) | loss: 3.921941 | lrm: 0.54 | dt: 1583.82ms | tok/sec: 10,344 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 60 | total time: 88.05m | eta: 44.3m
step 03332/05000 (66.64%) | loss: 3.906436 | lrm: 0.54 | dt: 1631.35ms | tok/sec: 10,043 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 60 | total time: 88.08m | eta: 44.2m
step 03333/05000 (66.66%) | loss: 3.912941 | lrm: 0.54 | dt: 1655.15ms | tok/sec: 9,898 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 60 | total time: 88.11m | eta: 44.2m
step 03334/05000 (66.68%) | loss: 3.916114 | lrm: 0.54 | dt: 1537.01ms | tok/sec: 10,659 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 60 | total time: 88.13m | eta: 44.2m
step 03335/05000 (66.70%) | loss: 3.922823 | lrm: 0.54 | dt: 1598.03ms | tok/sec: 10,252 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 60 | total time: 88.16m | eta: 44.1m
step 03336/05000 (66.72%) | loss: 3.928185 | lrm: 0.54 | dt: 1606.49ms | tok/sec: 10,198 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 60 | total time: 88.19m | eta: 44.1m
step 03337/05000 (66.74%) | loss: 3.925622 | lrm: 0.54 | dt: 1636.33ms | tok/sec: 10,012 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 60 | total time: 88.21m | eta: 44.1m
step 03338/05000 (66.76%) | loss: 3.923513 | lrm: 0.54 | dt: 1615.50ms | tok/sec: 10,141 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 60 | total time: 88.24m | eta: 44.1m
step 03339/05000 (66.78%) | loss: 3.915943 | lrm: 0.54 | dt: 1596.95ms | tok/sec: 10,259 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 60 | total time: 88.27m | eta: 44.0m
step 03340/05000 (66.80%) | loss: 3.931244 | lrm: 0.54 | dt: 1520.10ms | tok/sec: 10,778 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 61 | total time: 88.29m | eta: 44.0m
step 03341/05000 (66.82%) | loss: 3.930365 | lrm: 0.53 | dt: 1549.37ms | tok/sec: 10,574 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 61 | total time: 88.32m | eta: 44.0m
step 03342/05000 (66.84%) | loss: 3.931684 | lrm: 0.53 | dt: 1546.85ms | tok/sec: 10,591 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 61 | total time: 88.35m | eta: 44.0m
step 03343/05000 (66.86%) | loss: 3.920932 | lrm: 0.53 | dt: 1532.46ms | tok/sec: 10,691 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 61 | total time: 88.37m | eta: 43.9m
step 03344/05000 (66.88%) | loss: 3.907778 | lrm: 0.53 | dt: 1565.54ms | tok/sec: 10,465 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 61 | total time: 88.40m | eta: 43.9m
step 03345/05000 (66.90%) | loss: 3.907150 | lrm: 0.53 | dt: 1538.26ms | tok/sec: 10,651 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 61 | total time: 88.42m | eta: 43.9m
step 03346/05000 (66.92%) | loss: 3.900113 | lrm: 0.53 | dt: 1552.60ms | tok/sec: 10,552 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 61 | total time: 88.45m | eta: 43.9m
step 03347/05000 (66.94%) | loss: 3.914406 | lrm: 0.53 | dt: 1508.38ms | tok/sec: 10,861 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 61 | total time: 88.47m | eta: 43.8m
step 03348/05000 (66.96%) | loss: 3.911728 | lrm: 0.53 | dt: 1546.65ms | tok/sec: 10,593 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 61 | total time: 88.50m | eta: 43.8m
step 03349/05000 (66.98%) | loss: 3.911194 | lrm: 0.53 | dt: 1560.56ms | tok/sec: 10,498 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 61 | total time: 88.53m | eta: 43.8m
step 03350/05000 (67.00%) | loss: 3.902430 | lrm: 0.53 | dt: 1517.47ms | tok/sec: 10,796 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 61 | total time: 88.55m | eta: 43.7m
step 03351/05000 (67.02%) | loss: 3.879075 | lrm: 0.53 | dt: 1546.22ms | tok/sec: 10,596 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 61 | total time: 88.58m | eta: 43.7m
step 03352/05000 (67.04%) | loss: 3.881705 | lrm: 0.53 | dt: 1545.74ms | tok/sec: 10,599 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 61 | total time: 88.60m | eta: 43.7m
step 03353/05000 (67.06%) | loss: 3.891454 | lrm: 0.53 | dt: 1510.32ms | tok/sec: 10,848 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 61 | total time: 88.63m | eta: 43.7m
step 03354/05000 (67.08%) | loss: 3.906876 | lrm: 0.53 | dt: 1566.92ms | tok/sec: 10,456 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 61 | total time: 88.65m | eta: 43.6m
step 03355/05000 (67.10%) | loss: 3.889000 | lrm: 0.53 | dt: 1559.39ms | tok/sec: 10,506 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 61 | total time: 88.68m | eta: 43.6m
step 03356/05000 (67.12%) | loss: 3.898715 | lrm: 0.53 | dt: 1554.57ms | tok/sec: 10,539 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 61 | total time: 88.71m | eta: 43.6m
step 03357/05000 (67.14%) | loss: 3.901471 | lrm: 0.53 | dt: 1539.63ms | tok/sec: 10,641 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 61 | total time: 88.73m | eta: 43.6m
step 03358/05000 (67.16%) | loss: 3.907560 | lrm: 0.53 | dt: 1549.12ms | tok/sec: 10,576 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 61 | total time: 88.76m | eta: 43.5m
step 03359/05000 (67.18%) | loss: 3.901481 | lrm: 0.53 | dt: 1548.57ms | tok/sec: 10,580 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 61 | total time: 88.78m | eta: 43.5m
step 03360/05000 (67.20%) | loss: 3.889956 | lrm: 0.53 | dt: 1543.91ms | tok/sec: 10,612 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 61 | total time: 88.81m | eta: 43.5m
step 03361/05000 (67.22%) | loss: 3.883660 | lrm: 0.53 | dt: 1506.80ms | tok/sec: 10,873 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 61 | total time: 88.83m | eta: 43.4m
step 03362/05000 (67.24%) | loss: 3.882375 | lrm: 0.53 | dt: 1627.32ms | tok/sec: 10,068 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 61 | total time: 88.86m | eta: 43.4m
step 03363/05000 (67.26%) | loss: 3.891746 | lrm: 0.53 | dt: 1560.73ms | tok/sec: 10,497 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 62 | total time: 88.89m | eta: 43.4m
step 03364/05000 (67.28%) | loss: 3.898972 | lrm: 0.53 | dt: 1582.52ms | tok/sec: 10,353 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 62 | total time: 88.91m | eta: 43.4m
step 03365/05000 (67.30%) | loss: 3.905076 | lrm: 0.53 | dt: 1577.80ms | tok/sec: 10,384 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 62 | total time: 88.94m | eta: 43.3m
step 03366/05000 (67.32%) | loss: 3.897234 | lrm: 0.53 | dt: 1625.35ms | tok/sec: 10,080 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 62 | total time: 88.97m | eta: 43.3m
step 03367/05000 (67.34%) | loss: 3.913054 | lrm: 0.53 | dt: 1553.57ms | tok/sec: 10,546 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 62 | total time: 88.99m | eta: 43.3m
step 03368/05000 (67.36%) | loss: 3.897935 | lrm: 0.53 | dt: 1515.71ms | tok/sec: 10,809 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 62 | total time: 89.02m | eta: 43.3m
step 03369/05000 (67.38%) | loss: 3.882456 | lrm: 0.53 | dt: 1557.56ms | tok/sec: 10,518 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 62 | total time: 89.04m | eta: 43.2m
step 03370/05000 (67.40%) | loss: 3.879182 | lrm: 0.53 | dt: 1502.35ms | tok/sec: 10,905 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 62 | total time: 89.07m | eta: 43.2m
step 03371/05000 (67.42%) | loss: 3.884690 | lrm: 0.53 | dt: 1555.73ms | tok/sec: 10,531 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 62 | total time: 89.09m | eta: 43.2m
step 03372/05000 (67.44%) | loss: 3.879820 | lrm: 0.53 | dt: 1556.17ms | tok/sec: 10,528 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 62 | total time: 89.12m | eta: 43.2m
step 03373/05000 (67.46%) | loss: 3.874126 | lrm: 0.53 | dt: 1540.32ms | tok/sec: 10,636 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 62 | total time: 89.15m | eta: 43.1m
step 03374/05000 (67.48%) | loss: 3.870361 | lrm: 0.53 | dt: 1561.03ms | tok/sec: 10,495 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 62 | total time: 89.17m | eta: 43.1m
step 03375/05000 (67.50%) | loss: 3.878780 | lrm: 0.53 | dt: 1563.22ms | tok/sec: 10,480 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 62 | total time: 89.20m | eta: 43.1m
step 03376/05000 (67.52%) | loss: 3.893581 | lrm: 0.52 | dt: 1516.56ms | tok/sec: 10,803 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 62 | total time: 89.22m | eta: 43.0m
step 03377/05000 (67.54%) | loss: 3.884440 | lrm: 0.52 | dt: 1533.98ms | tok/sec: 10,680 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 62 | total time: 89.25m | eta: 43.0m
step 03378/05000 (67.56%) | loss: 3.885372 | lrm: 0.52 | dt: 1564.64ms | tok/sec: 10,471 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 62 | total time: 89.27m | eta: 43.0m
step 03379/05000 (67.58%) | loss: 3.879681 | lrm: 0.52 | dt: 1568.38ms | tok/sec: 10,446 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 62 | total time: 89.30m | eta: 43.0m
step 03380/05000 (67.60%) | loss: 3.884821 | lrm: 0.52 | dt: 1537.58ms | tok/sec: 10,655 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 62 | total time: 89.33m | eta: 42.9m
step 03381/05000 (67.62%) | loss: 3.891278 | lrm: 0.52 | dt: 1520.72ms | tok/sec: 10,773 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 62 | total time: 89.35m | eta: 42.9m
step 03382/05000 (67.64%) | loss: 3.897615 | lrm: 0.52 | dt: 1535.50ms | tok/sec: 10,670 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 62 | total time: 89.38m | eta: 42.9m
step 03383/05000 (67.66%) | loss: 3.894254 | lrm: 0.52 | dt: 1556.94ms | tok/sec: 10,523 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 62 | total time: 89.40m | eta: 42.9m
step 03384/05000 (67.68%) | loss: 3.879416 | lrm: 0.52 | dt: 1551.91ms | tok/sec: 10,557 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 62 | total time: 89.43m | eta: 42.8m
step 03385/05000 (67.70%) | loss: 3.894993 | lrm: 0.52 | dt: 1554.76ms | tok/sec: 10,537 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 62 | total time: 89.46m | eta: 42.8m
step 03386/05000 (67.72%) | loss: 3.888328 | lrm: 0.52 | dt: 1561.53ms | tok/sec: 10,492 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 63 | total time: 89.48m | eta: 42.8m
step 03387/05000 (67.74%) | loss: 3.890712 | lrm: 0.52 | dt: 1570.00ms | tok/sec: 10,435 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 63 | total time: 89.51m | eta: 42.8m
step 03388/05000 (67.76%) | loss: 3.879636 | lrm: 0.52 | dt: 1511.52ms | tok/sec: 10,839 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 63 | total time: 89.53m | eta: 42.7m
step 03389/05000 (67.78%) | loss: 3.886007 | lrm: 0.52 | dt: 1517.24ms | tok/sec: 10,798 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 63 | total time: 89.56m | eta: 42.7m
step 03390/05000 (67.80%) | loss: 3.894800 | lrm: 0.52 | dt: 1573.51ms | tok/sec: 10,412 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 63 | total time: 89.58m | eta: 42.7m
step 03391/05000 (67.82%) | loss: 3.909939 | lrm: 0.52 | dt: 1510.06ms | tok/sec: 10,849 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 63 | total time: 89.61m | eta: 42.6m
step 03392/05000 (67.84%) | loss: 3.912199 | lrm: 0.52 | dt: 1574.11ms | tok/sec: 10,408 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 63 | total time: 89.64m | eta: 42.6m
step 03393/05000 (67.86%) | loss: 3.907018 | lrm: 0.52 | dt: 1568.22ms | tok/sec: 10,447 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 63 | total time: 89.66m | eta: 42.6m
step 03394/05000 (67.88%) | loss: 3.888886 | lrm: 0.52 | dt: 1557.45ms | tok/sec: 10,519 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 63 | total time: 89.69m | eta: 42.6m
step 03395/05000 (67.90%) | loss: 3.889188 | lrm: 0.52 | dt: 1563.04ms | tok/sec: 10,482 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 63 | total time: 89.71m | eta: 42.5m
step 03396/05000 (67.92%) | loss: 3.884770 | lrm: 0.52 | dt: 1544.42ms | tok/sec: 10,608 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 63 | total time: 89.74m | eta: 42.5m
step 03397/05000 (67.94%) | loss: 3.872393 | lrm: 0.52 | dt: 1543.80ms | tok/sec: 10,612 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 63 | total time: 89.77m | eta: 42.5m
step 03398/05000 (67.96%) | loss: 3.874722 | lrm: 0.52 | dt: 1565.78ms | tok/sec: 10,463 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 63 | total time: 89.79m | eta: 42.5m
step 03399/05000 (67.98%) | loss: 3.876065 | lrm: 0.52 | dt: 1530.00ms | tok/sec: 10,708 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 63 | total time: 89.82m | eta: 42.4m
Step 03400 | Validation bpb: 1.208286
<|bos|>The capital of France is the largest city in the world. The city is the largest city in the world
<|bos|>The chemical symbol of gold is the gold, gold, gold, gold, gold, gold, gold, gold
<|bos|>If yesterday was Friday, then tomorrow will be the first time to see the world's largest world. The world's largest world
<|bos|>The opposite of hot is the same as the cold. The hot is the same as the cold. The
<|bos|>The planets of the solar system are: the solar system, the solar system, the solar system, the solar system,
<|bos|>My favorite color is the blue. I have a blue color. I have a blue color. I
<|bos|>If 5*x + 3 = 13, then x is 3.5*x + 3.5*x + 3
step 03400/05000 (68.00%) | loss: 3.876046 | lrm: 0.52 | dt: 1622.69ms | tok/sec: 10,096 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 63 | total time: 89.84m | eta: 42.4m
step 03401/05000 (68.02%) | loss: 3.885595 | lrm: 0.52 | dt: 1543.20ms | tok/sec: 10,616 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 63 | total time: 89.87m | eta: 42.4m
step 03402/05000 (68.04%) | loss: 3.892494 | lrm: 0.52 | dt: 1570.81ms | tok/sec: 10,430 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 63 | total time: 89.90m | eta: 42.4m
step 03403/05000 (68.06%) | loss: 3.891784 | lrm: 0.52 | dt: 1545.56ms | tok/sec: 10,600 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 63 | total time: 89.92m | eta: 42.3m
step 03404/05000 (68.08%) | loss: 3.899631 | lrm: 0.52 | dt: 1619.25ms | tok/sec: 10,118 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 63 | total time: 89.95m | eta: 42.3m
step 03405/05000 (68.10%) | loss: 3.900666 | lrm: 0.52 | dt: 1604.56ms | tok/sec: 10,210 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 63 | total time: 89.98m | eta: 42.3m
step 03406/05000 (68.12%) | loss: 3.898057 | lrm: 0.52 | dt: 1613.85ms | tok/sec: 10,152 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 63 | total time: 90.00m | eta: 42.2m
step 03407/05000 (68.14%) | loss: 3.894192 | lrm: 0.52 | dt: 1592.31ms | tok/sec: 10,289 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 63 | total time: 90.03m | eta: 42.2m
step 03408/05000 (68.16%) | loss: 3.911508 | lrm: 0.52 | dt: 1651.35ms | tok/sec: 9,921 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 63 | total time: 90.06m | eta: 42.2m
step 03409/05000 (68.18%) | loss: 3.930745 | lrm: 0.52 | dt: 1631.11ms | tok/sec: 10,044 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 63 | total time: 90.08m | eta: 42.2m
step 03410/05000 (68.20%) | loss: 3.930105 | lrm: 0.51 | dt: 1648.01ms | tok/sec: 9,941 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 64 | total time: 90.11m | eta: 42.1m
step 03411/05000 (68.22%) | loss: 3.916980 | lrm: 0.51 | dt: 1639.56ms | tok/sec: 9,992 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 64 | total time: 90.14m | eta: 42.1m
step 03412/05000 (68.24%) | loss: 3.916633 | lrm: 0.51 | dt: 1723.26ms | tok/sec: 9,507 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 64 | total time: 90.17m | eta: 42.1m
step 03413/05000 (68.26%) | loss: 3.926646 | lrm: 0.51 | dt: 1727.09ms | tok/sec: 9,486 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 64 | total time: 90.20m | eta: 42.1m
step 03414/05000 (68.28%) | loss: 3.928448 | lrm: 0.51 | dt: 1636.47ms | tok/sec: 10,011 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 64 | total time: 90.22m | eta: 42.0m
step 03415/05000 (68.30%) | loss: 3.914093 | lrm: 0.51 | dt: 1675.90ms | tok/sec: 9,776 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 64 | total time: 90.25m | eta: 42.0m
step 03416/05000 (68.32%) | loss: 3.914861 | lrm: 0.51 | dt: 1720.12ms | tok/sec: 9,524 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 64 | total time: 90.28m | eta: 42.0m
step 03417/05000 (68.34%) | loss: 3.925931 | lrm: 0.51 | dt: 1728.88ms | tok/sec: 9,476 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 64 | total time: 90.31m | eta: 42.0m
step 03418/05000 (68.36%) | loss: 3.927060 | lrm: 0.51 | dt: 1659.63ms | tok/sec: 9,872 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 64 | total time: 90.34m | eta: 41.9m
step 03419/05000 (68.38%) | loss: 3.927057 | lrm: 0.51 | dt: 1656.58ms | tok/sec: 9,890 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 64 | total time: 90.36m | eta: 41.9m
step 03420/05000 (68.40%) | loss: 3.934594 | lrm: 0.51 | dt: 1595.51ms | tok/sec: 10,268 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 64 | total time: 90.39m | eta: 41.9m
step 03421/05000 (68.42%) | loss: 3.938303 | lrm: 0.51 | dt: 1675.93ms | tok/sec: 9,776 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 64 | total time: 90.42m | eta: 41.9m
step 03422/05000 (68.44%) | loss: 3.925731 | lrm: 0.51 | dt: 1633.60ms | tok/sec: 10,029 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 64 | total time: 90.45m | eta: 41.8m
step 03423/05000 (68.46%) | loss: 3.918723 | lrm: 0.51 | dt: 1656.25ms | tok/sec: 9,892 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 64 | total time: 90.47m | eta: 41.8m
step 03424/05000 (68.48%) | loss: 3.918582 | lrm: 0.51 | dt: 1592.96ms | tok/sec: 10,285 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 64 | total time: 90.50m | eta: 41.8m
step 03425/05000 (68.50%) | loss: 3.902677 | lrm: 0.51 | dt: 1537.61ms | tok/sec: 10,655 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 64 | total time: 90.53m | eta: 41.8m
step 03426/05000 (68.52%) | loss: 3.912868 | lrm: 0.51 | dt: 1556.61ms | tok/sec: 10,525 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 64 | total time: 90.55m | eta: 41.7m
step 03427/05000 (68.54%) | loss: 3.910286 | lrm: 0.51 | dt: 1570.85ms | tok/sec: 10,430 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 64 | total time: 90.58m | eta: 41.7m
step 03428/05000 (68.56%) | loss: 3.892756 | lrm: 0.51 | dt: 1590.95ms | tok/sec: 10,298 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 64 | total time: 90.60m | eta: 41.7m
step 03429/05000 (68.58%) | loss: 3.907996 | lrm: 0.51 | dt: 1576.92ms | tok/sec: 10,389 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 64 | total time: 90.63m | eta: 41.6m
step 03430/05000 (68.60%) | loss: 3.913982 | lrm: 0.51 | dt: 1571.78ms | tok/sec: 10,423 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 64 | total time: 90.66m | eta: 41.6m
step 03431/05000 (68.62%) | loss: 3.906827 | lrm: 0.51 | dt: 1574.92ms | tok/sec: 10,403 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 64 | total time: 90.68m | eta: 41.6m
step 03432/05000 (68.64%) | loss: 3.916367 | lrm: 0.51 | dt: 1640.34ms | tok/sec: 9,988 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 64 | total time: 90.71m | eta: 41.6m
step 03433/05000 (68.66%) | loss: 3.913598 | lrm: 0.51 | dt: 1644.21ms | tok/sec: 9,964 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 65 | total time: 90.74m | eta: 41.5m
step 03434/05000 (68.68%) | loss: 3.916219 | lrm: 0.51 | dt: 1661.59ms | tok/sec: 9,860 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 65 | total time: 90.77m | eta: 41.5m
step 03435/05000 (68.70%) | loss: 3.905424 | lrm: 0.51 | dt: 1610.07ms | tok/sec: 10,175 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 65 | total time: 90.79m | eta: 41.5m
step 03436/05000 (68.72%) | loss: 3.903916 | lrm: 0.51 | dt: 1646.26ms | tok/sec: 9,952 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 65 | total time: 90.82m | eta: 41.5m
step 03437/05000 (68.74%) | loss: 3.907651 | lrm: 0.51 | dt: 1638.88ms | tok/sec: 9,997 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 65 | total time: 90.85m | eta: 41.4m
step 03438/05000 (68.76%) | loss: 3.908066 | lrm: 0.51 | dt: 1590.06ms | tok/sec: 10,304 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 65 | total time: 90.87m | eta: 41.4m
step 03439/05000 (68.78%) | loss: 3.894304 | lrm: 0.51 | dt: 1599.89ms | tok/sec: 10,240 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 65 | total time: 90.90m | eta: 41.4m
step 03440/05000 (68.80%) | loss: 3.894389 | lrm: 0.51 | dt: 1617.50ms | tok/sec: 10,129 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 65 | total time: 90.93m | eta: 41.4m
step 03441/05000 (68.82%) | loss: 3.901460 | lrm: 0.51 | dt: 1629.57ms | tok/sec: 10,054 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 65 | total time: 90.95m | eta: 41.3m
step 03442/05000 (68.84%) | loss: 3.887572 | lrm: 0.51 | dt: 1612.99ms | tok/sec: 10,157 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 65 | total time: 90.98m | eta: 41.3m
step 03443/05000 (68.86%) | loss: 3.894771 | lrm: 0.51 | dt: 1610.80ms | tok/sec: 10,171 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 65 | total time: 91.01m | eta: 41.3m
step 03444/05000 (68.88%) | loss: 3.887670 | lrm: 0.50 | dt: 1562.32ms | tok/sec: 10,486 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 65 | total time: 91.03m | eta: 41.2m
step 03445/05000 (68.90%) | loss: 3.871026 | lrm: 0.50 | dt: 1583.46ms | tok/sec: 10,346 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 65 | total time: 91.06m | eta: 41.2m
step 03446/05000 (68.92%) | loss: 3.880599 | lrm: 0.50 | dt: 1583.23ms | tok/sec: 10,348 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 65 | total time: 91.09m | eta: 41.2m
step 03447/05000 (68.94%) | loss: 3.879049 | lrm: 0.50 | dt: 1567.76ms | tok/sec: 10,450 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 65 | total time: 91.11m | eta: 41.2m
step 03448/05000 (68.96%) | loss: 3.891016 | lrm: 0.50 | dt: 1578.62ms | tok/sec: 10,378 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 65 | total time: 91.14m | eta: 41.1m
step 03449/05000 (68.98%) | loss: 3.896821 | lrm: 0.50 | dt: 1547.03ms | tok/sec: 10,590 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 65 | total time: 91.16m | eta: 41.1m
step 03450/05000 (69.00%) | loss: 3.887322 | lrm: 0.50 | dt: 1546.32ms | tok/sec: 10,595 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 65 | total time: 91.19m | eta: 41.1m
step 03451/05000 (69.02%) | loss: 3.867509 | lrm: 0.50 | dt: 1555.69ms | tok/sec: 10,531 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 65 | total time: 91.22m | eta: 41.1m
step 03452/05000 (69.04%) | loss: 3.881575 | lrm: 0.50 | dt: 1569.14ms | tok/sec: 10,441 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 65 | total time: 91.24m | eta: 41.0m
step 03453/05000 (69.06%) | loss: 3.867314 | lrm: 0.50 | dt: 1549.16ms | tok/sec: 10,576 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 65 | total time: 91.27m | eta: 41.0m
step 03454/05000 (69.08%) | loss: 3.883541 | lrm: 0.50 | dt: 1529.64ms | tok/sec: 10,710 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 65 | total time: 91.29m | eta: 41.0m
step 03455/05000 (69.10%) | loss: 3.881041 | lrm: 0.50 | dt: 1530.01ms | tok/sec: 10,708 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 65 | total time: 91.32m | eta: 41.0m
step 03456/05000 (69.12%) | loss: 3.859470 | lrm: 0.50 | dt: 1545.91ms | tok/sec: 10,598 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 66 | total time: 91.35m | eta: 40.9m
step 03457/05000 (69.14%) | loss: 3.858354 | lrm: 0.50 | dt: 1561.76ms | tok/sec: 10,490 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 66 | total time: 91.37m | eta: 40.9m
step 03458/05000 (69.16%) | loss: 3.876112 | lrm: 0.50 | dt: 1551.63ms | tok/sec: 10,559 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 66 | total time: 91.40m | eta: 40.9m
step 03459/05000 (69.18%) | loss: 3.870430 | lrm: 0.50 | dt: 1579.19ms | tok/sec: 10,374 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 66 | total time: 91.42m | eta: 40.8m
step 03460/05000 (69.20%) | loss: 3.873316 | lrm: 0.50 | dt: 1562.35ms | tok/sec: 10,486 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 66 | total time: 91.45m | eta: 40.8m
step 03461/05000 (69.22%) | loss: 3.877633 | lrm: 0.50 | dt: 1570.38ms | tok/sec: 10,433 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 66 | total time: 91.48m | eta: 40.8m
step 03462/05000 (69.24%) | loss: 3.875828 | lrm: 0.50 | dt: 1553.38ms | tok/sec: 10,547 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 66 | total time: 91.50m | eta: 40.8m
step 03463/05000 (69.26%) | loss: 3.881522 | lrm: 0.50 | dt: 1569.22ms | tok/sec: 10,440 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 66 | total time: 91.53m | eta: 40.7m
step 03464/05000 (69.28%) | loss: 3.887653 | lrm: 0.50 | dt: 1528.38ms | tok/sec: 10,719 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 66 | total time: 91.55m | eta: 40.7m
step 03465/05000 (69.30%) | loss: 3.893349 | lrm: 0.50 | dt: 1566.85ms | tok/sec: 10,456 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 66 | total time: 91.58m | eta: 40.7m
step 03466/05000 (69.32%) | loss: 3.886828 | lrm: 0.50 | dt: 1563.97ms | tok/sec: 10,475 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 66 | total time: 91.61m | eta: 40.7m
step 03467/05000 (69.34%) | loss: 3.899892 | lrm: 0.50 | dt: 1529.00ms | tok/sec: 10,715 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 66 | total time: 91.63m | eta: 40.6m
step 03468/05000 (69.36%) | loss: 3.906866 | lrm: 0.50 | dt: 1561.92ms | tok/sec: 10,489 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 66 | total time: 91.66m | eta: 40.6m
step 03469/05000 (69.38%) | loss: 3.905643 | lrm: 0.50 | dt: 1561.46ms | tok/sec: 10,492 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 66 | total time: 91.68m | eta: 40.6m
step 03470/05000 (69.40%) | loss: 3.900347 | lrm: 0.50 | dt: 1536.25ms | tok/sec: 10,664 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 66 | total time: 91.71m | eta: 40.6m
step 03471/05000 (69.42%) | loss: 3.883193 | lrm: 0.50 | dt: 1529.89ms | tok/sec: 10,709 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 66 | total time: 91.73m | eta: 40.5m
step 03472/05000 (69.44%) | loss: 3.882052 | lrm: 0.50 | dt: 1531.69ms | tok/sec: 10,696 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 66 | total time: 91.76m | eta: 40.5m
step 03473/05000 (69.46%) | loss: 3.887979 | lrm: 0.50 | dt: 1521.63ms | tok/sec: 10,767 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 66 | total time: 91.78m | eta: 40.5m
step 03474/05000 (69.48%) | loss: 3.911860 | lrm: 0.50 | dt: 1536.65ms | tok/sec: 10,662 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 66 | total time: 91.81m | eta: 40.4m
step 03475/05000 (69.50%) | loss: 3.925877 | lrm: 0.50 | dt: 1547.66ms | tok/sec: 10,586 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 66 | total time: 91.84m | eta: 40.4m
step 03476/05000 (69.52%) | loss: 3.904934 | lrm: 0.50 | dt: 1501.79ms | tok/sec: 10,909 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 66 | total time: 91.86m | eta: 40.4m
step 03477/05000 (69.54%) | loss: 3.898957 | lrm: 0.50 | dt: 1528.74ms | tok/sec: 10,717 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 66 | total time: 91.89m | eta: 40.4m
step 03478/05000 (69.56%) | loss: 3.903179 | lrm: 0.49 | dt: 1619.79ms | tok/sec: 10,114 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 66 | total time: 91.91m | eta: 40.3m
step 03479/05000 (69.58%) | loss: 3.908263 | lrm: 0.49 | dt: 1589.26ms | tok/sec: 10,309 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 67 | total time: 91.94m | eta: 40.3m
step 03480/05000 (69.60%) | loss: 3.910897 | lrm: 0.49 | dt: 1534.48ms | tok/sec: 10,677 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 67 | total time: 91.97m | eta: 40.3m
step 03481/05000 (69.62%) | loss: 3.904156 | lrm: 0.49 | dt: 1600.06ms | tok/sec: 10,239 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 67 | total time: 91.99m | eta: 40.3m
step 03482/05000 (69.64%) | loss: 3.888896 | lrm: 0.49 | dt: 1638.45ms | tok/sec: 9,999 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 67 | total time: 92.02m | eta: 40.2m
step 03483/05000 (69.66%) | loss: 3.901830 | lrm: 0.49 | dt: 1680.88ms | tok/sec: 9,747 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 67 | total time: 92.05m | eta: 40.2m
step 03484/05000 (69.68%) | loss: 3.910957 | lrm: 0.49 | dt: 1572.00ms | tok/sec: 10,422 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 67 | total time: 92.07m | eta: 40.2m
step 03485/05000 (69.70%) | loss: 3.898041 | lrm: 0.49 | dt: 1609.27ms | tok/sec: 10,181 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 67 | total time: 92.10m | eta: 40.2m
step 03486/05000 (69.72%) | loss: 3.896746 | lrm: 0.49 | dt: 1685.33ms | tok/sec: 9,721 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 67 | total time: 92.13m | eta: 40.1m
step 03487/05000 (69.74%) | loss: 3.896521 | lrm: 0.49 | dt: 1715.65ms | tok/sec: 9,549 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 67 | total time: 92.16m | eta: 40.1m
step 03488/05000 (69.76%) | loss: 3.881038 | lrm: 0.49 | dt: 1644.83ms | tok/sec: 9,960 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 67 | total time: 92.18m | eta: 40.1m
step 03489/05000 (69.78%) | loss: 3.878696 | lrm: 0.49 | dt: 1744.24ms | tok/sec: 9,393 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 67 | total time: 92.21m | eta: 40.1m
step 03490/05000 (69.80%) | loss: 3.886583 | lrm: 0.49 | dt: 1719.36ms | tok/sec: 9,529 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 67 | total time: 92.24m | eta: 40.0m
step 03491/05000 (69.82%) | loss: 3.902377 | lrm: 0.49 | dt: 1686.74ms | tok/sec: 9,713 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 67 | total time: 92.27m | eta: 40.0m
step 03492/05000 (69.84%) | loss: 3.903524 | lrm: 0.49 | dt: 1682.62ms | tok/sec: 9,737 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 67 | total time: 92.30m | eta: 40.0m
step 03493/05000 (69.86%) | loss: 3.901729 | lrm: 0.49 | dt: 1690.47ms | tok/sec: 9,692 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 67 | total time: 92.33m | eta: 39.9m
step 03494/05000 (69.88%) | loss: 3.887443 | lrm: 0.49 | dt: 1683.91ms | tok/sec: 9,729 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 67 | total time: 92.36m | eta: 39.9m
step 03495/05000 (69.90%) | loss: 3.907351 | lrm: 0.49 | dt: 1701.91ms | tok/sec: 9,626 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 67 | total time: 92.38m | eta: 39.9m
step 03496/05000 (69.92%) | loss: 3.896807 | lrm: 0.49 | dt: 1637.78ms | tok/sec: 10,003 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 67 | total time: 92.41m | eta: 39.9m
step 03497/05000 (69.94%) | loss: 3.901244 | lrm: 0.49 | dt: 1570.13ms | tok/sec: 10,434 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 67 | total time: 92.44m | eta: 39.8m
step 03498/05000 (69.96%) | loss: 3.903853 | lrm: 0.49 | dt: 1560.02ms | tok/sec: 10,502 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 67 | total time: 92.46m | eta: 39.8m
step 03499/05000 (69.98%) | loss: 3.895246 | lrm: 0.49 | dt: 1594.54ms | tok/sec: 10,275 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 67 | total time: 92.49m | eta: 39.8m
Step 03500 | Validation bpb: 1.204212
<|bos|>The capital of France is the capital of France, and the capital of France is the capital of France.
<|bos|>The chemical symbol of gold is the symbol of the gold's purity. It is a symbol of the purity of
<|bos|>If yesterday was Friday, then tomorrow will be Friday, Friday, Friday, Friday, Friday, Friday, Friday, Friday,
<|bos|>The opposite of hot is the same as the cold is the same as the cold is the same as the
<|bos|>The planets of the solar system are: the solar system, the solar system, the solar system, the solar system,
<|bos|>My favorite color is the white, but I'm not sure what I'm talking about. I'm
<|bos|>If 5*x + 3 = 13, then x is the number of the number of people in the universe. The number of people in
step 03500/05000 (70.00%) | loss: 3.875100 | lrm: 0.49 | dt: 1574.77ms | tok/sec: 10,404 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 67 | total time: 92.52m | eta: 39.8m
step 03501/05000 (70.02%) | loss: 3.860896 | lrm: 0.49 | dt: 1588.19ms | tok/sec: 10,316 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 67 | total time: 92.54m | eta: 39.7m
step 03502/05000 (70.04%) | loss: 3.860669 | lrm: 0.49 | dt: 1561.88ms | tok/sec: 10,489 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 67 | total time: 92.57m | eta: 39.7m
step 03503/05000 (70.06%) | loss: 3.863757 | lrm: 0.49 | dt: 1612.53ms | tok/sec: 10,160 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 68 | total time: 92.60m | eta: 39.7m
step 03504/05000 (70.08%) | loss: 3.875783 | lrm: 0.49 | dt: 1569.28ms | tok/sec: 10,440 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 68 | total time: 92.62m | eta: 39.7m
step 03505/05000 (70.10%) | loss: 3.881674 | lrm: 0.49 | dt: 1549.32ms | tok/sec: 10,574 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 68 | total time: 92.65m | eta: 39.6m
step 03506/05000 (70.12%) | loss: 3.880709 | lrm: 0.49 | dt: 1591.59ms | tok/sec: 10,294 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 68 | total time: 92.67m | eta: 39.6m
step 03507/05000 (70.14%) | loss: 3.891000 | lrm: 0.49 | dt: 1514.92ms | tok/sec: 10,815 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 68 | total time: 92.70m | eta: 39.6m
step 03508/05000 (70.16%) | loss: 3.897869 | lrm: 0.49 | dt: 1556.13ms | tok/sec: 10,528 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 68 | total time: 92.72m | eta: 39.5m
step 03509/05000 (70.18%) | loss: 3.882625 | lrm: 0.49 | dt: 1552.72ms | tok/sec: 10,551 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 68 | total time: 92.75m | eta: 39.5m
step 03510/05000 (70.20%) | loss: 3.874942 | lrm: 0.49 | dt: 1598.74ms | tok/sec: 10,248 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 68 | total time: 92.78m | eta: 39.5m
step 03511/05000 (70.22%) | loss: 3.886595 | lrm: 0.49 | dt: 1566.17ms | tok/sec: 10,461 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 68 | total time: 92.80m | eta: 39.5m
step 03512/05000 (70.24%) | loss: 3.893781 | lrm: 0.48 | dt: 1547.15ms | tok/sec: 10,589 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 68 | total time: 92.83m | eta: 39.4m
step 03513/05000 (70.26%) | loss: 3.890310 | lrm: 0.48 | dt: 1566.61ms | tok/sec: 10,458 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 68 | total time: 92.86m | eta: 39.4m
step 03514/05000 (70.28%) | loss: 3.880235 | lrm: 0.48 | dt: 1557.09ms | tok/sec: 10,522 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 68 | total time: 92.88m | eta: 39.4m
step 03515/05000 (70.30%) | loss: 3.878632 | lrm: 0.48 | dt: 1590.33ms | tok/sec: 10,302 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 68 | total time: 92.91m | eta: 39.4m
step 03516/05000 (70.32%) | loss: 3.898126 | lrm: 0.48 | dt: 1589.46ms | tok/sec: 10,307 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 68 | total time: 92.93m | eta: 39.3m
step 03517/05000 (70.34%) | loss: 3.887492 | lrm: 0.48 | dt: 1558.82ms | tok/sec: 10,510 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 68 | total time: 92.96m | eta: 39.3m
step 03518/05000 (70.36%) | loss: 3.897774 | lrm: 0.48 | dt: 1627.90ms | tok/sec: 10,064 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 68 | total time: 92.99m | eta: 39.3m
step 03519/05000 (70.38%) | loss: 3.900178 | lrm: 0.48 | dt: 1593.45ms | tok/sec: 10,282 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 68 | total time: 93.01m | eta: 39.3m
step 03520/05000 (70.40%) | loss: 3.879257 | lrm: 0.48 | dt: 1571.92ms | tok/sec: 10,422 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 68 | total time: 93.04m | eta: 39.2m
step 03521/05000 (70.42%) | loss: 3.887714 | lrm: 0.48 | dt: 1564.55ms | tok/sec: 10,472 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 68 | total time: 93.07m | eta: 39.2m
step 03522/05000 (70.44%) | loss: 3.886900 | lrm: 0.48 | dt: 1556.46ms | tok/sec: 10,526 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 68 | total time: 93.09m | eta: 39.2m
step 03523/05000 (70.46%) | loss: 3.901759 | lrm: 0.48 | dt: 1572.98ms | tok/sec: 10,415 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 68 | total time: 93.12m | eta: 39.2m
step 03524/05000 (70.48%) | loss: 3.912240 | lrm: 0.48 | dt: 1583.02ms | tok/sec: 10,349 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 68 | total time: 93.14m | eta: 39.1m
step 03525/05000 (70.50%) | loss: 3.910585 | lrm: 0.48 | dt: 1538.38ms | tok/sec: 10,650 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 68 | total time: 93.17m | eta: 39.1m
step 03526/05000 (70.52%) | loss: 3.901345 | lrm: 0.48 | dt: 1588.73ms | tok/sec: 10,312 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 69 | total time: 93.20m | eta: 39.1m
step 03527/05000 (70.54%) | loss: 3.892961 | lrm: 0.48 | dt: 1572.65ms | tok/sec: 10,418 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 69 | total time: 93.22m | eta: 39.0m
step 03528/05000 (70.56%) | loss: 3.907284 | lrm: 0.48 | dt: 1556.63ms | tok/sec: 10,525 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 69 | total time: 93.25m | eta: 39.0m
step 03529/05000 (70.58%) | loss: 3.887776 | lrm: 0.48 | dt: 1536.30ms | tok/sec: 10,664 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 69 | total time: 93.27m | eta: 39.0m
step 03530/05000 (70.60%) | loss: 3.886834 | lrm: 0.48 | dt: 1599.74ms | tok/sec: 10,241 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 69 | total time: 93.30m | eta: 39.0m
step 03531/05000 (70.62%) | loss: 3.879124 | lrm: 0.48 | dt: 1566.97ms | tok/sec: 10,455 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 69 | total time: 93.33m | eta: 38.9m
step 03532/05000 (70.64%) | loss: 3.855430 | lrm: 0.48 | dt: 1574.56ms | tok/sec: 10,405 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 69 | total time: 93.35m | eta: 38.9m
step 03533/05000 (70.66%) | loss: 3.862813 | lrm: 0.48 | dt: 1584.98ms | tok/sec: 10,337 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 69 | total time: 93.38m | eta: 38.9m
step 03534/05000 (70.68%) | loss: 3.854343 | lrm: 0.48 | dt: 1583.35ms | tok/sec: 10,347 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 69 | total time: 93.41m | eta: 38.9m
step 03535/05000 (70.70%) | loss: 3.833756 | lrm: 0.48 | dt: 1568.88ms | tok/sec: 10,443 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 69 | total time: 93.43m | eta: 38.8m
step 03536/05000 (70.72%) | loss: 3.832547 | lrm: 0.48 | dt: 1554.87ms | tok/sec: 10,537 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 69 | total time: 93.46m | eta: 38.8m
step 03537/05000 (70.74%) | loss: 3.841481 | lrm: 0.48 | dt: 1588.30ms | tok/sec: 10,315 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 69 | total time: 93.48m | eta: 38.8m
step 03538/05000 (70.76%) | loss: 3.848816 | lrm: 0.48 | dt: 1592.98ms | tok/sec: 10,285 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 69 | total time: 93.51m | eta: 38.8m
step 03539/05000 (70.78%) | loss: 3.860756 | lrm: 0.48 | dt: 1477.07ms | tok/sec: 11,092 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 69 | total time: 93.54m | eta: 38.7m
step 03540/05000 (70.80%) | loss: 3.859208 | lrm: 0.48 | dt: 1530.36ms | tok/sec: 10,705 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 69 | total time: 93.56m | eta: 38.7m
step 03541/05000 (70.82%) | loss: 3.849129 | lrm: 0.48 | dt: 1588.80ms | tok/sec: 10,312 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 69 | total time: 93.59m | eta: 38.7m
step 03542/05000 (70.84%) | loss: 3.844927 | lrm: 0.48 | dt: 1583.75ms | tok/sec: 10,345 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 69 | total time: 93.61m | eta: 38.6m
step 03543/05000 (70.86%) | loss: 3.854951 | lrm: 0.48 | dt: 1574.81ms | tok/sec: 10,403 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 69 | total time: 93.64m | eta: 38.6m
step 03544/05000 (70.88%) | loss: 3.853151 | lrm: 0.48 | dt: 1568.90ms | tok/sec: 10,442 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 69 | total time: 93.67m | eta: 38.6m
step 03545/05000 (70.90%) | loss: 3.869988 | lrm: 0.48 | dt: 1568.00ms | tok/sec: 10,448 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 69 | total time: 93.69m | eta: 38.6m
step 03546/05000 (70.92%) | loss: 3.862116 | lrm: 0.48 | dt: 1616.97ms | tok/sec: 10,132 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 69 | total time: 93.72m | eta: 38.5m
step 03547/05000 (70.94%) | loss: 3.857390 | lrm: 0.47 | dt: 1552.24ms | tok/sec: 10,555 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 69 | total time: 93.75m | eta: 38.5m
step 03548/05000 (70.96%) | loss: 3.874552 | lrm: 0.47 | dt: 1556.35ms | tok/sec: 10,527 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 69 | total time: 93.77m | eta: 38.5m
step 03549/05000 (70.98%) | loss: 3.884990 | lrm: 0.47 | dt: 1556.32ms | tok/sec: 10,527 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 70 | total time: 93.80m | eta: 38.5m
step 03550/05000 (71.00%) | loss: 3.886653 | lrm: 0.47 | dt: 1590.62ms | tok/sec: 10,300 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 70 | total time: 93.82m | eta: 38.4m
step 03551/05000 (71.02%) | loss: 3.888934 | lrm: 0.47 | dt: 1562.72ms | tok/sec: 10,484 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 70 | total time: 93.85m | eta: 38.4m
step 03552/05000 (71.04%) | loss: 3.888398 | lrm: 0.47 | dt: 1529.69ms | tok/sec: 10,710 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 70 | total time: 93.88m | eta: 38.4m
step 03553/05000 (71.06%) | loss: 3.892979 | lrm: 0.47 | dt: 1566.66ms | tok/sec: 10,457 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 70 | total time: 93.90m | eta: 38.4m
step 03554/05000 (71.08%) | loss: 3.875062 | lrm: 0.47 | dt: 1554.85ms | tok/sec: 10,537 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 70 | total time: 93.93m | eta: 38.3m
step 03555/05000 (71.10%) | loss: 3.884411 | lrm: 0.47 | dt: 1527.44ms | tok/sec: 10,726 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 70 | total time: 93.95m | eta: 38.3m
step 03556/05000 (71.12%) | loss: 3.876813 | lrm: 0.47 | dt: 1510.61ms | tok/sec: 10,845 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 70 | total time: 93.98m | eta: 38.3m
step 03557/05000 (71.14%) | loss: 3.868222 | lrm: 0.47 | dt: 1573.68ms | tok/sec: 10,411 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 70 | total time: 94.00m | eta: 38.2m
step 03558/05000 (71.16%) | loss: 3.879014 | lrm: 0.47 | dt: 1576.12ms | tok/sec: 10,395 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 70 | total time: 94.03m | eta: 38.2m
step 03559/05000 (71.18%) | loss: 3.878685 | lrm: 0.47 | dt: 1594.21ms | tok/sec: 10,277 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 70 | total time: 94.06m | eta: 38.2m
step 03560/05000 (71.20%) | loss: 3.867681 | lrm: 0.47 | dt: 1593.72ms | tok/sec: 10,280 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 70 | total time: 94.08m | eta: 38.2m
step 03561/05000 (71.22%) | loss: 3.877438 | lrm: 0.47 | dt: 1598.29ms | tok/sec: 10,250 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 70 | total time: 94.11m | eta: 38.1m
step 03562/05000 (71.24%) | loss: 3.885939 | lrm: 0.47 | dt: 1584.74ms | tok/sec: 10,338 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 70 | total time: 94.14m | eta: 38.1m
step 03563/05000 (71.26%) | loss: 3.895124 | lrm: 0.47 | dt: 1581.82ms | tok/sec: 10,357 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 70 | total time: 94.16m | eta: 38.1m
step 03564/05000 (71.28%) | loss: 3.890002 | lrm: 0.47 | dt: 1558.68ms | tok/sec: 10,511 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 70 | total time: 94.19m | eta: 38.1m
step 03565/05000 (71.30%) | loss: 3.894790 | lrm: 0.47 | dt: 1551.95ms | tok/sec: 10,557 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 70 | total time: 94.22m | eta: 38.0m
step 03566/05000 (71.32%) | loss: 3.883530 | lrm: 0.47 | dt: 1575.24ms | tok/sec: 10,400 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 70 | total time: 94.24m | eta: 38.0m
step 03567/05000 (71.34%) | loss: 3.868596 | lrm: 0.47 | dt: 1538.84ms | tok/sec: 10,646 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 70 | total time: 94.27m | eta: 38.0m
step 03568/05000 (71.36%) | loss: 3.871592 | lrm: 0.47 | dt: 1573.83ms | tok/sec: 10,410 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 70 | total time: 94.29m | eta: 38.0m
step 03569/05000 (71.38%) | loss: 3.869567 | lrm: 0.47 | dt: 1551.13ms | tok/sec: 10,562 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 70 | total time: 94.32m | eta: 37.9m
step 03570/05000 (71.40%) | loss: 3.859464 | lrm: 0.47 | dt: 1583.16ms | tok/sec: 10,348 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 70 | total time: 94.35m | eta: 37.9m
step 03571/05000 (71.42%) | loss: 3.855342 | lrm: 0.47 | dt: 1544.39ms | tok/sec: 10,608 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 70 | total time: 94.37m | eta: 37.9m
step 03572/05000 (71.44%) | loss: 3.848463 | lrm: 0.47 | dt: 1573.78ms | tok/sec: 10,410 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 71 | total time: 94.40m | eta: 37.8m
step 03573/05000 (71.46%) | loss: 3.848042 | lrm: 0.47 | dt: 1606.38ms | tok/sec: 10,199 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 71 | total time: 94.42m | eta: 37.8m
step 03574/05000 (71.48%) | loss: 3.839572 | lrm: 0.47 | dt: 1564.10ms | tok/sec: 10,475 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 71 | total time: 94.45m | eta: 37.8m
step 03575/05000 (71.50%) | loss: 3.845803 | lrm: 0.47 | dt: 1580.02ms | tok/sec: 10,369 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 71 | total time: 94.48m | eta: 37.8m
step 03576/05000 (71.52%) | loss: 3.853716 | lrm: 0.47 | dt: 1571.53ms | tok/sec: 10,425 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 71 | total time: 94.50m | eta: 37.7m
step 03577/05000 (71.54%) | loss: 3.852579 | lrm: 0.47 | dt: 1571.97ms | tok/sec: 10,422 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 71 | total time: 94.53m | eta: 37.7m
step 03578/05000 (71.56%) | loss: 3.869591 | lrm: 0.47 | dt: 1549.69ms | tok/sec: 10,572 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 71 | total time: 94.56m | eta: 37.7m
step 03579/05000 (71.58%) | loss: 3.862563 | lrm: 0.47 | dt: 1572.58ms | tok/sec: 10,418 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 71 | total time: 94.58m | eta: 37.7m
step 03580/05000 (71.60%) | loss: 3.879954 | lrm: 0.47 | dt: 1584.72ms | tok/sec: 10,338 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 71 | total time: 94.61m | eta: 37.6m
step 03581/05000 (71.62%) | loss: 3.881604 | lrm: 0.46 | dt: 1575.58ms | tok/sec: 10,398 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 71 | total time: 94.63m | eta: 37.6m
step 03582/05000 (71.64%) | loss: 3.873133 | lrm: 0.46 | dt: 1582.19ms | tok/sec: 10,355 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 71 | total time: 94.66m | eta: 37.6m
step 03583/05000 (71.66%) | loss: 3.881926 | lrm: 0.46 | dt: 1553.34ms | tok/sec: 10,547 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 71 | total time: 94.69m | eta: 37.6m
step 03584/05000 (71.68%) | loss: 3.872819 | lrm: 0.46 | dt: 1535.07ms | tok/sec: 10,673 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 71 | total time: 94.71m | eta: 37.5m
step 03585/05000 (71.70%) | loss: 3.879816 | lrm: 0.46 | dt: 1497.13ms | tok/sec: 10,943 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 71 | total time: 94.74m | eta: 37.5m
step 03586/05000 (71.72%) | loss: 3.884716 | lrm: 0.46 | dt: 1583.72ms | tok/sec: 10,345 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 71 | total time: 94.76m | eta: 37.5m
step 03587/05000 (71.74%) | loss: 3.867594 | lrm: 0.46 | dt: 1584.55ms | tok/sec: 10,339 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 71 | total time: 94.79m | eta: 37.4m
step 03588/05000 (71.76%) | loss: 3.876952 | lrm: 0.46 | dt: 1546.92ms | tok/sec: 10,591 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 71 | total time: 94.82m | eta: 37.4m
step 03589/05000 (71.78%) | loss: 3.858621 | lrm: 0.46 | dt: 1584.25ms | tok/sec: 10,341 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 71 | total time: 94.84m | eta: 37.4m
step 03590/05000 (71.80%) | loss: 3.853973 | lrm: 0.46 | dt: 1547.59ms | tok/sec: 10,586 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 71 | total time: 94.87m | eta: 37.4m
step 03591/05000 (71.82%) | loss: 3.863821 | lrm: 0.46 | dt: 1563.15ms | tok/sec: 10,481 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 71 | total time: 94.89m | eta: 37.3m
step 03592/05000 (71.84%) | loss: 3.871980 | lrm: 0.46 | dt: 1547.31ms | tok/sec: 10,588 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 71 | total time: 94.92m | eta: 37.3m
step 03593/05000 (71.86%) | loss: 3.878146 | lrm: 0.46 | dt: 1584.11ms | tok/sec: 10,342 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 71 | total time: 94.95m | eta: 37.3m
step 03594/05000 (71.88%) | loss: 3.879861 | lrm: 0.46 | dt: 1542.46ms | tok/sec: 10,621 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 71 | total time: 94.97m | eta: 37.3m
step 03595/05000 (71.90%) | loss: 3.875819 | lrm: 0.46 | dt: 1579.93ms | tok/sec: 10,370 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 72 | total time: 95.00m | eta: 37.2m
step 03596/05000 (71.92%) | loss: 3.863955 | lrm: 0.46 | dt: 1565.53ms | tok/sec: 10,465 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 72 | total time: 95.02m | eta: 37.2m
step 03597/05000 (71.94%) | loss: 3.878240 | lrm: 0.46 | dt: 1575.08ms | tok/sec: 10,402 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 72 | total time: 95.05m | eta: 37.2m
step 03598/05000 (71.96%) | loss: 3.871956 | lrm: 0.46 | dt: 1618.96ms | tok/sec: 10,120 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 72 | total time: 95.08m | eta: 37.2m
step 03599/05000 (71.98%) | loss: 3.879420 | lrm: 0.46 | dt: 1584.42ms | tok/sec: 10,340 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 72 | total time: 95.10m | eta: 37.1m
Step 03600 | Validation bpb: 1.200893
<|bos|>The capital of France is the capital of France. The capital of France is the capital of France. The
<|bos|>The chemical symbol of gold is the symbol of the gold's gold. The symbol of gold is the symbol of
<|bos|>If yesterday was Friday, then tomorrow will be the next day of the week. The next day will be Thursday, Friday,
<|bos|>The opposite of hot is the same as the cold is the same as the cold is the same as the
<|bos|>The planets of the solar system are: the solar system, the solar system, the solar system, the solar system,
<|bos|>My favorite color is the blue, but I'm not sure what I'm doing. I'm not
<|bos|>If 5*x + 3 = 13, then x is 3.5*x + 3.5*x + 3
step 03600/05000 (72.00%) | loss: 3.886284 | lrm: 0.46 | dt: 1536.06ms | tok/sec: 10,666 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 72 | total time: 95.13m | eta: 37.1m
step 03601/05000 (72.02%) | loss: 3.889234 | lrm: 0.46 | dt: 1513.97ms | tok/sec: 10,821 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 72 | total time: 95.15m | eta: 37.1m
step 03602/05000 (72.04%) | loss: 3.894696 | lrm: 0.46 | dt: 1518.13ms | tok/sec: 10,792 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 72 | total time: 95.18m | eta: 37.0m
step 03603/05000 (72.06%) | loss: 3.887454 | lrm: 0.46 | dt: 1569.47ms | tok/sec: 10,439 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 72 | total time: 95.21m | eta: 37.0m
step 03604/05000 (72.08%) | loss: 3.876900 | lrm: 0.46 | dt: 1617.71ms | tok/sec: 10,127 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 72 | total time: 95.23m | eta: 37.0m
step 03605/05000 (72.10%) | loss: 3.866428 | lrm: 0.46 | dt: 1601.66ms | tok/sec: 10,229 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 72 | total time: 95.26m | eta: 37.0m
step 03606/05000 (72.12%) | loss: 3.872658 | lrm: 0.46 | dt: 1616.84ms | tok/sec: 10,133 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 72 | total time: 95.29m | eta: 36.9m
step 03607/05000 (72.14%) | loss: 3.865623 | lrm: 0.46 | dt: 1571.50ms | tok/sec: 10,425 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 72 | total time: 95.31m | eta: 36.9m
step 03608/05000 (72.16%) | loss: 3.878454 | lrm: 0.46 | dt: 1584.93ms | tok/sec: 10,337 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 72 | total time: 95.34m | eta: 36.9m
step 03609/05000 (72.18%) | loss: 3.862989 | lrm: 0.46 | dt: 1591.38ms | tok/sec: 10,295 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 72 | total time: 95.37m | eta: 36.9m
step 03610/05000 (72.20%) | loss: 3.869390 | lrm: 0.46 | dt: 1595.61ms | tok/sec: 10,268 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 72 | total time: 95.39m | eta: 36.8m
step 03611/05000 (72.22%) | loss: 3.884946 | lrm: 0.46 | dt: 1544.75ms | tok/sec: 10,606 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 72 | total time: 95.42m | eta: 36.8m
step 03612/05000 (72.24%) | loss: 3.865149 | lrm: 0.46 | dt: 1555.61ms | tok/sec: 10,532 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 72 | total time: 95.44m | eta: 36.8m
step 03613/05000 (72.26%) | loss: 3.873027 | lrm: 0.46 | dt: 1561.65ms | tok/sec: 10,491 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 72 | total time: 95.47m | eta: 36.8m
step 03614/05000 (72.28%) | loss: 3.878972 | lrm: 0.46 | dt: 1565.75ms | tok/sec: 10,463 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 72 | total time: 95.50m | eta: 36.7m
step 03615/05000 (72.30%) | loss: 3.878457 | lrm: 0.45 | dt: 1604.17ms | tok/sec: 10,213 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 72 | total time: 95.52m | eta: 36.7m
step 03616/05000 (72.32%) | loss: 3.866668 | lrm: 0.45 | dt: 1579.62ms | tok/sec: 10,372 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 72 | total time: 95.55m | eta: 36.7m
step 03617/05000 (72.34%) | loss: 3.880566 | lrm: 0.45 | dt: 1612.77ms | tok/sec: 10,158 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 72 | total time: 95.58m | eta: 36.6m
step 03618/05000 (72.36%) | loss: 3.887887 | lrm: 0.45 | dt: 1522.65ms | tok/sec: 10,760 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 73 | total time: 95.60m | eta: 36.6m
step 03619/05000 (72.38%) | loss: 3.858802 | lrm: 0.45 | dt: 1575.74ms | tok/sec: 10,397 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 73 | total time: 95.63m | eta: 36.6m
step 03620/05000 (72.40%) | loss: 3.860792 | lrm: 0.45 | dt: 1587.22ms | tok/sec: 10,322 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 73 | total time: 95.65m | eta: 36.6m
step 03621/05000 (72.42%) | loss: 3.852073 | lrm: 0.45 | dt: 1555.95ms | tok/sec: 10,529 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 73 | total time: 95.68m | eta: 36.5m
step 03622/05000 (72.44%) | loss: 3.847503 | lrm: 0.45 | dt: 1600.32ms | tok/sec: 10,237 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 73 | total time: 95.71m | eta: 36.5m
step 03623/05000 (72.46%) | loss: 3.860827 | lrm: 0.45 | dt: 1578.20ms | tok/sec: 10,381 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 73 | total time: 95.73m | eta: 36.5m
step 03624/05000 (72.48%) | loss: 3.861090 | lrm: 0.45 | dt: 1536.07ms | tok/sec: 10,666 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 73 | total time: 95.76m | eta: 36.5m
step 03625/05000 (72.50%) | loss: 3.858937 | lrm: 0.45 | dt: 1584.50ms | tok/sec: 10,340 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 73 | total time: 95.78m | eta: 36.4m
step 03626/05000 (72.52%) | loss: 3.845219 | lrm: 0.45 | dt: 1550.86ms | tok/sec: 10,564 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 73 | total time: 95.81m | eta: 36.4m
step 03627/05000 (72.54%) | loss: 3.847999 | lrm: 0.45 | dt: 1535.53ms | tok/sec: 10,669 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 73 | total time: 95.84m | eta: 36.4m
step 03628/05000 (72.56%) | loss: 3.854143 | lrm: 0.45 | dt: 1527.36ms | tok/sec: 10,727 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 73 | total time: 95.86m | eta: 36.4m
step 03629/05000 (72.58%) | loss: 3.848780 | lrm: 0.45 | dt: 1591.31ms | tok/sec: 10,295 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 73 | total time: 95.89m | eta: 36.3m
step 03630/05000 (72.60%) | loss: 3.849408 | lrm: 0.45 | dt: 1550.84ms | tok/sec: 10,564 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 73 | total time: 95.91m | eta: 36.3m
step 03631/05000 (72.62%) | loss: 3.838895 | lrm: 0.45 | dt: 1557.14ms | tok/sec: 10,521 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 73 | total time: 95.94m | eta: 36.3m
step 03632/05000 (72.64%) | loss: 3.848870 | lrm: 0.45 | dt: 1578.91ms | tok/sec: 10,376 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 73 | total time: 95.97m | eta: 36.2m
step 03633/05000 (72.66%) | loss: 3.836088 | lrm: 0.45 | dt: 1539.59ms | tok/sec: 10,641 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 73 | total time: 95.99m | eta: 36.2m
step 03634/05000 (72.68%) | loss: 3.853498 | lrm: 0.45 | dt: 1580.22ms | tok/sec: 10,368 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 73 | total time: 96.02m | eta: 36.2m
step 03635/05000 (72.70%) | loss: 3.841811 | lrm: 0.45 | dt: 1528.48ms | tok/sec: 10,719 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 73 | total time: 96.04m | eta: 36.2m
step 03636/05000 (72.72%) | loss: 3.833178 | lrm: 0.45 | dt: 1591.02ms | tok/sec: 10,297 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 73 | total time: 96.07m | eta: 36.1m
step 03637/05000 (72.74%) | loss: 3.843464 | lrm: 0.45 | dt: 1559.46ms | tok/sec: 10,506 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 73 | total time: 96.10m | eta: 36.1m
step 03638/05000 (72.76%) | loss: 3.847615 | lrm: 0.45 | dt: 1559.82ms | tok/sec: 10,503 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 73 | total time: 96.12m | eta: 36.1m
step 03639/05000 (72.78%) | loss: 3.856018 | lrm: 0.45 | dt: 1560.38ms | tok/sec: 10,499 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 73 | total time: 96.15m | eta: 36.1m
step 03640/05000 (72.80%) | loss: 3.863345 | lrm: 0.45 | dt: 1556.78ms | tok/sec: 10,524 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 73 | total time: 96.17m | eta: 36.0m
step 03641/05000 (72.82%) | loss: 3.877962 | lrm: 0.45 | dt: 1575.16ms | tok/sec: 10,401 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 74 | total time: 96.20m | eta: 36.0m
step 03642/05000 (72.84%) | loss: 3.875002 | lrm: 0.45 | dt: 1581.20ms | tok/sec: 10,361 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 74 | total time: 96.23m | eta: 36.0m
step 03643/05000 (72.86%) | loss: 3.869200 | lrm: 0.45 | dt: 1546.12ms | tok/sec: 10,596 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 74 | total time: 96.25m | eta: 36.0m
step 03644/05000 (72.88%) | loss: 3.867477 | lrm: 0.45 | dt: 1569.27ms | tok/sec: 10,440 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 74 | total time: 96.28m | eta: 35.9m
step 03645/05000 (72.90%) | loss: 3.854931 | lrm: 0.45 | dt: 1547.89ms | tok/sec: 10,584 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 74 | total time: 96.30m | eta: 35.9m
step 03646/05000 (72.92%) | loss: 3.854911 | lrm: 0.45 | dt: 1560.72ms | tok/sec: 10,497 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 74 | total time: 96.33m | eta: 35.9m
step 03647/05000 (72.94%) | loss: 3.841207 | lrm: 0.45 | dt: 1560.23ms | tok/sec: 10,500 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 74 | total time: 96.36m | eta: 35.8m
step 03648/05000 (72.96%) | loss: 3.844677 | lrm: 0.45 | dt: 1544.06ms | tok/sec: 10,610 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 74 | total time: 96.38m | eta: 35.8m
step 03649/05000 (72.98%) | loss: 3.842424 | lrm: 0.44 | dt: 1571.43ms | tok/sec: 10,426 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 74 | total time: 96.41m | eta: 35.8m
step 03650/05000 (73.00%) | loss: 3.843016 | lrm: 0.44 | dt: 1581.87ms | tok/sec: 10,357 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 74 | total time: 96.43m | eta: 35.8m
step 03651/05000 (73.02%) | loss: 3.845145 | lrm: 0.44 | dt: 1564.74ms | tok/sec: 10,470 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 74 | total time: 96.46m | eta: 35.7m
step 03652/05000 (73.04%) | loss: 3.843452 | lrm: 0.44 | dt: 1554.66ms | tok/sec: 10,538 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 74 | total time: 96.49m | eta: 35.7m
step 03653/05000 (73.06%) | loss: 3.862422 | lrm: 0.44 | dt: 1557.01ms | tok/sec: 10,522 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 74 | total time: 96.51m | eta: 35.7m
step 03654/05000 (73.08%) | loss: 3.861673 | lrm: 0.44 | dt: 1549.60ms | tok/sec: 10,573 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 74 | total time: 96.54m | eta: 35.7m
step 03655/05000 (73.10%) | loss: 3.871933 | lrm: 0.44 | dt: 1591.46ms | tok/sec: 10,294 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 74 | total time: 96.57m | eta: 35.6m
step 03656/05000 (73.12%) | loss: 3.869036 | lrm: 0.44 | dt: 1562.33ms | tok/sec: 10,486 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 74 | total time: 96.59m | eta: 35.6m
step 03657/05000 (73.14%) | loss: 3.853563 | lrm: 0.44 | dt: 1590.02ms | tok/sec: 10,304 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 74 | total time: 96.62m | eta: 35.6m
step 03658/05000 (73.16%) | loss: 3.853116 | lrm: 0.44 | dt: 1542.43ms | tok/sec: 10,622 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 74 | total time: 96.64m | eta: 35.6m
step 03659/05000 (73.18%) | loss: 3.848365 | lrm: 0.44 | dt: 1546.13ms | tok/sec: 10,596 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 74 | total time: 96.67m | eta: 35.5m
step 03660/05000 (73.20%) | loss: 3.843347 | lrm: 0.44 | dt: 1584.07ms | tok/sec: 10,342 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 74 | total time: 96.70m | eta: 35.5m
step 03661/05000 (73.22%) | loss: 3.841988 | lrm: 0.44 | dt: 1602.29ms | tok/sec: 10,225 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 74 | total time: 96.72m | eta: 35.5m
step 03662/05000 (73.24%) | loss: 3.847652 | lrm: 0.44 | dt: 1537.55ms | tok/sec: 10,655 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 74 | total time: 96.75m | eta: 35.4m
step 03663/05000 (73.26%) | loss: 3.860848 | lrm: 0.44 | dt: 1545.20ms | tok/sec: 10,603 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 74 | total time: 96.77m | eta: 35.4m
step 03664/05000 (73.28%) | loss: 3.862237 | lrm: 0.44 | dt: 1541.44ms | tok/sec: 10,629 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 75 | total time: 96.80m | eta: 35.4m
step 03665/05000 (73.30%) | loss: 3.860393 | lrm: 0.44 | dt: 1537.84ms | tok/sec: 10,653 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 75 | total time: 96.83m | eta: 35.4m
step 03666/05000 (73.32%) | loss: 3.871272 | lrm: 0.44 | dt: 1550.32ms | tok/sec: 10,568 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 75 | total time: 96.85m | eta: 35.3m
step 03667/05000 (73.34%) | loss: 3.852850 | lrm: 0.44 | dt: 1551.62ms | tok/sec: 10,559 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 75 | total time: 96.88m | eta: 35.3m
step 03668/05000 (73.36%) | loss: 3.865180 | lrm: 0.44 | dt: 1566.65ms | tok/sec: 10,458 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 75 | total time: 96.90m | eta: 35.3m
step 03669/05000 (73.38%) | loss: 3.858101 | lrm: 0.44 | dt: 1583.04ms | tok/sec: 10,349 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 75 | total time: 96.93m | eta: 35.3m
step 03670/05000 (73.40%) | loss: 3.861000 | lrm: 0.44 | dt: 1550.56ms | tok/sec: 10,566 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 75 | total time: 96.96m | eta: 35.2m
step 03671/05000 (73.42%) | loss: 3.869251 | lrm: 0.44 | dt: 1606.82ms | tok/sec: 10,196 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 75 | total time: 96.98m | eta: 35.2m
step 03672/05000 (73.44%) | loss: 3.869955 | lrm: 0.44 | dt: 1611.20ms | tok/sec: 10,168 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 75 | total time: 97.01m | eta: 35.2m
step 03673/05000 (73.46%) | loss: 3.865053 | lrm: 0.44 | dt: 1561.64ms | tok/sec: 10,491 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 75 | total time: 97.03m | eta: 35.2m
step 03674/05000 (73.48%) | loss: 3.864359 | lrm: 0.44 | dt: 1565.32ms | tok/sec: 10,466 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 75 | total time: 97.06m | eta: 35.1m
step 03675/05000 (73.50%) | loss: 3.852554 | lrm: 0.44 | dt: 1581.30ms | tok/sec: 10,361 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 75 | total time: 97.09m | eta: 35.1m
step 03676/05000 (73.52%) | loss: 3.865266 | lrm: 0.44 | dt: 1559.49ms | tok/sec: 10,506 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 75 | total time: 97.11m | eta: 35.1m
step 03677/05000 (73.54%) | loss: 3.867071 | lrm: 0.44 | dt: 1567.73ms | tok/sec: 10,450 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 75 | total time: 97.14m | eta: 35.0m
step 03678/05000 (73.56%) | loss: 3.843970 | lrm: 0.44 | dt: 1569.29ms | tok/sec: 10,440 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 75 | total time: 97.17m | eta: 35.0m
step 03679/05000 (73.58%) | loss: 3.848271 | lrm: 0.44 | dt: 1597.51ms | tok/sec: 10,255 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 75 | total time: 97.19m | eta: 35.0m
step 03680/05000 (73.60%) | loss: 3.850857 | lrm: 0.44 | dt: 1556.66ms | tok/sec: 10,525 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 75 | total time: 97.22m | eta: 35.0m
step 03681/05000 (73.62%) | loss: 3.844312 | lrm: 0.44 | dt: 1562.64ms | tok/sec: 10,484 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 75 | total time: 97.24m | eta: 34.9m
step 03682/05000 (73.64%) | loss: 3.835592 | lrm: 0.44 | dt: 1583.76ms | tok/sec: 10,344 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 75 | total time: 97.27m | eta: 34.9m
step 03683/05000 (73.66%) | loss: 3.845183 | lrm: 0.43 | dt: 1558.29ms | tok/sec: 10,514 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 75 | total time: 97.30m | eta: 34.9m
step 03684/05000 (73.68%) | loss: 3.833844 | lrm: 0.43 | dt: 1567.77ms | tok/sec: 10,450 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 75 | total time: 97.32m | eta: 34.9m
step 03685/05000 (73.70%) | loss: 3.832443 | lrm: 0.43 | dt: 1550.29ms | tok/sec: 10,568 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 75 | total time: 97.35m | eta: 34.8m
step 03686/05000 (73.72%) | loss: 3.848170 | lrm: 0.43 | dt: 1564.96ms | tok/sec: 10,469 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 75 | total time: 97.37m | eta: 34.8m
step 03687/05000 (73.74%) | loss: 3.865654 | lrm: 0.43 | dt: 1560.57ms | tok/sec: 10,498 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 76 | total time: 97.40m | eta: 34.8m
step 03688/05000 (73.76%) | loss: 3.855630 | lrm: 0.43 | dt: 1553.57ms | tok/sec: 10,546 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 76 | total time: 97.43m | eta: 34.8m
step 03689/05000 (73.78%) | loss: 3.852919 | lrm: 0.43 | dt: 1573.21ms | tok/sec: 10,414 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 76 | total time: 97.45m | eta: 34.7m
step 03690/05000 (73.80%) | loss: 3.833252 | lrm: 0.43 | dt: 1537.47ms | tok/sec: 10,656 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 76 | total time: 97.48m | eta: 34.7m
step 03691/05000 (73.82%) | loss: 3.843279 | lrm: 0.43 | dt: 1564.29ms | tok/sec: 10,473 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 76 | total time: 97.50m | eta: 34.7m
step 03692/05000 (73.84%) | loss: 3.851121 | lrm: 0.43 | dt: 1547.47ms | tok/sec: 10,587 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 76 | total time: 97.53m | eta: 34.6m
step 03693/05000 (73.86%) | loss: 3.843021 | lrm: 0.43 | dt: 1583.88ms | tok/sec: 10,344 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 76 | total time: 97.56m | eta: 34.6m
step 03694/05000 (73.88%) | loss: 3.841663 | lrm: 0.43 | dt: 1568.50ms | tok/sec: 10,445 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 76 | total time: 97.58m | eta: 34.6m
step 03695/05000 (73.90%) | loss: 3.836713 | lrm: 0.43 | dt: 1585.36ms | tok/sec: 10,334 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 76 | total time: 97.61m | eta: 34.6m
step 03696/05000 (73.92%) | loss: 3.836549 | lrm: 0.43 | dt: 1583.66ms | tok/sec: 10,345 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 76 | total time: 97.64m | eta: 34.5m
step 03697/05000 (73.94%) | loss: 3.830188 | lrm: 0.43 | dt: 1577.73ms | tok/sec: 10,384 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 76 | total time: 97.66m | eta: 34.5m
step 03698/05000 (73.96%) | loss: 3.822241 | lrm: 0.43 | dt: 1583.61ms | tok/sec: 10,345 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 76 | total time: 97.69m | eta: 34.5m
step 03699/05000 (73.98%) | loss: 3.820335 | lrm: 0.43 | dt: 1551.52ms | tok/sec: 10,560 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 76 | total time: 97.71m | eta: 34.5m
Step 03700 | Validation bpb: 1.197689
<|bos|>The capital of France is the capital of France. The capital of France is the capital of France. The
<|bos|>The chemical symbol of gold is the symbol of the gold's gold. It is a symbol of the gold's
<|bos|>If yesterday was Friday, then tomorrow will be the next day of the week. The first day of the week is Friday,
<|bos|>The opposite of hot is the fact that the temperature of the air is 100 degrees Fahrenheit. The
<|bos|>The planets of the solar system are: the solar system, the solar system, the solar system, the solar system,
<|bos|>My favorite color is the blue, but I'm not sure what to do. I'm not sure
<|bos|>If 5*x + 3 = 13, then x is 2.5*x + 3.5*x + 3
step 03700/05000 (74.00%) | loss: 3.818552 | lrm: 0.43 | dt: 1607.10ms | tok/sec: 10,194 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 76 | total time: 97.74m | eta: 34.4m
step 03701/05000 (74.02%) | loss: 3.817278 | lrm: 0.43 | dt: 1597.11ms | tok/sec: 10,258 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 76 | total time: 97.77m | eta: 34.4m
step 03702/05000 (74.04%) | loss: 3.823087 | lrm: 0.43 | dt: 1576.23ms | tok/sec: 10,394 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 76 | total time: 97.79m | eta: 34.4m
step 03703/05000 (74.06%) | loss: 3.830422 | lrm: 0.43 | dt: 1587.51ms | tok/sec: 10,320 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 76 | total time: 97.82m | eta: 34.4m
step 03704/05000 (74.08%) | loss: 3.830510 | lrm: 0.43 | dt: 1560.01ms | tok/sec: 10,502 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 76 | total time: 97.85m | eta: 34.3m
step 03705/05000 (74.10%) | loss: 3.813994 | lrm: 0.43 | dt: 1650.02ms | tok/sec: 9,929 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 76 | total time: 97.87m | eta: 34.3m
step 03706/05000 (74.12%) | loss: 3.801577 | lrm: 0.43 | dt: 1597.99ms | tok/sec: 10,252 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 76 | total time: 97.90m | eta: 34.3m
step 03707/05000 (74.14%) | loss: 3.812786 | lrm: 0.43 | dt: 1629.62ms | tok/sec: 10,053 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 76 | total time: 97.93m | eta: 34.2m
step 03708/05000 (74.16%) | loss: 3.818298 | lrm: 0.43 | dt: 1677.96ms | tok/sec: 9,764 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 76 | total time: 97.96m | eta: 34.2m
step 03709/05000 (74.18%) | loss: 3.809962 | lrm: 0.43 | dt: 1653.36ms | tok/sec: 9,909 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 76 | total time: 97.98m | eta: 34.2m
step 03710/05000 (74.20%) | loss: 3.812391 | lrm: 0.43 | dt: 1675.00ms | tok/sec: 9,781 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 77 | total time: 98.01m | eta: 34.2m
step 03711/05000 (74.22%) | loss: 3.798538 | lrm: 0.43 | dt: 1699.91ms | tok/sec: 9,638 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 77 | total time: 98.04m | eta: 34.1m
step 03712/05000 (74.24%) | loss: 3.813824 | lrm: 0.43 | dt: 1694.80ms | tok/sec: 9,667 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 77 | total time: 98.07m | eta: 34.1m
step 03713/05000 (74.26%) | loss: 3.808869 | lrm: 0.43 | dt: 1701.97ms | tok/sec: 9,626 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 77 | total time: 98.10m | eta: 34.1m
step 03714/05000 (74.28%) | loss: 3.814204 | lrm: 0.43 | dt: 1670.28ms | tok/sec: 9,809 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 77 | total time: 98.12m | eta: 34.1m
step 03715/05000 (74.30%) | loss: 3.836454 | lrm: 0.43 | dt: 1618.23ms | tok/sec: 10,124 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 77 | total time: 98.15m | eta: 34.0m
step 03716/05000 (74.32%) | loss: 3.839858 | lrm: 0.43 | dt: 1603.53ms | tok/sec: 10,217 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 77 | total time: 98.18m | eta: 34.0m
step 03717/05000 (74.34%) | loss: 3.827692 | lrm: 0.43 | dt: 1639.86ms | tok/sec: 9,991 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 77 | total time: 98.20m | eta: 34.0m
step 03718/05000 (74.36%) | loss: 3.824969 | lrm: 0.42 | dt: 1570.28ms | tok/sec: 10,433 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 77 | total time: 98.23m | eta: 34.0m
step 03719/05000 (74.38%) | loss: 3.823910 | lrm: 0.42 | dt: 1563.75ms | tok/sec: 10,477 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 77 | total time: 98.26m | eta: 33.9m
step 03720/05000 (74.40%) | loss: 3.837349 | lrm: 0.42 | dt: 1549.09ms | tok/sec: 10,576 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 77 | total time: 98.28m | eta: 33.9m
step 03721/05000 (74.42%) | loss: 3.850578 | lrm: 0.42 | dt: 1557.36ms | tok/sec: 10,520 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 77 | total time: 98.31m | eta: 33.9m
step 03722/05000 (74.44%) | loss: 3.845562 | lrm: 0.42 | dt: 1552.61ms | tok/sec: 10,552 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 77 | total time: 98.33m | eta: 33.9m
step 03723/05000 (74.46%) | loss: 3.857217 | lrm: 0.42 | dt: 1562.93ms | tok/sec: 10,482 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 77 | total time: 98.36m | eta: 33.8m
step 03724/05000 (74.48%) | loss: 3.862073 | lrm: 0.42 | dt: 1529.35ms | tok/sec: 10,713 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 77 | total time: 98.39m | eta: 33.8m
step 03725/05000 (74.50%) | loss: 3.862133 | lrm: 0.42 | dt: 1519.15ms | tok/sec: 10,784 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 77 | total time: 98.41m | eta: 33.8m
step 03726/05000 (74.52%) | loss: 3.869098 | lrm: 0.42 | dt: 1492.99ms | tok/sec: 10,973 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 77 | total time: 98.44m | eta: 33.7m
step 03727/05000 (74.54%) | loss: 3.874282 | lrm: 0.42 | dt: 1559.91ms | tok/sec: 10,503 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 77 | total time: 98.46m | eta: 33.7m
step 03728/05000 (74.56%) | loss: 3.876544 | lrm: 0.42 | dt: 1527.08ms | tok/sec: 10,728 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 77 | total time: 98.49m | eta: 33.7m
step 03729/05000 (74.58%) | loss: 3.873748 | lrm: 0.42 | dt: 1554.14ms | tok/sec: 10,542 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 77 | total time: 98.51m | eta: 33.7m
step 03730/05000 (74.60%) | loss: 3.862967 | lrm: 0.42 | dt: 1559.23ms | tok/sec: 10,507 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 77 | total time: 98.54m | eta: 33.6m
step 03731/05000 (74.62%) | loss: 3.864341 | lrm: 0.42 | dt: 1580.61ms | tok/sec: 10,365 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 77 | total time: 98.57m | eta: 33.6m
step 03732/05000 (74.64%) | loss: 3.880039 | lrm: 0.42 | dt: 1584.28ms | tok/sec: 10,341 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 77 | total time: 98.59m | eta: 33.6m
step 03733/05000 (74.66%) | loss: 3.883931 | lrm: 0.42 | dt: 1593.83ms | tok/sec: 10,279 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 78 | total time: 98.62m | eta: 33.6m
step 03734/05000 (74.68%) | loss: 3.864902 | lrm: 0.42 | dt: 1557.34ms | tok/sec: 10,520 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 78 | total time: 98.64m | eta: 33.5m
step 03735/05000 (74.70%) | loss: 3.875441 | lrm: 0.42 | dt: 1554.73ms | tok/sec: 10,538 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 78 | total time: 98.67m | eta: 33.5m
step 03736/05000 (74.72%) | loss: 3.861018 | lrm: 0.42 | dt: 1536.36ms | tok/sec: 10,664 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 78 | total time: 98.70m | eta: 33.5m
step 03737/05000 (74.74%) | loss: 3.865425 | lrm: 0.42 | dt: 1514.90ms | tok/sec: 10,815 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 78 | total time: 98.72m | eta: 33.5m
step 03738/05000 (74.76%) | loss: 3.860786 | lrm: 0.42 | dt: 1553.34ms | tok/sec: 10,547 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 78 | total time: 98.75m | eta: 33.4m
step 03739/05000 (74.78%) | loss: 3.843992 | lrm: 0.42 | dt: 1542.76ms | tok/sec: 10,619 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 78 | total time: 98.77m | eta: 33.4m
step 03740/05000 (74.80%) | loss: 3.856556 | lrm: 0.42 | dt: 1529.64ms | tok/sec: 10,711 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 78 | total time: 98.80m | eta: 33.4m
step 03741/05000 (74.82%) | loss: 3.860903 | lrm: 0.42 | dt: 1559.81ms | tok/sec: 10,503 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 78 | total time: 98.82m | eta: 33.3m
step 03742/05000 (74.84%) | loss: 3.863622 | lrm: 0.42 | dt: 1522.95ms | tok/sec: 10,758 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 78 | total time: 98.85m | eta: 33.3m
step 03743/05000 (74.86%) | loss: 3.840007 | lrm: 0.42 | dt: 1558.12ms | tok/sec: 10,515 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 78 | total time: 98.88m | eta: 33.3m
step 03744/05000 (74.88%) | loss: 3.837876 | lrm: 0.42 | dt: 1522.86ms | tok/sec: 10,758 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 78 | total time: 98.90m | eta: 33.3m
step 03745/05000 (74.90%) | loss: 3.842914 | lrm: 0.42 | dt: 1550.94ms | tok/sec: 10,563 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 78 | total time: 98.93m | eta: 33.2m
step 03746/05000 (74.92%) | loss: 3.831718 | lrm: 0.42 | dt: 1524.03ms | tok/sec: 10,750 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 78 | total time: 98.95m | eta: 33.2m
step 03747/05000 (74.94%) | loss: 3.831894 | lrm: 0.42 | dt: 1546.96ms | tok/sec: 10,591 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 78 | total time: 98.98m | eta: 33.2m
step 03748/05000 (74.96%) | loss: 3.830515 | lrm: 0.42 | dt: 1574.40ms | tok/sec: 10,406 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 78 | total time: 99.00m | eta: 33.2m
step 03749/05000 (74.98%) | loss: 3.825226 | lrm: 0.42 | dt: 1543.07ms | tok/sec: 10,617 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 78 | total time: 99.03m | eta: 33.1m
step 03750/05000 (75.00%) | loss: 3.831142 | lrm: 0.42 | dt: 1527.48ms | tok/sec: 10,726 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 78 | total time: 99.06m | eta: 33.1m
step 03751/05000 (75.02%) | loss: 3.840115 | lrm: 0.42 | dt: 1548.24ms | tok/sec: 10,582 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 78 | total time: 99.08m | eta: 33.1m
step 03752/05000 (75.04%) | loss: 3.841101 | lrm: 0.41 | dt: 1562.08ms | tok/sec: 10,488 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 78 | total time: 99.11m | eta: 33.1m
step 03753/05000 (75.06%) | loss: 3.836655 | lrm: 0.41 | dt: 1554.84ms | tok/sec: 10,537 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 78 | total time: 99.13m | eta: 33.0m
step 03754/05000 (75.08%) | loss: 3.846856 | lrm: 0.41 | dt: 1563.83ms | tok/sec: 10,476 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 78 | total time: 99.16m | eta: 33.0m
step 03755/05000 (75.10%) | loss: 3.842122 | lrm: 0.41 | dt: 1557.59ms | tok/sec: 10,518 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 78 | total time: 99.19m | eta: 33.0m
step 03756/05000 (75.12%) | loss: 3.863158 | lrm: 0.41 | dt: 1563.15ms | tok/sec: 10,481 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 78 | total time: 99.21m | eta: 32.9m
step 03757/05000 (75.14%) | loss: 3.849369 | lrm: 0.41 | dt: 1553.18ms | tok/sec: 10,548 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 79 | total time: 99.24m | eta: 32.9m
step 03758/05000 (75.16%) | loss: 3.845164 | lrm: 0.41 | dt: 1483.87ms | tok/sec: 11,041 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 79 | total time: 99.26m | eta: 32.9m
step 03759/05000 (75.18%) | loss: 3.847086 | lrm: 0.41 | dt: 1534.96ms | tok/sec: 10,673 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 79 | total time: 99.29m | eta: 32.9m
step 03760/05000 (75.20%) | loss: 3.857044 | lrm: 0.41 | dt: 1543.53ms | tok/sec: 10,614 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 79 | total time: 99.31m | eta: 32.8m
step 03761/05000 (75.22%) | loss: 3.863862 | lrm: 0.41 | dt: 1538.04ms | tok/sec: 10,652 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 79 | total time: 99.34m | eta: 32.8m
step 03762/05000 (75.24%) | loss: 3.842679 | lrm: 0.41 | dt: 1571.49ms | tok/sec: 10,425 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 79 | total time: 99.37m | eta: 32.8m
step 03763/05000 (75.26%) | loss: 3.836180 | lrm: 0.41 | dt: 1558.24ms | tok/sec: 10,514 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 79 | total time: 99.39m | eta: 32.8m
step 03764/05000 (75.28%) | loss: 3.832533 | lrm: 0.41 | dt: 1574.90ms | tok/sec: 10,403 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 79 | total time: 99.42m | eta: 32.7m
step 03765/05000 (75.30%) | loss: 3.829096 | lrm: 0.41 | dt: 1519.76ms | tok/sec: 10,780 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 79 | total time: 99.44m | eta: 32.7m
step 03766/05000 (75.32%) | loss: 3.826863 | lrm: 0.41 | dt: 1547.37ms | tok/sec: 10,588 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 79 | total time: 99.47m | eta: 32.7m
step 03767/05000 (75.34%) | loss: 3.845165 | lrm: 0.41 | dt: 1555.07ms | tok/sec: 10,535 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 79 | total time: 99.49m | eta: 32.7m
step 03768/05000 (75.36%) | loss: 3.825410 | lrm: 0.41 | dt: 1548.49ms | tok/sec: 10,580 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 79 | total time: 99.52m | eta: 32.6m
step 03769/05000 (75.38%) | loss: 3.819613 | lrm: 0.41 | dt: 1517.59ms | tok/sec: 10,796 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 79 | total time: 99.55m | eta: 32.6m
step 03770/05000 (75.40%) | loss: 3.813546 | lrm: 0.41 | dt: 1519.23ms | tok/sec: 10,784 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 79 | total time: 99.57m | eta: 32.6m
step 03771/05000 (75.42%) | loss: 3.822824 | lrm: 0.41 | dt: 1522.22ms | tok/sec: 10,763 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 79 | total time: 99.60m | eta: 32.5m
step 03772/05000 (75.44%) | loss: 3.825403 | lrm: 0.41 | dt: 1545.90ms | tok/sec: 10,598 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 79 | total time: 99.62m | eta: 32.5m
step 03773/05000 (75.46%) | loss: 3.838475 | lrm: 0.41 | dt: 1554.52ms | tok/sec: 10,539 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 79 | total time: 99.65m | eta: 32.5m
step 03774/05000 (75.48%) | loss: 3.826365 | lrm: 0.41 | dt: 1572.55ms | tok/sec: 10,418 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 79 | total time: 99.67m | eta: 32.5m
step 03775/05000 (75.50%) | loss: 3.812699 | lrm: 0.41 | dt: 1606.98ms | tok/sec: 10,195 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 79 | total time: 99.70m | eta: 32.4m
step 03776/05000 (75.52%) | loss: 3.820103 | lrm: 0.41 | dt: 1600.46ms | tok/sec: 10,237 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 79 | total time: 99.73m | eta: 32.4m
step 03777/05000 (75.54%) | loss: 3.799109 | lrm: 0.41 | dt: 1561.97ms | tok/sec: 10,489 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 79 | total time: 99.75m | eta: 32.4m
step 03778/05000 (75.56%) | loss: 3.804718 | lrm: 0.41 | dt: 1548.39ms | tok/sec: 10,581 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 79 | total time: 99.78m | eta: 32.4m
step 03779/05000 (75.58%) | loss: 3.811815 | lrm: 0.41 | dt: 1554.87ms | tok/sec: 10,537 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 79 | total time: 99.81m | eta: 32.3m
step 03780/05000 (75.60%) | loss: 3.805440 | lrm: 0.41 | dt: 1540.29ms | tok/sec: 10,636 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 79 | total time: 99.83m | eta: 32.3m
step 03781/05000 (75.62%) | loss: 3.803959 | lrm: 0.41 | dt: 1532.42ms | tok/sec: 10,691 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 80 | total time: 99.86m | eta: 32.3m
step 03782/05000 (75.64%) | loss: 3.813134 | lrm: 0.41 | dt: 1570.28ms | tok/sec: 10,433 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 80 | total time: 99.88m | eta: 32.3m
step 03783/05000 (75.66%) | loss: 3.804944 | lrm: 0.41 | dt: 1524.91ms | tok/sec: 10,744 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 80 | total time: 99.91m | eta: 32.2m
step 03784/05000 (75.68%) | loss: 3.805001 | lrm: 0.41 | dt: 1532.74ms | tok/sec: 10,689 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 80 | total time: 99.93m | eta: 32.2m
step 03785/05000 (75.70%) | loss: 3.812411 | lrm: 0.41 | dt: 1534.15ms | tok/sec: 10,679 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 80 | total time: 99.96m | eta: 32.2m
step 03786/05000 (75.72%) | loss: 3.825359 | lrm: 0.40 | dt: 1545.37ms | tok/sec: 10,602 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 80 | total time: 99.99m | eta: 32.1m
step 03787/05000 (75.74%) | loss: 3.830477 | lrm: 0.40 | dt: 1556.78ms | tok/sec: 10,524 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 80 | total time: 100.01m | eta: 32.1m
step 03788/05000 (75.76%) | loss: 3.830333 | lrm: 0.40 | dt: 1545.77ms | tok/sec: 10,599 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 80 | total time: 100.04m | eta: 32.1m
step 03789/05000 (75.78%) | loss: 3.826795 | lrm: 0.40 | dt: 1541.20ms | tok/sec: 10,630 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 80 | total time: 100.06m | eta: 32.1m
step 03790/05000 (75.80%) | loss: 3.827608 | lrm: 0.40 | dt: 1556.00ms | tok/sec: 10,529 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 80 | total time: 100.09m | eta: 32.0m
step 03791/05000 (75.82%) | loss: 3.828391 | lrm: 0.40 | dt: 1558.59ms | tok/sec: 10,512 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 80 | total time: 100.11m | eta: 32.0m
step 03792/05000 (75.84%) | loss: 3.817109 | lrm: 0.40 | dt: 1583.44ms | tok/sec: 10,347 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 80 | total time: 100.14m | eta: 32.0m
step 03793/05000 (75.86%) | loss: 3.824374 | lrm: 0.40 | dt: 1585.86ms | tok/sec: 10,331 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 80 | total time: 100.17m | eta: 32.0m
step 03794/05000 (75.88%) | loss: 3.824385 | lrm: 0.40 | dt: 1541.36ms | tok/sec: 10,629 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 80 | total time: 100.19m | eta: 31.9m
step 03795/05000 (75.90%) | loss: 3.826246 | lrm: 0.40 | dt: 1541.04ms | tok/sec: 10,631 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 80 | total time: 100.22m | eta: 31.9m
step 03796/05000 (75.92%) | loss: 3.818600 | lrm: 0.40 | dt: 1543.23ms | tok/sec: 10,616 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 80 | total time: 100.24m | eta: 31.9m
step 03797/05000 (75.94%) | loss: 3.829446 | lrm: 0.40 | dt: 1557.82ms | tok/sec: 10,517 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 80 | total time: 100.27m | eta: 31.9m
step 03798/05000 (75.96%) | loss: 3.822085 | lrm: 0.40 | dt: 1608.66ms | tok/sec: 10,184 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 80 | total time: 100.30m | eta: 31.8m
step 03799/05000 (75.98%) | loss: 3.826137 | lrm: 0.40 | dt: 1563.20ms | tok/sec: 10,481 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 80 | total time: 100.32m | eta: 31.8m
Step 03800 | Validation bpb: 1.194168
<|bos|>The capital of France is the capital of France. The capital of France is the capital of France. The
<|bos|>The chemical symbol of gold is the symbol of the gold's gold. It is a symbol of the gold's
<|bos|>If yesterday was Friday, then tomorrow will be the next day of the week. The next day of the week will be Friday
<|bos|>The opposite of hot is the heat that is the heat that is the heat that is the heat that is
<|bos|>The planets of the solar system are: the solar system, the solar system, the solar system, the solar system,
<|bos|>My favorite color is the red, green, and blue. I have a red, green, and
<|bos|>If 5*x + 3 = 13, then x is 3.5*x + 3.5*x + 3
step 03800/05000 (76.00%) | loss: 3.828138 | lrm: 0.40 | dt: 1707.54ms | tok/sec: 9,595 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 80 | total time: 100.35m | eta: 31.8m
step 03801/05000 (76.02%) | loss: 3.837483 | lrm: 0.40 | dt: 1644.30ms | tok/sec: 9,964 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 80 | total time: 100.38m | eta: 31.7m
step 03802/05000 (76.04%) | loss: 3.838068 | lrm: 0.40 | dt: 1560.64ms | tok/sec: 10,498 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 80 | total time: 100.41m | eta: 31.7m
step 03803/05000 (76.06%) | loss: 3.835998 | lrm: 0.40 | dt: 1555.20ms | tok/sec: 10,534 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 81 | total time: 100.43m | eta: 31.7m
step 03804/05000 (76.08%) | loss: 3.842313 | lrm: 0.40 | dt: 1554.98ms | tok/sec: 10,536 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 81 | total time: 100.46m | eta: 31.7m
step 03805/05000 (76.10%) | loss: 3.842692 | lrm: 0.40 | dt: 1533.31ms | tok/sec: 10,685 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 81 | total time: 100.48m | eta: 31.6m
step 03806/05000 (76.12%) | loss: 3.844677 | lrm: 0.40 | dt: 1545.00ms | tok/sec: 10,604 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 81 | total time: 100.51m | eta: 31.6m
step 03807/05000 (76.14%) | loss: 3.830818 | lrm: 0.40 | dt: 1581.05ms | tok/sec: 10,362 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 81 | total time: 100.53m | eta: 31.6m
step 03808/05000 (76.16%) | loss: 3.831310 | lrm: 0.40 | dt: 1554.14ms | tok/sec: 10,542 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 81 | total time: 100.56m | eta: 31.6m
step 03809/05000 (76.18%) | loss: 3.826569 | lrm: 0.40 | dt: 1561.07ms | tok/sec: 10,495 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 81 | total time: 100.59m | eta: 31.5m
step 03810/05000 (76.20%) | loss: 3.824177 | lrm: 0.40 | dt: 1554.16ms | tok/sec: 10,542 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 81 | total time: 100.61m | eta: 31.5m
step 03811/05000 (76.22%) | loss: 3.837187 | lrm: 0.40 | dt: 1564.58ms | tok/sec: 10,471 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 81 | total time: 100.64m | eta: 31.5m
step 03812/05000 (76.24%) | loss: 3.827514 | lrm: 0.40 | dt: 1544.57ms | tok/sec: 10,607 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 81 | total time: 100.66m | eta: 31.5m
step 03813/05000 (76.26%) | loss: 3.848336 | lrm: 0.40 | dt: 1532.44ms | tok/sec: 10,691 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 81 | total time: 100.69m | eta: 31.4m
step 03814/05000 (76.28%) | loss: 3.852722 | lrm: 0.40 | dt: 1545.51ms | tok/sec: 10,600 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 81 | total time: 100.72m | eta: 31.4m
step 03815/05000 (76.30%) | loss: 3.857770 | lrm: 0.40 | dt: 1565.67ms | tok/sec: 10,464 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 81 | total time: 100.74m | eta: 31.4m
step 03816/05000 (76.32%) | loss: 3.847021 | lrm: 0.40 | dt: 1527.81ms | tok/sec: 10,723 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 81 | total time: 100.77m | eta: 31.3m
step 03817/05000 (76.34%) | loss: 3.839067 | lrm: 0.40 | dt: 1551.60ms | tok/sec: 10,559 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 81 | total time: 100.79m | eta: 31.3m
step 03818/05000 (76.36%) | loss: 3.840019 | lrm: 0.40 | dt: 1510.62ms | tok/sec: 10,845 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 81 | total time: 100.82m | eta: 31.3m
step 03819/05000 (76.38%) | loss: 3.834179 | lrm: 0.40 | dt: 1548.59ms | tok/sec: 10,579 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 81 | total time: 100.84m | eta: 31.3m
step 03820/05000 (76.40%) | loss: 3.846676 | lrm: 0.39 | dt: 1550.16ms | tok/sec: 10,569 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 81 | total time: 100.87m | eta: 31.2m
step 03821/05000 (76.42%) | loss: 3.844914 | lrm: 0.39 | dt: 1525.29ms | tok/sec: 10,741 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 81 | total time: 100.90m | eta: 31.2m
step 03822/05000 (76.44%) | loss: 3.836782 | lrm: 0.39 | dt: 1574.55ms | tok/sec: 10,405 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 81 | total time: 100.92m | eta: 31.2m
step 03823/05000 (76.46%) | loss: 3.839125 | lrm: 0.39 | dt: 1552.74ms | tok/sec: 10,551 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 81 | total time: 100.95m | eta: 31.2m
step 03824/05000 (76.48%) | loss: 3.818909 | lrm: 0.39 | dt: 1542.90ms | tok/sec: 10,618 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 81 | total time: 100.97m | eta: 31.1m
step 03825/05000 (76.50%) | loss: 3.826551 | lrm: 0.39 | dt: 1505.90ms | tok/sec: 10,879 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 81 | total time: 101.00m | eta: 31.1m
step 03826/05000 (76.52%) | loss: 3.830580 | lrm: 0.39 | dt: 1549.73ms | tok/sec: 10,572 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 81 | total time: 101.02m | eta: 31.1m
step 03827/05000 (76.54%) | loss: 3.847684 | lrm: 0.39 | dt: 1526.12ms | tok/sec: 10,735 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 82 | total time: 101.05m | eta: 31.1m
step 03828/05000 (76.56%) | loss: 3.855430 | lrm: 0.39 | dt: 1527.95ms | tok/sec: 10,722 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 82 | total time: 101.08m | eta: 31.0m
step 03829/05000 (76.58%) | loss: 3.864101 | lrm: 0.39 | dt: 1570.38ms | tok/sec: 10,433 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 82 | total time: 101.10m | eta: 31.0m
step 03830/05000 (76.60%) | loss: 3.857168 | lrm: 0.39 | dt: 1566.08ms | tok/sec: 10,461 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 82 | total time: 101.13m | eta: 31.0m
step 03831/05000 (76.62%) | loss: 3.863395 | lrm: 0.39 | dt: 1557.46ms | tok/sec: 10,519 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 82 | total time: 101.15m | eta: 30.9m
step 03832/05000 (76.64%) | loss: 3.843729 | lrm: 0.39 | dt: 1529.31ms | tok/sec: 10,713 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 82 | total time: 101.18m | eta: 30.9m
step 03833/05000 (76.66%) | loss: 3.840336 | lrm: 0.39 | dt: 1562.26ms | tok/sec: 10,487 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 82 | total time: 101.20m | eta: 30.9m
step 03834/05000 (76.68%) | loss: 3.839663 | lrm: 0.39 | dt: 1570.19ms | tok/sec: 10,434 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 82 | total time: 101.23m | eta: 30.9m
step 03835/05000 (76.70%) | loss: 3.843398 | lrm: 0.39 | dt: 1533.96ms | tok/sec: 10,680 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 82 | total time: 101.26m | eta: 30.8m
step 03836/05000 (76.72%) | loss: 3.842968 | lrm: 0.39 | dt: 1554.09ms | tok/sec: 10,542 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 82 | total time: 101.28m | eta: 30.8m
step 03837/05000 (76.74%) | loss: 3.846840 | lrm: 0.39 | dt: 1521.11ms | tok/sec: 10,771 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 82 | total time: 101.31m | eta: 30.8m
step 03838/05000 (76.76%) | loss: 3.840941 | lrm: 0.39 | dt: 1557.93ms | tok/sec: 10,516 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 82 | total time: 101.33m | eta: 30.8m
step 03839/05000 (76.78%) | loss: 3.856805 | lrm: 0.39 | dt: 1526.84ms | tok/sec: 10,730 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 82 | total time: 101.36m | eta: 30.7m
step 03840/05000 (76.80%) | loss: 3.843622 | lrm: 0.39 | dt: 1530.07ms | tok/sec: 10,707 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 82 | total time: 101.38m | eta: 30.7m
step 03841/05000 (76.82%) | loss: 3.849521 | lrm: 0.39 | dt: 1556.20ms | tok/sec: 10,528 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 82 | total time: 101.41m | eta: 30.7m
step 03842/05000 (76.84%) | loss: 3.854899 | lrm: 0.39 | dt: 1551.52ms | tok/sec: 10,559 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 82 | total time: 101.44m | eta: 30.7m
step 03843/05000 (76.86%) | loss: 3.869073 | lrm: 0.39 | dt: 1530.98ms | tok/sec: 10,701 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 82 | total time: 101.46m | eta: 30.6m
step 03844/05000 (76.88%) | loss: 3.860431 | lrm: 0.39 | dt: 1545.06ms | tok/sec: 10,604 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 82 | total time: 101.49m | eta: 30.6m
step 03845/05000 (76.90%) | loss: 3.883317 | lrm: 0.39 | dt: 1573.28ms | tok/sec: 10,413 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 82 | total time: 101.51m | eta: 30.6m
step 03846/05000 (76.92%) | loss: 3.876799 | lrm: 0.39 | dt: 1522.92ms | tok/sec: 10,758 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 82 | total time: 101.54m | eta: 30.5m
step 03847/05000 (76.94%) | loss: 3.881166 | lrm: 0.39 | dt: 1535.44ms | tok/sec: 10,670 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 82 | total time: 101.56m | eta: 30.5m
step 03848/05000 (76.96%) | loss: 3.872224 | lrm: 0.39 | dt: 1570.45ms | tok/sec: 10,432 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 82 | total time: 101.59m | eta: 30.5m
step 03849/05000 (76.98%) | loss: 3.859498 | lrm: 0.39 | dt: 1517.15ms | tok/sec: 10,799 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 82 | total time: 101.62m | eta: 30.5m
step 03850/05000 (77.00%) | loss: 3.847399 | lrm: 0.39 | dt: 1545.59ms | tok/sec: 10,600 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 83 | total time: 101.64m | eta: 30.4m
step 03851/05000 (77.02%) | loss: 3.838838 | lrm: 0.39 | dt: 1513.56ms | tok/sec: 10,824 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 83 | total time: 101.67m | eta: 30.4m
step 03852/05000 (77.04%) | loss: 3.837157 | lrm: 0.39 | dt: 1563.51ms | tok/sec: 10,479 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 83 | total time: 101.69m | eta: 30.4m
step 03853/05000 (77.06%) | loss: 3.850460 | lrm: 0.39 | dt: 1562.35ms | tok/sec: 10,486 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 83 | total time: 101.72m | eta: 30.4m
step 03854/05000 (77.08%) | loss: 3.851227 | lrm: 0.38 | dt: 1518.70ms | tok/sec: 10,788 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 83 | total time: 101.74m | eta: 30.3m
step 03855/05000 (77.10%) | loss: 3.847099 | lrm: 0.38 | dt: 1553.42ms | tok/sec: 10,547 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 83 | total time: 101.77m | eta: 30.3m
step 03856/05000 (77.12%) | loss: 3.825641 | lrm: 0.38 | dt: 1553.98ms | tok/sec: 10,543 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 83 | total time: 101.80m | eta: 30.3m
step 03857/05000 (77.14%) | loss: 3.824942 | lrm: 0.38 | dt: 1576.32ms | tok/sec: 10,393 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 83 | total time: 101.82m | eta: 30.3m
step 03858/05000 (77.16%) | loss: 3.826543 | lrm: 0.38 | dt: 1478.85ms | tok/sec: 11,078 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 83 | total time: 101.85m | eta: 30.2m
step 03859/05000 (77.18%) | loss: 3.837742 | lrm: 0.38 | dt: 1570.98ms | tok/sec: 10,429 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 83 | total time: 101.87m | eta: 30.2m
step 03860/05000 (77.20%) | loss: 3.848092 | lrm: 0.38 | dt: 1558.40ms | tok/sec: 10,513 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 83 | total time: 101.90m | eta: 30.2m
step 03861/05000 (77.22%) | loss: 3.864696 | lrm: 0.38 | dt: 1540.34ms | tok/sec: 10,636 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 83 | total time: 101.93m | eta: 30.1m
step 03862/05000 (77.24%) | loss: 3.863065 | lrm: 0.38 | dt: 1573.45ms | tok/sec: 10,412 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 83 | total time: 101.95m | eta: 30.1m
step 03863/05000 (77.26%) | loss: 3.858232 | lrm: 0.38 | dt: 1555.92ms | tok/sec: 10,530 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 83 | total time: 101.98m | eta: 30.1m
step 03864/05000 (77.28%) | loss: 3.849907 | lrm: 0.38 | dt: 1536.29ms | tok/sec: 10,664 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 83 | total time: 102.00m | eta: 30.1m
step 03865/05000 (77.30%) | loss: 3.846543 | lrm: 0.38 | dt: 1528.85ms | tok/sec: 10,716 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 83 | total time: 102.03m | eta: 30.0m
step 03866/05000 (77.32%) | loss: 3.845541 | lrm: 0.38 | dt: 1550.15ms | tok/sec: 10,569 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 83 | total time: 102.05m | eta: 30.0m
step 03867/05000 (77.34%) | loss: 3.853884 | lrm: 0.38 | dt: 1544.85ms | tok/sec: 10,605 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 83 | total time: 102.08m | eta: 30.0m
step 03868/05000 (77.36%) | loss: 3.845648 | lrm: 0.38 | dt: 1553.10ms | tok/sec: 10,549 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 83 | total time: 102.11m | eta: 30.0m
step 03869/05000 (77.38%) | loss: 3.852106 | lrm: 0.38 | dt: 1557.02ms | tok/sec: 10,522 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 83 | total time: 102.13m | eta: 29.9m
step 03870/05000 (77.40%) | loss: 3.834022 | lrm: 0.38 | dt: 1554.89ms | tok/sec: 10,537 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 83 | total time: 102.16m | eta: 29.9m
step 03871/05000 (77.42%) | loss: 3.835735 | lrm: 0.38 | dt: 1577.03ms | tok/sec: 10,389 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 83 | total time: 102.18m | eta: 29.9m
step 03872/05000 (77.44%) | loss: 3.847864 | lrm: 0.38 | dt: 1542.32ms | tok/sec: 10,622 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 83 | total time: 102.21m | eta: 29.9m
step 03873/05000 (77.46%) | loss: 3.848582 | lrm: 0.38 | dt: 1540.14ms | tok/sec: 10,638 | bf16_mfu: 0.00 | epoch: 1 pq: 1 rg: 83 | total time: 102.24m | eta: 29.8m
step 03874/05000 (77.48%) | loss: 3.830923 | lrm: 0.38 | dt: 1554.91ms | tok/sec: 10,536 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 0 | total time: 102.26m | eta: 29.8m
step 03875/05000 (77.50%) | loss: 3.819046 | lrm: 0.38 | dt: 1541.92ms | tok/sec: 10,625 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 0 | total time: 102.29m | eta: 29.8m
step 03876/05000 (77.52%) | loss: 3.821755 | lrm: 0.38 | dt: 1531.59ms | tok/sec: 10,697 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 0 | total time: 102.31m | eta: 29.7m
step 03877/05000 (77.54%) | loss: 3.799635 | lrm: 0.38 | dt: 1522.91ms | tok/sec: 10,758 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 0 | total time: 102.34m | eta: 29.7m
step 03878/05000 (77.56%) | loss: 3.806782 | lrm: 0.38 | dt: 1551.38ms | tok/sec: 10,560 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 0 | total time: 102.36m | eta: 29.7m
step 03879/05000 (77.58%) | loss: 3.824877 | lrm: 0.38 | dt: 1553.85ms | tok/sec: 10,544 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 0 | total time: 102.39m | eta: 29.7m
step 03880/05000 (77.60%) | loss: 3.810901 | lrm: 0.38 | dt: 1566.13ms | tok/sec: 10,461 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 0 | total time: 102.42m | eta: 29.6m
step 03881/05000 (77.62%) | loss: 3.821067 | lrm: 0.38 | dt: 1526.05ms | tok/sec: 10,736 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 0 | total time: 102.44m | eta: 29.6m
step 03882/05000 (77.64%) | loss: 3.801173 | lrm: 0.38 | dt: 1571.46ms | tok/sec: 10,425 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 0 | total time: 102.47m | eta: 29.6m
step 03883/05000 (77.66%) | loss: 3.796334 | lrm: 0.38 | dt: 1502.41ms | tok/sec: 10,905 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 0 | total time: 102.49m | eta: 29.6m
step 03884/05000 (77.68%) | loss: 3.817204 | lrm: 0.38 | dt: 1568.00ms | tok/sec: 10,448 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 0 | total time: 102.52m | eta: 29.5m
step 03885/05000 (77.70%) | loss: 3.821017 | lrm: 0.38 | dt: 1547.03ms | tok/sec: 10,590 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 0 | total time: 102.54m | eta: 29.5m
step 03886/05000 (77.72%) | loss: 3.821382 | lrm: 0.38 | dt: 1531.13ms | tok/sec: 10,700 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 0 | total time: 102.57m | eta: 29.5m
step 03887/05000 (77.74%) | loss: 3.834598 | lrm: 0.38 | dt: 1514.45ms | tok/sec: 10,818 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 0 | total time: 102.60m | eta: 29.5m
step 03888/05000 (77.76%) | loss: 3.828356 | lrm: 0.38 | dt: 1549.51ms | tok/sec: 10,573 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 0 | total time: 102.62m | eta: 29.4m
step 03889/05000 (77.78%) | loss: 3.824410 | lrm: 0.37 | dt: 1553.26ms | tok/sec: 10,548 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 0 | total time: 102.65m | eta: 29.4m
step 03890/05000 (77.80%) | loss: 3.826337 | lrm: 0.37 | dt: 1499.67ms | tok/sec: 10,925 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 0 | total time: 102.67m | eta: 29.4m
step 03891/05000 (77.82%) | loss: 3.818393 | lrm: 0.37 | dt: 1536.34ms | tok/sec: 10,664 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 0 | total time: 102.70m | eta: 29.3m
step 03892/05000 (77.84%) | loss: 3.821371 | lrm: 0.37 | dt: 1507.93ms | tok/sec: 10,865 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 0 | total time: 102.72m | eta: 29.3m
step 03893/05000 (77.86%) | loss: 3.817797 | lrm: 0.37 | dt: 1479.24ms | tok/sec: 11,075 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 0 | total time: 102.75m | eta: 29.3m
step 03894/05000 (77.88%) | loss: 3.819571 | lrm: 0.37 | dt: 1581.03ms | tok/sec: 10,362 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 0 | total time: 102.77m | eta: 29.3m
step 03895/05000 (77.90%) | loss: 3.814533 | lrm: 0.37 | dt: 1518.23ms | tok/sec: 10,791 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 0 | total time: 102.80m | eta: 29.2m
step 03896/05000 (77.92%) | loss: 3.809436 | lrm: 0.37 | dt: 1591.49ms | tok/sec: 10,294 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 0 | total time: 102.83m | eta: 29.2m
step 03897/05000 (77.94%) | loss: 3.806629 | lrm: 0.37 | dt: 1563.97ms | tok/sec: 10,475 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 1 | total time: 102.85m | eta: 29.2m
step 03898/05000 (77.96%) | loss: 3.805233 | lrm: 0.37 | dt: 1426.79ms | tok/sec: 11,483 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 1 | total time: 102.88m | eta: 29.2m
step 03899/05000 (77.98%) | loss: 3.800366 | lrm: 0.37 | dt: 1509.05ms | tok/sec: 10,857 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 1 | total time: 102.90m | eta: 29.1m
Step 03900 | Validation bpb: 1.191109
<|bos|>The capital of France is the capital of France. The capital of France is the capital of France. The
<|bos|>The chemical symbol of gold is the symbol of the gold's gold. The symbol of gold is the symbol of
<|bos|>If yesterday was Friday, then tomorrow will be the next day of the week. The next day will be Friday, Friday,
<|bos|>The opposite of hot is the heat that is absorbed by the body. The heat is absorbed by the body
<|bos|>The planets of the solar system are: the solar system, the solar system, the solar system, the solar system,
<|bos|>My favorite color is the red, but I don't know what color you are. I've seen
<|bos|>If 5*x + 3 = 13, then x is 1.5*x + 3.5*x + 3
step 03900/05000 (78.00%) | loss: 3.807944 | lrm: 0.37 | dt: 1506.04ms | tok/sec: 10,878 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 1 | total time: 102.93m | eta: 29.1m
step 03901/05000 (78.02%) | loss: 3.809852 | lrm: 0.37 | dt: 1444.11ms | tok/sec: 11,345 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 1 | total time: 102.95m | eta: 29.1m
step 03902/05000 (78.04%) | loss: 3.813993 | lrm: 0.37 | dt: 1435.58ms | tok/sec: 11,412 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 1 | total time: 102.97m | eta: 29.1m
step 03903/05000 (78.06%) | loss: 3.814223 | lrm: 0.37 | dt: 1422.48ms | tok/sec: 11,517 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 1 | total time: 103.00m | eta: 29.0m
step 03904/05000 (78.08%) | loss: 3.808359 | lrm: 0.37 | dt: 1447.77ms | tok/sec: 11,316 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 1 | total time: 103.02m | eta: 29.0m
step 03905/05000 (78.10%) | loss: 3.806388 | lrm: 0.37 | dt: 1482.02ms | tok/sec: 11,055 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 1 | total time: 103.05m | eta: 29.0m
step 03906/05000 (78.12%) | loss: 3.806302 | lrm: 0.37 | dt: 1475.88ms | tok/sec: 11,101 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 1 | total time: 103.07m | eta: 28.9m
step 03907/05000 (78.14%) | loss: 3.795179 | lrm: 0.37 | dt: 1467.46ms | tok/sec: 11,164 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 1 | total time: 103.10m | eta: 28.9m
step 03908/05000 (78.16%) | loss: 3.792097 | lrm: 0.37 | dt: 1498.68ms | tok/sec: 10,932 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 1 | total time: 103.12m | eta: 28.9m
step 03909/05000 (78.18%) | loss: 3.782208 | lrm: 0.37 | dt: 1475.07ms | tok/sec: 11,107 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 1 | total time: 103.14m | eta: 28.9m
step 03910/05000 (78.20%) | loss: 3.783236 | lrm: 0.37 | dt: 1465.66ms | tok/sec: 11,178 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 1 | total time: 103.17m | eta: 28.8m
step 03911/05000 (78.22%) | loss: 3.791591 | lrm: 0.37 | dt: 1501.31ms | tok/sec: 10,913 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 1 | total time: 103.19m | eta: 28.8m
step 03912/05000 (78.24%) | loss: 3.793208 | lrm: 0.37 | dt: 1583.86ms | tok/sec: 10,344 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 1 | total time: 103.22m | eta: 28.8m
step 03913/05000 (78.26%) | loss: 3.798138 | lrm: 0.37 | dt: 1585.91ms | tok/sec: 10,330 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 1 | total time: 103.25m | eta: 28.8m
step 03914/05000 (78.28%) | loss: 3.796443 | lrm: 0.37 | dt: 1610.16ms | tok/sec: 10,175 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 1 | total time: 103.27m | eta: 28.7m
step 03915/05000 (78.30%) | loss: 3.817695 | lrm: 0.37 | dt: 1616.57ms | tok/sec: 10,135 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 1 | total time: 103.30m | eta: 28.7m
step 03916/05000 (78.32%) | loss: 3.826401 | lrm: 0.37 | dt: 1529.61ms | tok/sec: 10,711 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 1 | total time: 103.33m | eta: 28.7m
step 03917/05000 (78.34%) | loss: 3.832005 | lrm: 0.37 | dt: 1544.26ms | tok/sec: 10,609 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 1 | total time: 103.35m | eta: 28.6m
step 03918/05000 (78.36%) | loss: 3.846721 | lrm: 0.37 | dt: 1580.96ms | tok/sec: 10,363 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 1 | total time: 103.38m | eta: 28.6m
step 03919/05000 (78.38%) | loss: 3.845718 | lrm: 0.37 | dt: 1715.95ms | tok/sec: 9,548 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 1 | total time: 103.41m | eta: 28.6m
step 03920/05000 (78.40%) | loss: 3.827227 | lrm: 0.37 | dt: 1554.72ms | tok/sec: 10,538 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 2 | total time: 103.43m | eta: 28.6m
step 03921/05000 (78.42%) | loss: 3.816221 | lrm: 0.37 | dt: 1636.26ms | tok/sec: 10,013 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 2 | total time: 103.46m | eta: 28.5m
step 03922/05000 (78.44%) | loss: 3.828569 | lrm: 0.37 | dt: 1516.57ms | tok/sec: 10,803 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 2 | total time: 103.49m | eta: 28.5m
step 03923/05000 (78.46%) | loss: 3.834186 | lrm: 0.36 | dt: 1580.98ms | tok/sec: 10,363 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 2 | total time: 103.51m | eta: 28.5m
step 03924/05000 (78.48%) | loss: 3.824879 | lrm: 0.36 | dt: 1438.56ms | tok/sec: 11,389 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 2 | total time: 103.54m | eta: 28.5m
step 03925/05000 (78.50%) | loss: 3.815060 | lrm: 0.36 | dt: 1675.14ms | tok/sec: 9,780 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 2 | total time: 103.56m | eta: 28.4m
step 03926/05000 (78.52%) | loss: 3.823125 | lrm: 0.36 | dt: 1538.80ms | tok/sec: 10,647 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 2 | total time: 103.59m | eta: 28.4m
step 03927/05000 (78.54%) | loss: 3.821492 | lrm: 0.36 | dt: 1469.39ms | tok/sec: 11,150 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 2 | total time: 103.61m | eta: 28.4m
step 03928/05000 (78.56%) | loss: 3.812314 | lrm: 0.36 | dt: 1509.17ms | tok/sec: 10,856 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 2 | total time: 103.64m | eta: 28.4m
step 03929/05000 (78.58%) | loss: 3.815571 | lrm: 0.36 | dt: 1444.29ms | tok/sec: 11,343 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 2 | total time: 103.66m | eta: 28.3m
step 03930/05000 (78.60%) | loss: 3.830473 | lrm: 0.36 | dt: 1608.57ms | tok/sec: 10,185 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 2 | total time: 103.69m | eta: 28.3m
step 03931/05000 (78.62%) | loss: 3.831588 | lrm: 0.36 | dt: 1538.30ms | tok/sec: 10,650 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 2 | total time: 103.72m | eta: 28.3m
step 03932/05000 (78.64%) | loss: 3.831091 | lrm: 0.36 | dt: 1611.33ms | tok/sec: 10,167 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 2 | total time: 103.74m | eta: 28.3m
step 03933/05000 (78.66%) | loss: 3.837697 | lrm: 0.36 | dt: 1657.42ms | tok/sec: 9,885 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 2 | total time: 103.77m | eta: 28.2m
step 03934/05000 (78.68%) | loss: 3.838225 | lrm: 0.36 | dt: 1411.57ms | tok/sec: 11,606 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 2 | total time: 103.79m | eta: 28.2m
step 03935/05000 (78.70%) | loss: 3.837645 | lrm: 0.36 | dt: 1495.85ms | tok/sec: 10,952 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 2 | total time: 103.82m | eta: 28.2m
step 03936/05000 (78.72%) | loss: 3.856064 | lrm: 0.36 | dt: 1520.25ms | tok/sec: 10,777 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 2 | total time: 103.84m | eta: 28.1m
step 03937/05000 (78.74%) | loss: 3.858165 | lrm: 0.36 | dt: 1511.57ms | tok/sec: 10,839 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 2 | total time: 103.87m | eta: 28.1m
step 03938/05000 (78.76%) | loss: 3.865428 | lrm: 0.36 | dt: 1609.42ms | tok/sec: 10,180 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 2 | total time: 103.90m | eta: 28.1m
step 03939/05000 (78.78%) | loss: 3.876603 | lrm: 0.36 | dt: 1436.23ms | tok/sec: 11,407 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 2 | total time: 103.92m | eta: 28.1m
step 03940/05000 (78.80%) | loss: 3.851552 | lrm: 0.36 | dt: 1601.85ms | tok/sec: 10,228 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 2 | total time: 103.95m | eta: 28.0m
step 03941/05000 (78.82%) | loss: 3.847283 | lrm: 0.36 | dt: 1603.44ms | tok/sec: 10,218 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 2 | total time: 103.97m | eta: 28.0m
step 03942/05000 (78.84%) | loss: 3.844875 | lrm: 0.36 | dt: 1564.76ms | tok/sec: 10,470 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 2 | total time: 104.00m | eta: 28.0m
step 03943/05000 (78.86%) | loss: 3.840026 | lrm: 0.36 | dt: 1558.03ms | tok/sec: 10,515 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 3 | total time: 104.03m | eta: 28.0m
step 03944/05000 (78.88%) | loss: 3.844576 | lrm: 0.36 | dt: 1612.20ms | tok/sec: 10,162 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 3 | total time: 104.05m | eta: 27.9m
step 03945/05000 (78.90%) | loss: 3.854845 | lrm: 0.36 | dt: 1605.66ms | tok/sec: 10,203 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 3 | total time: 104.08m | eta: 27.9m
step 03946/05000 (78.92%) | loss: 3.864901 | lrm: 0.36 | dt: 1540.68ms | tok/sec: 10,634 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 3 | total time: 104.10m | eta: 27.9m
step 03947/05000 (78.94%) | loss: 3.878375 | lrm: 0.36 | dt: 1535.65ms | tok/sec: 10,669 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 3 | total time: 104.13m | eta: 27.9m
step 03948/05000 (78.96%) | loss: 3.880680 | lrm: 0.36 | dt: 1538.28ms | tok/sec: 10,650 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 3 | total time: 104.16m | eta: 27.8m
step 03949/05000 (78.98%) | loss: 3.885185 | lrm: 0.36 | dt: 1732.73ms | tok/sec: 9,455 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 3 | total time: 104.18m | eta: 27.8m
step 03950/05000 (79.00%) | loss: 3.863978 | lrm: 0.36 | dt: 1567.65ms | tok/sec: 10,451 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 3 | total time: 104.21m | eta: 27.8m
step 03951/05000 (79.02%) | loss: 3.850742 | lrm: 0.36 | dt: 1462.38ms | tok/sec: 11,203 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 3 | total time: 104.24m | eta: 27.7m
step 03952/05000 (79.04%) | loss: 3.839563 | lrm: 0.36 | dt: 1552.26ms | tok/sec: 10,554 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 3 | total time: 104.26m | eta: 27.7m
step 03953/05000 (79.06%) | loss: 3.853653 | lrm: 0.36 | dt: 1398.56ms | tok/sec: 11,714 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 3 | total time: 104.28m | eta: 27.7m
step 03954/05000 (79.08%) | loss: 3.843962 | lrm: 0.36 | dt: 1523.96ms | tok/sec: 10,750 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 3 | total time: 104.31m | eta: 27.7m
step 03955/05000 (79.10%) | loss: 3.848602 | lrm: 0.36 | dt: 1557.97ms | tok/sec: 10,516 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 3 | total time: 104.34m | eta: 27.6m
step 03956/05000 (79.12%) | loss: 3.844601 | lrm: 0.36 | dt: 1542.15ms | tok/sec: 10,624 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 3 | total time: 104.36m | eta: 27.6m
step 03957/05000 (79.14%) | loss: 3.844433 | lrm: 0.35 | dt: 1539.51ms | tok/sec: 10,642 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 3 | total time: 104.39m | eta: 27.6m
step 03958/05000 (79.16%) | loss: 3.839287 | lrm: 0.35 | dt: 1638.23ms | tok/sec: 10,001 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 3 | total time: 104.41m | eta: 27.6m
step 03959/05000 (79.18%) | loss: 3.845632 | lrm: 0.35 | dt: 1570.99ms | tok/sec: 10,429 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 3 | total time: 104.44m | eta: 27.5m
step 03960/05000 (79.20%) | loss: 3.840502 | lrm: 0.35 | dt: 1549.24ms | tok/sec: 10,575 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 3 | total time: 104.47m | eta: 27.5m
step 03961/05000 (79.22%) | loss: 3.826885 | lrm: 0.35 | dt: 1580.51ms | tok/sec: 10,366 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 3 | total time: 104.49m | eta: 27.5m
step 03962/05000 (79.24%) | loss: 3.820884 | lrm: 0.35 | dt: 1609.98ms | tok/sec: 10,176 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 3 | total time: 104.52m | eta: 27.5m
step 03963/05000 (79.26%) | loss: 3.825053 | lrm: 0.35 | dt: 1583.48ms | tok/sec: 10,346 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 3 | total time: 104.55m | eta: 27.4m
step 03964/05000 (79.28%) | loss: 3.836906 | lrm: 0.35 | dt: 1504.99ms | tok/sec: 10,886 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 3 | total time: 104.57m | eta: 27.4m
step 03965/05000 (79.30%) | loss: 3.850438 | lrm: 0.35 | dt: 1635.79ms | tok/sec: 10,015 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 3 | total time: 104.60m | eta: 27.4m
step 03966/05000 (79.32%) | loss: 3.850410 | lrm: 0.35 | dt: 1563.06ms | tok/sec: 10,481 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 4 | total time: 104.62m | eta: 27.3m
step 03967/05000 (79.34%) | loss: 3.849126 | lrm: 0.35 | dt: 1622.14ms | tok/sec: 10,100 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 4 | total time: 104.65m | eta: 27.3m
step 03968/05000 (79.36%) | loss: 3.853702 | lrm: 0.35 | dt: 1655.66ms | tok/sec: 9,895 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 4 | total time: 104.68m | eta: 27.3m
step 03969/05000 (79.38%) | loss: 3.846550 | lrm: 0.35 | dt: 1606.46ms | tok/sec: 10,198 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 4 | total time: 104.71m | eta: 27.3m
step 03970/05000 (79.40%) | loss: 3.835103 | lrm: 0.35 | dt: 1613.02ms | tok/sec: 10,157 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 4 | total time: 104.73m | eta: 27.2m
step 03971/05000 (79.42%) | loss: 3.855936 | lrm: 0.35 | dt: 1623.52ms | tok/sec: 10,091 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 4 | total time: 104.76m | eta: 27.2m
step 03972/05000 (79.44%) | loss: 3.837061 | lrm: 0.35 | dt: 1645.13ms | tok/sec: 9,959 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 4 | total time: 104.79m | eta: 27.2m
step 03973/05000 (79.46%) | loss: 3.833716 | lrm: 0.35 | dt: 1600.79ms | tok/sec: 10,234 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 4 | total time: 104.81m | eta: 27.2m
step 03974/05000 (79.48%) | loss: 3.819700 | lrm: 0.35 | dt: 1704.31ms | tok/sec: 9,613 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 4 | total time: 104.84m | eta: 27.1m
step 03975/05000 (79.50%) | loss: 3.808457 | lrm: 0.35 | dt: 1666.58ms | tok/sec: 9,830 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 4 | total time: 104.87m | eta: 27.1m
step 03976/05000 (79.52%) | loss: 3.819538 | lrm: 0.35 | dt: 1695.24ms | tok/sec: 9,664 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 4 | total time: 104.90m | eta: 27.1m
step 03977/05000 (79.54%) | loss: 3.825982 | lrm: 0.35 | dt: 1637.54ms | tok/sec: 10,005 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 4 | total time: 104.93m | eta: 27.1m
step 03978/05000 (79.56%) | loss: 3.832132 | lrm: 0.35 | dt: 1600.04ms | tok/sec: 10,239 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 4 | total time: 104.95m | eta: 27.0m
step 03979/05000 (79.58%) | loss: 3.846394 | lrm: 0.35 | dt: 1590.11ms | tok/sec: 10,303 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 4 | total time: 104.98m | eta: 27.0m
step 03980/05000 (79.60%) | loss: 3.850516 | lrm: 0.35 | dt: 1633.54ms | tok/sec: 10,029 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 4 | total time: 105.01m | eta: 27.0m
step 03981/05000 (79.62%) | loss: 3.851886 | lrm: 0.35 | dt: 1700.47ms | tok/sec: 9,634 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 4 | total time: 105.03m | eta: 27.0m
step 03982/05000 (79.64%) | loss: 3.855093 | lrm: 0.35 | dt: 1637.20ms | tok/sec: 10,007 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 4 | total time: 105.06m | eta: 26.9m
step 03983/05000 (79.66%) | loss: 3.857652 | lrm: 0.35 | dt: 1625.89ms | tok/sec: 10,076 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 4 | total time: 105.09m | eta: 26.9m
step 03984/05000 (79.68%) | loss: 3.862837 | lrm: 0.35 | dt: 1584.04ms | tok/sec: 10,343 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 4 | total time: 105.12m | eta: 26.9m
step 03985/05000 (79.70%) | loss: 3.862476 | lrm: 0.35 | dt: 1629.58ms | tok/sec: 10,054 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 4 | total time: 105.14m | eta: 26.8m
step 03986/05000 (79.72%) | loss: 3.866223 | lrm: 0.35 | dt: 1597.02ms | tok/sec: 10,259 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 4 | total time: 105.17m | eta: 26.8m
step 03987/05000 (79.74%) | loss: 3.853090 | lrm: 0.35 | dt: 1588.51ms | tok/sec: 10,314 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 4 | total time: 105.20m | eta: 26.8m
step 03988/05000 (79.76%) | loss: 3.844208 | lrm: 0.35 | dt: 1649.54ms | tok/sec: 9,932 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 4 | total time: 105.22m | eta: 26.8m
step 03989/05000 (79.78%) | loss: 3.853519 | lrm: 0.35 | dt: 1608.41ms | tok/sec: 10,186 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 4 | total time: 105.25m | eta: 26.7m
step 03990/05000 (79.80%) | loss: 3.834039 | lrm: 0.35 | dt: 1587.35ms | tok/sec: 10,321 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 5 | total time: 105.28m | eta: 26.7m
step 03991/05000 (79.82%) | loss: 3.810558 | lrm: 0.34 | dt: 1573.47ms | tok/sec: 10,412 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 5 | total time: 105.30m | eta: 26.7m
step 03992/05000 (79.84%) | loss: 3.800984 | lrm: 0.34 | dt: 1567.60ms | tok/sec: 10,451 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 5 | total time: 105.33m | eta: 26.7m
step 03993/05000 (79.86%) | loss: 3.804130 | lrm: 0.34 | dt: 1547.70ms | tok/sec: 10,586 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 5 | total time: 105.35m | eta: 26.6m
step 03994/05000 (79.88%) | loss: 3.805769 | lrm: 0.34 | dt: 1599.30ms | tok/sec: 10,244 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 5 | total time: 105.38m | eta: 26.6m
step 03995/05000 (79.90%) | loss: 3.799736 | lrm: 0.34 | dt: 1625.21ms | tok/sec: 10,081 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 5 | total time: 105.41m | eta: 26.6m
step 03996/05000 (79.92%) | loss: 3.804275 | lrm: 0.34 | dt: 1660.83ms | tok/sec: 9,864 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 5 | total time: 105.44m | eta: 26.6m
step 03997/05000 (79.94%) | loss: 3.813431 | lrm: 0.34 | dt: 1517.90ms | tok/sec: 10,793 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 5 | total time: 105.46m | eta: 26.5m
step 03998/05000 (79.96%) | loss: 3.805018 | lrm: 0.34 | dt: 1571.75ms | tok/sec: 10,424 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 5 | total time: 105.49m | eta: 26.5m
step 03999/05000 (79.98%) | loss: 3.813768 | lrm: 0.34 | dt: 1663.66ms | tok/sec: 9,848 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 5 | total time: 105.51m | eta: 26.5m
Step 04000 | Validation bpb: 1.187774
<|bos|>The capital of France is the capital of France. The capital of France is the capital of France. The
<|bos|>The chemical symbol of gold is the gold, which is the gold, which is the gold, which is the
<|bos|>If yesterday was Friday, then tomorrow will be the next day of the week. The next day will be the next day of
<|bos|>The opposite of hot is the hot, hot, and cold. The hot is the hot, hot,
<|bos|>The planets of the solar system are: the solar system, the solar system, the solar system, the solar system,
<|bos|>My favorite color is the red, but I don't know what I'm doing. I'm not
<|bos|>If 5*x + 3 = 13, then x is 1.5*x + 1.5*x + 1
step 04000/05000 (80.00%) | loss: 3.812665 | lrm: 0.34 | dt: 1618.21ms | tok/sec: 10,124 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 5 | total time: 105.54m | eta: 26.5m
step 04001/05000 (80.02%) | loss: 3.803671 | lrm: 0.34 | dt: 1537.96ms | tok/sec: 10,653 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 5 | total time: 105.57m | eta: 26.4m
step 04002/05000 (80.04%) | loss: 3.802969 | lrm: 0.34 | dt: 1536.50ms | tok/sec: 10,663 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 5 | total time: 105.59m | eta: 26.4m
step 04003/05000 (80.06%) | loss: 3.806087 | lrm: 0.34 | dt: 1573.95ms | tok/sec: 10,409 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 5 | total time: 105.62m | eta: 26.4m
step 04004/05000 (80.08%) | loss: 3.816467 | lrm: 0.34 | dt: 1534.12ms | tok/sec: 10,679 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 5 | total time: 105.64m | eta: 26.3m
step 04005/05000 (80.10%) | loss: 3.825179 | lrm: 0.34 | dt: 1556.56ms | tok/sec: 10,525 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 5 | total time: 105.67m | eta: 26.3m
step 04006/05000 (80.12%) | loss: 3.812001 | lrm: 0.34 | dt: 1536.68ms | tok/sec: 10,661 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 5 | total time: 105.70m | eta: 26.3m
step 04007/05000 (80.14%) | loss: 3.825091 | lrm: 0.34 | dt: 1503.25ms | tok/sec: 10,899 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 5 | total time: 105.72m | eta: 26.3m
step 04008/05000 (80.16%) | loss: 3.840749 | lrm: 0.34 | dt: 1530.57ms | tok/sec: 10,704 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 5 | total time: 105.75m | eta: 26.2m
step 04009/05000 (80.18%) | loss: 3.831903 | lrm: 0.34 | dt: 1545.36ms | tok/sec: 10,602 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 5 | total time: 105.77m | eta: 26.2m
step 04010/05000 (80.20%) | loss: 3.823236 | lrm: 0.34 | dt: 1567.17ms | tok/sec: 10,454 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 5 | total time: 105.80m | eta: 26.2m
step 04011/05000 (80.22%) | loss: 3.814479 | lrm: 0.34 | dt: 1523.12ms | tok/sec: 10,756 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 5 | total time: 105.82m | eta: 26.2m
step 04012/05000 (80.24%) | loss: 3.814525 | lrm: 0.34 | dt: 1522.38ms | tok/sec: 10,762 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 5 | total time: 105.85m | eta: 26.1m
step 04013/05000 (80.26%) | loss: 3.819763 | lrm: 0.34 | dt: 1523.89ms | tok/sec: 10,751 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 6 | total time: 105.88m | eta: 26.1m
step 04014/05000 (80.28%) | loss: 3.831032 | lrm: 0.34 | dt: 1579.11ms | tok/sec: 10,375 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 6 | total time: 105.90m | eta: 26.1m
step 04015/05000 (80.30%) | loss: 3.844098 | lrm: 0.34 | dt: 1556.49ms | tok/sec: 10,526 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 6 | total time: 105.93m | eta: 26.1m
step 04016/05000 (80.32%) | loss: 3.824485 | lrm: 0.34 | dt: 1602.67ms | tok/sec: 10,222 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 6 | total time: 105.95m | eta: 26.0m
step 04017/05000 (80.34%) | loss: 3.826517 | lrm: 0.34 | dt: 1550.46ms | tok/sec: 10,567 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 6 | total time: 105.98m | eta: 26.0m
step 04018/05000 (80.36%) | loss: 3.815086 | lrm: 0.34 | dt: 1571.43ms | tok/sec: 10,426 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 6 | total time: 106.01m | eta: 26.0m
step 04019/05000 (80.38%) | loss: 3.821833 | lrm: 0.34 | dt: 1564.58ms | tok/sec: 10,471 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 6 | total time: 106.03m | eta: 25.9m
step 04020/05000 (80.40%) | loss: 3.828911 | lrm: 0.34 | dt: 1574.93ms | tok/sec: 10,402 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 6 | total time: 106.06m | eta: 25.9m
step 04021/05000 (80.42%) | loss: 3.818049 | lrm: 0.34 | dt: 1563.25ms | tok/sec: 10,480 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 6 | total time: 106.08m | eta: 25.9m
step 04022/05000 (80.44%) | loss: 3.823105 | lrm: 0.34 | dt: 1567.63ms | tok/sec: 10,451 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 6 | total time: 106.11m | eta: 25.9m
step 04023/05000 (80.46%) | loss: 3.835369 | lrm: 0.34 | dt: 1569.77ms | tok/sec: 10,437 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 6 | total time: 106.14m | eta: 25.8m
step 04024/05000 (80.48%) | loss: 3.841230 | lrm: 0.34 | dt: 1525.60ms | tok/sec: 10,739 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 6 | total time: 106.16m | eta: 25.8m
step 04025/05000 (80.50%) | loss: 3.833907 | lrm: 0.33 | dt: 1570.52ms | tok/sec: 10,432 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 6 | total time: 106.19m | eta: 25.8m
step 04026/05000 (80.52%) | loss: 3.833758 | lrm: 0.33 | dt: 1513.87ms | tok/sec: 10,822 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 6 | total time: 106.21m | eta: 25.8m
step 04027/05000 (80.54%) | loss: 3.834405 | lrm: 0.33 | dt: 1548.13ms | tok/sec: 10,583 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 6 | total time: 106.24m | eta: 25.7m
step 04028/05000 (80.56%) | loss: 3.836338 | lrm: 0.33 | dt: 1553.55ms | tok/sec: 10,546 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 6 | total time: 106.27m | eta: 25.7m
step 04029/05000 (80.58%) | loss: 3.843393 | lrm: 0.33 | dt: 1552.91ms | tok/sec: 10,550 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 6 | total time: 106.29m | eta: 25.7m
step 04030/05000 (80.60%) | loss: 3.837385 | lrm: 0.33 | dt: 1560.69ms | tok/sec: 10,497 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 6 | total time: 106.32m | eta: 25.7m
step 04031/05000 (80.62%) | loss: 3.837149 | lrm: 0.33 | dt: 1539.20ms | tok/sec: 10,644 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 6 | total time: 106.34m | eta: 25.6m
step 04032/05000 (80.64%) | loss: 3.826241 | lrm: 0.33 | dt: 1566.21ms | tok/sec: 10,460 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 6 | total time: 106.37m | eta: 25.6m
step 04033/05000 (80.66%) | loss: 3.820911 | lrm: 0.33 | dt: 1558.09ms | tok/sec: 10,515 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 6 | total time: 106.39m | eta: 25.6m
step 04034/05000 (80.68%) | loss: 3.813615 | lrm: 0.33 | dt: 1554.36ms | tok/sec: 10,540 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 6 | total time: 106.42m | eta: 25.5m
step 04035/05000 (80.70%) | loss: 3.822402 | lrm: 0.33 | dt: 1538.48ms | tok/sec: 10,649 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 6 | total time: 106.45m | eta: 25.5m
step 04036/05000 (80.72%) | loss: 3.813109 | lrm: 0.33 | dt: 1553.55ms | tok/sec: 10,546 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 7 | total time: 106.47m | eta: 25.5m
step 04037/05000 (80.74%) | loss: 3.817121 | lrm: 0.33 | dt: 1573.55ms | tok/sec: 10,412 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 7 | total time: 106.50m | eta: 25.5m
step 04038/05000 (80.76%) | loss: 3.823261 | lrm: 0.33 | dt: 1548.32ms | tok/sec: 10,581 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 7 | total time: 106.52m | eta: 25.4m
step 04039/05000 (80.78%) | loss: 3.813542 | lrm: 0.33 | dt: 1559.35ms | tok/sec: 10,506 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 7 | total time: 106.55m | eta: 25.4m
step 04040/05000 (80.80%) | loss: 3.806168 | lrm: 0.33 | dt: 1554.93ms | tok/sec: 10,536 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 7 | total time: 106.58m | eta: 25.4m
step 04041/05000 (80.82%) | loss: 3.821073 | lrm: 0.33 | dt: 1568.07ms | tok/sec: 10,448 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 7 | total time: 106.60m | eta: 25.4m
step 04042/05000 (80.84%) | loss: 3.819641 | lrm: 0.33 | dt: 1545.87ms | tok/sec: 10,598 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 7 | total time: 106.63m | eta: 25.3m
step 04043/05000 (80.86%) | loss: 3.824839 | lrm: 0.33 | dt: 1563.42ms | tok/sec: 10,479 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 7 | total time: 106.65m | eta: 25.3m
step 04044/05000 (80.88%) | loss: 3.822971 | lrm: 0.33 | dt: 1557.50ms | tok/sec: 10,519 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 7 | total time: 106.68m | eta: 25.3m
step 04045/05000 (80.90%) | loss: 3.812940 | lrm: 0.33 | dt: 1549.01ms | tok/sec: 10,577 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 7 | total time: 106.71m | eta: 25.3m
step 04046/05000 (80.92%) | loss: 3.803806 | lrm: 0.33 | dt: 1541.99ms | tok/sec: 10,625 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 7 | total time: 106.73m | eta: 25.2m
step 04047/05000 (80.94%) | loss: 3.808000 | lrm: 0.33 | dt: 1540.57ms | tok/sec: 10,635 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 7 | total time: 106.76m | eta: 25.2m
step 04048/05000 (80.96%) | loss: 3.826809 | lrm: 0.33 | dt: 1531.57ms | tok/sec: 10,697 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 7 | total time: 106.78m | eta: 25.2m
step 04049/05000 (80.98%) | loss: 3.822751 | lrm: 0.33 | dt: 1576.83ms | tok/sec: 10,390 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 7 | total time: 106.81m | eta: 25.1m
step 04050/05000 (81.00%) | loss: 3.834231 | lrm: 0.33 | dt: 1554.90ms | tok/sec: 10,536 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 7 | total time: 106.84m | eta: 25.1m
step 04051/05000 (81.02%) | loss: 3.837682 | lrm: 0.33 | dt: 1558.17ms | tok/sec: 10,514 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 7 | total time: 106.86m | eta: 25.1m
step 04052/05000 (81.04%) | loss: 3.836384 | lrm: 0.33 | dt: 1563.69ms | tok/sec: 10,477 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 7 | total time: 106.89m | eta: 25.1m
step 04053/05000 (81.06%) | loss: 3.835810 | lrm: 0.33 | dt: 1558.21ms | tok/sec: 10,514 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 7 | total time: 106.91m | eta: 25.0m
step 04054/05000 (81.08%) | loss: 3.833541 | lrm: 0.33 | dt: 1553.84ms | tok/sec: 10,544 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 7 | total time: 106.94m | eta: 25.0m
step 04055/05000 (81.10%) | loss: 3.849613 | lrm: 0.33 | dt: 1560.34ms | tok/sec: 10,500 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 7 | total time: 106.96m | eta: 25.0m
step 04056/05000 (81.12%) | loss: 3.844716 | lrm: 0.33 | dt: 1562.43ms | tok/sec: 10,486 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 7 | total time: 106.99m | eta: 25.0m
step 04057/05000 (81.14%) | loss: 3.849302 | lrm: 0.33 | dt: 1557.83ms | tok/sec: 10,517 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 7 | total time: 107.02m | eta: 24.9m
step 04058/05000 (81.16%) | loss: 3.846895 | lrm: 0.33 | dt: 1576.38ms | tok/sec: 10,393 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 7 | total time: 107.04m | eta: 24.9m
step 04059/05000 (81.18%) | loss: 3.840363 | lrm: 0.33 | dt: 1548.47ms | tok/sec: 10,580 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 8 | total time: 107.07m | eta: 24.9m
step 04060/05000 (81.20%) | loss: 3.830511 | lrm: 0.32 | dt: 1538.67ms | tok/sec: 10,648 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 8 | total time: 107.09m | eta: 24.9m
step 04061/05000 (81.22%) | loss: 3.837134 | lrm: 0.32 | dt: 1488.64ms | tok/sec: 11,006 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 8 | total time: 107.12m | eta: 24.8m
step 04062/05000 (81.24%) | loss: 3.826603 | lrm: 0.32 | dt: 1522.39ms | tok/sec: 10,762 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 8 | total time: 107.14m | eta: 24.8m
step 04063/05000 (81.26%) | loss: 3.821873 | lrm: 0.32 | dt: 1544.16ms | tok/sec: 10,610 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 8 | total time: 107.17m | eta: 24.8m
step 04064/05000 (81.28%) | loss: 3.823052 | lrm: 0.32 | dt: 1532.78ms | tok/sec: 10,689 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 8 | total time: 107.20m | eta: 24.7m
step 04065/05000 (81.30%) | loss: 3.805472 | lrm: 0.32 | dt: 1555.61ms | tok/sec: 10,532 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 8 | total time: 107.22m | eta: 24.7m
step 04066/05000 (81.32%) | loss: 3.804186 | lrm: 0.32 | dt: 1547.87ms | tok/sec: 10,584 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 8 | total time: 107.25m | eta: 24.7m
step 04067/05000 (81.34%) | loss: 3.807422 | lrm: 0.32 | dt: 1534.80ms | tok/sec: 10,674 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 8 | total time: 107.27m | eta: 24.7m
step 04068/05000 (81.36%) | loss: 3.791973 | lrm: 0.32 | dt: 1631.19ms | tok/sec: 10,044 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 8 | total time: 107.30m | eta: 24.6m
step 04069/05000 (81.38%) | loss: 3.804953 | lrm: 0.32 | dt: 1643.01ms | tok/sec: 9,971 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 8 | total time: 107.33m | eta: 24.6m
step 04070/05000 (81.40%) | loss: 3.806609 | lrm: 0.32 | dt: 1621.96ms | tok/sec: 10,101 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 8 | total time: 107.36m | eta: 24.6m
step 04071/05000 (81.42%) | loss: 3.803260 | lrm: 0.32 | dt: 1615.65ms | tok/sec: 10,140 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 8 | total time: 107.38m | eta: 24.6m
step 04072/05000 (81.44%) | loss: 3.803349 | lrm: 0.32 | dt: 1527.47ms | tok/sec: 10,726 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 8 | total time: 107.41m | eta: 24.5m
step 04073/05000 (81.46%) | loss: 3.813445 | lrm: 0.32 | dt: 1577.02ms | tok/sec: 10,389 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 8 | total time: 107.43m | eta: 24.5m
step 04074/05000 (81.48%) | loss: 3.815846 | lrm: 0.32 | dt: 1545.55ms | tok/sec: 10,600 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 8 | total time: 107.46m | eta: 24.5m
step 04075/05000 (81.50%) | loss: 3.811957 | lrm: 0.32 | dt: 1567.02ms | tok/sec: 10,455 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 8 | total time: 107.49m | eta: 24.5m
step 04076/05000 (81.52%) | loss: 3.814802 | lrm: 0.32 | dt: 1558.46ms | tok/sec: 10,512 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 8 | total time: 107.51m | eta: 24.4m
step 04077/05000 (81.54%) | loss: 3.812854 | lrm: 0.32 | dt: 1537.64ms | tok/sec: 10,655 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 8 | total time: 107.54m | eta: 24.4m
step 04078/05000 (81.56%) | loss: 3.797003 | lrm: 0.32 | dt: 1543.82ms | tok/sec: 10,612 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 8 | total time: 107.56m | eta: 24.4m
step 04079/05000 (81.58%) | loss: 3.800795 | lrm: 0.32 | dt: 1549.23ms | tok/sec: 10,575 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 8 | total time: 107.59m | eta: 24.4m
step 04080/05000 (81.60%) | loss: 3.783297 | lrm: 0.32 | dt: 1558.64ms | tok/sec: 10,511 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 8 | total time: 107.61m | eta: 24.3m
step 04081/05000 (81.62%) | loss: 3.799885 | lrm: 0.32 | dt: 1567.64ms | tok/sec: 10,451 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 8 | total time: 107.64m | eta: 24.3m
step 04082/05000 (81.64%) | loss: 3.801447 | lrm: 0.32 | dt: 1571.07ms | tok/sec: 10,428 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 9 | total time: 107.67m | eta: 24.3m
step 04083/05000 (81.66%) | loss: 3.797028 | lrm: 0.32 | dt: 1542.81ms | tok/sec: 10,619 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 9 | total time: 107.69m | eta: 24.2m
step 04084/05000 (81.68%) | loss: 3.805612 | lrm: 0.32 | dt: 1521.43ms | tok/sec: 10,768 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 9 | total time: 107.72m | eta: 24.2m
step 04085/05000 (81.70%) | loss: 3.807603 | lrm: 0.32 | dt: 1517.01ms | tok/sec: 10,800 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 9 | total time: 107.74m | eta: 24.2m
step 04086/05000 (81.72%) | loss: 3.800238 | lrm: 0.32 | dt: 1467.30ms | tok/sec: 11,166 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 9 | total time: 107.77m | eta: 24.2m
step 04087/05000 (81.74%) | loss: 3.805354 | lrm: 0.32 | dt: 1529.75ms | tok/sec: 10,710 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 9 | total time: 107.79m | eta: 24.1m
step 04088/05000 (81.76%) | loss: 3.818839 | lrm: 0.32 | dt: 1527.30ms | tok/sec: 10,727 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 9 | total time: 107.82m | eta: 24.1m
step 04089/05000 (81.78%) | loss: 3.824757 | lrm: 0.32 | dt: 1576.07ms | tok/sec: 10,395 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 9 | total time: 107.85m | eta: 24.1m
step 04090/05000 (81.80%) | loss: 3.825367 | lrm: 0.32 | dt: 1529.81ms | tok/sec: 10,709 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 9 | total time: 107.87m | eta: 24.1m
step 04091/05000 (81.82%) | loss: 3.824299 | lrm: 0.32 | dt: 1544.12ms | tok/sec: 10,610 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 9 | total time: 107.90m | eta: 24.0m
step 04092/05000 (81.84%) | loss: 3.822077 | lrm: 0.32 | dt: 1547.32ms | tok/sec: 10,588 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 9 | total time: 107.92m | eta: 24.0m
step 04093/05000 (81.86%) | loss: 3.823900 | lrm: 0.32 | dt: 1555.66ms | tok/sec: 10,531 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 9 | total time: 107.95m | eta: 24.0m
step 04094/05000 (81.88%) | loss: 3.828784 | lrm: 0.31 | dt: 1529.20ms | tok/sec: 10,714 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 9 | total time: 107.97m | eta: 24.0m
step 04095/05000 (81.90%) | loss: 3.824437 | lrm: 0.31 | dt: 1540.13ms | tok/sec: 10,638 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 9 | total time: 108.00m | eta: 23.9m
step 04096/05000 (81.92%) | loss: 3.820402 | lrm: 0.31 | dt: 1537.04ms | tok/sec: 10,659 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 9 | total time: 108.02m | eta: 23.9m
step 04097/05000 (81.94%) | loss: 3.811522 | lrm: 0.31 | dt: 1523.02ms | tok/sec: 10,757 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 9 | total time: 108.05m | eta: 23.9m
step 04098/05000 (81.96%) | loss: 3.822996 | lrm: 0.31 | dt: 1539.73ms | tok/sec: 10,640 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 9 | total time: 108.08m | eta: 23.8m
step 04099/05000 (81.98%) | loss: 3.803948 | lrm: 0.31 | dt: 1611.19ms | tok/sec: 10,168 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 9 | total time: 108.10m | eta: 23.8m
Step 04100 | Validation bpb: 1.185122
<|bos|>The capital of France is the capital of France. The capital of France is the capital of France. The
<|bos|>The chemical symbol of gold is the gold, and the gold is the gold. The gold is the gold,
<|bos|>If yesterday was Friday, then tomorrow will be Friday, and tomorrow will be Friday, and tomorrow will be Friday, and tomorrow
<|bos|>The opposite of hot is the hot, and the hot is the hot, and the hot is the hot
<|bos|>The planets of the solar system are: the solar system, the solar system, the solar system, the solar system,
<|bos|>My favorite color is the red, but I don't know what color you are. I have a
<|bos|>If 5*x + 3 = 13, then x is 3.0. If 5*x + 3 = 13
step 04100/05000 (82.00%) | loss: 3.814068 | lrm: 0.31 | dt: 2170.02ms | tok/sec: 7,550 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 9 | total time: 108.14m | eta: 23.8m
step 04101/05000 (82.02%) | loss: 3.820916 | lrm: 0.31 | dt: 1502.15ms | tok/sec: 10,907 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 9 | total time: 108.16m | eta: 23.8m
step 04102/05000 (82.04%) | loss: 3.818321 | lrm: 0.31 | dt: 1577.30ms | tok/sec: 10,387 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 9 | total time: 108.19m | eta: 23.7m
step 04103/05000 (82.06%) | loss: 3.806461 | lrm: 0.31 | dt: 1477.73ms | tok/sec: 11,087 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 9 | total time: 108.21m | eta: 23.7m
step 04104/05000 (82.08%) | loss: 3.816141 | lrm: 0.31 | dt: 1444.04ms | tok/sec: 11,345 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 9 | total time: 108.24m | eta: 23.7m
step 04105/05000 (82.10%) | loss: 3.809854 | lrm: 0.31 | dt: 1512.30ms | tok/sec: 10,833 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 10 | total time: 108.26m | eta: 23.7m
step 04106/05000 (82.12%) | loss: 3.798641 | lrm: 0.31 | dt: 1441.92ms | tok/sec: 11,362 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 10 | total time: 108.29m | eta: 23.6m
step 04107/05000 (82.14%) | loss: 3.807009 | lrm: 0.31 | dt: 1493.21ms | tok/sec: 10,972 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 10 | total time: 108.31m | eta: 23.6m
step 04108/05000 (82.16%) | loss: 3.809605 | lrm: 0.31 | dt: 1601.08ms | tok/sec: 10,233 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 10 | total time: 108.34m | eta: 23.6m
step 04109/05000 (82.18%) | loss: 3.798826 | lrm: 0.31 | dt: 1672.69ms | tok/sec: 9,795 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 10 | total time: 108.37m | eta: 23.6m
step 04110/05000 (82.20%) | loss: 3.807589 | lrm: 0.31 | dt: 1556.53ms | tok/sec: 10,525 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 10 | total time: 108.39m | eta: 23.5m
step 04111/05000 (82.22%) | loss: 3.792431 | lrm: 0.31 | dt: 1552.18ms | tok/sec: 10,555 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 10 | total time: 108.42m | eta: 23.5m
step 04112/05000 (82.24%) | loss: 3.790116 | lrm: 0.31 | dt: 1571.81ms | tok/sec: 10,423 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 10 | total time: 108.45m | eta: 23.5m
step 04113/05000 (82.26%) | loss: 3.799752 | lrm: 0.31 | dt: 1528.06ms | tok/sec: 10,722 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 10 | total time: 108.47m | eta: 23.4m
step 04114/05000 (82.28%) | loss: 3.806963 | lrm: 0.31 | dt: 1548.33ms | tok/sec: 10,581 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 10 | total time: 108.50m | eta: 23.4m
step 04115/05000 (82.30%) | loss: 3.817132 | lrm: 0.31 | dt: 1568.30ms | tok/sec: 10,446 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 10 | total time: 108.52m | eta: 23.4m
step 04116/05000 (82.32%) | loss: 3.799963 | lrm: 0.31 | dt: 1535.18ms | tok/sec: 10,672 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 10 | total time: 108.55m | eta: 23.4m
step 04117/05000 (82.34%) | loss: 3.784925 | lrm: 0.31 | dt: 1558.87ms | tok/sec: 10,510 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 10 | total time: 108.57m | eta: 23.3m
step 04118/05000 (82.36%) | loss: 3.796829 | lrm: 0.31 | dt: 1545.63ms | tok/sec: 10,600 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 10 | total time: 108.60m | eta: 23.3m
step 04119/05000 (82.38%) | loss: 3.789924 | lrm: 0.31 | dt: 1561.45ms | tok/sec: 10,492 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 10 | total time: 108.63m | eta: 23.3m
step 04120/05000 (82.40%) | loss: 3.801041 | lrm: 0.31 | dt: 1539.13ms | tok/sec: 10,644 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 10 | total time: 108.65m | eta: 23.3m
step 04121/05000 (82.42%) | loss: 3.805095 | lrm: 0.31 | dt: 1545.15ms | tok/sec: 10,603 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 10 | total time: 108.68m | eta: 23.2m
step 04122/05000 (82.44%) | loss: 3.813964 | lrm: 0.31 | dt: 1557.62ms | tok/sec: 10,518 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 10 | total time: 108.70m | eta: 23.2m
step 04123/05000 (82.46%) | loss: 3.819112 | lrm: 0.31 | dt: 1547.88ms | tok/sec: 10,584 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 10 | total time: 108.73m | eta: 23.2m
step 04124/05000 (82.48%) | loss: 3.803757 | lrm: 0.31 | dt: 1555.90ms | tok/sec: 10,530 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 10 | total time: 108.76m | eta: 23.2m
step 04125/05000 (82.50%) | loss: 3.803851 | lrm: 0.31 | dt: 1541.75ms | tok/sec: 10,626 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 10 | total time: 108.78m | eta: 23.1m
step 04126/05000 (82.52%) | loss: 3.795063 | lrm: 0.31 | dt: 1514.18ms | tok/sec: 10,820 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 10 | total time: 108.81m | eta: 23.1m
step 04127/05000 (82.54%) | loss: 3.796164 | lrm: 0.31 | dt: 1577.83ms | tok/sec: 10,383 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 10 | total time: 108.83m | eta: 23.1m
step 04128/05000 (82.56%) | loss: 3.815104 | lrm: 0.30 | dt: 1579.22ms | tok/sec: 10,374 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 10 | total time: 108.86m | eta: 23.1m
step 04129/05000 (82.58%) | loss: 3.804256 | lrm: 0.30 | dt: 1560.05ms | tok/sec: 10,502 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 11 | total time: 108.89m | eta: 23.0m
step 04130/05000 (82.60%) | loss: 3.799032 | lrm: 0.30 | dt: 1515.51ms | tok/sec: 10,810 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 11 | total time: 108.91m | eta: 23.0m
step 04131/05000 (82.62%) | loss: 3.787363 | lrm: 0.30 | dt: 1515.48ms | tok/sec: 10,811 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 11 | total time: 108.94m | eta: 23.0m
step 04132/05000 (82.64%) | loss: 3.785972 | lrm: 0.30 | dt: 1561.71ms | tok/sec: 10,491 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 11 | total time: 108.96m | eta: 22.9m
step 04133/05000 (82.66%) | loss: 3.787272 | lrm: 0.30 | dt: 1562.84ms | tok/sec: 10,483 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 11 | total time: 108.99m | eta: 22.9m
step 04134/05000 (82.68%) | loss: 3.782739 | lrm: 0.30 | dt: 1543.46ms | tok/sec: 10,615 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 11 | total time: 109.01m | eta: 22.9m
step 04135/05000 (82.70%) | loss: 3.774275 | lrm: 0.30 | dt: 1510.53ms | tok/sec: 10,846 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 11 | total time: 109.04m | eta: 22.9m
step 04136/05000 (82.72%) | loss: 3.791003 | lrm: 0.30 | dt: 1560.82ms | tok/sec: 10,497 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 11 | total time: 109.06m | eta: 22.8m
step 04137/05000 (82.74%) | loss: 3.786606 | lrm: 0.30 | dt: 1550.53ms | tok/sec: 10,566 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 11 | total time: 109.09m | eta: 22.8m
step 04138/05000 (82.76%) | loss: 3.792874 | lrm: 0.30 | dt: 1564.89ms | tok/sec: 10,469 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 11 | total time: 109.12m | eta: 22.8m
step 04139/05000 (82.78%) | loss: 3.802732 | lrm: 0.30 | dt: 1534.22ms | tok/sec: 10,679 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 11 | total time: 109.14m | eta: 22.8m
step 04140/05000 (82.80%) | loss: 3.801872 | lrm: 0.30 | dt: 1570.77ms | tok/sec: 10,430 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 11 | total time: 109.17m | eta: 22.7m
step 04141/05000 (82.82%) | loss: 3.792507 | lrm: 0.30 | dt: 1542.87ms | tok/sec: 10,619 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 11 | total time: 109.19m | eta: 22.7m
step 04142/05000 (82.84%) | loss: 3.806041 | lrm: 0.30 | dt: 1540.48ms | tok/sec: 10,635 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 11 | total time: 109.22m | eta: 22.7m
step 04143/05000 (82.86%) | loss: 3.800358 | lrm: 0.30 | dt: 1555.28ms | tok/sec: 10,534 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 11 | total time: 109.25m | eta: 22.7m
step 04144/05000 (82.88%) | loss: 3.810715 | lrm: 0.30 | dt: 1540.00ms | tok/sec: 10,638 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 11 | total time: 109.27m | eta: 22.6m
step 04145/05000 (82.90%) | loss: 3.828918 | lrm: 0.30 | dt: 1530.90ms | tok/sec: 10,702 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 11 | total time: 109.30m | eta: 22.6m
step 04146/05000 (82.92%) | loss: 3.808964 | lrm: 0.30 | dt: 1553.58ms | tok/sec: 10,545 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 11 | total time: 109.32m | eta: 22.6m
step 04147/05000 (82.94%) | loss: 3.803873 | lrm: 0.30 | dt: 1669.33ms | tok/sec: 9,814 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 11 | total time: 109.35m | eta: 22.5m
step 04148/05000 (82.96%) | loss: 3.804459 | lrm: 0.30 | dt: 1688.34ms | tok/sec: 9,704 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 11 | total time: 109.38m | eta: 22.5m
step 04149/05000 (82.98%) | loss: 3.824442 | lrm: 0.30 | dt: 1725.46ms | tok/sec: 9,495 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 11 | total time: 109.41m | eta: 22.5m
step 04150/05000 (83.00%) | loss: 3.827278 | lrm: 0.30 | dt: 1561.58ms | tok/sec: 10,491 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 11 | total time: 109.43m | eta: 22.5m
step 04151/05000 (83.02%) | loss: 3.821474 | lrm: 0.30 | dt: 1612.24ms | tok/sec: 10,162 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 11 | total time: 109.46m | eta: 22.4m
step 04152/05000 (83.04%) | loss: 3.822881 | lrm: 0.30 | dt: 1588.40ms | tok/sec: 10,314 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 12 | total time: 109.49m | eta: 22.4m
step 04153/05000 (83.06%) | loss: 3.800505 | lrm: 0.30 | dt: 1654.20ms | tok/sec: 9,904 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 12 | total time: 109.51m | eta: 22.4m
step 04154/05000 (83.08%) | loss: 3.815561 | lrm: 0.30 | dt: 1554.96ms | tok/sec: 10,536 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 12 | total time: 109.54m | eta: 22.4m
step 04155/05000 (83.10%) | loss: 3.801984 | lrm: 0.30 | dt: 1566.24ms | tok/sec: 10,460 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 12 | total time: 109.57m | eta: 22.3m
step 04156/05000 (83.12%) | loss: 3.801877 | lrm: 0.30 | dt: 1589.15ms | tok/sec: 10,309 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 12 | total time: 109.59m | eta: 22.3m
step 04157/05000 (83.14%) | loss: 3.800799 | lrm: 0.30 | dt: 1622.32ms | tok/sec: 10,099 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 12 | total time: 109.62m | eta: 22.3m
step 04158/05000 (83.16%) | loss: 3.797341 | lrm: 0.30 | dt: 1562.24ms | tok/sec: 10,487 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 12 | total time: 109.65m | eta: 22.3m
step 04159/05000 (83.18%) | loss: 3.808583 | lrm: 0.30 | dt: 1560.90ms | tok/sec: 10,496 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 12 | total time: 109.67m | eta: 22.2m
step 04160/05000 (83.20%) | loss: 3.810344 | lrm: 0.30 | dt: 1540.44ms | tok/sec: 10,635 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 12 | total time: 109.70m | eta: 22.2m
step 04161/05000 (83.22%) | loss: 3.815442 | lrm: 0.30 | dt: 1567.06ms | tok/sec: 10,455 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 12 | total time: 109.72m | eta: 22.2m
step 04162/05000 (83.24%) | loss: 3.801574 | lrm: 0.29 | dt: 1538.07ms | tok/sec: 10,652 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 12 | total time: 109.75m | eta: 22.2m
step 04163/05000 (83.26%) | loss: 3.807005 | lrm: 0.29 | dt: 1590.80ms | tok/sec: 10,299 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 12 | total time: 109.78m | eta: 22.1m
step 04164/05000 (83.28%) | loss: 3.807287 | lrm: 0.29 | dt: 1571.77ms | tok/sec: 10,423 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 12 | total time: 109.80m | eta: 22.1m
step 04165/05000 (83.30%) | loss: 3.794134 | lrm: 0.29 | dt: 1486.56ms | tok/sec: 11,021 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 12 | total time: 109.83m | eta: 22.1m
step 04166/05000 (83.32%) | loss: 3.786073 | lrm: 0.29 | dt: 1467.62ms | tok/sec: 11,163 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 12 | total time: 109.85m | eta: 22.0m
step 04167/05000 (83.34%) | loss: 3.789857 | lrm: 0.29 | dt: 1466.37ms | tok/sec: 11,173 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 12 | total time: 109.88m | eta: 22.0m
step 04168/05000 (83.36%) | loss: 3.791461 | lrm: 0.29 | dt: 1576.53ms | tok/sec: 10,392 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 12 | total time: 109.90m | eta: 22.0m
step 04169/05000 (83.38%) | loss: 3.800839 | lrm: 0.29 | dt: 1489.34ms | tok/sec: 11,000 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 12 | total time: 109.93m | eta: 22.0m
step 04170/05000 (83.40%) | loss: 3.795516 | lrm: 0.29 | dt: 1577.08ms | tok/sec: 10,388 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 12 | total time: 109.95m | eta: 21.9m
step 04171/05000 (83.42%) | loss: 3.798870 | lrm: 0.29 | dt: 1479.20ms | tok/sec: 11,076 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 12 | total time: 109.98m | eta: 21.9m
step 04172/05000 (83.44%) | loss: 3.807266 | lrm: 0.29 | dt: 1505.27ms | tok/sec: 10,884 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 12 | total time: 110.00m | eta: 21.9m
step 04173/05000 (83.46%) | loss: 3.814924 | lrm: 0.29 | dt: 1485.28ms | tok/sec: 11,030 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 12 | total time: 110.03m | eta: 21.9m
step 04174/05000 (83.48%) | loss: 3.790809 | lrm: 0.29 | dt: 1448.17ms | tok/sec: 11,313 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 12 | total time: 110.05m | eta: 21.8m
step 04175/05000 (83.50%) | loss: 3.791330 | lrm: 0.29 | dt: 1483.18ms | tok/sec: 11,046 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 12 | total time: 110.08m | eta: 21.8m
step 04176/05000 (83.52%) | loss: 3.792799 | lrm: 0.29 | dt: 1479.17ms | tok/sec: 11,076 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 13 | total time: 110.10m | eta: 21.8m
step 04177/05000 (83.54%) | loss: 3.794044 | lrm: 0.29 | dt: 1478.25ms | tok/sec: 11,083 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 13 | total time: 110.13m | eta: 21.8m
step 04178/05000 (83.56%) | loss: 3.803512 | lrm: 0.29 | dt: 1447.12ms | tok/sec: 11,321 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 13 | total time: 110.15m | eta: 21.7m
step 04179/05000 (83.58%) | loss: 3.804513 | lrm: 0.29 | dt: 1482.12ms | tok/sec: 11,054 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 13 | total time: 110.17m | eta: 21.7m
step 04180/05000 (83.60%) | loss: 3.792625 | lrm: 0.29 | dt: 1501.12ms | tok/sec: 10,914 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 13 | total time: 110.20m | eta: 21.7m
step 04181/05000 (83.62%) | loss: 3.797515 | lrm: 0.29 | dt: 1455.14ms | tok/sec: 11,259 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 13 | total time: 110.22m | eta: 21.6m
step 04182/05000 (83.64%) | loss: 3.791915 | lrm: 0.29 | dt: 1424.28ms | tok/sec: 11,503 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 13 | total time: 110.25m | eta: 21.6m
step 04183/05000 (83.66%) | loss: 3.805751 | lrm: 0.29 | dt: 1424.66ms | tok/sec: 11,500 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 13 | total time: 110.27m | eta: 21.6m
step 04184/05000 (83.68%) | loss: 3.808536 | lrm: 0.29 | dt: 1518.00ms | tok/sec: 10,793 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 13 | total time: 110.30m | eta: 21.6m
step 04185/05000 (83.70%) | loss: 3.810727 | lrm: 0.29 | dt: 1519.49ms | tok/sec: 10,782 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 13 | total time: 110.32m | eta: 21.5m
step 04186/05000 (83.72%) | loss: 3.840344 | lrm: 0.29 | dt: 1508.52ms | tok/sec: 10,860 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 13 | total time: 110.35m | eta: 21.5m
step 04187/05000 (83.74%) | loss: 3.841663 | lrm: 0.29 | dt: 1429.32ms | tok/sec: 11,462 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 13 | total time: 110.37m | eta: 21.5m
step 04188/05000 (83.76%) | loss: 3.827386 | lrm: 0.29 | dt: 1501.70ms | tok/sec: 10,910 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 13 | total time: 110.40m | eta: 21.5m
step 04189/05000 (83.78%) | loss: 3.807084 | lrm: 0.29 | dt: 1455.93ms | tok/sec: 11,253 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 13 | total time: 110.42m | eta: 21.4m
step 04190/05000 (83.80%) | loss: 3.811737 | lrm: 0.29 | dt: 1578.36ms | tok/sec: 10,380 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 13 | total time: 110.45m | eta: 21.4m
step 04191/05000 (83.82%) | loss: 3.819553 | lrm: 0.29 | dt: 1561.14ms | tok/sec: 10,494 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 13 | total time: 110.47m | eta: 21.4m
step 04192/05000 (83.84%) | loss: 3.808913 | lrm: 0.29 | dt: 1512.46ms | tok/sec: 10,832 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 13 | total time: 110.50m | eta: 21.3m
step 04193/05000 (83.86%) | loss: 3.799868 | lrm: 0.29 | dt: 1488.77ms | tok/sec: 11,005 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 13 | total time: 110.52m | eta: 21.3m
step 04194/05000 (83.88%) | loss: 3.792663 | lrm: 0.29 | dt: 1568.11ms | tok/sec: 10,448 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 13 | total time: 110.55m | eta: 21.3m
step 04195/05000 (83.90%) | loss: 3.808240 | lrm: 0.29 | dt: 1444.23ms | tok/sec: 11,344 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 13 | total time: 110.57m | eta: 21.3m
step 04196/05000 (83.92%) | loss: 3.806558 | lrm: 0.29 | dt: 1573.31ms | tok/sec: 10,413 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 13 | total time: 110.60m | eta: 21.2m
step 04197/05000 (83.94%) | loss: 3.816705 | lrm: 0.28 | dt: 1448.20ms | tok/sec: 11,313 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 13 | total time: 110.62m | eta: 21.2m
step 04198/05000 (83.96%) | loss: 3.820823 | lrm: 0.28 | dt: 1557.36ms | tok/sec: 10,520 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 13 | total time: 110.65m | eta: 21.2m
step 04199/05000 (83.98%) | loss: 3.813631 | lrm: 0.28 | dt: 1489.22ms | tok/sec: 11,001 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 13 | total time: 110.67m | eta: 21.2m
Step 04200 | Validation bpb: 1.182477
<|bos|>The capital of France is the capital of France. The capital of France is the capital of France. The
<|bos|>The chemical symbol of gold is the symbol of the symbol of the symbol of the symbol of the symbol of the
<|bos|>If yesterday was Friday, then tomorrow will be the next day of the week. The next day will be Friday, Friday,
<|bos|>The opposite of hot is the hot, and the hot is the hot, and the hot is the hot
<|bos|>The planets of the solar system are: the solar system, the solar system, the solar system, the solar system,
<|bos|>My favorite color is the white. I have a lot of color in my garden, but I have
<|bos|>If 5*x + 3 = 13, then x is 5*x + 3*x + 3*x +
step 04200/05000 (84.00%) | loss: 3.820058 | lrm: 0.28 | dt: 1528.73ms | tok/sec: 10,717 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 14 | total time: 110.70m | eta: 21.1m
step 04201/05000 (84.02%) | loss: 3.822144 | lrm: 0.28 | dt: 1433.81ms | tok/sec: 11,426 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 14 | total time: 110.72m | eta: 21.1m
step 04202/05000 (84.04%) | loss: 3.815502 | lrm: 0.28 | dt: 1469.62ms | tok/sec: 11,148 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 14 | total time: 110.75m | eta: 21.1m
step 04203/05000 (84.06%) | loss: 3.822371 | lrm: 0.28 | dt: 1454.03ms | tok/sec: 11,267 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 14 | total time: 110.77m | eta: 21.1m
step 04204/05000 (84.08%) | loss: 3.815983 | lrm: 0.28 | dt: 1473.89ms | tok/sec: 11,116 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 14 | total time: 110.80m | eta: 21.0m
step 04205/05000 (84.10%) | loss: 3.804891 | lrm: 0.28 | dt: 1454.76ms | tok/sec: 11,262 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 14 | total time: 110.82m | eta: 21.0m
step 04206/05000 (84.12%) | loss: 3.810518 | lrm: 0.28 | dt: 1556.58ms | tok/sec: 10,525 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 14 | total time: 110.85m | eta: 21.0m
step 04207/05000 (84.14%) | loss: 3.798209 | lrm: 0.28 | dt: 1669.83ms | tok/sec: 9,811 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 14 | total time: 110.87m | eta: 20.9m
step 04208/05000 (84.16%) | loss: 3.791950 | lrm: 0.28 | dt: 1622.45ms | tok/sec: 10,098 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 14 | total time: 110.90m | eta: 20.9m
step 04209/05000 (84.18%) | loss: 3.788110 | lrm: 0.28 | dt: 1545.41ms | tok/sec: 10,601 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 14 | total time: 110.93m | eta: 20.9m
step 04210/05000 (84.20%) | loss: 3.799223 | lrm: 0.28 | dt: 1586.89ms | tok/sec: 10,324 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 14 | total time: 110.95m | eta: 20.9m
step 04211/05000 (84.22%) | loss: 3.792618 | lrm: 0.28 | dt: 1564.29ms | tok/sec: 10,473 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 14 | total time: 110.98m | eta: 20.8m
step 04212/05000 (84.24%) | loss: 3.789493 | lrm: 0.28 | dt: 1592.93ms | tok/sec: 10,285 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 14 | total time: 111.01m | eta: 20.8m
step 04213/05000 (84.26%) | loss: 3.798179 | lrm: 0.28 | dt: 1559.06ms | tok/sec: 10,508 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 14 | total time: 111.03m | eta: 20.8m
step 04214/05000 (84.28%) | loss: 3.795379 | lrm: 0.28 | dt: 1503.47ms | tok/sec: 10,897 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 14 | total time: 111.06m | eta: 20.8m
step 04215/05000 (84.30%) | loss: 3.815915 | lrm: 0.28 | dt: 1558.11ms | tok/sec: 10,515 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 14 | total time: 111.08m | eta: 20.7m
step 04216/05000 (84.32%) | loss: 3.811901 | lrm: 0.28 | dt: 1539.65ms | tok/sec: 10,641 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 14 | total time: 111.11m | eta: 20.7m
step 04217/05000 (84.34%) | loss: 3.824380 | lrm: 0.28 | dt: 1544.93ms | tok/sec: 10,605 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 14 | total time: 111.13m | eta: 20.7m
step 04218/05000 (84.36%) | loss: 3.836040 | lrm: 0.28 | dt: 1545.54ms | tok/sec: 10,600 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 14 | total time: 111.16m | eta: 20.7m
step 04219/05000 (84.38%) | loss: 3.830153 | lrm: 0.28 | dt: 1540.43ms | tok/sec: 10,636 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 14 | total time: 111.19m | eta: 20.6m
step 04220/05000 (84.40%) | loss: 3.834708 | lrm: 0.28 | dt: 1578.18ms | tok/sec: 10,381 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 14 | total time: 111.21m | eta: 20.6m
step 04221/05000 (84.42%) | loss: 3.837693 | lrm: 0.28 | dt: 1540.97ms | tok/sec: 10,632 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 14 | total time: 111.24m | eta: 20.6m
step 04222/05000 (84.44%) | loss: 3.836803 | lrm: 0.28 | dt: 1557.52ms | tok/sec: 10,519 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 14 | total time: 111.26m | eta: 20.6m
step 04223/05000 (84.46%) | loss: 3.827076 | lrm: 0.28 | dt: 1522.51ms | tok/sec: 10,761 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 15 | total time: 111.29m | eta: 20.5m
step 04224/05000 (84.48%) | loss: 3.830727 | lrm: 0.28 | dt: 1551.87ms | tok/sec: 10,557 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 15 | total time: 111.32m | eta: 20.5m
step 04225/05000 (84.50%) | loss: 3.822067 | lrm: 0.28 | dt: 1533.09ms | tok/sec: 10,686 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 15 | total time: 111.34m | eta: 20.5m
step 04226/05000 (84.52%) | loss: 3.809119 | lrm: 0.28 | dt: 1517.69ms | tok/sec: 10,795 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 15 | total time: 111.37m | eta: 20.4m
step 04227/05000 (84.54%) | loss: 3.810992 | lrm: 0.28 | dt: 1537.72ms | tok/sec: 10,654 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 15 | total time: 111.39m | eta: 20.4m
step 04228/05000 (84.56%) | loss: 3.820762 | lrm: 0.28 | dt: 1568.98ms | tok/sec: 10,442 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 15 | total time: 111.42m | eta: 20.4m
step 04229/05000 (84.58%) | loss: 3.780804 | lrm: 0.28 | dt: 1542.00ms | tok/sec: 10,625 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 15 | total time: 111.44m | eta: 20.4m
step 04230/05000 (84.60%) | loss: 3.776017 | lrm: 0.28 | dt: 1529.18ms | tok/sec: 10,714 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 15 | total time: 111.47m | eta: 20.3m
step 04231/05000 (84.62%) | loss: 3.786355 | lrm: 0.27 | dt: 1545.57ms | tok/sec: 10,600 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 15 | total time: 111.49m | eta: 20.3m
step 04232/05000 (84.64%) | loss: 3.780932 | lrm: 0.27 | dt: 1561.39ms | tok/sec: 10,493 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 15 | total time: 111.52m | eta: 20.3m
step 04233/05000 (84.66%) | loss: 3.786867 | lrm: 0.27 | dt: 1558.87ms | tok/sec: 10,510 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 15 | total time: 111.55m | eta: 20.3m
step 04234/05000 (84.68%) | loss: 3.807777 | lrm: 0.27 | dt: 1563.32ms | tok/sec: 10,480 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 15 | total time: 111.57m | eta: 20.2m
step 04235/05000 (84.70%) | loss: 3.789410 | lrm: 0.27 | dt: 1582.93ms | tok/sec: 10,350 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 15 | total time: 111.60m | eta: 20.2m
step 04236/05000 (84.72%) | loss: 3.794197 | lrm: 0.27 | dt: 1546.89ms | tok/sec: 10,591 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 15 | total time: 111.63m | eta: 20.2m
step 04237/05000 (84.74%) | loss: 3.803392 | lrm: 0.27 | dt: 1547.58ms | tok/sec: 10,586 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 15 | total time: 111.65m | eta: 20.2m
step 04238/05000 (84.76%) | loss: 3.826723 | lrm: 0.27 | dt: 1524.06ms | tok/sec: 10,750 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 15 | total time: 111.68m | eta: 20.1m
step 04239/05000 (84.78%) | loss: 3.816404 | lrm: 0.27 | dt: 1576.29ms | tok/sec: 10,394 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 15 | total time: 111.70m | eta: 20.1m
step 04240/05000 (84.80%) | loss: 3.816168 | lrm: 0.27 | dt: 1563.69ms | tok/sec: 10,477 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 15 | total time: 111.73m | eta: 20.1m
step 04241/05000 (84.82%) | loss: 3.809931 | lrm: 0.27 | dt: 1570.43ms | tok/sec: 10,432 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 15 | total time: 111.75m | eta: 20.0m
step 04242/05000 (84.84%) | loss: 3.829848 | lrm: 0.27 | dt: 1575.52ms | tok/sec: 10,399 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 15 | total time: 111.78m | eta: 20.0m
step 04243/05000 (84.86%) | loss: 3.818052 | lrm: 0.27 | dt: 1549.97ms | tok/sec: 10,570 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 15 | total time: 111.81m | eta: 20.0m
step 04244/05000 (84.88%) | loss: 3.810514 | lrm: 0.27 | dt: 1545.58ms | tok/sec: 10,600 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 15 | total time: 111.83m | eta: 20.0m
step 04245/05000 (84.90%) | loss: 3.821387 | lrm: 0.27 | dt: 1559.57ms | tok/sec: 10,505 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 15 | total time: 111.86m | eta: 19.9m
step 04246/05000 (84.92%) | loss: 3.836639 | lrm: 0.27 | dt: 1559.86ms | tok/sec: 10,503 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 16 | total time: 111.88m | eta: 19.9m
step 04247/05000 (84.94%) | loss: 3.826634 | lrm: 0.27 | dt: 1630.28ms | tok/sec: 10,049 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 16 | total time: 111.91m | eta: 19.9m
step 04248/05000 (84.96%) | loss: 3.830065 | lrm: 0.27 | dt: 1635.76ms | tok/sec: 10,016 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 16 | total time: 111.94m | eta: 19.9m
step 04249/05000 (84.98%) | loss: 3.812596 | lrm: 0.27 | dt: 1572.69ms | tok/sec: 10,417 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 16 | total time: 111.97m | eta: 19.8m
step 04250/05000 (85.00%) | loss: 3.812713 | lrm: 0.27 | dt: 1530.04ms | tok/sec: 10,708 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 16 | total time: 111.99m | eta: 19.8m
step 04251/05000 (85.02%) | loss: 3.812513 | lrm: 0.27 | dt: 1516.32ms | tok/sec: 10,805 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 16 | total time: 112.02m | eta: 19.8m
step 04252/05000 (85.04%) | loss: 3.821328 | lrm: 0.27 | dt: 1533.22ms | tok/sec: 10,685 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 16 | total time: 112.04m | eta: 19.8m
step 04253/05000 (85.06%) | loss: 3.823475 | lrm: 0.27 | dt: 1543.90ms | tok/sec: 10,612 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 16 | total time: 112.07m | eta: 19.7m
step 04254/05000 (85.08%) | loss: 3.826028 | lrm: 0.27 | dt: 1531.75ms | tok/sec: 10,696 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 16 | total time: 112.09m | eta: 19.7m
step 04255/05000 (85.10%) | loss: 3.824702 | lrm: 0.27 | dt: 1545.10ms | tok/sec: 10,603 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 16 | total time: 112.12m | eta: 19.7m
step 04256/05000 (85.12%) | loss: 3.812018 | lrm: 0.27 | dt: 1511.60ms | tok/sec: 10,838 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 16 | total time: 112.14m | eta: 19.7m
step 04257/05000 (85.14%) | loss: 3.816709 | lrm: 0.27 | dt: 1538.85ms | tok/sec: 10,646 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 16 | total time: 112.17m | eta: 19.6m
step 04258/05000 (85.16%) | loss: 3.830315 | lrm: 0.27 | dt: 1542.38ms | tok/sec: 10,622 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 16 | total time: 112.20m | eta: 19.6m
step 04259/05000 (85.18%) | loss: 3.838430 | lrm: 0.27 | dt: 1554.10ms | tok/sec: 10,542 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 16 | total time: 112.22m | eta: 19.6m
step 04260/05000 (85.20%) | loss: 3.821162 | lrm: 0.27 | dt: 1530.78ms | tok/sec: 10,703 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 16 | total time: 112.25m | eta: 19.5m
step 04261/05000 (85.22%) | loss: 3.814188 | lrm: 0.27 | dt: 1534.80ms | tok/sec: 10,675 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 16 | total time: 112.27m | eta: 19.5m
step 04262/05000 (85.24%) | loss: 3.828225 | lrm: 0.27 | dt: 1538.27ms | tok/sec: 10,650 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 16 | total time: 112.30m | eta: 19.5m
step 04263/05000 (85.26%) | loss: 3.824066 | lrm: 0.27 | dt: 1546.06ms | tok/sec: 10,597 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 16 | total time: 112.32m | eta: 19.5m
step 04264/05000 (85.28%) | loss: 3.821119 | lrm: 0.27 | dt: 1564.64ms | tok/sec: 10,471 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 16 | total time: 112.35m | eta: 19.4m
step 04265/05000 (85.30%) | loss: 3.821681 | lrm: 0.26 | dt: 1585.11ms | tok/sec: 10,336 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 16 | total time: 112.38m | eta: 19.4m
step 04266/05000 (85.32%) | loss: 3.800015 | lrm: 0.26 | dt: 1558.99ms | tok/sec: 10,509 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 16 | total time: 112.40m | eta: 19.4m
step 04267/05000 (85.34%) | loss: 3.798798 | lrm: 0.26 | dt: 1526.40ms | tok/sec: 10,733 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 16 | total time: 112.43m | eta: 19.4m
step 04268/05000 (85.36%) | loss: 3.799950 | lrm: 0.26 | dt: 1542.21ms | tok/sec: 10,623 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 16 | total time: 112.45m | eta: 19.3m
step 04269/05000 (85.38%) | loss: 3.784801 | lrm: 0.26 | dt: 1559.85ms | tok/sec: 10,503 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 17 | total time: 112.48m | eta: 19.3m
step 04270/05000 (85.40%) | loss: 3.804720 | lrm: 0.26 | dt: 1570.99ms | tok/sec: 10,429 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 17 | total time: 112.51m | eta: 19.3m
step 04271/05000 (85.42%) | loss: 3.795218 | lrm: 0.26 | dt: 1545.10ms | tok/sec: 10,603 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 17 | total time: 112.53m | eta: 19.3m
step 04272/05000 (85.44%) | loss: 3.786523 | lrm: 0.26 | dt: 1553.10ms | tok/sec: 10,549 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 17 | total time: 112.56m | eta: 19.2m
step 04273/05000 (85.46%) | loss: 3.783690 | lrm: 0.26 | dt: 1546.58ms | tok/sec: 10,593 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 17 | total time: 112.58m | eta: 19.2m
step 04274/05000 (85.48%) | loss: 3.770930 | lrm: 0.26 | dt: 1571.93ms | tok/sec: 10,422 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 17 | total time: 112.61m | eta: 19.2m
step 04275/05000 (85.50%) | loss: 3.758853 | lrm: 0.26 | dt: 1536.46ms | tok/sec: 10,663 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 17 | total time: 112.63m | eta: 19.1m
step 04276/05000 (85.52%) | loss: 3.770265 | lrm: 0.26 | dt: 1577.98ms | tok/sec: 10,382 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 17 | total time: 112.66m | eta: 19.1m
step 04277/05000 (85.54%) | loss: 3.784029 | lrm: 0.26 | dt: 1554.44ms | tok/sec: 10,540 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 17 | total time: 112.69m | eta: 19.1m
step 04278/05000 (85.56%) | loss: 3.773554 | lrm: 0.26 | dt: 1555.59ms | tok/sec: 10,532 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 17 | total time: 112.71m | eta: 19.1m
step 04279/05000 (85.58%) | loss: 3.777012 | lrm: 0.26 | dt: 1573.21ms | tok/sec: 10,414 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 17 | total time: 112.74m | eta: 19.0m
step 04280/05000 (85.60%) | loss: 3.778262 | lrm: 0.26 | dt: 1560.91ms | tok/sec: 10,496 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 17 | total time: 112.77m | eta: 19.0m
step 04281/05000 (85.62%) | loss: 3.773585 | lrm: 0.26 | dt: 1558.29ms | tok/sec: 10,514 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 17 | total time: 112.79m | eta: 19.0m
step 04282/05000 (85.64%) | loss: 3.782802 | lrm: 0.26 | dt: 1542.31ms | tok/sec: 10,623 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 17 | total time: 112.82m | eta: 19.0m
step 04283/05000 (85.66%) | loss: 3.770724 | lrm: 0.26 | dt: 1518.66ms | tok/sec: 10,788 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 17 | total time: 112.84m | eta: 18.9m
step 04284/05000 (85.68%) | loss: 3.766062 | lrm: 0.26 | dt: 1544.10ms | tok/sec: 10,610 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 17 | total time: 112.87m | eta: 18.9m
step 04285/05000 (85.70%) | loss: 3.774356 | lrm: 0.26 | dt: 1550.59ms | tok/sec: 10,566 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 17 | total time: 112.89m | eta: 18.9m
step 04286/05000 (85.72%) | loss: 3.775662 | lrm: 0.26 | dt: 1552.39ms | tok/sec: 10,554 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 17 | total time: 112.92m | eta: 18.9m
step 04287/05000 (85.74%) | loss: 3.774860 | lrm: 0.26 | dt: 1554.18ms | tok/sec: 10,541 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 17 | total time: 112.95m | eta: 18.8m
step 04288/05000 (85.76%) | loss: 3.784024 | lrm: 0.26 | dt: 1491.21ms | tok/sec: 10,987 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 17 | total time: 112.97m | eta: 18.8m
step 04289/05000 (85.78%) | loss: 3.797045 | lrm: 0.26 | dt: 1576.63ms | tok/sec: 10,391 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 17 | total time: 113.00m | eta: 18.8m
step 04290/05000 (85.80%) | loss: 3.769049 | lrm: 0.26 | dt: 1550.74ms | tok/sec: 10,565 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 17 | total time: 113.02m | eta: 18.7m
step 04291/05000 (85.82%) | loss: 3.787059 | lrm: 0.26 | dt: 1514.37ms | tok/sec: 10,819 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 17 | total time: 113.05m | eta: 18.7m
step 04292/05000 (85.84%) | loss: 3.773944 | lrm: 0.26 | dt: 1557.76ms | tok/sec: 10,517 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 18 | total time: 113.07m | eta: 18.7m
step 04293/05000 (85.86%) | loss: 3.771667 | lrm: 0.26 | dt: 1524.85ms | tok/sec: 10,744 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 18 | total time: 113.10m | eta: 18.7m
step 04294/05000 (85.88%) | loss: 3.766368 | lrm: 0.26 | dt: 1553.74ms | tok/sec: 10,544 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 18 | total time: 113.12m | eta: 18.6m
step 04295/05000 (85.90%) | loss: 3.764358 | lrm: 0.26 | dt: 1566.50ms | tok/sec: 10,458 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 18 | total time: 113.15m | eta: 18.6m
step 04296/05000 (85.92%) | loss: 3.760875 | lrm: 0.26 | dt: 1533.52ms | tok/sec: 10,683 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 18 | total time: 113.18m | eta: 18.6m
step 04297/05000 (85.94%) | loss: 3.768983 | lrm: 0.26 | dt: 1524.75ms | tok/sec: 10,745 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 18 | total time: 113.20m | eta: 18.6m
step 04298/05000 (85.96%) | loss: 3.754363 | lrm: 0.26 | dt: 1561.26ms | tok/sec: 10,494 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 18 | total time: 113.23m | eta: 18.5m
step 04299/05000 (85.98%) | loss: 3.763929 | lrm: 0.25 | dt: 1559.94ms | tok/sec: 10,502 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 18 | total time: 113.25m | eta: 18.5m
Step 04300 | Validation bpb: 1.179723
<|bos|>The capital of France is the capital of France. The capital of France is the capital of France. The
<|bos|>The chemical symbol of gold is the symbol of the gold's gold. It is a symbol of the gold's
<|bos|>If yesterday was Friday, then tomorrow will be the next day. The next day will be Friday, then Thursday, then Thursday
<|bos|>The opposite of hot is the heat of the sun. The heat of the sun is the heat of the
<|bos|>The planets of the solar system are: the solar system, the solar system, the solar system, the solar system,
<|bos|>My favorite color is the white, but I don't know how to use it. I'm not
<|bos|>If 5*x + 3 = 13, then x is the number of points in the equation. The equation is the number of points in
step 04300/05000 (86.00%) | loss: 3.766881 | lrm: 0.25 | dt: 1528.64ms | tok/sec: 10,718 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 18 | total time: 113.28m | eta: 18.5m
step 04301/05000 (86.02%) | loss: 3.768366 | lrm: 0.25 | dt: 1517.44ms | tok/sec: 10,797 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 18 | total time: 113.30m | eta: 18.5m
step 04302/05000 (86.04%) | loss: 3.763002 | lrm: 0.25 | dt: 1537.22ms | tok/sec: 10,658 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 18 | total time: 113.33m | eta: 18.4m
step 04303/05000 (86.06%) | loss: 3.756183 | lrm: 0.25 | dt: 1544.34ms | tok/sec: 10,609 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 18 | total time: 113.36m | eta: 18.4m
step 04304/05000 (86.08%) | loss: 3.772438 | lrm: 0.25 | dt: 1537.93ms | tok/sec: 10,653 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 18 | total time: 113.38m | eta: 18.4m
step 04305/05000 (86.10%) | loss: 3.776039 | lrm: 0.25 | dt: 1566.46ms | tok/sec: 10,459 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 18 | total time: 113.41m | eta: 18.4m
step 04306/05000 (86.12%) | loss: 3.780666 | lrm: 0.25 | dt: 1545.55ms | tok/sec: 10,600 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 18 | total time: 113.43m | eta: 18.3m
step 04307/05000 (86.14%) | loss: 3.777848 | lrm: 0.25 | dt: 1568.36ms | tok/sec: 10,446 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 18 | total time: 113.46m | eta: 18.3m
step 04308/05000 (86.16%) | loss: 3.789819 | lrm: 0.25 | dt: 1461.25ms | tok/sec: 11,212 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 18 | total time: 113.48m | eta: 18.3m
step 04309/05000 (86.18%) | loss: 3.779253 | lrm: 0.25 | dt: 1561.61ms | tok/sec: 10,491 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 18 | total time: 113.51m | eta: 18.2m
step 04310/05000 (86.20%) | loss: 3.791084 | lrm: 0.25 | dt: 1529.50ms | tok/sec: 10,711 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 18 | total time: 113.54m | eta: 18.2m
step 04311/05000 (86.22%) | loss: 3.805854 | lrm: 0.25 | dt: 1534.83ms | tok/sec: 10,674 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 18 | total time: 113.56m | eta: 18.2m
step 04312/05000 (86.24%) | loss: 3.787024 | lrm: 0.25 | dt: 1614.72ms | tok/sec: 10,146 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 18 | total time: 113.59m | eta: 18.2m
step 04313/05000 (86.26%) | loss: 3.786134 | lrm: 0.25 | dt: 1615.20ms | tok/sec: 10,143 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 18 | total time: 113.62m | eta: 18.1m
step 04314/05000 (86.28%) | loss: 3.785033 | lrm: 0.25 | dt: 1603.01ms | tok/sec: 10,220 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 18 | total time: 113.64m | eta: 18.1m
step 04315/05000 (86.30%) | loss: 3.776967 | lrm: 0.25 | dt: 1474.08ms | tok/sec: 11,114 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 19 | total time: 113.67m | eta: 18.1m
step 04316/05000 (86.32%) | loss: 3.778820 | lrm: 0.25 | dt: 1568.19ms | tok/sec: 10,447 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 19 | total time: 113.69m | eta: 18.1m
step 04317/05000 (86.34%) | loss: 3.785614 | lrm: 0.25 | dt: 1569.05ms | tok/sec: 10,441 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 19 | total time: 113.72m | eta: 18.0m
step 04318/05000 (86.36%) | loss: 3.796374 | lrm: 0.25 | dt: 1604.81ms | tok/sec: 10,209 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 19 | total time: 113.75m | eta: 18.0m
step 04319/05000 (86.38%) | loss: 3.790271 | lrm: 0.25 | dt: 1510.42ms | tok/sec: 10,847 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 19 | total time: 113.77m | eta: 18.0m
step 04320/05000 (86.40%) | loss: 3.785475 | lrm: 0.25 | dt: 1472.30ms | tok/sec: 11,128 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 19 | total time: 113.80m | eta: 18.0m
step 04321/05000 (86.42%) | loss: 3.787976 | lrm: 0.25 | dt: 1475.61ms | tok/sec: 11,103 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 19 | total time: 113.82m | eta: 17.9m
step 04322/05000 (86.44%) | loss: 3.800758 | lrm: 0.25 | dt: 1494.14ms | tok/sec: 10,965 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 19 | total time: 113.84m | eta: 17.9m
step 04323/05000 (86.46%) | loss: 3.793094 | lrm: 0.25 | dt: 1692.78ms | tok/sec: 9,678 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 19 | total time: 113.87m | eta: 17.9m
step 04324/05000 (86.48%) | loss: 3.801263 | lrm: 0.25 | dt: 1631.88ms | tok/sec: 10,039 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 19 | total time: 113.90m | eta: 17.8m
step 04325/05000 (86.50%) | loss: 3.792619 | lrm: 0.25 | dt: 1604.69ms | tok/sec: 10,210 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 19 | total time: 113.93m | eta: 17.8m
step 04326/05000 (86.52%) | loss: 3.801225 | lrm: 0.25 | dt: 1472.81ms | tok/sec: 11,124 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 19 | total time: 113.95m | eta: 17.8m
step 04327/05000 (86.54%) | loss: 3.795085 | lrm: 0.25 | dt: 1652.14ms | tok/sec: 9,916 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 19 | total time: 113.98m | eta: 17.8m
step 04328/05000 (86.56%) | loss: 3.800572 | lrm: 0.25 | dt: 1572.55ms | tok/sec: 10,418 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 19 | total time: 114.01m | eta: 17.7m
step 04329/05000 (86.58%) | loss: 3.788572 | lrm: 0.25 | dt: 1503.83ms | tok/sec: 10,894 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 19 | total time: 114.03m | eta: 17.7m
step 04330/05000 (86.60%) | loss: 3.773988 | lrm: 0.25 | dt: 1678.12ms | tok/sec: 9,763 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 19 | total time: 114.06m | eta: 17.7m
step 04331/05000 (86.62%) | loss: 3.763532 | lrm: 0.25 | dt: 1737.88ms | tok/sec: 9,427 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 19 | total time: 114.09m | eta: 17.7m
step 04332/05000 (86.64%) | loss: 3.763943 | lrm: 0.25 | dt: 1692.14ms | tok/sec: 9,682 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 19 | total time: 114.12m | eta: 17.6m
step 04333/05000 (86.66%) | loss: 3.761137 | lrm: 0.24 | dt: 1710.59ms | tok/sec: 9,578 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 19 | total time: 114.14m | eta: 17.6m
step 04334/05000 (86.68%) | loss: 3.766983 | lrm: 0.24 | dt: 1654.27ms | tok/sec: 9,904 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 19 | total time: 114.17m | eta: 17.6m
step 04335/05000 (86.70%) | loss: 3.771562 | lrm: 0.24 | dt: 1592.26ms | tok/sec: 10,289 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 19 | total time: 114.20m | eta: 17.6m
step 04336/05000 (86.72%) | loss: 3.769494 | lrm: 0.24 | dt: 1654.92ms | tok/sec: 9,900 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 19 | total time: 114.23m | eta: 17.5m
step 04337/05000 (86.74%) | loss: 3.777055 | lrm: 0.24 | dt: 1574.43ms | tok/sec: 10,406 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 19 | total time: 114.25m | eta: 17.5m
step 04338/05000 (86.76%) | loss: 3.774829 | lrm: 0.24 | dt: 1533.84ms | tok/sec: 10,681 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 20 | total time: 114.28m | eta: 17.5m
step 04339/05000 (86.78%) | loss: 3.773632 | lrm: 0.24 | dt: 1555.78ms | tok/sec: 10,531 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 20 | total time: 114.30m | eta: 17.5m
step 04340/05000 (86.80%) | loss: 3.771573 | lrm: 0.24 | dt: 1530.02ms | tok/sec: 10,708 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 20 | total time: 114.33m | eta: 17.4m
step 04341/05000 (86.82%) | loss: 3.748552 | lrm: 0.24 | dt: 1564.58ms | tok/sec: 10,471 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 20 | total time: 114.35m | eta: 17.4m
step 04342/05000 (86.84%) | loss: 3.756196 | lrm: 0.24 | dt: 1517.86ms | tok/sec: 10,794 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 20 | total time: 114.38m | eta: 17.4m
step 04343/05000 (86.86%) | loss: 3.777378 | lrm: 0.24 | dt: 1537.70ms | tok/sec: 10,654 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 20 | total time: 114.41m | eta: 17.3m
step 04344/05000 (86.88%) | loss: 3.779811 | lrm: 0.24 | dt: 1502.66ms | tok/sec: 10,903 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 20 | total time: 114.43m | eta: 17.3m
step 04345/05000 (86.90%) | loss: 3.774536 | lrm: 0.24 | dt: 1528.88ms | tok/sec: 10,716 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 20 | total time: 114.46m | eta: 17.3m
step 04346/05000 (86.92%) | loss: 3.788232 | lrm: 0.24 | dt: 1518.58ms | tok/sec: 10,789 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 20 | total time: 114.48m | eta: 17.3m
step 04347/05000 (86.94%) | loss: 3.782753 | lrm: 0.24 | dt: 1512.10ms | tok/sec: 10,835 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 20 | total time: 114.51m | eta: 17.2m
step 04348/05000 (86.96%) | loss: 3.790407 | lrm: 0.24 | dt: 1543.36ms | tok/sec: 10,615 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 20 | total time: 114.53m | eta: 17.2m
step 04349/05000 (86.98%) | loss: 3.796736 | lrm: 0.24 | dt: 1545.14ms | tok/sec: 10,603 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 20 | total time: 114.56m | eta: 17.2m
step 04350/05000 (87.00%) | loss: 3.816668 | lrm: 0.24 | dt: 1545.93ms | tok/sec: 10,598 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 20 | total time: 114.58m | eta: 17.2m
step 04351/05000 (87.02%) | loss: 3.823477 | lrm: 0.24 | dt: 1537.95ms | tok/sec: 10,653 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 20 | total time: 114.61m | eta: 17.1m
step 04352/05000 (87.04%) | loss: 3.828237 | lrm: 0.24 | dt: 1545.46ms | tok/sec: 10,601 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 20 | total time: 114.64m | eta: 17.1m
step 04353/05000 (87.06%) | loss: 3.821066 | lrm: 0.24 | dt: 1532.10ms | tok/sec: 10,693 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 20 | total time: 114.66m | eta: 17.1m
step 04354/05000 (87.08%) | loss: 3.824571 | lrm: 0.24 | dt: 1567.96ms | tok/sec: 10,449 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 20 | total time: 114.69m | eta: 17.1m
step 04355/05000 (87.10%) | loss: 3.815278 | lrm: 0.24 | dt: 1547.54ms | tok/sec: 10,587 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 20 | total time: 114.71m | eta: 17.0m
step 04356/05000 (87.12%) | loss: 3.813679 | lrm: 0.24 | dt: 1557.86ms | tok/sec: 10,516 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 20 | total time: 114.74m | eta: 17.0m
step 04357/05000 (87.14%) | loss: 3.818902 | lrm: 0.24 | dt: 1542.19ms | tok/sec: 10,623 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 20 | total time: 114.76m | eta: 17.0m
step 04358/05000 (87.16%) | loss: 3.806978 | lrm: 0.24 | dt: 1566.78ms | tok/sec: 10,457 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 20 | total time: 114.79m | eta: 16.9m
step 04359/05000 (87.18%) | loss: 3.810104 | lrm: 0.24 | dt: 1554.07ms | tok/sec: 10,542 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 20 | total time: 114.82m | eta: 16.9m
step 04360/05000 (87.20%) | loss: 3.812025 | lrm: 0.24 | dt: 1540.55ms | tok/sec: 10,635 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 20 | total time: 114.84m | eta: 16.9m
step 04361/05000 (87.22%) | loss: 3.816501 | lrm: 0.24 | dt: 1586.97ms | tok/sec: 10,324 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 20 | total time: 114.87m | eta: 16.9m
step 04362/05000 (87.24%) | loss: 3.836565 | lrm: 0.24 | dt: 1560.45ms | tok/sec: 10,499 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 21 | total time: 114.89m | eta: 16.8m
step 04363/05000 (87.26%) | loss: 3.846390 | lrm: 0.24 | dt: 1553.90ms | tok/sec: 10,543 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 21 | total time: 114.92m | eta: 16.8m
step 04364/05000 (87.28%) | loss: 3.834435 | lrm: 0.24 | dt: 1565.33ms | tok/sec: 10,466 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 21 | total time: 114.95m | eta: 16.8m
step 04365/05000 (87.30%) | loss: 3.813345 | lrm: 0.24 | dt: 1601.25ms | tok/sec: 10,231 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 21 | total time: 114.97m | eta: 16.8m
step 04366/05000 (87.32%) | loss: 3.823375 | lrm: 0.24 | dt: 1566.07ms | tok/sec: 10,461 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 21 | total time: 115.00m | eta: 16.7m
step 04367/05000 (87.34%) | loss: 3.813465 | lrm: 0.24 | dt: 1580.69ms | tok/sec: 10,365 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 21 | total time: 115.03m | eta: 16.7m
step 04368/05000 (87.36%) | loss: 3.811557 | lrm: 0.23 | dt: 1544.13ms | tok/sec: 10,610 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 21 | total time: 115.05m | eta: 16.7m
step 04369/05000 (87.38%) | loss: 3.816641 | lrm: 0.23 | dt: 1532.11ms | tok/sec: 10,693 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 21 | total time: 115.08m | eta: 16.7m
step 04370/05000 (87.40%) | loss: 3.822239 | lrm: 0.23 | dt: 1547.59ms | tok/sec: 10,586 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 21 | total time: 115.10m | eta: 16.6m
step 04371/05000 (87.42%) | loss: 3.818399 | lrm: 0.23 | dt: 1539.08ms | tok/sec: 10,645 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 21 | total time: 115.13m | eta: 16.6m
step 04372/05000 (87.44%) | loss: 3.836894 | lrm: 0.23 | dt: 1530.67ms | tok/sec: 10,703 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 21 | total time: 115.15m | eta: 16.6m
step 04373/05000 (87.46%) | loss: 3.816757 | lrm: 0.23 | dt: 1572.08ms | tok/sec: 10,421 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 21 | total time: 115.18m | eta: 16.6m
step 04374/05000 (87.48%) | loss: 3.816468 | lrm: 0.23 | dt: 1535.19ms | tok/sec: 10,672 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 21 | total time: 115.21m | eta: 16.5m
step 04375/05000 (87.50%) | loss: 3.818369 | lrm: 0.23 | dt: 1544.03ms | tok/sec: 10,611 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 21 | total time: 115.23m | eta: 16.5m
step 04376/05000 (87.52%) | loss: 3.810214 | lrm: 0.23 | dt: 1514.48ms | tok/sec: 10,818 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 21 | total time: 115.26m | eta: 16.5m
step 04377/05000 (87.54%) | loss: 3.795476 | lrm: 0.23 | dt: 1511.22ms | tok/sec: 10,841 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 21 | total time: 115.28m | eta: 16.4m
step 04378/05000 (87.56%) | loss: 3.798895 | lrm: 0.23 | dt: 1549.80ms | tok/sec: 10,571 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 21 | total time: 115.31m | eta: 16.4m
step 04379/05000 (87.58%) | loss: 3.798477 | lrm: 0.23 | dt: 1414.86ms | tok/sec: 11,579 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 21 | total time: 115.33m | eta: 16.4m
step 04380/05000 (87.60%) | loss: 3.789144 | lrm: 0.23 | dt: 1471.00ms | tok/sec: 11,138 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 21 | total time: 115.36m | eta: 16.4m
step 04381/05000 (87.62%) | loss: 3.784331 | lrm: 0.23 | dt: 1495.15ms | tok/sec: 10,958 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 21 | total time: 115.38m | eta: 16.3m
step 04382/05000 (87.64%) | loss: 3.784381 | lrm: 0.23 | dt: 1475.57ms | tok/sec: 11,103 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 21 | total time: 115.41m | eta: 16.3m
step 04383/05000 (87.66%) | loss: 3.778743 | lrm: 0.23 | dt: 1430.62ms | tok/sec: 11,452 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 21 | total time: 115.43m | eta: 16.3m
step 04384/05000 (87.68%) | loss: 3.779760 | lrm: 0.23 | dt: 1425.86ms | tok/sec: 11,490 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 21 | total time: 115.45m | eta: 16.3m
step 04385/05000 (87.70%) | loss: 3.758397 | lrm: 0.23 | dt: 1453.35ms | tok/sec: 11,273 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 22 | total time: 115.48m | eta: 16.2m
step 04386/05000 (87.72%) | loss: 3.764314 | lrm: 0.23 | dt: 1635.56ms | tok/sec: 10,017 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 22 | total time: 115.50m | eta: 16.2m
step 04387/05000 (87.74%) | loss: 3.783277 | lrm: 0.23 | dt: 1613.17ms | tok/sec: 10,156 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 22 | total time: 115.53m | eta: 16.2m
step 04388/05000 (87.76%) | loss: 3.778232 | lrm: 0.23 | dt: 1542.18ms | tok/sec: 10,623 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 22 | total time: 115.56m | eta: 16.2m
step 04389/05000 (87.78%) | loss: 3.776873 | lrm: 0.23 | dt: 1545.77ms | tok/sec: 10,599 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 22 | total time: 115.58m | eta: 16.1m
step 04390/05000 (87.80%) | loss: 3.782897 | lrm: 0.23 | dt: 1536.54ms | tok/sec: 10,662 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 22 | total time: 115.61m | eta: 16.1m
step 04391/05000 (87.82%) | loss: 3.779055 | lrm: 0.23 | dt: 1638.02ms | tok/sec: 10,002 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 22 | total time: 115.64m | eta: 16.1m
step 04392/05000 (87.84%) | loss: 3.762098 | lrm: 0.23 | dt: 1602.56ms | tok/sec: 10,223 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 22 | total time: 115.66m | eta: 16.0m
step 04393/05000 (87.86%) | loss: 3.778593 | lrm: 0.23 | dt: 1629.60ms | tok/sec: 10,054 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 22 | total time: 115.69m | eta: 16.0m
step 04394/05000 (87.88%) | loss: 3.780209 | lrm: 0.23 | dt: 1605.16ms | tok/sec: 10,207 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 22 | total time: 115.72m | eta: 16.0m
step 04395/05000 (87.90%) | loss: 3.782813 | lrm: 0.23 | dt: 1564.68ms | tok/sec: 10,471 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 22 | total time: 115.74m | eta: 16.0m
step 04396/05000 (87.92%) | loss: 3.773476 | lrm: 0.23 | dt: 1551.11ms | tok/sec: 10,562 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 22 | total time: 115.77m | eta: 15.9m
step 04397/05000 (87.94%) | loss: 3.784555 | lrm: 0.23 | dt: 1522.78ms | tok/sec: 10,759 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 22 | total time: 115.79m | eta: 15.9m
step 04398/05000 (87.96%) | loss: 3.796286 | lrm: 0.23 | dt: 1564.82ms | tok/sec: 10,470 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 22 | total time: 115.82m | eta: 15.9m
step 04399/05000 (87.98%) | loss: 3.791538 | lrm: 0.23 | dt: 1578.32ms | tok/sec: 10,380 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 22 | total time: 115.85m | eta: 15.9m
Step 04400 | Validation bpb: 1.176897
<|bos|>The capital of France is the capital of France. The capital of France is the capital of France. The
<|bos|>The chemical symbol of gold is the symbol of the gold. It is a symbol of the gold's gold,
<|bos|>If yesterday was Friday, then tomorrow will be the last day of the week. The last day of the week is Friday,
<|bos|>The opposite of hot is the same as the hot, but the same as the hot, but the same
<|bos|>The planets of the solar system are: the solar system, the solar system, the solar system, the solar system,
<|bos|>My favorite color is the white, but I don't know what I'm doing. I'm not
<|bos|>If 5*x + 3 = 13, then x is 3.5*x + 3.5*x + 3
step 04400/05000 (88.00%) | loss: 3.793079 | lrm: 0.23 | dt: 1637.05ms | tok/sec: 10,008 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 22 | total time: 115.87m | eta: 15.8m
step 04401/05000 (88.02%) | loss: 3.791754 | lrm: 0.23 | dt: 1536.90ms | tok/sec: 10,660 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 22 | total time: 115.90m | eta: 15.8m
step 04402/05000 (88.04%) | loss: 3.782394 | lrm: 0.22 | dt: 1568.39ms | tok/sec: 10,446 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 22 | total time: 115.93m | eta: 15.8m
step 04403/05000 (88.06%) | loss: 3.763752 | lrm: 0.22 | dt: 1534.93ms | tok/sec: 10,674 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 22 | total time: 115.95m | eta: 15.8m
step 04404/05000 (88.08%) | loss: 3.772860 | lrm: 0.22 | dt: 1551.29ms | tok/sec: 10,561 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 22 | total time: 115.98m | eta: 15.7m
step 04405/05000 (88.10%) | loss: 3.764264 | lrm: 0.22 | dt: 1499.88ms | tok/sec: 10,923 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 22 | total time: 116.00m | eta: 15.7m
step 04406/05000 (88.12%) | loss: 3.770144 | lrm: 0.22 | dt: 1522.20ms | tok/sec: 10,763 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 22 | total time: 116.03m | eta: 15.7m
step 04407/05000 (88.14%) | loss: 3.780262 | lrm: 0.22 | dt: 1552.14ms | tok/sec: 10,555 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 22 | total time: 116.05m | eta: 15.7m
step 04408/05000 (88.16%) | loss: 3.795374 | lrm: 0.22 | dt: 1544.01ms | tok/sec: 10,611 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 23 | total time: 116.08m | eta: 15.6m
step 04409/05000 (88.18%) | loss: 3.797145 | lrm: 0.22 | dt: 1569.48ms | tok/sec: 10,439 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 23 | total time: 116.10m | eta: 15.6m
step 04410/05000 (88.20%) | loss: 3.805321 | lrm: 0.22 | dt: 1569.55ms | tok/sec: 10,438 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 23 | total time: 116.13m | eta: 15.6m
step 04411/05000 (88.22%) | loss: 3.796201 | lrm: 0.22 | dt: 1657.34ms | tok/sec: 9,885 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 23 | total time: 116.16m | eta: 15.5m
step 04412/05000 (88.24%) | loss: 3.793161 | lrm: 0.22 | dt: 1642.83ms | tok/sec: 9,973 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 23 | total time: 116.19m | eta: 15.5m
step 04413/05000 (88.26%) | loss: 3.794100 | lrm: 0.22 | dt: 1549.87ms | tok/sec: 10,571 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 23 | total time: 116.21m | eta: 15.5m
step 04414/05000 (88.28%) | loss: 3.799306 | lrm: 0.22 | dt: 1569.00ms | tok/sec: 10,442 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 23 | total time: 116.24m | eta: 15.5m
step 04415/05000 (88.30%) | loss: 3.788691 | lrm: 0.22 | dt: 1543.91ms | tok/sec: 10,612 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 23 | total time: 116.26m | eta: 15.4m
step 04416/05000 (88.32%) | loss: 3.784935 | lrm: 0.22 | dt: 1550.30ms | tok/sec: 10,568 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 23 | total time: 116.29m | eta: 15.4m
step 04417/05000 (88.34%) | loss: 3.794518 | lrm: 0.22 | dt: 1540.84ms | tok/sec: 10,633 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 23 | total time: 116.32m | eta: 15.4m
step 04418/05000 (88.36%) | loss: 3.794450 | lrm: 0.22 | dt: 1555.80ms | tok/sec: 10,530 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 23 | total time: 116.34m | eta: 15.4m
step 04419/05000 (88.38%) | loss: 3.805738 | lrm: 0.22 | dt: 1552.60ms | tok/sec: 10,552 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 23 | total time: 116.37m | eta: 15.3m
step 04420/05000 (88.40%) | loss: 3.798268 | lrm: 0.22 | dt: 1559.84ms | tok/sec: 10,503 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 23 | total time: 116.39m | eta: 15.3m
step 04421/05000 (88.42%) | loss: 3.789372 | lrm: 0.22 | dt: 1551.95ms | tok/sec: 10,557 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 23 | total time: 116.42m | eta: 15.3m
step 04422/05000 (88.44%) | loss: 3.801347 | lrm: 0.22 | dt: 1547.27ms | tok/sec: 10,588 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 23 | total time: 116.44m | eta: 15.3m
step 04423/05000 (88.46%) | loss: 3.792705 | lrm: 0.22 | dt: 1573.17ms | tok/sec: 10,414 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 23 | total time: 116.47m | eta: 15.2m
step 04424/05000 (88.48%) | loss: 3.795978 | lrm: 0.22 | dt: 1569.27ms | tok/sec: 10,440 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 23 | total time: 116.50m | eta: 15.2m
step 04425/05000 (88.50%) | loss: 3.802364 | lrm: 0.22 | dt: 1549.73ms | tok/sec: 10,572 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 23 | total time: 116.52m | eta: 15.2m
step 04426/05000 (88.52%) | loss: 3.805952 | lrm: 0.22 | dt: 1548.37ms | tok/sec: 10,581 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 23 | total time: 116.55m | eta: 15.1m
step 04427/05000 (88.54%) | loss: 3.815466 | lrm: 0.22 | dt: 1533.40ms | tok/sec: 10,684 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 23 | total time: 116.57m | eta: 15.1m
step 04428/05000 (88.56%) | loss: 3.802818 | lrm: 0.22 | dt: 1570.77ms | tok/sec: 10,430 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 23 | total time: 116.60m | eta: 15.1m
step 04429/05000 (88.58%) | loss: 3.800630 | lrm: 0.22 | dt: 1561.58ms | tok/sec: 10,491 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 23 | total time: 116.63m | eta: 15.1m
step 04430/05000 (88.60%) | loss: 3.792831 | lrm: 0.22 | dt: 1634.20ms | tok/sec: 10,025 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 23 | total time: 116.65m | eta: 15.0m
step 04431/05000 (88.62%) | loss: 3.804510 | lrm: 0.22 | dt: 1601.28ms | tok/sec: 10,231 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 23 | total time: 116.68m | eta: 15.0m
step 04432/05000 (88.64%) | loss: 3.811157 | lrm: 0.22 | dt: 1594.97ms | tok/sec: 10,272 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 24 | total time: 116.71m | eta: 15.0m
step 04433/05000 (88.66%) | loss: 3.812454 | lrm: 0.22 | dt: 1580.59ms | tok/sec: 10,365 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 24 | total time: 116.73m | eta: 15.0m
step 04434/05000 (88.68%) | loss: 3.818096 | lrm: 0.22 | dt: 1537.35ms | tok/sec: 10,657 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 24 | total time: 116.76m | eta: 14.9m
step 04435/05000 (88.70%) | loss: 3.806591 | lrm: 0.22 | dt: 1547.82ms | tok/sec: 10,585 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 24 | total time: 116.78m | eta: 14.9m
step 04436/05000 (88.72%) | loss: 3.811479 | lrm: 0.21 | dt: 1559.44ms | tok/sec: 10,506 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 24 | total time: 116.81m | eta: 14.9m
step 04437/05000 (88.74%) | loss: 3.801892 | lrm: 0.21 | dt: 1543.35ms | tok/sec: 10,615 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 24 | total time: 116.84m | eta: 14.9m
step 04438/05000 (88.76%) | loss: 3.810062 | lrm: 0.21 | dt: 1470.46ms | tok/sec: 11,142 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 24 | total time: 116.86m | eta: 14.8m
step 04439/05000 (88.78%) | loss: 3.820329 | lrm: 0.21 | dt: 1532.01ms | tok/sec: 10,694 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 24 | total time: 116.89m | eta: 14.8m
step 04440/05000 (88.80%) | loss: 3.813065 | lrm: 0.21 | dt: 1468.00ms | tok/sec: 11,160 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 24 | total time: 116.91m | eta: 14.8m
step 04441/05000 (88.82%) | loss: 3.804632 | lrm: 0.21 | dt: 1544.96ms | tok/sec: 10,604 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 24 | total time: 116.94m | eta: 14.8m
step 04442/05000 (88.84%) | loss: 3.805791 | lrm: 0.21 | dt: 1566.78ms | tok/sec: 10,457 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 24 | total time: 116.96m | eta: 14.7m
step 04443/05000 (88.86%) | loss: 3.804141 | lrm: 0.21 | dt: 1601.50ms | tok/sec: 10,230 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 24 | total time: 116.99m | eta: 14.7m
step 04444/05000 (88.88%) | loss: 3.806840 | lrm: 0.21 | dt: 1540.71ms | tok/sec: 10,634 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 24 | total time: 117.01m | eta: 14.7m
step 04445/05000 (88.90%) | loss: 3.800086 | lrm: 0.21 | dt: 1644.17ms | tok/sec: 9,964 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 24 | total time: 117.04m | eta: 14.6m
step 04446/05000 (88.92%) | loss: 3.817144 | lrm: 0.21 | dt: 1562.94ms | tok/sec: 10,482 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 24 | total time: 117.07m | eta: 14.6m
step 04447/05000 (88.94%) | loss: 3.817890 | lrm: 0.21 | dt: 1557.42ms | tok/sec: 10,519 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 24 | total time: 117.09m | eta: 14.6m
step 04448/05000 (88.96%) | loss: 3.813515 | lrm: 0.21 | dt: 1561.85ms | tok/sec: 10,490 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 24 | total time: 117.12m | eta: 14.6m
step 04449/05000 (88.98%) | loss: 3.818038 | lrm: 0.21 | dt: 1562.33ms | tok/sec: 10,486 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 24 | total time: 117.15m | eta: 14.5m
step 04450/05000 (89.00%) | loss: 3.830316 | lrm: 0.21 | dt: 1550.41ms | tok/sec: 10,567 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 24 | total time: 117.17m | eta: 14.5m
step 04451/05000 (89.02%) | loss: 3.816619 | lrm: 0.21 | dt: 1495.05ms | tok/sec: 10,958 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 24 | total time: 117.20m | eta: 14.5m
step 04452/05000 (89.04%) | loss: 3.799684 | lrm: 0.21 | dt: 1635.99ms | tok/sec: 10,014 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 24 | total time: 117.22m | eta: 14.5m
step 04453/05000 (89.06%) | loss: 3.805215 | lrm: 0.21 | dt: 1559.74ms | tok/sec: 10,504 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 24 | total time: 117.25m | eta: 14.4m
step 04454/05000 (89.08%) | loss: 3.791134 | lrm: 0.21 | dt: 1625.83ms | tok/sec: 10,077 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 25 | total time: 117.28m | eta: 14.4m
step 04455/05000 (89.10%) | loss: 3.777807 | lrm: 0.21 | dt: 1618.57ms | tok/sec: 10,122 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 25 | total time: 117.30m | eta: 14.4m
step 04456/05000 (89.12%) | loss: 3.774711 | lrm: 0.21 | dt: 1636.56ms | tok/sec: 10,011 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 25 | total time: 117.33m | eta: 14.4m
step 04457/05000 (89.14%) | loss: 3.758676 | lrm: 0.21 | dt: 1581.80ms | tok/sec: 10,357 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 25 | total time: 117.36m | eta: 14.3m
step 04458/05000 (89.16%) | loss: 3.748840 | lrm: 0.21 | dt: 1638.99ms | tok/sec: 9,996 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 25 | total time: 117.39m | eta: 14.3m
step 04459/05000 (89.18%) | loss: 3.733167 | lrm: 0.21 | dt: 1571.04ms | tok/sec: 10,428 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 25 | total time: 117.41m | eta: 14.3m
step 04460/05000 (89.20%) | loss: 3.737583 | lrm: 0.21 | dt: 1570.65ms | tok/sec: 10,431 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 25 | total time: 117.44m | eta: 14.3m
step 04461/05000 (89.22%) | loss: 3.731894 | lrm: 0.21 | dt: 1581.02ms | tok/sec: 10,362 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 25 | total time: 117.46m | eta: 14.2m
step 04462/05000 (89.24%) | loss: 3.737833 | lrm: 0.21 | dt: 1587.40ms | tok/sec: 10,321 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 25 | total time: 117.49m | eta: 14.2m
step 04463/05000 (89.26%) | loss: 3.741246 | lrm: 0.21 | dt: 1548.61ms | tok/sec: 10,579 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 25 | total time: 117.52m | eta: 14.2m
step 04464/05000 (89.28%) | loss: 3.733097 | lrm: 0.21 | dt: 1555.87ms | tok/sec: 10,530 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 25 | total time: 117.54m | eta: 14.1m
step 04465/05000 (89.30%) | loss: 3.733707 | lrm: 0.21 | dt: 1609.72ms | tok/sec: 10,178 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 25 | total time: 117.57m | eta: 14.1m
step 04466/05000 (89.32%) | loss: 3.725004 | lrm: 0.21 | dt: 1604.08ms | tok/sec: 10,213 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 25 | total time: 117.60m | eta: 14.1m
step 04467/05000 (89.34%) | loss: 3.732798 | lrm: 0.21 | dt: 1550.78ms | tok/sec: 10,564 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 25 | total time: 117.62m | eta: 14.1m
step 04468/05000 (89.36%) | loss: 3.758001 | lrm: 0.21 | dt: 1563.90ms | tok/sec: 10,476 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 25 | total time: 117.65m | eta: 14.0m
step 04469/05000 (89.38%) | loss: 3.772752 | lrm: 0.21 | dt: 1541.34ms | tok/sec: 10,629 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 25 | total time: 117.67m | eta: 14.0m
step 04470/05000 (89.40%) | loss: 3.778387 | lrm: 0.20 | dt: 1544.28ms | tok/sec: 10,609 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 25 | total time: 117.70m | eta: 14.0m
step 04471/05000 (89.42%) | loss: 3.790867 | lrm: 0.20 | dt: 1588.74ms | tok/sec: 10,312 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 25 | total time: 117.73m | eta: 14.0m
step 04472/05000 (89.44%) | loss: 3.779620 | lrm: 0.20 | dt: 1510.97ms | tok/sec: 10,843 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 25 | total time: 117.75m | eta: 13.9m
step 04473/05000 (89.46%) | loss: 3.771543 | lrm: 0.20 | dt: 1554.62ms | tok/sec: 10,538 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 25 | total time: 117.78m | eta: 13.9m
step 04474/05000 (89.48%) | loss: 3.780384 | lrm: 0.20 | dt: 1619.72ms | tok/sec: 10,115 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 25 | total time: 117.80m | eta: 13.9m
step 04475/05000 (89.50%) | loss: 3.789947 | lrm: 0.20 | dt: 1555.03ms | tok/sec: 10,536 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 25 | total time: 117.83m | eta: 13.9m
step 04476/05000 (89.52%) | loss: 3.771293 | lrm: 0.20 | dt: 1575.82ms | tok/sec: 10,397 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 25 | total time: 117.86m | eta: 13.8m
step 04477/05000 (89.54%) | loss: 3.775951 | lrm: 0.20 | dt: 1568.46ms | tok/sec: 10,445 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 25 | total time: 117.88m | eta: 13.8m
step 04478/05000 (89.56%) | loss: 3.770256 | lrm: 0.20 | dt: 1548.92ms | tok/sec: 10,577 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 26 | total time: 117.91m | eta: 13.8m
step 04479/05000 (89.58%) | loss: 3.773467 | lrm: 0.20 | dt: 1545.17ms | tok/sec: 10,603 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 26 | total time: 117.93m | eta: 13.7m
step 04480/05000 (89.60%) | loss: 3.788613 | lrm: 0.20 | dt: 1557.00ms | tok/sec: 10,522 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 26 | total time: 117.96m | eta: 13.7m
step 04481/05000 (89.62%) | loss: 3.783880 | lrm: 0.20 | dt: 1542.76ms | tok/sec: 10,619 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 26 | total time: 117.99m | eta: 13.7m
step 04482/05000 (89.64%) | loss: 3.789952 | lrm: 0.20 | dt: 1547.41ms | tok/sec: 10,587 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 26 | total time: 118.01m | eta: 13.7m
step 04483/05000 (89.66%) | loss: 3.796021 | lrm: 0.20 | dt: 1544.15ms | tok/sec: 10,610 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 26 | total time: 118.04m | eta: 13.6m
step 04484/05000 (89.68%) | loss: 3.796840 | lrm: 0.20 | dt: 1565.27ms | tok/sec: 10,467 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 26 | total time: 118.06m | eta: 13.6m
step 04485/05000 (89.70%) | loss: 3.805707 | lrm: 0.20 | dt: 1540.05ms | tok/sec: 10,638 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 26 | total time: 118.09m | eta: 13.6m
step 04486/05000 (89.72%) | loss: 3.796021 | lrm: 0.20 | dt: 1538.66ms | tok/sec: 10,648 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 26 | total time: 118.11m | eta: 13.6m
step 04487/05000 (89.74%) | loss: 3.788019 | lrm: 0.20 | dt: 1578.37ms | tok/sec: 10,380 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 26 | total time: 118.14m | eta: 13.5m
step 04488/05000 (89.76%) | loss: 3.795174 | lrm: 0.20 | dt: 1530.47ms | tok/sec: 10,705 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 26 | total time: 118.17m | eta: 13.5m
step 04489/05000 (89.78%) | loss: 3.797446 | lrm: 0.20 | dt: 1551.47ms | tok/sec: 10,560 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 26 | total time: 118.19m | eta: 13.5m
step 04490/05000 (89.80%) | loss: 3.789337 | lrm: 0.20 | dt: 1571.22ms | tok/sec: 10,427 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 26 | total time: 118.22m | eta: 13.5m
step 04491/05000 (89.82%) | loss: 3.795088 | lrm: 0.20 | dt: 1557.57ms | tok/sec: 10,518 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 26 | total time: 118.24m | eta: 13.4m
step 04492/05000 (89.84%) | loss: 3.786210 | lrm: 0.20 | dt: 1513.53ms | tok/sec: 10,825 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 26 | total time: 118.27m | eta: 13.4m
step 04493/05000 (89.86%) | loss: 3.792389 | lrm: 0.20 | dt: 1536.24ms | tok/sec: 10,664 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 26 | total time: 118.29m | eta: 13.4m
step 04494/05000 (89.88%) | loss: 3.790812 | lrm: 0.20 | dt: 1557.76ms | tok/sec: 10,517 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 26 | total time: 118.32m | eta: 13.4m
step 04495/05000 (89.90%) | loss: 3.782467 | lrm: 0.20 | dt: 1537.84ms | tok/sec: 10,653 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 26 | total time: 118.35m | eta: 13.3m
step 04496/05000 (89.92%) | loss: 3.775769 | lrm: 0.20 | dt: 1529.09ms | tok/sec: 10,714 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 26 | total time: 118.37m | eta: 13.3m
step 04497/05000 (89.94%) | loss: 3.792787 | lrm: 0.20 | dt: 1561.77ms | tok/sec: 10,490 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 26 | total time: 118.40m | eta: 13.3m
step 04498/05000 (89.96%) | loss: 3.790247 | lrm: 0.20 | dt: 1555.99ms | tok/sec: 10,529 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 26 | total time: 118.42m | eta: 13.2m
step 04499/05000 (89.98%) | loss: 3.798534 | lrm: 0.20 | dt: 1549.39ms | tok/sec: 10,574 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 26 | total time: 118.45m | eta: 13.2m
Step 04500 | Validation bpb: 1.174323
<|bos|>The capital of France is the capital of France. The capital of France is the capital of France. The
<|bos|>The chemical symbol of gold is the symbol of the gold. It is a symbol of the gold's gold.
<|bos|>If yesterday was Friday, then tomorrow will be the next day of the week. The next day will be Friday, Monday,
<|bos|>The opposite of hot is the heat of the sun. The sun is the most important part of the body
<|bos|>The planets of the solar system are: the solar system, the solar system, the solar system, the solar system,
<|bos|>My favorite color is the red, orange, and yellow. I have a red, orange, and
<|bos|>If 5*x + 3 = 13, then x is 3.5*x + 3.5*x + 3
step 04500/05000 (90.00%) | loss: 3.779664 | lrm: 0.20 | dt: 1659.40ms | tok/sec: 9,873 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 26 | total time: 118.48m | eta: 13.2m
step 04501/05000 (90.02%) | loss: 3.774418 | lrm: 0.20 | dt: 1534.24ms | tok/sec: 10,678 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 26 | total time: 118.50m | eta: 13.2m
step 04502/05000 (90.04%) | loss: 3.780792 | lrm: 0.20 | dt: 1566.83ms | tok/sec: 10,456 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 27 | total time: 118.53m | eta: 13.1m
step 04503/05000 (90.06%) | loss: 3.790501 | lrm: 0.20 | dt: 1581.70ms | tok/sec: 10,358 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 27 | total time: 118.56m | eta: 13.1m
step 04504/05000 (90.08%) | loss: 3.787689 | lrm: 0.19 | dt: 1543.45ms | tok/sec: 10,615 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 27 | total time: 118.58m | eta: 13.1m
step 04505/05000 (90.10%) | loss: 3.785434 | lrm: 0.19 | dt: 1563.89ms | tok/sec: 10,476 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 27 | total time: 118.61m | eta: 13.1m
step 04506/05000 (90.12%) | loss: 3.786740 | lrm: 0.19 | dt: 1557.65ms | tok/sec: 10,518 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 27 | total time: 118.63m | eta: 13.0m
step 04507/05000 (90.14%) | loss: 3.788829 | lrm: 0.19 | dt: 1564.27ms | tok/sec: 10,473 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 27 | total time: 118.66m | eta: 13.0m
step 04508/05000 (90.16%) | loss: 3.800623 | lrm: 0.19 | dt: 1561.21ms | tok/sec: 10,494 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 27 | total time: 118.69m | eta: 13.0m
step 04509/05000 (90.18%) | loss: 3.802400 | lrm: 0.19 | dt: 1585.49ms | tok/sec: 10,333 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 27 | total time: 118.71m | eta: 13.0m
step 04510/05000 (90.20%) | loss: 3.804794 | lrm: 0.19 | dt: 1541.01ms | tok/sec: 10,631 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 27 | total time: 118.74m | eta: 12.9m
step 04511/05000 (90.22%) | loss: 3.809234 | lrm: 0.19 | dt: 1571.21ms | tok/sec: 10,427 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 27 | total time: 118.76m | eta: 12.9m
step 04512/05000 (90.24%) | loss: 3.802619 | lrm: 0.19 | dt: 1539.25ms | tok/sec: 10,644 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 27 | total time: 118.79m | eta: 12.9m
step 04513/05000 (90.26%) | loss: 3.800148 | lrm: 0.19 | dt: 1563.15ms | tok/sec: 10,481 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 27 | total time: 118.82m | eta: 12.8m
step 04514/05000 (90.28%) | loss: 3.809927 | lrm: 0.19 | dt: 1568.33ms | tok/sec: 10,446 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 27 | total time: 118.84m | eta: 12.8m
step 04515/05000 (90.30%) | loss: 3.814484 | lrm: 0.19 | dt: 1533.24ms | tok/sec: 10,685 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 27 | total time: 118.87m | eta: 12.8m
step 04516/05000 (90.32%) | loss: 3.808230 | lrm: 0.19 | dt: 1562.68ms | tok/sec: 10,484 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 27 | total time: 118.89m | eta: 12.8m
step 04517/05000 (90.34%) | loss: 3.808175 | lrm: 0.19 | dt: 1550.08ms | tok/sec: 10,569 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 27 | total time: 118.92m | eta: 12.7m
step 04518/05000 (90.36%) | loss: 3.814335 | lrm: 0.19 | dt: 1567.93ms | tok/sec: 10,449 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 27 | total time: 118.95m | eta: 12.7m
step 04519/05000 (90.38%) | loss: 3.801175 | lrm: 0.19 | dt: 1561.95ms | tok/sec: 10,489 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 27 | total time: 118.97m | eta: 12.7m
step 04520/05000 (90.40%) | loss: 3.802552 | lrm: 0.19 | dt: 1526.54ms | tok/sec: 10,732 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 27 | total time: 119.00m | eta: 12.7m
step 04521/05000 (90.42%) | loss: 3.810472 | lrm: 0.19 | dt: 1558.67ms | tok/sec: 10,511 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 27 | total time: 119.02m | eta: 12.6m
step 04522/05000 (90.44%) | loss: 3.797620 | lrm: 0.19 | dt: 1583.89ms | tok/sec: 10,344 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 27 | total time: 119.05m | eta: 12.6m
step 04523/05000 (90.46%) | loss: 3.790719 | lrm: 0.19 | dt: 1548.44ms | tok/sec: 10,580 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 27 | total time: 119.07m | eta: 12.6m
step 04524/05000 (90.48%) | loss: 3.792008 | lrm: 0.19 | dt: 1530.26ms | tok/sec: 10,706 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 27 | total time: 119.10m | eta: 12.6m
step 04525/05000 (90.50%) | loss: 3.776972 | lrm: 0.19 | dt: 1517.97ms | tok/sec: 10,793 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 28 | total time: 119.13m | eta: 12.5m
step 04526/05000 (90.52%) | loss: 3.778544 | lrm: 0.19 | dt: 1530.82ms | tok/sec: 10,702 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 28 | total time: 119.15m | eta: 12.5m
step 04527/05000 (90.54%) | loss: 3.796043 | lrm: 0.19 | dt: 1558.55ms | tok/sec: 10,512 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 28 | total time: 119.18m | eta: 12.5m
step 04528/05000 (90.56%) | loss: 3.799990 | lrm: 0.19 | dt: 1568.44ms | tok/sec: 10,446 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 28 | total time: 119.20m | eta: 12.5m
step 04529/05000 (90.58%) | loss: 3.817470 | lrm: 0.19 | dt: 1557.38ms | tok/sec: 10,520 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 28 | total time: 119.23m | eta: 12.4m
step 04530/05000 (90.60%) | loss: 3.826286 | lrm: 0.19 | dt: 1527.00ms | tok/sec: 10,729 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 28 | total time: 119.25m | eta: 12.4m
step 04531/05000 (90.62%) | loss: 3.813792 | lrm: 0.19 | dt: 1544.94ms | tok/sec: 10,604 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 28 | total time: 119.28m | eta: 12.4m
step 04532/05000 (90.64%) | loss: 3.806418 | lrm: 0.19 | dt: 1566.55ms | tok/sec: 10,458 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 28 | total time: 119.31m | eta: 12.3m
step 04533/05000 (90.66%) | loss: 3.813779 | lrm: 0.19 | dt: 1529.98ms | tok/sec: 10,708 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 28 | total time: 119.33m | eta: 12.3m
step 04534/05000 (90.68%) | loss: 3.816476 | lrm: 0.19 | dt: 1576.46ms | tok/sec: 10,392 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 28 | total time: 119.36m | eta: 12.3m
step 04535/05000 (90.70%) | loss: 3.806919 | lrm: 0.19 | dt: 1571.41ms | tok/sec: 10,426 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 28 | total time: 119.38m | eta: 12.3m
step 04536/05000 (90.72%) | loss: 3.813063 | lrm: 0.19 | dt: 1563.38ms | tok/sec: 10,479 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 28 | total time: 119.41m | eta: 12.2m
step 04537/05000 (90.74%) | loss: 3.812294 | lrm: 0.19 | dt: 1530.90ms | tok/sec: 10,702 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 28 | total time: 119.44m | eta: 12.2m
step 04538/05000 (90.76%) | loss: 3.802112 | lrm: 0.19 | dt: 1524.79ms | tok/sec: 10,745 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 28 | total time: 119.46m | eta: 12.2m
step 04539/05000 (90.78%) | loss: 3.809227 | lrm: 0.18 | dt: 1550.38ms | tok/sec: 10,567 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 28 | total time: 119.49m | eta: 12.2m
step 04540/05000 (90.80%) | loss: 3.804744 | lrm: 0.18 | dt: 1571.04ms | tok/sec: 10,428 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 28 | total time: 119.51m | eta: 12.1m
step 04541/05000 (90.82%) | loss: 3.804490 | lrm: 0.18 | dt: 1547.59ms | tok/sec: 10,586 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 28 | total time: 119.54m | eta: 12.1m
step 04542/05000 (90.84%) | loss: 3.807958 | lrm: 0.18 | dt: 1561.75ms | tok/sec: 10,490 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 28 | total time: 119.57m | eta: 12.1m
step 04543/05000 (90.86%) | loss: 3.807557 | lrm: 0.18 | dt: 1550.37ms | tok/sec: 10,567 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 28 | total time: 119.59m | eta: 12.1m
step 04544/05000 (90.88%) | loss: 3.789604 | lrm: 0.18 | dt: 1521.76ms | tok/sec: 10,766 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 28 | total time: 119.62m | eta: 12.0m
step 04545/05000 (90.90%) | loss: 3.782828 | lrm: 0.18 | dt: 1545.85ms | tok/sec: 10,598 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 28 | total time: 119.64m | eta: 12.0m
step 04546/05000 (90.92%) | loss: 3.767421 | lrm: 0.18 | dt: 1557.82ms | tok/sec: 10,517 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 28 | total time: 119.67m | eta: 12.0m
step 04547/05000 (90.94%) | loss: 3.778121 | lrm: 0.18 | dt: 1563.14ms | tok/sec: 10,481 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 28 | total time: 119.69m | eta: 12.0m
step 04548/05000 (90.96%) | loss: 3.776809 | lrm: 0.18 | dt: 1575.36ms | tok/sec: 10,400 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 28 | total time: 119.72m | eta: 11.9m
step 04549/05000 (90.98%) | loss: 3.769941 | lrm: 0.18 | dt: 1555.83ms | tok/sec: 10,530 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 29 | total time: 119.75m | eta: 11.9m
step 04550/05000 (91.00%) | loss: 3.780168 | lrm: 0.18 | dt: 1564.44ms | tok/sec: 10,472 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 29 | total time: 119.77m | eta: 11.9m
step 04551/05000 (91.02%) | loss: 3.774933 | lrm: 0.18 | dt: 1544.06ms | tok/sec: 10,611 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 29 | total time: 119.80m | eta: 11.8m
step 04552/05000 (91.04%) | loss: 3.771118 | lrm: 0.18 | dt: 1566.49ms | tok/sec: 10,459 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 29 | total time: 119.82m | eta: 11.8m
step 04553/05000 (91.06%) | loss: 3.778844 | lrm: 0.18 | dt: 1561.54ms | tok/sec: 10,492 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 29 | total time: 119.85m | eta: 11.8m
step 04554/05000 (91.08%) | loss: 3.779972 | lrm: 0.18 | dt: 1565.99ms | tok/sec: 10,462 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 29 | total time: 119.88m | eta: 11.8m
step 04555/05000 (91.10%) | loss: 3.774559 | lrm: 0.18 | dt: 1546.19ms | tok/sec: 10,596 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 29 | total time: 119.90m | eta: 11.7m
step 04556/05000 (91.12%) | loss: 3.797826 | lrm: 0.18 | dt: 1545.14ms | tok/sec: 10,603 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 29 | total time: 119.93m | eta: 11.7m
step 04557/05000 (91.14%) | loss: 3.788834 | lrm: 0.18 | dt: 1553.86ms | tok/sec: 10,544 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 29 | total time: 119.95m | eta: 11.7m
step 04558/05000 (91.16%) | loss: 3.779920 | lrm: 0.18 | dt: 1558.53ms | tok/sec: 10,512 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 29 | total time: 119.98m | eta: 11.7m
step 04559/05000 (91.18%) | loss: 3.767077 | lrm: 0.18 | dt: 1573.42ms | tok/sec: 10,412 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 29 | total time: 120.01m | eta: 11.6m
step 04560/05000 (91.20%) | loss: 3.764902 | lrm: 0.18 | dt: 1547.95ms | tok/sec: 10,584 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 29 | total time: 120.03m | eta: 11.6m
step 04561/05000 (91.22%) | loss: 3.766341 | lrm: 0.18 | dt: 1603.76ms | tok/sec: 10,215 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 29 | total time: 120.06m | eta: 11.6m
step 04562/05000 (91.24%) | loss: 3.758824 | lrm: 0.18 | dt: 1552.49ms | tok/sec: 10,553 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 29 | total time: 120.08m | eta: 11.6m
step 04563/05000 (91.26%) | loss: 3.751827 | lrm: 0.18 | dt: 1548.74ms | tok/sec: 10,578 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 29 | total time: 120.11m | eta: 11.5m
step 04564/05000 (91.28%) | loss: 3.756330 | lrm: 0.18 | dt: 1537.74ms | tok/sec: 10,654 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 29 | total time: 120.14m | eta: 11.5m
step 04565/05000 (91.30%) | loss: 3.767542 | lrm: 0.18 | dt: 1568.58ms | tok/sec: 10,445 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 29 | total time: 120.16m | eta: 11.5m
step 04566/05000 (91.32%) | loss: 3.777042 | lrm: 0.18 | dt: 1576.28ms | tok/sec: 10,394 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 29 | total time: 120.19m | eta: 11.4m
step 04567/05000 (91.34%) | loss: 3.781852 | lrm: 0.18 | dt: 1560.70ms | tok/sec: 10,497 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 29 | total time: 120.21m | eta: 11.4m
step 04568/05000 (91.36%) | loss: 3.773775 | lrm: 0.18 | dt: 1547.46ms | tok/sec: 10,587 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 29 | total time: 120.24m | eta: 11.4m
step 04569/05000 (91.38%) | loss: 3.764840 | lrm: 0.18 | dt: 1549.66ms | tok/sec: 10,572 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 29 | total time: 120.27m | eta: 11.4m
step 04570/05000 (91.40%) | loss: 3.751013 | lrm: 0.18 | dt: 1570.53ms | tok/sec: 10,432 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 29 | total time: 120.29m | eta: 11.3m
step 04571/05000 (91.42%) | loss: 3.759558 | lrm: 0.18 | dt: 1530.69ms | tok/sec: 10,703 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 29 | total time: 120.32m | eta: 11.3m
step 04572/05000 (91.44%) | loss: 3.751414 | lrm: 0.18 | dt: 1531.86ms | tok/sec: 10,695 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 29 | total time: 120.34m | eta: 11.3m
step 04573/05000 (91.46%) | loss: 3.777991 | lrm: 0.17 | dt: 1538.16ms | tok/sec: 10,651 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 30 | total time: 120.37m | eta: 11.3m
step 04574/05000 (91.48%) | loss: 3.773493 | lrm: 0.17 | dt: 1538.45ms | tok/sec: 10,649 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 30 | total time: 120.39m | eta: 11.2m
step 04575/05000 (91.50%) | loss: 3.791657 | lrm: 0.17 | dt: 1526.05ms | tok/sec: 10,736 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 30 | total time: 120.42m | eta: 11.2m
step 04576/05000 (91.52%) | loss: 3.802034 | lrm: 0.17 | dt: 1570.47ms | tok/sec: 10,432 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 30 | total time: 120.45m | eta: 11.2m
step 04577/05000 (91.54%) | loss: 3.807924 | lrm: 0.17 | dt: 1564.21ms | tok/sec: 10,474 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 30 | total time: 120.47m | eta: 11.2m
step 04578/05000 (91.56%) | loss: 3.809042 | lrm: 0.17 | dt: 1591.65ms | tok/sec: 10,293 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 30 | total time: 120.50m | eta: 11.1m
step 04579/05000 (91.58%) | loss: 3.814275 | lrm: 0.17 | dt: 1553.75ms | tok/sec: 10,544 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 30 | total time: 120.52m | eta: 11.1m
step 04580/05000 (91.60%) | loss: 3.825016 | lrm: 0.17 | dt: 1554.52ms | tok/sec: 10,539 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 30 | total time: 120.55m | eta: 11.1m
step 04581/05000 (91.62%) | loss: 3.820390 | lrm: 0.17 | dt: 1544.58ms | tok/sec: 10,607 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 30 | total time: 120.58m | eta: 11.1m
step 04582/05000 (91.64%) | loss: 3.823527 | lrm: 0.17 | dt: 1561.99ms | tok/sec: 10,489 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 30 | total time: 120.60m | eta: 11.0m
step 04583/05000 (91.66%) | loss: 3.834169 | lrm: 0.17 | dt: 1583.14ms | tok/sec: 10,349 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 30 | total time: 120.63m | eta: 11.0m
step 04584/05000 (91.68%) | loss: 3.820211 | lrm: 0.17 | dt: 1550.34ms | tok/sec: 10,567 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 30 | total time: 120.65m | eta: 11.0m
step 04585/05000 (91.70%) | loss: 3.817282 | lrm: 0.17 | dt: 1570.04ms | tok/sec: 10,435 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 30 | total time: 120.68m | eta: 10.9m
step 04586/05000 (91.72%) | loss: 3.809246 | lrm: 0.17 | dt: 1558.24ms | tok/sec: 10,514 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 30 | total time: 120.71m | eta: 10.9m
step 04587/05000 (91.74%) | loss: 3.817030 | lrm: 0.17 | dt: 1558.97ms | tok/sec: 10,509 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 30 | total time: 120.73m | eta: 10.9m
step 04588/05000 (91.76%) | loss: 3.823982 | lrm: 0.17 | dt: 1558.91ms | tok/sec: 10,509 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 30 | total time: 120.76m | eta: 10.9m
step 04589/05000 (91.78%) | loss: 3.828890 | lrm: 0.17 | dt: 1550.23ms | tok/sec: 10,568 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 30 | total time: 120.78m | eta: 10.8m
step 04590/05000 (91.80%) | loss: 3.817381 | lrm: 0.17 | dt: 1563.82ms | tok/sec: 10,476 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 30 | total time: 120.81m | eta: 10.8m
step 04591/05000 (91.82%) | loss: 3.815240 | lrm: 0.17 | dt: 1520.53ms | tok/sec: 10,775 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 30 | total time: 120.84m | eta: 10.8m
step 04592/05000 (91.84%) | loss: 3.819901 | lrm: 0.17 | dt: 1538.27ms | tok/sec: 10,650 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 30 | total time: 120.86m | eta: 10.8m
step 04593/05000 (91.86%) | loss: 3.820031 | lrm: 0.17 | dt: 1563.93ms | tok/sec: 10,476 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 30 | total time: 120.89m | eta: 10.7m
step 04594/05000 (91.88%) | loss: 3.802892 | lrm: 0.17 | dt: 1551.67ms | tok/sec: 10,558 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 30 | total time: 120.91m | eta: 10.7m
step 04595/05000 (91.90%) | loss: 3.799358 | lrm: 0.17 | dt: 1553.51ms | tok/sec: 10,546 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 30 | total time: 120.94m | eta: 10.7m
step 04596/05000 (91.92%) | loss: 3.795494 | lrm: 0.17 | dt: 1558.62ms | tok/sec: 10,511 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 30 | total time: 120.97m | eta: 10.7m
step 04597/05000 (91.94%) | loss: 3.798116 | lrm: 0.17 | dt: 1539.57ms | tok/sec: 10,641 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 31 | total time: 120.99m | eta: 10.6m
step 04598/05000 (91.96%) | loss: 3.794455 | lrm: 0.17 | dt: 1515.40ms | tok/sec: 10,811 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 31 | total time: 121.02m | eta: 10.6m
step 04599/05000 (91.98%) | loss: 3.787548 | lrm: 0.17 | dt: 1561.09ms | tok/sec: 10,495 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 31 | total time: 121.04m | eta: 10.6m
Step 04600 | Validation bpb: 1.172274
<|bos|>The capital of France is the capital of France. The capital of France is the capital of France. The
<|bos|>The chemical symbol of gold is a symbol of the gold's gold. It is a symbol of the gold's
<|bos|>If yesterday was Friday, then tomorrow will be the next day of the week. The next day will be Friday, then Friday
<|bos|>The opposite of hot is the same as the hot, but the same as the hot, but the same
<|bos|>The planets of the solar system are: the solar system, the solar system, the solar system, the solar system,
<|bos|>My favorite color is the red, but I'm not sure what color you're looking for. I
<|bos|>If 5*x + 3 = 13, then x is the number of points that are given by the number of points that are given by
step 04600/05000 (92.00%) | loss: 3.810302 | lrm: 0.17 | dt: 1601.57ms | tok/sec: 10,229 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 31 | total time: 121.07m | eta: 10.6m
step 04601/05000 (92.02%) | loss: 3.815012 | lrm: 0.17 | dt: 1560.87ms | tok/sec: 10,496 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 31 | total time: 121.09m | eta: 10.5m
step 04602/05000 (92.04%) | loss: 3.822878 | lrm: 0.17 | dt: 1517.55ms | tok/sec: 10,796 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 31 | total time: 121.12m | eta: 10.5m
step 04603/05000 (92.06%) | loss: 3.812904 | lrm: 0.17 | dt: 1545.28ms | tok/sec: 10,602 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 31 | total time: 121.15m | eta: 10.5m
step 04604/05000 (92.08%) | loss: 3.817746 | lrm: 0.17 | dt: 1551.74ms | tok/sec: 10,558 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 31 | total time: 121.17m | eta: 10.4m
step 04605/05000 (92.10%) | loss: 3.800957 | lrm: 0.17 | dt: 1556.80ms | tok/sec: 10,524 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 31 | total time: 121.20m | eta: 10.4m
step 04606/05000 (92.12%) | loss: 3.805062 | lrm: 0.17 | dt: 1544.64ms | tok/sec: 10,607 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 31 | total time: 121.22m | eta: 10.4m
step 04607/05000 (92.14%) | loss: 3.806583 | lrm: 0.16 | dt: 1561.52ms | tok/sec: 10,492 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 31 | total time: 121.25m | eta: 10.4m
step 04608/05000 (92.16%) | loss: 3.806033 | lrm: 0.16 | dt: 1559.12ms | tok/sec: 10,508 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 31 | total time: 121.28m | eta: 10.3m
step 04609/05000 (92.18%) | loss: 3.796273 | lrm: 0.16 | dt: 1566.24ms | tok/sec: 10,460 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 31 | total time: 121.30m | eta: 10.3m
step 04610/05000 (92.20%) | loss: 3.793841 | lrm: 0.16 | dt: 1574.64ms | tok/sec: 10,404 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 31 | total time: 121.33m | eta: 10.3m
step 04611/05000 (92.22%) | loss: 3.801344 | lrm: 0.16 | dt: 1545.59ms | tok/sec: 10,600 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 31 | total time: 121.35m | eta: 10.3m
step 04612/05000 (92.24%) | loss: 3.795095 | lrm: 0.16 | dt: 1575.92ms | tok/sec: 10,396 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 31 | total time: 121.38m | eta: 10.2m
step 04613/05000 (92.26%) | loss: 3.796194 | lrm: 0.16 | dt: 1555.17ms | tok/sec: 10,535 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 31 | total time: 121.41m | eta: 10.2m
step 04614/05000 (92.28%) | loss: 3.790819 | lrm: 0.16 | dt: 1537.16ms | tok/sec: 10,658 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 31 | total time: 121.43m | eta: 10.2m
step 04615/05000 (92.30%) | loss: 3.793405 | lrm: 0.16 | dt: 1578.10ms | tok/sec: 10,382 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 31 | total time: 121.46m | eta: 10.2m
step 04616/05000 (92.32%) | loss: 3.797759 | lrm: 0.16 | dt: 1545.35ms | tok/sec: 10,602 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 31 | total time: 121.48m | eta: 10.1m
step 04617/05000 (92.34%) | loss: 3.791174 | lrm: 0.16 | dt: 1562.15ms | tok/sec: 10,488 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 31 | total time: 121.51m | eta: 10.1m
step 04618/05000 (92.36%) | loss: 3.795531 | lrm: 0.16 | dt: 1539.35ms | tok/sec: 10,643 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 31 | total time: 121.54m | eta: 10.1m
step 04619/05000 (92.38%) | loss: 3.793676 | lrm: 0.16 | dt: 1568.16ms | tok/sec: 10,447 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 31 | total time: 121.56m | eta: 10.0m
step 04620/05000 (92.40%) | loss: 3.787648 | lrm: 0.16 | dt: 1554.05ms | tok/sec: 10,542 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 32 | total time: 121.59m | eta: 10.0m
step 04621/05000 (92.42%) | loss: 3.779341 | lrm: 0.16 | dt: 1571.69ms | tok/sec: 10,424 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 32 | total time: 121.61m | eta: 10.0m
step 04622/05000 (92.44%) | loss: 3.794823 | lrm: 0.16 | dt: 1532.47ms | tok/sec: 10,691 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 32 | total time: 121.64m | eta: 10.0m
step 04623/05000 (92.46%) | loss: 3.801929 | lrm: 0.16 | dt: 1528.17ms | tok/sec: 10,721 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 32 | total time: 121.66m | eta: 9.9m
step 04624/05000 (92.48%) | loss: 3.786519 | lrm: 0.16 | dt: 1516.19ms | tok/sec: 10,806 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 32 | total time: 121.69m | eta: 9.9m
step 04625/05000 (92.50%) | loss: 3.798506 | lrm: 0.16 | dt: 1543.49ms | tok/sec: 10,614 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 32 | total time: 121.72m | eta: 9.9m
step 04626/05000 (92.52%) | loss: 3.806213 | lrm: 0.16 | dt: 1535.33ms | tok/sec: 10,671 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 32 | total time: 121.74m | eta: 9.9m
step 04627/05000 (92.54%) | loss: 3.791130 | lrm: 0.16 | dt: 1550.25ms | tok/sec: 10,568 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 32 | total time: 121.77m | eta: 9.8m
step 04628/05000 (92.56%) | loss: 3.799412 | lrm: 0.16 | dt: 1579.42ms | tok/sec: 10,373 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 32 | total time: 121.79m | eta: 9.8m
step 04629/05000 (92.58%) | loss: 3.809471 | lrm: 0.16 | dt: 1622.59ms | tok/sec: 10,097 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 32 | total time: 121.82m | eta: 9.8m
step 04630/05000 (92.60%) | loss: 3.815737 | lrm: 0.16 | dt: 1525.88ms | tok/sec: 10,737 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 32 | total time: 121.85m | eta: 9.8m
step 04631/05000 (92.62%) | loss: 3.803505 | lrm: 0.16 | dt: 1616.11ms | tok/sec: 10,137 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 32 | total time: 121.87m | eta: 9.7m
step 04632/05000 (92.64%) | loss: 3.790522 | lrm: 0.16 | dt: 1631.98ms | tok/sec: 10,039 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 32 | total time: 121.90m | eta: 9.7m
step 04633/05000 (92.66%) | loss: 3.800789 | lrm: 0.16 | dt: 1629.54ms | tok/sec: 10,054 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 32 | total time: 121.93m | eta: 9.7m
step 04634/05000 (92.68%) | loss: 3.781043 | lrm: 0.16 | dt: 1565.91ms | tok/sec: 10,462 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 32 | total time: 121.95m | eta: 9.7m
step 04635/05000 (92.70%) | loss: 3.788211 | lrm: 0.16 | dt: 1622.88ms | tok/sec: 10,095 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 32 | total time: 121.98m | eta: 9.6m
step 04636/05000 (92.72%) | loss: 3.799940 | lrm: 0.16 | dt: 1605.04ms | tok/sec: 10,207 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 32 | total time: 122.01m | eta: 9.6m
step 04637/05000 (92.74%) | loss: 3.782884 | lrm: 0.16 | dt: 1681.15ms | tok/sec: 9,745 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 32 | total time: 122.03m | eta: 9.6m
step 04638/05000 (92.76%) | loss: 3.793224 | lrm: 0.16 | dt: 1604.39ms | tok/sec: 10,211 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 32 | total time: 122.06m | eta: 9.5m
step 04639/05000 (92.78%) | loss: 3.784764 | lrm: 0.16 | dt: 1596.69ms | tok/sec: 10,261 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 32 | total time: 122.09m | eta: 9.5m
step 04640/05000 (92.80%) | loss: 3.784444 | lrm: 0.16 | dt: 1586.59ms | tok/sec: 10,326 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 32 | total time: 122.11m | eta: 9.5m
step 04641/05000 (92.82%) | loss: 3.794893 | lrm: 0.15 | dt: 1600.02ms | tok/sec: 10,239 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 32 | total time: 122.14m | eta: 9.5m
step 04642/05000 (92.84%) | loss: 3.772631 | lrm: 0.15 | dt: 1610.82ms | tok/sec: 10,171 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 32 | total time: 122.17m | eta: 9.4m
step 04643/05000 (92.86%) | loss: 3.784438 | lrm: 0.15 | dt: 1536.06ms | tok/sec: 10,666 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 33 | total time: 122.19m | eta: 9.4m
step 04644/05000 (92.88%) | loss: 3.785943 | lrm: 0.15 | dt: 1542.84ms | tok/sec: 10,619 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 33 | total time: 122.22m | eta: 9.4m
step 04645/05000 (92.90%) | loss: 3.796212 | lrm: 0.15 | dt: 1541.41ms | tok/sec: 10,629 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 33 | total time: 122.25m | eta: 9.4m
step 04646/05000 (92.92%) | loss: 3.811013 | lrm: 0.15 | dt: 1572.18ms | tok/sec: 10,421 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 33 | total time: 122.27m | eta: 9.3m
step 04647/05000 (92.94%) | loss: 3.802892 | lrm: 0.15 | dt: 1604.20ms | tok/sec: 10,213 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 33 | total time: 122.30m | eta: 9.3m
step 04648/05000 (92.96%) | loss: 3.801420 | lrm: 0.15 | dt: 1627.17ms | tok/sec: 10,069 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 33 | total time: 122.33m | eta: 9.3m
step 04649/05000 (92.98%) | loss: 3.817208 | lrm: 0.15 | dt: 1570.19ms | tok/sec: 10,434 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 33 | total time: 122.35m | eta: 9.3m
step 04650/05000 (93.00%) | loss: 3.814858 | lrm: 0.15 | dt: 1520.35ms | tok/sec: 10,776 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 33 | total time: 122.38m | eta: 9.2m
step 04651/05000 (93.02%) | loss: 3.800834 | lrm: 0.15 | dt: 1562.18ms | tok/sec: 10,487 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 33 | total time: 122.40m | eta: 9.2m
step 04652/05000 (93.04%) | loss: 3.788664 | lrm: 0.15 | dt: 1562.67ms | tok/sec: 10,484 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 33 | total time: 122.43m | eta: 9.2m
step 04653/05000 (93.06%) | loss: 3.794633 | lrm: 0.15 | dt: 1572.42ms | tok/sec: 10,419 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 33 | total time: 122.46m | eta: 9.2m
step 04654/05000 (93.08%) | loss: 3.792070 | lrm: 0.15 | dt: 1458.09ms | tok/sec: 11,236 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 33 | total time: 122.48m | eta: 9.1m
step 04655/05000 (93.10%) | loss: 3.777973 | lrm: 0.15 | dt: 1780.93ms | tok/sec: 9,199 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 33 | total time: 122.51m | eta: 9.1m
step 04656/05000 (93.12%) | loss: 3.787095 | lrm: 0.15 | dt: 1523.31ms | tok/sec: 10,755 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 33 | total time: 122.53m | eta: 9.1m
step 04657/05000 (93.14%) | loss: 3.798943 | lrm: 0.15 | dt: 1481.12ms | tok/sec: 11,061 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 33 | total time: 122.56m | eta: 9.0m
step 04658/05000 (93.16%) | loss: 3.782995 | lrm: 0.15 | dt: 1513.06ms | tok/sec: 10,828 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 33 | total time: 122.58m | eta: 9.0m
step 04659/05000 (93.18%) | loss: 3.781584 | lrm: 0.15 | dt: 1478.34ms | tok/sec: 11,082 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 33 | total time: 122.61m | eta: 9.0m
step 04660/05000 (93.20%) | loss: 3.789766 | lrm: 0.15 | dt: 1543.96ms | tok/sec: 10,611 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 33 | total time: 122.63m | eta: 9.0m
step 04661/05000 (93.22%) | loss: 3.785001 | lrm: 0.15 | dt: 1645.50ms | tok/sec: 9,956 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 33 | total time: 122.66m | eta: 8.9m
step 04662/05000 (93.24%) | loss: 3.794780 | lrm: 0.15 | dt: 1639.03ms | tok/sec: 9,996 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 33 | total time: 122.69m | eta: 8.9m
step 04663/05000 (93.26%) | loss: 3.782028 | lrm: 0.15 | dt: 1492.84ms | tok/sec: 10,975 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 33 | total time: 122.71m | eta: 8.9m
step 04664/05000 (93.28%) | loss: 3.774236 | lrm: 0.15 | dt: 1469.71ms | tok/sec: 11,147 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 33 | total time: 122.74m | eta: 8.9m
step 04665/05000 (93.30%) | loss: 3.775141 | lrm: 0.15 | dt: 1541.74ms | tok/sec: 10,626 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 33 | total time: 122.76m | eta: 8.8m
step 04666/05000 (93.32%) | loss: 3.759848 | lrm: 0.15 | dt: 1675.32ms | tok/sec: 9,779 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 34 | total time: 122.79m | eta: 8.8m
step 04667/05000 (93.34%) | loss: 3.761173 | lrm: 0.15 | dt: 1576.04ms | tok/sec: 10,395 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 34 | total time: 122.82m | eta: 8.8m
step 04668/05000 (93.36%) | loss: 3.762310 | lrm: 0.15 | dt: 1653.57ms | tok/sec: 9,908 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 34 | total time: 122.85m | eta: 8.8m
step 04669/05000 (93.38%) | loss: 3.772048 | lrm: 0.15 | dt: 1696.61ms | tok/sec: 9,656 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 34 | total time: 122.87m | eta: 8.7m
step 04670/05000 (93.40%) | loss: 3.786647 | lrm: 0.15 | dt: 1541.91ms | tok/sec: 10,625 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 34 | total time: 122.90m | eta: 8.7m
step 04671/05000 (93.42%) | loss: 3.794810 | lrm: 0.15 | dt: 1547.40ms | tok/sec: 10,588 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 34 | total time: 122.93m | eta: 8.7m
step 04672/05000 (93.44%) | loss: 3.807459 | lrm: 0.15 | dt: 1517.99ms | tok/sec: 10,793 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 34 | total time: 122.95m | eta: 8.7m
step 04673/05000 (93.46%) | loss: 3.800882 | lrm: 0.15 | dt: 1570.81ms | tok/sec: 10,430 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 34 | total time: 122.98m | eta: 8.6m
step 04674/05000 (93.48%) | loss: 3.795037 | lrm: 0.15 | dt: 1495.87ms | tok/sec: 10,952 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 34 | total time: 123.00m | eta: 8.6m
step 04675/05000 (93.50%) | loss: 3.789292 | lrm: 0.15 | dt: 1581.16ms | tok/sec: 10,362 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 34 | total time: 123.03m | eta: 8.6m
step 04676/05000 (93.52%) | loss: 3.792086 | lrm: 0.14 | dt: 1486.96ms | tok/sec: 11,018 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 34 | total time: 123.05m | eta: 8.5m
step 04677/05000 (93.54%) | loss: 3.810027 | lrm: 0.14 | dt: 1513.34ms | tok/sec: 10,826 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 34 | total time: 123.08m | eta: 8.5m
step 04678/05000 (93.56%) | loss: 3.789311 | lrm: 0.14 | dt: 1525.49ms | tok/sec: 10,740 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 34 | total time: 123.10m | eta: 8.5m
step 04679/05000 (93.58%) | loss: 3.791557 | lrm: 0.14 | dt: 1728.22ms | tok/sec: 9,480 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 34 | total time: 123.13m | eta: 8.5m
step 04680/05000 (93.60%) | loss: 3.789536 | lrm: 0.14 | dt: 1760.85ms | tok/sec: 9,304 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 34 | total time: 123.16m | eta: 8.4m
step 04681/05000 (93.62%) | loss: 3.773978 | lrm: 0.14 | dt: 1561.67ms | tok/sec: 10,491 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 34 | total time: 123.19m | eta: 8.4m
step 04682/05000 (93.64%) | loss: 3.770316 | lrm: 0.14 | dt: 1482.17ms | tok/sec: 11,054 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 34 | total time: 123.21m | eta: 8.4m
step 04683/05000 (93.66%) | loss: 3.761934 | lrm: 0.14 | dt: 1626.99ms | tok/sec: 10,070 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 34 | total time: 123.24m | eta: 8.4m
step 04684/05000 (93.68%) | loss: 3.766190 | lrm: 0.14 | dt: 1508.13ms | tok/sec: 10,863 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 34 | total time: 123.27m | eta: 8.3m
step 04685/05000 (93.70%) | loss: 3.761085 | lrm: 0.14 | dt: 1588.65ms | tok/sec: 10,313 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 34 | total time: 123.29m | eta: 8.3m
step 04686/05000 (93.72%) | loss: 3.765000 | lrm: 0.14 | dt: 1554.37ms | tok/sec: 10,540 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 34 | total time: 123.32m | eta: 8.3m
step 04687/05000 (93.74%) | loss: 3.753884 | lrm: 0.14 | dt: 1463.87ms | tok/sec: 11,192 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 34 | total time: 123.34m | eta: 8.3m
step 04688/05000 (93.76%) | loss: 3.771336 | lrm: 0.14 | dt: 1494.87ms | tok/sec: 10,960 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 34 | total time: 123.37m | eta: 8.2m
step 04689/05000 (93.78%) | loss: 3.771977 | lrm: 0.14 | dt: 1467.73ms | tok/sec: 11,162 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 35 | total time: 123.39m | eta: 8.2m
step 04690/05000 (93.80%) | loss: 3.774588 | lrm: 0.14 | dt: 1530.33ms | tok/sec: 10,706 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 35 | total time: 123.42m | eta: 8.2m
step 04691/05000 (93.82%) | loss: 3.786150 | lrm: 0.14 | dt: 1577.34ms | tok/sec: 10,387 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 35 | total time: 123.44m | eta: 8.1m
step 04692/05000 (93.84%) | loss: 3.787859 | lrm: 0.14 | dt: 1558.40ms | tok/sec: 10,513 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 35 | total time: 123.47m | eta: 8.1m
step 04693/05000 (93.86%) | loss: 3.784397 | lrm: 0.14 | dt: 1460.62ms | tok/sec: 11,217 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 35 | total time: 123.49m | eta: 8.1m
step 04694/05000 (93.88%) | loss: 3.777778 | lrm: 0.14 | dt: 1479.40ms | tok/sec: 11,074 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 35 | total time: 123.52m | eta: 8.1m
step 04695/05000 (93.90%) | loss: 3.768278 | lrm: 0.14 | dt: 1428.67ms | tok/sec: 11,468 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 35 | total time: 123.54m | eta: 8.0m
step 04696/05000 (93.92%) | loss: 3.768663 | lrm: 0.14 | dt: 1562.92ms | tok/sec: 10,482 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 35 | total time: 123.57m | eta: 8.0m
step 04697/05000 (93.94%) | loss: 3.784888 | lrm: 0.14 | dt: 1491.21ms | tok/sec: 10,987 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 35 | total time: 123.59m | eta: 8.0m
step 04698/05000 (93.96%) | loss: 3.779859 | lrm: 0.14 | dt: 1495.44ms | tok/sec: 10,955 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 35 | total time: 123.62m | eta: 8.0m
step 04699/05000 (93.98%) | loss: 3.777817 | lrm: 0.14 | dt: 1511.80ms | tok/sec: 10,837 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 35 | total time: 123.64m | eta: 7.9m
Step 04700 | Validation bpb: 1.170340
<|bos|>The capital of France is the capital of France. The capital of France is the capital of France. The
<|bos|>The chemical symbol of gold is the gold, which is the gold that is used to make gold. It is
<|bos|>If yesterday was Friday, then tomorrow will be the next day. The next day will be Friday, and the next day will
<|bos|>The opposite of hot is the same as the cold. The heat is the same as the hot, but
<|bos|>The planets of the solar system are: the solar system, the solar system, the solar system, the solar system,
<|bos|>My favorite color is the red, but I don't know what I'm doing. I'm not
<|bos|>If 5*x + 3 = 13, then x is the number of points that are in the same direction. The number of points that
step 04700/05000 (94.00%) | loss: 3.785789 | lrm: 0.14 | dt: 1552.73ms | tok/sec: 10,551 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 35 | total time: 123.67m | eta: 7.9m
step 04701/05000 (94.02%) | loss: 3.795769 | lrm: 0.14 | dt: 1535.57ms | tok/sec: 10,669 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 35 | total time: 123.69m | eta: 7.9m
step 04702/05000 (94.04%) | loss: 3.801085 | lrm: 0.14 | dt: 1491.32ms | tok/sec: 10,986 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 35 | total time: 123.72m | eta: 7.9m
step 04703/05000 (94.06%) | loss: 3.811888 | lrm: 0.14 | dt: 1465.42ms | tok/sec: 11,180 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 35 | total time: 123.74m | eta: 7.8m
step 04704/05000 (94.08%) | loss: 3.793359 | lrm: 0.14 | dt: 1470.92ms | tok/sec: 11,138 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 35 | total time: 123.77m | eta: 7.8m
step 04705/05000 (94.10%) | loss: 3.801859 | lrm: 0.14 | dt: 1516.15ms | tok/sec: 10,806 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 35 | total time: 123.79m | eta: 7.8m
step 04706/05000 (94.12%) | loss: 3.786259 | lrm: 0.14 | dt: 1445.05ms | tok/sec: 11,338 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 35 | total time: 123.82m | eta: 7.8m
step 04707/05000 (94.14%) | loss: 3.776078 | lrm: 0.14 | dt: 1541.19ms | tok/sec: 10,630 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 35 | total time: 123.84m | eta: 7.7m
step 04708/05000 (94.16%) | loss: 3.767489 | lrm: 0.14 | dt: 1459.52ms | tok/sec: 11,225 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 35 | total time: 123.87m | eta: 7.7m
step 04709/05000 (94.18%) | loss: 3.774156 | lrm: 0.14 | dt: 1548.22ms | tok/sec: 10,582 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 35 | total time: 123.89m | eta: 7.7m
step 04710/05000 (94.20%) | loss: 3.777535 | lrm: 0.13 | dt: 1498.80ms | tok/sec: 10,931 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 35 | total time: 123.92m | eta: 7.6m
step 04711/05000 (94.22%) | loss: 3.760966 | lrm: 0.13 | dt: 1463.45ms | tok/sec: 11,195 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 35 | total time: 123.94m | eta: 7.6m
step 04712/05000 (94.24%) | loss: 3.764296 | lrm: 0.13 | dt: 1460.40ms | tok/sec: 11,218 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 35 | total time: 123.97m | eta: 7.6m
step 04713/05000 (94.26%) | loss: 3.755710 | lrm: 0.13 | dt: 1500.51ms | tok/sec: 10,918 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 36 | total time: 123.99m | eta: 7.6m
step 04714/05000 (94.28%) | loss: 3.758269 | lrm: 0.13 | dt: 1447.75ms | tok/sec: 11,316 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 36 | total time: 124.02m | eta: 7.5m
step 04715/05000 (94.30%) | loss: 3.755714 | lrm: 0.13 | dt: 1514.12ms | tok/sec: 10,820 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 36 | total time: 124.04m | eta: 7.5m
step 04716/05000 (94.32%) | loss: 3.746322 | lrm: 0.13 | dt: 1434.26ms | tok/sec: 11,423 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 36 | total time: 124.07m | eta: 7.5m
step 04717/05000 (94.34%) | loss: 3.762221 | lrm: 0.13 | dt: 1497.69ms | tok/sec: 10,939 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 36 | total time: 124.09m | eta: 7.5m
step 04718/05000 (94.36%) | loss: 3.770727 | lrm: 0.13 | dt: 1465.84ms | tok/sec: 11,177 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 36 | total time: 124.11m | eta: 7.4m
step 04719/05000 (94.38%) | loss: 3.783808 | lrm: 0.13 | dt: 1469.68ms | tok/sec: 11,148 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 36 | total time: 124.14m | eta: 7.4m
step 04720/05000 (94.40%) | loss: 3.782869 | lrm: 0.13 | dt: 1430.21ms | tok/sec: 11,455 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 36 | total time: 124.16m | eta: 7.4m
step 04721/05000 (94.42%) | loss: 3.788152 | lrm: 0.13 | dt: 1439.74ms | tok/sec: 11,379 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 36 | total time: 124.19m | eta: 7.4m
step 04722/05000 (94.44%) | loss: 3.788168 | lrm: 0.13 | dt: 1505.53ms | tok/sec: 10,882 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 36 | total time: 124.21m | eta: 7.3m
step 04723/05000 (94.46%) | loss: 3.777081 | lrm: 0.13 | dt: 1458.80ms | tok/sec: 11,231 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 36 | total time: 124.24m | eta: 7.3m
step 04724/05000 (94.48%) | loss: 3.771212 | lrm: 0.13 | dt: 1447.99ms | tok/sec: 11,314 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 36 | total time: 124.26m | eta: 7.3m
step 04725/05000 (94.50%) | loss: 3.771533 | lrm: 0.13 | dt: 1500.82ms | tok/sec: 10,916 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 36 | total time: 124.29m | eta: 7.2m
step 04726/05000 (94.52%) | loss: 3.796303 | lrm: 0.13 | dt: 1498.88ms | tok/sec: 10,930 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 36 | total time: 124.31m | eta: 7.2m
step 04727/05000 (94.54%) | loss: 3.792619 | lrm: 0.13 | dt: 1556.98ms | tok/sec: 10,522 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 36 | total time: 124.34m | eta: 7.2m
step 04728/05000 (94.56%) | loss: 3.759946 | lrm: 0.13 | dt: 1534.70ms | tok/sec: 10,675 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 36 | total time: 124.36m | eta: 7.2m
step 04729/05000 (94.58%) | loss: 3.748891 | lrm: 0.13 | dt: 1544.26ms | tok/sec: 10,609 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 36 | total time: 124.39m | eta: 7.1m
step 04730/05000 (94.60%) | loss: 3.760082 | lrm: 0.13 | dt: 1562.80ms | tok/sec: 10,483 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 36 | total time: 124.41m | eta: 7.1m
step 04731/05000 (94.62%) | loss: 3.760599 | lrm: 0.13 | dt: 1563.14ms | tok/sec: 10,481 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 36 | total time: 124.44m | eta: 7.1m
step 04732/05000 (94.64%) | loss: 3.766844 | lrm: 0.13 | dt: 1555.40ms | tok/sec: 10,533 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 36 | total time: 124.47m | eta: 7.1m
step 04733/05000 (94.66%) | loss: 3.770040 | lrm: 0.13 | dt: 1636.78ms | tok/sec: 10,009 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 36 | total time: 124.49m | eta: 7.0m
step 04734/05000 (94.68%) | loss: 3.787729 | lrm: 0.13 | dt: 1644.98ms | tok/sec: 9,960 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 36 | total time: 124.52m | eta: 7.0m
step 04735/05000 (94.70%) | loss: 3.765524 | lrm: 0.13 | dt: 1622.24ms | tok/sec: 10,099 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 36 | total time: 124.55m | eta: 7.0m
step 04736/05000 (94.72%) | loss: 3.765874 | lrm: 0.13 | dt: 1641.05ms | tok/sec: 9,983 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 36 | total time: 124.58m | eta: 7.0m
step 04737/05000 (94.74%) | loss: 3.758978 | lrm: 0.13 | dt: 1636.95ms | tok/sec: 10,008 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 37 | total time: 124.60m | eta: 6.9m
step 04738/05000 (94.76%) | loss: 3.757174 | lrm: 0.13 | dt: 1681.73ms | tok/sec: 9,742 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 37 | total time: 124.63m | eta: 6.9m
step 04739/05000 (94.78%) | loss: 3.772147 | lrm: 0.13 | dt: 1607.35ms | tok/sec: 10,193 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 37 | total time: 124.66m | eta: 6.9m
step 04740/05000 (94.80%) | loss: 3.766210 | lrm: 0.13 | dt: 1580.17ms | tok/sec: 10,368 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 37 | total time: 124.68m | eta: 6.9m
step 04741/05000 (94.82%) | loss: 3.758454 | lrm: 0.13 | dt: 1606.43ms | tok/sec: 10,198 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 37 | total time: 124.71m | eta: 6.8m
step 04742/05000 (94.84%) | loss: 3.755105 | lrm: 0.13 | dt: 1541.70ms | tok/sec: 10,627 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 37 | total time: 124.74m | eta: 6.8m
step 04743/05000 (94.86%) | loss: 3.755962 | lrm: 0.13 | dt: 1557.74ms | tok/sec: 10,517 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 37 | total time: 124.76m | eta: 6.8m
step 04744/05000 (94.88%) | loss: 3.769410 | lrm: 0.12 | dt: 1615.48ms | tok/sec: 10,141 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 37 | total time: 124.79m | eta: 6.7m
step 04745/05000 (94.90%) | loss: 3.765502 | lrm: 0.12 | dt: 1570.82ms | tok/sec: 10,430 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 37 | total time: 124.82m | eta: 6.7m
step 04746/05000 (94.92%) | loss: 3.777254 | lrm: 0.12 | dt: 1577.19ms | tok/sec: 10,388 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 37 | total time: 124.84m | eta: 6.7m
step 04747/05000 (94.94%) | loss: 3.773610 | lrm: 0.12 | dt: 1565.35ms | tok/sec: 10,466 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 37 | total time: 124.87m | eta: 6.7m
step 04748/05000 (94.96%) | loss: 3.767202 | lrm: 0.12 | dt: 1654.09ms | tok/sec: 9,905 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 37 | total time: 124.90m | eta: 6.6m
step 04749/05000 (94.98%) | loss: 3.760263 | lrm: 0.12 | dt: 1446.68ms | tok/sec: 11,325 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 37 | total time: 124.92m | eta: 6.6m
step 04750/05000 (95.00%) | loss: 3.776258 | lrm: 0.12 | dt: 1591.29ms | tok/sec: 10,296 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 37 | total time: 124.95m | eta: 6.6m
step 04751/05000 (95.02%) | loss: 3.778094 | lrm: 0.12 | dt: 1568.18ms | tok/sec: 10,447 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 37 | total time: 124.97m | eta: 6.6m
step 04752/05000 (95.04%) | loss: 3.762150 | lrm: 0.12 | dt: 1524.28ms | tok/sec: 10,748 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 37 | total time: 125.00m | eta: 6.5m
step 04753/05000 (95.06%) | loss: 3.763717 | lrm: 0.12 | dt: 1456.58ms | tok/sec: 11,248 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 37 | total time: 125.02m | eta: 6.5m
step 04754/05000 (95.08%) | loss: 3.776926 | lrm: 0.12 | dt: 1485.67ms | tok/sec: 11,028 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 37 | total time: 125.05m | eta: 6.5m
step 04755/05000 (95.10%) | loss: 3.769906 | lrm: 0.12 | dt: 1581.89ms | tok/sec: 10,357 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 37 | total time: 125.07m | eta: 6.5m
step 04756/05000 (95.12%) | loss: 3.788414 | lrm: 0.12 | dt: 1483.82ms | tok/sec: 11,041 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 37 | total time: 125.10m | eta: 6.4m
step 04757/05000 (95.14%) | loss: 3.776837 | lrm: 0.12 | dt: 1445.67ms | tok/sec: 11,333 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 37 | total time: 125.12m | eta: 6.4m
step 04758/05000 (95.16%) | loss: 3.785980 | lrm: 0.12 | dt: 1459.83ms | tok/sec: 11,223 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 37 | total time: 125.15m | eta: 6.4m
step 04759/05000 (95.18%) | loss: 3.789069 | lrm: 0.12 | dt: 1453.71ms | tok/sec: 11,270 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 37 | total time: 125.17m | eta: 6.4m
step 04760/05000 (95.20%) | loss: 3.793815 | lrm: 0.12 | dt: 1535.31ms | tok/sec: 10,671 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 38 | total time: 125.20m | eta: 6.3m
step 04761/05000 (95.22%) | loss: 3.796706 | lrm: 0.12 | dt: 1495.19ms | tok/sec: 10,957 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 38 | total time: 125.22m | eta: 6.3m
step 04762/05000 (95.24%) | loss: 3.779962 | lrm: 0.12 | dt: 1441.01ms | tok/sec: 11,369 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 38 | total time: 125.24m | eta: 6.3m
step 04763/05000 (95.26%) | loss: 3.761414 | lrm: 0.12 | dt: 1526.96ms | tok/sec: 10,729 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 38 | total time: 125.27m | eta: 6.2m
step 04764/05000 (95.28%) | loss: 3.765911 | lrm: 0.12 | dt: 1499.64ms | tok/sec: 10,925 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 38 | total time: 125.29m | eta: 6.2m
step 04765/05000 (95.30%) | loss: 3.754806 | lrm: 0.12 | dt: 1544.83ms | tok/sec: 10,605 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 38 | total time: 125.32m | eta: 6.2m
step 04766/05000 (95.32%) | loss: 3.769284 | lrm: 0.12 | dt: 1526.23ms | tok/sec: 10,734 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 38 | total time: 125.35m | eta: 6.2m
step 04767/05000 (95.34%) | loss: 3.777716 | lrm: 0.12 | dt: 1571.44ms | tok/sec: 10,426 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 38 | total time: 125.37m | eta: 6.1m
step 04768/05000 (95.36%) | loss: 3.765961 | lrm: 0.12 | dt: 1622.25ms | tok/sec: 10,099 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 38 | total time: 125.40m | eta: 6.1m
step 04769/05000 (95.38%) | loss: 3.761033 | lrm: 0.12 | dt: 1566.26ms | tok/sec: 10,460 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 38 | total time: 125.43m | eta: 6.1m
step 04770/05000 (95.40%) | loss: 3.773858 | lrm: 0.12 | dt: 1531.20ms | tok/sec: 10,700 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 38 | total time: 125.45m | eta: 6.1m
step 04771/05000 (95.42%) | loss: 3.779159 | lrm: 0.12 | dt: 1742.30ms | tok/sec: 9,403 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 38 | total time: 125.48m | eta: 6.0m
step 04772/05000 (95.44%) | loss: 3.778253 | lrm: 0.12 | dt: 1625.41ms | tok/sec: 10,079 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 38 | total time: 125.51m | eta: 6.0m
step 04773/05000 (95.46%) | loss: 3.789025 | lrm: 0.12 | dt: 1556.93ms | tok/sec: 10,523 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 38 | total time: 125.53m | eta: 6.0m
step 04774/05000 (95.48%) | loss: 3.797128 | lrm: 0.12 | dt: 1742.88ms | tok/sec: 9,400 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 38 | total time: 125.56m | eta: 6.0m
step 04775/05000 (95.50%) | loss: 3.808036 | lrm: 0.12 | dt: 1634.17ms | tok/sec: 10,025 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 38 | total time: 125.59m | eta: 5.9m
step 04776/05000 (95.52%) | loss: 3.811988 | lrm: 0.12 | dt: 1794.92ms | tok/sec: 9,127 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 38 | total time: 125.62m | eta: 5.9m
step 04777/05000 (95.54%) | loss: 3.805438 | lrm: 0.12 | dt: 1579.50ms | tok/sec: 10,372 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 38 | total time: 125.65m | eta: 5.9m
step 04778/05000 (95.56%) | loss: 3.795852 | lrm: 0.11 | dt: 1511.34ms | tok/sec: 10,840 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 38 | total time: 125.67m | eta: 5.9m
step 04779/05000 (95.58%) | loss: 3.769377 | lrm: 0.11 | dt: 1598.64ms | tok/sec: 10,248 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 38 | total time: 125.70m | eta: 5.8m
step 04780/05000 (95.60%) | loss: 3.773730 | lrm: 0.11 | dt: 1585.69ms | tok/sec: 10,332 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 38 | total time: 125.72m | eta: 5.8m
step 04781/05000 (95.62%) | loss: 3.782764 | lrm: 0.11 | dt: 1604.04ms | tok/sec: 10,214 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 38 | total time: 125.75m | eta: 5.8m
step 04782/05000 (95.64%) | loss: 3.787535 | lrm: 0.11 | dt: 1591.44ms | tok/sec: 10,295 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 38 | total time: 125.78m | eta: 5.7m
step 04783/05000 (95.66%) | loss: 3.788675 | lrm: 0.11 | dt: 1537.88ms | tok/sec: 10,653 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 39 | total time: 125.80m | eta: 5.7m
step 04784/05000 (95.68%) | loss: 3.784802 | lrm: 0.11 | dt: 1530.21ms | tok/sec: 10,707 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 39 | total time: 125.83m | eta: 5.7m
step 04785/05000 (95.70%) | loss: 3.776420 | lrm: 0.11 | dt: 1550.97ms | tok/sec: 10,563 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 39 | total time: 125.85m | eta: 5.7m
step 04786/05000 (95.72%) | loss: 3.777196 | lrm: 0.11 | dt: 1507.90ms | tok/sec: 10,865 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 39 | total time: 125.88m | eta: 5.6m
step 04787/05000 (95.74%) | loss: 3.786237 | lrm: 0.11 | dt: 1506.95ms | tok/sec: 10,872 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 39 | total time: 125.90m | eta: 5.6m
step 04788/05000 (95.76%) | loss: 3.765180 | lrm: 0.11 | dt: 1595.68ms | tok/sec: 10,267 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 39 | total time: 125.93m | eta: 5.6m
step 04789/05000 (95.78%) | loss: 3.775842 | lrm: 0.11 | dt: 1579.07ms | tok/sec: 10,375 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 39 | total time: 125.96m | eta: 5.6m
step 04790/05000 (95.80%) | loss: 3.774619 | lrm: 0.11 | dt: 1560.20ms | tok/sec: 10,501 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 39 | total time: 125.98m | eta: 5.5m
step 04791/05000 (95.82%) | loss: 3.782380 | lrm: 0.11 | dt: 1597.23ms | tok/sec: 10,257 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 39 | total time: 126.01m | eta: 5.5m
step 04792/05000 (95.84%) | loss: 3.776343 | lrm: 0.11 | dt: 1614.38ms | tok/sec: 10,148 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 39 | total time: 126.04m | eta: 5.5m
step 04793/05000 (95.86%) | loss: 3.782865 | lrm: 0.11 | dt: 1606.88ms | tok/sec: 10,196 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 39 | total time: 126.06m | eta: 5.5m
step 04794/05000 (95.88%) | loss: 3.789933 | lrm: 0.11 | dt: 1517.32ms | tok/sec: 10,797 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 39 | total time: 126.09m | eta: 5.4m
step 04795/05000 (95.90%) | loss: 3.780719 | lrm: 0.11 | dt: 1544.25ms | tok/sec: 10,609 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 39 | total time: 126.11m | eta: 5.4m
step 04796/05000 (95.92%) | loss: 3.783527 | lrm: 0.11 | dt: 1553.46ms | tok/sec: 10,546 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 39 | total time: 126.14m | eta: 5.4m
step 04797/05000 (95.94%) | loss: 3.780716 | lrm: 0.11 | dt: 1558.03ms | tok/sec: 10,515 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 39 | total time: 126.17m | eta: 5.4m
step 04798/05000 (95.96%) | loss: 3.782900 | lrm: 0.11 | dt: 1540.77ms | tok/sec: 10,633 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 39 | total time: 126.19m | eta: 5.3m
step 04799/05000 (95.98%) | loss: 3.783608 | lrm: 0.11 | dt: 1516.24ms | tok/sec: 10,805 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 39 | total time: 126.22m | eta: 5.3m
Step 04800 | Validation bpb: 1.168470
<|bos|>The capital of France is the capital of France. The capital of France is the capital of France. The
<|bos|>The chemical symbol of gold is the gold, which is the gold that is used to make gold. The gold
<|bos|>If yesterday was Friday, then tomorrow will be the next day of the week. The next day will be Friday, and the
<|bos|>The opposite of hot is the same as the cold. The heat is the same as the hot, but
<|bos|>The planets of the solar system are: the solar system, the solar system, the solar system, the solar system,
<|bos|>My favorite color is the red, orange, and yellow. I have a red, orange, and
<|bos|>If 5*x + 3 = 13, then x is the number of points in the equation. The equation is the number of points in
step 04800/05000 (96.00%) | loss: 3.793469 | lrm: 0.11 | dt: 1634.36ms | tok/sec: 10,024 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 39 | total time: 126.24m | eta: 5.3m
step 04801/05000 (96.02%) | loss: 3.802238 | lrm: 0.11 | dt: 1534.49ms | tok/sec: 10,677 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 39 | total time: 126.27m | eta: 5.2m
step 04802/05000 (96.04%) | loss: 3.791575 | lrm: 0.11 | dt: 1567.37ms | tok/sec: 10,453 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 39 | total time: 126.30m | eta: 5.2m
step 04803/05000 (96.06%) | loss: 3.787696 | lrm: 0.11 | dt: 1517.16ms | tok/sec: 10,799 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 39 | total time: 126.32m | eta: 5.2m
step 04804/05000 (96.08%) | loss: 3.782454 | lrm: 0.11 | dt: 1563.50ms | tok/sec: 10,479 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 39 | total time: 126.35m | eta: 5.2m
step 04805/05000 (96.10%) | loss: 3.767879 | lrm: 0.11 | dt: 1558.99ms | tok/sec: 10,509 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 39 | total time: 126.37m | eta: 5.1m
step 04806/05000 (96.12%) | loss: 3.753160 | lrm: 0.11 | dt: 1575.17ms | tok/sec: 10,401 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 40 | total time: 126.40m | eta: 5.1m
step 04807/05000 (96.14%) | loss: 3.762366 | lrm: 0.11 | dt: 1558.31ms | tok/sec: 10,513 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 40 | total time: 126.43m | eta: 5.1m
step 04808/05000 (96.16%) | loss: 3.768111 | lrm: 0.11 | dt: 1586.67ms | tok/sec: 10,326 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 40 | total time: 126.45m | eta: 5.1m
step 04809/05000 (96.18%) | loss: 3.779721 | lrm: 0.11 | dt: 1463.51ms | tok/sec: 11,195 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 40 | total time: 126.48m | eta: 5.0m
step 04810/05000 (96.20%) | loss: 3.788062 | lrm: 0.11 | dt: 1488.90ms | tok/sec: 11,004 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 40 | total time: 126.50m | eta: 5.0m
step 04811/05000 (96.22%) | loss: 3.775012 | lrm: 0.11 | dt: 1479.21ms | tok/sec: 11,076 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 40 | total time: 126.53m | eta: 5.0m
step 04812/05000 (96.24%) | loss: 3.772484 | lrm: 0.10 | dt: 1526.37ms | tok/sec: 10,733 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 40 | total time: 126.55m | eta: 5.0m
step 04813/05000 (96.26%) | loss: 3.763962 | lrm: 0.10 | dt: 1469.04ms | tok/sec: 11,152 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 40 | total time: 126.58m | eta: 4.9m
step 04814/05000 (96.28%) | loss: 3.754384 | lrm: 0.10 | dt: 1477.96ms | tok/sec: 11,085 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 40 | total time: 126.60m | eta: 4.9m
step 04815/05000 (96.30%) | loss: 3.762494 | lrm: 0.10 | dt: 1475.98ms | tok/sec: 11,100 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 40 | total time: 126.63m | eta: 4.9m
step 04816/05000 (96.32%) | loss: 3.768331 | lrm: 0.10 | dt: 1493.43ms | tok/sec: 10,970 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 40 | total time: 126.65m | eta: 4.8m
step 04817/05000 (96.34%) | loss: 3.754226 | lrm: 0.10 | dt: 1478.69ms | tok/sec: 11,080 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 40 | total time: 126.67m | eta: 4.8m
step 04818/05000 (96.36%) | loss: 3.756433 | lrm: 0.10 | dt: 1505.73ms | tok/sec: 10,881 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 40 | total time: 126.70m | eta: 4.8m
step 04819/05000 (96.38%) | loss: 3.741205 | lrm: 0.10 | dt: 1473.53ms | tok/sec: 11,118 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 40 | total time: 126.72m | eta: 4.8m
step 04820/05000 (96.40%) | loss: 3.741391 | lrm: 0.10 | dt: 1489.33ms | tok/sec: 11,000 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 40 | total time: 126.75m | eta: 4.7m
step 04821/05000 (96.42%) | loss: 3.736642 | lrm: 0.10 | dt: 1405.63ms | tok/sec: 11,655 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 40 | total time: 126.77m | eta: 4.7m
step 04822/05000 (96.44%) | loss: 3.743830 | lrm: 0.10 | dt: 1417.87ms | tok/sec: 11,555 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 40 | total time: 126.80m | eta: 4.7m
step 04823/05000 (96.46%) | loss: 3.753644 | lrm: 0.10 | dt: 1529.10ms | tok/sec: 10,714 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 40 | total time: 126.82m | eta: 4.7m
step 04824/05000 (96.48%) | loss: 3.745678 | lrm: 0.10 | dt: 1480.22ms | tok/sec: 11,068 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 40 | total time: 126.85m | eta: 4.6m
step 04825/05000 (96.50%) | loss: 3.761120 | lrm: 0.10 | dt: 1515.93ms | tok/sec: 10,807 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 40 | total time: 126.87m | eta: 4.6m
step 04826/05000 (96.52%) | loss: 3.756953 | lrm: 0.10 | dt: 1487.78ms | tok/sec: 11,012 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 40 | total time: 126.90m | eta: 4.6m
step 04827/05000 (96.54%) | loss: 3.763193 | lrm: 0.10 | dt: 1521.25ms | tok/sec: 10,770 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 40 | total time: 126.92m | eta: 4.6m
step 04828/05000 (96.56%) | loss: 3.764380 | lrm: 0.10 | dt: 1503.63ms | tok/sec: 10,896 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 40 | total time: 126.95m | eta: 4.5m
step 04829/05000 (96.58%) | loss: 3.755340 | lrm: 0.10 | dt: 1548.19ms | tok/sec: 10,582 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 41 | total time: 126.97m | eta: 4.5m
step 04830/05000 (96.60%) | loss: 3.737104 | lrm: 0.10 | dt: 1513.03ms | tok/sec: 10,828 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 41 | total time: 127.00m | eta: 4.5m
step 04831/05000 (96.62%) | loss: 3.745601 | lrm: 0.10 | dt: 1610.18ms | tok/sec: 10,175 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 41 | total time: 127.02m | eta: 4.5m
step 04832/05000 (96.64%) | loss: 3.729634 | lrm: 0.10 | dt: 1585.06ms | tok/sec: 10,336 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 41 | total time: 127.05m | eta: 4.4m
step 04833/05000 (96.66%) | loss: 3.719928 | lrm: 0.10 | dt: 1588.53ms | tok/sec: 10,313 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 41 | total time: 127.08m | eta: 4.4m
step 04834/05000 (96.68%) | loss: 3.712213 | lrm: 0.10 | dt: 1457.17ms | tok/sec: 11,243 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 41 | total time: 127.10m | eta: 4.4m
step 04835/05000 (96.70%) | loss: 3.727772 | lrm: 0.10 | dt: 1534.58ms | tok/sec: 10,676 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 41 | total time: 127.13m | eta: 4.3m
step 04836/05000 (96.72%) | loss: 3.721008 | lrm: 0.10 | dt: 1574.55ms | tok/sec: 10,405 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 41 | total time: 127.15m | eta: 4.3m
step 04837/05000 (96.74%) | loss: 3.725218 | lrm: 0.10 | dt: 1559.06ms | tok/sec: 10,508 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 41 | total time: 127.18m | eta: 4.3m
step 04838/05000 (96.76%) | loss: 3.713660 | lrm: 0.10 | dt: 1488.53ms | tok/sec: 11,006 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 41 | total time: 127.20m | eta: 4.3m
step 04839/05000 (96.78%) | loss: 3.726224 | lrm: 0.10 | dt: 1492.40ms | tok/sec: 10,978 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 41 | total time: 127.23m | eta: 4.2m
step 04840/05000 (96.80%) | loss: 3.726591 | lrm: 0.10 | dt: 1545.79ms | tok/sec: 10,599 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 41 | total time: 127.26m | eta: 4.2m
step 04841/05000 (96.82%) | loss: 3.724851 | lrm: 0.10 | dt: 1613.75ms | tok/sec: 10,152 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 41 | total time: 127.28m | eta: 4.2m
step 04842/05000 (96.84%) | loss: 3.719262 | lrm: 0.10 | dt: 1517.84ms | tok/sec: 10,794 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 41 | total time: 127.31m | eta: 4.2m
step 04843/05000 (96.86%) | loss: 3.709701 | lrm: 0.10 | dt: 1569.77ms | tok/sec: 10,437 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 41 | total time: 127.33m | eta: 4.1m
step 04844/05000 (96.88%) | loss: 3.714535 | lrm: 0.10 | dt: 1541.28ms | tok/sec: 10,630 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 41 | total time: 127.36m | eta: 4.1m
step 04845/05000 (96.90%) | loss: 3.722792 | lrm: 0.10 | dt: 1559.83ms | tok/sec: 10,503 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 41 | total time: 127.39m | eta: 4.1m
step 04846/05000 (96.92%) | loss: 3.730816 | lrm: 0.10 | dt: 1584.42ms | tok/sec: 10,340 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 41 | total time: 127.41m | eta: 4.1m
step 04847/05000 (96.94%) | loss: 3.737139 | lrm: 0.09 | dt: 1641.77ms | tok/sec: 9,979 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 41 | total time: 127.44m | eta: 4.0m
step 04848/05000 (96.96%) | loss: 3.745132 | lrm: 0.09 | dt: 1650.15ms | tok/sec: 9,928 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 41 | total time: 127.47m | eta: 4.0m
step 04849/05000 (96.98%) | loss: 3.733890 | lrm: 0.09 | dt: 1702.90ms | tok/sec: 9,621 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 41 | total time: 127.50m | eta: 4.0m
step 04850/05000 (97.00%) | loss: 3.724835 | lrm: 0.09 | dt: 1669.97ms | tok/sec: 9,810 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 41 | total time: 127.52m | eta: 4.0m
step 04851/05000 (97.02%) | loss: 3.719976 | lrm: 0.09 | dt: 1718.29ms | tok/sec: 9,535 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 41 | total time: 127.55m | eta: 3.9m
step 04852/05000 (97.04%) | loss: 3.721442 | lrm: 0.09 | dt: 1648.47ms | tok/sec: 9,938 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 42 | total time: 127.58m | eta: 3.9m
step 04853/05000 (97.06%) | loss: 3.734308 | lrm: 0.09 | dt: 1657.53ms | tok/sec: 9,884 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 42 | total time: 127.61m | eta: 3.9m
step 04854/05000 (97.08%) | loss: 3.740105 | lrm: 0.09 | dt: 1623.43ms | tok/sec: 10,092 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 42 | total time: 127.63m | eta: 3.8m
step 04855/05000 (97.10%) | loss: 3.737250 | lrm: 0.09 | dt: 1585.99ms | tok/sec: 10,330 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 42 | total time: 127.66m | eta: 3.8m
step 04856/05000 (97.12%) | loss: 3.731555 | lrm: 0.09 | dt: 1583.72ms | tok/sec: 10,345 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 42 | total time: 127.69m | eta: 3.8m
step 04857/05000 (97.14%) | loss: 3.722779 | lrm: 0.09 | dt: 1513.84ms | tok/sec: 10,822 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 42 | total time: 127.71m | eta: 3.8m
step 04858/05000 (97.16%) | loss: 3.737262 | lrm: 0.09 | dt: 1652.81ms | tok/sec: 9,912 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 42 | total time: 127.74m | eta: 3.7m
step 04859/05000 (97.18%) | loss: 3.732018 | lrm: 0.09 | dt: 1684.81ms | tok/sec: 9,724 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 42 | total time: 127.77m | eta: 3.7m
step 04860/05000 (97.20%) | loss: 3.747903 | lrm: 0.09 | dt: 1662.02ms | tok/sec: 9,857 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 42 | total time: 127.80m | eta: 3.7m
step 04861/05000 (97.22%) | loss: 3.736030 | lrm: 0.09 | dt: 1659.54ms | tok/sec: 9,872 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 42 | total time: 127.82m | eta: 3.7m
step 04862/05000 (97.24%) | loss: 3.750577 | lrm: 0.09 | dt: 1683.47ms | tok/sec: 9,732 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 42 | total time: 127.85m | eta: 3.6m
step 04863/05000 (97.26%) | loss: 3.757681 | lrm: 0.09 | dt: 1654.98ms | tok/sec: 9,899 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 42 | total time: 127.88m | eta: 3.6m
step 04864/05000 (97.28%) | loss: 3.760246 | lrm: 0.09 | dt: 1645.98ms | tok/sec: 9,953 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 42 | total time: 127.91m | eta: 3.6m
step 04865/05000 (97.30%) | loss: 3.770483 | lrm: 0.09 | dt: 1655.31ms | tok/sec: 9,897 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 42 | total time: 127.93m | eta: 3.6m
step 04866/05000 (97.32%) | loss: 3.762315 | lrm: 0.09 | dt: 1677.46ms | tok/sec: 9,767 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 42 | total time: 127.96m | eta: 3.5m
step 04867/05000 (97.34%) | loss: 3.773392 | lrm: 0.09 | dt: 1680.07ms | tok/sec: 9,751 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 42 | total time: 127.99m | eta: 3.5m
step 04868/05000 (97.36%) | loss: 3.773775 | lrm: 0.09 | dt: 1677.77ms | tok/sec: 9,765 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 42 | total time: 128.02m | eta: 3.5m
step 04869/05000 (97.38%) | loss: 3.764779 | lrm: 0.09 | dt: 1628.82ms | tok/sec: 10,058 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 42 | total time: 128.04m | eta: 3.5m
step 04870/05000 (97.40%) | loss: 3.762745 | lrm: 0.09 | dt: 1641.46ms | tok/sec: 9,981 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 42 | total time: 128.07m | eta: 3.4m
step 04871/05000 (97.42%) | loss: 3.756675 | lrm: 0.09 | dt: 1634.18ms | tok/sec: 10,025 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 42 | total time: 128.10m | eta: 3.4m
step 04872/05000 (97.44%) | loss: 3.765051 | lrm: 0.09 | dt: 1639.39ms | tok/sec: 9,993 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 42 | total time: 128.13m | eta: 3.4m
step 04873/05000 (97.46%) | loss: 3.768684 | lrm: 0.09 | dt: 1663.34ms | tok/sec: 9,850 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 42 | total time: 128.15m | eta: 3.3m
step 04874/05000 (97.48%) | loss: 3.763398 | lrm: 0.09 | dt: 1667.82ms | tok/sec: 9,823 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 42 | total time: 128.18m | eta: 3.3m
step 04875/05000 (97.50%) | loss: 3.757030 | lrm: 0.09 | dt: 1625.23ms | tok/sec: 10,081 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 43 | total time: 128.21m | eta: 3.3m
step 04876/05000 (97.52%) | loss: 3.771752 | lrm: 0.09 | dt: 1625.96ms | tok/sec: 10,076 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 43 | total time: 128.24m | eta: 3.3m
step 04877/05000 (97.54%) | loss: 3.771572 | lrm: 0.09 | dt: 1644.95ms | tok/sec: 9,960 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 43 | total time: 128.26m | eta: 3.2m
step 04878/05000 (97.56%) | loss: 3.774807 | lrm: 0.09 | dt: 1658.67ms | tok/sec: 9,877 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 43 | total time: 128.29m | eta: 3.2m
step 04879/05000 (97.58%) | loss: 3.783894 | lrm: 0.09 | dt: 1635.98ms | tok/sec: 10,014 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 43 | total time: 128.32m | eta: 3.2m
step 04880/05000 (97.60%) | loss: 3.764071 | lrm: 0.09 | dt: 1677.65ms | tok/sec: 9,766 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 43 | total time: 128.35m | eta: 3.2m
step 04881/05000 (97.62%) | loss: 3.776415 | lrm: 0.08 | dt: 1659.90ms | tok/sec: 9,870 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 43 | total time: 128.37m | eta: 3.1m
step 04882/05000 (97.64%) | loss: 3.779111 | lrm: 0.08 | dt: 1624.89ms | tok/sec: 10,083 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 43 | total time: 128.40m | eta: 3.1m
step 04883/05000 (97.66%) | loss: 3.775753 | lrm: 0.08 | dt: 1670.64ms | tok/sec: 9,807 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 43 | total time: 128.43m | eta: 3.1m
step 04884/05000 (97.68%) | loss: 3.780112 | lrm: 0.08 | dt: 1610.20ms | tok/sec: 10,175 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 43 | total time: 128.46m | eta: 3.1m
step 04885/05000 (97.70%) | loss: 3.784159 | lrm: 0.08 | dt: 1639.60ms | tok/sec: 9,992 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 43 | total time: 128.48m | eta: 3.0m
step 04886/05000 (97.72%) | loss: 3.782932 | lrm: 0.08 | dt: 1797.38ms | tok/sec: 9,115 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 43 | total time: 128.51m | eta: 3.0m
step 04887/05000 (97.74%) | loss: 3.768234 | lrm: 0.08 | dt: 1581.28ms | tok/sec: 10,361 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 43 | total time: 128.54m | eta: 3.0m
step 04888/05000 (97.76%) | loss: 3.767691 | lrm: 0.08 | dt: 1600.24ms | tok/sec: 10,238 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 43 | total time: 128.57m | eta: 3.0m
step 04889/05000 (97.78%) | loss: 3.770290 | lrm: 0.08 | dt: 1568.46ms | tok/sec: 10,445 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 43 | total time: 128.59m | eta: 2.9m
step 04890/05000 (97.80%) | loss: 3.776416 | lrm: 0.08 | dt: 1673.01ms | tok/sec: 9,793 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 43 | total time: 128.62m | eta: 2.9m
step 04891/05000 (97.82%) | loss: 3.782478 | lrm: 0.08 | dt: 1589.21ms | tok/sec: 10,309 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 43 | total time: 128.65m | eta: 2.9m
step 04892/05000 (97.84%) | loss: 3.774198 | lrm: 0.08 | dt: 1658.72ms | tok/sec: 9,877 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 43 | total time: 128.67m | eta: 2.8m
step 04893/05000 (97.86%) | loss: 3.775861 | lrm: 0.08 | dt: 1573.80ms | tok/sec: 10,410 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 43 | total time: 128.70m | eta: 2.8m
step 04894/05000 (97.88%) | loss: 3.776109 | lrm: 0.08 | dt: 1711.83ms | tok/sec: 9,571 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 43 | total time: 128.73m | eta: 2.8m
step 04895/05000 (97.90%) | loss: 3.783492 | lrm: 0.08 | dt: 1653.61ms | tok/sec: 9,908 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 43 | total time: 128.76m | eta: 2.8m
step 04896/05000 (97.92%) | loss: 3.778531 | lrm: 0.08 | dt: 1836.85ms | tok/sec: 8,919 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 43 | total time: 128.79m | eta: 2.7m
step 04897/05000 (97.94%) | loss: 3.781213 | lrm: 0.08 | dt: 1560.40ms | tok/sec: 10,499 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 43 | total time: 128.81m | eta: 2.7m
step 04898/05000 (97.96%) | loss: 3.764825 | lrm: 0.08 | dt: 1534.90ms | tok/sec: 10,674 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 44 | total time: 128.84m | eta: 2.7m
step 04899/05000 (97.98%) | loss: 3.760433 | lrm: 0.08 | dt: 1753.82ms | tok/sec: 9,341 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 44 | total time: 128.87m | eta: 2.7m
Step 04900 | Validation bpb: 1.167120
<|bos|>The capital of France is the capital of France. The capital of France is the capital of France. The
<|bos|>The chemical symbol of gold is the gold, which is the gold that is used to make gold. The gold
<|bos|>If yesterday was Friday, then tomorrow will be the next day of the week. The next day will be Friday, and the
<|bos|>The opposite of hot is the same as the hot, but the same as the hot, but the same
<|bos|>The planets of the solar system are: the solar system, the solar system, the solar system, the solar system,
<|bos|>My favorite color is the red, orange, and yellow. I have a lot of red, orange
<|bos|>If 5*x + 3 = 13, then x is the number of points that are in the equation. The equation for the number of
step 04900/05000 (98.00%) | loss: 3.772723 | lrm: 0.08 | dt: 2499.76ms | tok/sec: 6,554 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 44 | total time: 128.91m | eta: 2.6m
step 04901/05000 (98.02%) | loss: 3.773870 | lrm: 0.08 | dt: 1579.75ms | tok/sec: 10,371 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 44 | total time: 128.94m | eta: 2.6m
step 04902/05000 (98.04%) | loss: 3.765679 | lrm: 0.08 | dt: 1575.04ms | tok/sec: 10,402 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 44 | total time: 128.96m | eta: 2.6m
step 04903/05000 (98.06%) | loss: 3.770858 | lrm: 0.08 | dt: 1619.95ms | tok/sec: 10,113 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 44 | total time: 128.99m | eta: 2.6m
step 04904/05000 (98.08%) | loss: 3.754838 | lrm: 0.08 | dt: 1609.46ms | tok/sec: 10,179 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 44 | total time: 129.02m | eta: 2.5m
step 04905/05000 (98.10%) | loss: 3.764533 | lrm: 0.08 | dt: 1849.09ms | tok/sec: 8,860 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 44 | total time: 129.05m | eta: 2.5m
step 04906/05000 (98.12%) | loss: 3.764154 | lrm: 0.08 | dt: 1673.23ms | tok/sec: 9,791 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 44 | total time: 129.07m | eta: 2.5m
step 04907/05000 (98.14%) | loss: 3.763734 | lrm: 0.08 | dt: 1667.52ms | tok/sec: 9,825 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 44 | total time: 129.10m | eta: 2.5m
step 04908/05000 (98.16%) | loss: 3.767862 | lrm: 0.08 | dt: 1568.93ms | tok/sec: 10,442 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 44 | total time: 129.13m | eta: 2.4m
step 04909/05000 (98.18%) | loss: 3.761272 | lrm: 0.08 | dt: 1583.22ms | tok/sec: 10,348 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 44 | total time: 129.16m | eta: 2.4m
step 04910/05000 (98.20%) | loss: 3.755150 | lrm: 0.08 | dt: 1564.25ms | tok/sec: 10,474 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 44 | total time: 129.18m | eta: 2.4m
step 04911/05000 (98.22%) | loss: 3.756232 | lrm: 0.08 | dt: 1628.23ms | tok/sec: 10,062 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 44 | total time: 129.21m | eta: 2.3m
step 04912/05000 (98.24%) | loss: 3.755031 | lrm: 0.08 | dt: 1654.91ms | tok/sec: 9,900 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 44 | total time: 129.24m | eta: 2.3m
step 04913/05000 (98.26%) | loss: 3.761974 | lrm: 0.08 | dt: 1607.24ms | tok/sec: 10,193 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 44 | total time: 129.26m | eta: 2.3m
step 04914/05000 (98.28%) | loss: 3.765524 | lrm: 0.08 | dt: 1585.63ms | tok/sec: 10,332 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 44 | total time: 129.29m | eta: 2.3m
step 04915/05000 (98.30%) | loss: 3.765619 | lrm: 0.07 | dt: 1587.86ms | tok/sec: 10,318 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 44 | total time: 129.32m | eta: 2.2m
step 04916/05000 (98.32%) | loss: 3.765193 | lrm: 0.07 | dt: 1621.31ms | tok/sec: 10,105 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 44 | total time: 129.34m | eta: 2.2m
step 04917/05000 (98.34%) | loss: 3.765202 | lrm: 0.07 | dt: 1573.63ms | tok/sec: 10,411 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 44 | total time: 129.37m | eta: 2.2m
step 04918/05000 (98.36%) | loss: 3.749106 | lrm: 0.07 | dt: 1593.79ms | tok/sec: 10,279 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 44 | total time: 129.40m | eta: 2.2m
step 04919/05000 (98.38%) | loss: 3.761764 | lrm: 0.07 | dt: 1560.25ms | tok/sec: 10,500 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 44 | total time: 129.42m | eta: 2.1m
step 04920/05000 (98.40%) | loss: 3.770559 | lrm: 0.07 | dt: 1670.75ms | tok/sec: 9,806 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 44 | total time: 129.45m | eta: 2.1m
step 04921/05000 (98.42%) | loss: 3.761019 | lrm: 0.07 | dt: 1785.54ms | tok/sec: 9,175 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 45 | total time: 129.48m | eta: 2.1m
step 04922/05000 (98.44%) | loss: 3.752117 | lrm: 0.07 | dt: 1924.22ms | tok/sec: 8,514 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 45 | total time: 129.51m | eta: 2.1m
step 04923/05000 (98.46%) | loss: 3.743576 | lrm: 0.07 | dt: 1529.69ms | tok/sec: 10,710 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 45 | total time: 129.54m | eta: 2.0m
step 04924/05000 (98.48%) | loss: 3.746361 | lrm: 0.07 | dt: 1625.20ms | tok/sec: 10,081 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 45 | total time: 129.56m | eta: 2.0m
step 04925/05000 (98.50%) | loss: 3.741600 | lrm: 0.07 | dt: 1586.64ms | tok/sec: 10,326 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 45 | total time: 129.59m | eta: 2.0m
step 04926/05000 (98.52%) | loss: 3.750475 | lrm: 0.07 | dt: 1633.85ms | tok/sec: 10,027 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 45 | total time: 129.62m | eta: 2.0m
step 04927/05000 (98.54%) | loss: 3.727202 | lrm: 0.07 | dt: 1591.21ms | tok/sec: 10,296 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 45 | total time: 129.64m | eta: 1.9m
step 04928/05000 (98.56%) | loss: 3.728790 | lrm: 0.07 | dt: 1566.46ms | tok/sec: 10,459 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 45 | total time: 129.67m | eta: 1.9m
step 04929/05000 (98.58%) | loss: 3.748988 | lrm: 0.07 | dt: 1585.83ms | tok/sec: 10,331 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 45 | total time: 129.70m | eta: 1.9m
step 04930/05000 (98.60%) | loss: 3.743237 | lrm: 0.07 | dt: 1571.81ms | tok/sec: 10,423 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 45 | total time: 129.72m | eta: 1.8m
step 04931/05000 (98.62%) | loss: 3.738012 | lrm: 0.07 | dt: 1600.13ms | tok/sec: 10,239 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 45 | total time: 129.75m | eta: 1.8m
step 04932/05000 (98.64%) | loss: 3.740972 | lrm: 0.07 | dt: 1593.39ms | tok/sec: 10,282 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 45 | total time: 129.78m | eta: 1.8m
step 04933/05000 (98.66%) | loss: 3.752548 | lrm: 0.07 | dt: 1624.28ms | tok/sec: 10,086 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 45 | total time: 129.80m | eta: 1.8m
step 04934/05000 (98.68%) | loss: 3.753011 | lrm: 0.07 | dt: 1739.79ms | tok/sec: 9,417 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 45 | total time: 129.83m | eta: 1.7m
step 04935/05000 (98.70%) | loss: 3.735949 | lrm: 0.07 | dt: 1809.48ms | tok/sec: 9,054 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 45 | total time: 129.86m | eta: 1.7m
step 04936/05000 (98.72%) | loss: 3.733689 | lrm: 0.07 | dt: 1655.44ms | tok/sec: 9,897 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 45 | total time: 129.89m | eta: 1.7m
step 04937/05000 (98.74%) | loss: 3.737236 | lrm: 0.07 | dt: 1693.59ms | tok/sec: 9,674 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 45 | total time: 129.92m | eta: 1.7m
step 04938/05000 (98.76%) | loss: 3.721923 | lrm: 0.07 | dt: 1723.78ms | tok/sec: 9,504 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 45 | total time: 129.95m | eta: 1.6m
step 04939/05000 (98.78%) | loss: 3.724606 | lrm: 0.07 | dt: 1725.62ms | tok/sec: 9,494 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 45 | total time: 129.98m | eta: 1.6m
step 04940/05000 (98.80%) | loss: 3.737340 | lrm: 0.07 | dt: 1675.99ms | tok/sec: 9,775 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 45 | total time: 130.00m | eta: 1.6m
step 04941/05000 (98.82%) | loss: 3.728170 | lrm: 0.07 | dt: 1700.96ms | tok/sec: 9,632 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 45 | total time: 130.03m | eta: 1.6m
step 04942/05000 (98.84%) | loss: 3.724711 | lrm: 0.07 | dt: 1656.66ms | tok/sec: 9,889 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 45 | total time: 130.06m | eta: 1.5m
step 04943/05000 (98.86%) | loss: 3.745375 | lrm: 0.07 | dt: 1693.02ms | tok/sec: 9,677 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 45 | total time: 130.09m | eta: 1.5m
step 04944/05000 (98.88%) | loss: 3.728512 | lrm: 0.07 | dt: 1688.11ms | tok/sec: 9,705 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 46 | total time: 130.12m | eta: 1.5m
step 04945/05000 (98.90%) | loss: 3.743629 | lrm: 0.07 | dt: 1679.71ms | tok/sec: 9,754 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 46 | total time: 130.14m | eta: 1.5m
step 04946/05000 (98.92%) | loss: 3.745840 | lrm: 0.07 | dt: 1649.29ms | tok/sec: 9,933 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 46 | total time: 130.17m | eta: 1.4m
step 04947/05000 (98.94%) | loss: 3.747163 | lrm: 0.07 | dt: 1648.16ms | tok/sec: 9,940 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 46 | total time: 130.20m | eta: 1.4m
step 04948/05000 (98.96%) | loss: 3.740559 | lrm: 0.07 | dt: 1661.26ms | tok/sec: 9,862 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 46 | total time: 130.23m | eta: 1.4m
step 04949/05000 (98.98%) | loss: 3.740370 | lrm: 0.06 | dt: 1629.84ms | tok/sec: 10,052 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 46 | total time: 130.25m | eta: 1.3m
step 04950/05000 (99.00%) | loss: 3.734345 | lrm: 0.06 | dt: 1700.88ms | tok/sec: 9,632 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 46 | total time: 130.28m | eta: 1.3m
step 04951/05000 (99.02%) | loss: 3.745479 | lrm: 0.06 | dt: 1666.08ms | tok/sec: 9,833 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 46 | total time: 130.31m | eta: 1.3m
step 04952/05000 (99.04%) | loss: 3.736407 | lrm: 0.06 | dt: 1649.90ms | tok/sec: 9,930 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 46 | total time: 130.34m | eta: 1.3m
step 04953/05000 (99.06%) | loss: 3.743969 | lrm: 0.06 | dt: 1478.26ms | tok/sec: 11,083 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 46 | total time: 130.36m | eta: 1.2m
step 04954/05000 (99.08%) | loss: 3.752969 | lrm: 0.06 | dt: 1559.38ms | tok/sec: 10,506 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 46 | total time: 130.39m | eta: 1.2m
step 04955/05000 (99.10%) | loss: 3.734971 | lrm: 0.06 | dt: 1473.69ms | tok/sec: 11,117 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 46 | total time: 130.41m | eta: 1.2m
step 04956/05000 (99.12%) | loss: 3.741801 | lrm: 0.06 | dt: 1468.11ms | tok/sec: 11,159 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 46 | total time: 130.44m | eta: 1.2m
step 04957/05000 (99.14%) | loss: 3.746506 | lrm: 0.06 | dt: 1556.29ms | tok/sec: 10,527 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 46 | total time: 130.46m | eta: 1.1m
step 04958/05000 (99.16%) | loss: 3.748536 | lrm: 0.06 | dt: 1528.08ms | tok/sec: 10,721 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 46 | total time: 130.49m | eta: 1.1m
step 04959/05000 (99.18%) | loss: 3.755783 | lrm: 0.06 | dt: 1592.05ms | tok/sec: 10,291 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 46 | total time: 130.51m | eta: 1.1m
step 04960/05000 (99.20%) | loss: 3.765304 | lrm: 0.06 | dt: 1554.97ms | tok/sec: 10,536 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 46 | total time: 130.54m | eta: 1.1m
step 04961/05000 (99.22%) | loss: 3.772858 | lrm: 0.06 | dt: 1558.05ms | tok/sec: 10,515 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 46 | total time: 130.57m | eta: 1.0m
step 04962/05000 (99.24%) | loss: 3.746264 | lrm: 0.06 | dt: 1610.27ms | tok/sec: 10,174 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 46 | total time: 130.59m | eta: 1.0m
step 04963/05000 (99.26%) | loss: 3.739525 | lrm: 0.06 | dt: 1603.73ms | tok/sec: 10,216 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 46 | total time: 130.62m | eta: 1.0m
step 04964/05000 (99.28%) | loss: 3.742742 | lrm: 0.06 | dt: 1652.97ms | tok/sec: 9,911 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 46 | total time: 130.65m | eta: 0.9m
step 04965/05000 (99.30%) | loss: 3.733752 | lrm: 0.06 | dt: 1581.02ms | tok/sec: 10,362 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 46 | total time: 130.67m | eta: 0.9m
step 04966/05000 (99.32%) | loss: 3.715918 | lrm: 0.06 | dt: 1575.59ms | tok/sec: 10,398 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 46 | total time: 130.70m | eta: 0.9m
step 04967/05000 (99.34%) | loss: 3.718592 | lrm: 0.06 | dt: 1478.60ms | tok/sec: 11,080 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 46 | total time: 130.72m | eta: 0.9m
step 04968/05000 (99.36%) | loss: 3.716334 | lrm: 0.06 | dt: 1587.30ms | tok/sec: 10,321 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 47 | total time: 130.75m | eta: 0.8m
step 04969/05000 (99.38%) | loss: 3.732480 | lrm: 0.06 | dt: 1746.50ms | tok/sec: 9,381 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 47 | total time: 130.78m | eta: 0.8m
step 04970/05000 (99.40%) | loss: 3.728067 | lrm: 0.06 | dt: 1639.24ms | tok/sec: 9,994 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 47 | total time: 130.81m | eta: 0.8m
step 04971/05000 (99.42%) | loss: 3.724363 | lrm: 0.06 | dt: 1487.63ms | tok/sec: 11,013 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 47 | total time: 130.83m | eta: 0.8m
step 04972/05000 (99.44%) | loss: 3.722713 | lrm: 0.06 | dt: 1531.99ms | tok/sec: 10,694 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 47 | total time: 130.86m | eta: 0.7m
step 04973/05000 (99.46%) | loss: 3.716354 | lrm: 0.06 | dt: 1779.93ms | tok/sec: 9,204 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 47 | total time: 130.89m | eta: 0.7m
step 04974/05000 (99.48%) | loss: 3.714774 | lrm: 0.06 | dt: 1632.13ms | tok/sec: 10,038 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 47 | total time: 130.91m | eta: 0.7m
step 04975/05000 (99.50%) | loss: 3.735691 | lrm: 0.06 | dt: 1623.74ms | tok/sec: 10,090 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 47 | total time: 130.94m | eta: 0.7m
step 04976/05000 (99.52%) | loss: 3.737060 | lrm: 0.06 | dt: 1896.22ms | tok/sec: 8,640 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 47 | total time: 130.97m | eta: 0.6m
step 04977/05000 (99.54%) | loss: 3.744247 | lrm: 0.06 | dt: 1824.49ms | tok/sec: 8,980 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 47 | total time: 131.00m | eta: 0.6m
step 04978/05000 (99.56%) | loss: 3.740520 | lrm: 0.06 | dt: 1522.63ms | tok/sec: 10,760 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 47 | total time: 131.03m | eta: 0.6m
step 04979/05000 (99.58%) | loss: 3.743188 | lrm: 0.06 | dt: 1441.17ms | tok/sec: 11,368 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 47 | total time: 131.05m | eta: 0.6m
step 04980/05000 (99.60%) | loss: 3.735855 | lrm: 0.06 | dt: 1502.02ms | tok/sec: 10,907 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 47 | total time: 131.08m | eta: 0.5m
step 04981/05000 (99.62%) | loss: 3.742889 | lrm: 0.06 | dt: 1543.26ms | tok/sec: 10,616 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 47 | total time: 131.10m | eta: 0.5m
step 04982/05000 (99.64%) | loss: 3.750700 | lrm: 0.06 | dt: 1480.75ms | tok/sec: 11,064 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 47 | total time: 131.13m | eta: 0.5m
step 04983/05000 (99.66%) | loss: 3.739219 | lrm: 0.05 | dt: 1531.98ms | tok/sec: 10,694 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 47 | total time: 131.15m | eta: 0.4m
step 04984/05000 (99.68%) | loss: 3.748037 | lrm: 0.05 | dt: 1475.38ms | tok/sec: 11,104 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 47 | total time: 131.18m | eta: 0.4m
step 04985/05000 (99.70%) | loss: 3.750059 | lrm: 0.05 | dt: 1530.70ms | tok/sec: 10,703 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 47 | total time: 131.20m | eta: 0.4m
step 04986/05000 (99.72%) | loss: 3.750349 | lrm: 0.05 | dt: 1501.65ms | tok/sec: 10,910 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 47 | total time: 131.23m | eta: 0.4m
step 04987/05000 (99.74%) | loss: 3.754770 | lrm: 0.05 | dt: 1532.40ms | tok/sec: 10,691 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 47 | total time: 131.26m | eta: 0.3m
step 04988/05000 (99.76%) | loss: 3.748440 | lrm: 0.05 | dt: 1481.66ms | tok/sec: 11,057 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 47 | total time: 131.28m | eta: 0.3m
step 04989/05000 (99.78%) | loss: 3.756800 | lrm: 0.05 | dt: 1444.53ms | tok/sec: 11,342 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 47 | total time: 131.30m | eta: 0.3m
step 04990/05000 (99.80%) | loss: 3.753862 | lrm: 0.05 | dt: 1503.56ms | tok/sec: 10,896 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 47 | total time: 131.33m | eta: 0.3m
step 04991/05000 (99.82%) | loss: 3.752691 | lrm: 0.05 | dt: 1527.92ms | tok/sec: 10,723 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 48 | total time: 131.35m | eta: 0.2m
step 04992/05000 (99.84%) | loss: 3.754924 | lrm: 0.05 | dt: 1520.94ms | tok/sec: 10,772 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 48 | total time: 131.38m | eta: 0.2m
step 04993/05000 (99.86%) | loss: 3.756582 | lrm: 0.05 | dt: 1512.41ms | tok/sec: 10,833 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 48 | total time: 131.40m | eta: 0.2m
step 04994/05000 (99.88%) | loss: 3.748885 | lrm: 0.05 | dt: 1466.38ms | tok/sec: 11,173 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 48 | total time: 131.43m | eta: 0.2m
step 04995/05000 (99.90%) | loss: 3.753573 | lrm: 0.05 | dt: 1460.28ms | tok/sec: 11,219 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 48 | total time: 131.45m | eta: 0.1m
step 04996/05000 (99.92%) | loss: 3.749403 | lrm: 0.05 | dt: 1496.99ms | tok/sec: 10,944 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 48 | total time: 131.48m | eta: 0.1m
step 04997/05000 (99.94%) | loss: 3.737026 | lrm: 0.05 | dt: 1522.64ms | tok/sec: 10,760 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 48 | total time: 131.50m | eta: 0.1m
step 04998/05000 (99.96%) | loss: 3.747258 | lrm: 0.05 | dt: 1499.94ms | tok/sec: 10,923 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 48 | total time: 131.53m | eta: 0.1m
step 04999/05000 (99.98%) | loss: 3.751467 | lrm: 0.05 | dt: 1482.88ms | tok/sec: 11,048 | bf16_mfu: 0.00 | epoch: 1 pq: 2 rg: 48 | total time: 131.55m | eta: 0.0m
Step 05000 | Validation bpb: 1.165758
<|bos|>The capital of France is the capital of France. The capital of France is the capital of France. The
<|bos|>The chemical symbol of gold is the gold, which is the gold that is used to make gold. The gold
<|bos|>If yesterday was Friday, then tomorrow will be the next day of the week. The next day will be Friday, April
<|bos|>The opposite of hot is the fact that the sun is not a hot object. The sun is not a
<|bos|>The planets of the solar system are: the solar system, the solar system, the solar system, the solar system,
<|bos|>My favorite color is the red, orange, and yellow. I have a lot of red, orange
<|bos|>If 5*x + 3 = 13, then x is the number of points that are in the equation. The equation for the number of
2026-08-04 13:17:02,074 - nanochat.checkpoint_manager - INFO - Saved model parameters to: $ORX_RUN_DIR/repo/.cache/nanochat/base_checkpoints/d6/model_005000.pt
2026-08-04 13:17:02,076 - nanochat.checkpoint_manager - INFO - Saved metadata to: $ORX_RUN_DIR/repo/.cache/nanochat/base_checkpoints/d6/meta_005000.json
2026-08-04 13:17:04,102 - nanochat.checkpoint_manager - INFO - Saved optimizer state to: $ORX_RUN_DIR/repo/.cache/nanochat/base_checkpoints/d6/optim_005000_rank0.pt
Peak memory usage: 0.00MiB
Total training time: 131.55m
Minimum validation bpb: 1.165758
Autodetected device type: mps
2026-08-04 13:17:07,268 - nanochat.common - INFO - Distributed world size: 1
2026-08-04 13:17:07,269 - nanochat.checkpoint_manager - INFO - No model tag provided, guessing model tag: d6
2026-08-04 13:17:07,269 - nanochat.checkpoint_manager - INFO - Loading model from $ORX_RUN_DIR/repo/.cache/nanochat/base_checkpoints/d6 with step 5000
2026-08-04 13:17:07,381 - nanochat.checkpoint_manager - INFO - Building model with config: {'sequence_len': 512, 'vocab_size': 32768, 'n_layer': 6, 'n_head': 6, 'n_kv_head': 6, 'n_embd': 384, 'window_pattern': 'L'}
Evaluating model: base_model (step 5000)
Eval modes: bpb, core, sample

================================================================================
Model Samples
================================================================================

Conditioned samples:
--------------------------------------------------------------------------------
<|bos|>The capital of France is the capital of France. The capital of France is the capital of France. The
--------------------------------------------------------------------------------
<|bos|>The chemical symbol of gold is the gold, which is the gold that is used to make gold. The gold
--------------------------------------------------------------------------------
<|bos|>If yesterday was Friday, then tomorrow will be the next day of the week. The next day will be Friday, April
--------------------------------------------------------------------------------
<|bos|>The opposite of hot is the fact that the sun is not a hot object. The sun is not a
--------------------------------------------------------------------------------
<|bos|>The planets of the solar system are: the solar system, the solar system, the solar system, the solar system,
--------------------------------------------------------------------------------
<|bos|>My favorite color is the red, orange, and yellow. I have a lot of red, orange
--------------------------------------------------------------------------------
<|bos|>If 5*x + 3 = 13, then x is the number of points that are in the equation. The equation for the number of

Unconditioned samples:
--------------------------------------------------------------------------------
<|bos|>On March 19th harvesters from Mlli made crass through the lateimportant railway operators Charles perfected with thin branches lined with state-owned railways founded during the late 19th and early 20th centuries. After 7—but did Richard make a member, then to make him essentially the most orbitten fans, the first beer. This one depicts a whole explosion stylish way to make the preface to the Start controlling thevagant rail features. In this allagram article I used some sharp tricks from one of the Artists into making woodworking work-shaded something as exciting as with a young player.
The effect of
--------------------------------------------------------------------------------
<|bos|>scientific study showing the relationship between heat Normal heat core heat flux and Earth balanced

Cæ ~ I vitamins and minerals have much more than just the ability to generate energy from heat sorted dye. utilize chemicalWow transistors. this is only possible at one time the real load of red-Extin HLA team, you know the following is easy to find.

p filmsed-shore plasma cyan converting woven eighth formky with ironing from molten iron brown with become metallic in addition to pure iron. crystal is made Campbell soy brewer Puppy, his labman and labman are currently designing a master nanostructured video of electricity machines
--------------------------------------------------------------------------------
<|bos|>NUTDREATE!

Abstract

When people friction a element, their eyes get mL of all wavelength contact. Is this the point of contact for the copper industry? Maybe 10 g diverse.

 preserved protected areas

The only point where these units fit neatly depends on their use. put traces of ore into. Themission was independent. can earn this through a steaks base.

oderms (dry mass); steel, coal and trees (b wafer, soy meat; lumber), nouns materials., WiFi, timber, put nighttime cleaners, dead steel on steel cap; glass; modern glass; double deck; with lead and handlers. only
--------------------------------------------------------------------------------
<|bos|>I want toican historical flavour in the process of World structur. This recent story requires that clienteleamo financially fits this mark making time and so far this tribute can be viewed at any time.

Rivers of the renewable energy and energy industry, as well as any immigration and transportation. Almost each of us in Western Europe runs the economy next season, strictly requiring nothing to live. To do so each day, others are given a chance forilling renewable energy, which is less expensive because the economy takes less energy, eroding the shortcomings of almost every country and getting zeroing in by 10 months here is twice the energy
--------------------------------------------------------------------------------
<|bos|>Lostmonst little garden thrives in extreme temperatures: FOR bark!

Meet your neighbors: FOR bark! pirate friends Come to boil your livingress King

Usually, you're just fighting by telling your neighbors what he is doing and saying what he is doing in the home. He doesn't fly out so fast or loud because his tone language has got downtime. In reality, he doesn't take up the opportunity to call him ease, he makes something noise.

This includes the annual Holy Roman Taundath Run, which is open to smell of knowledge about tomato insignificant plants. Featuring places like Agrageo Elementary School, 3 feet
--------------------------------------------------------------------------------
<|bos|>Related Blog

On the recent transportation of the UK's waterways and fishing fleet, the Infections of Why and Why septic tourism is the harder and rentier method of paranoia and the more complex it is and more eco-friendly is not generally regarded as expensive, but the main difficulties posed by laws and practices that are important to the environment pose issues. Weather is rich, and many environmental laws exist. From the destruction and…

Structural overspunts for economic and environmental shifts, the demands of jobs can be divided directly into two categories. For example, in an economy occupied by a campsite, a culture is investigated to check for
--------------------------------------------------------------------------------
<|bos|>Bob and heal and heal wounds across the inner part of the body.
A graceful wound, wound banded to help heal wounds across the outer parts of the body.
What can I do when a wound healing wound healing wounds wound healing wounds healing wound healing wound healing wound healing wound delicious wound healing wound healing wound healing wound healing wound free wound heal wound healing wound healing wound wound wound healing wound healing wound healing wound wound healing wound wound tissue healing wound healing injury keeps massage heal wounds healing wound insertion wound wound wound healing wound healing wound healing wound wound wound healingSkin wound wound healing Arm supply wound wound healing wound wound healing wound healing wound healing wound healing wound
--------------------------------------------------------------------------------
<|bos|>There are probably no possibly secrets that you will gnarly other people, but you can count advantage these but you will require a little else to know which overarching grayscal players explain them. Read on to find out which players they will do hanging with over calls for you cover. As long as you do not include followers, they may be limited if you have started the review season without a stock.
The process of creating makes it easy to let demons get good things down there usually. centres o

.This Hurst Col metrics, st DP pevish match sets le seedling only upon lithles extinction be a problem if you

================================================================================
BPB Evaluation
================================================================================
train bpb: 1.152185
val bpb: 1.119301

================================================================================
CORE Evaluation
================================================================================
Downloading https://karpathy-public.s3.us-west-2.amazonaws.com/eval_bundle.zip...
Downloaded to $ORX_RUN_DIR/repo/.cache/nanochat/eval_bundle.zip
Placed eval_bundle directory at $ORX_RUN_DIR/repo/.cache/nanochat/eval_bundle
Evaluating: hellaswag_zeroshot (0-shot, type: multiple_choice)...
Evaluating: jeopardy (10-shot, type: language_modeling)...
Evaluating: bigbench_qa_wikidata (10-shot, type: language_modeling)... accuracy: 0.0000 | centered: 0.0000 | time: 0.82s
Evaluating: arc_easy (10-shot, type: multiple_choice)...
Evaluating: arc_challenge (10-shot, type: multiple_choice)...
Evaluating: copa (0-shot, type: multiple_choice)...
Evaluating: commonsense_qa (10-shot, type: multiple_choice)...
Evaluating: piqa (10-shot, type: multiple_choice)...
Evaluating: openbook_qa (0-shot, type: multiple_choice)... accuracy: 0.2500 | centered: 0.0000 | time: 1.00s
Evaluating: lambada_openai (0-shot, type: language_modeling)...
Evaluating: hellaswag (10-shot, type: multiple_choice)...
Evaluating: winograd (0-shot, type: schema)...
Evaluating: winogrande (0-shot, type: schema)... accuracy: 0.5625 | centered: 0.1250 | time: 0.79s
Evaluating: bigbench_dyck_languages (10-shot, type: language_modeling)...
Evaluating: agi_eval_lsat_ar (3-shot, type: multiple_choice)...
Evaluating: bigbench_cs_algorithms (10-shot, type: language_modeling)...
Evaluating: bigbench_operators (10-shot, type: language_modeling)... accuracy: 0.0000 | centered: 0.0000 | time: 1.20s
Evaluating: bigbench_repeat_copy_logic (10-shot, type: language_modeling)...
Evaluating: squad (10-shot, type: language_modeling)...
Evaluating: coqa (0-shot, type: language_modeling)...
Evaluating: boolq (10-shot, type: multiple_choice)...
Evaluating: bigbench_language_identification (10-shot, type: multiple_choice)...

=== Supervised fine-tuning ===
Autodetected device type: mps
2026-08-04 13:48:21,362 - nanochat.common - INFO - Distributed world size: 1
COMPUTE_DTYPE: torch.float32 (auto-detected: no CUDA (CPU/MPS))
WARNING: Flash Attention 3 not available, using PyTorch SDPA fallback. Training will be less efficient.
2026-08-04 13:48:21,363 - nanochat.checkpoint_manager - INFO - No model tag provided, guessing model tag: d6
2026-08-04 13:48:21,363 - nanochat.checkpoint_manager - INFO - Loading model from $ORX_RUN_DIR/repo/.cache/nanochat/base_checkpoints/d6 with step 5000
2026-08-04 13:48:21,554 - nanochat.checkpoint_manager - INFO - Building model with config: {'sequence_len': 512, 'vocab_size': 32768, 'n_layer': 6, 'n_head': 6, 'n_kv_head': 6, 'n_embd': 384, 'window_pattern': 'L'}
Inherited max_seq_len=512 from pretrained checkpoint
Inherited device_batch_size=32 from pretrained checkpoint
Inherited total_batch_size=16384 from pretrained checkpoint
Inherited embedding_lr=0.3 from pretrained checkpoint
Inherited unembedding_lr=0.008 from pretrained checkpoint
Inherited matrix_lr=0.02 from pretrained checkpoint
Tokens / micro-batch / rank: 32 x 512 = 16,384
Tokens / micro-batch: 16,384
Total batch size 16,384 => gradient accumulation steps: 1
Scaling the LR for the AdamW parameters ∝1/√(384/768) = 1.414214
Training mixture: 789,759 rows (MMLU x3, GSM8K x4)
Step 00000 | Validation bpb: 1.0174
step 00001 (0.13%) | loss: 1.817033 | lrm: 1.00 | dt: 2522.10ms | tok/sec: 6,496 | mfu: 0.00 | epoch: 1 | total time: 0.00m
step 00002 (0.20%) | loss: 5.473115 | lrm: 1.00 | dt: 1482.41ms | tok/sec: 11,052 | mfu: 0.00 | epoch: 1 | total time: 0.00m
step 00003 (0.27%) | loss: 6.019664 | lrm: 1.00 | dt: 1488.04ms | tok/sec: 11,010 | mfu: 0.00 | epoch: 1 | total time: 0.00m
step 00004 (0.33%) | loss: 6.121520 | lrm: 1.00 | dt: 1479.17ms | tok/sec: 11,076 | mfu: 0.00 | epoch: 1 | total time: 0.00m
step 00005 (0.40%) | loss: 6.169227 | lrm: 1.00 | dt: 1498.15ms | tok/sec: 10,936 | mfu: 0.00 | epoch: 1 | total time: 0.00m
step 00006 (0.47%) | loss: 6.144121 | lrm: 1.00 | dt: 1489.45ms | tok/sec: 11,000 | mfu: 0.00 | epoch: 1 | total time: 0.00m
step 00007 (0.53%) | loss: 6.049717 | lrm: 1.00 | dt: 1507.93ms | tok/sec: 10,865 | mfu: 0.00 | epoch: 1 | total time: 0.00m
step 00008 (0.60%) | loss: 5.944581 | lrm: 1.00 | dt: 1483.85ms | tok/sec: 11,041 | mfu: 0.00 | epoch: 1 | total time: 0.00m
step 00009 (0.67%) | loss: 5.902769 | lrm: 1.00 | dt: 1490.85ms | tok/sec: 10,989 | mfu: 0.00 | epoch: 1 | total time: 0.00m
step 00010 (0.73%) | loss: 5.852226 | lrm: 1.00 | dt: 1501.87ms | tok/sec: 10,909 | mfu: 0.00 | epoch: 1 | total time: 0.00m
step 00011 (0.80%) | loss: 5.776022 | lrm: 1.00 | dt: 1506.82ms | tok/sec: 10,873 | mfu: 0.00 | epoch: 1 | total time: 0.03m
step 00012 (0.87%) | loss: 5.739937 | lrm: 1.00 | dt: 1500.87ms | tok/sec: 10,916 | mfu: 0.00 | epoch: 1 | total time: 0.05m
step 00013 (0.93%) | loss: 5.657535 | lrm: 1.00 | dt: 1519.51ms | tok/sec: 10,782 | mfu: 0.00 | epoch: 1 | total time: 0.08m
step 00014 (1.00%) | loss: 5.550800 | lrm: 1.00 | dt: 1512.46ms | tok/sec: 10,832 | mfu: 0.00 | epoch: 1 | total time: 0.10m
step 00015 (1.07%) | loss: 5.474723 | lrm: 1.00 | dt: 1509.14ms | tok/sec: 10,856 | mfu: 0.00 | epoch: 1 | total time: 0.13m
step 00016 (1.13%) | loss: 5.417098 | lrm: 1.00 | dt: 1494.45ms | tok/sec: 10,963 | mfu: 0.00 | epoch: 1 | total time: 0.15m
step 00017 (1.20%) | loss: 5.378817 | lrm: 1.00 | dt: 1458.62ms | tok/sec: 11,232 | mfu: 0.00 | epoch: 1 | total time: 0.18m
step 00018 (1.27%) | loss: 5.343087 | lrm: 1.00 | dt: 1519.85ms | tok/sec: 10,780 | mfu: 0.00 | epoch: 1 | total time: 0.20m
step 00019 (1.33%) | loss: 5.375785 | lrm: 1.00 | dt: 1483.97ms | tok/sec: 11,040 | mfu: 0.00 | epoch: 1 | total time: 0.23m
step 00020 (1.40%) | loss: 5.329651 | lrm: 1.00 | dt: 1487.10ms | tok/sec: 11,017 | mfu: 0.00 | epoch: 1 | total time: 0.25m
step 00021 (1.47%) | loss: 5.250349 | lrm: 1.00 | dt: 1553.69ms | tok/sec: 10,545 | mfu: 0.00 | epoch: 1 | total time: 0.28m
step 00022 (1.53%) | loss: 5.141745 | lrm: 1.00 | dt: 1638.61ms | tok/sec: 9,998 | mfu: 0.00 | epoch: 1 | total time: 0.30m
step 00023 (1.60%) | loss: 5.104595 | lrm: 1.00 | dt: 1727.58ms | tok/sec: 9,483 | mfu: 0.00 | epoch: 1 | total time: 0.33m
step 00024 (1.67%) | loss: 5.089206 | lrm: 1.00 | dt: 1705.65ms | tok/sec: 9,605 | mfu: 0.00 | epoch: 1 | total time: 0.36m
step 00025 (1.73%) | loss: 5.034736 | lrm: 1.00 | dt: 1718.92ms | tok/sec: 9,531 | mfu: 0.00 | epoch: 1 | total time: 0.39m
step 00026 (1.80%) | loss: 4.974130 | lrm: 1.00 | dt: 1716.91ms | tok/sec: 9,542 | mfu: 0.00 | epoch: 1 | total time: 0.42m
step 00027 (1.87%) | loss: 4.915598 | lrm: 1.00 | dt: 1683.17ms | tok/sec: 9,733 | mfu: 0.00 | epoch: 1 | total time: 0.45m
step 00028 (1.93%) | loss: 4.878025 | lrm: 1.00 | dt: 1716.27ms | tok/sec: 9,546 | mfu: 0.00 | epoch: 1 | total time: 0.47m
step 00029 (2.00%) | loss: 4.836676 | lrm: 1.00 | dt: 1484.90ms | tok/sec: 11,033 | mfu: 0.00 | epoch: 1 | total time: 0.50m
step 00030 (2.07%) | loss: 4.815304 | lrm: 1.00 | dt: 1618.10ms | tok/sec: 10,125 | mfu: 0.00 | epoch: 1 | total time: 0.53m
step 00031 (2.13%) | loss: 4.794586 | lrm: 1.00 | dt: 1645.86ms | tok/sec: 9,954 | mfu: 0.00 | epoch: 1 | total time: 0.55m
step 00032 (2.20%) | loss: 4.772627 | lrm: 1.00 | dt: 1536.36ms | tok/sec: 10,664 | mfu: 0.00 | epoch: 1 | total time: 0.58m
step 00033 (2.27%) | loss: 4.670185 | lrm: 1.00 | dt: 1529.12ms | tok/sec: 10,714 | mfu: 0.00 | epoch: 1 | total time: 0.60m
step 00034 (2.33%) | loss: 4.689180 | lrm: 1.00 | dt: 1519.70ms | tok/sec: 10,781 | mfu: 0.00 | epoch: 1 | total time: 0.63m
step 00035 (2.40%) | loss: 4.652431 | lrm: 1.00 | dt: 1521.98ms | tok/sec: 10,764 | mfu: 0.00 | epoch: 1 | total time: 0.66m
step 00036 (2.47%) | loss: 4.608195 | lrm: 1.00 | dt: 1540.44ms | tok/sec: 10,635 | mfu: 0.00 | epoch: 1 | total time: 0.68m
step 00037 (2.53%) | loss: 4.559339 | lrm: 1.00 | dt: 1562.52ms | tok/sec: 10,485 | mfu: 0.00 | epoch: 1 | total time: 0.71m
step 00038 (2.60%) | loss: 4.523541 | lrm: 1.00 | dt: 1541.48ms | tok/sec: 10,628 | mfu: 0.00 | epoch: 1 | total time: 0.73m
step 00039 (2.67%) | loss: 4.489213 | lrm: 1.00 | dt: 1547.22ms | tok/sec: 10,589 | mfu: 0.00 | epoch: 1 | total time: 0.76m
step 00040 (2.73%) | loss: 4.453238 | lrm: 1.00 | dt: 1512.21ms | tok/sec: 10,834 | mfu: 0.00 | epoch: 1 | total time: 0.78m
step 00041 (2.80%) | loss: 4.386787 | lrm: 1.00 | dt: 1621.39ms | tok/sec: 10,104 | mfu: 0.00 | epoch: 1 | total time: 0.81m
step 00042 (2.87%) | loss: 4.365549 | lrm: 1.00 | dt: 1530.76ms | tok/sec: 10,703 | mfu: 0.00 | epoch: 1 | total time: 0.84m
step 00043 (2.93%) | loss: 4.349801 | lrm: 1.00 | dt: 1677.61ms | tok/sec: 9,766 | mfu: 0.00 | epoch: 1 | total time: 0.86m
step 00044 (3.00%) | loss: 4.367675 | lrm: 1.00 | dt: 1556.86ms | tok/sec: 10,523 | mfu: 0.00 | epoch: 1 | total time: 0.89m
step 00045 (3.07%) | loss: 4.319615 | lrm: 1.00 | dt: 1524.04ms | tok/sec: 10,750 | mfu: 0.00 | epoch: 1 | total time: 0.92m
step 00046 (3.13%) | loss: 4.229251 | lrm: 1.00 | dt: 1501.02ms | tok/sec: 10,915 | mfu: 0.00 | epoch: 1 | total time: 0.94m
step 00047 (3.20%) | loss: 4.176152 | lrm: 1.00 | dt: 1525.29ms | tok/sec: 10,741 | mfu: 0.00 | epoch: 1 | total time: 0.97m
step 00048 (3.27%) | loss: 4.133491 | lrm: 1.00 | dt: 1518.29ms | tok/sec: 10,791 | mfu: 0.00 | epoch: 1 | total time: 0.99m
step 00049 (3.33%) | loss: 4.129563 | lrm: 1.00 | dt: 1531.97ms | tok/sec: 10,694 | mfu: 0.00 | epoch: 1 | total time: 1.02m
step 00050 (3.40%) | loss: 4.112618 | lrm: 1.00 | dt: 1569.18ms | tok/sec: 10,441 | mfu: 0.00 | epoch: 1 | total time: 1.04m
step 00051 (3.47%) | loss: 4.091719 | lrm: 1.00 | dt: 1577.98ms | tok/sec: 10,382 | mfu: 0.00 | epoch: 1 | total time: 1.07m
step 00052 (3.53%) | loss: 4.147897 | lrm: 1.00 | dt: 1549.35ms | tok/sec: 10,574 | mfu: 0.00 | epoch: 1 | total time: 1.09m
step 00053 (3.60%) | loss: 4.080249 | lrm: 1.00 | dt: 1551.27ms | tok/sec: 10,561 | mfu: 0.00 | epoch: 1 | total time: 1.12m
step 00054 (3.67%) | loss: 4.054996 | lrm: 1.00 | dt: 1603.41ms | tok/sec: 10,218 | mfu: 0.00 | epoch: 1 | total time: 1.15m
step 00055 (3.73%) | loss: 4.045061 | lrm: 1.00 | dt: 1565.02ms | tok/sec: 10,468 | mfu: 0.00 | epoch: 1 | total time: 1.17m
step 00056 (3.80%) | loss: 3.991729 | lrm: 1.00 | dt: 1635.05ms | tok/sec: 10,020 | mfu: 0.00 | epoch: 1 | total time: 1.20m
step 00057 (3.87%) | loss: 3.957480 | lrm: 1.00 | dt: 1689.87ms | tok/sec: 9,695 | mfu: 0.00 | epoch: 1 | total time: 1.23m
step 00058 (3.93%) | loss: 3.918715 | lrm: 1.00 | dt: 1565.74ms | tok/sec: 10,464 | mfu: 0.00 | epoch: 1 | total time: 1.26m
step 00059 (4.00%) | loss: 3.925298 | lrm: 1.00 | dt: 1575.55ms | tok/sec: 10,398 | mfu: 0.00 | epoch: 1 | total time: 1.28m
step 00060 (4.07%) | loss: 3.924037 | lrm: 1.00 | dt: 1568.19ms | tok/sec: 10,447 | mfu: 0.00 | epoch: 1 | total time: 1.31m
step 00061 (4.13%) | loss: 3.922783 | lrm: 1.00 | dt: 1622.70ms | tok/sec: 10,096 | mfu: 0.00 | epoch: 1 | total time: 1.33m
step 00062 (4.20%) | loss: 3.902232 | lrm: 1.00 | dt: 1559.52ms | tok/sec: 10,505 | mfu: 0.00 | epoch: 1 | total time: 1.36m
step 00063 (4.27%) | loss: 3.833745 | lrm: 1.00 | dt: 1590.69ms | tok/sec: 10,299 | mfu: 0.00 | epoch: 1 | total time: 1.39m
step 00064 (4.33%) | loss: 3.811280 | lrm: 1.00 | dt: 1567.29ms | tok/sec: 10,453 | mfu: 0.00 | epoch: 1 | total time: 1.41m
step 00065 (4.40%) | loss: 3.816868 | lrm: 1.00 | dt: 1644.88ms | tok/sec: 9,960 | mfu: 0.00 | epoch: 1 | total time: 1.44m
step 00066 (4.47%) | loss: 3.759453 | lrm: 1.00 | dt: 1572.98ms | tok/sec: 10,415 | mfu: 0.00 | epoch: 1 | total time: 1.47m
step 00067 (4.53%) | loss: 3.749534 | lrm: 1.00 | dt: 1542.83ms | tok/sec: 10,619 | mfu: 0.00 | epoch: 1 | total time: 1.49m
step 00068 (4.60%) | loss: 3.663668 | lrm: 1.00 | dt: 1554.05ms | tok/sec: 10,542 | mfu: 0.00 | epoch: 1 | total time: 1.52m
step 00069 (4.67%) | loss: 3.696500 | lrm: 1.00 | dt: 1577.52ms | tok/sec: 10,385 | mfu: 0.00 | epoch: 1 | total time: 1.54m
step 00070 (4.73%) | loss: 3.676621 | lrm: 1.00 | dt: 1532.14ms | tok/sec: 10,693 | mfu: 0.00 | epoch: 1 | total time: 1.57m
step 00071 (4.80%) | loss: 3.682109 | lrm: 1.00 | dt: 1596.45ms | tok/sec: 10,262 | mfu: 0.00 | epoch: 1 | total time: 1.60m
step 00072 (4.87%) | loss: 3.689631 | lrm: 1.00 | dt: 1558.31ms | tok/sec: 10,513 | mfu: 0.00 | epoch: 1 | total time: 1.62m
step 00073 (4.93%) | loss: 3.634130 | lrm: 1.00 | dt: 1576.68ms | tok/sec: 10,391 | mfu: 0.00 | epoch: 1 | total time: 1.65m
step 00074 (5.00%) | loss: 3.569631 | lrm: 1.00 | dt: 1621.58ms | tok/sec: 10,103 | mfu: 0.00 | epoch: 1 | total time: 1.68m
step 00075 (5.07%) | loss: 3.534563 | lrm: 1.00 | dt: 1596.61ms | tok/sec: 10,261 | mfu: 0.00 | epoch: 1 | total time: 1.70m
step 00076 (5.13%) | loss: 3.537962 | lrm: 1.00 | dt: 1551.73ms | tok/sec: 10,558 | mfu: 0.00 | epoch: 1 | total time: 1.73m
step 00077 (5.20%) | loss: 3.544795 | lrm: 1.00 | dt: 1541.41ms | tok/sec: 10,629 | mfu: 0.00 | epoch: 1 | total time: 1.75m
step 00078 (5.27%) | loss: 3.488586 | lrm: 1.00 | dt: 1536.79ms | tok/sec: 10,661 | mfu: 0.00 | epoch: 1 | total time: 1.78m
step 00079 (5.33%) | loss: 3.524447 | lrm: 1.00 | dt: 1536.21ms | tok/sec: 10,665 | mfu: 0.00 | epoch: 1 | total time: 1.81m
step 00080 (5.40%) | loss: 3.557363 | lrm: 1.00 | dt: 1518.58ms | tok/sec: 10,789 | mfu: 0.00 | epoch: 1 | total time: 1.83m
step 00081 (5.47%) | loss: 3.525948 | lrm: 1.00 | dt: 1566.52ms | tok/sec: 10,458 | mfu: 0.00 | epoch: 1 | total time: 1.86m
step 00082 (5.53%) | loss: 3.473403 | lrm: 1.00 | dt: 1545.83ms | tok/sec: 10,598 | mfu: 0.00 | epoch: 1 | total time: 1.88m
step 00083 (5.60%) | loss: 3.440575 | lrm: 1.00 | dt: 1510.73ms | tok/sec: 10,845 | mfu: 0.00 | epoch: 1 | total time: 1.91m
step 00084 (5.67%) | loss: 3.488826 | lrm: 1.00 | dt: 1559.33ms | tok/sec: 10,507 | mfu: 0.00 | epoch: 1 | total time: 1.93m
step 00085 (5.73%) | loss: 3.473699 | lrm: 1.00 | dt: 1580.65ms | tok/sec: 10,365 | mfu: 0.00 | epoch: 1 | total time: 1.96m
step 00086 (5.80%) | loss: 3.439534 | lrm: 1.00 | dt: 1582.63ms | tok/sec: 10,352 | mfu: 0.00 | epoch: 1 | total time: 1.99m
step 00087 (5.87%) | loss: 3.434460 | lrm: 1.00 | dt: 1574.55ms | tok/sec: 10,405 | mfu: 0.00 | epoch: 1 | total time: 2.01m
step 00088 (5.93%) | loss: 3.398289 | lrm: 1.00 | dt: 1565.71ms | tok/sec: 10,464 | mfu: 0.00 | epoch: 1 | total time: 2.04m
step 00089 (6.00%) | loss: 3.412527 | lrm: 1.00 | dt: 1576.10ms | tok/sec: 10,395 | mfu: 0.00 | epoch: 1 | total time: 2.07m
step 00090 (6.07%) | loss: 3.390738 | lrm: 1.00 | dt: 1563.67ms | tok/sec: 10,477 | mfu: 0.00 | epoch: 1 | total time: 2.09m
step 00091 (6.13%) | loss: 3.337335 | lrm: 1.00 | dt: 1547.75ms | tok/sec: 10,585 | mfu: 0.00 | epoch: 1 | total time: 2.12m
step 00092 (6.20%) | loss: 3.332206 | lrm: 1.00 | dt: 1550.48ms | tok/sec: 10,567 | mfu: 0.00 | epoch: 1 | total time: 2.14m
step 00093 (6.27%) | loss: 3.340259 | lrm: 1.00 | dt: 1580.19ms | tok/sec: 10,368 | mfu: 0.00 | epoch: 1 | total time: 2.17m
step 00094 (6.33%) | loss: 3.353429 | lrm: 1.00 | dt: 1571.74ms | tok/sec: 10,424 | mfu: 0.00 | epoch: 1 | total time: 2.20m
step 00095 (6.40%) | loss: 3.328557 | lrm: 1.00 | dt: 1548.81ms | tok/sec: 10,578 | mfu: 0.00 | epoch: 1 | total time: 2.22m
step 00096 (6.47%) | loss: 3.307132 | lrm: 1.00 | dt: 1562.29ms | tok/sec: 10,487 | mfu: 0.00 | epoch: 1 | total time: 2.25m
step 00097 (6.53%) | loss: 3.306049 | lrm: 1.00 | dt: 1546.09ms | tok/sec: 10,597 | mfu: 0.00 | epoch: 1 | total time: 2.27m
step 00098 (6.60%) | loss: 3.308425 | lrm: 1.00 | dt: 1567.24ms | tok/sec: 10,454 | mfu: 0.00 | epoch: 1 | total time: 2.30m
step 00099 (6.67%) | loss: 3.292488 | lrm: 1.00 | dt: 1583.32ms | tok/sec: 10,347 | mfu: 0.00 | epoch: 1 | total time: 2.33m
step 00100 (6.73%) | loss: 3.258907 | lrm: 1.00 | dt: 1560.74ms | tok/sec: 10,497 | mfu: 0.00 | epoch: 1 | total time: 2.35m
step 00101 (6.80%) | loss: 3.271363 | lrm: 1.00 | dt: 1551.60ms | tok/sec: 10,559 | mfu: 0.00 | epoch: 1 | total time: 2.38m
step 00102 (6.87%) | loss: 3.314409 | lrm: 1.00 | dt: 1538.50ms | tok/sec: 10,649 | mfu: 0.00 | epoch: 1 | total time: 2.40m
step 00103 (6.93%) | loss: 3.314323 | lrm: 1.00 | dt: 1580.13ms | tok/sec: 10,368 | mfu: 0.00 | epoch: 1 | total time: 2.43m
step 00104 (7.00%) | loss: 3.263994 | lrm: 1.00 | dt: 1564.72ms | tok/sec: 10,470 | mfu: 0.00 | epoch: 1 | total time: 2.46m
step 00105 (7.07%) | loss: 3.276106 | lrm: 1.00 | dt: 1555.24ms | tok/sec: 10,534 | mfu: 0.00 | epoch: 1 | total time: 2.48m
step 00106 (7.13%) | loss: 3.299020 | lrm: 1.00 | dt: 1552.89ms | tok/sec: 10,550 | mfu: 0.00 | epoch: 1 | total time: 2.51m
step 00107 (7.20%) | loss: 3.266227 | lrm: 1.00 | dt: 1560.42ms | tok/sec: 10,499 | mfu: 0.00 | epoch: 1 | total time: 2.53m
step 00108 (7.27%) | loss: 3.228880 | lrm: 1.00 | dt: 1555.49ms | tok/sec: 10,533 | mfu: 0.00 | epoch: 1 | total time: 2.56m
step 00109 (7.33%) | loss: 3.218070 | lrm: 1.00 | dt: 1586.85ms | tok/sec: 10,324 | mfu: 0.00 | epoch: 1 | total time: 2.59m
step 00110 (7.40%) | loss: 3.196243 | lrm: 1.00 | dt: 1530.41ms | tok/sec: 10,705 | mfu: 0.00 | epoch: 1 | total time: 2.61m
step 00111 (7.47%) | loss: 3.202894 | lrm: 1.00 | dt: 1556.02ms | tok/sec: 10,529 | mfu: 0.00 | epoch: 1 | total time: 2.64m
step 00112 (7.53%) | loss: 3.143111 | lrm: 1.00 | dt: 1551.72ms | tok/sec: 10,558 | mfu: 0.00 | epoch: 1 | total time: 2.66m
step 00113 (7.60%) | loss: 3.128542 | lrm: 1.00 | dt: 1581.85ms | tok/sec: 10,357 | mfu: 0.00 | epoch: 1 | total time: 2.69m
step 00114 (7.67%) | loss: 3.155872 | lrm: 1.00 | dt: 1538.31ms | tok/sec: 10,650 | mfu: 0.00 | epoch: 1 | total time: 2.71m
step 00115 (7.73%) | loss: 3.143340 | lrm: 1.00 | dt: 1551.63ms | tok/sec: 10,559 | mfu: 0.00 | epoch: 1 | total time: 2.74m
step 00116 (7.80%) | loss: 3.123210 | lrm: 1.00 | dt: 1560.42ms | tok/sec: 10,499 | mfu: 0.00 | epoch: 1 | total time: 2.77m
step 00117 (7.87%) | loss: 3.175933 | lrm: 1.00 | dt: 1524.03ms | tok/sec: 10,750 | mfu: 0.00 | epoch: 1 | total time: 2.79m
step 00118 (7.93%) | loss: 3.194324 | lrm: 1.00 | dt: 1572.28ms | tok/sec: 10,420 | mfu: 0.00 | epoch: 1 | total time: 2.82m
step 00119 (8.00%) | loss: 3.144614 | lrm: 1.00 | dt: 1527.59ms | tok/sec: 10,725 | mfu: 0.00 | epoch: 1 | total time: 2.84m
step 00120 (8.07%) | loss: 3.132595 | lrm: 1.00 | dt: 1537.71ms | tok/sec: 10,654 | mfu: 0.00 | epoch: 1 | total time: 2.87m
step 00121 (8.13%) | loss: 3.117175 | lrm: 1.00 | dt: 1560.45ms | tok/sec: 10,499 | mfu: 0.00 | epoch: 1 | total time: 2.90m
step 00122 (8.20%) | loss: 3.147308 | lrm: 1.00 | dt: 1550.98ms | tok/sec: 10,563 | mfu: 0.00 | epoch: 1 | total time: 2.92m
step 00123 (8.27%) | loss: 3.132386 | lrm: 1.00 | dt: 1544.10ms | tok/sec: 10,610 | mfu: 0.00 | epoch: 1 | total time: 2.95m
step 00124 (8.33%) | loss: 3.188586 | lrm: 1.00 | dt: 1546.61ms | tok/sec: 10,593 | mfu: 0.00 | epoch: 1 | total time: 2.97m
step 00125 (8.40%) | loss: 3.155346 | lrm: 1.00 | dt: 1563.08ms | tok/sec: 10,481 | mfu: 0.00 | epoch: 1 | total time: 3.00m
step 00126 (8.47%) | loss: 3.156122 | lrm: 1.00 | dt: 1547.93ms | tok/sec: 10,584 | mfu: 0.00 | epoch: 1 | total time: 3.02m
step 00127 (8.53%) | loss: 3.155630 | lrm: 1.00 | dt: 1550.22ms | tok/sec: 10,568 | mfu: 0.00 | epoch: 1 | total time: 3.05m
step 00128 (8.60%) | loss: 3.124457 | lrm: 1.00 | dt: 1556.20ms | tok/sec: 10,528 | mfu: 0.00 | epoch: 1 | total time: 3.08m
step 00129 (8.67%) | loss: 3.105679 | lrm: 1.00 | dt: 1551.93ms | tok/sec: 10,557 | mfu: 0.00 | epoch: 1 | total time: 3.10m
step 00130 (8.73%) | loss: 3.122985 | lrm: 1.00 | dt: 1530.02ms | tok/sec: 10,708 | mfu: 0.00 | epoch: 1 | total time: 3.13m
step 00131 (8.80%) | loss: 3.111942 | lrm: 1.00 | dt: 1581.67ms | tok/sec: 10,358 | mfu: 0.00 | epoch: 1 | total time: 3.15m
step 00132 (8.87%) | loss: 3.145066 | lrm: 1.00 | dt: 1569.00ms | tok/sec: 10,442 | mfu: 0.00 | epoch: 1 | total time: 3.18m
step 00133 (8.93%) | loss: 3.131240 | lrm: 1.00 | dt: 1624.40ms | tok/sec: 10,086 | mfu: 0.00 | epoch: 1 | total time: 3.21m
step 00134 (9.00%) | loss: 3.124197 | lrm: 1.00 | dt: 1563.92ms | tok/sec: 10,476 | mfu: 0.00 | epoch: 1 | total time: 3.23m
step 00135 (9.07%) | loss: 3.103801 | lrm: 1.00 | dt: 1567.44ms | tok/sec: 10,452 | mfu: 0.00 | epoch: 1 | total time: 3.26m
step 00136 (9.13%) | loss: 3.087369 | lrm: 1.00 | dt: 1631.58ms | tok/sec: 10,041 | mfu: 0.00 | epoch: 1 | total time: 3.29m
step 00137 (9.20%) | loss: 3.040292 | lrm: 1.00 | dt: 1543.28ms | tok/sec: 10,616 | mfu: 0.00 | epoch: 1 | total time: 3.31m
step 00138 (9.27%) | loss: 3.075346 | lrm: 1.00 | dt: 1537.78ms | tok/sec: 10,654 | mfu: 0.00 | epoch: 1 | total time: 3.34m
step 00139 (9.33%) | loss: 3.059785 | lrm: 1.00 | dt: 1702.35ms | tok/sec: 9,624 | mfu: 0.00 | epoch: 1 | total time: 3.37m
step 00140 (9.40%) | loss: 3.072881 | lrm: 1.00 | dt: 1623.68ms | tok/sec: 10,090 | mfu: 0.00 | epoch: 1 | total time: 3.39m
step 00141 (9.47%) | loss: 3.062308 | lrm: 1.00 | dt: 1575.54ms | tok/sec: 10,398 | mfu: 0.00 | epoch: 1 | total time: 3.42m
step 00142 (9.53%) | loss: 3.119131 | lrm: 1.00 | dt: 1676.85ms | tok/sec: 9,770 | mfu: 0.00 | epoch: 1 | total time: 3.45m
step 00143 (9.60%) | loss: 3.078310 | lrm: 1.00 | dt: 1654.89ms | tok/sec: 9,900 | mfu: 0.00 | epoch: 1 | total time: 3.48m
step 00144 (9.67%) | loss: 3.047946 | lrm: 1.00 | dt: 1700.74ms | tok/sec: 9,633 | mfu: 0.00 | epoch: 1 | total time: 3.50m
step 00145 (9.73%) | loss: 3.048648 | lrm: 1.00 | dt: 1537.44ms | tok/sec: 10,656 | mfu: 0.00 | epoch: 1 | total time: 3.53m
step 00146 (9.80%) | loss: 3.173942 | lrm: 1.00 | dt: 1510.10ms | tok/sec: 10,849 | mfu: 0.00 | epoch: 1 | total time: 3.55m
step 00147 (9.87%) | loss: 3.165353 | lrm: 1.00 | dt: 1504.10ms | tok/sec: 10,892 | mfu: 0.00 | epoch: 1 | total time: 3.58m
step 00148 (9.93%) | loss: 3.179993 | lrm: 1.00 | dt: 1537.23ms | tok/sec: 10,658 | mfu: 0.00 | epoch: 1 | total time: 3.61m
step 00149 (10.00%) | loss: 3.187072 | lrm: 1.00 | dt: 1531.69ms | tok/sec: 10,696 | mfu: 0.00 | epoch: 1 | total time: 3.63m
step 00150 (10.07%) | loss: 3.194895 | lrm: 1.00 | dt: 1549.66ms | tok/sec: 10,572 | mfu: 0.00 | epoch: 1 | total time: 3.66m
step 00151 (10.13%) | loss: 3.178232 | lrm: 1.00 | dt: 1504.41ms | tok/sec: 10,890 | mfu: 0.00 | epoch: 1 | total time: 3.68m
step 00152 (10.20%) | loss: 3.194859 | lrm: 1.00 | dt: 1577.54ms | tok/sec: 10,385 | mfu: 0.00 | epoch: 1 | total time: 3.71m
step 00153 (10.27%) | loss: 3.188694 | lrm: 1.00 | dt: 1517.93ms | tok/sec: 10,793 | mfu: 0.00 | epoch: 1 | total time: 3.73m
step 00154 (10.33%) | loss: 3.194731 | lrm: 1.00 | dt: 1596.38ms | tok/sec: 10,263 | mfu: 0.00 | epoch: 1 | total time: 3.76m
step 00155 (10.40%) | loss: 3.188180 | lrm: 1.00 | dt: 1536.78ms | tok/sec: 10,661 | mfu: 0.00 | epoch: 1 | total time: 3.79m
step 00156 (10.47%) | loss: 3.158290 | lrm: 1.00 | dt: 1580.38ms | tok/sec: 10,367 | mfu: 0.00 | epoch: 1 | total time: 3.81m
step 00157 (10.53%) | loss: 3.151262 | lrm: 1.00 | dt: 1662.22ms | tok/sec: 9,856 | mfu: 0.00 | epoch: 1 | total time: 3.84m
step 00158 (10.60%) | loss: 3.099948 | lrm: 1.00 | dt: 1659.82ms | tok/sec: 9,870 | mfu: 0.00 | epoch: 1 | total time: 3.87m
step 00159 (10.67%) | loss: 3.124621 | lrm: 1.00 | dt: 1650.97ms | tok/sec: 9,923 | mfu: 0.00 | epoch: 1 | total time: 3.89m
step 00160 (10.73%) | loss: 3.152373 | lrm: 1.00 | dt: 1672.33ms | tok/sec: 9,797 | mfu: 0.00 | epoch: 1 | total time: 3.92m
step 00161 (10.80%) | loss: 3.133019 | lrm: 1.00 | dt: 1668.70ms | tok/sec: 9,818 | mfu: 0.00 | epoch: 1 | total time: 3.95m
step 00162 (10.87%) | loss: 3.132608 | lrm: 1.00 | dt: 1531.89ms | tok/sec: 10,695 | mfu: 0.00 | epoch: 1 | total time: 3.98m
step 00163 (10.93%) | loss: 3.135633 | lrm: 1.00 | dt: 1528.50ms | tok/sec: 10,718 | mfu: 0.00 | epoch: 1 | total time: 4.00m
step 00164 (11.00%) | loss: 3.129328 | lrm: 1.00 | dt: 1530.20ms | tok/sec: 10,707 | mfu: 0.00 | epoch: 1 | total time: 4.03m
step 00165 (11.07%) | loss: 3.056259 | lrm: 1.00 | dt: 1527.71ms | tok/sec: 10,724 | mfu: 0.00 | epoch: 1 | total time: 4.05m
step 00166 (11.13%) | loss: 3.012520 | lrm: 1.00 | dt: 1542.11ms | tok/sec: 10,624 | mfu: 0.00 | epoch: 1 | total time: 4.08m
step 00167 (11.20%) | loss: 2.986730 | lrm: 1.00 | dt: 1509.03ms | tok/sec: 10,857 | mfu: 0.00 | epoch: 1 | total time: 4.10m
step 00168 (11.27%) | loss: 2.945832 | lrm: 1.00 | dt: 1532.93ms | tok/sec: 10,688 | mfu: 0.00 | epoch: 1 | total time: 4.13m
step 00169 (11.33%) | loss: 2.924382 | lrm: 1.00 | dt: 1518.23ms | tok/sec: 10,791 | mfu: 0.00 | epoch: 1 | total time: 4.15m
step 00170 (11.40%) | loss: 2.952373 | lrm: 1.00 | dt: 1525.82ms | tok/sec: 10,737 | mfu: 0.00 | epoch: 1 | total time: 4.18m
step 00171 (11.47%) | loss: 2.960551 | lrm: 1.00 | dt: 1532.27ms | tok/sec: 10,692 | mfu: 0.00 | epoch: 1 | total time: 4.21m
step 00172 (11.53%) | loss: 2.971364 | lrm: 1.00 | dt: 1510.04ms | tok/sec: 10,850 | mfu: 0.00 | epoch: 1 | total time: 4.23m
step 00173 (11.60%) | loss: 2.984204 | lrm: 1.00 | dt: 1715.52ms | tok/sec: 9,550 | mfu: 0.00 | epoch: 1 | total time: 4.26m
step 00174 (11.67%) | loss: 2.962117 | lrm: 1.00 | dt: 1796.24ms | tok/sec: 9,121 | mfu: 0.00 | epoch: 1 | total time: 4.29m
step 00175 (11.73%) | loss: 3.015728 | lrm: 1.00 | dt: 1696.14ms | tok/sec: 9,659 | mfu: 0.00 | epoch: 1 | total time: 4.32m
step 00176 (11.80%) | loss: 2.995240 | lrm: 1.00 | dt: 1775.79ms | tok/sec: 9,226 | mfu: 0.00 | epoch: 1 | total time: 4.35m
step 00177 (11.87%) | loss: 2.952391 | lrm: 1.00 | dt: 1721.61ms | tok/sec: 9,516 | mfu: 0.00 | epoch: 1 | total time: 4.38m
step 00178 (11.93%) | loss: 2.980049 | lrm: 1.00 | dt: 1746.61ms | tok/sec: 9,380 | mfu: 0.00 | epoch: 1 | total time: 4.40m
step 00179 (12.00%) | loss: 3.027064 | lrm: 1.00 | dt: 1695.49ms | tok/sec: 9,663 | mfu: 0.00 | epoch: 1 | total time: 4.43m
step 00180 (12.07%) | loss: 2.991604 | lrm: 1.00 | dt: 1516.97ms | tok/sec: 10,800 | mfu: 0.00 | epoch: 1 | total time: 4.46m
step 00181 (12.13%) | loss: 2.974462 | lrm: 1.00 | dt: 1777.03ms | tok/sec: 9,219 | mfu: 0.00 | epoch: 1 | total time: 4.49m
step 00182 (12.20%) | loss: 2.922804 | lrm: 1.00 | dt: 1628.61ms | tok/sec: 10,060 | mfu: 0.00 | epoch: 1 | total time: 4.51m
step 00183 (12.27%) | loss: 2.898653 | lrm: 1.00 | dt: 1578.72ms | tok/sec: 10,378 | mfu: 0.00 | epoch: 1 | total time: 4.54m
step 00184 (12.33%) | loss: 2.914735 | lrm: 1.00 | dt: 1507.99ms | tok/sec: 10,864 | mfu: 0.00 | epoch: 1 | total time: 4.57m
step 00185 (12.40%) | loss: 2.966376 | lrm: 1.00 | dt: 1489.13ms | tok/sec: 11,002 | mfu: 0.00 | epoch: 1 | total time: 4.59m
step 00186 (12.47%) | loss: 2.923186 | lrm: 1.00 | dt: 1517.40ms | tok/sec: 10,797 | mfu: 0.00 | epoch: 1 | total time: 4.62m
step 00187 (12.53%) | loss: 2.905501 | lrm: 1.00 | dt: 1523.44ms | tok/sec: 10,754 | mfu: 0.00 | epoch: 1 | total time: 4.64m
step 00188 (12.60%) | loss: 2.893168 | lrm: 1.00 | dt: 1535.74ms | tok/sec: 10,668 | mfu: 0.00 | epoch: 1 | total time: 4.67m
step 00189 (12.67%) | loss: 2.887417 | lrm: 1.00 | dt: 1516.27ms | tok/sec: 10,805 | mfu: 0.00 | epoch: 1 | total time: 4.69m
step 00190 (12.73%) | loss: 2.906503 | lrm: 1.00 | dt: 1530.34ms | tok/sec: 10,706 | mfu: 0.00 | epoch: 1 | total time: 4.72m
step 00191 (12.80%) | loss: 2.888997 | lrm: 1.00 | dt: 1572.65ms | tok/sec: 10,418 | mfu: 0.00 | epoch: 1 | total time: 4.74m
step 00192 (12.87%) | loss: 2.895453 | lrm: 1.00 | dt: 1628.19ms | tok/sec: 10,062 | mfu: 0.00 | epoch: 1 | total time: 4.77m
step 00193 (12.93%) | loss: 2.920827 | lrm: 1.00 | dt: 1525.19ms | tok/sec: 10,742 | mfu: 0.00 | epoch: 1 | total time: 4.80m
step 00194 (13.00%) | loss: 2.920472 | lrm: 1.00 | dt: 1609.30ms | tok/sec: 10,180 | mfu: 0.00 | epoch: 1 | total time: 4.82m
step 00195 (13.07%) | loss: 2.914865 | lrm: 1.00 | dt: 1717.34ms | tok/sec: 9,540 | mfu: 0.00 | epoch: 1 | total time: 4.85m
step 00196 (13.13%) | loss: 2.902953 | lrm: 1.00 | dt: 1580.80ms | tok/sec: 10,364 | mfu: 0.00 | epoch: 1 | total time: 4.88m
step 00197 (13.20%) | loss: 2.885326 | lrm: 1.00 | dt: 1505.11ms | tok/sec: 10,885 | mfu: 0.00 | epoch: 1 | total time: 4.90m
step 00198 (13.27%) | loss: 2.839747 | lrm: 1.00 | dt: 1519.80ms | tok/sec: 10,780 | mfu: 0.00 | epoch: 1 | total time: 4.93m
step 00199 (13.33%) | loss: 2.805334 | lrm: 1.00 | dt: 1520.21ms | tok/sec: 10,777 | mfu: 0.00 | epoch: 1 | total time: 4.95m
step 00200 (13.40%) | loss: 2.783189 | lrm: 1.00 | dt: 1541.57ms | tok/sec: 10,628 | mfu: 0.00 | epoch: 1 | total time: 4.98m
Step 00200 | Validation bpb: 1.0580
step 00201 (13.47%) | loss: 2.871624 | lrm: 1.00 | dt: 1568.79ms | tok/sec: 10,443 | mfu: 0.00 | epoch: 1 | total time: 5.01m
step 00202 (13.53%) | loss: 2.894222 | lrm: 1.00 | dt: 1545.91ms | tok/sec: 10,598 | mfu: 0.00 | epoch: 1 | total time: 5.03m
step 00203 (13.60%) | loss: 2.936373 | lrm: 1.00 | dt: 1635.87ms | tok/sec: 10,015 | mfu: 0.00 | epoch: 1 | total time: 5.06m
step 00204 (13.67%) | loss: 3.018064 | lrm: 1.00 | dt: 1678.53ms | tok/sec: 9,760 | mfu: 0.00 | epoch: 1 | total time: 5.09m
step 00205 (13.73%) | loss: 3.016622 | lrm: 1.00 | dt: 1818.93ms | tok/sec: 9,007 | mfu: 0.00 | epoch: 1 | total time: 5.12m
step 00206 (13.80%) | loss: 3.044480 | lrm: 1.00 | dt: 1599.65ms | tok/sec: 10,242 | mfu: 0.00 | epoch: 1 | total time: 5.14m
step 00207 (13.87%) | loss: 3.005719 | lrm: 1.00 | dt: 1635.85ms | tok/sec: 10,015 | mfu: 0.00 | epoch: 1 | total time: 5.17m
step 00208 (13.93%) | loss: 3.004216 | lrm: 1.00 | dt: 1683.60ms | tok/sec: 9,731 | mfu: 0.00 | epoch: 1 | total time: 5.20m
step 00209 (14.00%) | loss: 2.974139 | lrm: 1.00 | dt: 1596.57ms | tok/sec: 10,261 | mfu: 0.00 | epoch: 1 | total time: 5.23m
step 00210 (14.07%) | loss: 2.949579 | lrm: 1.00 | dt: 1573.96ms | tok/sec: 10,409 | mfu: 0.00 | epoch: 1 | total time: 5.25m
step 00211 (14.13%) | loss: 2.952695 | lrm: 1.00 | dt: 1580.55ms | tok/sec: 10,366 | mfu: 0.00 | epoch: 1 | total time: 5.28m
step 00212 (14.20%) | loss: 2.934181 | lrm: 1.00 | dt: 1716.10ms | tok/sec: 9,547 | mfu: 0.00 | epoch: 1 | total time: 5.31m
step 00213 (14.27%) | loss: 2.956267 | lrm: 1.00 | dt: 1559.34ms | tok/sec: 10,507 | mfu: 0.00 | epoch: 1 | total time: 5.33m
step 00214 (14.33%) | loss: 2.970707 | lrm: 1.00 | dt: 1599.18ms | tok/sec: 10,245 | mfu: 0.00 | epoch: 1 | total time: 5.36m
step 00215 (14.40%) | loss: 2.959653 | lrm: 1.00 | dt: 1604.03ms | tok/sec: 10,214 | mfu: 0.00 | epoch: 1 | total time: 5.39m
step 00216 (14.47%) | loss: 2.921104 | lrm: 1.00 | dt: 1609.53ms | tok/sec: 10,179 | mfu: 0.00 | epoch: 1 | total time: 5.41m
step 00217 (14.53%) | loss: 2.945268 | lrm: 1.00 | dt: 1587.05ms | tok/sec: 10,323 | mfu: 0.00 | epoch: 1 | total time: 5.44m
step 00218 (14.60%) | loss: 2.889144 | lrm: 1.00 | dt: 1595.44ms | tok/sec: 10,269 | mfu: 0.00 | epoch: 1 | total time: 5.47m
step 00219 (14.67%) | loss: 2.937231 | lrm: 1.00 | dt: 1581.72ms | tok/sec: 10,358 | mfu: 0.00 | epoch: 1 | total time: 5.49m
step 00220 (14.73%) | loss: 3.008325 | lrm: 1.00 | dt: 1609.57ms | tok/sec: 10,179 | mfu: 0.00 | epoch: 1 | total time: 5.52m
step 00221 (14.80%) | loss: 3.010158 | lrm: 1.00 | dt: 1587.22ms | tok/sec: 10,322 | mfu: 0.00 | epoch: 1 | total time: 5.55m
step 00222 (14.87%) | loss: 2.982749 | lrm: 1.00 | dt: 1602.25ms | tok/sec: 10,225 | mfu: 0.00 | epoch: 1 | total time: 5.57m
step 00223 (14.93%) | loss: 2.924951 | lrm: 1.00 | dt: 1758.92ms | tok/sec: 9,314 | mfu: 0.00 | epoch: 1 | total time: 5.60m
step 00224 (15.00%) | loss: 2.917959 | lrm: 1.00 | dt: 1662.11ms | tok/sec: 9,857 | mfu: 0.00 | epoch: 1 | total time: 5.63m
step 00225 (15.07%) | loss: 2.898410 | lrm: 1.00 | dt: 1683.88ms | tok/sec: 9,729 | mfu: 0.00 | epoch: 1 | total time: 5.66m
step 00226 (15.13%) | loss: 2.872910 | lrm: 1.00 | dt: 1571.08ms | tok/sec: 10,428 | mfu: 0.00 | epoch: 1 | total time: 5.68m
step 00227 (15.20%) | loss: 2.875106 | lrm: 1.00 | dt: 1627.30ms | tok/sec: 10,068 | mfu: 0.00 | epoch: 1 | total time: 5.71m
step 00228 (15.27%) | loss: 2.873695 | lrm: 1.00 | dt: 1651.29ms | tok/sec: 9,921 | mfu: 0.00 | epoch: 1 | total time: 5.74m
step 00229 (15.33%) | loss: 2.854641 | lrm: 1.00 | dt: 1578.30ms | tok/sec: 10,380 | mfu: 0.00 | epoch: 1 | total time: 5.77m
step 00230 (15.40%) | loss: 2.837480 | lrm: 1.00 | dt: 1811.77ms | tok/sec: 9,043 | mfu: 0.00 | epoch: 1 | total time: 5.80m
step 00231 (15.47%) | loss: 2.876753 | lrm: 1.00 | dt: 1572.13ms | tok/sec: 10,421 | mfu: 0.00 | epoch: 1 | total time: 5.82m
step 00232 (15.53%) | loss: 2.889380 | lrm: 1.00 | dt: 1591.05ms | tok/sec: 10,297 | mfu: 0.00 | epoch: 1 | total time: 5.85m
step 00233 (15.60%) | loss: 2.894027 | lrm: 1.00 | dt: 1600.80ms | tok/sec: 10,234 | mfu: 0.00 | epoch: 1 | total time: 5.87m
step 00234 (15.67%) | loss: 2.906358 | lrm: 1.00 | dt: 1559.69ms | tok/sec: 10,504 | mfu: 0.00 | epoch: 1 | total time: 5.90m
step 00235 (15.73%) | loss: 2.929235 | lrm: 1.00 | dt: 1573.94ms | tok/sec: 10,409 | mfu: 0.00 | epoch: 1 | total time: 5.93m
step 00236 (15.80%) | loss: 2.836455 | lrm: 1.00 | dt: 1557.96ms | tok/sec: 10,516 | mfu: 0.00 | epoch: 1 | total time: 5.95m
step 00237 (15.87%) | loss: 2.834266 | lrm: 1.00 | dt: 1849.53ms | tok/sec: 8,858 | mfu: 0.00 | epoch: 1 | total time: 5.98m
step 00238 (15.93%) | loss: 2.814363 | lrm: 1.00 | dt: 1740.23ms | tok/sec: 9,414 | mfu: 0.00 | epoch: 1 | total time: 6.01m
step 00239 (16.00%) | loss: 2.811871 | lrm: 1.00 | dt: 1696.42ms | tok/sec: 9,657 | mfu: 0.00 | epoch: 1 | total time: 6.04m
step 00240 (16.07%) | loss: 2.811750 | lrm: 1.00 | dt: 1747.97ms | tok/sec: 9,373 | mfu: 0.00 | epoch: 1 | total time: 6.07m
step 00241 (16.13%) | loss: 2.821767 | lrm: 1.00 | dt: 1721.13ms | tok/sec: 9,519 | mfu: 0.00 | epoch: 1 | total time: 6.10m
step 00242 (16.20%) | loss: 2.812065 | lrm: 1.00 | dt: 1841.87ms | tok/sec: 8,895 | mfu: 0.00 | epoch: 1 | total time: 6.13m
step 00243 (16.27%) | loss: 2.790957 | lrm: 1.00 | dt: 1667.71ms | tok/sec: 9,824 | mfu: 0.00 | epoch: 1 | total time: 6.16m
step 00244 (16.33%) | loss: 2.781618 | lrm: 1.00 | dt: 1636.88ms | tok/sec: 10,009 | mfu: 0.00 | epoch: 1 | total time: 6.18m
step 00245 (16.40%) | loss: 2.805697 | lrm: 1.00 | dt: 1630.34ms | tok/sec: 10,049 | mfu: 0.00 | epoch: 1 | total time: 6.21m
step 00246 (16.47%) | loss: 2.870517 | lrm: 1.00 | dt: 1674.14ms | tok/sec: 9,786 | mfu: 0.00 | epoch: 1 | total time: 6.24m
step 00247 (16.53%) | loss: 2.904862 | lrm: 1.00 | dt: 1669.95ms | tok/sec: 9,811 | mfu: 0.00 | epoch: 1 | total time: 6.27m
step 00248 (16.60%) | loss: 2.921827 | lrm: 1.00 | dt: 1595.26ms | tok/sec: 10,270 | mfu: 0.00 | epoch: 1 | total time: 6.29m
step 00249 (16.67%) | loss: 2.958981 | lrm: 1.00 | dt: 1620.79ms | tok/sec: 10,108 | mfu: 0.00 | epoch: 1 | total time: 6.32m
step 00250 (16.73%) | loss: 2.917663 | lrm: 1.00 | dt: 1613.11ms | tok/sec: 10,156 | mfu: 0.00 | epoch: 1 | total time: 6.35m
step 00251 (16.80%) | loss: 2.911009 | lrm: 1.00 | dt: 1619.25ms | tok/sec: 10,118 | mfu: 0.00 | epoch: 1 | total time: 6.37m
step 00252 (16.87%) | loss: 2.894506 | lrm: 1.00 | dt: 1660.83ms | tok/sec: 9,864 | mfu: 0.00 | epoch: 1 | total time: 6.40m
step 00253 (16.93%) | loss: 2.963329 | lrm: 1.00 | dt: 1618.89ms | tok/sec: 10,120 | mfu: 0.00 | epoch: 1 | total time: 6.43m
step 00254 (17.00%) | loss: 2.956710 | lrm: 1.00 | dt: 1653.41ms | tok/sec: 9,909 | mfu: 0.00 | epoch: 1 | total time: 6.46m
step 00255 (17.07%) | loss: 2.946617 | lrm: 1.00 | dt: 1581.82ms | tok/sec: 10,357 | mfu: 0.00 | epoch: 1 | total time: 6.48m
step 00256 (17.13%) | loss: 2.958855 | lrm: 1.00 | dt: 1607.79ms | tok/sec: 10,190 | mfu: 0.00 | epoch: 1 | total time: 6.51m
step 00257 (17.20%) | loss: 2.911668 | lrm: 1.00 | dt: 1693.01ms | tok/sec: 9,677 | mfu: 0.00 | epoch: 1 | total time: 6.54m
step 00258 (17.27%) | loss: 2.915556 | lrm: 1.00 | dt: 1644.46ms | tok/sec: 9,963 | mfu: 0.00 | epoch: 1 | total time: 6.57m
step 00259 (17.33%) | loss: 2.906665 | lrm: 1.00 | dt: 1624.39ms | tok/sec: 10,086 | mfu: 0.00 | epoch: 1 | total time: 6.59m
step 00260 (17.40%) | loss: 2.898575 | lrm: 1.00 | dt: 1644.64ms | tok/sec: 9,962 | mfu: 0.00 | epoch: 1 | total time: 6.62m
step 00261 (17.47%) | loss: 2.923086 | lrm: 1.00 | dt: 1589.95ms | tok/sec: 10,304 | mfu: 0.00 | epoch: 1 | total time: 6.65m
step 00262 (17.53%) | loss: 2.950778 | lrm: 1.00 | dt: 1641.22ms | tok/sec: 9,982 | mfu: 0.00 | epoch: 1 | total time: 6.67m
step 00263 (17.60%) | loss: 2.958178 | lrm: 1.00 | dt: 1646.39ms | tok/sec: 9,951 | mfu: 0.00 | epoch: 1 | total time: 6.70m
step 00264 (17.67%) | loss: 2.972020 | lrm: 1.00 | dt: 1648.27ms | tok/sec: 9,940 | mfu: 0.00 | epoch: 1 | total time: 6.73m
step 00265 (17.73%) | loss: 2.971793 | lrm: 1.00 | dt: 1528.79ms | tok/sec: 10,717 | mfu: 0.00 | epoch: 1 | total time: 6.75m
step 00266 (17.80%) | loss: 2.956003 | lrm: 1.00 | dt: 1553.70ms | tok/sec: 10,545 | mfu: 0.00 | epoch: 1 | total time: 6.78m
step 00267 (17.87%) | loss: 2.977153 | lrm: 1.00 | dt: 1486.06ms | tok/sec: 11,025 | mfu: 0.00 | epoch: 1 | total time: 6.81m
step 00268 (17.93%) | loss: 2.943753 | lrm: 1.00 | dt: 1524.58ms | tok/sec: 10,746 | mfu: 0.00 | epoch: 1 | total time: 6.83m
step 00269 (18.00%) | loss: 2.945521 | lrm: 1.00 | dt: 1589.73ms | tok/sec: 10,306 | mfu: 0.00 | epoch: 1 | total time: 6.86m
step 00270 (18.07%) | loss: 2.881599 | lrm: 1.00 | dt: 1580.44ms | tok/sec: 10,366 | mfu: 0.00 | epoch: 1 | total time: 6.88m
step 00271 (18.13%) | loss: 2.861891 | lrm: 1.00 | dt: 1563.53ms | tok/sec: 10,478 | mfu: 0.00 | epoch: 1 | total time: 6.91m
step 00272 (18.20%) | loss: 2.833660 | lrm: 1.00 | dt: 1531.75ms | tok/sec: 10,696 | mfu: 0.00 | epoch: 1 | total time: 6.94m
step 00273 (18.27%) | loss: 2.860089 | lrm: 1.00 | dt: 1529.40ms | tok/sec: 10,712 | mfu: 0.00 | epoch: 1 | total time: 6.96m
step 00274 (18.33%) | loss: 2.895677 | lrm: 1.00 | dt: 1520.29ms | tok/sec: 10,776 | mfu: 0.00 | epoch: 1 | total time: 6.99m
step 00275 (18.40%) | loss: 2.886650 | lrm: 1.00 | dt: 1557.98ms | tok/sec: 10,516 | mfu: 0.00 | epoch: 1 | total time: 7.01m
step 00276 (18.47%) | loss: 2.875699 | lrm: 1.00 | dt: 1557.46ms | tok/sec: 10,519 | mfu: 0.00 | epoch: 1 | total time: 7.04m
step 00277 (18.53%) | loss: 2.813724 | lrm: 1.00 | dt: 1555.33ms | tok/sec: 10,534 | mfu: 0.00 | epoch: 1 | total time: 7.06m
step 00278 (18.60%) | loss: 2.829065 | lrm: 1.00 | dt: 1533.74ms | tok/sec: 10,682 | mfu: 0.00 | epoch: 1 | total time: 7.09m
step 00279 (18.67%) | loss: 2.857910 | lrm: 1.00 | dt: 1583.44ms | tok/sec: 10,347 | mfu: 0.00 | epoch: 1 | total time: 7.12m
step 00280 (18.73%) | loss: 2.846124 | lrm: 1.00 | dt: 1564.18ms | tok/sec: 10,474 | mfu: 0.00 | epoch: 1 | total time: 7.14m
step 00281 (18.80%) | loss: 2.799914 | lrm: 1.00 | dt: 1606.73ms | tok/sec: 10,197 | mfu: 0.00 | epoch: 1 | total time: 7.17m
step 00282 (18.87%) | loss: 2.818476 | lrm: 1.00 | dt: 1549.10ms | tok/sec: 10,576 | mfu: 0.00 | epoch: 1 | total time: 7.19m
step 00283 (18.93%) | loss: 2.836727 | lrm: 1.00 | dt: 1521.86ms | tok/sec: 10,765 | mfu: 0.00 | epoch: 1 | total time: 7.22m
step 00284 (19.00%) | loss: 2.826883 | lrm: 1.00 | dt: 1519.31ms | tok/sec: 10,783 | mfu: 0.00 | epoch: 1 | total time: 7.25m
step 00285 (19.07%) | loss: 2.800353 | lrm: 1.00 | dt: 1497.82ms | tok/sec: 10,938 | mfu: 0.00 | epoch: 1 | total time: 7.27m
step 00286 (19.13%) | loss: 2.790878 | lrm: 1.00 | dt: 1534.74ms | tok/sec: 10,675 | mfu: 0.00 | epoch: 1 | total time: 7.30m
step 00287 (19.20%) | loss: 2.814557 | lrm: 1.00 | dt: 1552.67ms | tok/sec: 10,552 | mfu: 0.00 | epoch: 1 | total time: 7.32m
step 00288 (19.27%) | loss: 2.835043 | lrm: 1.00 | dt: 1620.90ms | tok/sec: 10,107 | mfu: 0.00 | epoch: 1 | total time: 7.35m
step 00289 (19.33%) | loss: 2.794027 | lrm: 1.00 | dt: 1507.06ms | tok/sec: 10,871 | mfu: 0.00 | epoch: 1 | total time: 7.37m
step 00290 (19.40%) | loss: 2.762979 | lrm: 1.00 | dt: 1526.03ms | tok/sec: 10,736 | mfu: 0.00 | epoch: 1 | total time: 7.40m
step 00291 (19.47%) | loss: 2.790389 | lrm: 1.00 | dt: 1530.58ms | tok/sec: 10,704 | mfu: 0.00 | epoch: 1 | total time: 7.42m
step 00292 (19.53%) | loss: 2.791335 | lrm: 1.00 | dt: 1519.55ms | tok/sec: 10,782 | mfu: 0.00 | epoch: 1 | total time: 7.45m
step 00293 (19.60%) | loss: 2.788002 | lrm: 1.00 | dt: 1547.73ms | tok/sec: 10,585 | mfu: 0.00 | epoch: 1 | total time: 7.48m
step 00294 (19.67%) | loss: 2.783547 | lrm: 1.00 | dt: 1614.02ms | tok/sec: 10,151 | mfu: 0.00 | epoch: 1 | total time: 7.50m
step 00295 (19.73%) | loss: 2.737591 | lrm: 1.00 | dt: 1560.05ms | tok/sec: 10,502 | mfu: 0.00 | epoch: 1 | total time: 7.53m
step 00296 (19.80%) | loss: 2.736220 | lrm: 1.00 | dt: 1532.33ms | tok/sec: 10,692 | mfu: 0.00 | epoch: 1 | total time: 7.55m
step 00297 (19.87%) | loss: 2.735536 | lrm: 1.00 | dt: 1525.07ms | tok/sec: 10,743 | mfu: 0.00 | epoch: 1 | total time: 7.58m
step 00298 (19.93%) | loss: 2.782258 | lrm: 1.00 | dt: 1637.92ms | tok/sec: 10,002 | mfu: 0.00 | epoch: 1 | total time: 7.61m
step 00299 (20.00%) | loss: 2.736696 | lrm: 1.00 | dt: 1588.08ms | tok/sec: 10,316 | mfu: 0.00 | epoch: 1 | total time: 7.63m
step 00300 (20.07%) | loss: 2.792406 | lrm: 1.00 | dt: 1554.12ms | tok/sec: 10,542 | mfu: 0.00 | epoch: 1 | total time: 7.66m
step 00301 (20.13%) | loss: 2.767717 | lrm: 1.00 | dt: 1529.12ms | tok/sec: 10,714 | mfu: 0.00 | epoch: 1 | total time: 7.68m
step 00302 (20.20%) | loss: 2.743280 | lrm: 1.00 | dt: 1639.20ms | tok/sec: 9,995 | mfu: 0.00 | epoch: 1 | total time: 7.71m
step 00303 (20.27%) | loss: 2.749701 | lrm: 1.00 | dt: 1514.33ms | tok/sec: 10,819 | mfu: 0.00 | epoch: 1 | total time: 7.74m
step 00304 (20.33%) | loss: 2.759765 | lrm: 1.00 | dt: 1548.39ms | tok/sec: 10,581 | mfu: 0.00 | epoch: 1 | total time: 7.76m
step 00305 (20.40%) | loss: 2.784703 | lrm: 1.00 | dt: 1544.75ms | tok/sec: 10,606 | mfu: 0.00 | epoch: 1 | total time: 7.79m
step 00306 (20.47%) | loss: 2.841440 | lrm: 1.00 | dt: 1612.04ms | tok/sec: 10,163 | mfu: 0.00 | epoch: 1 | total time: 7.82m
step 00307 (20.53%) | loss: 2.827801 | lrm: 1.00 | dt: 1550.13ms | tok/sec: 10,569 | mfu: 0.00 | epoch: 1 | total time: 7.84m
step 00308 (20.60%) | loss: 2.865152 | lrm: 1.00 | dt: 1522.47ms | tok/sec: 10,761 | mfu: 0.00 | epoch: 1 | total time: 7.87m
step 00309 (20.67%) | loss: 2.977392 | lrm: 1.00 | dt: 1509.30ms | tok/sec: 10,855 | mfu: 0.00 | epoch: 1 | total time: 7.89m
step 00310 (20.73%) | loss: 2.969238 | lrm: 1.00 | dt: 1506.06ms | tok/sec: 10,878 | mfu: 0.00 | epoch: 1 | total time: 7.92m
step 00311 (20.80%) | loss: 2.959256 | lrm: 1.00 | dt: 1520.78ms | tok/sec: 10,773 | mfu: 0.00 | epoch: 1 | total time: 7.94m
step 00312 (20.87%) | loss: 2.976041 | lrm: 1.00 | dt: 1554.20ms | tok/sec: 10,541 | mfu: 0.00 | epoch: 1 | total time: 7.97m
step 00313 (20.93%) | loss: 2.981821 | lrm: 1.00 | dt: 1543.75ms | tok/sec: 10,613 | mfu: 0.00 | epoch: 1 | total time: 7.99m
step 00314 (21.00%) | loss: 2.943648 | lrm: 1.00 | dt: 1507.10ms | tok/sec: 10,871 | mfu: 0.00 | epoch: 1 | total time: 8.02m
step 00315 (21.07%) | loss: 3.005204 | lrm: 1.00 | dt: 1503.07ms | tok/sec: 10,900 | mfu: 0.00 | epoch: 1 | total time: 8.04m
step 00316 (21.13%) | loss: 3.009540 | lrm: 1.00 | dt: 1521.00ms | tok/sec: 10,771 | mfu: 0.00 | epoch: 1 | total time: 8.07m
step 00317 (21.20%) | loss: 2.966867 | lrm: 1.00 | dt: 1512.53ms | tok/sec: 10,832 | mfu: 0.00 | epoch: 1 | total time: 8.09m
step 00318 (21.27%) | loss: 3.004019 | lrm: 1.00 | dt: 1527.23ms | tok/sec: 10,727 | mfu: 0.00 | epoch: 1 | total time: 8.12m
step 00319 (21.33%) | loss: 2.985806 | lrm: 1.00 | dt: 1520.04ms | tok/sec: 10,778 | mfu: 0.00 | epoch: 1 | total time: 8.15m
step 00320 (21.40%) | loss: 2.969923 | lrm: 1.00 | dt: 1521.94ms | tok/sec: 10,765 | mfu: 0.00 | epoch: 1 | total time: 8.17m
step 00321 (21.47%) | loss: 2.989585 | lrm: 1.00 | dt: 1513.83ms | tok/sec: 10,822 | mfu: 0.00 | epoch: 1 | total time: 8.20m
step 00322 (21.53%) | loss: 2.993615 | lrm: 1.00 | dt: 1508.15ms | tok/sec: 10,863 | mfu: 0.00 | epoch: 1 | total time: 8.22m
step 00323 (21.60%) | loss: 3.023099 | lrm: 1.00 | dt: 1529.14ms | tok/sec: 10,714 | mfu: 0.00 | epoch: 1 | total time: 8.25m
step 00324 (21.67%) | loss: 3.013757 | lrm: 1.00 | dt: 1484.30ms | tok/sec: 11,038 | mfu: 0.00 | epoch: 1 | total time: 8.27m
step 00325 (21.73%) | loss: 3.067621 | lrm: 1.00 | dt: 1525.70ms | tok/sec: 10,738 | mfu: 0.00 | epoch: 1 | total time: 8.30m
step 00326 (21.80%) | loss: 3.038048 | lrm: 1.00 | dt: 1509.37ms | tok/sec: 10,854 | mfu: 0.00 | epoch: 1 | total time: 8.32m
step 00327 (21.87%) | loss: 3.072232 | lrm: 1.00 | dt: 1521.64ms | tok/sec: 10,767 | mfu: 0.00 | epoch: 1 | total time: 8.35m
step 00328 (21.93%) | loss: 3.033771 | lrm: 1.00 | dt: 1504.02ms | tok/sec: 10,893 | mfu: 0.00 | epoch: 1 | total time: 8.37m
step 00329 (22.00%) | loss: 2.974915 | lrm: 1.00 | dt: 1508.30ms | tok/sec: 10,862 | mfu: 0.00 | epoch: 1 | total time: 8.40m
step 00330 (22.07%) | loss: 2.993157 | lrm: 1.00 | dt: 1507.15ms | tok/sec: 10,870 | mfu: 0.00 | epoch: 1 | total time: 8.42m
step 00331 (22.13%) | loss: 2.952978 | lrm: 1.00 | dt: 1531.13ms | tok/sec: 10,700 | mfu: 0.00 | epoch: 1 | total time: 8.45m
step 00332 (22.20%) | loss: 2.963865 | lrm: 1.00 | dt: 1501.57ms | tok/sec: 10,911 | mfu: 0.00 | epoch: 1 | total time: 8.47m
step 00333 (22.27%) | loss: 2.955783 | lrm: 1.00 | dt: 1531.95ms | tok/sec: 10,694 | mfu: 0.00 | epoch: 1 | total time: 8.50m
step 00334 (22.33%) | loss: 2.947417 | lrm: 1.00 | dt: 1510.44ms | tok/sec: 10,847 | mfu: 0.00 | epoch: 1 | total time: 8.52m
step 00335 (22.40%) | loss: 2.930746 | lrm: 1.00 | dt: 1499.34ms | tok/sec: 10,927 | mfu: 0.00 | epoch: 1 | total time: 8.55m
step 00336 (22.47%) | loss: 2.917858 | lrm: 1.00 | dt: 1503.87ms | tok/sec: 10,894 | mfu: 0.00 | epoch: 1 | total time: 8.57m
step 00337 (22.53%) | loss: 2.933221 | lrm: 1.00 | dt: 1518.31ms | tok/sec: 10,790 | mfu: 0.00 | epoch: 1 | total time: 8.60m
step 00338 (22.60%) | loss: 2.920569 | lrm: 1.00 | dt: 1522.84ms | tok/sec: 10,758 | mfu: 0.00 | epoch: 1 | total time: 8.62m
step 00339 (22.67%) | loss: 2.914428 | lrm: 1.00 | dt: 1511.15ms | tok/sec: 10,842 | mfu: 0.00 | epoch: 1 | total time: 8.65m
step 00340 (22.73%) | loss: 2.886931 | lrm: 1.00 | dt: 1482.52ms | tok/sec: 11,051 | mfu: 0.00 | epoch: 1 | total time: 8.67m
step 00341 (22.80%) | loss: 2.880877 | lrm: 1.00 | dt: 1497.42ms | tok/sec: 10,941 | mfu: 0.00 | epoch: 1 | total time: 8.70m
step 00342 (22.87%) | loss: 2.898653 | lrm: 1.00 | dt: 1510.18ms | tok/sec: 10,849 | mfu: 0.00 | epoch: 1 | total time: 8.72m
step 00343 (22.93%) | loss: 2.925420 | lrm: 1.00 | dt: 1495.75ms | tok/sec: 10,953 | mfu: 0.00 | epoch: 1 | total time: 8.75m
step 00344 (23.00%) | loss: 2.915956 | lrm: 1.00 | dt: 1493.72ms | tok/sec: 10,968 | mfu: 0.00 | epoch: 1 | total time: 8.77m
step 00345 (23.07%) | loss: 2.983470 | lrm: 1.00 | dt: 1505.32ms | tok/sec: 10,884 | mfu: 0.00 | epoch: 1 | total time: 8.80m
step 00346 (23.13%) | loss: 2.975642 | lrm: 1.00 | dt: 1495.51ms | tok/sec: 10,955 | mfu: 0.00 | epoch: 1 | total time: 8.82m
step 00347 (23.20%) | loss: 2.951365 | lrm: 1.00 | dt: 1513.81ms | tok/sec: 10,823 | mfu: 0.00 | epoch: 1 | total time: 8.85m
step 00348 (23.27%) | loss: 2.868174 | lrm: 1.00 | dt: 1495.89ms | tok/sec: 10,952 | mfu: 0.00 | epoch: 1 | total time: 8.87m
step 00349 (23.33%) | loss: 2.860149 | lrm: 1.00 | dt: 1504.16ms | tok/sec: 10,892 | mfu: 0.00 | epoch: 1 | total time: 8.90m
step 00350 (23.40%) | loss: 2.813166 | lrm: 1.00 | dt: 1492.41ms | tok/sec: 10,978 | mfu: 0.00 | epoch: 1 | total time: 8.92m
step 00351 (23.47%) | loss: 2.831534 | lrm: 1.00 | dt: 1502.47ms | tok/sec: 10,904 | mfu: 0.00 | epoch: 1 | total time: 8.95m
step 00352 (23.53%) | loss: 2.794870 | lrm: 1.00 | dt: 1508.16ms | tok/sec: 10,863 | mfu: 0.00 | epoch: 1 | total time: 8.98m
step 00353 (23.60%) | loss: 2.783435 | lrm: 1.00 | dt: 1513.60ms | tok/sec: 10,824 | mfu: 0.00 | epoch: 1 | total time: 9.00m
step 00354 (23.67%) | loss: 2.791307 | lrm: 1.00 | dt: 1516.32ms | tok/sec: 10,805 | mfu: 0.00 | epoch: 1 | total time: 9.03m
step 00355 (23.73%) | loss: 2.797779 | lrm: 1.00 | dt: 1524.90ms | tok/sec: 10,744 | mfu: 0.00 | epoch: 1 | total time: 9.05m
step 00356 (23.80%) | loss: 2.823055 | lrm: 1.00 | dt: 1521.98ms | tok/sec: 10,764 | mfu: 0.00 | epoch: 1 | total time: 9.08m
step 00357 (23.87%) | loss: 2.796249 | lrm: 1.00 | dt: 1509.46ms | tok/sec: 10,854 | mfu: 0.00 | epoch: 1 | total time: 9.10m
step 00358 (23.93%) | loss: 2.778057 | lrm: 1.00 | dt: 1483.20ms | tok/sec: 11,046 | mfu: 0.00 | epoch: 1 | total time: 9.13m
step 00359 (24.00%) | loss: 2.821656 | lrm: 1.00 | dt: 1492.10ms | tok/sec: 10,980 | mfu: 0.00 | epoch: 1 | total time: 9.15m
step 00360 (24.07%) | loss: 2.764969 | lrm: 1.00 | dt: 1484.09ms | tok/sec: 11,039 | mfu: 0.00 | epoch: 1 | total time: 9.18m
step 00361 (24.13%) | loss: 2.796293 | lrm: 1.00 | dt: 1505.85ms | tok/sec: 10,880 | mfu: 0.00 | epoch: 1 | total time: 9.20m
step 00362 (24.20%) | loss: 2.814234 | lrm: 1.00 | dt: 1495.80ms | tok/sec: 10,953 | mfu: 0.00 | epoch: 1 | total time: 9.23m
step 00363 (24.27%) | loss: 2.863612 | lrm: 1.00 | dt: 1508.32ms | tok/sec: 10,862 | mfu: 0.00 | epoch: 1 | total time: 9.25m
step 00364 (24.33%) | loss: 2.874921 | lrm: 1.00 | dt: 1506.02ms | tok/sec: 10,879 | mfu: 0.00 | epoch: 1 | total time: 9.28m
step 00365 (24.40%) | loss: 2.881324 | lrm: 1.00 | dt: 1511.25ms | tok/sec: 10,841 | mfu: 0.00 | epoch: 1 | total time: 9.30m
step 00366 (24.47%) | loss: 2.839450 | lrm: 1.00 | dt: 1528.91ms | tok/sec: 10,716 | mfu: 0.00 | epoch: 1 | total time: 9.33m
step 00367 (24.53%) | loss: 2.833173 | lrm: 1.00 | dt: 1525.24ms | tok/sec: 10,741 | mfu: 0.00 | epoch: 1 | total time: 9.35m
step 00368 (24.60%) | loss: 2.833604 | lrm: 1.00 | dt: 1512.68ms | tok/sec: 10,831 | mfu: 0.00 | epoch: 1 | total time: 9.38m
step 00369 (24.67%) | loss: 2.834391 | lrm: 1.00 | dt: 1515.76ms | tok/sec: 10,809 | mfu: 0.00 | epoch: 1 | total time: 9.40m
step 00370 (24.73%) | loss: 2.778362 | lrm: 1.00 | dt: 1505.48ms | tok/sec: 10,882 | mfu: 0.00 | epoch: 1 | total time: 9.43m
step 00371 (24.80%) | loss: 2.847523 | lrm: 1.00 | dt: 1496.19ms | tok/sec: 10,950 | mfu: 0.00 | epoch: 1 | total time: 9.45m
step 00372 (24.87%) | loss: 2.828562 | lrm: 1.00 | dt: 1514.08ms | tok/sec: 10,821 | mfu: 0.00 | epoch: 1 | total time: 9.48m
step 00373 (24.93%) | loss: 2.824599 | lrm: 1.00 | dt: 1498.99ms | tok/sec: 10,930 | mfu: 0.00 | epoch: 1 | total time: 9.50m
step 00374 (25.00%) | loss: 2.860367 | lrm: 1.00 | dt: 1515.88ms | tok/sec: 10,808 | mfu: 0.00 | epoch: 1 | total time: 9.53m
step 00375 (25.07%) | loss: 2.820628 | lrm: 1.00 | dt: 1510.50ms | tok/sec: 10,846 | mfu: 0.00 | epoch: 1 | total time: 9.55m
step 00376 (25.13%) | loss: 2.802159 | lrm: 1.00 | dt: 1501.59ms | tok/sec: 10,911 | mfu: 0.00 | epoch: 1 | total time: 9.58m
step 00377 (25.20%) | loss: 2.837138 | lrm: 1.00 | dt: 1519.32ms | tok/sec: 10,783 | mfu: 0.00 | epoch: 1 | total time: 9.60m
step 00378 (25.27%) | loss: 2.819165 | lrm: 1.00 | dt: 1511.16ms | tok/sec: 10,841 | mfu: 0.00 | epoch: 1 | total time: 9.63m
step 00379 (25.33%) | loss: 2.800532 | lrm: 1.00 | dt: 1531.62ms | tok/sec: 10,697 | mfu: 0.00 | epoch: 1 | total time: 9.65m
step 00380 (25.40%) | loss: 2.804149 | lrm: 1.00 | dt: 1503.69ms | tok/sec: 10,895 | mfu: 0.00 | epoch: 1 | total time: 9.68m
step 00381 (25.47%) | loss: 2.803761 | lrm: 1.00 | dt: 1523.41ms | tok/sec: 10,754 | mfu: 0.00 | epoch: 1 | total time: 9.70m
step 00382 (25.53%) | loss: 2.826453 | lrm: 1.00 | dt: 1516.88ms | tok/sec: 10,801 | mfu: 0.00 | epoch: 1 | total time: 9.73m
step 00383 (25.60%) | loss: 2.805169 | lrm: 1.00 | dt: 1497.46ms | tok/sec: 10,941 | mfu: 0.00 | epoch: 1 | total time: 9.76m
step 00384 (25.67%) | loss: 2.808560 | lrm: 1.00 | dt: 1516.16ms | tok/sec: 10,806 | mfu: 0.00 | epoch: 1 | total time: 9.78m
step 00385 (25.73%) | loss: 2.817573 | lrm: 1.00 | dt: 1517.55ms | tok/sec: 10,796 | mfu: 0.00 | epoch: 1 | total time: 9.81m
step 00386 (25.80%) | loss: 2.808762 | lrm: 1.00 | dt: 1514.84ms | tok/sec: 10,815 | mfu: 0.00 | epoch: 1 | total time: 9.83m
step 00387 (25.87%) | loss: 2.763828 | lrm: 1.00 | dt: 1510.31ms | tok/sec: 10,848 | mfu: 0.00 | epoch: 1 | total time: 9.86m
step 00388 (25.93%) | loss: 2.813977 | lrm: 1.00 | dt: 1517.47ms | tok/sec: 10,796 | mfu: 0.00 | epoch: 1 | total time: 9.88m
step 00389 (26.00%) | loss: 2.823822 | lrm: 1.00 | dt: 1490.10ms | tok/sec: 10,995 | mfu: 0.00 | epoch: 1 | total time: 9.91m
step 00390 (26.07%) | loss: 2.775746 | lrm: 1.00 | dt: 1523.21ms | tok/sec: 10,756 | mfu: 0.00 | epoch: 1 | total time: 9.93m
step 00391 (26.13%) | loss: 2.764919 | lrm: 1.00 | dt: 1491.05ms | tok/sec: 10,988 | mfu: 0.00 | epoch: 1 | total time: 9.96m
step 00392 (26.20%) | loss: 2.761341 | lrm: 1.00 | dt: 1512.05ms | tok/sec: 10,835 | mfu: 0.00 | epoch: 1 | total time: 9.98m
step 00393 (26.27%) | loss: 2.725036 | lrm: 1.00 | dt: 1499.29ms | tok/sec: 10,927 | mfu: 0.00 | epoch: 1 | total time: 10.01m
step 00394 (26.33%) | loss: 2.712968 | lrm: 1.00 | dt: 1520.21ms | tok/sec: 10,777 | mfu: 0.00 | epoch: 1 | total time: 10.03m
step 00395 (26.40%) | loss: 2.684755 | lrm: 1.00 | dt: 1517.02ms | tok/sec: 10,800 | mfu: 0.00 | epoch: 1 | total time: 10.06m
step 00396 (26.47%) | loss: 2.714863 | lrm: 1.00 | dt: 1524.24ms | tok/sec: 10,748 | mfu: 0.00 | epoch: 1 | total time: 10.08m
step 00397 (26.53%) | loss: 2.713876 | lrm: 1.00 | dt: 1532.54ms | tok/sec: 10,690 | mfu: 0.00 | epoch: 1 | total time: 10.11m
step 00398 (26.60%) | loss: 2.725199 | lrm: 1.00 | dt: 1498.47ms | tok/sec: 10,933 | mfu: 0.00 | epoch: 1 | total time: 10.13m
step 00399 (26.67%) | loss: 2.744146 | lrm: 1.00 | dt: 1494.42ms | tok/sec: 10,963 | mfu: 0.00 | epoch: 1 | total time: 10.16m
step 00400 (26.73%) | loss: 2.772887 | lrm: 1.00 | dt: 1496.00ms | tok/sec: 10,951 | mfu: 0.00 | epoch: 1 | total time: 10.18m
Step 00400 | Validation bpb: 0.9914
step 00401 (26.80%) | loss: 2.744857 | lrm: 1.00 | dt: 1501.11ms | tok/sec: 10,914 | mfu: 0.00 | epoch: 1 | total time: 10.21m
step 00402 (26.87%) | loss: 2.682859 | lrm: 1.00 | dt: 1512.10ms | tok/sec: 10,835 | mfu: 0.00 | epoch: 1 | total time: 10.23m
step 00403 (26.93%) | loss: 2.670237 | lrm: 1.00 | dt: 1507.83ms | tok/sec: 10,865 | mfu: 0.00 | epoch: 1 | total time: 10.26m
step 00404 (27.00%) | loss: 2.644767 | lrm: 1.00 | dt: 1508.68ms | tok/sec: 10,859 | mfu: 0.00 | epoch: 1 | total time: 10.28m
step 00405 (27.07%) | loss: 2.704053 | lrm: 1.00 | dt: 1502.53ms | tok/sec: 10,904 | mfu: 0.00 | epoch: 1 | total time: 10.31m
step 00406 (27.13%) | loss: 2.760858 | lrm: 1.00 | dt: 1506.46ms | tok/sec: 10,875 | mfu: 0.00 | epoch: 1 | total time: 10.33m
step 00407 (27.20%) | loss: 2.799783 | lrm: 1.00 | dt: 1487.87ms | tok/sec: 11,011 | mfu: 0.00 | epoch: 1 | total time: 10.36m
step 00408 (27.27%) | loss: 2.789597 | lrm: 1.00 | dt: 1503.65ms | tok/sec: 10,896 | mfu: 0.00 | epoch: 1 | total time: 10.38m
step 00409 (27.33%) | loss: 2.805987 | lrm: 1.00 | dt: 1474.67ms | tok/sec: 11,110 | mfu: 0.00 | epoch: 1 | total time: 10.41m
step 00410 (27.40%) | loss: 2.794021 | lrm: 1.00 | dt: 1520.15ms | tok/sec: 10,777 | mfu: 0.00 | epoch: 1 | total time: 10.43m
step 00411 (27.47%) | loss: 2.826715 | lrm: 1.00 | dt: 1506.92ms | tok/sec: 10,872 | mfu: 0.00 | epoch: 1 | total time: 10.46m
step 00412 (27.53%) | loss: 2.831830 | lrm: 1.00 | dt: 1495.61ms | tok/sec: 10,954 | mfu: 0.00 | epoch: 1 | total time: 10.48m
step 00413 (27.60%) | loss: 2.788379 | lrm: 1.00 | dt: 1517.60ms | tok/sec: 10,795 | mfu: 0.00 | epoch: 1 | total time: 10.51m
step 00414 (27.67%) | loss: 2.820278 | lrm: 1.00 | dt: 1486.63ms | tok/sec: 11,020 | mfu: 0.00 | epoch: 1 | total time: 10.53m
step 00415 (27.73%) | loss: 2.821269 | lrm: 1.00 | dt: 1498.68ms | tok/sec: 10,932 | mfu: 0.00 | epoch: 1 | total time: 10.56m
step 00416 (27.80%) | loss: 2.865136 | lrm: 1.00 | dt: 1481.16ms | tok/sec: 11,061 | mfu: 0.00 | epoch: 1 | total time: 10.58m
step 00417 (27.87%) | loss: 2.888118 | lrm: 1.00 | dt: 1504.70ms | tok/sec: 10,888 | mfu: 0.00 | epoch: 1 | total time: 10.61m
step 00418 (27.93%) | loss: 2.904073 | lrm: 1.00 | dt: 1578.60ms | tok/sec: 10,378 | mfu: 0.00 | epoch: 1 | total time: 10.63m
step 00419 (28.00%) | loss: 2.886319 | lrm: 1.00 | dt: 1534.57ms | tok/sec: 10,676 | mfu: 0.00 | epoch: 1 | total time: 10.66m
step 00420 (28.07%) | loss: 2.891409 | lrm: 1.00 | dt: 1543.70ms | tok/sec: 10,613 | mfu: 0.00 | epoch: 1 | total time: 10.69m
step 00421 (28.13%) | loss: 2.872274 | lrm: 1.00 | dt: 1529.94ms | tok/sec: 10,708 | mfu: 0.00 | epoch: 1 | total time: 10.71m
step 00422 (28.20%) | loss: 2.924096 | lrm: 1.00 | dt: 1521.83ms | tok/sec: 10,765 | mfu: 0.00 | epoch: 1 | total time: 10.74m
step 00423 (28.27%) | loss: 2.851694 | lrm: 1.00 | dt: 1496.48ms | tok/sec: 10,948 | mfu: 0.00 | epoch: 1 | total time: 10.76m
step 00424 (28.33%) | loss: 2.857127 | lrm: 1.00 | dt: 1515.85ms | tok/sec: 10,808 | mfu: 0.00 | epoch: 1 | total time: 10.79m
step 00425 (28.40%) | loss: 2.822691 | lrm: 1.00 | dt: 1488.39ms | tok/sec: 11,007 | mfu: 0.00 | epoch: 1 | total time: 10.81m
step 00426 (28.47%) | loss: 2.867509 | lrm: 1.00 | dt: 1515.68ms | tok/sec: 10,809 | mfu: 0.00 | epoch: 1 | total time: 10.84m
step 00427 (28.53%) | loss: 2.863832 | lrm: 1.00 | dt: 1517.48ms | tok/sec: 10,796 | mfu: 0.00 | epoch: 1 | total time: 10.86m
step 00428 (28.60%) | loss: 2.798334 | lrm: 1.00 | dt: 1512.35ms | tok/sec: 10,833 | mfu: 0.00 | epoch: 1 | total time: 10.89m
step 00429 (28.67%) | loss: 2.756826 | lrm: 1.00 | dt: 1499.80ms | tok/sec: 10,924 | mfu: 0.00 | epoch: 1 | total time: 10.91m
step 00430 (28.73%) | loss: 2.749014 | lrm: 1.00 | dt: 1500.79ms | tok/sec: 10,916 | mfu: 0.00 | epoch: 1 | total time: 10.94m
step 00431 (28.80%) | loss: 2.756509 | lrm: 1.00 | dt: 1512.18ms | tok/sec: 10,834 | mfu: 0.00 | epoch: 1 | total time: 10.96m
step 00432 (28.87%) | loss: 2.780013 | lrm: 1.00 | dt: 1558.05ms | tok/sec: 10,515 | mfu: 0.00 | epoch: 1 | total time: 10.99m
step 00433 (28.93%) | loss: 2.751880 | lrm: 1.00 | dt: 1510.72ms | tok/sec: 10,845 | mfu: 0.00 | epoch: 1 | total time: 11.01m
step 00434 (29.00%) | loss: 2.750537 | lrm: 1.00 | dt: 1481.79ms | tok/sec: 11,056 | mfu: 0.00 | epoch: 1 | total time: 11.04m
step 00435 (29.07%) | loss: 2.740214 | lrm: 1.00 | dt: 1513.41ms | tok/sec: 10,825 | mfu: 0.00 | epoch: 1 | total time: 11.06m
step 00436 (29.13%) | loss: 2.737264 | lrm: 1.00 | dt: 1498.13ms | tok/sec: 10,936 | mfu: 0.00 | epoch: 1 | total time: 11.09m
step 00437 (29.20%) | loss: 2.739659 | lrm: 1.00 | dt: 1522.11ms | tok/sec: 10,764 | mfu: 0.00 | epoch: 1 | total time: 11.11m
step 00438 (29.27%) | loss: 2.725492 | lrm: 1.00 | dt: 1523.30ms | tok/sec: 10,755 | mfu: 0.00 | epoch: 1 | total time: 11.14m
step 00439 (29.33%) | loss: 2.719952 | lrm: 1.00 | dt: 1518.84ms | tok/sec: 10,787 | mfu: 0.00 | epoch: 1 | total time: 11.16m
step 00440 (29.40%) | loss: 2.706259 | lrm: 1.00 | dt: 1511.64ms | tok/sec: 10,838 | mfu: 0.00 | epoch: 1 | total time: 11.19m
step 00441 (29.47%) | loss: 2.699848 | lrm: 1.00 | dt: 1517.13ms | tok/sec: 10,799 | mfu: 0.00 | epoch: 1 | total time: 11.22m
step 00442 (29.53%) | loss: 2.741598 | lrm: 1.00 | dt: 1504.77ms | tok/sec: 10,888 | mfu: 0.00 | epoch: 1 | total time: 11.24m
step 00443 (29.60%) | loss: 2.722269 | lrm: 1.00 | dt: 1503.76ms | tok/sec: 10,895 | mfu: 0.00 | epoch: 1 | total time: 11.27m
step 00444 (29.67%) | loss: 2.730911 | lrm: 1.00 | dt: 1500.90ms | tok/sec: 10,916 | mfu: 0.00 | epoch: 1 | total time: 11.29m
step 00445 (29.73%) | loss: 2.728124 | lrm: 1.00 | dt: 1512.11ms | tok/sec: 10,835 | mfu: 0.00 | epoch: 1 | total time: 11.32m
step 00446 (29.80%) | loss: 2.813650 | lrm: 1.00 | dt: 1527.08ms | tok/sec: 10,729 | mfu: 0.00 | epoch: 1 | total time: 11.34m
step 00447 (29.87%) | loss: 2.821153 | lrm: 1.00 | dt: 1530.57ms | tok/sec: 10,704 | mfu: 0.00 | epoch: 1 | total time: 11.37m
step 00448 (29.93%) | loss: 2.746175 | lrm: 1.00 | dt: 1524.68ms | tok/sec: 10,745 | mfu: 0.00 | epoch: 1 | total time: 11.39m
step 00449 (30.00%) | loss: 2.697512 | lrm: 1.00 | dt: 1512.15ms | tok/sec: 10,834 | mfu: 0.00 | epoch: 1 | total time: 11.42m
step 00450 (30.07%) | loss: 2.727454 | lrm: 1.00 | dt: 1507.59ms | tok/sec: 10,867 | mfu: 0.00 | epoch: 1 | total time: 11.44m
step 00451 (30.13%) | loss: 2.681549 | lrm: 1.00 | dt: 1513.21ms | tok/sec: 10,827 | mfu: 0.00 | epoch: 1 | total time: 11.47m
step 00452 (30.20%) | loss: 2.685231 | lrm: 1.00 | dt: 1508.60ms | tok/sec: 10,860 | mfu: 0.00 | epoch: 1 | total time: 11.49m
step 00453 (30.27%) | loss: 2.626052 | lrm: 1.00 | dt: 1607.41ms | tok/sec: 10,192 | mfu: 0.00 | epoch: 1 | total time: 11.52m
step 00454 (30.33%) | loss: 2.625728 | lrm: 1.00 | dt: 1522.47ms | tok/sec: 10,761 | mfu: 0.00 | epoch: 1 | total time: 11.54m
step 00455 (30.40%) | loss: 2.641852 | lrm: 1.00 | dt: 1542.46ms | tok/sec: 10,622 | mfu: 0.00 | epoch: 1 | total time: 11.57m
step 00456 (30.47%) | loss: 2.628288 | lrm: 1.00 | dt: 1529.52ms | tok/sec: 10,711 | mfu: 0.00 | epoch: 1 | total time: 11.60m
step 00457 (30.53%) | loss: 2.617580 | lrm: 1.00 | dt: 1566.20ms | tok/sec: 10,460 | mfu: 0.00 | epoch: 1 | total time: 11.62m
step 00458 (30.60%) | loss: 2.613184 | lrm: 1.00 | dt: 1525.27ms | tok/sec: 10,741 | mfu: 0.00 | epoch: 1 | total time: 11.65m
step 00459 (30.67%) | loss: 2.573656 | lrm: 1.00 | dt: 1517.34ms | tok/sec: 10,797 | mfu: 0.00 | epoch: 1 | total time: 11.67m
step 00460 (30.73%) | loss: 2.596646 | lrm: 1.00 | dt: 1534.32ms | tok/sec: 10,678 | mfu: 0.00 | epoch: 1 | total time: 11.70m
step 00461 (30.80%) | loss: 2.622456 | lrm: 1.00 | dt: 1499.45ms | tok/sec: 10,926 | mfu: 0.00 | epoch: 1 | total time: 11.72m
step 00462 (30.87%) | loss: 2.628126 | lrm: 1.00 | dt: 1543.71ms | tok/sec: 10,613 | mfu: 0.00 | epoch: 1 | total time: 11.75m
step 00463 (30.93%) | loss: 2.589236 | lrm: 1.00 | dt: 1508.44ms | tok/sec: 10,861 | mfu: 0.00 | epoch: 1 | total time: 11.77m
step 00464 (31.00%) | loss: 2.655935 | lrm: 1.00 | dt: 1492.94ms | tok/sec: 10,974 | mfu: 0.00 | epoch: 1 | total time: 11.80m
step 00465 (31.07%) | loss: 2.662390 | lrm: 1.00 | dt: 1531.87ms | tok/sec: 10,695 | mfu: 0.00 | epoch: 1 | total time: 11.82m
step 00466 (31.13%) | loss: 2.668578 | lrm: 1.00 | dt: 1515.01ms | tok/sec: 10,814 | mfu: 0.00 | epoch: 1 | total time: 11.85m
step 00467 (31.20%) | loss: 2.652761 | lrm: 1.00 | dt: 1501.78ms | tok/sec: 10,909 | mfu: 0.00 | epoch: 1 | total time: 11.88m
step 00468 (31.27%) | loss: 2.630653 | lrm: 1.00 | dt: 1522.81ms | tok/sec: 10,759 | mfu: 0.00 | epoch: 1 | total time: 11.90m
step 00469 (31.33%) | loss: 2.701908 | lrm: 1.00 | dt: 1541.06ms | tok/sec: 10,631 | mfu: 0.00 | epoch: 1 | total time: 11.93m
step 00470 (31.40%) | loss: 2.677271 | lrm: 1.00 | dt: 1502.74ms | tok/sec: 10,902 | mfu: 0.00 | epoch: 1 | total time: 11.95m
step 00471 (31.47%) | loss: 2.671502 | lrm: 1.00 | dt: 1489.32ms | tok/sec: 11,001 | mfu: 0.00 | epoch: 1 | total time: 11.98m
step 00472 (31.53%) | loss: 2.672157 | lrm: 1.00 | dt: 1522.03ms | tok/sec: 10,764 | mfu: 0.00 | epoch: 1 | total time: 12.00m
step 00473 (31.60%) | loss: 2.638912 | lrm: 1.00 | dt: 1511.06ms | tok/sec: 10,842 | mfu: 0.00 | epoch: 1 | total time: 12.03m
step 00474 (31.67%) | loss: 2.665716 | lrm: 1.00 | dt: 1524.34ms | tok/sec: 10,748 | mfu: 0.00 | epoch: 1 | total time: 12.05m
step 00475 (31.73%) | loss: 2.642139 | lrm: 1.00 | dt: 1529.48ms | tok/sec: 10,712 | mfu: 0.00 | epoch: 1 | total time: 12.08m
step 00476 (31.80%) | loss: 2.604290 | lrm: 1.00 | dt: 1524.46ms | tok/sec: 10,747 | mfu: 0.00 | epoch: 1 | total time: 12.10m
step 00477 (31.87%) | loss: 2.625198 | lrm: 1.00 | dt: 1505.61ms | tok/sec: 10,881 | mfu: 0.00 | epoch: 1 | total time: 12.13m
step 00478 (31.93%) | loss: 2.633410 | lrm: 1.00 | dt: 1565.83ms | tok/sec: 10,463 | mfu: 0.00 | epoch: 1 | total time: 12.15m
step 00479 (32.00%) | loss: 2.646172 | lrm: 1.00 | dt: 1535.93ms | tok/sec: 10,667 | mfu: 0.00 | epoch: 1 | total time: 12.18m
step 00480 (32.07%) | loss: 2.636051 | lrm: 1.00 | dt: 1547.95ms | tok/sec: 10,584 | mfu: 0.00 | epoch: 1 | total time: 12.21m
step 00481 (32.13%) | loss: 2.632605 | lrm: 1.00 | dt: 1551.02ms | tok/sec: 10,563 | mfu: 0.00 | epoch: 1 | total time: 12.23m
step 00482 (32.20%) | loss: 2.628583 | lrm: 1.00 | dt: 1552.82ms | tok/sec: 10,551 | mfu: 0.00 | epoch: 1 | total time: 12.26m
step 00483 (32.27%) | loss: 2.654480 | lrm: 1.00 | dt: 1523.84ms | tok/sec: 10,751 | mfu: 0.00 | epoch: 1 | total time: 12.28m
step 00484 (32.33%) | loss: 2.724096 | lrm: 1.00 | dt: 1529.67ms | tok/sec: 10,710 | mfu: 0.00 | epoch: 1 | total time: 12.31m
step 00485 (32.40%) | loss: 2.697166 | lrm: 1.00 | dt: 1520.41ms | tok/sec: 10,776 | mfu: 0.00 | epoch: 1 | total time: 12.33m
step 00486 (32.47%) | loss: 2.784136 | lrm: 1.00 | dt: 1521.63ms | tok/sec: 10,767 | mfu: 0.00 | epoch: 1 | total time: 12.36m
step 00487 (32.53%) | loss: 2.800572 | lrm: 1.00 | dt: 1544.70ms | tok/sec: 10,606 | mfu: 0.00 | epoch: 1 | total time: 12.38m
step 00488 (32.60%) | loss: 2.799652 | lrm: 1.00 | dt: 1561.55ms | tok/sec: 10,492 | mfu: 0.00 | epoch: 1 | total time: 12.41m
step 00489 (32.67%) | loss: 2.755921 | lrm: 1.00 | dt: 1529.60ms | tok/sec: 10,711 | mfu: 0.00 | epoch: 1 | total time: 12.44m
step 00490 (32.73%) | loss: 2.765104 | lrm: 1.00 | dt: 1545.02ms | tok/sec: 10,604 | mfu: 0.00 | epoch: 1 | total time: 12.46m
step 00491 (32.80%) | loss: 2.745325 | lrm: 1.00 | dt: 1514.28ms | tok/sec: 10,819 | mfu: 0.00 | epoch: 1 | total time: 12.49m
step 00492 (32.87%) | loss: 2.779094 | lrm: 1.00 | dt: 1544.13ms | tok/sec: 10,610 | mfu: 0.00 | epoch: 1 | total time: 12.51m
step 00493 (32.93%) | loss: 2.747904 | lrm: 1.00 | dt: 1552.96ms | tok/sec: 10,550 | mfu: 0.00 | epoch: 1 | total time: 12.54m
step 00494 (33.00%) | loss: 2.730346 | lrm: 1.00 | dt: 1739.54ms | tok/sec: 9,418 | mfu: 0.00 | epoch: 1 | total time: 12.57m
step 00495 (33.07%) | loss: 2.703912 | lrm: 1.00 | dt: 1653.73ms | tok/sec: 9,907 | mfu: 0.00 | epoch: 1 | total time: 12.60m
step 00496 (33.13%) | loss: 2.673907 | lrm: 1.00 | dt: 1614.21ms | tok/sec: 10,149 | mfu: 0.00 | epoch: 1 | total time: 12.62m
step 00497 (33.20%) | loss: 2.667141 | lrm: 1.00 | dt: 1680.50ms | tok/sec: 9,749 | mfu: 0.00 | epoch: 1 | total time: 12.65m
step 00498 (33.27%) | loss: 2.649772 | lrm: 1.00 | dt: 1631.76ms | tok/sec: 10,040 | mfu: 0.00 | epoch: 1 | total time: 12.68m
step 00499 (33.33%) | loss: 2.708163 | lrm: 1.00 | dt: 1623.42ms | tok/sec: 10,092 | mfu: 0.00 | epoch: 1 | total time: 12.70m
step 00500 (33.40%) | loss: 2.689994 | lrm: 1.00 | dt: 1701.05ms | tok/sec: 9,631 | mfu: 0.00 | epoch: 1 | total time: 12.73m
step 00501 (33.47%) | loss: 2.672964 | lrm: 1.00 | dt: 1677.16ms | tok/sec: 9,768 | mfu: 0.00 | epoch: 1 | total time: 12.76m
step 00502 (33.53%) | loss: 2.646944 | lrm: 1.00 | dt: 1609.16ms | tok/sec: 10,181 | mfu: 0.00 | epoch: 1 | total time: 12.79m
step 00503 (33.60%) | loss: 2.658405 | lrm: 1.00 | dt: 1607.00ms | tok/sec: 10,195 | mfu: 0.00 | epoch: 1 | total time: 12.81m
step 00504 (33.67%) | loss: 2.613331 | lrm: 1.00 | dt: 1571.46ms | tok/sec: 10,425 | mfu: 0.00 | epoch: 1 | total time: 12.84m
step 00505 (33.73%) | loss: 2.635495 | lrm: 1.00 | dt: 1586.16ms | tok/sec: 10,329 | mfu: 0.00 | epoch: 1 | total time: 12.87m
step 00506 (33.80%) | loss: 2.630779 | lrm: 1.00 | dt: 1552.66ms | tok/sec: 10,552 | mfu: 0.00 | epoch: 1 | total time: 12.89m
step 00507 (33.87%) | loss: 2.619154 | lrm: 1.00 | dt: 1562.07ms | tok/sec: 10,488 | mfu: 0.00 | epoch: 1 | total time: 12.92m
step 00508 (33.93%) | loss: 2.676370 | lrm: 1.00 | dt: 1542.84ms | tok/sec: 10,619 | mfu: 0.00 | epoch: 1 | total time: 12.94m
step 00509 (34.00%) | loss: 2.603503 | lrm: 1.00 | dt: 1568.07ms | tok/sec: 10,448 | mfu: 0.00 | epoch: 1 | total time: 12.97m
step 00510 (34.07%) | loss: 2.620506 | lrm: 1.00 | dt: 1582.56ms | tok/sec: 10,352 | mfu: 0.00 | epoch: 1 | total time: 13.00m
step 00511 (34.13%) | loss: 2.616098 | lrm: 1.00 | dt: 1567.78ms | tok/sec: 10,450 | mfu: 0.00 | epoch: 1 | total time: 13.02m
step 00512 (34.20%) | loss: 2.598622 | lrm: 1.00 | dt: 1599.08ms | tok/sec: 10,245 | mfu: 0.00 | epoch: 1 | total time: 13.05m
step 00513 (34.27%) | loss: 2.605674 | lrm: 1.00 | dt: 1557.78ms | tok/sec: 10,517 | mfu: 0.00 | epoch: 1 | total time: 13.08m
step 00514 (34.33%) | loss: 2.659803 | lrm: 1.00 | dt: 1541.53ms | tok/sec: 10,628 | mfu: 0.00 | epoch: 1 | total time: 13.10m
step 00515 (34.40%) | loss: 2.652420 | lrm: 1.00 | dt: 1553.09ms | tok/sec: 10,549 | mfu: 0.00 | epoch: 1 | total time: 13.13m
step 00516 (34.47%) | loss: 2.640053 | lrm: 1.00 | dt: 1547.74ms | tok/sec: 10,585 | mfu: 0.00 | epoch: 1 | total time: 13.15m
step 00517 (34.53%) | loss: 2.607561 | lrm: 1.00 | dt: 1542.06ms | tok/sec: 10,624 | mfu: 0.00 | epoch: 1 | total time: 13.18m
step 00518 (34.60%) | loss: 2.604008 | lrm: 1.00 | dt: 1563.91ms | tok/sec: 10,476 | mfu: 0.00 | epoch: 1 | total time: 13.20m
step 00519 (34.67%) | loss: 2.631202 | lrm: 1.00 | dt: 1527.24ms | tok/sec: 10,727 | mfu: 0.00 | epoch: 1 | total time: 13.23m
step 00520 (34.73%) | loss: 2.648417 | lrm: 1.00 | dt: 1586.77ms | tok/sec: 10,325 | mfu: 0.00 | epoch: 1 | total time: 13.26m
step 00521 (34.80%) | loss: 2.674723 | lrm: 1.00 | dt: 1567.42ms | tok/sec: 10,452 | mfu: 0.00 | epoch: 1 | total time: 13.28m
step 00522 (34.87%) | loss: 2.669008 | lrm: 1.00 | dt: 1538.04ms | tok/sec: 10,652 | mfu: 0.00 | epoch: 1 | total time: 13.31m
step 00523 (34.93%) | loss: 2.703769 | lrm: 1.00 | dt: 1544.31ms | tok/sec: 10,609 | mfu: 0.00 | epoch: 1 | total time: 13.33m
step 00524 (35.00%) | loss: 2.690919 | lrm: 1.00 | dt: 1579.59ms | tok/sec: 10,372 | mfu: 0.00 | epoch: 1 | total time: 13.36m
step 00525 (35.07%) | loss: 2.713313 | lrm: 1.00 | dt: 1581.24ms | tok/sec: 10,361 | mfu: 0.00 | epoch: 1 | total time: 13.39m
step 00526 (35.13%) | loss: 2.735793 | lrm: 1.00 | dt: 1546.24ms | tok/sec: 10,596 | mfu: 0.00 | epoch: 1 | total time: 13.41m
step 00527 (35.20%) | loss: 2.707700 | lrm: 1.00 | dt: 1502.53ms | tok/sec: 10,904 | mfu: 0.00 | epoch: 1 | total time: 13.44m
step 00528 (35.27%) | loss: 2.711592 | lrm: 1.00 | dt: 1528.35ms | tok/sec: 10,720 | mfu: 0.00 | epoch: 1 | total time: 13.46m
step 00529 (35.33%) | loss: 2.738105 | lrm: 1.00 | dt: 1507.69ms | tok/sec: 10,866 | mfu: 0.00 | epoch: 1 | total time: 13.49m
step 00530 (35.40%) | loss: 2.728075 | lrm: 1.00 | dt: 1511.06ms | tok/sec: 10,842 | mfu: 0.00 | epoch: 1 | total time: 13.51m
step 00531 (35.47%) | loss: 2.709116 | lrm: 1.00 | dt: 1530.44ms | tok/sec: 10,705 | mfu: 0.00 | epoch: 1 | total time: 13.54m
step 00532 (35.53%) | loss: 2.755170 | lrm: 1.00 | dt: 1670.40ms | tok/sec: 9,808 | mfu: 0.00 | epoch: 1 | total time: 13.57m
step 00533 (35.60%) | loss: 2.713932 | lrm: 1.00 | dt: 1644.90ms | tok/sec: 9,960 | mfu: 0.00 | epoch: 1 | total time: 13.59m
step 00534 (35.67%) | loss: 2.704693 | lrm: 1.00 | dt: 1608.14ms | tok/sec: 10,188 | mfu: 0.00 | epoch: 1 | total time: 13.62m
step 00535 (35.73%) | loss: 2.722845 | lrm: 1.00 | dt: 1567.34ms | tok/sec: 10,453 | mfu: 0.00 | epoch: 1 | total time: 13.65m
step 00536 (35.80%) | loss: 2.700946 | lrm: 1.00 | dt: 1600.28ms | tok/sec: 10,238 | mfu: 0.00 | epoch: 1 | total time: 13.67m
step 00537 (35.87%) | loss: 2.747172 | lrm: 1.00 | dt: 1602.82ms | tok/sec: 10,221 | mfu: 0.00 | epoch: 1 | total time: 13.70m
step 00538 (35.93%) | loss: 2.735829 | lrm: 1.00 | dt: 1597.93ms | tok/sec: 10,253 | mfu: 0.00 | epoch: 1 | total time: 13.73m
step 00539 (36.00%) | loss: 2.763906 | lrm: 1.00 | dt: 1590.82ms | tok/sec: 10,299 | mfu: 0.00 | epoch: 1 | total time: 13.75m
step 00540 (36.07%) | loss: 2.777875 | lrm: 1.00 | dt: 1625.27ms | tok/sec: 10,080 | mfu: 0.00 | epoch: 1 | total time: 13.78m
step 00541 (36.13%) | loss: 2.762409 | lrm: 1.00 | dt: 1515.42ms | tok/sec: 10,811 | mfu: 0.00 | epoch: 1 | total time: 13.81m
step 00542 (36.20%) | loss: 2.735969 | lrm: 1.00 | dt: 1530.65ms | tok/sec: 10,703 | mfu: 0.00 | epoch: 1 | total time: 13.83m
step 00543 (36.27%) | loss: 2.702759 | lrm: 1.00 | dt: 1496.67ms | tok/sec: 10,946 | mfu: 0.00 | epoch: 1 | total time: 13.86m
step 00544 (36.33%) | loss: 2.713949 | lrm: 1.00 | dt: 1594.95ms | tok/sec: 10,272 | mfu: 0.00 | epoch: 1 | total time: 13.88m
step 00545 (36.40%) | loss: 2.700038 | lrm: 1.00 | dt: 1651.00ms | tok/sec: 9,923 | mfu: 0.00 | epoch: 1 | total time: 13.91m
step 00546 (36.47%) | loss: 2.753315 | lrm: 1.00 | dt: 1628.64ms | tok/sec: 10,059 | mfu: 0.00 | epoch: 1 | total time: 13.94m
step 00547 (36.53%) | loss: 2.754693 | lrm: 1.00 | dt: 1609.91ms | tok/sec: 10,176 | mfu: 0.00 | epoch: 1 | total time: 13.96m
step 00548 (36.60%) | loss: 2.729389 | lrm: 1.00 | dt: 1575.67ms | tok/sec: 10,398 | mfu: 0.00 | epoch: 1 | total time: 13.99m
step 00549 (36.67%) | loss: 2.772348 | lrm: 1.00 | dt: 1586.41ms | tok/sec: 10,327 | mfu: 0.00 | epoch: 1 | total time: 14.02m
step 00550 (36.73%) | loss: 2.810376 | lrm: 1.00 | dt: 1579.07ms | tok/sec: 10,375 | mfu: 0.00 | epoch: 1 | total time: 14.04m
step 00551 (36.80%) | loss: 2.786991 | lrm: 1.00 | dt: 1616.98ms | tok/sec: 10,132 | mfu: 0.00 | epoch: 1 | total time: 14.07m
step 00552 (36.87%) | loss: 2.769187 | lrm: 1.00 | dt: 1540.56ms | tok/sec: 10,635 | mfu: 0.00 | epoch: 1 | total time: 14.10m
step 00553 (36.93%) | loss: 2.767683 | lrm: 1.00 | dt: 1538.01ms | tok/sec: 10,652 | mfu: 0.00 | epoch: 1 | total time: 14.12m
step 00554 (37.00%) | loss: 2.769629 | lrm: 1.00 | dt: 1563.33ms | tok/sec: 10,480 | mfu: 0.00 | epoch: 1 | total time: 14.15m
step 00555 (37.07%) | loss: 2.712794 | lrm: 1.00 | dt: 1622.29ms | tok/sec: 10,099 | mfu: 0.00 | epoch: 1 | total time: 14.18m
step 00556 (37.13%) | loss: 2.672382 | lrm: 1.00 | dt: 1653.19ms | tok/sec: 9,910 | mfu: 0.00 | epoch: 1 | total time: 14.20m
step 00557 (37.20%) | loss: 2.640938 | lrm: 1.00 | dt: 1641.54ms | tok/sec: 9,980 | mfu: 0.00 | epoch: 1 | total time: 14.23m
step 00558 (37.27%) | loss: 2.617033 | lrm: 1.00 | dt: 1616.23ms | tok/sec: 10,137 | mfu: 0.00 | epoch: 1 | total time: 14.26m
step 00559 (37.33%) | loss: 2.603166 | lrm: 1.00 | dt: 1618.15ms | tok/sec: 10,125 | mfu: 0.00 | epoch: 1 | total time: 14.28m
step 00560 (37.40%) | loss: 2.611399 | lrm: 1.00 | dt: 1643.00ms | tok/sec: 9,971 | mfu: 0.00 | epoch: 1 | total time: 14.31m
step 00561 (37.47%) | loss: 2.643289 | lrm: 1.00 | dt: 1649.84ms | tok/sec: 9,930 | mfu: 0.00 | epoch: 1 | total time: 14.34m
step 00562 (37.53%) | loss: 2.642547 | lrm: 1.00 | dt: 1656.75ms | tok/sec: 9,889 | mfu: 0.00 | epoch: 1 | total time: 14.37m
step 00563 (37.60%) | loss: 2.609064 | lrm: 1.00 | dt: 1622.09ms | tok/sec: 10,100 | mfu: 0.00 | epoch: 1 | total time: 14.39m
step 00564 (37.67%) | loss: 2.630896 | lrm: 1.00 | dt: 1625.63ms | tok/sec: 10,078 | mfu: 0.00 | epoch: 1 | total time: 14.42m
step 00565 (37.73%) | loss: 2.637474 | lrm: 1.00 | dt: 1673.84ms | tok/sec: 9,788 | mfu: 0.00 | epoch: 1 | total time: 14.45m
step 00566 (37.80%) | loss: 2.667267 | lrm: 1.00 | dt: 1633.57ms | tok/sec: 10,029 | mfu: 0.00 | epoch: 1 | total time: 14.48m
step 00567 (37.87%) | loss: 2.699633 | lrm: 1.00 | dt: 1629.26ms | tok/sec: 10,056 | mfu: 0.00 | epoch: 1 | total time: 14.50m
step 00568 (37.93%) | loss: 2.671111 | lrm: 1.00 | dt: 1721.20ms | tok/sec: 9,518 | mfu: 0.00 | epoch: 1 | total time: 14.53m
step 00569 (38.00%) | loss: 2.600678 | lrm: 1.00 | dt: 1559.89ms | tok/sec: 10,503 | mfu: 0.00 | epoch: 1 | total time: 14.56m
step 00570 (38.07%) | loss: 2.626125 | lrm: 1.00 | dt: 1527.07ms | tok/sec: 10,729 | mfu: 0.00 | epoch: 1 | total time: 14.58m
step 00571 (38.13%) | loss: 2.659205 | lrm: 1.00 | dt: 1548.32ms | tok/sec: 10,581 | mfu: 0.00 | epoch: 1 | total time: 14.61m
step 00572 (38.20%) | loss: 2.629917 | lrm: 1.00 | dt: 1530.23ms | tok/sec: 10,706 | mfu: 0.00 | epoch: 1 | total time: 14.63m
step 00573 (38.27%) | loss: 2.631036 | lrm: 1.00 | dt: 1529.95ms | tok/sec: 10,708 | mfu: 0.00 | epoch: 1 | total time: 14.66m
step 00574 (38.33%) | loss: 2.662584 | lrm: 1.00 | dt: 1562.13ms | tok/sec: 10,488 | mfu: 0.00 | epoch: 1 | total time: 14.69m
step 00575 (38.40%) | loss: 2.619189 | lrm: 1.00 | dt: 1543.31ms | tok/sec: 10,616 | mfu: 0.00 | epoch: 1 | total time: 14.71m
step 00576 (38.47%) | loss: 2.589221 | lrm: 1.00 | dt: 1552.48ms | tok/sec: 10,553 | mfu: 0.00 | epoch: 1 | total time: 14.74m
step 00577 (38.53%) | loss: 2.590890 | lrm: 1.00 | dt: 1522.24ms | tok/sec: 10,763 | mfu: 0.00 | epoch: 1 | total time: 14.76m
step 00578 (38.60%) | loss: 2.646483 | lrm: 1.00 | dt: 1529.30ms | tok/sec: 10,713 | mfu: 0.00 | epoch: 1 | total time: 14.79m
step 00579 (38.67%) | loss: 2.617279 | lrm: 1.00 | dt: 1568.76ms | tok/sec: 10,443 | mfu: 0.00 | epoch: 1 | total time: 14.81m
step 00580 (38.73%) | loss: 2.705578 | lrm: 1.00 | dt: 1529.76ms | tok/sec: 10,710 | mfu: 0.00 | epoch: 1 | total time: 14.84m
step 00581 (38.80%) | loss: 2.719059 | lrm: 1.00 | dt: 1554.35ms | tok/sec: 10,540 | mfu: 0.00 | epoch: 1 | total time: 14.87m
step 00582 (38.87%) | loss: 2.728466 | lrm: 1.00 | dt: 1546.15ms | tok/sec: 10,596 | mfu: 0.00 | epoch: 1 | total time: 14.89m
step 00583 (38.93%) | loss: 2.684978 | lrm: 1.00 | dt: 1544.24ms | tok/sec: 10,609 | mfu: 0.00 | epoch: 1 | total time: 14.92m
step 00584 (39.00%) | loss: 2.783963 | lrm: 1.00 | dt: 1527.66ms | tok/sec: 10,724 | mfu: 0.00 | epoch: 1 | total time: 14.94m
step 00585 (39.07%) | loss: 2.766675 | lrm: 1.00 | dt: 1536.21ms | tok/sec: 10,665 | mfu: 0.00 | epoch: 1 | total time: 14.97m
step 00586 (39.13%) | loss: 2.752759 | lrm: 1.00 | dt: 1511.85ms | tok/sec: 10,837 | mfu: 0.00 | epoch: 1 | total time: 14.99m
step 00587 (39.20%) | loss: 2.742832 | lrm: 1.00 | dt: 1519.40ms | tok/sec: 10,783 | mfu: 0.00 | epoch: 1 | total time: 15.02m
step 00588 (39.27%) | loss: 2.792051 | lrm: 1.00 | dt: 1504.02ms | tok/sec: 10,893 | mfu: 0.00 | epoch: 1 | total time: 15.04m
step 00589 (39.33%) | loss: 2.742647 | lrm: 1.00 | dt: 1546.94ms | tok/sec: 10,591 | mfu: 0.00 | epoch: 1 | total time: 15.07m
step 00590 (39.40%) | loss: 2.755302 | lrm: 1.00 | dt: 1553.58ms | tok/sec: 10,545 | mfu: 0.00 | epoch: 1 | total time: 15.10m
step 00591 (39.47%) | loss: 2.724998 | lrm: 1.00 | dt: 1533.96ms | tok/sec: 10,680 | mfu: 0.00 | epoch: 1 | total time: 15.12m
step 00592 (39.53%) | loss: 2.791908 | lrm: 1.00 | dt: 1526.12ms | tok/sec: 10,735 | mfu: 0.00 | epoch: 1 | total time: 15.15m
step 00593 (39.60%) | loss: 2.824020 | lrm: 1.00 | dt: 1552.35ms | tok/sec: 10,554 | mfu: 0.00 | epoch: 1 | total time: 15.17m
step 00594 (39.67%) | loss: 2.809594 | lrm: 1.00 | dt: 1504.60ms | tok/sec: 10,889 | mfu: 0.00 | epoch: 1 | total time: 15.20m
step 00595 (39.73%) | loss: 2.772917 | lrm: 1.00 | dt: 1504.12ms | tok/sec: 10,892 | mfu: 0.00 | epoch: 1 | total time: 15.22m
step 00596 (39.80%) | loss: 2.743743 | lrm: 1.00 | dt: 1496.47ms | tok/sec: 10,948 | mfu: 0.00 | epoch: 1 | total time: 15.25m
step 00597 (39.87%) | loss: 2.739369 | lrm: 1.00 | dt: 1485.18ms | tok/sec: 11,031 | mfu: 0.00 | epoch: 1 | total time: 15.27m
step 00598 (39.93%) | loss: 2.770219 | lrm: 1.00 | dt: 1570.25ms | tok/sec: 10,434 | mfu: 0.00 | epoch: 1 | total time: 15.30m
step 00599 (40.00%) | loss: 2.722914 | lrm: 1.00 | dt: 1502.41ms | tok/sec: 10,905 | mfu: 0.00 | epoch: 1 | total time: 15.32m
step 00600 (40.07%) | loss: 2.755285 | lrm: 1.00 | dt: 1542.03ms | tok/sec: 10,624 | mfu: 0.00 | epoch: 1 | total time: 15.35m
Step 00600 | Validation bpb: 0.9513
step 00601 (40.13%) | loss: 2.695409 | lrm: 1.00 | dt: 1537.49ms | tok/sec: 10,656 | mfu: 0.00 | epoch: 1 | total time: 15.37m
step 00602 (40.20%) | loss: 2.708766 | lrm: 1.00 | dt: 1546.66ms | tok/sec: 10,593 | mfu: 0.00 | epoch: 1 | total time: 15.40m
step 00603 (40.27%) | loss: 2.712026 | lrm: 1.00 | dt: 1555.21ms | tok/sec: 10,534 | mfu: 0.00 | epoch: 1 | total time: 15.43m
step 00604 (40.33%) | loss: 2.687170 | lrm: 1.00 | dt: 1533.96ms | tok/sec: 10,680 | mfu: 0.00 | epoch: 1 | total time: 15.45m
step 00605 (40.40%) | loss: 2.602564 | lrm: 1.00 | dt: 1527.27ms | tok/sec: 10,727 | mfu: 0.00 | epoch: 1 | total time: 15.48m
step 00606 (40.47%) | loss: 2.669161 | lrm: 1.00 | dt: 1558.76ms | tok/sec: 10,510 | mfu: 0.00 | epoch: 1 | total time: 15.50m
step 00607 (40.53%) | loss: 2.658800 | lrm: 1.00 | dt: 1546.84ms | tok/sec: 10,591 | mfu: 0.00 | epoch: 1 | total time: 15.53m
step 00608 (40.60%) | loss: 2.665947 | lrm: 1.00 | dt: 1529.95ms | tok/sec: 10,708 | mfu: 0.00 | epoch: 1 | total time: 15.55m
step 00609 (40.67%) | loss: 2.706855 | lrm: 1.00 | dt: 1516.74ms | tok/sec: 10,802 | mfu: 0.00 | epoch: 1 | total time: 15.58m
step 00610 (40.73%) | loss: 2.657886 | lrm: 1.00 | dt: 1551.86ms | tok/sec: 10,557 | mfu: 0.00 | epoch: 1 | total time: 15.61m
step 00611 (40.80%) | loss: 2.719219 | lrm: 1.00 | dt: 1554.79ms | tok/sec: 10,537 | mfu: 0.00 | epoch: 1 | total time: 15.63m
step 00612 (40.87%) | loss: 2.691215 | lrm: 1.00 | dt: 1623.68ms | tok/sec: 10,090 | mfu: 0.00 | epoch: 1 | total time: 15.66m
step 00613 (40.93%) | loss: 2.632929 | lrm: 1.00 | dt: 1616.54ms | tok/sec: 10,135 | mfu: 0.00 | epoch: 1 | total time: 15.69m
step 00614 (41.00%) | loss: 2.644710 | lrm: 1.00 | dt: 1683.77ms | tok/sec: 9,730 | mfu: 0.00 | epoch: 1 | total time: 15.71m
step 00615 (41.07%) | loss: 2.709387 | lrm: 1.00 | dt: 1651.36ms | tok/sec: 9,921 | mfu: 0.00 | epoch: 1 | total time: 15.74m
step 00616 (41.13%) | loss: 2.672022 | lrm: 1.00 | dt: 1625.87ms | tok/sec: 10,077 | mfu: 0.00 | epoch: 1 | total time: 15.77m
step 00617 (41.20%) | loss: 2.701685 | lrm: 1.00 | dt: 1622.48ms | tok/sec: 10,098 | mfu: 0.00 | epoch: 1 | total time: 15.80m
step 00618 (41.27%) | loss: 2.697123 | lrm: 1.00 | dt: 1625.84ms | tok/sec: 10,077 | mfu: 0.00 | epoch: 1 | total time: 15.82m
step 00619 (41.33%) | loss: 2.645871 | lrm: 1.00 | dt: 1656.45ms | tok/sec: 9,891 | mfu: 0.00 | epoch: 1 | total time: 15.85m
step 00620 (41.40%) | loss: 2.677406 | lrm: 1.00 | dt: 1591.32ms | tok/sec: 10,295 | mfu: 0.00 | epoch: 1 | total time: 15.88m
step 00621 (41.47%) | loss: 2.672103 | lrm: 1.00 | dt: 1660.13ms | tok/sec: 9,869 | mfu: 0.00 | epoch: 1 | total time: 15.90m
step 00622 (41.53%) | loss: 2.713436 | lrm: 1.00 | dt: 1649.19ms | tok/sec: 9,934 | mfu: 0.00 | epoch: 1 | total time: 15.93m
step 00623 (41.60%) | loss: 2.785579 | lrm: 1.00 | dt: 1660.53ms | tok/sec: 9,866 | mfu: 0.00 | epoch: 1 | total time: 15.96m
step 00624 (41.67%) | loss: 2.776521 | lrm: 1.00 | dt: 1621.21ms | tok/sec: 10,106 | mfu: 0.00 | epoch: 1 | total time: 15.99m
step 00625 (41.73%) | loss: 2.759780 | lrm: 1.00 | dt: 1624.96ms | tok/sec: 10,082 | mfu: 0.00 | epoch: 1 | total time: 16.01m
step 00626 (41.80%) | loss: 2.749755 | lrm: 1.00 | dt: 1626.02ms | tok/sec: 10,076 | mfu: 0.00 | epoch: 1 | total time: 16.04m
step 00627 (41.87%) | loss: 2.777849 | lrm: 1.00 | dt: 1615.61ms | tok/sec: 10,141 | mfu: 0.00 | epoch: 1 | total time: 16.07m
step 00628 (41.93%) | loss: 2.725506 | lrm: 1.00 | dt: 1807.54ms | tok/sec: 9,064 | mfu: 0.00 | epoch: 1 | total time: 16.10m
step 00629 (42.00%) | loss: 2.696992 | lrm: 1.00 | dt: 1682.24ms | tok/sec: 9,739 | mfu: 0.00 | epoch: 1 | total time: 16.13m
step 00630 (42.07%) | loss: 2.659400 | lrm: 1.00 | dt: 1658.33ms | tok/sec: 9,879 | mfu: 0.00 | epoch: 1 | total time: 16.15m
step 00631 (42.13%) | loss: 2.581626 | lrm: 1.00 | dt: 1740.98ms | tok/sec: 9,410 | mfu: 0.00 | epoch: 1 | total time: 16.18m
step 00632 (42.20%) | loss: 2.659227 | lrm: 1.00 | dt: 1657.81ms | tok/sec: 9,882 | mfu: 0.00 | epoch: 1 | total time: 16.21m
step 00633 (42.27%) | loss: 2.677710 | lrm: 1.00 | dt: 1655.00ms | tok/sec: 9,899 | mfu: 0.00 | epoch: 1 | total time: 16.24m
step 00634 (42.33%) | loss: 2.686441 | lrm: 1.00 | dt: 1690.22ms | tok/sec: 9,693 | mfu: 0.00 | epoch: 1 | total time: 16.27m
step 00635 (42.40%) | loss: 2.618589 | lrm: 1.00 | dt: 1657.73ms | tok/sec: 9,883 | mfu: 0.00 | epoch: 1 | total time: 16.29m
step 00636 (42.47%) | loss: 2.654166 | lrm: 1.00 | dt: 1577.28ms | tok/sec: 10,387 | mfu: 0.00 | epoch: 1 | total time: 16.32m
step 00637 (42.53%) | loss: 2.686935 | lrm: 1.00 | dt: 1622.25ms | tok/sec: 10,099 | mfu: 0.00 | epoch: 1 | total time: 16.35m
step 00638 (42.60%) | loss: 2.667104 | lrm: 1.00 | dt: 1657.51ms | tok/sec: 9,884 | mfu: 0.00 | epoch: 1 | total time: 16.37m
step 00639 (42.67%) | loss: 2.725983 | lrm: 1.00 | dt: 1675.60ms | tok/sec: 9,778 | mfu: 0.00 | epoch: 1 | total time: 16.40m
step 00640 (42.73%) | loss: 2.687232 | lrm: 1.00 | dt: 1689.39ms | tok/sec: 9,698 | mfu: 0.00 | epoch: 1 | total time: 16.43m
step 00641 (42.80%) | loss: 2.673190 | lrm: 1.00 | dt: 1620.64ms | tok/sec: 10,109 | mfu: 0.00 | epoch: 1 | total time: 16.46m
step 00642 (42.87%) | loss: 2.656780 | lrm: 1.00 | dt: 1724.42ms | tok/sec: 9,501 | mfu: 0.00 | epoch: 1 | total time: 16.49m
step 00643 (42.93%) | loss: 2.689655 | lrm: 1.00 | dt: 1667.78ms | tok/sec: 9,823 | mfu: 0.00 | epoch: 1 | total time: 16.51m
step 00644 (43.00%) | loss: 2.706252 | lrm: 1.00 | dt: 1673.46ms | tok/sec: 9,790 | mfu: 0.00 | epoch: 1 | total time: 16.54m
step 00645 (43.07%) | loss: 2.753296 | lrm: 1.00 | dt: 1720.78ms | tok/sec: 9,521 | mfu: 0.00 | epoch: 1 | total time: 16.57m
step 00646 (43.13%) | loss: 2.717909 | lrm: 1.00 | dt: 1707.09ms | tok/sec: 9,597 | mfu: 0.00 | epoch: 1 | total time: 16.60m
step 00647 (43.20%) | loss: 2.693735 | lrm: 1.00 | dt: 1656.71ms | tok/sec: 9,889 | mfu: 0.00 | epoch: 1 | total time: 16.63m
step 00648 (43.27%) | loss: 2.679098 | lrm: 1.00 | dt: 1704.75ms | tok/sec: 9,610 | mfu: 0.00 | epoch: 1 | total time: 16.66m
step 00649 (43.33%) | loss: 2.688400 | lrm: 1.00 | dt: 1684.70ms | tok/sec: 9,725 | mfu: 0.00 | epoch: 1 | total time: 16.68m
step 00650 (43.40%) | loss: 2.700297 | lrm: 1.00 | dt: 1667.81ms | tok/sec: 9,823 | mfu: 0.00 | epoch: 1 | total time: 16.71m
step 00651 (43.47%) | loss: 2.675509 | lrm: 1.00 | dt: 1689.55ms | tok/sec: 9,697 | mfu: 0.00 | epoch: 1 | total time: 16.74m
step 00652 (43.53%) | loss: 2.702653 | lrm: 1.00 | dt: 1692.98ms | tok/sec: 9,677 | mfu: 0.00 | epoch: 1 | total time: 16.77m
step 00653 (43.60%) | loss: 2.719211 | lrm: 1.00 | dt: 1744.09ms | tok/sec: 9,394 | mfu: 0.00 | epoch: 1 | total time: 16.80m
step 00654 (43.67%) | loss: 2.722809 | lrm: 1.00 | dt: 1664.86ms | tok/sec: 9,841 | mfu: 0.00 | epoch: 1 | total time: 16.82m
step 00655 (43.73%) | loss: 2.678247 | lrm: 1.00 | dt: 1646.61ms | tok/sec: 9,950 | mfu: 0.00 | epoch: 1 | total time: 16.85m
step 00656 (43.80%) | loss: 2.619624 | lrm: 1.00 | dt: 1615.27ms | tok/sec: 10,143 | mfu: 0.00 | epoch: 1 | total time: 16.88m
step 00657 (43.87%) | loss: 2.672057 | lrm: 1.00 | dt: 1604.67ms | tok/sec: 10,210 | mfu: 0.00 | epoch: 1 | total time: 16.91m
step 00658 (43.93%) | loss: 2.700160 | lrm: 1.00 | dt: 1637.54ms | tok/sec: 10,005 | mfu: 0.00 | epoch: 1 | total time: 16.93m
step 00659 (44.00%) | loss: 2.675396 | lrm: 1.00 | dt: 1547.12ms | tok/sec: 10,589 | mfu: 0.00 | epoch: 1 | total time: 16.96m
step 00660 (44.07%) | loss: 2.649217 | lrm: 1.00 | dt: 1687.20ms | tok/sec: 9,710 | mfu: 0.00 | epoch: 1 | total time: 16.99m
step 00661 (44.13%) | loss: 2.670329 | lrm: 1.00 | dt: 1638.81ms | tok/sec: 9,997 | mfu: 0.00 | epoch: 1 | total time: 17.01m
step 00662 (44.20%) | loss: 2.644344 | lrm: 1.00 | dt: 1629.61ms | tok/sec: 10,053 | mfu: 0.00 | epoch: 1 | total time: 17.04m
step 00663 (44.27%) | loss: 2.641971 | lrm: 1.00 | dt: 1679.70ms | tok/sec: 9,754 | mfu: 0.00 | epoch: 1 | total time: 17.07m
step 00664 (44.33%) | loss: 2.634107 | lrm: 1.00 | dt: 1650.26ms | tok/sec: 9,928 | mfu: 0.00 | epoch: 1 | total time: 17.10m
step 00665 (44.40%) | loss: 2.607525 | lrm: 1.00 | dt: 1703.01ms | tok/sec: 9,620 | mfu: 0.00 | epoch: 1 | total time: 17.12m
step 00666 (44.47%) | loss: 2.614809 | lrm: 1.00 | dt: 1656.34ms | tok/sec: 9,891 | mfu: 0.00 | epoch: 1 | total time: 17.15m
step 00667 (44.53%) | loss: 2.658726 | lrm: 1.00 | dt: 1624.36ms | tok/sec: 10,086 | mfu: 0.00 | epoch: 1 | total time: 17.18m
step 00668 (44.60%) | loss: 2.613162 | lrm: 1.00 | dt: 1666.84ms | tok/sec: 9,829 | mfu: 0.00 | epoch: 1 | total time: 17.21m
step 00669 (44.67%) | loss: 2.588738 | lrm: 1.00 | dt: 1554.67ms | tok/sec: 10,538 | mfu: 0.00 | epoch: 1 | total time: 17.23m
step 00670 (44.73%) | loss: 2.590008 | lrm: 1.00 | dt: 1512.98ms | tok/sec: 10,828 | mfu: 0.00 | epoch: 1 | total time: 17.26m
step 00671 (44.80%) | loss: 2.593386 | lrm: 1.00 | dt: 1521.32ms | tok/sec: 10,769 | mfu: 0.00 | epoch: 1 | total time: 17.28m
step 00672 (44.87%) | loss: 2.621424 | lrm: 1.00 | dt: 1593.78ms | tok/sec: 10,279 | mfu: 0.00 | epoch: 1 | total time: 17.31m
step 00673 (44.93%) | loss: 2.626003 | lrm: 1.00 | dt: 1508.13ms | tok/sec: 10,863 | mfu: 0.00 | epoch: 1 | total time: 17.34m
step 00674 (45.00%) | loss: 2.614916 | lrm: 1.00 | dt: 1558.86ms | tok/sec: 10,510 | mfu: 0.00 | epoch: 1 | total time: 17.36m
step 00675 (45.07%) | loss: 2.620925 | lrm: 1.00 | dt: 1496.25ms | tok/sec: 10,950 | mfu: 0.00 | epoch: 1 | total time: 17.39m
step 00676 (45.13%) | loss: 2.570736 | lrm: 1.00 | dt: 1531.56ms | tok/sec: 10,697 | mfu: 0.00 | epoch: 1 | total time: 17.41m
step 00677 (45.20%) | loss: 2.601037 | lrm: 1.00 | dt: 1533.08ms | tok/sec: 10,687 | mfu: 0.00 | epoch: 1 | total time: 17.44m
step 00678 (45.27%) | loss: 2.621454 | lrm: 1.00 | dt: 1541.01ms | tok/sec: 10,632 | mfu: 0.00 | epoch: 1 | total time: 17.46m
step 00679 (45.33%) | loss: 2.620461 | lrm: 1.00 | dt: 1539.26ms | tok/sec: 10,644 | mfu: 0.00 | epoch: 1 | total time: 17.49m
step 00680 (45.40%) | loss: 2.601411 | lrm: 1.00 | dt: 1496.90ms | tok/sec: 10,945 | mfu: 0.00 | epoch: 1 | total time: 17.51m
step 00681 (45.47%) | loss: 2.607199 | lrm: 1.00 | dt: 1531.89ms | tok/sec: 10,695 | mfu: 0.00 | epoch: 1 | total time: 17.54m
step 00682 (45.53%) | loss: 2.580833 | lrm: 1.00 | dt: 1496.01ms | tok/sec: 10,951 | mfu: 0.00 | epoch: 1 | total time: 17.56m
step 00683 (45.60%) | loss: 2.585976 | lrm: 1.00 | dt: 1524.43ms | tok/sec: 10,747 | mfu: 0.00 | epoch: 1 | total time: 17.59m
step 00684 (45.67%) | loss: 2.547500 | lrm: 1.00 | dt: 1513.01ms | tok/sec: 10,828 | mfu: 0.00 | epoch: 1 | total time: 17.61m
step 00685 (45.73%) | loss: 2.530064 | lrm: 1.00 | dt: 1505.34ms | tok/sec: 10,883 | mfu: 0.00 | epoch: 1 | total time: 17.64m
step 00686 (45.80%) | loss: 2.553569 | lrm: 1.00 | dt: 1543.33ms | tok/sec: 10,616 | mfu: 0.00 | epoch: 1 | total time: 17.67m
step 00687 (45.87%) | loss: 2.535266 | lrm: 1.00 | dt: 1516.18ms | tok/sec: 10,806 | mfu: 0.00 | epoch: 1 | total time: 17.69m
step 00688 (45.93%) | loss: 2.508510 | lrm: 1.00 | dt: 1530.83ms | tok/sec: 10,702 | mfu: 0.00 | epoch: 1 | total time: 17.72m
step 00689 (46.00%) | loss: 2.472649 | lrm: 1.00 | dt: 1513.81ms | tok/sec: 10,823 | mfu: 0.00 | epoch: 1 | total time: 17.74m
step 00690 (46.07%) | loss: 2.496863 | lrm: 1.00 | dt: 1508.41ms | tok/sec: 10,861 | mfu: 0.00 | epoch: 1 | total time: 17.77m
step 00691 (46.13%) | loss: 2.539944 | lrm: 1.00 | dt: 1510.62ms | tok/sec: 10,845 | mfu: 0.00 | epoch: 1 | total time: 17.79m
step 00692 (46.20%) | loss: 2.509772 | lrm: 1.00 | dt: 1517.38ms | tok/sec: 10,797 | mfu: 0.00 | epoch: 1 | total time: 17.82m
step 00693 (46.27%) | loss: 2.478121 | lrm: 1.00 | dt: 1534.02ms | tok/sec: 10,680 | mfu: 0.00 | epoch: 1 | total time: 17.84m
step 00694 (46.33%) | loss: 2.507317 | lrm: 1.00 | dt: 1514.84ms | tok/sec: 10,815 | mfu: 0.00 | epoch: 1 | total time: 17.87m
step 00695 (46.40%) | loss: 2.508695 | lrm: 1.00 | dt: 1517.28ms | tok/sec: 10,798 | mfu: 0.00 | epoch: 1 | total time: 17.89m
step 00696 (46.47%) | loss: 2.569475 | lrm: 1.00 | dt: 1485.11ms | tok/sec: 11,032 | mfu: 0.00 | epoch: 1 | total time: 17.92m
step 00697 (46.53%) | loss: 2.579010 | lrm: 1.00 | dt: 1501.45ms | tok/sec: 10,912 | mfu: 0.00 | epoch: 1 | total time: 17.94m
step 00698 (46.60%) | loss: 2.594429 | lrm: 1.00 | dt: 1495.32ms | tok/sec: 10,956 | mfu: 0.00 | epoch: 1 | total time: 17.97m
step 00699 (46.67%) | loss: 2.589746 | lrm: 1.00 | dt: 1490.85ms | tok/sec: 10,989 | mfu: 0.00 | epoch: 1 | total time: 17.99m
step 00700 (46.73%) | loss: 2.549885 | lrm: 1.00 | dt: 1513.22ms | tok/sec: 10,827 | mfu: 0.00 | epoch: 1 | total time: 18.02m
step 00701 (46.80%) | loss: 2.547100 | lrm: 1.00 | dt: 1489.57ms | tok/sec: 10,999 | mfu: 0.00 | epoch: 1 | total time: 18.04m
step 00702 (46.87%) | loss: 2.560428 | lrm: 1.00 | dt: 1529.49ms | tok/sec: 10,712 | mfu: 0.00 | epoch: 1 | total time: 18.07m
step 00703 (46.93%) | loss: 2.525123 | lrm: 1.00 | dt: 1504.38ms | tok/sec: 10,890 | mfu: 0.00 | epoch: 1 | total time: 18.09m
step 00704 (47.00%) | loss: 2.528587 | lrm: 1.00 | dt: 1535.86ms | tok/sec: 10,667 | mfu: 0.00 | epoch: 1 | total time: 18.12m
step 00705 (47.07%) | loss: 2.517651 | lrm: 1.00 | dt: 1510.73ms | tok/sec: 10,845 | mfu: 0.00 | epoch: 1 | total time: 18.14m
step 00706 (47.13%) | loss: 2.525228 | lrm: 1.00 | dt: 1495.79ms | tok/sec: 10,953 | mfu: 0.00 | epoch: 1 | total time: 18.17m
step 00707 (47.20%) | loss: 2.514650 | lrm: 1.00 | dt: 1526.00ms | tok/sec: 10,736 | mfu: 0.00 | epoch: 1 | total time: 18.19m
step 00708 (47.27%) | loss: 2.504568 | lrm: 1.00 | dt: 1491.27ms | tok/sec: 10,986 | mfu: 0.00 | epoch: 1 | total time: 18.22m
step 00709 (47.33%) | loss: 2.476278 | lrm: 1.00 | dt: 1522.72ms | tok/sec: 10,759 | mfu: 0.00 | epoch: 1 | total time: 18.25m
step 00710 (47.40%) | loss: 2.451612 | lrm: 1.00 | dt: 1529.81ms | tok/sec: 10,709 | mfu: 0.00 | epoch: 1 | total time: 18.27m
step 00711 (47.47%) | loss: 2.450857 | lrm: 1.00 | dt: 1518.71ms | tok/sec: 10,788 | mfu: 0.00 | epoch: 1 | total time: 18.30m
step 00712 (47.53%) | loss: 2.474304 | lrm: 1.00 | dt: 1528.04ms | tok/sec: 10,722 | mfu: 0.00 | epoch: 1 | total time: 18.32m
step 00713 (47.60%) | loss: 2.524094 | lrm: 1.00 | dt: 1509.66ms | tok/sec: 10,852 | mfu: 0.00 | epoch: 1 | total time: 18.35m
step 00714 (47.67%) | loss: 2.561957 | lrm: 1.00 | dt: 1506.38ms | tok/sec: 10,876 | mfu: 0.00 | epoch: 1 | total time: 18.37m
step 00715 (47.73%) | loss: 2.662907 | lrm: 1.00 | dt: 1502.80ms | tok/sec: 10,902 | mfu: 0.00 | epoch: 1 | total time: 18.40m
step 00716 (47.80%) | loss: 2.678016 | lrm: 1.00 | dt: 1497.81ms | tok/sec: 10,938 | mfu: 0.00 | epoch: 1 | total time: 18.42m
step 00717 (47.87%) | loss: 2.682038 | lrm: 1.00 | dt: 1506.52ms | tok/sec: 10,875 | mfu: 0.00 | epoch: 1 | total time: 18.45m
step 00718 (47.93%) | loss: 2.729030 | lrm: 1.00 | dt: 1500.27ms | tok/sec: 10,920 | mfu: 0.00 | epoch: 1 | total time: 18.47m
step 00719 (48.00%) | loss: 2.748644 | lrm: 1.00 | dt: 1507.45ms | tok/sec: 10,868 | mfu: 0.00 | epoch: 1 | total time: 18.50m
step 00720 (48.07%) | loss: 2.724778 | lrm: 1.00 | dt: 1546.64ms | tok/sec: 10,593 | mfu: 0.00 | epoch: 1 | total time: 18.52m
step 00721 (48.13%) | loss: 2.687723 | lrm: 1.00 | dt: 1508.64ms | tok/sec: 10,860 | mfu: 0.00 | epoch: 1 | total time: 18.55m
step 00722 (48.20%) | loss: 2.713436 | lrm: 1.00 | dt: 1527.65ms | tok/sec: 10,724 | mfu: 0.00 | epoch: 1 | total time: 18.57m
step 00723 (48.27%) | loss: 2.699334 | lrm: 1.00 | dt: 1602.25ms | tok/sec: 10,225 | mfu: 0.00 | epoch: 1 | total time: 18.60m
step 00724 (48.33%) | loss: 2.643227 | lrm: 1.00 | dt: 1523.52ms | tok/sec: 10,754 | mfu: 0.00 | epoch: 1 | total time: 18.63m
step 00725 (48.40%) | loss: 2.629517 | lrm: 1.00 | dt: 1547.31ms | tok/sec: 10,588 | mfu: 0.00 | epoch: 1 | total time: 18.65m
step 00726 (48.47%) | loss: 2.627442 | lrm: 1.00 | dt: 1573.22ms | tok/sec: 10,414 | mfu: 0.00 | epoch: 1 | total time: 18.68m
step 00727 (48.53%) | loss: 2.634469 | lrm: 1.00 | dt: 1565.31ms | tok/sec: 10,466 | mfu: 0.00 | epoch: 1 | total time: 18.70m
step 00728 (48.60%) | loss: 2.635684 | lrm: 1.00 | dt: 1574.61ms | tok/sec: 10,405 | mfu: 0.00 | epoch: 1 | total time: 18.73m
step 00729 (48.67%) | loss: 2.626115 | lrm: 1.00 | dt: 1510.15ms | tok/sec: 10,849 | mfu: 0.00 | epoch: 1 | total time: 18.75m
step 00730 (48.73%) | loss: 2.606128 | lrm: 1.00 | dt: 1538.67ms | tok/sec: 10,648 | mfu: 0.00 | epoch: 1 | total time: 18.78m
step 00731 (48.80%) | loss: 2.656038 | lrm: 1.00 | dt: 1540.02ms | tok/sec: 10,638 | mfu: 0.00 | epoch: 1 | total time: 18.81m
step 00732 (48.87%) | loss: 2.664077 | lrm: 1.00 | dt: 1536.53ms | tok/sec: 10,663 | mfu: 0.00 | epoch: 1 | total time: 18.83m
step 00733 (48.93%) | loss: 2.700407 | lrm: 1.00 | dt: 1530.25ms | tok/sec: 10,706 | mfu: 0.00 | epoch: 1 | total time: 18.86m
step 00734 (49.00%) | loss: 2.686150 | lrm: 1.00 | dt: 1535.80ms | tok/sec: 10,668 | mfu: 0.00 | epoch: 1 | total time: 18.88m
step 00735 (49.07%) | loss: 2.647143 | lrm: 1.00 | dt: 1519.08ms | tok/sec: 10,785 | mfu: 0.00 | epoch: 1 | total time: 18.91m
step 00736 (49.13%) | loss: 2.649495 | lrm: 1.00 | dt: 1531.25ms | tok/sec: 10,699 | mfu: 0.00 | epoch: 1 | total time: 18.93m
step 00737 (49.20%) | loss: 2.658831 | lrm: 1.00 | dt: 1493.88ms | tok/sec: 10,967 | mfu: 0.00 | epoch: 1 | total time: 18.96m
step 00738 (49.27%) | loss: 2.659196 | lrm: 1.00 | dt: 1503.52ms | tok/sec: 10,897 | mfu: 0.00 | epoch: 1 | total time: 18.98m
step 00739 (49.33%) | loss: 2.695345 | lrm: 1.00 | dt: 1564.91ms | tok/sec: 10,469 | mfu: 0.00 | epoch: 1 | total time: 19.01m
step 00740 (49.40%) | loss: 2.709167 | lrm: 1.00 | dt: 1532.30ms | tok/sec: 10,692 | mfu: 0.00 | epoch: 1 | total time: 19.04m
step 00741 (49.47%) | loss: 2.721359 | lrm: 1.00 | dt: 1540.48ms | tok/sec: 10,635 | mfu: 0.00 | epoch: 1 | total time: 19.06m
step 00742 (49.53%) | loss: 2.721199 | lrm: 1.00 | dt: 1522.59ms | tok/sec: 10,760 | mfu: 0.00 | epoch: 1 | total time: 19.09m
step 00743 (49.60%) | loss: 2.720870 | lrm: 1.00 | dt: 1548.75ms | tok/sec: 10,578 | mfu: 0.00 | epoch: 1 | total time: 19.11m
step 00744 (49.67%) | loss: 2.664131 | lrm: 1.00 | dt: 1565.28ms | tok/sec: 10,467 | mfu: 0.00 | epoch: 1 | total time: 19.14m
step 00745 (49.73%) | loss: 2.642988 | lrm: 1.00 | dt: 1518.52ms | tok/sec: 10,789 | mfu: 0.00 | epoch: 1 | total time: 19.16m
step 00746 (49.80%) | loss: 2.622763 | lrm: 1.00 | dt: 1519.99ms | tok/sec: 10,779 | mfu: 0.00 | epoch: 1 | total time: 19.19m
step 00747 (49.87%) | loss: 2.614923 | lrm: 1.00 | dt: 1535.61ms | tok/sec: 10,669 | mfu: 0.00 | epoch: 1 | total time: 19.21m
step 00748 (49.93%) | loss: 2.587969 | lrm: 1.00 | dt: 1538.50ms | tok/sec: 10,649 | mfu: 0.00 | epoch: 1 | total time: 19.24m
step 00749 (50.00%) | loss: 2.660727 | lrm: 1.00 | dt: 1686.25ms | tok/sec: 9,716 | mfu: 0.00 | epoch: 1 | total time: 19.27m
step 00750 (50.07%) | loss: 2.654309 | lrm: 1.00 | dt: 1856.40ms | tok/sec: 8,825 | mfu: 0.00 | epoch: 1 | total time: 19.30m
step 00751 (50.13%) | loss: 2.622968 | lrm: 1.00 | dt: 1705.81ms | tok/sec: 9,604 | mfu: 0.00 | epoch: 1 | total time: 19.33m
step 00752 (50.20%) | loss: 2.615504 | lrm: 1.00 | dt: 1598.01ms | tok/sec: 10,252 | mfu: 0.00 | epoch: 1 | total time: 19.35m
step 00753 (50.27%) | loss: 2.594878 | lrm: 0.99 | dt: 1690.59ms | tok/sec: 9,691 | mfu: 0.00 | epoch: 1 | total time: 19.38m
step 00754 (50.33%) | loss: 2.688150 | lrm: 0.99 | dt: 1739.62ms | tok/sec: 9,418 | mfu: 0.00 | epoch: 1 | total time: 19.41m
step 00755 (50.40%) | loss: 2.669211 | lrm: 0.99 | dt: 1563.66ms | tok/sec: 10,477 | mfu: 0.00 | epoch: 1 | total time: 19.44m
step 00756 (50.47%) | loss: 2.658013 | lrm: 0.99 | dt: 1569.25ms | tok/sec: 10,440 | mfu: 0.00 | epoch: 1 | total time: 19.46m
step 00757 (50.53%) | loss: 2.640401 | lrm: 0.99 | dt: 1609.67ms | tok/sec: 10,178 | mfu: 0.00 | epoch: 1 | total time: 19.49m
step 00758 (50.60%) | loss: 2.648192 | lrm: 0.99 | dt: 1740.79ms | tok/sec: 9,411 | mfu: 0.00 | epoch: 1 | total time: 19.52m
step 00759 (50.67%) | loss: 2.646998 | lrm: 0.99 | dt: 1638.74ms | tok/sec: 9,997 | mfu: 0.00 | epoch: 1 | total time: 19.55m
step 00760 (50.73%) | loss: 2.595888 | lrm: 0.99 | dt: 1615.97ms | tok/sec: 10,138 | mfu: 0.00 | epoch: 1 | total time: 19.57m
step 00761 (50.80%) | loss: 2.607493 | lrm: 0.98 | dt: 1581.67ms | tok/sec: 10,358 | mfu: 0.00 | epoch: 1 | total time: 19.60m
step 00762 (50.87%) | loss: 2.553690 | lrm: 0.98 | dt: 1568.71ms | tok/sec: 10,444 | mfu: 0.00 | epoch: 1 | total time: 19.63m
step 00763 (50.93%) | loss: 2.545920 | lrm: 0.98 | dt: 1560.20ms | tok/sec: 10,501 | mfu: 0.00 | epoch: 1 | total time: 19.65m
step 00764 (51.00%) | loss: 2.508829 | lrm: 0.98 | dt: 1617.84ms | tok/sec: 10,127 | mfu: 0.00 | epoch: 1 | total time: 19.68m
step 00765 (51.07%) | loss: 2.480734 | lrm: 0.98 | dt: 1590.83ms | tok/sec: 10,299 | mfu: 0.00 | epoch: 1 | total time: 19.71m
step 00766 (51.13%) | loss: 2.469832 | lrm: 0.98 | dt: 1564.32ms | tok/sec: 10,473 | mfu: 0.00 | epoch: 1 | total time: 19.73m
step 00767 (51.20%) | loss: 2.427290 | lrm: 0.98 | dt: 1540.99ms | tok/sec: 10,632 | mfu: 0.00 | epoch: 1 | total time: 19.76m
step 00768 (51.27%) | loss: 2.443900 | lrm: 0.97 | dt: 1634.15ms | tok/sec: 10,026 | mfu: 0.00 | epoch: 1 | total time: 19.78m
step 00769 (51.33%) | loss: 2.443312 | lrm: 0.97 | dt: 1635.76ms | tok/sec: 10,016 | mfu: 0.00 | epoch: 1 | total time: 19.81m
step 00770 (51.40%) | loss: 2.517464 | lrm: 0.97 | dt: 1589.77ms | tok/sec: 10,305 | mfu: 0.00 | epoch: 1 | total time: 19.84m
step 00771 (51.47%) | loss: 2.489948 | lrm: 0.97 | dt: 1614.42ms | tok/sec: 10,148 | mfu: 0.00 | epoch: 1 | total time: 19.87m
step 00772 (51.53%) | loss: 2.479664 | lrm: 0.97 | dt: 1646.58ms | tok/sec: 9,950 | mfu: 0.00 | epoch: 1 | total time: 19.89m
step 00773 (51.60%) | loss: 2.457817 | lrm: 0.97 | dt: 1646.57ms | tok/sec: 9,950 | mfu: 0.00 | epoch: 1 | total time: 19.92m
step 00774 (51.67%) | loss: 2.391414 | lrm: 0.97 | dt: 1617.49ms | tok/sec: 10,129 | mfu: 0.00 | epoch: 1 | total time: 19.95m
step 00775 (51.73%) | loss: 2.452830 | lrm: 0.97 | dt: 1639.67ms | tok/sec: 9,992 | mfu: 0.00 | epoch: 1 | total time: 19.97m
step 00776 (51.80%) | loss: 2.449308 | lrm: 0.96 | dt: 1623.67ms | tok/sec: 10,090 | mfu: 0.00 | epoch: 1 | total time: 20.00m
step 00777 (51.87%) | loss: 2.501610 | lrm: 0.96 | dt: 1596.94ms | tok/sec: 10,259 | mfu: 0.00 | epoch: 1 | total time: 20.03m
step 00778 (51.93%) | loss: 2.568180 | lrm: 0.96 | dt: 1598.07ms | tok/sec: 10,252 | mfu: 0.00 | epoch: 1 | total time: 20.05m
step 00779 (52.00%) | loss: 2.615627 | lrm: 0.96 | dt: 1590.94ms | tok/sec: 10,298 | mfu: 0.00 | epoch: 1 | total time: 20.08m
step 00780 (52.07%) | loss: 2.605285 | lrm: 0.96 | dt: 1651.17ms | tok/sec: 9,922 | mfu: 0.00 | epoch: 1 | total time: 20.11m
step 00781 (52.13%) | loss: 2.583863 | lrm: 0.96 | dt: 1770.03ms | tok/sec: 9,256 | mfu: 0.00 | epoch: 1 | total time: 20.14m
step 00782 (52.20%) | loss: 2.610176 | lrm: 0.96 | dt: 1533.64ms | tok/sec: 10,683 | mfu: 0.00 | epoch: 1 | total time: 20.16m
step 00783 (52.27%) | loss: 2.615704 | lrm: 0.95 | dt: 1586.30ms | tok/sec: 10,328 | mfu: 0.00 | epoch: 1 | total time: 20.19m
step 00784 (52.33%) | loss: 2.666977 | lrm: 0.95 | dt: 1526.16ms | tok/sec: 10,735 | mfu: 0.00 | epoch: 1 | total time: 20.22m
step 00785 (52.40%) | loss: 2.663237 | lrm: 0.95 | dt: 1587.41ms | tok/sec: 10,321 | mfu: 0.00 | epoch: 1 | total time: 20.24m
step 00786 (52.47%) | loss: 2.650189 | lrm: 0.95 | dt: 1532.53ms | tok/sec: 10,690 | mfu: 0.00 | epoch: 1 | total time: 20.27m
step 00787 (52.53%) | loss: 2.634247 | lrm: 0.95 | dt: 1592.74ms | tok/sec: 10,286 | mfu: 0.00 | epoch: 1 | total time: 20.29m
step 00788 (52.60%) | loss: 2.721857 | lrm: 0.95 | dt: 1561.57ms | tok/sec: 10,491 | mfu: 0.00 | epoch: 1 | total time: 20.32m
step 00789 (52.67%) | loss: 2.714047 | lrm: 0.95 | dt: 1536.41ms | tok/sec: 10,663 | mfu: 0.00 | epoch: 1 | total time: 20.35m
step 00790 (52.73%) | loss: 2.699430 | lrm: 0.95 | dt: 1595.41ms | tok/sec: 10,269 | mfu: 0.00 | epoch: 1 | total time: 20.37m
step 00791 (52.80%) | loss: 2.666831 | lrm: 0.94 | dt: 1556.38ms | tok/sec: 10,527 | mfu: 0.00 | epoch: 1 | total time: 20.40m
step 00792 (52.87%) | loss: 2.680800 | lrm: 0.94 | dt: 1504.96ms | tok/sec: 10,886 | mfu: 0.00 | epoch: 1 | total time: 20.42m
step 00793 (52.93%) | loss: 2.694724 | lrm: 0.94 | dt: 1582.20ms | tok/sec: 10,355 | mfu: 0.00 | epoch: 1 | total time: 20.45m
step 00794 (53.00%) | loss: 2.656246 | lrm: 0.94 | dt: 1592.45ms | tok/sec: 10,288 | mfu: 0.00 | epoch: 1 | total time: 20.48m
step 00795 (53.07%) | loss: 2.625351 | lrm: 0.94 | dt: 1542.18ms | tok/sec: 10,623 | mfu: 0.00 | epoch: 1 | total time: 20.50m
step 00796 (53.13%) | loss: 2.629922 | lrm: 0.94 | dt: 1559.35ms | tok/sec: 10,506 | mfu: 0.00 | epoch: 1 | total time: 20.53m
step 00797 (53.20%) | loss: 2.581207 | lrm: 0.94 | dt: 1537.32ms | tok/sec: 10,657 | mfu: 0.00 | epoch: 1 | total time: 20.55m
step 00798 (53.27%) | loss: 2.541359 | lrm: 0.93 | dt: 1514.99ms | tok/sec: 10,814 | mfu: 0.00 | epoch: 1 | total time: 20.58m
step 00799 (53.33%) | loss: 2.575574 | lrm: 0.93 | dt: 1551.14ms | tok/sec: 10,562 | mfu: 0.00 | epoch: 1 | total time: 20.60m
step 00800 (53.40%) | loss: 2.546944 | lrm: 0.93 | dt: 1567.19ms | tok/sec: 10,454 | mfu: 0.00 | epoch: 1 | total time: 20.63m
Step 00800 | Validation bpb: 0.9141
step 00801 (53.47%) | loss: 2.506786 | lrm: 0.93 | dt: 1524.26ms | tok/sec: 10,748 | mfu: 0.00 | epoch: 1 | total time: 20.66m
step 00802 (53.53%) | loss: 2.505285 | lrm: 0.93 | dt: 1487.43ms | tok/sec: 11,014 | mfu: 0.00 | epoch: 1 | total time: 20.68m
step 00803 (53.60%) | loss: 2.471501 | lrm: 0.93 | dt: 1523.29ms | tok/sec: 10,755 | mfu: 0.00 | epoch: 1 | total time: 20.71m
step 00804 (53.67%) | loss: 2.449767 | lrm: 0.93 | dt: 1502.58ms | tok/sec: 10,903 | mfu: 0.00 | epoch: 1 | total time: 20.73m
step 00805 (53.73%) | loss: 2.466024 | lrm: 0.93 | dt: 1515.58ms | tok/sec: 10,810 | mfu: 0.00 | epoch: 1 | total time: 20.76m
step 00806 (53.80%) | loss: 2.501885 | lrm: 0.92 | dt: 1525.76ms | tok/sec: 10,738 | mfu: 0.00 | epoch: 1 | total time: 20.78m
step 00807 (53.87%) | loss: 2.548119 | lrm: 0.92 | dt: 1547.30ms | tok/sec: 10,588 | mfu: 0.00 | epoch: 1 | total time: 20.81m
step 00808 (53.93%) | loss: 2.578706 | lrm: 0.92 | dt: 1530.71ms | tok/sec: 10,703 | mfu: 0.00 | epoch: 1 | total time: 20.83m
step 00809 (54.00%) | loss: 2.500512 | lrm: 0.92 | dt: 1541.82ms | tok/sec: 10,626 | mfu: 0.00 | epoch: 1 | total time: 20.86m
step 00810 (54.07%) | loss: 2.498185 | lrm: 0.92 | dt: 1562.38ms | tok/sec: 10,486 | mfu: 0.00 | epoch: 1 | total time: 20.89m
step 00811 (54.13%) | loss: 2.533717 | lrm: 0.92 | dt: 1543.57ms | tok/sec: 10,614 | mfu: 0.00 | epoch: 1 | total time: 20.91m
step 00812 (54.20%) | loss: 2.533620 | lrm: 0.92 | dt: 1544.86ms | tok/sec: 10,605 | mfu: 0.00 | epoch: 1 | total time: 20.94m
step 00813 (54.27%) | loss: 2.630403 | lrm: 0.91 | dt: 1556.78ms | tok/sec: 10,524 | mfu: 0.00 | epoch: 1 | total time: 20.96m
step 00814 (54.33%) | loss: 2.628302 | lrm: 0.91 | dt: 1521.10ms | tok/sec: 10,771 | mfu: 0.00 | epoch: 1 | total time: 20.99m
step 00815 (54.40%) | loss: 2.690580 | lrm: 0.91 | dt: 1617.62ms | tok/sec: 10,128 | mfu: 0.00 | epoch: 1 | total time: 21.02m
step 00816 (54.47%) | loss: 2.641287 | lrm: 0.91 | dt: 1639.05ms | tok/sec: 9,996 | mfu: 0.00 | epoch: 1 | total time: 21.04m
step 00817 (54.53%) | loss: 2.556172 | lrm: 0.91 | dt: 1623.71ms | tok/sec: 10,090 | mfu: 0.00 | epoch: 1 | total time: 21.07m
step 00818 (54.60%) | loss: 2.506228 | lrm: 0.91 | dt: 1634.35ms | tok/sec: 10,024 | mfu: 0.00 | epoch: 1 | total time: 21.10m
step 00819 (54.67%) | loss: 2.591624 | lrm: 0.91 | dt: 1604.62ms | tok/sec: 10,210 | mfu: 0.00 | epoch: 1 | total time: 21.12m
step 00820 (54.73%) | loss: 2.589183 | lrm: 0.91 | dt: 1619.75ms | tok/sec: 10,115 | mfu: 0.00 | epoch: 1 | total time: 21.15m
step 00821 (54.80%) | loss: 2.584342 | lrm: 0.90 | dt: 1634.94ms | tok/sec: 10,021 | mfu: 0.00 | epoch: 1 | total time: 21.18m
step 00822 (54.87%) | loss: 2.635101 | lrm: 0.90 | dt: 1610.01ms | tok/sec: 10,176 | mfu: 0.00 | epoch: 1 | total time: 21.20m
step 00823 (54.93%) | loss: 2.589384 | lrm: 0.90 | dt: 1634.73ms | tok/sec: 10,022 | mfu: 0.00 | epoch: 1 | total time: 21.23m
step 00824 (55.00%) | loss: 2.607225 | lrm: 0.90 | dt: 1634.47ms | tok/sec: 10,024 | mfu: 0.00 | epoch: 1 | total time: 21.26m
step 00825 (55.07%) | loss: 2.571849 | lrm: 0.90 | dt: 1620.63ms | tok/sec: 10,109 | mfu: 0.00 | epoch: 1 | total time: 21.29m
step 00826 (55.13%) | loss: 2.501248 | lrm: 0.90 | dt: 1644.00ms | tok/sec: 9,965 | mfu: 0.00 | epoch: 1 | total time: 21.31m
step 00827 (55.20%) | loss: 2.518254 | lrm: 0.90 | dt: 1633.58ms | tok/sec: 10,029 | mfu: 0.00 | epoch: 1 | total time: 21.34m
step 00828 (55.27%) | loss: 2.503104 | lrm: 0.89 | dt: 1624.56ms | tok/sec: 10,085 | mfu: 0.00 | epoch: 1 | total time: 21.37m
step 00829 (55.33%) | loss: 2.495123 | lrm: 0.89 | dt: 1635.33ms | tok/sec: 10,018 | mfu: 0.00 | epoch: 1 | total time: 21.39m
step 00830 (55.40%) | loss: 2.486436 | lrm: 0.89 | dt: 1621.44ms | tok/sec: 10,104 | mfu: 0.00 | epoch: 1 | total time: 21.42m
step 00831 (55.47%) | loss: 2.601810 | lrm: 0.89 | dt: 1643.83ms | tok/sec: 9,966 | mfu: 0.00 | epoch: 1 | total time: 21.45m
step 00832 (55.53%) | loss: 2.530127 | lrm: 0.89 | dt: 1635.55ms | tok/sec: 10,017 | mfu: 0.00 | epoch: 1 | total time: 21.48m
step 00833 (55.60%) | loss: 2.488122 | lrm: 0.89 | dt: 1607.08ms | tok/sec: 10,194 | mfu: 0.00 | epoch: 1 | total time: 21.50m
step 00834 (55.67%) | loss: 2.470278 | lrm: 0.89 | dt: 1643.00ms | tok/sec: 9,971 | mfu: 0.00 | epoch: 1 | total time: 21.53m
step 00835 (55.73%) | loss: 2.549592 | lrm: 0.89 | dt: 1642.69ms | tok/sec: 9,973 | mfu: 0.00 | epoch: 1 | total time: 21.56m
step 00836 (55.80%) | loss: 2.583580 | lrm: 0.88 | dt: 1617.15ms | tok/sec: 10,131 | mfu: 0.00 | epoch: 1 | total time: 21.59m
step 00837 (55.87%) | loss: 2.535299 | lrm: 0.88 | dt: 1635.57ms | tok/sec: 10,017 | mfu: 0.00 | epoch: 1 | total time: 21.61m
step 00838 (55.93%) | loss: 2.541570 | lrm: 0.88 | dt: 1648.12ms | tok/sec: 9,941 | mfu: 0.00 | epoch: 1 | total time: 21.64m
step 00839 (56.00%) | loss: 2.518295 | lrm: 0.88 | dt: 1649.43ms | tok/sec: 9,933 | mfu: 0.00 | epoch: 1 | total time: 21.67m
step 00840 (56.07%) | loss: 2.490199 | lrm: 0.88 | dt: 1619.39ms | tok/sec: 10,117 | mfu: 0.00 | epoch: 1 | total time: 21.69m
step 00841 (56.13%) | loss: 2.483685 | lrm: 0.88 | dt: 1617.42ms | tok/sec: 10,129 | mfu: 0.00 | epoch: 1 | total time: 21.72m
step 00842 (56.20%) | loss: 2.549303 | lrm: 0.88 | dt: 1631.30ms | tok/sec: 10,043 | mfu: 0.00 | epoch: 1 | total time: 21.75m
step 00843 (56.27%) | loss: 2.486238 | lrm: 0.87 | dt: 1661.46ms | tok/sec: 9,861 | mfu: 0.00 | epoch: 1 | total time: 21.78m
step 00844 (56.33%) | loss: 2.463921 | lrm: 0.87 | dt: 1725.12ms | tok/sec: 9,497 | mfu: 0.00 | epoch: 1 | total time: 21.80m
step 00845 (56.40%) | loss: 2.500647 | lrm: 0.87 | dt: 1669.47ms | tok/sec: 9,813 | mfu: 0.00 | epoch: 1 | total time: 21.83m
step 00846 (56.47%) | loss: 2.534159 | lrm: 0.87 | dt: 1615.74ms | tok/sec: 10,140 | mfu: 0.00 | epoch: 1 | total time: 21.86m
step 00847 (56.53%) | loss: 2.534668 | lrm: 0.87 | dt: 1597.23ms | tok/sec: 10,257 | mfu: 0.00 | epoch: 1 | total time: 21.89m
step 00848 (56.60%) | loss: 2.543687 | lrm: 0.87 | dt: 1643.43ms | tok/sec: 9,969 | mfu: 0.00 | epoch: 1 | total time: 21.91m
step 00849 (56.67%) | loss: 2.542417 | lrm: 0.87 | dt: 1534.72ms | tok/sec: 10,675 | mfu: 0.00 | epoch: 1 | total time: 21.94m
step 00850 (56.73%) | loss: 2.507413 | lrm: 0.87 | dt: 1627.63ms | tok/sec: 10,066 | mfu: 0.00 | epoch: 1 | total time: 21.97m
step 00851 (56.80%) | loss: 2.483119 | lrm: 0.86 | dt: 1627.69ms | tok/sec: 10,065 | mfu: 0.00 | epoch: 1 | total time: 21.99m
step 00852 (56.87%) | loss: 2.470462 | lrm: 0.86 | dt: 1642.57ms | tok/sec: 9,974 | mfu: 0.00 | epoch: 1 | total time: 22.02m
step 00853 (56.93%) | loss: 2.464630 | lrm: 0.86 | dt: 1534.18ms | tok/sec: 10,679 | mfu: 0.00 | epoch: 1 | total time: 22.05m
step 00854 (57.00%) | loss: 2.430880 | lrm: 0.86 | dt: 1514.54ms | tok/sec: 10,817 | mfu: 0.00 | epoch: 1 | total time: 22.07m
step 00855 (57.07%) | loss: 2.421355 | lrm: 0.86 | dt: 1540.72ms | tok/sec: 10,633 | mfu: 0.00 | epoch: 1 | total time: 22.10m
step 00856 (57.13%) | loss: 2.400980 | lrm: 0.86 | dt: 1544.38ms | tok/sec: 10,608 | mfu: 0.00 | epoch: 1 | total time: 22.12m
step 00857 (57.20%) | loss: 2.469633 | lrm: 0.86 | dt: 1509.69ms | tok/sec: 10,852 | mfu: 0.00 | epoch: 1 | total time: 22.15m
step 00858 (57.27%) | loss: 2.490280 | lrm: 0.85 | dt: 1511.57ms | tok/sec: 10,839 | mfu: 0.00 | epoch: 1 | total time: 22.17m
step 00859 (57.33%) | loss: 2.522444 | lrm: 0.85 | dt: 1531.45ms | tok/sec: 10,698 | mfu: 0.00 | epoch: 1 | total time: 22.20m
step 00860 (57.40%) | loss: 2.469000 | lrm: 0.85 | dt: 1538.58ms | tok/sec: 10,648 | mfu: 0.00 | epoch: 1 | total time: 22.22m
step 00861 (57.47%) | loss: 2.488110 | lrm: 0.85 | dt: 1518.84ms | tok/sec: 10,787 | mfu: 0.00 | epoch: 1 | total time: 22.25m
step 00862 (57.53%) | loss: 2.530627 | lrm: 0.85 | dt: 1517.33ms | tok/sec: 10,797 | mfu: 0.00 | epoch: 1 | total time: 22.28m
step 00863 (57.60%) | loss: 2.565592 | lrm: 0.85 | dt: 1537.15ms | tok/sec: 10,658 | mfu: 0.00 | epoch: 1 | total time: 22.30m
step 00864 (57.67%) | loss: 2.557293 | lrm: 0.85 | dt: 1503.84ms | tok/sec: 10,894 | mfu: 0.00 | epoch: 1 | total time: 22.33m
step 00865 (57.73%) | loss: 2.551699 | lrm: 0.85 | dt: 1575.52ms | tok/sec: 10,399 | mfu: 0.00 | epoch: 1 | total time: 22.35m
step 00866 (57.80%) | loss: 2.573321 | lrm: 0.84 | dt: 1495.98ms | tok/sec: 10,952 | mfu: 0.00 | epoch: 1 | total time: 22.38m
step 00867 (57.87%) | loss: 2.554382 | lrm: 0.84 | dt: 1523.90ms | tok/sec: 10,751 | mfu: 0.00 | epoch: 1 | total time: 22.40m
step 00868 (57.93%) | loss: 2.514794 | lrm: 0.84 | dt: 1502.16ms | tok/sec: 10,906 | mfu: 0.00 | epoch: 1 | total time: 22.43m
step 00869 (58.00%) | loss: 2.531451 | lrm: 0.84 | dt: 1501.48ms | tok/sec: 10,911 | mfu: 0.00 | epoch: 1 | total time: 22.45m
step 00870 (58.07%) | loss: 2.465310 | lrm: 0.84 | dt: 1519.03ms | tok/sec: 10,785 | mfu: 0.00 | epoch: 1 | total time: 22.48m
step 00871 (58.13%) | loss: 2.438525 | lrm: 0.84 | dt: 1524.64ms | tok/sec: 10,746 | mfu: 0.00 | epoch: 1 | total time: 22.50m
step 00872 (58.20%) | loss: 2.487534 | lrm: 0.84 | dt: 1520.34ms | tok/sec: 10,776 | mfu: 0.00 | epoch: 1 | total time: 22.53m
step 00873 (58.27%) | loss: 2.501286 | lrm: 0.83 | dt: 1506.68ms | tok/sec: 10,874 | mfu: 0.00 | epoch: 1 | total time: 22.55m
step 00874 (58.33%) | loss: 2.460813 | lrm: 0.83 | dt: 1506.83ms | tok/sec: 10,873 | mfu: 0.00 | epoch: 1 | total time: 22.58m
step 00875 (58.40%) | loss: 2.447735 | lrm: 0.83 | dt: 1496.14ms | tok/sec: 10,950 | mfu: 0.00 | epoch: 1 | total time: 22.60m
step 00876 (58.47%) | loss: 2.427928 | lrm: 0.83 | dt: 1526.21ms | tok/sec: 10,735 | mfu: 0.00 | epoch: 1 | total time: 22.63m
step 00877 (58.53%) | loss: 2.501668 | lrm: 0.83 | dt: 1497.24ms | tok/sec: 10,942 | mfu: 0.00 | epoch: 1 | total time: 22.65m
step 00878 (58.60%) | loss: 2.504743 | lrm: 0.83 | dt: 1537.35ms | tok/sec: 10,657 | mfu: 0.00 | epoch: 1 | total time: 22.68m
step 00879 (58.67%) | loss: 2.481842 | lrm: 0.83 | dt: 1490.18ms | tok/sec: 10,994 | mfu: 0.00 | epoch: 1 | total time: 22.70m
step 00880 (58.73%) | loss: 2.476214 | lrm: 0.83 | dt: 1524.60ms | tok/sec: 10,746 | mfu: 0.00 | epoch: 1 | total time: 22.73m
step 00881 (58.80%) | loss: 2.450649 | lrm: 0.82 | dt: 1511.43ms | tok/sec: 10,840 | mfu: 0.00 | epoch: 1 | total time: 22.76m
step 00882 (58.87%) | loss: 2.471862 | lrm: 0.82 | dt: 1503.36ms | tok/sec: 10,898 | mfu: 0.00 | epoch: 1 | total time: 22.78m
step 00883 (58.93%) | loss: 2.460784 | lrm: 0.82 | dt: 1498.19ms | tok/sec: 10,935 | mfu: 0.00 | epoch: 1 | total time: 22.81m
step 00884 (59.00%) | loss: 2.451250 | lrm: 0.82 | dt: 1506.46ms | tok/sec: 10,875 | mfu: 0.00 | epoch: 1 | total time: 22.83m
step 00885 (59.07%) | loss: 2.492951 | lrm: 0.82 | dt: 1537.14ms | tok/sec: 10,658 | mfu: 0.00 | epoch: 1 | total time: 22.86m
step 00886 (59.13%) | loss: 2.476748 | lrm: 0.82 | dt: 1524.46ms | tok/sec: 10,747 | mfu: 0.00 | epoch: 1 | total time: 22.88m
step 00887 (59.20%) | loss: 2.455706 | lrm: 0.82 | dt: 1512.61ms | tok/sec: 10,831 | mfu: 0.00 | epoch: 1 | total time: 22.91m
step 00888 (59.27%) | loss: 2.472691 | lrm: 0.81 | dt: 1534.81ms | tok/sec: 10,674 | mfu: 0.00 | epoch: 1 | total time: 22.93m
step 00889 (59.33%) | loss: 2.445528 | lrm: 0.81 | dt: 1511.16ms | tok/sec: 10,842 | mfu: 0.00 | epoch: 1 | total time: 22.96m
step 00890 (59.40%) | loss: 2.436633 | lrm: 0.81 | dt: 1499.58ms | tok/sec: 10,925 | mfu: 0.00 | epoch: 1 | total time: 22.98m
step 00891 (59.47%) | loss: 2.536087 | lrm: 0.81 | dt: 1506.68ms | tok/sec: 10,874 | mfu: 0.00 | epoch: 1 | total time: 23.01m
step 00892 (59.53%) | loss: 2.568812 | lrm: 0.81 | dt: 1534.88ms | tok/sec: 10,674 | mfu: 0.00 | epoch: 1 | total time: 23.03m
step 00893 (59.60%) | loss: 2.564946 | lrm: 0.81 | dt: 1502.60ms | tok/sec: 10,903 | mfu: 0.00 | epoch: 1 | total time: 23.06m
step 00894 (59.67%) | loss: 2.535627 | lrm: 0.81 | dt: 1545.87ms | tok/sec: 10,598 | mfu: 0.00 | epoch: 1 | total time: 23.08m
step 00895 (59.73%) | loss: 2.496581 | lrm: 0.81 | dt: 1511.37ms | tok/sec: 10,840 | mfu: 0.00 | epoch: 1 | total time: 23.11m
step 00896 (59.80%) | loss: 2.516986 | lrm: 0.80 | dt: 1513.49ms | tok/sec: 10,825 | mfu: 0.00 | epoch: 1 | total time: 23.13m
step 00897 (59.87%) | loss: 2.436323 | lrm: 0.80 | dt: 1504.80ms | tok/sec: 10,887 | mfu: 0.00 | epoch: 1 | total time: 23.16m
step 00898 (59.93%) | loss: 2.417509 | lrm: 0.80 | dt: 1519.99ms | tok/sec: 10,779 | mfu: 0.00 | epoch: 1 | total time: 23.18m
step 00899 (60.00%) | loss: 2.414592 | lrm: 0.80 | dt: 1544.28ms | tok/sec: 10,609 | mfu: 0.00 | epoch: 1 | total time: 23.21m
step 00900 (60.07%) | loss: 2.411735 | lrm: 0.80 | dt: 1559.68ms | tok/sec: 10,504 | mfu: 0.00 | epoch: 1 | total time: 23.24m
step 00901 (60.13%) | loss: 2.416000 | lrm: 0.80 | dt: 1575.32ms | tok/sec: 10,400 | mfu: 0.00 | epoch: 1 | total time: 23.26m
step 00902 (60.20%) | loss: 2.463930 | lrm: 0.80 | dt: 1597.12ms | tok/sec: 10,258 | mfu: 0.00 | epoch: 1 | total time: 23.29m
step 00903 (60.27%) | loss: 2.428188 | lrm: 0.79 | dt: 1527.90ms | tok/sec: 10,723 | mfu: 0.00 | epoch: 1 | total time: 23.31m
step 00904 (60.33%) | loss: 2.481903 | lrm: 0.79 | dt: 1534.61ms | tok/sec: 10,676 | mfu: 0.00 | epoch: 1 | total time: 23.34m
step 00905 (60.40%) | loss: 2.460529 | lrm: 0.79 | dt: 1541.63ms | tok/sec: 10,627 | mfu: 0.00 | epoch: 1 | total time: 23.37m
step 00906 (60.47%) | loss: 2.531766 | lrm: 0.79 | dt: 1557.74ms | tok/sec: 10,517 | mfu: 0.00 | epoch: 1 | total time: 23.39m
step 00907 (60.53%) | loss: 2.512229 | lrm: 0.79 | dt: 1539.69ms | tok/sec: 10,641 | mfu: 0.00 | epoch: 1 | total time: 23.42m
step 00908 (60.60%) | loss: 2.479158 | lrm: 0.79 | dt: 1530.22ms | tok/sec: 10,706 | mfu: 0.00 | epoch: 1 | total time: 23.44m
step 00909 (60.67%) | loss: 2.473828 | lrm: 0.79 | dt: 1539.58ms | tok/sec: 10,641 | mfu: 0.00 | epoch: 1 | total time: 23.47m
step 00910 (60.73%) | loss: 2.489352 | lrm: 0.79 | dt: 1524.02ms | tok/sec: 10,750 | mfu: 0.00 | epoch: 1 | total time: 23.49m
step 00911 (60.80%) | loss: 2.467877 | lrm: 0.78 | dt: 1553.17ms | tok/sec: 10,548 | mfu: 0.00 | epoch: 1 | total time: 23.52m
step 00912 (60.87%) | loss: 2.464971 | lrm: 0.78 | dt: 1543.01ms | tok/sec: 10,618 | mfu: 0.00 | epoch: 1 | total time: 23.55m
step 00913 (60.93%) | loss: 2.465191 | lrm: 0.78 | dt: 1539.95ms | tok/sec: 10,639 | mfu: 0.00 | epoch: 1 | total time: 23.57m
step 00914 (61.00%) | loss: 2.482750 | lrm: 0.78 | dt: 1568.97ms | tok/sec: 10,442 | mfu: 0.00 | epoch: 1 | total time: 23.60m
step 00915 (61.07%) | loss: 2.497961 | lrm: 0.78 | dt: 1533.95ms | tok/sec: 10,680 | mfu: 0.00 | epoch: 1 | total time: 23.62m
step 00916 (61.13%) | loss: 2.490945 | lrm: 0.78 | dt: 1559.72ms | tok/sec: 10,504 | mfu: 0.00 | epoch: 1 | total time: 23.65m
step 00917 (61.20%) | loss: 2.483941 | lrm: 0.78 | dt: 1556.23ms | tok/sec: 10,528 | mfu: 0.00 | epoch: 1 | total time: 23.68m
step 00918 (61.27%) | loss: 2.497090 | lrm: 0.77 | dt: 1534.43ms | tok/sec: 10,677 | mfu: 0.00 | epoch: 1 | total time: 23.70m
step 00919 (61.33%) | loss: 2.490191 | lrm: 0.77 | dt: 1597.81ms | tok/sec: 10,254 | mfu: 0.00 | epoch: 1 | total time: 23.73m
step 00920 (61.40%) | loss: 2.476645 | lrm: 0.77 | dt: 1627.77ms | tok/sec: 10,065 | mfu: 0.00 | epoch: 1 | total time: 23.75m
step 00921 (61.47%) | loss: 2.468461 | lrm: 0.77 | dt: 1625.17ms | tok/sec: 10,081 | mfu: 0.00 | epoch: 1 | total time: 23.78m
step 00922 (61.53%) | loss: 2.576745 | lrm: 0.77 | dt: 1666.98ms | tok/sec: 9,828 | mfu: 0.00 | epoch: 1 | total time: 23.81m
step 00923 (61.60%) | loss: 2.600718 | lrm: 0.77 | dt: 1641.38ms | tok/sec: 9,981 | mfu: 0.00 | epoch: 1 | total time: 23.84m
step 00924 (61.67%) | loss: 2.655502 | lrm: 0.77 | dt: 1655.41ms | tok/sec: 9,897 | mfu: 0.00 | epoch: 1 | total time: 23.86m
step 00925 (61.73%) | loss: 2.631243 | lrm: 0.77 | dt: 1628.85ms | tok/sec: 10,058 | mfu: 0.00 | epoch: 1 | total time: 23.89m
step 00926 (61.80%) | loss: 2.610830 | lrm: 0.76 | dt: 1641.34ms | tok/sec: 9,982 | mfu: 0.00 | epoch: 1 | total time: 23.92m
step 00927 (61.87%) | loss: 2.644502 | lrm: 0.76 | dt: 1596.78ms | tok/sec: 10,260 | mfu: 0.00 | epoch: 1 | total time: 23.95m
step 00928 (61.93%) | loss: 2.642875 | lrm: 0.76 | dt: 1611.91ms | tok/sec: 10,164 | mfu: 0.00 | epoch: 1 | total time: 23.97m
step 00929 (62.00%) | loss: 2.610113 | lrm: 0.76 | dt: 1650.94ms | tok/sec: 9,924 | mfu: 0.00 | epoch: 1 | total time: 24.00m
step 00930 (62.07%) | loss: 2.660750 | lrm: 0.76 | dt: 1624.89ms | tok/sec: 10,083 | mfu: 0.00 | epoch: 1 | total time: 24.03m
step 00931 (62.13%) | loss: 2.690307 | lrm: 0.76 | dt: 1612.92ms | tok/sec: 10,157 | mfu: 0.00 | epoch: 1 | total time: 24.05m
step 00932 (62.20%) | loss: 2.658938 | lrm: 0.76 | dt: 1596.75ms | tok/sec: 10,260 | mfu: 0.00 | epoch: 1 | total time: 24.08m
step 00933 (62.27%) | loss: 2.620443 | lrm: 0.75 | dt: 1607.25ms | tok/sec: 10,193 | mfu: 0.00 | epoch: 1 | total time: 24.11m
step 00934 (62.33%) | loss: 2.591196 | lrm: 0.75 | dt: 1640.99ms | tok/sec: 9,984 | mfu: 0.00 | epoch: 1 | total time: 24.13m
step 00935 (62.40%) | loss: 2.563213 | lrm: 0.75 | dt: 1682.73ms | tok/sec: 9,736 | mfu: 0.00 | epoch: 1 | total time: 24.16m
step 00936 (62.47%) | loss: 2.528086 | lrm: 0.75 | dt: 1622.50ms | tok/sec: 10,098 | mfu: 0.00 | epoch: 1 | total time: 24.19m
step 00937 (62.53%) | loss: 2.473712 | lrm: 0.75 | dt: 1575.39ms | tok/sec: 10,399 | mfu: 0.00 | epoch: 1 | total time: 24.22m
step 00938 (62.60%) | loss: 2.498976 | lrm: 0.75 | dt: 1545.93ms | tok/sec: 10,598 | mfu: 0.00 | epoch: 1 | total time: 24.24m
step 00939 (62.67%) | loss: 2.467696 | lrm: 0.75 | dt: 1499.28ms | tok/sec: 10,927 | mfu: 0.00 | epoch: 1 | total time: 24.27m
step 00940 (62.73%) | loss: 2.470930 | lrm: 0.75 | dt: 1496.95ms | tok/sec: 10,944 | mfu: 0.00 | epoch: 1 | total time: 24.29m
step 00941 (62.80%) | loss: 2.449057 | lrm: 0.74 | dt: 1505.40ms | tok/sec: 10,883 | mfu: 0.00 | epoch: 1 | total time: 24.32m
step 00942 (62.87%) | loss: 2.486916 | lrm: 0.74 | dt: 1514.50ms | tok/sec: 10,818 | mfu: 0.00 | epoch: 1 | total time: 24.34m
step 00943 (62.93%) | loss: 2.442046 | lrm: 0.74 | dt: 1508.19ms | tok/sec: 10,863 | mfu: 0.00 | epoch: 1 | total time: 24.37m
step 00944 (63.00%) | loss: 2.451680 | lrm: 0.74 | dt: 1516.98ms | tok/sec: 10,800 | mfu: 0.00 | epoch: 1 | total time: 24.39m
step 00945 (63.07%) | loss: 2.397047 | lrm: 0.74 | dt: 1528.64ms | tok/sec: 10,718 | mfu: 0.00 | epoch: 1 | total time: 24.42m
step 00946 (63.13%) | loss: 2.411614 | lrm: 0.74 | dt: 1518.68ms | tok/sec: 10,788 | mfu: 0.00 | epoch: 1 | total time: 24.44m
step 00947 (63.20%) | loss: 2.407636 | lrm: 0.74 | dt: 1511.46ms | tok/sec: 10,839 | mfu: 0.00 | epoch: 1 | total time: 24.47m
step 00948 (63.27%) | loss: 2.379009 | lrm: 0.73 | dt: 1525.64ms | tok/sec: 10,739 | mfu: 0.00 | epoch: 1 | total time: 24.49m
step 00949 (63.33%) | loss: 2.435756 | lrm: 0.73 | dt: 1490.99ms | tok/sec: 10,988 | mfu: 0.00 | epoch: 1 | total time: 24.52m
step 00950 (63.40%) | loss: 2.416846 | lrm: 0.73 | dt: 1508.75ms | tok/sec: 10,859 | mfu: 0.00 | epoch: 1 | total time: 24.54m
step 00951 (63.47%) | loss: 2.424099 | lrm: 0.73 | dt: 1505.89ms | tok/sec: 10,879 | mfu: 0.00 | epoch: 1 | total time: 24.57m
step 00952 (63.53%) | loss: 2.422621 | lrm: 0.73 | dt: 1523.76ms | tok/sec: 10,752 | mfu: 0.00 | epoch: 1 | total time: 24.59m
step 00953 (63.60%) | loss: 2.434743 | lrm: 0.73 | dt: 1580.29ms | tok/sec: 10,367 | mfu: 0.00 | epoch: 1 | total time: 24.62m
step 00954 (63.67%) | loss: 2.396480 | lrm: 0.73 | dt: 1587.81ms | tok/sec: 10,318 | mfu: 0.00 | epoch: 1 | total time: 24.65m
step 00955 (63.73%) | loss: 2.409380 | lrm: 0.73 | dt: 1567.83ms | tok/sec: 10,450 | mfu: 0.00 | epoch: 1 | total time: 24.67m
step 00956 (63.80%) | loss: 2.396210 | lrm: 0.72 | dt: 1555.25ms | tok/sec: 10,534 | mfu: 0.00 | epoch: 1 | total time: 24.70m
step 00957 (63.87%) | loss: 2.364708 | lrm: 0.72 | dt: 1506.40ms | tok/sec: 10,876 | mfu: 0.00 | epoch: 1 | total time: 24.72m
step 00958 (63.93%) | loss: 2.396249 | lrm: 0.72 | dt: 1534.82ms | tok/sec: 10,674 | mfu: 0.00 | epoch: 1 | total time: 24.75m
step 00959 (64.00%) | loss: 2.445648 | lrm: 0.72 | dt: 1555.87ms | tok/sec: 10,530 | mfu: 0.00 | epoch: 1 | total time: 24.78m
step 00960 (64.07%) | loss: 2.372667 | lrm: 0.72 | dt: 1515.47ms | tok/sec: 10,811 | mfu: 0.00 | epoch: 1 | total time: 24.80m
step 00961 (64.13%) | loss: 2.415226 | lrm: 0.72 | dt: 1537.81ms | tok/sec: 10,654 | mfu: 0.00 | epoch: 1 | total time: 24.83m
step 00962 (64.20%) | loss: 2.437632 | lrm: 0.72 | dt: 1599.56ms | tok/sec: 10,242 | mfu: 0.00 | epoch: 1 | total time: 24.85m
step 00963 (64.27%) | loss: 2.495759 | lrm: 0.71 | dt: 1621.00ms | tok/sec: 10,107 | mfu: 0.00 | epoch: 1 | total time: 24.88m
step 00964 (64.33%) | loss: 2.484504 | lrm: 0.71 | dt: 1667.77ms | tok/sec: 9,823 | mfu: 0.00 | epoch: 1 | total time: 24.91m
step 00965 (64.40%) | loss: 2.522604 | lrm: 0.71 | dt: 1638.47ms | tok/sec: 9,999 | mfu: 0.00 | epoch: 1 | total time: 24.94m
step 00966 (64.47%) | loss: 2.520195 | lrm: 0.71 | dt: 1606.77ms | tok/sec: 10,196 | mfu: 0.00 | epoch: 1 | total time: 24.96m
step 00967 (64.53%) | loss: 2.536431 | lrm: 0.71 | dt: 1674.22ms | tok/sec: 9,786 | mfu: 0.00 | epoch: 1 | total time: 24.99m
step 00968 (64.60%) | loss: 2.564180 | lrm: 0.71 | dt: 1638.94ms | tok/sec: 9,996 | mfu: 0.00 | epoch: 1 | total time: 25.02m
step 00969 (64.67%) | loss: 2.508673 | lrm: 0.71 | dt: 1653.63ms | tok/sec: 9,907 | mfu: 0.00 | epoch: 1 | total time: 25.04m
step 00970 (64.73%) | loss: 2.459162 | lrm: 0.71 | dt: 1692.75ms | tok/sec: 9,678 | mfu: 0.00 | epoch: 1 | total time: 25.07m
step 00971 (64.80%) | loss: 2.436362 | lrm: 0.70 | dt: 1603.59ms | tok/sec: 10,217 | mfu: 0.00 | epoch: 1 | total time: 25.10m
step 00972 (64.87%) | loss: 2.428892 | lrm: 0.70 | dt: 1640.35ms | tok/sec: 9,988 | mfu: 0.00 | epoch: 1 | total time: 25.13m
step 00973 (64.93%) | loss: 2.400003 | lrm: 0.70 | dt: 1614.81ms | tok/sec: 10,146 | mfu: 0.00 | epoch: 1 | total time: 25.15m
step 00974 (65.00%) | loss: 2.467873 | lrm: 0.70 | dt: 1585.93ms | tok/sec: 10,330 | mfu: 0.00 | epoch: 1 | total time: 25.18m
step 00975 (65.07%) | loss: 2.458332 | lrm: 0.70 | dt: 1682.36ms | tok/sec: 9,738 | mfu: 0.00 | epoch: 1 | total time: 25.21m
step 00976 (65.13%) | loss: 2.451184 | lrm: 0.70 | dt: 1634.60ms | tok/sec: 10,023 | mfu: 0.00 | epoch: 1 | total time: 25.24m
step 00977 (65.20%) | loss: 2.447732 | lrm: 0.70 | dt: 1512.48ms | tok/sec: 10,832 | mfu: 0.00 | epoch: 1 | total time: 25.26m
step 00978 (65.27%) | loss: 2.400340 | lrm: 0.69 | dt: 1593.21ms | tok/sec: 10,283 | mfu: 0.00 | epoch: 1 | total time: 25.29m
step 00979 (65.33%) | loss: 2.401679 | lrm: 0.69 | dt: 1533.14ms | tok/sec: 10,686 | mfu: 0.00 | epoch: 1 | total time: 25.31m
step 00980 (65.40%) | loss: 2.405541 | lrm: 0.69 | dt: 1521.81ms | tok/sec: 10,766 | mfu: 0.00 | epoch: 1 | total time: 25.34m
step 00981 (65.47%) | loss: 2.357467 | lrm: 0.69 | dt: 1571.04ms | tok/sec: 10,428 | mfu: 0.00 | epoch: 1 | total time: 25.36m
step 00982 (65.53%) | loss: 2.374232 | lrm: 0.69 | dt: 1517.02ms | tok/sec: 10,800 | mfu: 0.00 | epoch: 1 | total time: 25.39m
step 00983 (65.60%) | loss: 2.346272 | lrm: 0.69 | dt: 1559.95ms | tok/sec: 10,502 | mfu: 0.00 | epoch: 1 | total time: 25.42m
step 00984 (65.67%) | loss: 2.303306 | lrm: 0.69 | dt: 1559.49ms | tok/sec: 10,505 | mfu: 0.00 | epoch: 1 | total time: 25.44m
step 00985 (65.73%) | loss: 2.269850 | lrm: 0.69 | dt: 1513.27ms | tok/sec: 10,826 | mfu: 0.00 | epoch: 1 | total time: 25.47m
step 00986 (65.80%) | loss: 2.278315 | lrm: 0.68 | dt: 1512.96ms | tok/sec: 10,829 | mfu: 0.00 | epoch: 1 | total time: 25.49m
step 00987 (65.87%) | loss: 2.301948 | lrm: 0.68 | dt: 1537.20ms | tok/sec: 10,658 | mfu: 0.00 | epoch: 1 | total time: 25.52m
step 00988 (65.93%) | loss: 2.309677 | lrm: 0.68 | dt: 1572.17ms | tok/sec: 10,421 | mfu: 0.00 | epoch: 1 | total time: 25.54m
step 00989 (66.00%) | loss: 2.351740 | lrm: 0.68 | dt: 1743.57ms | tok/sec: 9,396 | mfu: 0.00 | epoch: 1 | total time: 25.57m
step 00990 (66.07%) | loss: 2.345083 | lrm: 0.68 | dt: 1517.30ms | tok/sec: 10,798 | mfu: 0.00 | epoch: 1 | total time: 25.60m
step 00991 (66.13%) | loss: 2.376066 | lrm: 0.68 | dt: 1505.18ms | tok/sec: 10,885 | mfu: 0.00 | epoch: 1 | total time: 25.62m
step 00992 (66.20%) | loss: 2.367534 | lrm: 0.68 | dt: 1505.82ms | tok/sec: 10,880 | mfu: 0.00 | epoch: 1 | total time: 25.65m
step 00993 (66.27%) | loss: 2.421230 | lrm: 0.67 | dt: 1531.93ms | tok/sec: 10,695 | mfu: 0.00 | epoch: 1 | total time: 25.67m
step 00994 (66.33%) | loss: 2.433877 | lrm: 0.67 | dt: 1609.07ms | tok/sec: 10,182 | mfu: 0.00 | epoch: 1 | total time: 25.70m
step 00995 (66.40%) | loss: 2.387872 | lrm: 0.67 | dt: 1513.55ms | tok/sec: 10,824 | mfu: 0.00 | epoch: 1 | total time: 25.73m
step 00996 (66.47%) | loss: 2.371026 | lrm: 0.67 | dt: 1501.26ms | tok/sec: 10,913 | mfu: 0.00 | epoch: 1 | total time: 25.75m
step 00997 (66.53%) | loss: 2.383563 | lrm: 0.67 | dt: 1496.39ms | tok/sec: 10,948 | mfu: 0.00 | epoch: 1 | total time: 25.78m
step 00998 (66.60%) | loss: 2.363582 | lrm: 0.67 | dt: 1529.20ms | tok/sec: 10,714 | mfu: 0.00 | epoch: 1 | total time: 25.80m
step 00999 (66.67%) | loss: 2.420187 | lrm: 0.67 | dt: 1574.71ms | tok/sec: 10,404 | mfu: 0.00 | epoch: 1 | total time: 25.83m
step 01000 (66.73%) | loss: 2.418937 | lrm: 0.67 | dt: 1533.33ms | tok/sec: 10,685 | mfu: 0.00 | epoch: 1 | total time: 25.85m
Step 01000 | Validation bpb: 0.8483
step 01001 (66.80%) | loss: 2.423689 | lrm: 0.66 | dt: 1489.18ms | tok/sec: 11,002 | mfu: 0.00 | epoch: 1 | total time: 25.88m
step 01002 (66.87%) | loss: 2.420547 | lrm: 0.66 | dt: 1521.31ms | tok/sec: 10,769 | mfu: 0.00 | epoch: 1 | total time: 25.90m
step 01003 (66.93%) | loss: 2.349110 | lrm: 0.66 | dt: 1508.75ms | tok/sec: 10,859 | mfu: 0.00 | epoch: 1 | total time: 25.93m
step 01004 (67.00%) | loss: 2.337367 | lrm: 0.66 | dt: 1505.09ms | tok/sec: 10,885 | mfu: 0.00 | epoch: 1 | total time: 25.95m
step 01005 (67.07%) | loss: 2.317232 | lrm: 0.66 | dt: 1536.83ms | tok/sec: 10,660 | mfu: 0.00 | epoch: 1 | total time: 25.98m
step 01006 (67.13%) | loss: 2.312525 | lrm: 0.66 | dt: 1503.48ms | tok/sec: 10,897 | mfu: 0.00 | epoch: 1 | total time: 26.00m
step 01007 (67.20%) | loss: 2.290884 | lrm: 0.66 | dt: 1520.93ms | tok/sec: 10,772 | mfu: 0.00 | epoch: 1 | total time: 26.03m
step 01008 (67.27%) | loss: 2.328919 | lrm: 0.65 | dt: 1509.93ms | tok/sec: 10,850 | mfu: 0.00 | epoch: 1 | total time: 26.06m
step 01009 (67.33%) | loss: 2.328042 | lrm: 0.65 | dt: 1486.32ms | tok/sec: 11,023 | mfu: 0.00 | epoch: 1 | total time: 26.08m
step 01010 (67.40%) | loss: 2.337021 | lrm: 0.65 | dt: 1505.40ms | tok/sec: 10,883 | mfu: 0.00 | epoch: 1 | total time: 26.11m
step 01011 (67.47%) | loss: 2.342883 | lrm: 0.65 | dt: 1482.47ms | tok/sec: 11,051 | mfu: 0.00 | epoch: 1 | total time: 26.13m
step 01012 (67.53%) | loss: 2.342077 | lrm: 0.65 | dt: 1521.74ms | tok/sec: 10,766 | mfu: 0.00 | epoch: 1 | total time: 26.16m
step 01013 (67.60%) | loss: 2.342804 | lrm: 0.65 | dt: 1481.17ms | tok/sec: 11,061 | mfu: 0.00 | epoch: 1 | total time: 26.18m
step 01014 (67.67%) | loss: 2.347181 | lrm: 0.65 | dt: 1522.83ms | tok/sec: 10,758 | mfu: 0.00 | epoch: 1 | total time: 26.21m
step 01015 (67.73%) | loss: 2.323359 | lrm: 0.65 | dt: 1500.38ms | tok/sec: 10,919 | mfu: 0.00 | epoch: 1 | total time: 26.23m
step 01016 (67.80%) | loss: 2.362453 | lrm: 0.64 | dt: 1539.61ms | tok/sec: 10,641 | mfu: 0.00 | epoch: 1 | total time: 26.26m
step 01017 (67.87%) | loss: 2.389226 | lrm: 0.64 | dt: 1505.48ms | tok/sec: 10,882 | mfu: 0.00 | epoch: 1 | total time: 26.28m
step 01018 (67.93%) | loss: 2.383459 | lrm: 0.64 | dt: 1524.97ms | tok/sec: 10,743 | mfu: 0.00 | epoch: 1 | total time: 26.31m
step 01019 (68.00%) | loss: 2.392079 | lrm: 0.64 | dt: 1501.04ms | tok/sec: 10,915 | mfu: 0.00 | epoch: 1 | total time: 26.33m
step 01020 (68.07%) | loss: 2.351132 | lrm: 0.64 | dt: 1516.78ms | tok/sec: 10,801 | mfu: 0.00 | epoch: 1 | total time: 26.36m
step 01021 (68.13%) | loss: 2.346650 | lrm: 0.64 | dt: 1513.51ms | tok/sec: 10,825 | mfu: 0.00 | epoch: 1 | total time: 26.38m
step 01022 (68.20%) | loss: 2.325925 | lrm: 0.64 | dt: 1530.77ms | tok/sec: 10,703 | mfu: 0.00 | epoch: 1 | total time: 26.41m
step 01023 (68.27%) | loss: 2.272545 | lrm: 0.63 | dt: 1517.15ms | tok/sec: 10,799 | mfu: 0.00 | epoch: 1 | total time: 26.43m
step 01024 (68.33%) | loss: 2.245445 | lrm: 0.63 | dt: 1513.17ms | tok/sec: 10,827 | mfu: 0.00 | epoch: 1 | total time: 26.46m
step 01025 (68.40%) | loss: 2.217132 | lrm: 0.63 | dt: 1507.41ms | tok/sec: 10,868 | mfu: 0.00 | epoch: 1 | total time: 26.48m
step 01026 (68.47%) | loss: 2.228241 | lrm: 0.63 | dt: 1501.54ms | tok/sec: 10,911 | mfu: 0.00 | epoch: 1 | total time: 26.51m
step 01027 (68.53%) | loss: 2.222990 | lrm: 0.63 | dt: 1570.49ms | tok/sec: 10,432 | mfu: 0.00 | epoch: 1 | total time: 26.53m
step 01028 (68.60%) | loss: 2.239057 | lrm: 0.63 | dt: 1494.37ms | tok/sec: 10,963 | mfu: 0.00 | epoch: 1 | total time: 26.56m
step 01029 (68.67%) | loss: 2.254067 | lrm: 0.63 | dt: 1529.23ms | tok/sec: 10,713 | mfu: 0.00 | epoch: 1 | total time: 26.58m
step 01030 (68.73%) | loss: 2.243183 | lrm: 0.63 | dt: 1527.02ms | tok/sec: 10,729 | mfu: 0.00 | epoch: 1 | total time: 26.61m
step 01031 (68.80%) | loss: 2.253080 | lrm: 0.62 | dt: 1543.93ms | tok/sec: 10,611 | mfu: 0.00 | epoch: 1 | total time: 26.64m
step 01032 (68.87%) | loss: 2.196311 | lrm: 0.62 | dt: 1509.27ms | tok/sec: 10,855 | mfu: 0.00 | epoch: 1 | total time: 26.66m
step 01033 (68.93%) | loss: 2.182757 | lrm: 0.62 | dt: 1524.33ms | tok/sec: 10,748 | mfu: 0.00 | epoch: 1 | total time: 26.69m
step 01034 (69.00%) | loss: 2.196484 | lrm: 0.62 | dt: 1521.16ms | tok/sec: 10,770 | mfu: 0.00 | epoch: 1 | total time: 26.71m
step 01035 (69.07%) | loss: 2.191742 | lrm: 0.62 | dt: 1552.69ms | tok/sec: 10,552 | mfu: 0.00 | epoch: 1 | total time: 26.74m
step 01036 (69.13%) | loss: 2.220368 | lrm: 0.62 | dt: 1529.68ms | tok/sec: 10,710 | mfu: 0.00 | epoch: 1 | total time: 26.76m
step 01037 (69.20%) | loss: 2.319220 | lrm: 0.62 | dt: 1532.15ms | tok/sec: 10,693 | mfu: 0.00 | epoch: 1 | total time: 26.79m
step 01038 (69.27%) | loss: 2.299905 | lrm: 0.61 | dt: 1511.69ms | tok/sec: 10,838 | mfu: 0.00 | epoch: 1 | total time: 26.81m
step 01039 (69.33%) | loss: 2.313615 | lrm: 0.61 | dt: 1513.99ms | tok/sec: 10,821 | mfu: 0.00 | epoch: 1 | total time: 26.84m
step 01040 (69.40%) | loss: 2.313257 | lrm: 0.61 | dt: 1524.44ms | tok/sec: 10,747 | mfu: 0.00 | epoch: 1 | total time: 26.86m
step 01041 (69.47%) | loss: 2.315079 | lrm: 0.61 | dt: 1490.23ms | tok/sec: 10,994 | mfu: 0.00 | epoch: 1 | total time: 26.89m
step 01042 (69.53%) | loss: 2.322481 | lrm: 0.61 | dt: 1522.55ms | tok/sec: 10,760 | mfu: 0.00 | epoch: 1 | total time: 26.91m
step 01043 (69.60%) | loss: 2.371319 | lrm: 0.61 | dt: 1516.71ms | tok/sec: 10,802 | mfu: 0.00 | epoch: 1 | total time: 26.94m
step 01044 (69.67%) | loss: 2.325966 | lrm: 0.61 | dt: 1511.38ms | tok/sec: 10,840 | mfu: 0.00 | epoch: 1 | total time: 26.97m
step 01045 (69.73%) | loss: 2.360958 | lrm: 0.61 | dt: 1533.07ms | tok/sec: 10,687 | mfu: 0.00 | epoch: 1 | total time: 26.99m
step 01046 (69.80%) | loss: 2.369659 | lrm: 0.60 | dt: 1533.18ms | tok/sec: 10,686 | mfu: 0.00 | epoch: 1 | total time: 27.02m
step 01047 (69.87%) | loss: 2.373295 | lrm: 0.60 | dt: 1519.20ms | tok/sec: 10,784 | mfu: 0.00 | epoch: 1 | total time: 27.04m
step 01048 (69.93%) | loss: 2.397920 | lrm: 0.60 | dt: 1511.18ms | tok/sec: 10,841 | mfu: 0.00 | epoch: 1 | total time: 27.07m
step 01049 (70.00%) | loss: 2.376373 | lrm: 0.60 | dt: 1529.97ms | tok/sec: 10,708 | mfu: 0.00 | epoch: 1 | total time: 27.09m
step 01050 (70.07%) | loss: 2.354106 | lrm: 0.60 | dt: 1507.47ms | tok/sec: 10,868 | mfu: 0.00 | epoch: 1 | total time: 27.12m
step 01051 (70.13%) | loss: 2.379413 | lrm: 0.60 | dt: 1515.11ms | tok/sec: 10,813 | mfu: 0.00 | epoch: 1 | total time: 27.14m
step 01052 (70.20%) | loss: 2.331680 | lrm: 0.60 | dt: 1512.98ms | tok/sec: 10,828 | mfu: 0.00 | epoch: 1 | total time: 27.17m
step 01053 (70.27%) | loss: 2.290267 | lrm: 0.59 | dt: 1519.74ms | tok/sec: 10,780 | mfu: 0.00 | epoch: 1 | total time: 27.19m
step 01054 (70.33%) | loss: 2.342404 | lrm: 0.59 | dt: 1519.51ms | tok/sec: 10,782 | mfu: 0.00 | epoch: 1 | total time: 27.22m
step 01055 (70.40%) | loss: 2.383651 | lrm: 0.59 | dt: 1503.27ms | tok/sec: 10,898 | mfu: 0.00 | epoch: 1 | total time: 27.24m
step 01056 (70.47%) | loss: 2.349686 | lrm: 0.59 | dt: 1545.68ms | tok/sec: 10,599 | mfu: 0.00 | epoch: 1 | total time: 27.27m
step 01057 (70.53%) | loss: 2.377372 | lrm: 0.59 | dt: 1522.31ms | tok/sec: 10,762 | mfu: 0.00 | epoch: 1 | total time: 27.29m
step 01058 (70.60%) | loss: 2.393810 | lrm: 0.59 | dt: 1514.78ms | tok/sec: 10,816 | mfu: 0.00 | epoch: 1 | total time: 27.32m
step 01059 (70.67%) | loss: 2.431911 | lrm: 0.59 | dt: 1520.53ms | tok/sec: 10,775 | mfu: 0.00 | epoch: 1 | total time: 27.35m
step 01060 (70.73%) | loss: 2.409806 | lrm: 0.59 | dt: 1534.79ms | tok/sec: 10,675 | mfu: 0.00 | epoch: 1 | total time: 27.37m
step 01061 (70.80%) | loss: 2.424065 | lrm: 0.58 | dt: 1501.84ms | tok/sec: 10,909 | mfu: 0.00 | epoch: 1 | total time: 27.40m
step 01062 (70.87%) | loss: 2.440808 | lrm: 0.58 | dt: 1545.78ms | tok/sec: 10,599 | mfu: 0.00 | epoch: 1 | total time: 27.42m
step 01063 (70.93%) | loss: 2.488960 | lrm: 0.58 | dt: 1559.03ms | tok/sec: 10,509 | mfu: 0.00 | epoch: 1 | total time: 27.45m
step 01064 (71.00%) | loss: 2.516227 | lrm: 0.58 | dt: 1579.82ms | tok/sec: 10,370 | mfu: 0.00 | epoch: 1 | total time: 27.47m
step 01065 (71.07%) | loss: 2.571607 | lrm: 0.58 | dt: 1600.38ms | tok/sec: 10,237 | mfu: 0.00 | epoch: 1 | total time: 27.50m
step 01066 (71.13%) | loss: 2.524014 | lrm: 0.58 | dt: 1621.22ms | tok/sec: 10,105 | mfu: 0.00 | epoch: 1 | total time: 27.53m
step 01067 (71.20%) | loss: 2.499731 | lrm: 0.58 | dt: 1525.87ms | tok/sec: 10,737 | mfu: 0.00 | epoch: 1 | total time: 27.55m
step 01068 (71.27%) | loss: 2.433826 | lrm: 0.57 | dt: 1533.26ms | tok/sec: 10,685 | mfu: 0.00 | epoch: 1 | total time: 27.58m
step 01069 (71.33%) | loss: 2.451533 | lrm: 0.57 | dt: 1454.09ms | tok/sec: 11,267 | mfu: 0.00 | epoch: 1 | total time: 27.60m
step 01070 (71.40%) | loss: 2.411789 | lrm: 0.57 | dt: 1524.40ms | tok/sec: 10,747 | mfu: 0.00 | epoch: 1 | total time: 27.63m
step 01071 (71.47%) | loss: 2.365256 | lrm: 0.57 | dt: 1520.88ms | tok/sec: 10,772 | mfu: 0.00 | epoch: 1 | total time: 27.65m
step 01072 (71.53%) | loss: 2.289308 | lrm: 0.57 | dt: 1506.65ms | tok/sec: 10,874 | mfu: 0.00 | epoch: 1 | total time: 27.68m
step 01073 (71.60%) | loss: 2.296925 | lrm: 0.57 | dt: 1531.42ms | tok/sec: 10,698 | mfu: 0.00 | epoch: 1 | total time: 27.70m
step 01074 (71.67%) | loss: 2.242555 | lrm: 0.57 | dt: 1513.31ms | tok/sec: 10,826 | mfu: 0.00 | epoch: 1 | total time: 27.73m
step 01075 (71.73%) | loss: 2.253531 | lrm: 0.57 | dt: 1520.42ms | tok/sec: 10,775 | mfu: 0.00 | epoch: 1 | total time: 27.75m
step 01076 (71.80%) | loss: 2.272585 | lrm: 0.56 | dt: 1527.06ms | tok/sec: 10,729 | mfu: 0.00 | epoch: 1 | total time: 27.78m
step 01077 (71.87%) | loss: 2.213287 | lrm: 0.56 | dt: 1515.97ms | tok/sec: 10,807 | mfu: 0.00 | epoch: 1 | total time: 27.81m
step 01078 (71.93%) | loss: 2.231105 | lrm: 0.56 | dt: 1526.60ms | tok/sec: 10,732 | mfu: 0.00 | epoch: 1 | total time: 27.83m
step 01079 (72.00%) | loss: 2.252314 | lrm: 0.56 | dt: 1515.63ms | tok/sec: 10,810 | mfu: 0.00 | epoch: 1 | total time: 27.86m
step 01080 (72.07%) | loss: 2.211047 | lrm: 0.56 | dt: 1535.80ms | tok/sec: 10,668 | mfu: 0.00 | epoch: 1 | total time: 27.88m
step 01081 (72.13%) | loss: 2.222454 | lrm: 0.56 | dt: 1506.14ms | tok/sec: 10,878 | mfu: 0.00 | epoch: 1 | total time: 27.91m
step 01082 (72.20%) | loss: 2.216552 | lrm: 0.56 | dt: 1522.19ms | tok/sec: 10,763 | mfu: 0.00 | epoch: 1 | total time: 27.93m
step 01083 (72.27%) | loss: 2.174751 | lrm: 0.55 | dt: 1528.31ms | tok/sec: 10,720 | mfu: 0.00 | epoch: 1 | total time: 27.96m
step 01084 (72.33%) | loss: 2.216961 | lrm: 0.55 | dt: 1512.32ms | tok/sec: 10,833 | mfu: 0.00 | epoch: 1 | total time: 27.98m
step 01085 (72.40%) | loss: 2.221432 | lrm: 0.55 | dt: 1522.29ms | tok/sec: 10,762 | mfu: 0.00 | epoch: 1 | total time: 28.01m
step 01086 (72.47%) | loss: 2.249143 | lrm: 0.55 | dt: 1534.37ms | tok/sec: 10,678 | mfu: 0.00 | epoch: 1 | total time: 28.03m
step 01087 (72.53%) | loss: 2.294339 | lrm: 0.55 | dt: 1622.70ms | tok/sec: 10,096 | mfu: 0.00 | epoch: 1 | total time: 28.06m
step 01088 (72.60%) | loss: 2.339942 | lrm: 0.55 | dt: 1541.22ms | tok/sec: 10,630 | mfu: 0.00 | epoch: 1 | total time: 28.09m
step 01089 (72.67%) | loss: 2.349120 | lrm: 0.55 | dt: 1566.20ms | tok/sec: 10,460 | mfu: 0.00 | epoch: 1 | total time: 28.11m
step 01090 (72.73%) | loss: 2.293622 | lrm: 0.55 | dt: 1584.02ms | tok/sec: 10,343 | mfu: 0.00 | epoch: 1 | total time: 28.14m
step 01091 (72.80%) | loss: 2.270536 | lrm: 0.54 | dt: 1588.21ms | tok/sec: 10,316 | mfu: 0.00 | epoch: 1 | total time: 28.17m
step 01092 (72.87%) | loss: 2.298774 | lrm: 0.54 | dt: 1664.81ms | tok/sec: 9,841 | mfu: 0.00 | epoch: 1 | total time: 28.19m
step 01093 (72.93%) | loss: 2.343297 | lrm: 0.54 | dt: 1618.40ms | tok/sec: 10,123 | mfu: 0.00 | epoch: 1 | total time: 28.22m
step 01094 (73.00%) | loss: 2.305013 | lrm: 0.54 | dt: 1597.37ms | tok/sec: 10,256 | mfu: 0.00 | epoch: 1 | total time: 28.25m
step 01095 (73.07%) | loss: 2.241029 | lrm: 0.54 | dt: 1591.71ms | tok/sec: 10,293 | mfu: 0.00 | epoch: 1 | total time: 28.27m
step 01096 (73.13%) | loss: 2.265088 | lrm: 0.54 | dt: 1549.11ms | tok/sec: 10,576 | mfu: 0.00 | epoch: 1 | total time: 28.30m
step 01097 (73.20%) | loss: 2.285222 | lrm: 0.54 | dt: 1533.87ms | tok/sec: 10,681 | mfu: 0.00 | epoch: 1 | total time: 28.32m
step 01098 (73.27%) | loss: 2.316698 | lrm: 0.53 | dt: 1506.00ms | tok/sec: 10,879 | mfu: 0.00 | epoch: 1 | total time: 28.35m
step 01099 (73.33%) | loss: 2.327646 | lrm: 0.53 | dt: 1528.16ms | tok/sec: 10,721 | mfu: 0.00 | epoch: 1 | total time: 28.38m
step 01100 (73.40%) | loss: 2.320218 | lrm: 0.53 | dt: 1524.91ms | tok/sec: 10,744 | mfu: 0.00 | epoch: 1 | total time: 28.40m
step 01101 (73.47%) | loss: 2.288746 | lrm: 0.53 | dt: 1508.28ms | tok/sec: 10,862 | mfu: 0.00 | epoch: 1 | total time: 28.43m
step 01102 (73.53%) | loss: 2.298780 | lrm: 0.53 | dt: 1519.50ms | tok/sec: 10,782 | mfu: 0.00 | epoch: 1 | total time: 28.45m
step 01103 (73.60%) | loss: 2.295591 | lrm: 0.53 | dt: 1493.02ms | tok/sec: 10,973 | mfu: 0.00 | epoch: 1 | total time: 28.48m
step 01104 (73.67%) | loss: 2.292834 | lrm: 0.53 | dt: 1519.22ms | tok/sec: 10,784 | mfu: 0.00 | epoch: 1 | total time: 28.50m
step 01105 (73.73%) | loss: 2.293294 | lrm: 0.53 | dt: 1535.40ms | tok/sec: 10,670 | mfu: 0.00 | epoch: 1 | total time: 28.53m
step 01106 (73.80%) | loss: 2.232086 | lrm: 0.52 | dt: 1546.96ms | tok/sec: 10,591 | mfu: 0.00 | epoch: 1 | total time: 28.55m
step 01107 (73.87%) | loss: 2.251651 | lrm: 0.52 | dt: 1511.63ms | tok/sec: 10,838 | mfu: 0.00 | epoch: 1 | total time: 28.58m
step 01108 (73.93%) | loss: 2.251587 | lrm: 0.52 | dt: 1495.69ms | tok/sec: 10,954 | mfu: 0.00 | epoch: 1 | total time: 28.60m
step 01109 (74.00%) | loss: 2.269016 | lrm: 0.52 | dt: 1512.48ms | tok/sec: 10,832 | mfu: 0.00 | epoch: 1 | total time: 28.63m
step 01110 (74.07%) | loss: 2.236282 | lrm: 0.52 | dt: 1494.41ms | tok/sec: 10,963 | mfu: 0.00 | epoch: 1 | total time: 28.65m
step 01111 (74.13%) | loss: 2.259047 | lrm: 0.52 | dt: 1541.39ms | tok/sec: 10,629 | mfu: 0.00 | epoch: 1 | total time: 28.68m
step 01112 (74.20%) | loss: 2.238418 | lrm: 0.52 | dt: 1563.74ms | tok/sec: 10,477 | mfu: 0.00 | epoch: 1 | total time: 28.70m
step 01113 (74.27%) | loss: 2.230673 | lrm: 0.51 | dt: 1531.11ms | tok/sec: 10,700 | mfu: 0.00 | epoch: 1 | total time: 28.73m
step 01114 (74.33%) | loss: 2.210677 | lrm: 0.51 | dt: 1502.68ms | tok/sec: 10,903 | mfu: 0.00 | epoch: 1 | total time: 28.76m
step 01115 (74.40%) | loss: 2.204548 | lrm: 0.51 | dt: 1528.88ms | tok/sec: 10,716 | mfu: 0.00 | epoch: 1 | total time: 28.78m
step 01116 (74.47%) | loss: 2.249534 | lrm: 0.51 | dt: 1519.56ms | tok/sec: 10,782 | mfu: 0.00 | epoch: 1 | total time: 28.81m
step 01117 (74.53%) | loss: 2.237143 | lrm: 0.51 | dt: 1545.21ms | tok/sec: 10,603 | mfu: 0.00 | epoch: 1 | total time: 28.83m
step 01118 (74.60%) | loss: 2.251012 | lrm: 0.51 | dt: 1611.06ms | tok/sec: 10,169 | mfu: 0.00 | epoch: 1 | total time: 28.86m
step 01119 (74.67%) | loss: 2.245847 | lrm: 0.51 | dt: 1543.03ms | tok/sec: 10,618 | mfu: 0.00 | epoch: 1 | total time: 28.88m
step 01120 (74.73%) | loss: 2.241599 | lrm: 0.51 | dt: 1544.90ms | tok/sec: 10,605 | mfu: 0.00 | epoch: 1 | total time: 28.91m
step 01121 (74.80%) | loss: 2.202734 | lrm: 0.50 | dt: 1573.74ms | tok/sec: 10,410 | mfu: 0.00 | epoch: 1 | total time: 28.94m
step 01122 (74.87%) | loss: 2.263993 | lrm: 0.50 | dt: 1591.51ms | tok/sec: 10,294 | mfu: 0.00 | epoch: 1 | total time: 28.96m
step 01123 (74.93%) | loss: 2.306717 | lrm: 0.50 | dt: 1637.71ms | tok/sec: 10,004 | mfu: 0.00 | epoch: 1 | total time: 28.99m
step 01124 (75.00%) | loss: 2.289353 | lrm: 0.50 | dt: 1551.18ms | tok/sec: 10,562 | mfu: 0.00 | epoch: 1 | total time: 29.02m
step 01125 (75.07%) | loss: 2.278986 | lrm: 0.50 | dt: 1580.39ms | tok/sec: 10,367 | mfu: 0.00 | epoch: 1 | total time: 29.04m
step 01126 (75.13%) | loss: 2.227315 | lrm: 0.50 | dt: 1548.90ms | tok/sec: 10,577 | mfu: 0.00 | epoch: 1 | total time: 29.07m
step 01127 (75.20%) | loss: 2.210742 | lrm: 0.50 | dt: 1519.76ms | tok/sec: 10,780 | mfu: 0.00 | epoch: 1 | total time: 29.09m
step 01128 (75.27%) | loss: 2.190515 | lrm: 0.49 | dt: 1610.46ms | tok/sec: 10,173 | mfu: 0.00 | epoch: 1 | total time: 29.12m
step 01129 (75.33%) | loss: 2.203914 | lrm: 0.49 | dt: 1579.55ms | tok/sec: 10,372 | mfu: 0.00 | epoch: 1 | total time: 29.15m
step 01130 (75.40%) | loss: 2.231641 | lrm: 0.49 | dt: 1572.06ms | tok/sec: 10,421 | mfu: 0.00 | epoch: 1 | total time: 29.17m
step 01131 (75.47%) | loss: 2.274303 | lrm: 0.49 | dt: 1570.03ms | tok/sec: 10,435 | mfu: 0.00 | epoch: 1 | total time: 29.20m
step 01132 (75.53%) | loss: 2.286254 | lrm: 0.49 | dt: 1484.69ms | tok/sec: 11,035 | mfu: 0.00 | epoch: 1 | total time: 29.22m
step 01133 (75.60%) | loss: 2.236371 | lrm: 0.49 | dt: 1602.45ms | tok/sec: 10,224 | mfu: 0.00 | epoch: 1 | total time: 29.25m
step 01134 (75.67%) | loss: 2.244166 | lrm: 0.49 | dt: 1568.58ms | tok/sec: 10,445 | mfu: 0.00 | epoch: 1 | total time: 29.28m
step 01135 (75.73%) | loss: 2.297613 | lrm: 0.49 | dt: 1532.04ms | tok/sec: 10,694 | mfu: 0.00 | epoch: 1 | total time: 29.30m
step 01136 (75.80%) | loss: 2.297163 | lrm: 0.48 | dt: 1577.58ms | tok/sec: 10,385 | mfu: 0.00 | epoch: 1 | total time: 29.33m
step 01137 (75.87%) | loss: 2.264649 | lrm: 0.48 | dt: 1626.96ms | tok/sec: 10,070 | mfu: 0.00 | epoch: 1 | total time: 29.36m
step 01138 (75.93%) | loss: 2.229852 | lrm: 0.48 | dt: 1664.88ms | tok/sec: 9,840 | mfu: 0.00 | epoch: 1 | total time: 29.38m
step 01139 (76.00%) | loss: 2.237071 | lrm: 0.48 | dt: 1612.97ms | tok/sec: 10,157 | mfu: 0.00 | epoch: 1 | total time: 29.41m
step 01140 (76.07%) | loss: 2.243236 | lrm: 0.48 | dt: 1580.79ms | tok/sec: 10,364 | mfu: 0.00 | epoch: 1 | total time: 29.44m
step 01141 (76.13%) | loss: 2.233169 | lrm: 0.48 | dt: 1599.54ms | tok/sec: 10,242 | mfu: 0.00 | epoch: 1 | total time: 29.46m
step 01142 (76.20%) | loss: 2.180758 | lrm: 0.48 | dt: 1602.44ms | tok/sec: 10,224 | mfu: 0.00 | epoch: 1 | total time: 29.49m
step 01143 (76.27%) | loss: 2.208505 | lrm: 0.47 | dt: 1652.81ms | tok/sec: 9,912 | mfu: 0.00 | epoch: 1 | total time: 29.52m
step 01144 (76.33%) | loss: 2.273544 | lrm: 0.47 | dt: 1630.27ms | tok/sec: 10,049 | mfu: 0.00 | epoch: 1 | total time: 29.54m
step 01145 (76.40%) | loss: 2.247774 | lrm: 0.47 | dt: 1648.04ms | tok/sec: 9,941 | mfu: 0.00 | epoch: 1 | total time: 29.57m
step 01146 (76.47%) | loss: 2.275256 | lrm: 0.47 | dt: 1587.80ms | tok/sec: 10,318 | mfu: 0.00 | epoch: 1 | total time: 29.60m
step 01147 (76.53%) | loss: 2.294019 | lrm: 0.47 | dt: 1586.98ms | tok/sec: 10,324 | mfu: 0.00 | epoch: 1 | total time: 29.63m
step 01148 (76.60%) | loss: 2.278979 | lrm: 0.47 | dt: 1505.09ms | tok/sec: 10,885 | mfu: 0.00 | epoch: 1 | total time: 29.65m
step 01149 (76.67%) | loss: 2.265956 | lrm: 0.47 | dt: 1539.31ms | tok/sec: 10,643 | mfu: 0.00 | epoch: 1 | total time: 29.68m
step 01150 (76.73%) | loss: 2.279938 | lrm: 0.47 | dt: 1622.81ms | tok/sec: 10,096 | mfu: 0.00 | epoch: 1 | total time: 29.70m
step 01151 (76.80%) | loss: 2.323973 | lrm: 0.46 | dt: 1612.41ms | tok/sec: 10,161 | mfu: 0.00 | epoch: 1 | total time: 29.73m
step 01152 (76.87%) | loss: 2.274525 | lrm: 0.46 | dt: 1552.95ms | tok/sec: 10,550 | mfu: 0.00 | epoch: 1 | total time: 29.76m
step 01153 (76.93%) | loss: 2.257563 | lrm: 0.46 | dt: 1495.22ms | tok/sec: 10,957 | mfu: 0.00 | epoch: 1 | total time: 29.78m
step 01154 (77.00%) | loss: 2.233934 | lrm: 0.46 | dt: 1509.18ms | tok/sec: 10,856 | mfu: 0.00 | epoch: 1 | total time: 29.81m
step 01155 (77.07%) | loss: 2.251147 | lrm: 0.46 | dt: 1512.29ms | tok/sec: 10,833 | mfu: 0.00 | epoch: 1 | total time: 29.83m
step 01156 (77.13%) | loss: 2.249642 | lrm: 0.46 | dt: 1515.23ms | tok/sec: 10,812 | mfu: 0.00 | epoch: 1 | total time: 29.86m
step 01157 (77.20%) | loss: 2.272582 | lrm: 0.46 | dt: 1637.24ms | tok/sec: 10,007 | mfu: 0.00 | epoch: 1 | total time: 29.88m
step 01158 (77.27%) | loss: 2.252621 | lrm: 0.45 | dt: 1543.24ms | tok/sec: 10,616 | mfu: 0.00 | epoch: 1 | total time: 29.91m
step 01159 (77.33%) | loss: 2.284360 | lrm: 0.45 | dt: 1601.89ms | tok/sec: 10,227 | mfu: 0.00 | epoch: 1 | total time: 29.94m
step 01160 (77.40%) | loss: 2.326729 | lrm: 0.45 | dt: 1517.02ms | tok/sec: 10,800 | mfu: 0.00 | epoch: 1 | total time: 29.96m
step 01161 (77.47%) | loss: 2.312888 | lrm: 0.45 | dt: 1560.19ms | tok/sec: 10,501 | mfu: 0.00 | epoch: 1 | total time: 29.99m
step 01162 (77.53%) | loss: 2.324950 | lrm: 0.45 | dt: 1676.03ms | tok/sec: 9,775 | mfu: 0.00 | epoch: 1 | total time: 30.02m
step 01163 (77.60%) | loss: 2.307905 | lrm: 0.45 | dt: 1538.07ms | tok/sec: 10,652 | mfu: 0.00 | epoch: 1 | total time: 30.04m
step 01164 (77.67%) | loss: 2.312307 | lrm: 0.45 | dt: 1564.47ms | tok/sec: 10,472 | mfu: 0.00 | epoch: 1 | total time: 30.07m
step 01165 (77.73%) | loss: 2.345450 | lrm: 0.45 | dt: 1623.64ms | tok/sec: 10,090 | mfu: 0.00 | epoch: 1 | total time: 30.09m
step 01166 (77.80%) | loss: 2.409636 | lrm: 0.44 | dt: 1647.07ms | tok/sec: 9,947 | mfu: 0.00 | epoch: 1 | total time: 30.12m
step 01167 (77.87%) | loss: 2.368613 | lrm: 0.44 | dt: 1681.18ms | tok/sec: 9,745 | mfu: 0.00 | epoch: 1 | total time: 30.15m
step 01168 (77.93%) | loss: 2.365247 | lrm: 0.44 | dt: 1719.18ms | tok/sec: 9,530 | mfu: 0.00 | epoch: 1 | total time: 30.18m
step 01169 (78.00%) | loss: 2.350128 | lrm: 0.44 | dt: 1707.78ms | tok/sec: 9,593 | mfu: 0.00 | epoch: 1 | total time: 30.21m
step 01170 (78.07%) | loss: 2.404484 | lrm: 0.44 | dt: 1657.49ms | tok/sec: 9,884 | mfu: 0.00 | epoch: 1 | total time: 30.23m
step 01171 (78.13%) | loss: 2.368233 | lrm: 0.44 | dt: 1732.97ms | tok/sec: 9,454 | mfu: 0.00 | epoch: 1 | total time: 30.26m
step 01172 (78.20%) | loss: 2.346163 | lrm: 0.44 | dt: 1631.38ms | tok/sec: 10,043 | mfu: 0.00 | epoch: 1 | total time: 30.29m
step 01173 (78.27%) | loss: 2.376643 | lrm: 0.43 | dt: 1640.55ms | tok/sec: 9,986 | mfu: 0.00 | epoch: 1 | total time: 30.32m
step 01174 (78.33%) | loss: 2.359850 | lrm: 0.43 | dt: 1633.32ms | tok/sec: 10,031 | mfu: 0.00 | epoch: 1 | total time: 30.34m
step 01175 (78.40%) | loss: 2.317048 | lrm: 0.43 | dt: 1645.64ms | tok/sec: 9,955 | mfu: 0.00 | epoch: 1 | total time: 30.37m
step 01176 (78.47%) | loss: 2.272078 | lrm: 0.43 | dt: 1847.83ms | tok/sec: 8,866 | mfu: 0.00 | epoch: 1 | total time: 30.40m
step 01177 (78.53%) | loss: 2.228941 | lrm: 0.43 | dt: 1605.84ms | tok/sec: 10,202 | mfu: 0.00 | epoch: 1 | total time: 30.43m
step 01178 (78.60%) | loss: 2.181373 | lrm: 0.43 | dt: 1599.67ms | tok/sec: 10,242 | mfu: 0.00 | epoch: 1 | total time: 30.46m
step 01179 (78.67%) | loss: 2.190571 | lrm: 0.43 | dt: 1674.78ms | tok/sec: 9,782 | mfu: 0.00 | epoch: 1 | total time: 30.48m
step 01180 (78.73%) | loss: 2.190508 | lrm: 0.43 | dt: 1667.62ms | tok/sec: 9,824 | mfu: 0.00 | epoch: 1 | total time: 30.51m
step 01181 (78.80%) | loss: 2.261856 | lrm: 0.42 | dt: 1583.94ms | tok/sec: 10,343 | mfu: 0.00 | epoch: 1 | total time: 30.54m
step 01182 (78.87%) | loss: 2.271409 | lrm: 0.42 | dt: 1509.70ms | tok/sec: 10,852 | mfu: 0.00 | epoch: 1 | total time: 30.56m
step 01183 (78.93%) | loss: 2.287166 | lrm: 0.42 | dt: 1501.48ms | tok/sec: 10,911 | mfu: 0.00 | epoch: 1 | total time: 30.59m
step 01184 (79.00%) | loss: 2.256822 | lrm: 0.42 | dt: 1557.44ms | tok/sec: 10,519 | mfu: 0.00 | epoch: 1 | total time: 30.61m
step 01185 (79.07%) | loss: 2.257958 | lrm: 0.42 | dt: 1532.07ms | tok/sec: 10,694 | mfu: 0.00 | epoch: 1 | total time: 30.64m
step 01186 (79.13%) | loss: 2.247370 | lrm: 0.42 | dt: 1508.40ms | tok/sec: 10,861 | mfu: 0.00 | epoch: 1 | total time: 30.67m
step 01187 (79.20%) | loss: 2.247189 | lrm: 0.42 | dt: 1578.24ms | tok/sec: 10,381 | mfu: 0.00 | epoch: 1 | total time: 30.69m
step 01188 (79.27%) | loss: 2.335970 | lrm: 0.41 | dt: 1570.96ms | tok/sec: 10,429 | mfu: 0.00 | epoch: 1 | total time: 30.72m
step 01189 (79.33%) | loss: 2.304151 | lrm: 0.41 | dt: 1606.71ms | tok/sec: 10,197 | mfu: 0.00 | epoch: 1 | total time: 30.74m
step 01190 (79.40%) | loss: 2.307617 | lrm: 0.41 | dt: 1649.93ms | tok/sec: 9,930 | mfu: 0.00 | epoch: 1 | total time: 30.77m
step 01191 (79.47%) | loss: 2.276654 | lrm: 0.41 | dt: 1568.67ms | tok/sec: 10,444 | mfu: 0.00 | epoch: 1 | total time: 30.80m
step 01192 (79.53%) | loss: 2.287543 | lrm: 0.41 | dt: 1491.95ms | tok/sec: 10,981 | mfu: 0.00 | epoch: 1 | total time: 30.82m
step 01193 (79.60%) | loss: 2.232386 | lrm: 0.41 | dt: 1547.53ms | tok/sec: 10,587 | mfu: 0.00 | epoch: 1 | total time: 30.85m
step 01194 (79.67%) | loss: 2.202938 | lrm: 0.41 | dt: 1513.36ms | tok/sec: 10,826 | mfu: 0.00 | epoch: 1 | total time: 30.87m
step 01195 (79.73%) | loss: 2.184316 | lrm: 0.41 | dt: 1587.94ms | tok/sec: 10,317 | mfu: 0.00 | epoch: 1 | total time: 30.90m
step 01196 (79.80%) | loss: 2.183039 | lrm: 0.40 | dt: 1522.65ms | tok/sec: 10,760 | mfu: 0.00 | epoch: 1 | total time: 30.93m
step 01197 (79.87%) | loss: 2.165409 | lrm: 0.40 | dt: 1502.32ms | tok/sec: 10,905 | mfu: 0.00 | epoch: 1 | total time: 30.95m
step 01198 (79.93%) | loss: 2.164174 | lrm: 0.40 | dt: 1619.91ms | tok/sec: 10,114 | mfu: 0.00 | epoch: 1 | total time: 30.98m
step 01199 (80.00%) | loss: 2.216155 | lrm: 0.40 | dt: 1639.15ms | tok/sec: 9,995 | mfu: 0.00 | epoch: 1 | total time: 31.01m
step 01200 (80.07%) | loss: 2.246522 | lrm: 0.40 | dt: 1554.02ms | tok/sec: 10,542 | mfu: 0.00 | epoch: 1 | total time: 31.03m
Step 01200 | Validation bpb: 0.7950
step 01201 (80.13%) | loss: 2.264945 | lrm: 0.40 | dt: 1687.79ms | tok/sec: 9,707 | mfu: 0.00 | epoch: 1 | total time: 31.06m
step 01202 (80.20%) | loss: 2.299183 | lrm: 0.40 | dt: 1562.25ms | tok/sec: 10,487 | mfu: 0.00 | epoch: 1 | total time: 31.09m
step 01203 (80.27%) | loss: 2.272415 | lrm: 0.39 | dt: 1644.00ms | tok/sec: 9,965 | mfu: 0.00 | epoch: 1 | total time: 31.11m
step 01204 (80.33%) | loss: 2.257225 | lrm: 0.39 | dt: 1645.91ms | tok/sec: 9,954 | mfu: 0.00 | epoch: 1 | total time: 31.14m
step 01205 (80.40%) | loss: 2.238836 | lrm: 0.39 | dt: 1632.43ms | tok/sec: 10,036 | mfu: 0.00 | epoch: 1 | total time: 31.17m
step 01206 (80.47%) | loss: 2.200745 | lrm: 0.39 | dt: 1674.41ms | tok/sec: 9,784 | mfu: 0.00 | epoch: 1 | total time: 31.20m
step 01207 (80.53%) | loss: 2.211602 | lrm: 0.39 | dt: 1593.79ms | tok/sec: 10,279 | mfu: 0.00 | epoch: 1 | total time: 31.22m
step 01208 (80.60%) | loss: 2.164103 | lrm: 0.39 | dt: 1632.47ms | tok/sec: 10,036 | mfu: 0.00 | epoch: 1 | total time: 31.25m
step 01209 (80.67%) | loss: 2.191713 | lrm: 0.39 | dt: 1691.75ms | tok/sec: 9,684 | mfu: 0.00 | epoch: 1 | total time: 31.28m
step 01210 (80.73%) | loss: 2.213326 | lrm: 0.39 | dt: 1609.01ms | tok/sec: 10,182 | mfu: 0.00 | epoch: 1 | total time: 31.30m
step 01211 (80.80%) | loss: 2.231350 | lrm: 0.38 | dt: 1647.78ms | tok/sec: 9,943 | mfu: 0.00 | epoch: 1 | total time: 31.33m
step 01212 (80.87%) | loss: 2.244605 | lrm: 0.38 | dt: 1562.07ms | tok/sec: 10,488 | mfu: 0.00 | epoch: 1 | total time: 31.36m
step 01213 (80.93%) | loss: 2.262168 | lrm: 0.38 | dt: 1763.14ms | tok/sec: 9,292 | mfu: 0.00 | epoch: 1 | total time: 31.39m
step 01214 (81.00%) | loss: 2.238924 | lrm: 0.38 | dt: 1548.73ms | tok/sec: 10,579 | mfu: 0.00 | epoch: 1 | total time: 31.41m
step 01215 (81.07%) | loss: 2.236893 | lrm: 0.38 | dt: 1654.49ms | tok/sec: 9,902 | mfu: 0.00 | epoch: 1 | total time: 31.44m
step 01216 (81.13%) | loss: 2.239146 | lrm: 0.38 | dt: 1522.51ms | tok/sec: 10,761 | mfu: 0.00 | epoch: 1 | total time: 31.47m
step 01217 (81.20%) | loss: 2.260495 | lrm: 0.38 | dt: 1526.14ms | tok/sec: 10,735 | mfu: 0.00 | epoch: 1 | total time: 31.49m
step 01218 (81.27%) | loss: 2.281981 | lrm: 0.37 | dt: 1518.57ms | tok/sec: 10,789 | mfu: 0.00 | epoch: 1 | total time: 31.52m
step 01219 (81.33%) | loss: 2.224980 | lrm: 0.37 | dt: 1506.42ms | tok/sec: 10,876 | mfu: 0.00 | epoch: 1 | total time: 31.54m
step 01220 (81.40%) | loss: 2.209042 | lrm: 0.37 | dt: 1529.83ms | tok/sec: 10,709 | mfu: 0.00 | epoch: 1 | total time: 31.57m
step 01221 (81.47%) | loss: 2.185883 | lrm: 0.37 | dt: 1521.54ms | tok/sec: 10,768 | mfu: 0.00 | epoch: 1 | total time: 31.59m
step 01222 (81.53%) | loss: 2.149956 | lrm: 0.37 | dt: 1511.95ms | tok/sec: 10,836 | mfu: 0.00 | epoch: 1 | total time: 31.62m
step 01223 (81.60%) | loss: 2.159549 | lrm: 0.37 | dt: 1517.85ms | tok/sec: 10,794 | mfu: 0.00 | epoch: 1 | total time: 31.64m
step 01224 (81.67%) | loss: 2.197013 | lrm: 0.37 | dt: 1531.52ms | tok/sec: 10,697 | mfu: 0.00 | epoch: 1 | total time: 31.67m
step 01225 (81.73%) | loss: 2.232692 | lrm: 0.37 | dt: 1556.43ms | tok/sec: 10,526 | mfu: 0.00 | epoch: 1 | total time: 31.69m
step 01226 (81.80%) | loss: 2.188515 | lrm: 0.36 | dt: 1542.09ms | tok/sec: 10,624 | mfu: 0.00 | epoch: 1 | total time: 31.72m
step 01227 (81.87%) | loss: 2.251070 | lrm: 0.36 | dt: 1563.44ms | tok/sec: 10,479 | mfu: 0.00 | epoch: 1 | total time: 31.75m
step 01228 (81.93%) | loss: 2.237197 | lrm: 0.36 | dt: 1550.46ms | tok/sec: 10,567 | mfu: 0.00 | epoch: 1 | total time: 31.77m
step 01229 (82.00%) | loss: 2.198035 | lrm: 0.36 | dt: 1568.90ms | tok/sec: 10,442 | mfu: 0.00 | epoch: 1 | total time: 31.80m
step 01230 (82.07%) | loss: 2.146933 | lrm: 0.36 | dt: 1554.91ms | tok/sec: 10,536 | mfu: 0.00 | epoch: 1 | total time: 31.82m
step 01231 (82.13%) | loss: 2.161540 | lrm: 0.36 | dt: 1500.15ms | tok/sec: 10,921 | mfu: 0.00 | epoch: 1 | total time: 31.85m
step 01232 (82.20%) | loss: 2.181957 | lrm: 0.36 | dt: 1560.29ms | tok/sec: 10,500 | mfu: 0.00 | epoch: 1 | total time: 31.88m
step 01233 (82.27%) | loss: 2.149501 | lrm: 0.35 | dt: 1540.54ms | tok/sec: 10,635 | mfu: 0.00 | epoch: 1 | total time: 31.90m
step 01234 (82.33%) | loss: 2.154226 | lrm: 0.35 | dt: 1640.17ms | tok/sec: 9,989 | mfu: 0.00 | epoch: 1 | total time: 31.93m
step 01235 (82.40%) | loss: 2.200502 | lrm: 0.35 | dt: 1649.78ms | tok/sec: 9,931 | mfu: 0.00 | epoch: 1 | total time: 31.96m
step 01236 (82.47%) | loss: 2.205113 | lrm: 0.35 | dt: 1663.63ms | tok/sec: 9,848 | mfu: 0.00 | epoch: 1 | total time: 31.98m
step 01237 (82.53%) | loss: 2.204501 | lrm: 0.35 | dt: 1641.99ms | tok/sec: 9,978 | mfu: 0.00 | epoch: 1 | total time: 32.01m
step 01238 (82.60%) | loss: 2.178326 | lrm: 0.35 | dt: 1625.04ms | tok/sec: 10,082 | mfu: 0.00 | epoch: 1 | total time: 32.04m
step 01239 (82.67%) | loss: 2.169965 | lrm: 0.35 | dt: 1849.11ms | tok/sec: 8,860 | mfu: 0.00 | epoch: 1 | total time: 32.07m
step 01240 (82.73%) | loss: 2.208282 | lrm: 0.35 | dt: 1666.84ms | tok/sec: 9,829 | mfu: 0.00 | epoch: 1 | total time: 32.10m
step 01241 (82.80%) | loss: 2.216183 | lrm: 0.34 | dt: 1623.16ms | tok/sec: 10,093 | mfu: 0.00 | epoch: 1 | total time: 32.12m
step 01242 (82.87%) | loss: 2.253784 | lrm: 0.34 | dt: 1653.61ms | tok/sec: 9,908 | mfu: 0.00 | epoch: 1 | total time: 32.15m
step 01243 (82.93%) | loss: 2.291340 | lrm: 0.34 | dt: 1719.88ms | tok/sec: 9,526 | mfu: 0.00 | epoch: 1 | total time: 32.18m
step 01244 (83.00%) | loss: 2.289196 | lrm: 0.34 | dt: 1607.68ms | tok/sec: 10,191 | mfu: 0.00 | epoch: 1 | total time: 32.21m
step 01245 (83.07%) | loss: 2.339823 | lrm: 0.34 | dt: 1701.84ms | tok/sec: 9,627 | mfu: 0.00 | epoch: 1 | total time: 32.23m
step 01246 (83.13%) | loss: 2.337177 | lrm: 0.34 | dt: 1676.04ms | tok/sec: 9,775 | mfu: 0.00 | epoch: 1 | total time: 32.26m
step 01247 (83.20%) | loss: 2.342678 | lrm: 0.34 | dt: 1649.74ms | tok/sec: 9,931 | mfu: 0.00 | epoch: 1 | total time: 32.29m
step 01248 (83.27%) | loss: 2.306965 | lrm: 0.33 | dt: 1749.40ms | tok/sec: 9,365 | mfu: 0.00 | epoch: 1 | total time: 32.32m
step 01249 (83.33%) | loss: 2.285811 | lrm: 0.33 | dt: 1610.90ms | tok/sec: 10,170 | mfu: 0.00 | epoch: 1 | total time: 32.35m
step 01250 (83.40%) | loss: 2.284112 | lrm: 0.33 | dt: 1621.05ms | tok/sec: 10,107 | mfu: 0.00 | epoch: 1 | total time: 32.37m
step 01251 (83.47%) | loss: 2.299917 | lrm: 0.33 | dt: 1619.83ms | tok/sec: 10,114 | mfu: 0.00 | epoch: 1 | total time: 32.40m
step 01252 (83.53%) | loss: 2.307186 | lrm: 0.33 | dt: 1591.35ms | tok/sec: 10,295 | mfu: 0.00 | epoch: 1 | total time: 32.43m
step 01253 (83.60%) | loss: 2.323337 | lrm: 0.33 | dt: 1500.95ms | tok/sec: 10,915 | mfu: 0.00 | epoch: 1 | total time: 32.45m
step 01254 (83.67%) | loss: 2.328294 | lrm: 0.33 | dt: 1483.22ms | tok/sec: 11,046 | mfu: 0.00 | epoch: 1 | total time: 32.48m
step 01255 (83.73%) | loss: 2.275048 | lrm: 0.33 | dt: 1524.84ms | tok/sec: 10,744 | mfu: 0.00 | epoch: 1 | total time: 32.50m
step 01256 (83.80%) | loss: 2.263942 | lrm: 0.32 | dt: 1592.94ms | tok/sec: 10,285 | mfu: 0.00 | epoch: 1 | total time: 32.53m
step 01257 (83.87%) | loss: 2.278524 | lrm: 0.32 | dt: 1529.42ms | tok/sec: 10,712 | mfu: 0.00 | epoch: 1 | total time: 32.55m
step 01258 (83.93%) | loss: 2.255116 | lrm: 0.32 | dt: 1568.67ms | tok/sec: 10,444 | mfu: 0.00 | epoch: 1 | total time: 32.58m
step 01259 (84.00%) | loss: 2.278548 | lrm: 0.32 | dt: 1536.06ms | tok/sec: 10,666 | mfu: 0.00 | epoch: 1 | total time: 32.61m
step 01260 (84.07%) | loss: 2.277696 | lrm: 0.32 | dt: 1541.09ms | tok/sec: 10,631 | mfu: 0.00 | epoch: 1 | total time: 32.63m
step 01261 (84.13%) | loss: 2.259807 | lrm: 0.32 | dt: 1535.76ms | tok/sec: 10,668 | mfu: 0.00 | epoch: 1 | total time: 32.66m
step 01262 (84.20%) | loss: 2.220540 | lrm: 0.32 | dt: 1557.17ms | tok/sec: 10,521 | mfu: 0.00 | epoch: 1 | total time: 32.68m
step 01263 (84.27%) | loss: 2.205980 | lrm: 0.31 | dt: 1543.62ms | tok/sec: 10,614 | mfu: 0.00 | epoch: 1 | total time: 32.71m
step 01264 (84.33%) | loss: 2.206590 | lrm: 0.31 | dt: 1550.30ms | tok/sec: 10,568 | mfu: 0.00 | epoch: 1 | total time: 32.73m
step 01265 (84.40%) | loss: 2.219423 | lrm: 0.31 | dt: 1564.49ms | tok/sec: 10,472 | mfu: 0.00 | epoch: 1 | total time: 32.76m
step 01266 (84.47%) | loss: 2.268150 | lrm: 0.31 | dt: 1635.55ms | tok/sec: 10,017 | mfu: 0.00 | epoch: 1 | total time: 32.79m
step 01267 (84.53%) | loss: 2.282664 | lrm: 0.31 | dt: 1848.73ms | tok/sec: 8,862 | mfu: 0.00 | epoch: 1 | total time: 32.82m
step 01268 (84.60%) | loss: 2.259200 | lrm: 0.31 | dt: 1631.45ms | tok/sec: 10,042 | mfu: 0.00 | epoch: 1 | total time: 32.85m
step 01269 (84.67%) | loss: 2.200715 | lrm: 0.31 | dt: 1622.75ms | tok/sec: 10,096 | mfu: 0.00 | epoch: 1 | total time: 32.87m
step 01270 (84.73%) | loss: 2.149606 | lrm: 0.31 | dt: 1606.53ms | tok/sec: 10,198 | mfu: 0.00 | epoch: 1 | total time: 32.90m
step 01271 (84.80%) | loss: 2.149778 | lrm: 0.30 | dt: 1635.04ms | tok/sec: 10,020 | mfu: 0.00 | epoch: 1 | total time: 32.93m
step 01272 (84.87%) | loss: 2.107573 | lrm: 0.30 | dt: 1716.63ms | tok/sec: 9,544 | mfu: 0.00 | epoch: 1 | total time: 32.96m
step 01273 (84.93%) | loss: 2.119035 | lrm: 0.30 | dt: 1693.15ms | tok/sec: 9,676 | mfu: 0.00 | epoch: 1 | total time: 32.98m
step 01274 (85.00%) | loss: 2.133450 | lrm: 0.30 | dt: 1605.17ms | tok/sec: 10,206 | mfu: 0.00 | epoch: 1 | total time: 33.01m
step 01275 (85.07%) | loss: 2.122646 | lrm: 0.30 | dt: 1596.76ms | tok/sec: 10,260 | mfu: 0.00 | epoch: 1 | total time: 33.04m
step 01276 (85.13%) | loss: 2.099623 | lrm: 0.30 | dt: 1637.75ms | tok/sec: 10,003 | mfu: 0.00 | epoch: 1 | total time: 33.06m
step 01277 (85.20%) | loss: 2.122997 | lrm: 0.30 | dt: 1635.36ms | tok/sec: 10,018 | mfu: 0.00 | epoch: 1 | total time: 33.09m
step 01278 (85.27%) | loss: 2.147364 | lrm: 0.29 | dt: 1632.44ms | tok/sec: 10,036 | mfu: 0.00 | epoch: 1 | total time: 33.12m
step 01279 (85.33%) | loss: 2.104627 | lrm: 0.29 | dt: 1568.26ms | tok/sec: 10,447 | mfu: 0.00 | epoch: 1 | total time: 33.15m
step 01280 (85.40%) | loss: 2.057661 | lrm: 0.29 | dt: 1568.00ms | tok/sec: 10,448 | mfu: 0.00 | epoch: 1 | total time: 33.17m
step 01281 (85.47%) | loss: 2.095753 | lrm: 0.29 | dt: 1616.19ms | tok/sec: 10,137 | mfu: 0.00 | epoch: 1 | total time: 33.20m
step 01282 (85.53%) | loss: 2.062293 | lrm: 0.29 | dt: 1605.14ms | tok/sec: 10,207 | mfu: 0.00 | epoch: 1 | total time: 33.22m
step 01283 (85.60%) | loss: 2.097277 | lrm: 0.29 | dt: 1702.96ms | tok/sec: 9,620 | mfu: 0.00 | epoch: 1 | total time: 33.25m
step 01284 (85.67%) | loss: 2.102794 | lrm: 0.29 | dt: 1618.49ms | tok/sec: 10,122 | mfu: 0.00 | epoch: 1 | total time: 33.28m
step 01285 (85.73%) | loss: 2.092459 | lrm: 0.29 | dt: 1579.84ms | tok/sec: 10,370 | mfu: 0.00 | epoch: 1 | total time: 33.31m
step 01286 (85.80%) | loss: 2.053828 | lrm: 0.28 | dt: 1578.37ms | tok/sec: 10,380 | mfu: 0.00 | epoch: 1 | total time: 33.33m
step 01287 (85.87%) | loss: 2.051559 | lrm: 0.28 | dt: 1625.22ms | tok/sec: 10,081 | mfu: 0.00 | epoch: 1 | total time: 33.36m
step 01288 (85.93%) | loss: 2.123950 | lrm: 0.28 | dt: 1566.95ms | tok/sec: 10,456 | mfu: 0.00 | epoch: 1 | total time: 33.39m
step 01289 (86.00%) | loss: 2.098858 | lrm: 0.28 | dt: 1575.81ms | tok/sec: 10,397 | mfu: 0.00 | epoch: 1 | total time: 33.41m
step 01290 (86.07%) | loss: 2.103339 | lrm: 0.28 | dt: 1566.77ms | tok/sec: 10,457 | mfu: 0.00 | epoch: 1 | total time: 33.44m
step 01291 (86.13%) | loss: 2.146585 | lrm: 0.28 | dt: 1541.02ms | tok/sec: 10,631 | mfu: 0.00 | epoch: 1 | total time: 33.46m
step 01292 (86.20%) | loss: 2.163310 | lrm: 0.28 | dt: 1566.79ms | tok/sec: 10,457 | mfu: 0.00 | epoch: 1 | total time: 33.49m
step 01293 (86.27%) | loss: 2.207155 | lrm: 0.27 | dt: 1570.32ms | tok/sec: 10,433 | mfu: 0.00 | epoch: 1 | total time: 33.52m
step 01294 (86.33%) | loss: 2.207443 | lrm: 0.27 | dt: 1551.42ms | tok/sec: 10,560 | mfu: 0.00 | epoch: 1 | total time: 33.54m
step 01295 (86.40%) | loss: 2.153260 | lrm: 0.27 | dt: 1568.63ms | tok/sec: 10,444 | mfu: 0.00 | epoch: 1 | total time: 33.57m
step 01296 (86.47%) | loss: 2.144146 | lrm: 0.27 | dt: 1561.18ms | tok/sec: 10,494 | mfu: 0.00 | epoch: 1 | total time: 33.59m
step 01297 (86.53%) | loss: 2.204231 | lrm: 0.27 | dt: 1537.03ms | tok/sec: 10,659 | mfu: 0.00 | epoch: 1 | total time: 33.62m
step 01298 (86.60%) | loss: 2.178855 | lrm: 0.27 | dt: 1554.59ms | tok/sec: 10,539 | mfu: 0.00 | epoch: 1 | total time: 33.65m
step 01299 (86.67%) | loss: 2.124145 | lrm: 0.27 | dt: 1635.51ms | tok/sec: 10,017 | mfu: 0.00 | epoch: 1 | total time: 33.67m
step 01300 (86.73%) | loss: 2.126637 | lrm: 0.27 | dt: 1559.84ms | tok/sec: 10,503 | mfu: 0.00 | epoch: 1 | total time: 33.70m
step 01301 (86.80%) | loss: 2.116299 | lrm: 0.26 | dt: 1567.06ms | tok/sec: 10,455 | mfu: 0.00 | epoch: 1 | total time: 33.73m
step 01302 (86.87%) | loss: 2.143660 | lrm: 0.26 | dt: 1607.88ms | tok/sec: 10,189 | mfu: 0.00 | epoch: 1 | total time: 33.75m
step 01303 (86.93%) | loss: 2.114452 | lrm: 0.26 | dt: 1533.77ms | tok/sec: 10,682 | mfu: 0.00 | epoch: 1 | total time: 33.78m
step 01304 (87.00%) | loss: 2.159742 | lrm: 0.26 | dt: 1553.21ms | tok/sec: 10,548 | mfu: 0.00 | epoch: 1 | total time: 33.80m
step 01305 (87.07%) | loss: 2.167791 | lrm: 0.26 | dt: 1741.49ms | tok/sec: 9,408 | mfu: 0.00 | epoch: 1 | total time: 33.83m
step 01306 (87.13%) | loss: 2.180222 | lrm: 0.26 | dt: 2009.41ms | tok/sec: 8,153 | mfu: 0.00 | epoch: 1 | total time: 33.87m
step 01307 (87.20%) | loss: 2.169058 | lrm: 0.26 | dt: 1777.07ms | tok/sec: 9,219 | mfu: 0.00 | epoch: 1 | total time: 33.90m
step 01308 (87.27%) | loss: 2.170736 | lrm: 0.25 | dt: 1610.40ms | tok/sec: 10,173 | mfu: 0.00 | epoch: 1 | total time: 33.92m
step 01309 (87.33%) | loss: 2.128306 | lrm: 0.25 | dt: 1697.70ms | tok/sec: 9,650 | mfu: 0.00 | epoch: 1 | total time: 33.95m
step 01310 (87.40%) | loss: 2.078163 | lrm: 0.25 | dt: 1617.26ms | tok/sec: 10,130 | mfu: 0.00 | epoch: 1 | total time: 33.98m
step 01311 (87.47%) | loss: 2.103658 | lrm: 0.25 | dt: 1625.17ms | tok/sec: 10,081 | mfu: 0.00 | epoch: 1 | total time: 34.01m
step 01312 (87.53%) | loss: 2.140099 | lrm: 0.25 | dt: 1640.22ms | tok/sec: 9,988 | mfu: 0.00 | epoch: 1 | total time: 34.03m
step 01313 (87.60%) | loss: 2.142000 | lrm: 0.25 | dt: 1641.89ms | tok/sec: 9,978 | mfu: 0.00 | epoch: 1 | total time: 34.06m
step 01314 (87.67%) | loss: 2.134390 | lrm: 0.25 | dt: 1689.71ms | tok/sec: 9,696 | mfu: 0.00 | epoch: 1 | total time: 34.09m
step 01315 (87.73%) | loss: 2.138571 | lrm: 0.25 | dt: 1624.92ms | tok/sec: 10,082 | mfu: 0.00 | epoch: 1 | total time: 34.11m
step 01316 (87.80%) | loss: 2.117295 | lrm: 0.24 | dt: 1656.40ms | tok/sec: 9,891 | mfu: 0.00 | epoch: 1 | total time: 34.14m
step 01317 (87.87%) | loss: 2.133060 | lrm: 0.24 | dt: 1545.72ms | tok/sec: 10,599 | mfu: 0.00 | epoch: 1 | total time: 34.17m
step 01318 (87.93%) | loss: 2.101352 | lrm: 0.24 | dt: 1506.76ms | tok/sec: 10,873 | mfu: 0.00 | epoch: 1 | total time: 34.19m
step 01319 (88.00%) | loss: 2.097891 | lrm: 0.24 | dt: 1559.89ms | tok/sec: 10,503 | mfu: 0.00 | epoch: 1 | total time: 34.22m
step 01320 (88.07%) | loss: 2.101608 | lrm: 0.24 | dt: 1542.07ms | tok/sec: 10,624 | mfu: 0.00 | epoch: 1 | total time: 34.25m
step 01321 (88.13%) | loss: 2.046535 | lrm: 0.24 | dt: 1539.14ms | tok/sec: 10,644 | mfu: 0.00 | epoch: 1 | total time: 34.27m
step 01322 (88.20%) | loss: 2.030204 | lrm: 0.24 | dt: 1551.19ms | tok/sec: 10,562 | mfu: 0.00 | epoch: 1 | total time: 34.30m
step 01323 (88.27%) | loss: 2.101617 | lrm: 0.23 | dt: 1682.49ms | tok/sec: 9,737 | mfu: 0.00 | epoch: 1 | total time: 34.32m
step 01324 (88.33%) | loss: 2.144211 | lrm: 0.23 | dt: 1639.59ms | tok/sec: 9,992 | mfu: 0.00 | epoch: 1 | total time: 34.35m
step 01325 (88.40%) | loss: 2.128916 | lrm: 0.23 | dt: 1527.04ms | tok/sec: 10,729 | mfu: 0.00 | epoch: 1 | total time: 34.38m
step 01326 (88.47%) | loss: 2.094388 | lrm: 0.23 | dt: 1555.34ms | tok/sec: 10,534 | mfu: 0.00 | epoch: 1 | total time: 34.40m
step 01327 (88.53%) | loss: 2.118366 | lrm: 0.23 | dt: 1551.27ms | tok/sec: 10,561 | mfu: 0.00 | epoch: 1 | total time: 34.43m
step 01328 (88.60%) | loss: 2.094992 | lrm: 0.23 | dt: 1532.29ms | tok/sec: 10,692 | mfu: 0.00 | epoch: 1 | total time: 34.45m
step 01329 (88.67%) | loss: 2.095386 | lrm: 0.23 | dt: 1543.24ms | tok/sec: 10,616 | mfu: 0.00 | epoch: 1 | total time: 34.48m
step 01330 (88.73%) | loss: 2.079214 | lrm: 0.23 | dt: 1528.76ms | tok/sec: 10,717 | mfu: 0.00 | epoch: 1 | total time: 34.51m
step 01331 (88.80%) | loss: 2.049501 | lrm: 0.22 | dt: 1527.92ms | tok/sec: 10,723 | mfu: 0.00 | epoch: 1 | total time: 34.53m
step 01332 (88.87%) | loss: 2.056902 | lrm: 0.22 | dt: 1525.83ms | tok/sec: 10,737 | mfu: 0.00 | epoch: 1 | total time: 34.56m
step 01333 (88.93%) | loss: 2.078713 | lrm: 0.22 | dt: 1552.80ms | tok/sec: 10,551 | mfu: 0.00 | epoch: 1 | total time: 34.58m
step 01334 (89.00%) | loss: 2.084922 | lrm: 0.22 | dt: 1562.56ms | tok/sec: 10,485 | mfu: 0.00 | epoch: 1 | total time: 34.61m
step 01335 (89.07%) | loss: 2.090232 | lrm: 0.22 | dt: 1528.76ms | tok/sec: 10,717 | mfu: 0.00 | epoch: 1 | total time: 34.63m
step 01336 (89.13%) | loss: 2.060750 | lrm: 0.22 | dt: 1536.94ms | tok/sec: 10,660 | mfu: 0.00 | epoch: 1 | total time: 34.66m
step 01337 (89.20%) | loss: 2.039437 | lrm: 0.22 | dt: 1537.87ms | tok/sec: 10,653 | mfu: 0.00 | epoch: 1 | total time: 34.69m
step 01338 (89.27%) | loss: 2.044322 | lrm: 0.21 | dt: 1546.83ms | tok/sec: 10,592 | mfu: 0.00 | epoch: 1 | total time: 34.71m
step 01339 (89.33%) | loss: 2.046451 | lrm: 0.21 | dt: 1539.52ms | tok/sec: 10,642 | mfu: 0.00 | epoch: 1 | total time: 34.74m
step 01340 (89.40%) | loss: 2.051409 | lrm: 0.21 | dt: 1618.20ms | tok/sec: 10,124 | mfu: 0.00 | epoch: 1 | total time: 34.76m
step 01341 (89.47%) | loss: 2.084111 | lrm: 0.21 | dt: 1637.10ms | tok/sec: 10,007 | mfu: 0.00 | epoch: 1 | total time: 34.79m
step 01342 (89.53%) | loss: 2.105712 | lrm: 0.21 | dt: 1647.56ms | tok/sec: 9,944 | mfu: 0.00 | epoch: 1 | total time: 34.82m
step 01343 (89.60%) | loss: 2.100955 | lrm: 0.21 | dt: 1646.68ms | tok/sec: 9,949 | mfu: 0.00 | epoch: 1 | total time: 34.85m
step 01344 (89.67%) | loss: 2.070755 | lrm: 0.21 | dt: 1656.82ms | tok/sec: 9,888 | mfu: 0.00 | epoch: 1 | total time: 34.87m
step 01345 (89.73%) | loss: 2.109252 | lrm: 0.21 | dt: 1661.60ms | tok/sec: 9,860 | mfu: 0.00 | epoch: 1 | total time: 34.90m
step 01346 (89.80%) | loss: 2.085168 | lrm: 0.20 | dt: 1752.02ms | tok/sec: 9,351 | mfu: 0.00 | epoch: 1 | total time: 34.93m
step 01347 (89.87%) | loss: 2.088278 | lrm: 0.20 | dt: 1648.53ms | tok/sec: 9,938 | mfu: 0.00 | epoch: 1 | total time: 34.96m
step 01348 (89.93%) | loss: 2.100411 | lrm: 0.20 | dt: 1623.90ms | tok/sec: 10,089 | mfu: 0.00 | epoch: 1 | total time: 34.99m
step 01349 (90.00%) | loss: 2.174334 | lrm: 0.20 | dt: 1640.48ms | tok/sec: 9,987 | mfu: 0.00 | epoch: 1 | total time: 35.01m
step 01350 (90.07%) | loss: 2.183311 | lrm: 0.20 | dt: 1618.02ms | tok/sec: 10,125 | mfu: 0.00 | epoch: 1 | total time: 35.04m
step 01351 (90.13%) | loss: 2.196888 | lrm: 0.20 | dt: 1630.25ms | tok/sec: 10,049 | mfu: 0.00 | epoch: 1 | total time: 35.07m
step 01352 (90.20%) | loss: 2.203906 | lrm: 0.20 | dt: 1632.41ms | tok/sec: 10,036 | mfu: 0.00 | epoch: 1 | total time: 35.09m
step 01353 (90.27%) | loss: 2.174773 | lrm: 0.19 | dt: 1600.05ms | tok/sec: 10,239 | mfu: 0.00 | epoch: 1 | total time: 35.12m
step 01354 (90.33%) | loss: 2.179435 | lrm: 0.19 | dt: 1647.25ms | tok/sec: 9,946 | mfu: 0.00 | epoch: 1 | total time: 35.15m
step 01355 (90.40%) | loss: 2.142313 | lrm: 0.19 | dt: 1652.95ms | tok/sec: 9,911 | mfu: 0.00 | epoch: 1 | total time: 35.18m
step 01356 (90.47%) | loss: 2.139904 | lrm: 0.19 | dt: 1623.94ms | tok/sec: 10,089 | mfu: 0.00 | epoch: 1 | total time: 35.20m
step 01357 (90.53%) | loss: 2.118623 | lrm: 0.19 | dt: 1635.77ms | tok/sec: 10,016 | mfu: 0.00 | epoch: 1 | total time: 35.23m
step 01358 (90.60%) | loss: 2.125115 | lrm: 0.19 | dt: 1619.30ms | tok/sec: 10,117 | mfu: 0.00 | epoch: 1 | total time: 35.26m
step 01359 (90.67%) | loss: 2.118281 | lrm: 0.19 | dt: 1609.26ms | tok/sec: 10,181 | mfu: 0.00 | epoch: 1 | total time: 35.28m
step 01360 (90.73%) | loss: 2.112583 | lrm: 0.19 | dt: 1649.12ms | tok/sec: 9,934 | mfu: 0.00 | epoch: 1 | total time: 35.31m
step 01361 (90.80%) | loss: 2.178239 | lrm: 0.18 | dt: 1625.15ms | tok/sec: 10,081 | mfu: 0.00 | epoch: 1 | total time: 35.34m
step 01362 (90.87%) | loss: 2.159390 | lrm: 0.18 | dt: 1624.94ms | tok/sec: 10,082 | mfu: 0.00 | epoch: 1 | total time: 35.37m
step 01363 (90.93%) | loss: 2.180243 | lrm: 0.18 | dt: 1630.50ms | tok/sec: 10,048 | mfu: 0.00 | epoch: 1 | total time: 35.39m
step 01364 (91.00%) | loss: 2.249493 | lrm: 0.18 | dt: 1610.62ms | tok/sec: 10,172 | mfu: 0.00 | epoch: 1 | total time: 35.42m
step 01365 (91.07%) | loss: 2.236007 | lrm: 0.18 | dt: 1627.46ms | tok/sec: 10,067 | mfu: 0.00 | epoch: 1 | total time: 35.45m
step 01366 (91.13%) | loss: 2.261398 | lrm: 0.18 | dt: 1640.58ms | tok/sec: 9,986 | mfu: 0.00 | epoch: 1 | total time: 35.47m
step 01367 (91.20%) | loss: 2.224478 | lrm: 0.18 | dt: 1641.17ms | tok/sec: 9,983 | mfu: 0.00 | epoch: 1 | total time: 35.50m
step 01368 (91.27%) | loss: 2.183627 | lrm: 0.17 | dt: 1701.49ms | tok/sec: 9,629 | mfu: 0.00 | epoch: 1 | total time: 35.53m
step 01369 (91.33%) | loss: 2.253360 | lrm: 0.17 | dt: 1691.11ms | tok/sec: 9,688 | mfu: 0.00 | epoch: 1 | total time: 35.56m
step 01370 (91.40%) | loss: 2.239534 | lrm: 0.17 | dt: 1644.55ms | tok/sec: 9,962 | mfu: 0.00 | epoch: 1 | total time: 35.59m
step 01371 (91.47%) | loss: 2.195817 | lrm: 0.17 | dt: 1681.44ms | tok/sec: 9,744 | mfu: 0.00 | epoch: 1 | total time: 35.61m
step 01372 (91.53%) | loss: 2.200581 | lrm: 0.17 | dt: 1687.31ms | tok/sec: 9,710 | mfu: 0.00 | epoch: 1 | total time: 35.64m
step 01373 (91.60%) | loss: 2.142342 | lrm: 0.17 | dt: 1662.21ms | tok/sec: 9,856 | mfu: 0.00 | epoch: 1 | total time: 35.67m
step 01374 (91.67%) | loss: 2.120920 | lrm: 0.17 | dt: 1665.40ms | tok/sec: 9,837 | mfu: 0.00 | epoch: 1 | total time: 35.70m
step 01375 (91.73%) | loss: 2.173443 | lrm: 0.17 | dt: 1721.70ms | tok/sec: 9,516 | mfu: 0.00 | epoch: 1 | total time: 35.73m
step 01376 (91.80%) | loss: 2.175649 | lrm: 0.16 | dt: 1729.66ms | tok/sec: 9,472 | mfu: 0.00 | epoch: 1 | total time: 35.75m
step 01377 (91.87%) | loss: 2.131496 | lrm: 0.16 | dt: 1704.62ms | tok/sec: 9,611 | mfu: 0.00 | epoch: 1 | total time: 35.78m
step 01378 (91.93%) | loss: 2.096909 | lrm: 0.16 | dt: 1660.24ms | tok/sec: 9,868 | mfu: 0.00 | epoch: 1 | total time: 35.81m
step 01379 (92.00%) | loss: 2.094062 | lrm: 0.16 | dt: 1666.69ms | tok/sec: 9,830 | mfu: 0.00 | epoch: 1 | total time: 35.84m
step 01380 (92.07%) | loss: 2.052499 | lrm: 0.16 | dt: 1676.27ms | tok/sec: 9,774 | mfu: 0.00 | epoch: 1 | total time: 35.87m
step 01381 (92.13%) | loss: 2.040930 | lrm: 0.16 | dt: 1673.32ms | tok/sec: 9,791 | mfu: 0.00 | epoch: 1 | total time: 35.89m
step 01382 (92.20%) | loss: 2.091157 | lrm: 0.16 | dt: 1694.28ms | tok/sec: 9,670 | mfu: 0.00 | epoch: 1 | total time: 35.92m
step 01383 (92.27%) | loss: 2.107777 | lrm: 0.15 | dt: 1663.91ms | tok/sec: 9,846 | mfu: 0.00 | epoch: 1 | total time: 35.95m
step 01384 (92.33%) | loss: 2.116868 | lrm: 0.15 | dt: 1666.56ms | tok/sec: 9,831 | mfu: 0.00 | epoch: 1 | total time: 35.98m
step 01385 (92.40%) | loss: 2.182498 | lrm: 0.15 | dt: 1664.96ms | tok/sec: 9,840 | mfu: 0.00 | epoch: 1 | total time: 36.01m
step 01386 (92.47%) | loss: 2.203717 | lrm: 0.15 | dt: 1752.32ms | tok/sec: 9,349 | mfu: 0.00 | epoch: 1 | total time: 36.03m
step 01387 (92.53%) | loss: 2.182814 | lrm: 0.15 | dt: 1699.77ms | tok/sec: 9,638 | mfu: 0.00 | epoch: 1 | total time: 36.06m
step 01388 (92.60%) | loss: 2.154381 | lrm: 0.15 | dt: 1701.60ms | tok/sec: 9,628 | mfu: 0.00 | epoch: 1 | total time: 36.09m
step 01389 (92.67%) | loss: 2.127697 | lrm: 0.15 | dt: 1671.72ms | tok/sec: 9,800 | mfu: 0.00 | epoch: 1 | total time: 36.12m
step 01390 (92.73%) | loss: 2.141222 | lrm: 0.15 | dt: 1653.59ms | tok/sec: 9,908 | mfu: 0.00 | epoch: 1 | total time: 36.15m
step 01391 (92.80%) | loss: 2.117934 | lrm: 0.14 | dt: 1668.90ms | tok/sec: 9,817 | mfu: 0.00 | epoch: 1 | total time: 36.17m
step 01392 (92.87%) | loss: 2.107858 | lrm: 0.14 | dt: 1614.61ms | tok/sec: 10,147 | mfu: 0.00 | epoch: 1 | total time: 36.20m
step 01393 (92.93%) | loss: 2.120962 | lrm: 0.14 | dt: 1705.95ms | tok/sec: 9,604 | mfu: 0.00 | epoch: 1 | total time: 36.23m
step 01394 (93.00%) | loss: 2.147943 | lrm: 0.14 | dt: 1670.60ms | tok/sec: 9,807 | mfu: 0.00 | epoch: 1 | total time: 36.26m
step 01395 (93.07%) | loss: 2.165507 | lrm: 0.14 | dt: 1558.88ms | tok/sec: 10,510 | mfu: 0.00 | epoch: 1 | total time: 36.28m
step 01396 (93.13%) | loss: 2.224345 | lrm: 0.14 | dt: 1562.58ms | tok/sec: 10,485 | mfu: 0.00 | epoch: 1 | total time: 36.31m
step 01397 (93.20%) | loss: 2.199819 | lrm: 0.14 | dt: 1607.06ms | tok/sec: 10,195 | mfu: 0.00 | epoch: 1 | total time: 36.34m
step 01398 (93.27%) | loss: 2.158992 | lrm: 0.13 | dt: 1542.24ms | tok/sec: 10,623 | mfu: 0.00 | epoch: 1 | total time: 36.36m
step 01399 (93.33%) | loss: 2.113627 | lrm: 0.13 | dt: 1550.68ms | tok/sec: 10,565 | mfu: 0.00 | epoch: 1 | total time: 36.39m
step 01400 (93.40%) | loss: 2.112651 | lrm: 0.13 | dt: 1502.35ms | tok/sec: 10,905 | mfu: 0.00 | epoch: 1 | total time: 36.41m
Step 01400 | Validation bpb: 0.7486
step 01401 (93.47%) | loss: 2.128661 | lrm: 0.13 | dt: 1540.75ms | tok/sec: 10,633 | mfu: 0.00 | epoch: 1 | total time: 36.44m
step 01402 (93.53%) | loss: 2.086680 | lrm: 0.13 | dt: 1533.12ms | tok/sec: 10,686 | mfu: 0.00 | epoch: 1 | total time: 36.46m
step 01403 (93.60%) | loss: 2.061133 | lrm: 0.13 | dt: 1532.95ms | tok/sec: 10,687 | mfu: 0.00 | epoch: 1 | total time: 36.49m
step 01404 (93.67%) | loss: 2.093825 | lrm: 0.13 | dt: 1523.09ms | tok/sec: 10,757 | mfu: 0.00 | epoch: 1 | total time: 36.52m
step 01405 (93.73%) | loss: 2.053725 | lrm: 0.13 | dt: 1544.22ms | tok/sec: 10,609 | mfu: 0.00 | epoch: 1 | total time: 36.54m
step 01406 (93.80%) | loss: 2.048801 | lrm: 0.12 | dt: 1540.21ms | tok/sec: 10,637 | mfu: 0.00 | epoch: 1 | total time: 36.57m
step 01407 (93.87%) | loss: 2.048138 | lrm: 0.12 | dt: 1507.42ms | tok/sec: 10,868 | mfu: 0.00 | epoch: 1 | total time: 36.59m
step 01408 (93.93%) | loss: 2.084118 | lrm: 0.12 | dt: 1531.82ms | tok/sec: 10,695 | mfu: 0.00 | epoch: 1 | total time: 36.62m
step 01409 (94.00%) | loss: 2.090783 | lrm: 0.12 | dt: 1537.99ms | tok/sec: 10,652 | mfu: 0.00 | epoch: 1 | total time: 36.64m
step 01410 (94.07%) | loss: 2.142468 | lrm: 0.12 | dt: 1545.02ms | tok/sec: 10,604 | mfu: 0.00 | epoch: 1 | total time: 36.67m
step 01411 (94.13%) | loss: 2.102462 | lrm: 0.12 | dt: 1526.78ms | tok/sec: 10,731 | mfu: 0.00 | epoch: 1 | total time: 36.69m
step 01412 (94.20%) | loss: 2.129962 | lrm: 0.12 | dt: 1522.64ms | tok/sec: 10,760 | mfu: 0.00 | epoch: 1 | total time: 36.72m
step 01413 (94.27%) | loss: 2.152641 | lrm: 0.11 | dt: 1550.09ms | tok/sec: 10,569 | mfu: 0.00 | epoch: 1 | total time: 36.75m
step 01414 (94.33%) | loss: 2.149210 | lrm: 0.11 | dt: 1525.96ms | tok/sec: 10,736 | mfu: 0.00 | epoch: 1 | total time: 36.77m
step 01415 (94.40%) | loss: 2.167593 | lrm: 0.11 | dt: 1522.44ms | tok/sec: 10,761 | mfu: 0.00 | epoch: 1 | total time: 36.80m
step 01416 (94.47%) | loss: 2.125535 | lrm: 0.11 | dt: 1549.91ms | tok/sec: 10,570 | mfu: 0.00 | epoch: 1 | total time: 36.82m
step 01417 (94.53%) | loss: 2.117370 | lrm: 0.11 | dt: 1536.32ms | tok/sec: 10,664 | mfu: 0.00 | epoch: 1 | total time: 36.85m
step 01418 (94.60%) | loss: 2.121282 | lrm: 0.11 | dt: 1573.44ms | tok/sec: 10,412 | mfu: 0.00 | epoch: 1 | total time: 36.87m
step 01419 (94.67%) | loss: 2.119615 | lrm: 0.11 | dt: 1530.66ms | tok/sec: 10,703 | mfu: 0.00 | epoch: 1 | total time: 36.90m
step 01420 (94.73%) | loss: 2.114158 | lrm: 0.11 | dt: 1542.83ms | tok/sec: 10,619 | mfu: 0.00 | epoch: 1 | total time: 36.93m
step 01421 (94.80%) | loss: 2.122190 | lrm: 0.10 | dt: 1523.66ms | tok/sec: 10,753 | mfu: 0.00 | epoch: 1 | total time: 36.95m
step 01422 (94.87%) | loss: 2.099435 | lrm: 0.10 | dt: 1543.61ms | tok/sec: 10,614 | mfu: 0.00 | epoch: 1 | total time: 36.98m
step 01423 (94.93%) | loss: 2.089699 | lrm: 0.10 | dt: 1554.94ms | tok/sec: 10,536 | mfu: 0.00 | epoch: 1 | total time: 37.00m
step 01424 (95.00%) | loss: 2.103743 | lrm: 0.10 | dt: 1511.97ms | tok/sec: 10,836 | mfu: 0.00 | epoch: 1 | total time: 37.03m
step 01425 (95.07%) | loss: 2.090156 | lrm: 0.10 | dt: 1747.11ms | tok/sec: 9,377 | mfu: 0.00 | epoch: 1 | total time: 37.06m
step 01426 (95.13%) | loss: 2.094877 | lrm: 0.10 | dt: 1618.11ms | tok/sec: 10,125 | mfu: 0.00 | epoch: 1 | total time: 37.08m
step 01427 (95.20%) | loss: 2.076190 | lrm: 0.10 | dt: 1565.04ms | tok/sec: 10,468 | mfu: 0.00 | epoch: 1 | total time: 37.11m
step 01428 (95.27%) | loss: 2.051798 | lrm: 0.09 | dt: 1581.12ms | tok/sec: 10,362 | mfu: 0.00 | epoch: 1 | total time: 37.14m
step 01429 (95.33%) | loss: 2.002917 | lrm: 0.09 | dt: 1667.29ms | tok/sec: 9,826 | mfu: 0.00 | epoch: 1 | total time: 37.16m
step 01430 (95.40%) | loss: 1.994453 | lrm: 0.09 | dt: 1670.99ms | tok/sec: 9,804 | mfu: 0.00 | epoch: 1 | total time: 37.19m
step 01431 (95.47%) | loss: 1.941805 | lrm: 0.09 | dt: 1580.49ms | tok/sec: 10,366 | mfu: 0.00 | epoch: 1 | total time: 37.22m
step 01432 (95.53%) | loss: 1.947912 | lrm: 0.09 | dt: 1559.36ms | tok/sec: 10,506 | mfu: 0.00 | epoch: 1 | total time: 37.24m
step 01433 (95.60%) | loss: 1.997159 | lrm: 0.09 | dt: 1625.14ms | tok/sec: 10,081 | mfu: 0.00 | epoch: 1 | total time: 37.27m
step 01434 (95.67%) | loss: 2.014664 | lrm: 0.09 | dt: 1590.01ms | tok/sec: 10,304 | mfu: 0.00 | epoch: 1 | total time: 37.30m
step 01435 (95.73%) | loss: 2.070365 | lrm: 0.09 | dt: 1651.82ms | tok/sec: 9,918 | mfu: 0.00 | epoch: 1 | total time: 37.32m
step 01436 (95.80%) | loss: 2.027401 | lrm: 0.08 | dt: 1573.89ms | tok/sec: 10,409 | mfu: 0.00 | epoch: 1 | total time: 37.35m
step 01437 (95.87%) | loss: 2.034600 | lrm: 0.08 | dt: 1761.88ms | tok/sec: 9,299 | mfu: 0.00 | epoch: 1 | total time: 37.38m
step 01438 (95.93%) | loss: 1.995190 | lrm: 0.08 | dt: 1585.08ms | tok/sec: 10,336 | mfu: 0.00 | epoch: 1 | total time: 37.41m
step 01439 (96.00%) | loss: 1.994835 | lrm: 0.08 | dt: 1682.84ms | tok/sec: 9,735 | mfu: 0.00 | epoch: 1 | total time: 37.44m
step 01440 (96.07%) | loss: 1.999576 | lrm: 0.08 | dt: 1575.58ms | tok/sec: 10,398 | mfu: 0.00 | epoch: 1 | total time: 37.46m
step 01441 (96.13%) | loss: 1.961749 | lrm: 0.08 | dt: 1541.35ms | tok/sec: 10,629 | mfu: 0.00 | epoch: 1 | total time: 37.49m
step 01442 (96.20%) | loss: 1.943555 | lrm: 0.08 | dt: 1534.06ms | tok/sec: 10,680 | mfu: 0.00 | epoch: 1 | total time: 37.51m
step 01443 (96.27%) | loss: 1.978611 | lrm: 0.07 | dt: 1531.36ms | tok/sec: 10,698 | mfu: 0.00 | epoch: 1 | total time: 37.54m
step 01444 (96.33%) | loss: 1.952200 | lrm: 0.07 | dt: 1513.31ms | tok/sec: 10,826 | mfu: 0.00 | epoch: 1 | total time: 37.56m
step 01445 (96.40%) | loss: 1.906117 | lrm: 0.07 | dt: 1636.60ms | tok/sec: 10,010 | mfu: 0.00 | epoch: 1 | total time: 37.59m
step 01446 (96.47%) | loss: 1.905336 | lrm: 0.07 | dt: 1681.25ms | tok/sec: 9,745 | mfu: 0.00 | epoch: 1 | total time: 37.62m
step 01447 (96.53%) | loss: 1.884408 | lrm: 0.07 | dt: 1678.07ms | tok/sec: 9,763 | mfu: 0.00 | epoch: 1 | total time: 37.65m
step 01448 (96.60%) | loss: 1.881189 | lrm: 0.07 | dt: 1537.15ms | tok/sec: 10,658 | mfu: 0.00 | epoch: 1 | total time: 37.67m
step 01449 (96.67%) | loss: 1.876938 | lrm: 0.07 | dt: 1750.56ms | tok/sec: 9,359 | mfu: 0.00 | epoch: 1 | total time: 37.70m
step 01450 (96.73%) | loss: 1.901120 | lrm: 0.07 | dt: 1871.32ms | tok/sec: 8,755 | mfu: 0.00 | epoch: 1 | total time: 37.73m
step 01451 (96.80%) | loss: 1.884345 | lrm: 0.06 | dt: 1686.95ms | tok/sec: 9,712 | mfu: 0.00 | epoch: 1 | total time: 37.76m
step 01452 (96.87%) | loss: 1.939825 | lrm: 0.06 | dt: 1545.95ms | tok/sec: 10,598 | mfu: 0.00 | epoch: 1 | total time: 37.79m
step 01453 (96.93%) | loss: 1.940070 | lrm: 0.06 | dt: 1617.52ms | tok/sec: 10,129 | mfu: 0.00 | epoch: 1 | total time: 37.81m
step 01454 (97.00%) | loss: 2.011664 | lrm: 0.06 | dt: 1786.91ms | tok/sec: 9,168 | mfu: 0.00 | epoch: 1 | total time: 37.84m
step 01455 (97.07%) | loss: 2.042305 | lrm: 0.06 | dt: 1596.61ms | tok/sec: 10,261 | mfu: 0.00 | epoch: 1 | total time: 37.87m
step 01456 (97.13%) | loss: 2.053365 | lrm: 0.06 | dt: 1583.70ms | tok/sec: 10,345 | mfu: 0.00 | epoch: 1 | total time: 37.90m
step 01457 (97.20%) | loss: 2.030436 | lrm: 0.06 | dt: 1614.99ms | tok/sec: 10,144 | mfu: 0.00 | epoch: 1 | total time: 37.92m
step 01458 (97.27%) | loss: 2.004280 | lrm: 0.05 | dt: 1562.16ms | tok/sec: 10,488 | mfu: 0.00 | epoch: 1 | total time: 37.95m
step 01459 (97.33%) | loss: 2.025508 | lrm: 0.05 | dt: 1566.21ms | tok/sec: 10,460 | mfu: 0.00 | epoch: 1 | total time: 37.98m
step 01460 (97.40%) | loss: 2.064194 | lrm: 0.05 | dt: 1577.28ms | tok/sec: 10,387 | mfu: 0.00 | epoch: 1 | total time: 38.00m
step 01461 (97.47%) | loss: 2.086715 | lrm: 0.05 | dt: 1526.02ms | tok/sec: 10,736 | mfu: 0.00 | epoch: 1 | total time: 38.03m
step 01462 (97.53%) | loss: 2.106520 | lrm: 0.05 | dt: 1577.97ms | tok/sec: 10,382 | mfu: 0.00 | epoch: 1 | total time: 38.05m
step 01463 (97.60%) | loss: 2.094211 | lrm: 0.05 | dt: 1554.10ms | tok/sec: 10,542 | mfu: 0.00 | epoch: 1 | total time: 38.08m
step 01464 (97.67%) | loss: 2.127970 | lrm: 0.05 | dt: 1568.82ms | tok/sec: 10,443 | mfu: 0.00 | epoch: 1 | total time: 38.11m
step 01465 (97.73%) | loss: 2.156082 | lrm: 0.05 | dt: 1579.40ms | tok/sec: 10,373 | mfu: 0.00 | epoch: 1 | total time: 38.13m
step 01466 (97.80%) | loss: 2.142095 | lrm: 0.04 | dt: 1552.54ms | tok/sec: 10,553 | mfu: 0.00 | epoch: 1 | total time: 38.16m
step 01467 (97.87%) | loss: 2.145328 | lrm: 0.04 | dt: 1581.24ms | tok/sec: 10,361 | mfu: 0.00 | epoch: 1 | total time: 38.18m
step 01468 (97.93%) | loss: 2.111837 | lrm: 0.04 | dt: 1590.27ms | tok/sec: 10,302 | mfu: 0.00 | epoch: 1 | total time: 38.21m
step 01469 (98.00%) | loss: 2.126673 | lrm: 0.04 | dt: 1658.97ms | tok/sec: 9,876 | mfu: 0.00 | epoch: 1 | total time: 38.24m
step 01470 (98.07%) | loss: 2.073390 | lrm: 0.04 | dt: 1680.41ms | tok/sec: 9,749 | mfu: 0.00 | epoch: 1 | total time: 38.27m
step 01471 (98.13%) | loss: 2.124494 | lrm: 0.04 | dt: 1643.54ms | tok/sec: 9,968 | mfu: 0.00 | epoch: 1 | total time: 38.29m
step 01472 (98.20%) | loss: 2.122321 | lrm: 0.04 | dt: 1658.59ms | tok/sec: 9,878 | mfu: 0.00 | epoch: 1 | total time: 38.32m
step 01473 (98.27%) | loss: 2.109634 | lrm: 0.03 | dt: 1635.48ms | tok/sec: 10,017 | mfu: 0.00 | epoch: 1 | total time: 38.35m
step 01474 (98.33%) | loss: 2.123900 | lrm: 0.03 | dt: 1663.87ms | tok/sec: 9,846 | mfu: 0.00 | epoch: 1 | total time: 38.38m
step 01475 (98.40%) | loss: 2.153559 | lrm: 0.03 | dt: 1640.12ms | tok/sec: 9,989 | mfu: 0.00 | epoch: 1 | total time: 38.40m
step 01476 (98.47%) | loss: 2.150526 | lrm: 0.03 | dt: 1652.16ms | tok/sec: 9,916 | mfu: 0.00 | epoch: 1 | total time: 38.43m
step 01477 (98.53%) | loss: 2.160388 | lrm: 0.03 | dt: 1645.06ms | tok/sec: 9,959 | mfu: 0.00 | epoch: 1 | total time: 38.46m
step 01478 (98.60%) | loss: 2.147634 | lrm: 0.03 | dt: 1636.73ms | tok/sec: 10,010 | mfu: 0.00 | epoch: 1 | total time: 38.49m
step 01479 (98.67%) | loss: 2.145989 | lrm: 0.03 | dt: 1600.38ms | tok/sec: 10,237 | mfu: 0.00 | epoch: 1 | total time: 38.51m
step 01480 (98.73%) | loss: 2.177011 | lrm: 0.03 | dt: 1669.25ms | tok/sec: 9,815 | mfu: 0.00 | epoch: 1 | total time: 38.54m
step 01481 (98.80%) | loss: 2.170497 | lrm: 0.02 | dt: 1610.20ms | tok/sec: 10,175 | mfu: 0.00 | epoch: 1 | total time: 38.57m
step 01482 (98.87%) | loss: 2.145211 | lrm: 0.02 | dt: 1580.63ms | tok/sec: 10,365 | mfu: 0.00 | epoch: 1 | total time: 38.59m
step 01483 (98.93%) | loss: 2.101080 | lrm: 0.02 | dt: 1662.60ms | tok/sec: 9,854 | mfu: 0.00 | epoch: 1 | total time: 38.62m
step 01484 (99.00%) | loss: 2.084803 | lrm: 0.02 | dt: 1626.57ms | tok/sec: 10,072 | mfu: 0.00 | epoch: 1 | total time: 38.65m
step 01485 (99.07%) | loss: 2.091653 | lrm: 0.02 | dt: 1697.37ms | tok/sec: 9,652 | mfu: 0.00 | epoch: 1 | total time: 38.68m
step 01486 (99.13%) | loss: 2.096558 | lrm: 0.02 | dt: 1664.25ms | tok/sec: 9,844 | mfu: 0.00 | epoch: 1 | total time: 38.70m
step 01487 (99.20%) | loss: 2.107173 | lrm: 0.02 | dt: 1642.90ms | tok/sec: 9,972 | mfu: 0.00 | epoch: 1 | total time: 38.73m
step 01488 (99.27%) | loss: 2.150076 | lrm: 0.01 | dt: 1660.01ms | tok/sec: 9,869 | mfu: 0.00 | epoch: 1 | total time: 38.76m
step 01489 (99.33%) | loss: 2.155721 | lrm: 0.01 | dt: 1652.26ms | tok/sec: 9,916 | mfu: 0.00 | epoch: 1 | total time: 38.79m
step 01490 (99.40%) | loss: 2.140055 | lrm: 0.01 | dt: 1720.81ms | tok/sec: 9,521 | mfu: 0.00 | epoch: 1 | total time: 38.82m
step 01491 (99.47%) | loss: 2.161542 | lrm: 0.01 | dt: 1725.60ms | tok/sec: 9,494 | mfu: 0.00 | epoch: 1 | total time: 38.84m
step 01492 (99.53%) | loss: 2.137895 | lrm: 0.01 | dt: 1683.90ms | tok/sec: 9,729 | mfu: 0.00 | epoch: 1 | total time: 38.87m
step 01493 (99.60%) | loss: 2.073720 | lrm: 0.01 | dt: 1629.53ms | tok/sec: 10,054 | mfu: 0.00 | epoch: 1 | total time: 38.90m
step 01494 (99.67%) | loss: 2.105495 | lrm: 0.01 | dt: 1693.42ms | tok/sec: 9,675 | mfu: 0.00 | epoch: 1 | total time: 38.93m
step 01495 (99.73%) | loss: 2.101427 | lrm: 0.01 | dt: 1686.90ms | tok/sec: 9,712 | mfu: 0.00 | epoch: 1 | total time: 38.96m
step 01496 (99.80%) | loss: 2.094071 | lrm: 0.00 | dt: 1645.63ms | tok/sec: 9,956 | mfu: 0.00 | epoch: 1 | total time: 38.98m
step 01497 (99.87%) | loss: 2.095949 | lrm: 0.00 | dt: 1683.80ms | tok/sec: 9,730 | mfu: 0.00 | epoch: 1 | total time: 39.01m
step 01498 (99.93%) | loss: 2.052968 | lrm: 0.00 | dt: 1591.57ms | tok/sec: 10,294 | mfu: 0.00 | epoch: 1 | total time: 39.04m
step 01499 (100.00%) | loss: 2.026530 | lrm: 0.00 | dt: 1756.95ms | tok/sec: 9,325 | mfu: 0.00 | epoch: 1 | total time: 39.07m
Step 01499 | Validation bpb: 0.7389
2026-08-04 14:31:08,605 - nanochat.checkpoint_manager - INFO - Saved model parameters to: $ORX_RUN_DIR/repo/.cache/nanochat/chatsft_checkpoints/d6/model_001499.pt
2026-08-04 14:31:08,607 - nanochat.checkpoint_manager - INFO - Saved metadata to: $ORX_RUN_DIR/repo/.cache/nanochat/chatsft_checkpoints/d6/meta_001499.json
2026-08-04 14:31:12,505 - nanochat.checkpoint_manager - INFO - Saved optimizer state to: $ORX_RUN_DIR/repo/.cache/nanochat/chatsft_checkpoints/d6/optim_001499_rank0.pt
Peak memory usage: 0.00MiB
Total training time: 39.07m
Minimum validation bpb: 0.7389

=== Chat confirmation ===
Using completed SFT checkpoint model_001499.pt
Autodetected device type: mps
2026-08-04 14:33:21,843 - nanochat.common - INFO - Distributed world size: 1
2026-08-04 14:33:21,843 - nanochat.checkpoint_manager - INFO - No model tag provided, guessing model tag: d6
2026-08-04 14:33:21,844 - nanochat.checkpoint_manager - INFO - Loading model from $ORX_RUN_DIR/repo/.cache/nanochat/chatsft_checkpoints/d6 with step 1499
2026-08-04 14:33:21,965 - nanochat.checkpoint_manager - INFO - Building model with config: {'sequence_len': 512, 'vocab_size': 32768, 'n_layer': 6, 'n_head': 6, 'n_kv_head': 6, 'n_embd': 384, 'window_pattern': 'L'}

NanoChat Interactive Mode
--------------------------------------------------
Type 'quit' or 'exit' to end the conversation
Type 'clear' to start a new conversation
--------------------------------------------------

Assistant:

``` Paris
Paris is a city known for its historical and cultural significance. The capital of France is Paris, a 1715,345,345,345,345,345,345,345,345,345,345,345,345,345,345,345,345,345,345,345,345,345,345,345,345,345,345,345,345,

run completed successfully
