LLM360 has made a significant leap in the world of open-source large language models (LLMs) with the release of K2 65B, a fully reproducible model that rivals the performance of Llama 2 70B. Trained on an impressive 1.4 trillion tokens, K2 65B sets a new standard for transparency and performance in the LLM community. The training process was meticulously documented, with 140 model checkpoints and the entire 1.3 trillion token dataset from the first training stage available on HuggingFace, ensuring that researchers and developers can fully understand and replicate the model's development.
The training of K2 65B was executed in two stages. The first stage involved training with a context length of 2048 tokens for 1.3 trillion tokens. In the second stage, the context length was extended to 8192 tokens, and an additional 69.3 billion tokens were used. This approach not only enhanced the model's capabilities but also provided valuable insights into the dynamics of training large-scale models. The base model checkpoints and finetuned chat model are both accessible under the Apache 2.0 license, while the data is provided under the ODC-By license, promoting open collaboration and innovation.
One of the standout features of K2 65B is its fully transparent training process, which is a rarity in the realm of large-scale LLMs. This transparency is crucial for the research community, as it allows for a deeper understanding of how model weights evolve during training and the impact of different datasets on model performance. The detailed technical report, available on LLM360's website, outlines the challenges encountered during the model's development and the solutions implemented, making it a valuable resource for anyone looking to delve into LLM pretraining behavior at a large scale.
For those interested in exploring the model further, the training data for the base model and the finetuned chat model can be found on HuggingFace. Additionally, HuggingFace user legraphista has uploaded GGUFs of both models, providing even more flexibility for researchers and developers. This level of accessibility and openness is a testament to LLM360's commitment to advancing the field of artificial intelligence through collaboration and transparency.
