We are happy to announce that our Novus Research The Turkish LLM has topped the OpenLLM Turkey leaderboard! 🏆

‍

Our model, NovusResearch/Novus-7b-tr_v1, is a fully fine-tuned model that has undergone extensive training on various Turkish datasets. These datasets mainly consist of translated versions from the teknium/OpenHermes-2.5 and Open-Orca/SlimOrca datasets.

In our initial experiments, we found that traditional LoRA-based fine-tuning does not improve performance benchmarks. In fact, performance degraded in many runs, especially in the GSM8K benchmark.

Looking at competitors, we found that Trendyol uses Low Rank Adaptation (LoRA) but we had more success using the full fine-tuning model.

‍

What makes LoRA different from fine-tuning, and why did we decide to go with fine-tuning?

‍

Low Rank Adaptation (LoRA) is an innovative approach to fine-tuning deep learning models. It achieves this by reducing the number of trainable parameters, which not only improves efficiency but also enables seamless switching between different tasks.

‍

Full fine-tuning, on the other hand, involves fine-tuning all of the parameters of the pre-trained model on a specific task or dataset. This approach allows the model to learn task-specific features and nuances, potentially leading to better performance on the target task. However, full fine-tuning may require more computational resources and time compared to LoRA-based fine-tuning. This is the reason why we decided to go for full fine-tuning.

Our focus has been on incorporating knowledge through pre-training and fully fine-tuning models. We believe that traditional LoRA-based fine tuning only allows LLMs to adapt to different styles without adding additional information.

With the addition of new GPUs, we are expanding our efforts on continuous pre-education and aim to contribute more to the Turkish open-source community!

We are very excited to be a part of this journey and look forward to more to come. 🚀