Bigger Isn’t Always Better: Rethinking Model Size in AI

Every day I look at my news feed and see yet another headline about a new LLM model that is supposed to be even better and more powerfull because it is larger and trained on with more parameters and is bigger than its predecessor. It feels like every company is constantly trying to release a new large model so they can claim the number one spot in benchmarks or their own comparisons. But is model size really always a direct indicator of whether a model is "better"? Unfortunately, this question is asked far too rarely when models are actually used, because there is a persistent assumption that scaling models automatically leads to better performance.

To understand what actually determines the size of a model and what influence this has, we need to take a look at what "larger" really means on a technical level. With this understanding, we can see exactly how the model size affects training and usage.

What "Parameters" really are

A model generally consists of computational operations that can vary depending on the type of model. In the case of a neural network which are the most base for LLMs, these would be matrix multiplications, for example. An input to such a neural model is therefore represented as a vector and multiplied by so-called weight matrices. The values in these matrices are the parameters. So when we say a model has billions of parameters, we mean billions of learned numerical weights that transform input vectors into output vectors. The resulting output is then further processed by an activation function.

Let’s stick with neural networks as the architecture. These consist of many neurons. Each neuron can be thought of as a weighted sum of its inputs followed by an activation function. In practice, however, this is implemented as vector-matrix multiplication rather than as isolated biological neurons.

More parameters increase the model’s ability to represent complex transformations. So one might think, quite simply, that choosing the maximum number of parameters allowed by technical limitations would yield the best and most powerful model. However, this is exactly where the problem arises when the number of parameters becomes large. This additional capacity is only relevant if it is effectively trained and utilized. A higher number of parameters also increases computational cost, but in practice the total training effort is determined by both the model size and the amount of data processed, often roughly proportional to the product of parameters.

If we now consider LLMs with the Transformer architecture, this means that, in addition to the weights as parameters, there is another very significant factor that determines the model’s size. The data used as input and output for the LLM is represented as training tokens in an abstract form. In text data, a token is a part of a word; for example, the word "transformer" is represented by the two tokens "transform" and "er". Thus, all words and sentences in the input and output data are mapped to the corresponding tokens. The sentence "Transformers are revolutionizing AI research" gets represented as the respective tokens as you can see in the following image.

Thus, a sentence consisting of 5 words is represented by 7 tokens with the same meaning. The number of tokens is therefore determined by the tokenization strategy and thus has a direct impact on the resulting model size and performance.

Scaling-law research such as DeepMind’s Chinchilla paper has shown that for example a ratio of 20:1 between training tokens and parameters represents a good balance between training time and model performance. However, the specific values depend heavily on the experience, intuition, and testing conducted by the individual AI engineer. So you could also say that, for developers, choosing the initial number is just as difficult as choosing variable names.

It's clear to see how the trend in numbers has developed over the past few years. To illustrate this, let's take a look at a few models from recent years:

Unfortunately, the major providers no longer publish their exact numbers, but the extreme trend toward ever-increasing numbers of parameters and training tokens is very clear. It can also be seen that the providers themselves have realized that it makes sense to release smaller models with significantly fewer parameters.

The training, even though it should be impossible

Training a model involves adjusting its parameters through an optimization process (gradient descent). Simplified, this is an optimization process in which the model calculates how inaccurate its prediction is and then slightly adjusts its weights to reduce this error.

For larger models with more parameters and training tokens, the optimization landscape becomes more complex, as higher-dimensional parameter spaces introduce more potential problems, such as local minima and saddle points. This phenomenon is known as the curse of dimensionality. It describes the observation that data distribution in very high dimensions is not uniform but sparse.

It’s easy to visualize this using the example of looking for an apartment. When you’re searching for a new apartment based on criteria like price and square footage, you’ll initially find many results that meet those criteria. But if we add more criteria - such as the number of rooms, whether pets are allowed, and whether there’s a garage or balcony - we increase the dimensionality with each new criterion. The number of apartments that meet all criteria simultaneously becomes smaller and smaller.

The curse of dimensionality is therefore a complex and difficult problem to solve in classical machine learning methods. However, when we look at the LLM architecture, we need to view the problem somewhat differently to understand why it isn’t as apparent and what exactly the architecture does differently to counteract it.

LLMs are trained on extremely large amounts of data; for example, top models like LLaMA 3 from 2024 use more than 15 trillion tokens of raw data. This scale of training data was unthinkable just a few years ago and is now giving rise to entirely new problems and solutions, which we will examine in more detail later in this article. Back to the LLMs: The curse of dimensionality problem manifests itself precisely through this symptom of modern models’ extreme “data hunger.” The reason for this is that LLMs operate in very high-dimensional representation spaces where data points are extremely sparsely distributed, and only a small fraction of all possible combinations is actually observed during training. We recall the example of the apartment search and the criteria that come into play here.

To simplify our understanding of the problem, let’s consider another example using the word “pitch.” In a simple, low-dimensional framework, similarity could still be explained along a few axes, such as “music” vs. “sports.” In the high-dimensional linguistic space of an LLM, however, “pitch” is described simultaneously by a great many factors:

• the sentence context
• typical neighboring words
• grammatical function
• thematic context
• stylistic cues

This results in very different meanings:

• "The singer has perfect pitch."
• "He threw a fast pitch."
• "She made a startup pitch to investors."

Although the same word is used, its representations in high-dimensional space are far apart because many dimensions (topic, context, typical co-occurring words) differ. At the same time, completely different words can be “closer” in certain contexts if they share similar patterns across many of these dimensions.

As a result, the LLM model requires a very large number of training examples to learn stable statistical patterns across words, contexts, and semantic spaces, as rare or complex combinations would otherwise be barely covered. At the same time, classical intuitions about “proximity” or “similarity” lose their meaning in high-dimensional space, which further complicates learning. Thus, the LLM architecture has not solved the problem and remains subject to it, though not in the same way as traditional machine learning methods.

More data is all that matters!?

So we could simply solve the curse of dimensionality problem by using even more data with even larger models. This is a naive approach, however, one that does not address the root cause but merely attempts to mitigate the symptom. But a larger dataset cannot resolve the symptom, since it does not specifically address the root cause of the problem. The problem lies in the language itself, as it is highly diverse and complex. While using more data may seem sensible at first glance, it also means that poor data points end up in the training data unchecked simply due to the sheer size of the dataset.

Thus, simply scaling the data volume blindly would be counterproductive and lead to a poorer overall modeling result. Let’s consider a simple example: If we read 100 specialized books or 1,000 random internet posts, we gain significantly more knowledge from the specialized books than from the 10 times larger number of completely different internet posts. Let’s also assume that the information in the internet posts is repetitive or even contradictory. Then we would increasingly memorize duplicates and incorrect information. This is exactly what happens with a training dataset for an LLM. This makes it all the more important to select the right data beforehand during training and not simply prioritize quantity over quality, since quantity can naturally be scaled much more easily through automated data collection than quality, which in the most labor-intensive cases requires human curation. The goal is therefore to find the sweet spot between quantity and quality, allowing the model size to be chosen appropriately so that the information from the training data can be meaningfully represented internally. A model that is too large therefore always requires a correspondingly large amount of high-quality data and not simply a large volume of low-quality data collected through automated processes.

Modern architectures such as Transformer models, however, partially mitigate this problem by using attention mechanisms to selectively target relevant information, thereby narrowing the effective search space. Additionally, very large, well-filtered datasets as well as scaling laws such as the aforementioned Chinchilla Scaling Law help to make the relationship between model size and data volume more efficient. Instead of a direct “computational explosion” across all dimensions, the effect is significantly mitigated through architectural design, data strategy, and scaling, and transformed into a manageable training problem.

Clean data is the key

A well-filtered dataset can therefore help mitigate the problem, even if it doesn’t solve it entirely. This naturally raises the question: aside from incorrect and duplicate data points, what makes a dataset good or bad, and how does one even assess whether a data point is good or bad when dealing with such a massive volume of data?

There are four main characteristics that define good data points for an LLM:

1. Error/Noise: meaningless or incorrect data
2. Relevance: Is a data point relevant to the use case?
3. Bias: Data exhibits a strong bias in one direction
4. Balance: The number of data points from different subject areas is not evenly distributed

Thus, erroneous or noisy data points would interfere with the model’s ability to learn the correct patterns during training, and data points irrelevant to the use case would reduce the model’s capacity to process the data points that are actually important. An example would be a model designed to generate medical diagnoses that is trained on patent data and novels. In this case, the model’s capacity for pattern recognition, such as identifying disease symptoms in patent data, is wasted on patterns found in novel content. Another major problem is biased data, as it is not only difficult to detect but also causes the model to learn patterns only in a predetermined direction, thereby limiting generalization. The same occurs with unbalanced datasets, for example, when one topic area is significantly overrepresented, which can again lead to model bias.

Common solution strategies therefore include:

• Errors/Noise:
  • Data deduplication
  • Automatic correction of formatting errors
  • Removal of spam/incorrect data points
  • Correction of data points
• Relevance:
  • Human relevance assessment
  • Filtering of the data source
  • Thematic grouping/filtering
• Bias:
  • Use of multiple different data sources
• Balancing:
  • Generation of artificial data points based on existing ones
  • Weighting of data points

Some of these strategies can be implemented automatically, such as selecting different data sources during automatic data collection. However, strategies such as weighting data points or assessing relevance clearly require some manual effort, which means that as datasets grow larger, it naturally becomes increasingly difficult to apply these techniques effectively across the entire dataset. The higher the number of parameters, the larger the clean training dataset must be in order to train a model that delivers good results. Otherwise, we’re wasting the potential offered by having more parameters, and we might actually achieve the same or even better model performance with fewer parameters and a smaller training dataset.

Influence on training and usage

We have now taken a closer look at the architecture of an LLM and understand why its architecture and the number of parameters require large amounts of data, as well as the requirements for the dataset. Now that we have a sufficiently good dataset, we need to determine how the model size and the corresponding data size influence the training process and the model’s performance after training.

Naturally, the more data points there are in a dataset, the more storage space the dataset requires. Let’s take the LLaMA 3 model as an example, which was trained on more than 15 trillion tokens. Based on a typical estimate of 4 bytes per token, this amounts to roughly 60TB of training data. To process this volume of data within a reasonable timeframe, more than 16,000 GPUs were used for training over a period of just over 50 days. This scale highlights precisely the resulting problem: scaling with LLM architecture due to ever-increasing data volumes and model sizes. Thus, in addition to the extreme data hunger, the associated training costs are a major issue, meaning that only larger companies can afford to directly develop their own models of this size. For example, training the LLaMa 3 model required more than $320 million in hardware costs alone, at approximately $20,000 per GPU (NVIDIA H100 80 GB). On top of that, an additional $2–3 million is needed to operate the cards, covering power consumption, cooling, and the remaining server components. When considering these costs against the backdrop that new models of this scale are released at least every 2–3 months and smaller model updates occur at significantly shorter, nearly weekly intervals, training LLMs with this large number of parameters results in extremely high costs and computation times.

In addition to training costs, the operational costs of the models are also a major factor and are, of course, heavily influenced by the number of parameters. While training requires extremely large amounts of data, which significantly drives up hardware requirements for processing alone, the use of a trained model with many parameters still necessitates a significant amount of fast memory.

Sticking with the LLaMA 3 model, if we want to run the 70B version, we would need about 140 GB of fast GPU memory; for the LLaMA 3.1 405B version, we would need 810 GB. That would be two NVIDIA H100s for the 70B model and 11 for the 405B model, or €40,000 and €220,000 in pure acquisition costs. And that doesn't even include the costs for the remaining components, such as the CPU and RAM. This scale is therefore far beyond what smaller companies can easily and quickly handle, even if we set aside the current general shortage of RAM caused by the situation.

However, the pure costs of operation or training, along with the model’s performance, are not the only major factors determining how well a model ultimately will be received by users. This is because a model with more parameters and the exact same architecture as a model with fewer parameters will run slower on the same hardware. Therefore, a model with more parameters always runs the risk of causing excessively high latency in real-world applications.

Take, for example, a voice assistant based on an LLM with Transformer architecture, which is currently the most popular. A model with fewer parameters would have a delay of only 0.5 seconds, while one with more parameters would have a delay of 1 second. That 1-second delay is already quite noticeable and feels unnatural. This is why the model with fewer parameters might feel better in terms of pure usability.

That is why there are already many techniques available for running models with a large number of parameters quickly or with lower hardware requirements. We will take a closer look at these in a future blog post.

Blunt scaling doesn't help anyone

At first glance, it may seem obvious that a model with more parameters should also be the better one. But as we have seen throughout this article, that assumption is far too simplistic. Model size by itself does not determine quality, because performance always depends on how well architecture, training data, and computational resources fit together. A larger model may have a greater theoretical capacity, but this capacity only becomes useful when it is supported by sufficiently large and carefully filtered training data. Without that foundation, the additional parameters do not create better generalization automatically. Instead, they can increase training complexity, demand more compute, raise inference latency, and make deployment significantly more expensive.

This is precisely why scaling laws, attention mechanisms, and better data curation matter so much. They do not eliminate the fundamental problem, but they help turn a brute force scaling problem into a more manageable engineering task. In practice, this means that the relationship between parameter count, token volume, and data quality is far more important than parameter count alone. The real challenge, then, is not to build the largest model possible, but to find the point at which model size, data quality, and practical usability are in balance. In many cases, a smaller model that has been trained on better data will be more efficient, more usable, and ultimately more effective than a much larger model that was scaled without enough care.

In other words, the future of LLM development will not be decided by blind scaling, but by intelligent design, disciplined data selection, and a realistic understanding of what a model actually needs in order to perform well.

This blog post is part of the ‘AI Myths’ series. You can find all other posts here.