In a recent study titled “The use of model-generated content in training large language models (LLMs)”, the authors delve into a critical issue that has significant implications for the field of machine learning and artificial intelligence. The paper discusses a phenomenon known as “model collapse,” which refers to the disappearance of the tails of the original content distribution in the resulting models due to the use of model-generated content in training large language models.
This issue is not isolated but is ubiquitous amongst all learned generative models. It is a matter of serious concern, especially considering the benefits derived from training with large-scale data scraped from the web.
The authors emphasize the increasing value of data collected from genuine human interactions with systems, especially in the context of the presence of content generated by large language models in data crawled from the Internet.
The paper suggests that the use of model-generated content in training large language models can lead to irreversible defects. These defects can significantly affect the performance and reliability of these models, making it a crucial area of research and development in the field of AI and machine learning.
The document provides a comprehensive analysis of the issue and offers valuable insights into the challenges and potential solutions associated with training large language models. It is a must-read for researchers, data scientists, and AI enthusiasts who are keen on understanding the intricacies of large language model training and the impact of model-generated content on these processes.
The cause of model collapse is primarily attributed to two types of errors: statistical approximation error and functional approximation error.
Statistical approximation error is the primary type of error, which arises due to the number of samples being finite, and disappears as the number of samples tends to infinity. This occurs due to a non-zero probability that information can get lost at every step of re-sampling. For instance, a single-dimensional Gaussian being approximated from a finite number of samples can still have significant errors, despite using a very large number of points.
Functional approximation error is a secondary type of error, which stems from our function approximators being insufficiently expressive (or sometimes too expressive outside of the original distribution support). For example, a neural network can introduce non-zero likelihood outside of the support of the original distribution. A simple example of this error is if we were to try fitting a mixture of two Gaussians with a single Gaussian. Even if we have perfect information about the data distribution, model errors will be inevitable.
These errors can cause model collapse to get worse or better. Better approximation power can even be a double-edged sword – better expressiveness may counteract statistical noise, resulting in a good approximation of the true distribution, but it can equally compound this noise. More often then not, we get a cascading effect where combined individual inaccuracy causes the overall error to grow. Overfitting the density model will cause the model to extrapolate incorrectly and might give high density to low-density regions not covered in the training set support; these will then be sampled with arbitrary frequency.
It’s also worth mentioning that modern computers also have a further computational error coming from the way floating point numbers are represented. This error is not evenly spread across different floating point ranges, making it hard to estimate the precise value of a given number. Such errors are smaller in magnitude and are fixable with more precise hardware, making them less influential on model collapse.
For more detailed insights, you can access the full paper here.