Introduction and Research Gap
Developing AI-detection software has helped combat online misinformation campaigns that render users vulnerable to political, social, or even economic manipulation. However, there are still many false positives that turn users away from trusting or even using such software. This problem can be attributed to bias in unrepresentative data samples, inadequate models due to collaborative restraints, and lack of manpower for the proper facilitation of model evaluation. Thus, our proposal aims to implement a standardized system with automatic data augmentation, model stacking, and feedback loop mechanisms to address these problems respectively and improve the accuracy of AI-detection software.
Data Augmentation Techniques
Data augmentation is an innovative technique which can play a crucial role in reducing the number of false positives. It enhances the accuracy of the machine and AI learning models. This is done through artificially generating diverse versions of a dataset, increasing its diversity and size. This consequently allows the model to learn from a bigger range of examples. These various techniques can be used at different levels, including word, text, character and audio data. For image data, data augmentation can be used by applying color space and geometric transformations. This involves resizing, cropping, flipping and adjusting the contrast and brightness, and allows the model to generalize better. In text and audio data, techniques are similar and focus on creating a more diverse representation of contexts and patterns. Overall, by introducing the additional realistic artificial samples and variations automatically, the machine learning models can learn and increase its diversity. Thus, it helps mitigate false positives as the model has increased its ability to handle distinct variations and generalize.
Model Stacking
In the process of model stacking, predictions are not generated solely by a single model. Instead, multiple diverse models are employed to produce predictions, and these predictions are subsequently utilized as features for a more advanced meta-model. This approach proves particularly effective when employing a mix of lower-level learners with distinct strengths that collectively enhance the performance of the meta-model. The strength of model stacking lies in the concept of the central limit theorem, which is that as the sample size increases, the sample mean becomes a more accurate estimate of the population mean, and the variance decreases. When a single model is relied upon to predict the dependent variable, the prediction may exhibit some degree of bias, either being too high or too low. However, by employing multiple models, that each produces different predictions, the high predictions from some models are likely to counterbalance the low errors from others for any given observation. The net result is a tendency for the average of predictions to converge toward “the truth.” Generally, this leads to ensembles outperforming the best single model. Therefore, implementing a mechanism that automatically assesses a certain task, weighs the advantages and disadvantages of each possible model, and assigns the most optimal model stacking combination would be ideal.
Instantaneous Feedback Loops
Significantly automating the AI development and fine-tuning process, instantaneous feedback loops allow tools to improve faster, requiring less developer input. They work by comparing the outcome of processes to pre-set benchmarks, and then feeding the prompt back into the tool, until desired benchmarks are achieved. For example, ad-generating AI tools could have benchmark interaction rates, aiming to maximize interaction-per-ad.
Until now, AI development followed the human-in-the-loop model, where after each outcome, a human compared it to the benchmark and fine-tuned the tool. We have now moved towards a human-supervising-the-loop model, where the human continuously ups the benchmark, but allows the tool to finetune itself, by learning from its own more successful generations.
The desired goal is to achieve a human-out-of-the-loop model, where AI also sets its own benchmark. The initial benchmark would nevertheless be set by a human, but as it progresses, the AI would set iterative benchmarks once it saw that it was regularly hitting them.
Limitations
Our proposed production model faces several limitations. Firstly, the effectiveness of automatic model stacking relies heavily on the availability of suitable and unbiased models. This limitation may hinder the overall improvement in performance that we aim to achieve. Secondly, the evaluation of our model is complicated by the existence of instantaneous feedback loops, which necessitate constant and vigilant monitoring and validation processes to prevent the introduction of new biases or errors. Moreover, our model is susceptible to adversarial attacks, as malicious actors can exploit vulnerabilities in data augmentation, model stacking, and feedback loops to deceive AI detection systems. Lastly, ethical considerations surrounding the implementation of AI detection systems raise concerns related to privacy, surveillance, and freedom of expression. Striking a balance between combating misinformation and safeguarding individual rights and liberties is crucial.
References:
Baheti, P. (2022, January 31). Overfitting vs Underfitting: What’s the Difference? V7 Labs. https://www.v7labs.com/blog/overfitting-vs-underfitting
Chukhray, N.; Shakhovska, N.; Mrykhina, O.; Lisovska, L.; Izonin, I. (2022, 11 March) Stacking Machine Learning Model for the Assessment of R&D Product’s Readiness and Method for Its Cost Estimation. https://doi.org/10.3390/math10091466
Feinberg, M. (2021, May 24). The Problems with Turnitin. Highland Piper. https://highlandpiper-sc.com/1897/opinion/the-problems-with-turnitin/
Frąckiewicz, M. (2023, April 6). Data Augmentation and Bias Mitigation: Addressing Fairness and Ethics. TS2 SPACE. https://ts2.space/en/data-augmentation-and-bias-mitigation-addressing-fairness-and-ethics/
Kreps, S. (2020, June). The Role of Technology in Online Misinformation. Brookings Institution.
Lu, M.; Hou, Q.; Qin, S.; Zhou, L.; Hua, D.; Wang, X.; Cheng, L. A. (2023, 2 March) Stacking Ensemble Model of Various Machine Learning Models for Daily Runoff Forecasting.
Manyika, J., Silberg, J., & Presten, B. (2019, October 25). What Do We Do About the Biases in AI? Harvard Business Review. https://hbr.org/2019/10/what-do-we-do-about-the-biases-in-ai
Raitaluoto, T. (2023, August 29). Exploring the Accuracy of AI Detectors: A Comprehensive Analysis. AIContentfy. https://aicontentfy.com/en/blog/exploring-accuracy-of-ai-detectors-comprehensive-analysis
Soni, D. (2022, September 20). Feedback Loops in Machine Learning Systems. Medium; Towards Data Science. https://towardsdatascience.com/feedback-loops-in-machine-learning-systems-701296c91787
