Introduction
Imagine yourself a decade ago, jumping directly into the present shock of conversing naturally with an encyclopedic AI that crafts images, writes code, and debates philosophy. Won’t this technology almost certainly transform society — and hasn’t AI’s impact on us so far been a mixed-bag? Thus it’s no surprise that so many conversations these days circle around an era-defining question: How do we ensure AI benefits humanity? These conversations often devolve into strident optimism or pessimism about AI, and our earnest aim is to walk a pragmatic middle path, though no doubt we will not perfectly succeed.
While it’s fashionable to handwave towards “beneficial AI,” and many of us want to contribute towards its development — it’s not easy to pin down what beneficial AI concretely means in practice. This essay represents our attempt to demystify beneficial AI, through grounding it in the wellbeing of individuals and the health of society. In doing so, we hope to promote opportunities for AI research and products to benefit our flourishing, and along the way to share ways of thinking about AI’s coming impact that motivate our conclusions.
The Big Picture
By trade, we’re closer in background to AI than to the fields where human flourishing is most-discussed, such as wellbeing economics, positive psychology, or philosophy, and in our journey to find productive connections between such fields and the technical world of AI, we found ourselves often confused (what even is human flourishing, or wellbeing, anyways?) and from that confusion, often stuck (maybe there is nothing to be done? — the problem is too multifarious and diffuse). We imagine that others aiming to create prosocial technology might share our experience, and the hope here is to shine a partial path through the confusion to a place where there’s much interesting and useful work to be done. We start with some of our main conclusions, and then dive into more detail in what follows.
One conclusion we came to is that it’s okay that we can’t conclusively define human wellbeing. It’s been debated by philosophers, economists, psychotherapists, psychologists, and religious thinkers, for many years, and there’s no consensus. At the same time, there’s agreement around many concrete factors that make our lives go well, like: supportive intimate relationships, meaningful and engaging work, a sense of growth and achievement, and positive emotional experiences. And there’s clear understanding, too, that beyond momentary wellbeing, we must consider how to secure and improve wellbeing across years and decades — through what we could call societal infrastructure: important institutions such as education, government, the market, and academia.
One benefit of this wellbeing lens is to wake us to an almost-paradoxical fact: While the deep purpose behind nearly everything our species does is wellbeing, we’ve tragically lost sight of it. Both by common measures of individual wellbeing (suicide rate, loneliness, meaningful work) and societal wellbeing (trust in our institutions, shared sense of reality, political divisiveness), we’re not doing well, and our impression is that AI is complicit in that decline. The central benefit of this wellbeing view, however, is the insight that no fundamental obstacle prevents us from synthesizing the science of wellbeing with machine learning to our collective benefit.
This leads to our second conclusion: We need plausible positive visions of a society with capable AI, grounded in wellbeing. Like other previous transformative technologies, AI will shock our societal infrastructure — dramatically altering the character of our daily lives, whether we want it to or not. For example, Facebook launched only twenty years ago, and yet social media’s shockwaves have already upended much in society — subverting news media and our informational commons, addicting us to likes, and displacing meaningful human connection with its shell. We believe capable AI’s impact will exceed that of social media. As a result, it’s vital that we strive to explore, envision, and move towards the AI-infused worlds we’d flourish within — ones perhaps in which it revitalizes our institutions, empowers us to pursue what we find most meaningful, and helps us cultivate our relationships. This is no simple task, requiring imagination, groundedness, and technical plausibility — to somehow dance through the minefields illuminated by previous critiques of technology. Yet now is the time to dream and build if we want to actively shape what is to come.
This segues into our final conclusion: Foundation models and the arc of their future deployment is critical. Even for those of us in the thick of the field, it’s hard to internalize how quickly models have improved, and how capable they might become given several more years. Recall that GPT-2 — barely functional by today’s standards — was released only in 2019. If future models are much more capable than today’s, and competently engage with more of the world with greater autonomy, we can expect their entanglement with our lives and society to rachet skywards. So, at minimum, we’d like to enable these models to understand our wellbeing and how to support it, potentially through new algorithms, wellbeing-based evaluations of models and wellbeing training data. Of course, we also want to realize human benefit in practice — the last section of this blog post highlights what we believe are strong leverage points towards that end.
The rest of this post describes in more detail (1) what we mean by AI that benefits our wellbeing, (2) the need for positive visions for AI grounded in wellbeing, and (3) concrete leverage points to aid in the development and deployment of AI in service of such positive visions. We’ve designed this essay such that the individual parts are mostly independent, so if you are interested most in concrete research directions, feel free to skip there.
Beneficial AI grounds out in human wellbeing
Discussion about AI for human benefit is often high-minded, but not particularly actionable, as in unarguable but content-free phrases like “We should make sure AI is in service of humanity.” But to meaningfully implement such ideas in AI or policy requires enough precision and clarity to translate them into code or law. So we set out to survey what science has discovered about the ground of human benefit, as a step towards being able to measure and support it through AI.
Often, when we think about beneficial impact, we focus on abstract pillars like democracy, education, fairness, or the economy. However important, none of these are valuable intrinsically. We care about them because of how they affect our collective lived experience, over the short and long-term. We care about increasing society’s GDP to the extent it aligns with actual improvement of our lives and future, but when treated as an end in itself, it becomes disconnected from what matters: improving human (and potentially all species’) experience.
In looking for fields that most directly study the root of human flourishing, we found the scientific literature on wellbeing. The literature is vast, spanning many disciplines, each with their own abstractions and theories — and, as you might expect, there’s no true consensus on what wellbeing actually is. In diving into the philosophy of flourishing, wellbeing economics, or psychological theories of human wellbeing, one encounters many interesting, compelling, but seemingly incompatible ideas.
For example, theories of hedonism in philosophy claim that pleasure and the absence of suffering is the core of wellbeing; while desire satisfaction theories instead claim that wellbeing is about the fulfillment of our desires, no matter how we feel emotionally. There’s a wealth of literature on measuring subjective wellbeing (broadly, how we experience and feel about our life), and many different frameworks of what variables characterize flourishing. For example, Martin Seligman’s PERMA framework claims that wellbeing consists of positive emotions, engagement, relationships, meaning, and achievement. There are theories that say that the core of wellbeing is satisfying psychological needs, like the need for autonomy, competence, and relatedness. Other theories claim that wellbeing comes from living by our values. In economics, frameworks rhyme with those in philosophy and psychology, but diverge enough to complicate an exact bridge. For example, the wellbeing economics movement largely focuses on subjective wellbeing and explores many different proxies of it, like income, quality of relationships, job stability, etc.
After the excitement from surveying so many interesting ideas began to fade, perhaps unsurprisingly, we remained fundamentally confused about what “the right theory” was. But, we recognized that in fact this has always been the human situation when it comes to wellbeing, and just as a lack of an incontrovertible theory of flourishing has not prevented humanity from flourishing in the past, it need not stand as a fundamental obstacle for beneficial AI. In other words, our attempts to guide AI to support human flourishing must take this lack of certainty seriously, just as all sophisticated societal efforts to support flourishing must do.
In the end, we came to a simple workable understanding, not far from the view of wellbeing economics: Human benefit ultimately must ground out in the lived experience of humans. We want to live happy, meaningful, healthy, full lives — and it’s not so difficult to imagine ways AI might assist in that aim. For example, the development of low-cost but proficient AI coaches, intelligent journals that help us to self-reflect, or apps that help us to find friends, romantic partners, or to connect with loved ones. We can ground these efforts in imperfect but workable measures of wellbeing from the literature (e.g. PERMA), taking as first-class concerns that the map (wellbeing measurement) is not the territory (actual wellbeing), and that humanity itself continues to explore and refine its vision of wellbeing.
More broadly our wellbeing relies on a healthy society, and we care not only about our own lives, but also want beautiful lives for our neighbors, community, country, and world, and for our children, and their children as well. The infrastructure of society (institutions like government, art, science, military, education, news, and markets) is what supports this broader, longer-term vision of wellbeing.
Each of these institutions have important roles to play in society, and we can also imagine ways that AI could support or improve them; for example, generative AI may catalyze education through personal tutors that help us develop a richer worldview, may help us to better hold our politicians to account through sifting through what they are actually up to, or accelerate meaningful science through helping researchers make novel connections. Thus in short, beneficial AI would meaningfully support our quest for lives worth living, in both the immediate and long-term sense.
So, from the lofty confusion of conflicting grand theories, we arrive at something sounding more like common sense. Let’s not take this for granted, however — it cuts through the cruft of abstractions to firmly recenter what is ultimately important: the psychological experience of humans. This view points us towards the ingredients of wellbeing that are both well-supported scientifically and could be made measurable and actionable through AI (e.g. there exist instruments to measure many of these ingredients). Further, wellbeing across the short and long-term provides the common currency that bridges divergent approaches to beneficial AI, whether mitigating societal harms like discrimination in the AI ethics community, to attempting to reinvigorate democracy through AI-driven deliberation, to creating a world where humans live more meaningful lives, to creating low-cost emotional support and self-growth tools, to reducing the likelihood of existential risks from AI, to using AI to reinvigorate our institutions — wellbeing is the ultimate ground.
Finally, focusing on wellbeing helps to highlight where we currently fall short. Current AI development is driven by our existing incentive systems: Profit, research novelty, engagement, with little explicit focus on what fundamentally is more important (human flourishing). We need to find tractable ways to shift incentives towards wellbeing-supportive models (something we’ll discuss later), and positive directions to move toward (discussed next).
We need positive visions for AI
Technology is a shockingly powerful societal force. While nearly all new technologies bring only limited change, like an improved toothbrush, sometimes they upend the world. Like the proverbial slowly-boiling frog, we forget how in short order the internet and cellphones have overhauled our lived experience: the rise of dating apps, podcasts, social networks, our constant messaging, cross-continental video calls, massive online games, the rise of influencers, on-demand limitless entertainment, etc. Our lives as a whole — our relationships, our leisure, how we work and collaborate, how news and politics work — have dramatically shifted, for both the better and worse.
AI is transformative, and the mixed bag of its impacts are poised to reshape society in mundane and profound ways; we might doubt it, but that was also our naivety at the advent of social media and the cell-phone. We don’t see it coming, and once it’s here we take it for granted. Generative AI translates applications from science fiction into rapid adoption: AI romantic companions; automated writing and coding assistants; automatic generation of high-quality images, music, and videos; low-cost personalized AI tutors; highly-persuasive personalized ads; and so on.
In this way, transformative impact is happening now — it does not require AI with superhuman intelligence — see the rise of LLM-based social media bots; ChatGPT as the fastest-adopted consumer app; LLMs requiring fundamental changes to homework in school. Much greater impact will yet come, as the technology (and the business around it) matures, and as AI is integrated more pervasively throughout society.
Our institutions were understandably not designed with this latest wave of AI in mind, and it’s unclear that many of them will adapt quickly enough to keep up with AI's rapid deployment. For example, an important function of news is to keep a democracy’s citizens well-informed, so their vote is meaningful. But news these days spreads through AI-driven algorithms on social media, which amplifies emotional virality and confirmation bias at the expense of meaningful debate. And so, the public square and the sense of a shared reality is being undercut, as AI degrades an important institution devised without foresight of this novel technological development.
Thus in practice, it may not be possible to play defense by simply “mitigating harms” from a technology; often, a new technology demands that we creatively and skillfully apply our existing values to a radically new situation. We don’t want AI to, for example, undermine the livelihood of artists, yet how do we want our relationship to creativity to look like in a world where AI can, easily and cheaply, produce compelling art or write symphonies and novels, in the style of your favorite artist? There’s no easy answer. We need to debate, understand, and capture what we believe is the spirit of our institutions and systems given this new technology.
For example, what’s truly important about education? We can reduce harms that AI imposes on the current education paradigm by banning use of AI in students’ essays, or apply AI in service of existing metrics (for example, to increase high school graduation rates). But the paradigm itself must adapt: The world that schooling currently prepares our children for is not the world they’ll graduate into, nor does it prepare us generally to flourish and find meaning in our lives. We must ask ourselves what we really value in education that we want AI to enable: Perhaps teaching critical thinking, enabling agency, and creating a sense of social belonging and civic responsibility?
To anticipate critique, we agree that there will be no global consensus on what education is for, or on the underlying essence of any particular institution, at root because different communities and societies have distinct values and visions. But that’s okay: Let’s empower communities to fit AI systems to local societal contexts; for example, algorithms like constitutional AI enable creating different constitutions that embody flourishing for different communities. This kind of cheap flexibility is an exciting affordance, meaning we no longer must sacrifice nuance and context-sensitivity for scalability and efficiency, a bitter pill technology often pushes us to swallow.
And while of course we have always wanted education to create critical thinkers, our past metrics (like standardized tests) have been so coarse that scoring high is easily gamed without critical thinking. But generative AI enables new affordances here, too: just as a teacher can socratically question a student to evaluate their independent thought, advances in generative AI open up the door for similarly qualitative and interactive measures, like personalized AI tutors that meaningfully gauge critical thinking.
We hope to tow a delicate line beyond broken dichotomies, whether between naive optimism and pessimism, or idealism and cynicism. Change is coming, and we must channel it towards refined visions of what we want, which is a profound opportunity, rather than to assume that by default technology will deliver us (or doom us), or that we will be able to wholly resist the transformation it brings (or are entirely helpless against it). For example, we must temper naive optimism (“AI will save the world if only we deploy it everywhere!”) by integrating lessons from the long line of work that studies the social drivers and consequences of technology, often from a critical angle. But neither should cynical concerns so paralyze us that we remain only as critics on the sidelines.
So, what can we do?
The case so far is that we need positive visions for society with capable AI, grounded in individual and societal wellbeing. But what concrete work can actually support this? We propose the following break-down:
- Understanding where we want to go
- Measuring how AI impacts our wellbeing
- Training models that can support wellbeing
- Deploying models in service of wellbeing
The overall idea is to support an ongoing, iterative process of exploring the positive directions we want to go and deploying and adapting models in service of them.
We need to understand where we want to go in the age of AI
This point follows closely from the need to explore the positive futures we want with AI. What directions of work and research can help us to clarify where is possible to go, and is worth going to, in the age of AI?
For starters, it’s more important now than ever to have productive and grounded discussions about questions like: What makes us human? How do we want to live? What do we want the future to feel like? What values are important to us? What do we want to retain as AI transformations sweep through society? Rather than being centered on the machine learning community, this should be an interdisciplinary, international effort, spanning psychology, philosophy, political science, art, economics, sociology, and neuroscience (and many other fields!), and bridging diverse intra- and international cultures.
Of course, it’s easy to call for such a dialogue, but the real question is how such interdisciplinary discussions can be convened in a meaningful, grounded, and action-guiding way — rather than leading only to cross-field squabbles or agreeable but vacuous aspirations. Perhaps through participatory design that pairs citizens with disciplinary experts to explore these questions, with machine learning experts mainly serving to ground technological plausibility. Perhaps AI itself could be of service: For example, research in AI-driven deliberative democracy and plurality may help involve more people in navigating these questions; as might research into meaning alignment, by helping us describe and aggregate what is meaningful and worth preserving to us. It’s important here to look beyond cynicism or idealism (suggestive of meta-modern political philosophy): Yes, mapping exciting positive futures is not a cure-all, as there are powerful societal forces, like regulatory capture, institutional momentum, and the profit motive, that resist their realization, and yet, societal movements all have to start somewhere, and some really do succeed.
Beyond visions for big-picture questions about the future, much work is needed to understand where we want to go in narrower contexts. For example, while it might at first seem trivial, how can we reimagine online dating with capable AI, given that healthy romantic partnership is such an important individual and societal good? Almost certainly, we will look back at swipe-based apps as misguided means for finding long-term partners. And many of our institutions, small and large, can be re-visioned in this way, from tutoring to academic journals to local newspapers. AI will make possible a much richer set of design possibilities, and we can work to identify which of those possibilities are workable and well-represent the desired essence of an institution’s role in our lives and society.
Finally, continued basic and applied research into the factors that contribute and characterize human wellbeing and societal health also are highly important, as these are what ultimately ground our visions. And as the next section explores, having better measures of such factors can help us to change incentives and work towards our desired futures.
We need to develop measures for how AI affects wellbeing
For better and worse, we often navigate through what we measure. We’ve seen this play out before: Measure GDP, and nations orient towards increasing it at great expense. Measure clicks and engagement, and we develop platforms that are terrifyingly adept at keeping people hooked. A natural question is, what prevents us from similarly measuring aspects of wellbeing to guide our development and deployment of AI? And if we do develop wellbeing measures, can we avoid the pitfalls that have derailed other well-intended measures, like GDP or engagement?
One central problem for measurement is that wellbeing is more complex and qualitative than GDP or engagement. Time-on-site is a very straightforwardmeasure of engagement. In contrast, properties relevant to wellbeing, like the felt sense of meaning or the quality of healthy relationships, are difficult to pin down quantitatively, especially from the limited viewpoint of how a user interacts with a particular app.
Wellbeing depends on the broader context of a user’s life in messy ways, meaning it’s harder to isolate how any small intervention impacts it. And so, wellbeing measures are more expensive and less standardized to apply, end up less measured, and less guide our development of technology. However, foundation models are beginning to have the exciting ability to work with qualitative aspects of wellbeing. For example, present-day language models can (with caveats) infer emotions from user messages and detect conflict; or conduct qualitative interviews with users about its impact on their experience.
So one promising direction of research, though not easy, is to explore how foundation models themselves can be applied to more reliably measure facets of individual and societal wellbeing, and ideally, help to identify how AI products and services are impacting that wellbeing. The mechanisms of impact are two-fold: One, companies may currently lack means of measuring wellbeing even though all-things-equal they want their products to help humans; two, where the profit motive conflicts with encouraging wellbeing, if a product’s impact can be externally audited and published, it can help hold the company to account by consumers and regulators, shifting corporate incentives towards societal good.
Another powerful way that wellbeing-related measures can have impact is as evaluation benchmarks for foundation models. In machine learning, evaluations are a powerful lever for channeling research effort through competitive pressure. For example, model providers and academics continuously develop new models that perform better and better on benchmarks like TruthfulQA. Once you have legible outcomes, you often spur innovation to improve upon them. We currently have very few benchmarks focused on how AI affects our wellbeing, or how well they can understand our emotions, make wise decisions, or respect our autonomy: We need to develop these benchmarks.
Finally, as mentioned briefly above, metrics can also create accountability and enable regulations. Recent efforts like the Stanford Foundational Model Transparency Index have created public accountability for AI labs, and initiatives like Responsible Scaling Policies are premised on evaluations of model capabilities, as are evaluations by government bodies such as AI safety institutes in both the UK and US. Are there similar metrics and initiatives to encourage accountability around AI’s impact on wellbeing?
To anticipate a natural concern, unanticipated side-effects are nearly universal when attempting to improve important qualities through quantitative measures. What if in measuring wellbeing, the second-order consequence is perversely to undermine it? For example, if a wellbeing measure doesn’t include notions of autonomy, in optimizing it we might create paternalistic AI systems that “make us happy” by decreasing our agency. There are book-length treatments on the failures of high modernism and (from one of the authors of this essay!) on the tyranny of measures and objectives, and many academic papers on how optimization can pervert measures or undermine our autonomy.
The trick is to look beyond binaries. Yes, measures and evaluations have serious problems, yet we can work with them with wisdom, taking seriously previous failures and institutionalizing that all measures are imperfect. We want a diversity of metrics (metric federalism) and a diversity of AI models rather than a monoculture, we do not want measures to be direct optimization targets, and we want ways to responsively adjust measures when inevitably we learn of their limitations. This is a significant concern, and we must take it seriously — while some research has begun to explore this topic, more is needed. Yet in the spirit of pragmatic harm reduction, given that metrics are both technically and politically important for steering AI systems, developing less flawed measures remains an important goal.
Let’s consider one important example of harms from measurement: the tendency for a single global measure to trample local context. Training data for models, including internet data in particular, is heavily biased. Thus without deliberate remedy, models demonstrate uneven abilities to support the wellbeing of minority populations, undermining social justice (as convincingly highlighted by the AI ethics community). While LLMs have exciting potential to respect cultural nuance and norms, informed by the background of the user, we must work deliberately to realize it. One important direction is to develop measures of wellbeing specific to diverse cultural contexts, to drive accountability and reward progress.
To tie these ideas about measurement together, we suggest a taxonomy, looking at measures of AI capabilities, behaviors, usage, and impacts. Similar to this DeepMind paper, the idea is to examine spheres of expanding context, from testing a model in isolation (both what it is capable of and what behaviors it demonstrates), all the way to understanding what happens when a model meets the real world (how humans use it, and what its impact is on them and society).
The idea is that we need a complementary ecosystem of measures fit to different stages of model development and deployment. In more detail:
- AI capabilities refers to what models are able to do. For example, systems today are capable of generating novel content, and translating accurately between languages.
- AI behaviors refers to how an AI system responds to different concrete situations. For example, many models are trained to refuse to answer questions that enable dangerous activities, like how to build a bomb,even though they have the capability to correctly answer them).
- AI usage refers to how models are used in practice when deployed. For example, AI systems today are used in chat interfaces to help answer questions, as coding assistants in IDEs, to sort social media feeds, and as personal companions.
- AI impacts refers to how AI impacts our experience or society. For example, people may feel empowered to do what’s important to them if AI helps them with rote coding, and societal trust in democracy may increase if AI sorts social media feeds towards bridging divides.
As an example of applying this framework to an important quality that contributes to wellbeing, here is a sketch of how we might design measures of human autonomy:
Let’s work through this example: we take a quality with strong scientific links to wellbeing, autonomy, and create measures of it and what enables it, all along the pipeline from model development to when it’s deployed at scale.
Starting from the right side of the table (Impact), there exist validated psychological surveys that measure autonomy, which can be adapted and given to users of an AI app to measure its impact on their autonomy. Then, moving leftwards, these changes in autonomy could be linked to more specific types of usage, through additional survey questions. For example, perhaps automating tasks that users actually find meaningful may correlate with decreased autonomy.
Moving further left on the table, the behaviors of models that are needed to enable beneficial usage and impact can be gauged through more focused benchmarks. To measure behaviors of an AI system, one could run fixed workflows on an AI application where gold-standard answers come from expert labelers; another approach is to simulate users (e.g. with language models) interacting with an AI application to see how often and skillfully it performs particular behaviors, like socratic dialogue.
Finally, capabilities of a particular AI model could be similarly measured through benchmark queries input directly to the model, in a way very similar to how LLMs are benchmarked for capabilities like reasoning or question-answering. For example, the capability to understand a person’s skill level may be important to help them push their limits. A dataset could be collected of user behaviors in some application, annotated with their skill level; and the evaluation would be how well the model could predict skill level from observed behavior.
At each stage, the hope is to link what is measured through evidence and reasoning to what lies above and below it in the stack. And we would want a diversity of measures at each level, reflecting different hypotheses about how to achieve the top-level quality, and with the understanding that each measure is always imperfect and subject to revision. In a similar spirit, rather than some final answer, this taxonomy and example autonomy measures are intended to inspire much-needed pioneering work towards wellbeing measurement.
We need to train models to improve their ability to support wellbeing
Foundation models are becoming increasingly capable and in the future we believe most applications will not train models from scratch. Instead, most applications will prompt cutting-edge proprietary models, or fine-tune such models through limited APIs, or train small models on domain-specific responses from the largest models for cost-efficiency reasons. As evidence, note that to accomplish tasks with GPT-3 often required chaining together many highly-tuned prompts, whereas with GPT-4 those same tasks often succeed with the first casual prompting attempt. Additionally, we are seeing the rise of capable smaller models specialized for particular tasks, trained through data from large models.
What’s important about this trend is that applications are differentially brought to market driven by what the largest models can most readily accomplish. For example, if frontier models excel at viral persuasion from being trained on Twitter data, but struggle with the depths of positive psychology, it will be easier to create persuasive apps than supportive ones, and there will be more of them, sooner, on the market.
Thus we believe it’s crucial that the most capable foundation models themselves understand what contributes to our wellbeing — an understanding granted to them through their training process. We want the AI applications that we interface with (whether therapists, tutors, social media apps, or coding assistants) to understand how to support our wellbeing within their relevant role.
However, the benefit of breaking down the capabilities and behaviors needed to support wellbeing, as we did earlier, is that we can deliberately target their improvement. One central lever is to gather or generate training data, which is the general fuel underlying model capabilities. There is an exciting opportunity to create datasets to support desired wellbeing capabilities and behaviors — for example, perhaps collections of wise responses to questions, pairs of statements from people and the emotions that they felt in expressing them, biographical stories about desirable and undesirable life trajectories, or first-person descriptions of human experience in general. The effect of these datasets can be grounded in the measures discussed above.
To better ground our thinking, we can examine how wellbeing data could improve the common phases of foundation model training: pretraining, fine-tuning, and alignment.
Pretraining
The first training phase (confusingly called pretraining) establishes a model’s base abilities. It does so by training on vast amounts of variable-quality data, like a scrape of the internet. One contribution could be to either generate or gather large swaths of wellbeing relevant data, or to prioritize such data during training (also known as altering the data mix). For example, data could be sourced from subreddits relevant to mental health or life decisions, collections of biographies, books about psychology, or transcripts of supportive conversations. Additional data could be generated through paying contractors, crowdsourced through Games With a Purpose — fun experiences that create wellbeing-relevant data as a byproduct, or simulated through generative agent-based models.
Fine-tuning
The next stage of model training is fine-tuning. Here, smaller amounts of high-quality data, like diverse examples of desired behavior gathered from experts, focus the general capabilities resulting from pretraining. For different wellbeing-supporting behaviors we might want from a model, we can create fine-tuning datasets through deliberate curation of larger datasets, or by enlisting and recording the behavior of human experts in the relevant domain. We hope that the companies training the largest models place more emphasis on wellbeing in this phase of training, which is often driven by tasks with more obvious economic implications, like coding.
Alignment
The final stage of model training is alignment, often achieved through techniques like reinforcement learning through human feedback (RLHF), where human contractors give feedback on AI responses to guide the model towards better ones. Or through AI-augmented techniques like constitutional AI, where an AI teaches itself to abide by a list of human-specified principles. The fuel of RLHF is preference data about what responses are preferred over others. Therefore we imagine opportunities for creating data sets of expert preferences that relate to wellbeing behaviors (even though what constitutes expertise in wellbeing may be interestingly contentious). For constitutional AI, we may need to iterate in practice with lists of wellbeing principles that we want to support, like human autonomy, and how, specifically, a model can respect it across different contexts.
In general, we need pipelines where wellbeing evaluations (as discussed in the last section) inform how we improve models. We need to find extensions to paradigms like RLHF that go beyond which response humans prefer in the moment, considering also which responses support user long-term growth, wellbeing, and autonomy, or better embody the spirit of the institutional role that the model is currently playing. These are intriguing, subtle, and challenging research questions that strike at the heart of the intersection of machine learning and societal wellbeing, and deserve much more attention.
For example, we care about wellbeing over spans of years or decades, but it is impractical to apply RLHF directly on human feedback to such ends, as we cannot wait decades to gather human feedback for a model; instead, we need research that helps integrate validated short-term proxies for long-term wellbeing (e.g. quality of intimate relationships, time spent in flow, etc.), ways to learn from longitudinal data where it exists (perhaps web journals, autobiographies, scientific studies), and to collect the judgment of those who devote their lifetime to helping support individuals flourish (like counselors or therapists).
We need to deploy AI models in a way that supports wellbeing
Ultimately we want AI models deployed in the world to benefit us. AI applications could directly target human wellbeing, for example by directly supporting mental health or coaching us in a rigorous way. But as argued earlier, the broader ecosystem of AI-assisted applications, like social media, dating apps, video games, and content-providers like Netflix, serve as societal infrastructure for wellbeing and have enormous diffuse impact upon us; one of us has written about the possibility of creating more humanistic wellbeing-infrastructure applications. While difficult, dramatic societal benefits could result from, for example, new social media networks that better align with short and long-term wellbeing.
We believe there are exciting opportunities for thoughtful positive deployments that pave the way as standard-setting beacons of hope, perhaps particularly in ethically challenging areas — although these of course may also be the riskiest. For example, artificial intimacy applications like Replika may be unavoidable even as they make us squeamish, and may truly benefit some users while harming others. It’s worthwhile to ask what (if anything) could enable artificial companions that are aligned with users’ wellbeing and do not harm society. Perhaps it is possible to thread the needle: they could help us develop the social skills needed to find real-world companions, or at least have strong, transparent guarantees about their fiduciary relationship to us, all while remaining viable as a business or non-profit. Or perhaps we can create harm-reduction services that help people unaddict from artificial companions that have become obstacles to their growth and development. Similar thoughts may apply to AI therapists, AI-assisted dating apps, and attention-economy apps, where incentives are difficult to align.
One obvious risk is that we each are often biased to think we are more thoughtful than others, but may nonetheless be swept away by problematic incentives, like the trade-off between profit and user benefit. Legal structures like public benefit corporations, non-profits, or innovative new structures may help minimize this risk, as may value-driven investors or exceedingly careful design of internal culture.
Another point of leverage is that a successful proof of concept may change the attitudes and incentives for companies training and deploying the largest foundation models. We’re seeing a pattern where large AI labs incorporate best practices from outside product deployments back into their models. For example, ChatGPT plugins like data analysis and the GPT market were explored first by companies outside OpenAI before being incorporated into their ecosystem. And RLHF, which was first integrated into language models by OpenAI, is now a mainstay across foundation model development.
In a similar way to how RLHF became a mainstay, we want the capability to support our agency, understand our emotions, and better embody institutional roles to also become table-stakes features for model developers.This could happen through research advances outside of the big companies, making it much easier for such features to be adopted within them — though adoption may require pressure, through regulation, advocacy, or competition.
Initiatives
We believe there’s much concrete work to be done in the present. Here are a sampling of initiatives to seed thinking about what could move the field forward:
Conclusion: A call to action
AI will transform society in ways that we cannot yet predict. If we continue on the present track, we risk AI reshaping our interactions and institutions in ways that erode our wellbeing and what makes our lives meaningful. Instead, challenging as it may be, we need to develop AI systems that understand and support wellbeing, both individual and societal. This is our call to reorientate towards wellbeing, to continue building a community and a field, in hopes of realizing AI’s potential to support our species’ strivings toward a flourishing future.
