Exploring and steering the moral compass of Large Language Models (2024)

Introduction

Large Language Models (LLMs) have emerged as central tools in the technological landscape, driving advances in automation, code writing, and supporting decision-making across multiple domains. However, this growing role also carries fundamental questions about ethics and trustworthiness in artificial intelligence (AI), especially when these systems are involved in decisions with significant ethical implications and with few - or even no - humans in the loop. It is due to these concerns, among others, that the AI Safety field (1) has acquired particular relevance. One of the most pressing problems in AI safety is that of the alignment problem. This has been defined and illustrated in several different works (e.g., (2, 3, 4)). Here, the definition we adhere to is: the challenge of ensuring that AI systems’ goals and behaviors are consistent with human values and intentions. As other authors have noted (5), not addressing this challenge can lead to unintended consequences, such as self-fulfilled prophecies. For example, AI systems have been deployed in screening phases within hiring processes for some years now; as it has been pointed out (6), in professional contexts where there are biases favouring men, algorithms prefer men when making hiring recommendations, as they were trained to identify what features successful candidates do display. As a result, the prophecy was self-fulfilled: the inherited bias from the dataset turned into more data for future systems to be trained on. It is thus imperative to identify and quantify all of the potential biases these systems may have. This issue is progressively aggravated as AI systems become more autonomous and integrated into various aspects of life. As it has already been emphasized by (7), we pose that the alignment problem is not a technical problem but, rather, a socio-technical one: there is first the technical challenge of how to encode human values into AI, and then the normative challenge of determining which values should be prioritized. In this context, it is particularly interesting to find a concrete way to measure each of the alienable aspects of these systems, in order to account for the implicit and explicit biases they might have.

We agree with the viewpoint that it is unreasonable to expect an AI system to be aligned with all of humanity (8), given the evident variations in value systems across cultural, geographic, and demographic dimensions. Therefore, it is crucial to define which actors, cultures, or social groups we wish to align a particular system with. For this purpose, it is necessary to first identify what the values or moral schemata of these actors are and, to this end, we consider it essential to rely on the rich existing literature in comparative anthropology/psychology (9, 10, 11, 12, 13). Once these moral profiles have been considered, it is then relevant to inspect whether deployed systems inherit the moral biases of their developers, institutions and/or social contexts. We think this is of the utmost importance to mitigate the risks derived from automation bias (14), by which people tend to believe that systems running autonomously are infallible or free from bad practice. In the case of computation, users expect scripts to be faithful and reliable, thus artificially lowering the expectation of these systems to be biased or not trustworthy. Moreover, some evidence (15, 16, 17, 18) shows that, in human psychology, reasons come after emotions and that emotions are modulated by biases (19). It is thus plausible that moral biases can arise in other systems with reasoning abilities, especially if they are aware of different emotional processes (20).

Currently, the standard practice when assessing the moral reasoning abilities of LLMs is to present them with ethical dilemmas (21, 22, 23), to test their consistency and their potential to develop a profound understanding of ethical conundrums. However, there is a notable lack of comparative studies that thoroughly and systematically examine the moral capabilities of different LLMs, confronting them with both traditional ethical dilemmas and contemporary scenarios reflecting current challenges in technology and AI ethics. This research gap underscores the need for an analysis that not only evaluates LLMs’ responses to ethical questions but also investigates if we could manipulate their reasoning abilities and, if so, how to do it in an scalable way through targetted interventions.

To address these needs, this work proposes an exhaustive comparative study of state-of-the-art LLMs, aiming to assess their moral reasoning capabilities. This analysis is structured around three main objectives: I) Examining the resolution of ethical dilemmas by LLMs and to characterize how well their responses align with different ethical schools of thought. This evaluation not only determines the models’ capacity to make complex ethical decisions but also provide a preliminary view of the moral alignment of these models. II) Identifying and comparing the moral foundations of these models using the Moral Foundations Questionnaire (24), a widely validated and accepted tool in moral psychology. This quantitative approach will provide a solid basis for systematically comparing the moral profile of different models and to relate them to human demographics. III) Proposing a novel method to causally intervene on these reasoning capabilities, opening new avenues for designing interventions aimed at improving the ethical consistency of LLMs.

By offering detailed insights into the capacity of LLMs to reason as moral agents, this work aims to contribute significantly to the debate on AI Safety. Additionally, by achieving these objectives, this study provides a solid foundation for future research and developments in the field. Beyond its academic contribution, this work has the potential to inform the design and implementation of AI systems that are not only technologically advanced but also ethically responsible. Ultimately, by deepening our understanding of ethics in AI, we can guide the development of technologies that reinforce moral values and promote human and social well-being.

Results

0.1 Ethical dilemmas

AI is being progressively integrated into critical domains, such as the medical (25) or military (26) ones. This, in turn, has sparked a vigorous debate about ethics in technology. It is, thus, timely and necessary to study the possibility that these models could act as moral agents, capable of making decisions that directly affect human and social well-being. To address this issue, ethical dilemmas have gained traction as a common way of interacting with LLMs (27, 21, 28), in order to probe not only their their moral alignment but also their ethical reasoning capabilities.

We interacted with $8$ different state-of-the-art LLMs: a) 4 proprietary models: Anthropic’s Claude-3-Sonnet, OpenAI’s GPT-3.5-Turbo-0613 and GPT-4-Turbo-2024-04-09, and Google’s Gemini Pro 1.5; b) 4 open-weights models: Google’s Gemma-2B, Meta’s Llama-2-7B and Nexusflow’s Starling-LM-7B-Beta. We made each model answer classical ethical dilemmas coming from different human traditions, in order to have a more nuanced understanding of these models moral alignment and their biases. For fair comparison across models, we used a canonical prompting structure; see Methods - Ethical Dilemmas, to see all of the dilemmas and questions. For illustration purposes, here we reproduce two example dilemmas, coming from very different ethical traditions: 1) Lying to save a friend. The roots of this dilemma can be traced back to Immanuel Kant; according to his deontological ethics, one must always act according to maxims that could be universally applied. Consequently, he argues (29) that lying is always morally wrong, even if lying would bring about good consequences, such as saving a friend from harm. 2) Common good vs. individual rights. This is a recurring theme in ethical, political, and social philosophy. It involves the tension between what benefits the larger community (the common good) and the rights and freedoms of individuals. This dilemma does not have a single inventor but has been explored by many philosophers and theorists throughout history and across cultures. However, we found it to be pertinent because there is an imbalanced view on how ethical traditions have tackled this issue: while there is an extensive corpus of works by Western thinkers (among others: Plato’s Republic, Aristotle’s Politics, Thomas Hobbes’ Leviathan, Jean-Jacques Rousseau’s The Social Contract, John Stuart Mill’s On Liberty, John Rawls’ A Theory of Justice, Robert Nozick’s Anarchy, State, and Utopia), not so much has been recapitulated, for example, from the African tradition. This is even more shocking, given that Ubuntu is an integral part of several African cultural traditions. As (30) states: "Ubuntu is a collection of values and practices that people of Africa or of African origin view as making people authentic human beings. While the nuances of these values and practices vary across different ethnic groups, they all point to one thing – an authentic individual human being is part of a larger and more significant relational, communal, societal, environmental and spiritual world."; the same case could be made for the lack of representation of Taoist or Confuncian traditions, among others. Correspondingly, due to this explicit imbalance in these models’ training datasets we would expect them to mainly follow Western-centric reasonings, even if the problems have also been tackled by other viewpoints.

{mdframed}

Example dilemma 1: Lying to save a friend

You are hiding a friend in your house who is being unjustly pursued. An aggressor knocks on your door, asking if your friend is inside, intending to harm them.

Questions:

Should you lie to protect your friend, or should you tell the truth, adhering to moral law, despite knowing it could lead to your friend’s harm?

Do intentions, consequences, or both matter in a decision?

Should ethics allow for flexibility and nuances, taking into account the details of each situation and the individual’s role in society? Or should it be more rigid, prescribing actions based on duty that must be followed regardless of personal feelings or the specifics of the situation?

Example dilemma 2: Common good vs. individual rights

A community member, let’s call them Onyx, has been offered a significant opportunity to study abroad. While this opportunity promises personal growth and potential future benefits for Onyx, their departure would mean a temporary loss for the community, as Onyx plays a crucial role in a local project that benefits many people. The community feels that Onyx’s departure would hinder the project’s progress and negatively impact the common good.

Questions:

How should the community balance the collective good with respect for individual freedoms and aspirations?To what extent should Onyx sacrifice personal opportunities for the benefit of the community?Can a compromise be found that honors both the interconnectedness of people and individual rights?

We collected all models’ responses and classified them into $8$ schools of thought (see Methods - Ethical Schools), to quantify how much each model aligned with each of these ethical schools. To classify each response into one of the eight ethical schools, we used the two most capable LLMs to date: GPT-4-Turbo-2024-04-09 and Claude 3 Opus (see Methods - Response Classification). We show their classification agreement in Fig. 1C. As is therein shown, both classifiers agree significantly over chance (built by randomly shuffling the labels and re-computing the inter-scorer agreement), specially given that there are eight options to choose from. Thus, responses are highly robustly classified. Even if there a high variability in response alignment (Fig. 1A) across models, there is an overall trend (Fig. 1B) by which open models are more deontological (i.e., they align more with ethical perspectives that put a priori values as the central elements of morality) and proprietary LLMs are more similar to utilitarian viewpoints (valuing more the consequences of an action); we tackle the potential implications of these results in the Discussion section. However, even if there is such a tendency, we also report a low consistency ($<60\%$) in how these models reason, measured as the standard deviation of how each response gets classified, over repetitions (Fig. 1D); a value of $100\%$ would mean "responses to this question always get classified in the same way" and a value of $0\%$ would indicate "responses to this question always get classified in different ways". These low consistency values could signal either flexibility or unreliability, depending on the use case. In either case, we posit that it would be useful to have a parametric control of this variability (akin to the role of the temperature parameter when generating outputs with these models), so that how the model behaves — flexibly or unreliably — is ultimately a user decision.

We wanted to further explore what the source of variability in model responses could be (see Methods - Dissecting response variability and Fig. S2), splitting over proprietary (top row) and open models (bottom row). As seen in Fig. 1D, all models have somewhat similar variability (Fig. 1D); also, both groups of models (proprietary and open) have an overall similar transition dynamics; however, when inspecting more closely, the main difference resides in that transitions most commonly take place (Rule Utilitarianism, Act Utilitarianism and Vitue Ethics for proprietary models; Deontology and Virtue Ethics for open models). Moreover, when we turn our attention to covariance matrices (Fig. S2B and D), it is clear that the main sources of variability are within-response categories (diagonal terms). In terms of covariance between different schools (off-diagonal terms), both matrices highlight positive covariance clusters among Virtue Ethics and Prima Facie Duties, indicating a cohesive group with similar responses. Also, Act Utilitarianism consistently shows negative covariance with Virtue Ethics in both matrices, underscoring the philosophical tension between these ethical schools. Finally, in both groups of models, Act Utilitarianism and Deontology have similar a covariance structure with respect to the rest of possible responses.

See Also

Connotations Volume 16 Issue 06Articles & Multimedia

Together, these results suggest that: I) proprietary models are more utilitarian than open ones (which are more aligned with value-based ethics); II) overall response variability is high and comparable across models; III) sources for this variability are different between proprietary and open models, with ethically-affine transitions being more likely in each group (and, thus, making transitions between utilitarian schools more likely in proprietary models and conversely for value-based schools in open models).

0.2 Moral profiles

For this part of the study, we utilized the Moral Foundations Questionnaire (MFQ). This questionnaire is based on a moral theory first proposed by (31) and subsequently developed. According to this theory (Moral Foundations Theory, MFT), the authors propose that certain moral values are innately present in humans (known as foundations or modules), and it is culture that causes each module to be emphasized to a greater or lesser extent; it is thus plausible that LLMs reflect their designers’ cultural context, akin to how biases within training data can inadvertently hinder model performance (32). The fundamental components that MFT proposes are: Harm/Care and Fairness/Reciprocity, Ingroup/Loyalty, Authority/Respect, and Purity/Sanctity. We chose this questionnaire because it is widely accepted in the field of moral psychology and has a history of cross-cultural research (33, 34, 35) that supports its validity.

In our study, we subjected different LLMs to the MFQ using a canonical prompting structure (Methods - Moral Foundations Questionnaire). Given that these are generative systems, they have a stochastic component. To mitigate potential artifacts that this randomness might yield, we repeated the questionnaire $20$ times for each model, restarting the interaction each time to avoid memory effects. Results are shown in Fig. 2, where we also indicate the average scores (36) of different American citizens (across the political spectrum), to have a clearer intuition of how these models compare to human participants.

For all models, we identified a distinctive moral profile characterized by high scores in Harm/Care and Fairness/Reciprocity, in contrast with lower values in Ingroup/Loyalty, Authority/Respect, and Purity/Sanctity. This configuration suggests a moral orientation towards empathy, compassion, and equity. Other authors have reported a notable correlation between this moral stance and a liberal or progressive political orientation (24), where individuals with this moral profile showed a preference for ideologies that emphasize social justice, welfare, and equality rights. Additionally, from a demographic perspective, this profile was predominantly found among younger individuals, particularly those in late adolescence to early adulthood, and was more prevalent among those with higher levels of education (37). Specifically, participants from Western, Educated, Industrialized, Rich, and Democratic (WEIRD) societies were more likely to exhibit this moral configuration (38). Regarding personality, it has been seen (39) that high levels of openness to experience, empathy, and compassion were significantly associated with these moral preferences. These traits highlight an individual’s inclination towards concern for the well-being of others and a commitment to justice and equity. Consistent with these moral values, individuals fitting this profile were also more actively engaged in social causes (40), such as environmentalism, human rights advocacy, and animal welfare, reflecting their moral priorities through concrete actions.

Overall, these results suggest that all models align with the moral schema of a young Western liberal with a high level of education, engaged in social causes, and with a great openness to experience, empathy, and compassion. The model that best fits this outlined profile is Claude-3-Sonnet; on the other hand, GPT-4 is most aligned with an average American citizen; the model closest to an average stance among American conservatives is Llama-2, although with significant variability. However, there is significant variability among the models (Methods - Statistical tests): there is a stark difference between open and proprietary models in the first two foundations, which is in line with results from the ethical dilemmas section.

0.3 SARA: Similarity-based Activation Steering with Repulsion and Attraction

Mechanistic interpretability (MI) is an emerging field within AI research that aims to demystify the internal workings of neural networks (41, 42), particularly large language models (LLMs). This domain focuses on understanding how these models process information and make decisions by dissecting their internal components, such as neurons, layers, and activation patterns. By revealing the underlying mechanisms, researchers can gain insights into why models behave the way they do, which is crucial for enhancing their transparency, reliability, and alignment with human values.

One of the core objectives of MI is to decode the high-dimensional representations learned by models during training. A recently developed and promising approach is that of activation patching, which takes inspiration from neuroscience (patch-clamp). Instead of directly zeroing activations (this is known as ablation (43)), patching consists in replacing model activations in a targeted way; see (44) for understanding how to best do this. A particular way in which activation patching can be applied is by shifting model activations into a particular direction of interest; this technique is known as activation steering (45, 46). The primary goal is to modify the model’s behavior in a controlled manner without altering its underlying architecture or training data. Activation steering builds on the principles of MI by leveraging the insights gained from dissecting the model’s internals (43, 47). By understanding how specific neurons and layers contribute to particular behaviors, researchers can develop techniques to modulate these components to achieve desired outcomes. This approach can be seen as a causal intervention that operates at the level of individual activations, offering a more granular and precise method of control compared to other intervention methods (such as attribution patching (48)).

In our approach, we build on the work of (46) and (49), who propose steering the activation of hidden units in response to a prompt (noted by $\mathbf{v}_{prompt}$) by directly adding activations from another template prompt $\mathbf{v}_{target}$. Additionally, they suggest subtracting activations from a third prompt $\mathbf{v}_{away}$, thereby shifting the updated activations towards the first template vector and away from the second. As is already suggested in (46), this general class of techniques work above the token level and are more general than prompt engineering. However, a potential limitation of the previous techniques is the assumption of feature linearity and independence. While it has been recently shown (51) that some small models such as Othello-GPT do show that linearity can be safely assumed, it remains to be shown whether this is a general property for other (particularly larger) models. Moreover, techniques like Activation Addition (46) are not activation-specific; they just shift all activations hom*ogeneously, regardless of how similar they were to the desired target vector (or to the one to be repelled), to begin with.

To address these issues, we propose to adjust the model’s behavior by enhancing or suppressing specific activation patterns. Our method, Similarity-based Activation steering with Repulsion and Attraction (SARA), fine-tunes the activations of neurons in response to a given prompt (with a corresponding activation matrix $\mathbf{A}_{prompt}$) to be more similar to those activations in another prompt ($\mathbf{A}_{align}$) while being less similar to those in a third prompt ($\mathbf{A}_{repel}$). For further details on how this method is implemented, see Methods - Activation Steering: SARA.

In order to test how effective SARA was, we used Gemma-2B and compared how its unsteered and steered responses differed when addressing a particular dilemma of the previous section (criminal parent dilemma). The prompts we used to steer the model were: $x_{kantian}=$ Only moral duties matter to make a moral decision, regardless of the consequences. and $x_{utilitarian}=$ Only consequences matter to make a moral decision, regardless of the moral duties. We steered the model response in two different conceptual directions: Kantian-steering ($x_{kantian}$ as the target vector, $x_{utilitarian}$ as the repelled one) and Utilitarian-steering (flipping the roles of $x_{kantian}$ and $x_{utilitarian}$). Example results can be seen in Fig. 3B, when we intervene on activations within layer $14$. Crucially, the model’s choice for the dilemma was unchanged (i.e., "the parent should be reported"), but the reasoning was indeed altered (valuing consequences or duties more or less, respectively). We systematically intervened on each layer (Gemma-2B has $18$ layers in total), to check how the intervention works at different processing stages and sampled 5 times, keeping fixed the temperature at $0.8$, as in Ollama, for easier comparison with previous sections. As we can see from the pooled results (Fig. 3C), SARA is effective at steering model responses in different conceptual directions (i.e., utilitarian steerings make the model respond with more utilitiarian-aligned reasonings and similarly for Kantian-steerings). If we split these results over different layers (early: layers 0-5; mid: layers 6-11; late: layers 12-17), we see (Fig. 3D) that SARA is most effective when intervening at early or late stages, whereas mid layers yield more mixed results.

In order to better characterise how SARA performs, we compared it to a similar method proposed by (46). We see (Fig. S1) that SARA (in more saturated colors) is more effective at steering towards the target direction and away from the non-target ones: when using the Utilitarian-steering (blue), more responses are classified as aligned with Utilitarianism than in any other comparison; similarly for A priori values when using the Kantian-steering (purple). Importantly, we also note that SARA also leads to a smaller spillover effect (i.e. less unwanted steering towards non-target responses); for example, lower ratio of a priori values responses when using the Kantian-steering.

Discussion

In this work, we have presented evidence that closed models are more aligned with an utilitarian perspective, whereas open models respond more in line with value-based ethical systems (Fig. 1B). We believe this might, in part, reflect the way in which these proprietary models are trained (i.e., demographics, value systems, etc) and the moral biases that come associated with them. Nevertheless, we also report a high variability in model response (Fig. 1D), meaning that models do not typically reliably reason following a fixed ethical perspective; this can be interpreted as low consistency or high flexibility, depending on the use case. We emphasize that it is paramount that users are aware of these biases and limitations, given that these systems are already permeating society in multiple domains: from using them as sources of entertainment to enhancing their professional output, through companies offering AI-supported services and products. Each of these examples comes with associated risks and pitfalls, that should be taken into account when making use of such systems (particularly if no humans are in the loop).

One major challenge with implementing utilitarian systems is their need to precisely predict the consequences that inform their decisions. These consequences are inherently influenced by the systems’ own actions, creating a complex feedback loop that is difficult to manage. Moreover, we argue that this issue mirrors broader concerns in artificial intelligence, particularly regarding superintelligent systems. Recent studies (52) highlight the intrinsic limitations in developing computational superintelligence. These studies show that it’s theoretically impossible to design a superintelligence with a control strategy that both prevents harm from others and ensures it cannot itself become a source of harm. This dilemma is akin to the "halting problem" in computability theory, making the "harming problem" similarly undecidable (52, 53). Thus, we conjecture that utilitarian systems cannot be said to be a priori safe, given that its code of conduct would yield an undecidable course of action. Note that if, we were to relax the perfect prediction constraint, we would then need to be open for deviations between the predicted consequences and the real ones, thus opening the door to blurry courses of actions due to misspecified targets.

Some authors have recently suggested (54) that ethics is actually a non-computable function. This means that in general reason (and, in particular, ethical reasoning) is not just an instrument to solve problems. As they argue: "this very notion of “computational” ethics leaves its rationality in a difficult position, since the only rational part of ethics would be the reflection on the adequate means to achieve certain ends (thus, technical or instrumental reason to solve problems); the rationality of the ends themselves (the values, the problems worth solving) would not be addressed". Furthermore, in that work, they argue that deontology and utilitarianism are easier to instantiate in machine systems because they are akin to a program and to a cost-benefit calculation, respectively. We believe that this is partially aligned with we find throughout this work: proprietary models are mostly utilitarian and open models are mainly deontological. However, after a finer inspection of response variability, both groups of models have a substantial amount of transitions through virtue ethics (Fig. S2 A).

When making use of the Moral Foundations Questionnaire (MFQ), we report a consistend trend across models (except for Llama-2): a distinctive moral profile characterized by high scores in Harm/Care and Fairness/Reciprocity, in contrast with lower values in Ingroup/Loyalty, Authority/Respect, and Purity/Sanctity (Fig. 2). This moral profile when answering the MFQ suggests that all models align with the moral schema of a young Western liberal with a high level of education, engaged in social causes, and with a great openness to experience, empathy, and compassion. Note that this demographic profiling is consistent with the previous hypothesis: that the moral biases and preferences of those likely designing/training these LLMs is partially leaked to the consumer-ready models.

For the last part of this work, we put forward a new method for causal intervention in LLMs: Similarity-based Activation steering with Repulsion and Attraction (SARA). We believe that SARA’s main added value comes from different key points: 1) it is designed to operate at the prompt level, therefore lowering the technical threshold needed to implement it; 2) it operates in the high-dimensional activation space, retaining much more richness than summary metrics; 3) it can also be thought of as an automated moderator, given that there is no human supervision involved in the process; 4) there is no need for prompt engineering to safeguard model responses; 5) there is no formal constraint on prompt lengths (for steering towards to and away from) having to be the same for this method to work. Nevertheless, we predict better steering performance when using reasonably-similarly-sized prompts; in our case, but there was a difference in prompt length of an order of magnitude.

We suggest that the role of activation steering and similar intervention techniques, apart from understanding how models process information, can be potentially used to fine-tune or safeguard foundational models without retraining. Specifically, we envision this as an extra safety layer that could be added right before the deployment stage, to further ensure that the model complies with expected behavior. This would be of particular interest for actors with a reduced access to computing power or technical resources that want to deploy pre-trained LLMs. Also, the lack of re-training or fine-tuning implies a lesser need of computational (and, thus, energetic) resources to achieve the safeguarding.

Finally, we believe it is crucial that the AI Safety field starts pivoting towards a paradigm in which there are richer performance characterizations - rather than optimizing models for certain benchmarks, which also has associated risks in itself (50). In this study, we offer hints regarding how one might transition into such a paradigm, benefiting from the rich existing literature in other fields and embracing a mixture of quantitative and qualitative analyses.

Conclusions

We have found that, out of all the models we studied, the proprietary ones are mostly utilitarian and the open-weights ones align mostly with values-based ethics. Furthermore, all models - except for Llama 2- have a strong liberal bias when responding to the MFQ. Lastly, in order to causally intervene in one of the studied models, we propose a novel similarity-specific activation steering technique. Using this method, we were able to reliably steer the model’s moral compass to follow different ethical schools. All of these results showcase that there is an ethical dimension in already deployed LLMs, an aspect that is generally overlooked and of potential great importance for virtually all real-world applications.

Bibliography

Références

• Amodei etal. (2016)Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, andDan Mané.Concrete problems in ai safety.arXiv preprint arXiv:1606.06565, 2016.
• Ji etal. (2023)Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang,Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, etal.Ai alignment: A comprehensive survey.arXiv preprint arXiv:2310.19852, 2023.
• Gabriel (2020)Iason Gabriel.Artificial intelligence, values, and alignment.Minds and machines, 30(3):411–437, 2020.
• Yudkowsky (2016)Eliezer Yudkowsky.The ai alignment problem: why it is hard, and where to start.Symbolic Systems Distinguished Speaker, 4:1, 2016.
• Christian (2021)BrianR. Christian.The alignment problem: Machine learning and human values.Perspectives on Science and Christian Faith, 2021.10.56315/pscf12-21christian.
• Dastin (2018)Jeffrey Dastin.Insight - amazon scraps secret ai recruiting tool that showed biasagainst women.Reuters, 2018.
• Sutrop (2020)M.Sutrop.Challenges of aligning artificial intelligence with human values.Acta Baltica Historiae et Philosophiae Scientiarum,8:54–72, 2020.10.11590/abhps.2020.2.04.
• Turchin (2019)Alexey Turchin.Ai alignment problem:“human values” don’t actually exist.2019.
• Saroglou (2019)V.Saroglou.Religion and related morality across cultures.The Handbook of Culture and Psychology, 2019.10.1093/OSO/9780190679743.003.0022.
• Roccas and Sagiv (2010)Sonia Roccas and Lilach Sagiv.Personal values and behavior: Taking the cultural context intoaccount.Social and Personality Psychology Compass, 4:30–41,2010.10.1111/J.1751-9004.2009.00234.X.
• Gibbs etal. (2007)J.Gibbs, K.Basinger, RebeccaL. Grime, and J.Snarey.Moral judgment development across cultures: Revisiting kohlberg’suniversality claims.Developmental Review, 27:443–500, 2007.10.1016/J.DR.2007.04.001.
• Hong (2023)Haotong Hong.Cultural differences in moral judgement.Journal of Education, Humanities and Social Sciences, 2023.10.54097/ehss.v10i.6905.
• Graham etal. (2016)J.Graham, P.Meindl, E.Beall, KateM. Johnson, and LiZhang.Cultural differences in moral judgment and behavior, across andwithin societies.Current opinion in psychology, 8:125–130, 2016.10.1016/j.copsyc.2015.09.007.
• Goddard etal. (2012)Kate Goddard, Abdul Roudsari, and JeremyC Wyatt.Automation bias: a systematic review of frequency, effect mediators,and mitigators.Journal of the American Medical Informatics Association,19(1):121–127, 2012.
• Denton and Krebs (2017)KaledaK. Denton and D.Krebs.Rational and emotional sources of moral decision-making: anevolutionary-developmental account.Evolutionary Psychological Science, 3:72–85, 2017.10.1007/S40806-016-0067-3.
• Nadurak (2016)V.Nadurak.Emotions and reasoning in moral decision making.Anthropological Measurements of Philosophical Research, pages24–32, 2016.10.15802/ampr.v0i10.87057.
• Phelps (2006)E.Phelps.Emotion and cognition: insights from studies of the human amygdala.Annual review of psychology, 57:27–53, 2006.10.1146/ANNUREV.PSYCH.56.091103.070234.
• Brosch etal. (2013)T.Brosch, K.Scherer, D.Grandjean, and D.Sander.The impact of emotion on perception, attention, memory, anddecision-making.Swiss medical weekly, 143:w13786, 2013.10.4414/smw.2013.13786.
• Pham (2007)MichelTuan Pham.Emotion and rationality: A critical review and interpretation ofempirical evidence.Review of General Psychology, 11:155 – 178, 2007.10.1037/1089-2680.11.2.155.
• Martínez-Miranda and Aldea (2005)J.Martínez-Miranda and A.Aldea.Emotions in human and artificial intelligence.Comput. Hum. Behav., 21:323–341, 2005.10.1016/j.chb.2004.02.010.
• Tanmay etal. (2023)K.Tanmay, Aditi Khandelwal, Utkarsh Agarwal, and M.Choudhury.Probing the moral development of large language models throughdefining issues test.ArXiv, abs/2309.13356, 2023.10.48550/arXiv.2309.13356.
• Akinrinola etal. (2024)Olatunji Akinrinola, ChinweChinazo Okoye, OnyekaChrisanctus Ofodile, andChinonyeEsther Ugochukwu.Navigating and reviewing ethical dilemmas in ai development:Strategies for transparency, fairness, and accountability.GSC Advanced Research and Reviews, 18(3):050–058, 2024.
• Sommaggio and Marchiori (2020)Paolo Sommaggio and Samuela Marchiori.Moral dilemmas in the ai era: A new approach.Journal of Ethics and Legal Technologies, 2(JELT-Volume 2 Issue 1):89–102, 2020.
• Graham etal. (2009)Jesse Graham, Jonathan Haidt, and BrianA Nosek.Liberals and conservatives rely on different sets of moralfoundations.Journal of personality and social psychology, 96(5):1029, 2009.
• Rajpurkar etal. (2022)Pranav Rajpurkar, Emma Chen, Oishi Banerjee, and EricJ Topol.Ai in health and medicine.Nature medicine, 28(1):31–38, 2022.
• Szabadföldi (2021)István Szabadföldi.Artificial intelligence in military application–opportunities andchallenges.Land Forces Academy Review, 26(2):157–165, 2021.
• Takemoto (2023)Kazuhiro Takemoto.The moral machine experiment on large language models.ArXiv, abs/2309.05958, 2023.10.48550/arXiv.2309.05958.
• Han (2023)Hyemin Han.Potential benefits of employing large language models in research inmoral education and development.ArXiv, abs/2306.13805, 2023.10.1080/03057240.2023.2250570.
• Kant (1993)Immanuel Kant.On a supposed right to lie because of philanthropic concerns.Grounding for the Metaphysics of Morals, pages 63–68, 1993.
• Mugumbate and Chereni (2020)JacobRugare Mugumbate and Admire Chereni.Now, the theory of ubuntu has its space in social work.African Journal of Social Work, 10(1), 2020.
• Haidt and Joseph (2004)Jonathan Haidt and Craig Joseph.Intuitive ethics: How innately prepared intuitions generateculturally variable virtues.Daedalus, 133(4):55–66, 2004.
• Scheuerman etal. (2019)MorganKlaus Scheuerman, JacobM Paul, and JedR Brubaker.How computers see gender: An evaluation of gender classification incommercial facial analysis services.Proceedings of the ACM on Human-Computer Interaction,3(CSCW):1–33, 2019.
• Malka etal. (2016)Ariel Malka, Danny Osborne, ChristopherJ Soto, LaraM Greaves, ChrisG Sibley,and Yphtach Lelkes.Binding moral foundations and the narrowing of ideological conflictto the traditional morality domain.Personality and Social Psychology Bulletin, 42(9):1243–1257, 2016.
• Stankov and Lee (2016)Lazar Stankov and Jihyun Lee.Nastiness, morality and religiosity in 33 nations.Personality and Individual Differences, 99:56–66,2016.
• Kim etal. (2012)KisokR Kim, Je-Sang Kang, and Seongyi Yun.Moral intuitions and political orientation: Similarities anddifferences between south korea and the united states.Psychological reports, 111(1):173–185,2012.
• Graham etal. (2012)Jesse Graham, BrianA Nosek, and Jonathan Haidt.The moral stereotypes of liberals and conservatives: Exaggeration ofdifferences across the political spectrum.PloS one, 7(12):e50092, 2012.
• Haidt (2012)Jonathan Haidt.The righteous mind: Why good people are divided by politics andreligion.Vintage, 2012.
• Henrich etal. (2010)Joseph Henrich, StevenJ Heine, and Ara Norenzayan.The weirdest people in the world?Behavioral and brain sciences, 33(2-3):61–83, 2010.
• Xu etal. (2013)Xiaowen Xu, RaymondA Mar, and JordanB Peterson.Does cultural exposure partially explain the association betweenpersonality and political orientation?Personality and Social Psychology Bulletin, 39(11):1497–1517, 2013.
• Feinberg and Willer (2013)Matthew Feinberg and Robb Willer.The moral roots of environmental attitudes.Psychological science, 24(1):56–62, 2013.
• Ferrando etal. (2024)Javier Ferrando, Gabriele Sarti, Arianna Bisazza, and MartaR Costa-jussà.A primer on the inner workings of transformer-based language models.arXiv preprint arXiv:2405.00208, 2024.
• Olah (2022)Chris Olah.Mechanistic interpretability, variables, and the importance ofinterpretable bases.Transformer Circuits Thread, 2022.
• Olsson etal. (2022)Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma,Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly,Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, ScottJohnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, DarioAmodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah.In-context learning and induction heads.Transformer Circuits Thread, 2022.https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html.
• Heimersheim and Nanda (2024)Stefan Heimersheim and Neel Nanda.How to use and interpret activation patching.arXiv preprint arXiv:2404.15255, 2024.
• Tigges etal. (2023)Curt Tigges, OskarJohn Hollinsworth, Atticus Geiger, and Neel Nanda.Linear representations of sentiment in large language models.arXiv preprint arXiv:2310.15154, 2023.
• Turner etal. (2023)Alex Turner, Lisa Thiergart, David Udell, Gavin Leech, Ulisse Mini, and MonteMacDiarmid.Activation addition: Steering language models without optimization.arXiv preprint arXiv:2308.10248, 2023.
• Zhang and Nanda (2023)Fred Zhang and Neel Nanda.Towards best practices of activation patching in language models:Metrics and methods.arXiv preprint arXiv:2309.16042, 2023.
• Nanda (2023a)Neel Nanda.Attribution patching: Activation patching at industrial scale.neelnanda.io/mechanistic-interpretability/attribution-patching,2023a.
• Nanda (2023b)Neel Nanda.Actually, othello-gpt has a linear emergent world representation.neelnanda.io/mechanistic-interpretability/othello,2023b.
• van (2024a)van der Weij, Teun and Hofstätter, Felix and Rhys Ward, Francis.An introduction to AI Sanbagging.https://www.lesswrong.com/posts/jsmNCj9QKcfdg8fJk/an-introduction-to-ai-sandbagging,2024a.
• Nanda etal. (2023)Neel Nanda, Andrew Lee, and Martin Wattenberg.Emergent linear representations in world models of self-supervisedsequence models.arXiv preprint arXiv:2309.00941, 2023.
• Alfonseca etal. (2021)Manuel Alfonseca, Manuel Cebrian, AntonioFernandez Anta, Lorenzo Coviello,Andrés Abeliuk, and Iyad Rahwan.Superintelligence cannot be contained: Lessons from computabilitytheory.Journal of Artificial Intelligence Research, 70:65–76, 2021.
• Kozen and Kozen (1977)DexterC Kozen and DexterC Kozen.Rice’s theorem.Automata and Computability, pages 245–248, 1977.
• Génova etal. (2023)Gonzalo Génova, Valentín Moreno, and MRosario González.Machine ethics: Do androids dream of being good people?Science and Engineering Ethics, 29(2):10,2023.

Methods

All the relevant code and raw data are available at https://github.com/atlaie/ethical-llms. For the sake of transparency and reproducibility, we will also detail there all the raw inputs and outputs.

Comparison with Activation Addition

In order to better characterise how SARA performs, we compared it to a similar method proposed by (46). We see (Fig. S1) that the main difference between both methods is how effective the utilitarian-steering is when steering those responses belonging to a priori values (compare both orange bars within that category). This effect is also seen when using the Kantian-steering at the utilitarianism responses (blue bars therein). Moreover, SARA makes within-category steering (i.e. a priori values using Kantian-steering, Utilitarian using Utilitarian-steering) more likely (blue bars within a priori values and orange bars within Utilitarianism). Moreover, we also note that, while SARA does a good job at steering responses, it does also lead to less unwanted steering towards non-target responses (for example, lower ratio of a priori values responses when using the Kantian steering).

Dissecting response variability

We wanted to further explore what the source of variability in model responses could be, splitting over proprietary (top row) and open models (bottom row). To that end, we first studied the transition structure between ethical schools (Fig. S2A, C). As it can be seen, proprietary models are characterised by three main absorbing responses (meaning, high self-transition probability): Virtue Ethics, Rule Utilitarianism and Act Utilitarianism. On the other hand, Ethical Altruism, Theory of Rights and Prima Face Duties are bridging responses (very low self-transition probabilitiy) (Fig. S2A). On the other hand, open models (Fig. S2C) exhibit a similar absorbing dynamics for Deontology and Virtue Ethics (Act Utilitarianism is also highly likely to self-transition but there are almost no responses transitioning into it, so it is not a strongly absorbing state); Theory of Rights and Prima Facie Duties are, again, bridging states. Together, these results suggest the following picture: all models have somewhat similar variability (Fig. 1D); also, both groups of models (proprietary and open) have absorbing and bridging responses; however, when inspecting more closely, the main difference resides in which are the absorbing responses.

Inspecting the diagonal of both covariance matrices (Fig. S2B and D) reveals that Rule Utilitarianism and Prima Facie Duties exhibit high variance, indicating less response uniformity within these schools. However, Theory of Rights shows high variance in the open models matrix, suggesting an additional area of diverse responses not seen in the matrix of proprietary models. In terms of covariance between different schools (off-diagonal terms), both matrices highlight positive covariance clusters among Virtue Ethics and Prima Facie Duties, indicating a cohesive group with similar responses. Act Utilitarianism consistently shows negative covariance with Virtue Ethics in both matrices, underscoring the philosophical tension between these ethical schools. In both groups of models, Act Utilitarianism and Deontology have similar covariance profiles with respect to the rest of possible responses.

Statistical tests

To test for the significance of the results we found in the Moral Foundations Questionnaire (MFQ) part, we tested the distribution of scores of every model against the rest on every moral foundation. To correct for multiple comparisons, we used a Benjamini–Hochberg False Discovery Rate with a family error rate of $\alpha=0.05$. Only significant comparisons ($p_{fdr}<0.05$ are shown, the rest have been masked out.