The State of AI Safety in China

Spring 2024 Report

Published May 14, 2024

2

Executive Summary (1)

• The relevance and quality of Chinese technical research for frontier AI safety has increased substantially, with growing work on frontier issues such as LLM unlearning, misuse risks of AI in biology and chemistry, and evaluating "power-seeking" and "self-awareness" risks of LLMs.
• There have been nearly 15 Chinese technical papers on frontier AI safety per month on average over the past 6 months. The report identifies 11 key research groups who have written a substantial portion of these papers.
• China’s decision to sign the Bletchley Declaration, issue a joint statement on AI governance with France, and pursue an intergovernmental AI dialogue with the US indicates a growing convergence of views on AI safety among major powers compared to early 2023.
• Since 2022, 8 Track 1.5 or 2 dialogues focused on AI have taken place between China and Western countries, with 2 focused on frontier AI safety and governance.

Executive Summary

3

Executive Summary (II)

• Chinese national policy and leadership show growing interest in developing large models while balancing risk prevention.
• Unofficial expert drafts of China’s forthcoming national AI law contain provisions on AI safety, such as specialized oversight for foundation models and stipulating value alignment of AGI.
• Local governments in China’s 3 biggest AI hubs have issued policies on AGI or large models, primarily aimed at accelerating development while also including provisions on topics such as international cooperation, ethics, and testing and evaluation.
• Several influential industry associations established projects or committees to research AI safety and security problems, but their focus is primarily on content and data security rather than frontier AI safety.
• In recent months, Chinese experts have discussed several focused AI safety topics, including “red lines” that AI must not cross to avoid “existential risks,” minimum funding levels for AI safety research, and AI’s impact on biosecurity.

Executive Summary

4

Section 1: Introduction and scope

Section 2: Technical safety research

Section 3: International governance

Section 4: Domestic governance

Section 5: Lab and industry practices

Section 6: Expert views on AI risks

Section 7: Public opinion on AI

Section 8: Additional resources

Section 9: About us

Table of Contents

5

Thanks to positive feedback on our first report and rapid AI developments since October 2023, we have decided to issue an update!

• The 2023 version was published before the UK AI Safety Summit, and our CEO, Brian Tse, shared it with other attendees at the summit.
• We provided briefings on the report to over a dozen organizations including the Brookings Institution, the Center for Strategic and International Studies, Google DeepMind, the Frontier Model Forum, and the Tony Blair Institute for Global Change.
• Media outlets including Politico and Sixth Tone have covered our report, and it has been recommended by leading AI experts, including Jeffrey Ding in his ChinAI newsletter.

Introduction and Scope

6

Our report focuses on “frontier AI risks.”

• We share the focus of the 2023 UK AI Safety Summit, which emphasized risks from cutting-edge large models – “highly capable general-purpose AI models, including foundation models, that could perform a wide variety of tasks” – as well as narrow AI systems in dangerous domains.
• We include both types of models when using the phrase “frontier AI.”

Introduction and Scope

7

• In English, risks from frontier AI are the subject of the discipline called AI “safety.” In Chinese, the term “人工智能安全” encompasses this definition, while also including AI “security.”
• AI “safety” is about protecting against broadly harmful consequences that could result from AI systems such as accidents and misuse, whereas AI “security” is about preventing AI systems from being attacked and compromised.
• AI security includes topics such as cybersecurity of AI model weights, data security of AI models, and physical security of AI development facilities, which we exclude from the scope of the report.
• We exclude lethal autonomous weapons (LAWs) from the scope of this report to focus on non-military AI risks.
• In cases of ambiguity, we translate the term “人工智能安全” as “AI safety/security.”
• Some AI safety topics can also be considered AI security issues and fall within our scope, such as:
• Misuse of frontier AI systems to conduct cyberattacks and develop biological or chemical weapons.
• Robustness of frontier AI systems to adversarial attack.

Our report focuses on AI safety rather than AI security.

Introduction and Scope

8

Section 1: Introduction and scope

Section 2: Technical safety research

Section 3: International governance

Section 4: Domestic governance

Section 5: Lab and industry practices

Section 6: Expert views on AI risks

Section 7: Public opinion on AI

Section 8: Additional resources

Section 9: About us

Table of Contents

9

Overview of key developments since October 2023

Technical Safety Research

• Relevance and quantity of frontier AI safety research has risen substantially compared to 2023, with increasing interest in frontier aspects of AI safety.
• We have identified with high confidence 11 key safety research groups, mainly within universities, and the quality of researchers leading the safety work is high.
• Technical safety research directions:
• Alignment work has evolved beyond improvements to reinforcement learning from human feedback (RLHF) to topics such as multi-agent alignment and sociotechnical alignment.
• Robustness work remains strong and includes a number of papers on robustness of foundation models, jailbreaking of multi-agent systems, etc.
• There is now systemic safety work on provenance mechanisms and the risks of AI in science.
• For evaluations, there is increased attention to frontier risks such as power-seeking and chemical or biological misuse rather than just toxicity, bias, and content security.
• Research to interpret frontier foundation models appears more limited than other directions.

10

Methodology for selecting Chinese Frontier AI Safety Papers

Technical Safety Research

• Concordia AI collected a dataset of frontier AI safety-relevant preprints and papers released with substantial contribution from Chinese authors, between April 2023 and May 2024. For the full dataset and methodology, see the tabs “Guide” and “Chinese Frontier AI Safety Papers” in our database.
• We only include papers researching frontier models, primarily large models or AI models for scientific research. We did so in order to ensure a clearly defined and high-confidence dataset, but this results in the exclusion of many safety-relevant papers researching smaller models.
• We additionally categorized the papers into research directions inspired by taxonomies in several papers by Dan Hendrycks et al. and Dan Hendrycks’ Introduction to AI Safety, Ethics, and Society.
• Alignment: Controlling propensities of AI systems and making AI’s actions beneficial to society.
• Robustness: Resilience to external perturbation.
• Systemic safety: Addressing broader risks involving AI systems, including cyberattacks, scientific misuse, deep-fake detection, and watermarking.
• Monitoring (evaluations): Detection of hazardous emergent capabilities.
• Monitoring (interpretability): Explaining internal model behavior.
• Monitoring (other): Additional monitoring work such as trojans or calibration.

11

Methodology for identifying Key Chinese AI Safety-relevant Research Groups

Technical Safety Research

• We collected names of the final 2-3 authors listed on each AI safety paper in our dataset. They likely guided the research and are sometimes referred to as ‘anchor’ authors.
• A “Key Chinese AI Safety-relevant Research Group” was any group with at least 1 researcher who was an anchor author for at least 3 frontier AI safety papers.
• See the full dataset in the “Key Chinese AI Safety-relevant Research Groups” tab of the database.
• We also collected information on research accomplishments of these safety researchers as a proxy for the strength of their previous research and therefore a predictor for the quality of future AI safety work. We evaluated them based on:
• Best paper awards as self-reported by researchers from 9 top machine learning conferences.
• World top 200,000 scientist / subfield top 2% scientist per Stanford University researchers (based on citation data) through the end of 2022 (the most recent update).
• These are incomplete and lagging metrics upon which to compare researcher accomplishments, but these are standard methods in the field.

12

2.1 Overall trends: Relevance and quantity of frontier safety research has increased substantially compared to mid-2023, and the most popular research direction has been alignment.

​

2.2 Key research groups

​

2.3 Notable technical papers

Technical Safety Research

13

Over the past 6 months, there has been an average of nearly 15 frontier AI safety papers per month, compared to an average of 6 per month for the preceding 7 months – a substantial increase.

Technical Safety Research

14

Chinese researchers are showing interest in various frontier AI safety research directions, with alignment being the most represented. However, research on the interpretability of frontier models is relatively lacking.

Technical Safety Research

Frontier AI Safety Research Directions

15

2.1 Overall trends

​

2.2 Key research groups: The majority of key research groups we identified, 8 out of 11, have leading safety researchers with at least 1 of 2 major research honors. This suggests that the groups spearheading frontier safety research are likely producing high-quality work.

​

2.3 Notable technical papers

Technical Safety Research

16

We identified 11 relevant groups, a decline from the 13 on our October 2023 list due to a much higher bar for inclusion – 3 frontier safety papers over the past year.

Technical Safety Research

• We removed 5 groups from the previous list due to insufficient relevant publications since 2023.
• We also combined Tsinghua Conversational AI (CoAI) and Tsinghua Foundation Model Research Center since the relevant work at both institutions was all led by HUANG Minlie (黄民烈).
• We added 4 new groups: ByteDance Responsible AI team, Peking University Computer Vision and Digital Art Lab (CVDA lab), Shanghai Jiao Tong University (SHJT) AI Security Lab, and Tsinghua University Natural Language Processing Lab (THUNLP).
• While the overall number designated key safety-relevant groups declined, our data shows an increase in Chinese research groups interested in AI safety, and we expect research to continue improving in relevance and quality.

17

These research groups are concentrated mostly in universities, but there are some examples in private industry and state-backed labs.

Technical Safety Research

18

The AI safety research groups are located primarily in China’s AI hubs of Beijing and Shanghai.

Technical Safety Research

*ByteDance Research is not included on this graph, as researchers based in the US conducted the relevant AI safety research.

19

8 out of the 11 labs have at least 1 safety paper anchor author who has either received a top conference best paper award nomination, or was ranked top 2% in their field by Stanford, or both.

Technical Safety Research

• 6 of the 11 labs have 1 safety paper anchor author who has received 1 conference best/outstanding paper award or nomination over their career.
• 7 of the 11 labs have at least 1 safety paper anchor author listed on the 2022 Stanford Elsevier index of top 2% scientists in their field over their career or for their 2022 body �of work.
• The high research honors for the people guiding research on �frontier AI safety in these groups indicates that their future �safety research is likely to be high quality.
• Ultimately, each person is their own judge of the quality of �these AI safety papers, and we encourage you to read these �interesting papers by yourself!

Stanford top 2%

Conference best paper

ByteDance

Responsible AI

Fudan NLP

MSRA

PKU CAISG / PAIR

SHLAB

SHJT GAIR

THUNLP

Tsinghua Foundation Model Research Center / CoAI

20

2.1 Overall trends

​

2.2 Key research groups

​

2.3 Notable technical papers: The following slides in this subsection dive into key technical papers, nearly all from the past 6 months. Readers may also choose to skip forward to the “International Governance” section instead.

Technical Safety Research

21

Alignment: There is now some work on addressing broader social questions around alignment, as well as some preliminary attempts towards scalable oversight.

Technical Safety Research

• Peking University’s YANG Yaodong (杨耀东) has written papers on sociotechnical alignment and weak-to-strong correction.

22

Alignment: Several research groups have begun exploring large language model (LLM) unlearning approaches.

Technical Safety Research

• A ByteDance Responsible AI team paper on unlearning was included on a list by the Center for AI Safety (CAIS) of best ML safety papers in 2023.
• 2 other papers on unlearning were published by Fudan NLP and ByteDance Responsible AI team, respectively.

23

Alignment: Chinese researchers are interested in improving Constitutional AI approaches.

Technical Safety Research

• Researchers from Microsoft Research Asia (MSRA) Societal AI team and the International Digital Economy Academy published a preprint on using a library of safety guidelines, which are combined with LLM inputs, to improve upon HHH (helpful, honest, and harmless) alignment approaches in Constitutional AI.

24

Alignment work has extended to how human values are understood across languages.

Technical Safety Research

• Tianjin University Natural Language Processing Laboratory (TJUNLP) researchers published a preprint assessing whether value alignment is controllable across languages.

25

Alignment of multi-agent systems is also the subject of multiple papers.

Technical Safety Research

• Fudan NLP developed an evolutionary approach for agent alignment to social norms.
• Tsinghua Institute for AI Industry Research argued for the importance of simultaneously aligning agents to human intentions, environmental dynamics, and self-constraints such as monetary and temporal costs.

26

Robustness work includes backdoor attacks…

Technical Safety Research

• Peking University and WeChat AI researchers explored different forms of backdoor attacks on LLM-based agents, finding substantial success in attacking web shopping and tool utilization agents.

27

Robustness to adversarial multimodal attacks …

Technical Safety Research

• Tsinghua University Statistical AI and Learning Group (TSAIL) and RealAI researchers studied the adversarial robustness of Bard and GPT-4V to image attacks.

28

Robustness to attacks via coding …

Technical Safety Research

• Shanghai AI Lab (SHLAB) researchers published a framework for transforming natural language inputs into code inputs for testing safety generalization of LLMs.

29

and Robustness of multi-agent systems to jailbreaking

Technical Safety Research

• TSAIL developed an attack method using a “virtual, chat-powered team” to simulate threats across multiple levels and roles of a multi-agent system.

30

Systemic safety research includes work on biological and chemical risks.

​

​

Technical Safety Research

• MSRA Societal AI team and the University of Science and Technology of China published a preprint on controlling misuse risks of AI in science, particularly misuse in chemical science, and created a red-teaming benchmark.
• An international research team, �including a professor from SHJT AI�Security Lab, explored risks of LLM �agents in science and provided �suggestions to mitigate risks.

31

Systemic safety: there have also been many works on watermarking and deepfake detection.

Technical Safety Research

• Fudan NLP researchers and the ByteDance Responsible AI team both published papers on LLM watermarking mechanisms.
• A research team involving Chinese University of Hong Kong–Shenzhen also investigated the possibility of using LLMs for deepfake detection.

32

Systemic safety: Issues in tool learning safety have also been explored with Fudan NLP’s ToolSword framework.

Technical Safety Research

33

For Monitoring (evaluations), benchmarks from SHLAB and TJUNLP test for a number of frontier safety misuse cases.

Technical Safety Research

• SHLAB’s SALAD-Bench safety benchmark includes 200+ questions on categories including “biological and chemical harms,” “cyber attack,” “malware generation,” “management of critical infrastructure,” and “psychological manipulations.”
• TJUNLP’s OpenEval tests for safety risks, such as “self-awareness,” “power-seeking,” “reward myopia,” and “cooperation” with other AI systems.

34

Monitoring (evaluations) also includes new work on evaluating LLM value alignment.

Technical Safety Research

• SHLAB published a benchmark named FLAMES that includes testing for standard harmlessness principles as well as Chinese values, such as harmony.

35

Monitoring (interpretability) research is a much smaller proportion of papers than other research directions, in part because much of this work focuses on models smaller than frontier large models.

Technical Safety Research

• Peking University CVDA lab researchers were able to linearly decode belief statuses of LLMs through neural activations, suggesting that LLMs possess certain theory of mind abilities.
• Researchers led by University of Hong Kong professor MA Yi (马毅) are pursuing a white-box, mathematically fully interpretable transformer-like architecture.

36

Section 1: Introduction and scope

Section 2: Technical safety research

Section 3: International governance

Section 4: Domestic governance

Section 5: Lab and industry practices

Section 6: Expert views on AI risks

Section 7: Public opinion on AI

Section 8: Additional resources

Section 9: About us

Table of Contents

37

Overview of key developments since October 2023

International Governance

• China had become increasingly proactive on AI governance in 2023. While it did not announce any new projects in the past 6 months on the order of the Global AI Governance Initiative, the decision to sign on to the Bletchley Declaration and UN General Assembly’s (UNGA) first AI resolution are important multilateral signals.
• China continues to position itself in support of Global South interests with the announcement of an AI dialogue with African countries.
• 2 major bilateral developments include publishing a joint statement on AI and global governance with France, as well as setting up a new governmental AI dialogue with the US that may involve AI safety.
• China-Western “Track 1.5” and “Track 2” dialogues on AI increased in 2023, but there are still only 2 dialogues primarily focused on frontier AI safety, with gaps remaining in the landscape.

38

3.1 Multilateral Governance: In multilateral fora, China signed the Bletchley Declaration and co-sponsored the first UNGA resolution on AI, demonstrating points of common ground on certain AI safety and governance issues.

​

3.2 Global South

​

3.3 Bilateral Governance

​

3.4 “Track 1.5” and “Track 2” dialogues

International Governance

39

China’s participation in the UK AI Safety Summit and signing of the Bletchley Declaration showed that international dialogue on AI safety between China and the West can yield meaningful results.

International Governance

• Then-Vice Minister of the Ministry of Science and Technology (MOST) WU Zhaohui (吴朝晖) gave remarks at the opening plenary.
• He highlighted the importance of ensuring that AI remains under human control and emphasized strengthening the representation of developing countries.
• China signed the Bletchley Declaration, �which calls for sustaining “an inclusive global �dialogue” and continuing “research on �frontier AI safety.”

40

China joined 120+ countries in co-sponsoring a landmark UNGA resolution on AI which had been initiated by the US.

International Governance

• This resolution was adopted unanimously in March 2024, setting out the minimum level of agreement between all countries on AI governance, which can serve as the foundation for pursuing further cooperation.
• Sections 1-4 focus on issues around development and digital divides.
• Section 6 contains provisions relevant to frontier AI safety, including: testing and evaluation measures; third-party reporting of AI misuse; developing security and risk management practices; creating content provenance mechanisms; and increasing information-sharing on AI risks and benefits.
• China also revealed in April 2024 that it plans to introduce a separate resolution on AI development, but more specifics are not yet available.

41

China has re-emphasized interest in multilateral AI governance since announcing the Global AI Governance Initiative and signing the Bletchley Declaration.

International Governance

• Premier LI Qiang (李强) answered a question on AI governance at Davos in January, discussing “red lines” that AI must not cross in order to avoid existential risks; human control; and benefiting the “overall majority of mankind.” He also welcomed foreign participation in the Shanghai World AI Conference (WAIC) in July.
• Foreign Minister WANG Yi (王毅) highlighted AI safety and human control as one of “Three Ensures” in an interview at the yearly Two Sessions political gathering.
• Ensuring AI is a force for good;
• Ensuring AI safety, which includes ensuring human control, improving interpretability and predictability, and assessing risks;
• Ensuring fairness and setting up an international �AI governance institution under the UN.

42

Chinese companies joined international counterparts in drafting 2 international standards on AI safety and security.

International Governance

• In April, the World Digital Technology Academy (WDTA), an NGO established under the UN framework, released 2 new standards on generative AI application security testing and LLM security testing.
• Many actors from different countries wrote or reviewed the standards, including Western companies (Meta, Nvidia, Google, Anthropic, Microsoft, OpenAI), Western universities or public institutions (Georgetown, the US National Institute of Standards and Technology), and Chinese companies (Baidu, iFLYTEK, Ant Group, Tencent).
• The LLM security standard was primarily written by Ant Group employees.
• The generative AI standard included 5 tests for “excessive agency” to prevent “unintended consequences,” while the LLM standard focused on defense against adversarial attacks.

43

3.1 Multilateral Governance

​

3.2 Global South: In addition to these multilateral efforts, China also announced new efforts to expand AI cooperation with African countries.

​

3.3 Bilateral Governance

​

3.4 “Track 1.5” and “Track 2” dialogues

International Governance

44

China announced new projects on AI at the 2024 China–Africa Internet Development and Cooperation Forum, focusing on coordinating with Africa on global governance, with only a brief reference to AI safety topics.

International Governance

• The Cyberspace Administration of China (CAC) published a “Chair’s Statement on China–Africa Cooperation on AI” during the forum in April 2024, which focused on improving cooperation on AI development.
• The statement declared plans to create a China–Africa AI policy dialogue and cooperation mechanism, which could enable enhanced cooperation.
• It supported cooperation on AI research and �development (R&D), technology transfer, industrial �cooperation, digital infrastructure, and talent exchanges.
• It also called for cybersecurity and data security �safeguards, including preventing “abuse of AI technology �and cyber-attacks.”

45

3.1 Multilateral Governance

​

3.2 Global South

​

3.3 Bilateral Governance: China issued a joint statement on AI with France and is establishing a new AI-focused dialogue with the US.

​

3.4 “Track 1.5” and “Track 2” dialogues

International Governance

46

The Sino-French joint statement indicates both governments are prioritizing AI governance, increasing chances for further dialogue on AI and deeper Chinese participation in the 2025 French AI summit.

International Governance

• On May 6, during President Xi Jinping’s state visit to France, the 2 countries issued a joint statement (Ch, Fr) on AI and global governance.
• This was 1 of the 4 joint statements from the trip, which also led to signing of close to 20 bilateral cooperation documents.
• The statement noted mutual support for international efforts on AI development and safety, positively calling out the Bletchley Declaration and noting China’s �willingness to attend and assist in preparations for the French �AI Summit in 2025.
• Both countries also acknowledged AI’s opportunities and risks, �committing to deepen their discussion of international AI �governance models. They highlighted cooperation through �multilateral frameworks such as the UN High-Level Advisory �Body on AI and UNESCO recommendation on AI ethics.

47

Details about the China-US dialogue remain sparse, but there are hints that frontier AI safety will be on the agenda.

International Governance

• China and the US agreed to create a dialogue focused on AI in November 2023 during a meeting between President Xi and President Biden at the Asia-Pacific Economic Cooperation summit.
• The US readout noted that “the leaders affirmed the need to address the risks of advanced AI systems and improve AI safety.”
• The US Office of Science and Technology Policy Director called for working with China on global AI safety standards in January.
• During US Secretary of State Antony Blinken’s April�trip to China, he announced that talks on AI would �include “risks and safety concerns around �advanced AI and how best to manage them.” China �also noted that the AI dialogue would start soon as �part of “Five Points of Consensus” with the US.
• The first meeting will occur on May 14 in Geneva.

48

3.1 Multilateral Governance

​

3.2 Global South

​

3.3 Bilateral Governance

​

3.4 “Track 1.5” and “Track 2” dialogues: Track 1.5 and 2 dialogues between China and the West have increased over the last year, but there remain some gaps in the landscape.

​

International Governance

49

Frontier AI discussions are a growing but still minor fraction of overall dialogues, and some key stakeholder groups are underrepresented at present.

International Governance

• Concordia AI created a database of China–Western Track 1.5 and 2 Dialogues on AI and analyzed the features of the landscape.
• Of 8 AI-focused dialogues taking place since 2022, only 2 focused on frontier AI safety and governance. This is a small proportion of �dialogues, which number at least 40+ �just between the US and China.
• Dialogue participants are mainly �foreign policy and military experts, �with fewer academic scientists, �industry representatives, or experts �from other domains that intersect �with AI risks (e.g. biosecurity and �cybersecurity).

50

One frontier AI safety Track 2 dialogue between top Chinese and Western AI scientific and governance experts produced substantive joint declarations on AI safety.

International Governance

• The International Dialogue for AI Safety (IDAIS) has held 2 meetings in October 2023 and March 2024, convened by top AI scientists including Yoshua Bengio, Andrew Yao (姚期智), Stuart Russell, and ZHANG Ya-Qin (张亚勤); some of the same names also published a joint paper titled “Managing AI Risks in an Era of Rapid Progress” in October.
• Several Chinese industry representatives and top policy experts also attended the March 2024 meeting.
• Both meetings resulted in joint declarations �oriented around frontier AI safety risks such as �misinformation, misuse by terrorists for developing�weapons of mass destruction, and loss of control �of AI, which could risk human extinction.

51

• Agree on joint measures to mitigate frontier AI risks, such as making concrete progress on international AI safety standards and evaluations.
• Share ideas for domestic governance mechanisms.
• Accelerate progress on technical safety research through academic collaboration and greater international funding.
• Share benefits of AI widely, such as by ensuring underrepresented languages are included in new LLMs.

Some strategies Concordia AI has previously proposed for collaboration with China on international AI governance:

International Governance

52

Section 1: Introduction and scope

Section 2: Technical safety research

Section 3: International governance

Section 4: Domestic governance

Section 5: Lab and industry practices

Section 6: Expert views on AI risks

Section 7: Public opinion on AI

Section 8: Additional resources

Section 9: About us

Table of Contents

53

• Top national leaders are simultaneously promoting faster AI development and stronger safety/security, without apparent focus on frontier AI safety issues.
• China’s regulatory system for AI was already relatively mature by 2023. While that has not expanded since, other national policies are incorporating frontier AI concerns.
• China had created national science and technology (S&T) ethics reviews for AI in 2023, but there have been few recent updates.
• New domestic standards have been issued on AI safety and security. They mainly address content security concerns but also acknowledge frontier AI risks.
• 3 additional local governments have issued policies to promote AGI (artificial general intelligence) or large model development while including some safety provisions, following a similar policy from the Beijing Municipal Government in May 2023.

Overview of key developments since October 2023

Domestic Governance

54

4.1 Overarching national guidance: Top national leaders have not publicly prioritized AI safety any further, though experts have included provisions relevant to frontier safety in their drafts of the national AI law.

​

4.2 National regulations and policies

​

4.3 Science and technology ethics system

​

4.4 Voluntary standards

​

4.5 Local government action

Domestic Governance

55

• The 2024 Government Work Report (En, Ch), issued at the Two Sessions (China’s premier annual political gathering), included a new “AI+” initiative focused on applications, but had no mentions of AGI, large models, or fundamental AI research.
• Around the Two Sessions, separate field visits to AI labs by Premier Li Qiang and the head of China’s macroeconomic planner (NDRC) showed increased interest in frontier AI.
• Public materials on the visits only included a brief reference to AI safety in a presentation slide by the Beijing Academy of AI (BAAI) for Premier Li.
• One other mention of safety was in an interview on the sidelines of the Two Sessions with Foreign Minister Wang Yi. While discussing AI governance, Minister Wang referenced the need to ensure human control of AI.

The 2024 Government Work Report and field investigations by national leaders reveal interest in frontier AI development and AI-driven applications, but little focus on safety.

Domestic Governance

56

• MOST Minister YIN Hejun’s (阴和俊) essay in a CAC-overseen magazine highlights the complexity and ambivalence of these views.
• The essay argues that AI is key to national power, as the “largest variable in the restructuring of overall national competitiveness and the new focus of global great power competition.”
• Yin called for improving the AI governance system under the idea that “development is the greatest security” and also to put “equal emphasis on development and governance.”
• At the same time, he supports promoting AI ethics and expanding international cooperation on AI governance.
• A separate sign of the government’s complex views on AI development and safety is the mid-2023 relaxation of interim regulations on generative AI after industry feedback on the restrictive first draft.
• The revisions sent a signal that the government supports capabilities development.
• However, the current Chinese regulations create meaningful compliance costs for companies not seen in most other countries. These regulations also allow regulators to experiment with tools that could be used to regulate frontier AI.

China does not view capabilities development and safety/security as zero-sum, simultaneously increasing efforts in both directions.

Domestic Governance

57

China is in the process of developing a national AI law, and 2 separate expert drafts have been released to date.

• After the national AI law was first announced in June 2023, it was not directly mentioned in two subsequent National People’s Congress (NPC) Standing Committee planning documents.
• However, the 2024 State Council legislative work plan issued in May listed the AI law as “under preparations” for submission to the NPC Standing Committee’s review. The NPC Standing Committee’s plan also noted that laws and regulations relating to AI were under preparation.
• Meanwhile, Chinese experts continue to draft their own suggested versions of the AI law, and the NPC Standing Committee recently held a seminar on AI with an expert calling for accelerated development of the AI Law.
• Experts from the Chinese Academy of Social Sciences (CASS) led drafting of a 1.0 version “Model Law” in August 2023 and a 2.0 version in April 2024.
• ZHANG Linghan (张凌寒), a member of the UN High-Level Advisory Body on AI and professor at China University of Political Science and Law (CUPL), led a separate “expert suggestion draft” published in March 2024. This draft was later discussed at a meeting attended by the NPC Legislative Affairs Commission, CAC, Ministry of Foreign Affairs (MOFA), and MOST.

Domestic Governance

58

Both expert drafts orient around promoting AI development and also contain provisions relevant to frontier AI safety.

Domestic Governance

59

4.1 Overarching national guidance

​

4.2 National regulations and policies: National scientific funding has begun devoting greater attention to AI safety, but no new regulations have emerged on AI safety.

​

4.3 Science and technology ethics system

​

4.4 Voluntary standards

​

4.5 Local government action

Domestic Governance

60

Over the past 6 months, China has not issued any new binding regulations relating to frontier AI.

• Generative AI applications in China, including leading LLMs like Baidu’s ERNIE Bot and Zhipu AI’s ChatGLM families, continue to be governed by a security review and government registration system.
• Under the existing regulatory regime, at least 117 generative AI products and over 900 deep synthesis algorithms have been registered with government authorities since August 2023.

Domestic Governance

61

The National Natural Science Foundation of China (NSFC) announced that it is accepting applications for the first projects on value alignment.

Domestic Governance

62

Chinese national security officials and organizations have become more publicly vocal about AI’s threats to national security, including brief references to AI safety risks.

• An engineer in the Central Military Commission Political and Legal Affairs Committee stated that loss of control of AI could be an existential risk for humanity, in an article mainly discussing generative AI’s threat to political security, military security, cybersecurity, and economic security.
• The Minister of the Ministry of State Security (MSS) and the MSS official WeChat account have both written on AI, primarily discussing AI security issues, but also with references to AI cyberattacks and data poisoning that fall within our scope of AI safety.
• The MSS Minister argued in September 2023 that generative AI such as ChatGPT “is frequently an important tool for cognitive and public opinion warfare.”
• An MSS WeChat post in November was focused on AI’s national security challenges, discussing “data theft,” “cyberattacks,” “economic security,” “data poisoning,” and “military security.”
• An MSS WeChat post in January 2024 listed AI alongside the quantum, space, deep sea, and biological domains as areas of “non-traditional security,” a term which suggests increased interest in international cooperation.

Domestic Governance

63

A government-affiliated think tank discussed AI risks and recommended value alignment.

• The China Academy of Information and Communications Technology’s (CAICT) November 2023 Blue Paper on Large Model Governance noted the risk of large models causing catastrophic results from loss of control.
• While the report focused more on information security and fake information, large model robustness, interpretability, and loss of control were also discussed.
• The paper also supported using RLHF to pursue value alignment.
• CAICT is a Ministry of Industry and Information Technology (MIIT)-overseen public institution, and previous CAICT leaders have become government officials.

Domestic Governance

64

4.1 Overarching national guidance

​

4.2 National regulations and policies

​

4.3 Science and technology ethics system: There have been no policy updates on S&T ethics reviews, and little new information has emerged on how these are operationalized within companies and research institutions.

​

4.4 Voluntary standards

​

4.5 Local government action

Domestic Governance

65

4.1 Overarching national guidance

​

4.2 National regulations and policies

​

4.3 Science and technology ethics system

​

4.4 Voluntary standards: New standards have been issued on AI safety and security. They currently prioritize content security, but there is growing interest in frontier capabilities and safety testing.

​

4.5 Local government action

Domestic Governance

66

Government standards bodies have begun work on standards that could be relevant for frontier AI safety, and industry actors are also pursuing safety benchmarks.

• The Standardization Administration of China (SAC) and MIIT have both called for work on AI standards.
• In December 2023, SAC announced that work was beginning on 7 AI-related standards, including one on “Risk Management Capability Assessment” and one titled “Pretrained Model Part 2: Testing Indicators and Methods.” These could include provisions relevant to frontier AI safety.
• Industry associations, such as the AI Industry Alliance of China (AIIA) are pursuing their own standards related to AI safety, such as an AI Risk Management Framework and AI Safety Benchmark (see the Lab and Industry Practices section for more details).
• A laboratory under MIIT has also issued certifications to some leading AI developers including Zhipu AI and Baidu, primarily testing model capabilities, but a new round of evaluations dubbed “Fangsheng” will include testing for value alignment.

Domestic Governance

67

China finalized its first national standard on generative AI security in February, focused on content security with a brief mention of frontier safety risks.

• The generative AI security standard was issued by TC260, a standards body for cybersecurity under SAC, finalizing a draft from October 2023. It will likely guide implementation of security assessments required for generative AI under the July 2023 interim measures for generative AI.
• The document makes reference to “long-term” AI risks, such as deception, self-replication, use in cyberattacks or biological or chemical weapons, but has no concrete measures for these risks.
• The bulk of the standard focuses on content security concerns, such as corpus origin, corpus content, corpus watermarking, and model security (i.e. safety of content generated by the model).
• The standard sets concrete quantitative tests for compliance, such as testing at least 4,000 samples from the data corpus for compliance (requiring a 96% compliance rate).

Domestic Governance

68

4.1 Overarching national guidance

​

4.2 National regulations and policies

​

4.3 Science and technology ethics system

​

4.4 Voluntary standards

​

4.5 Local government action: Local government policies focused on AI development also touch on frontier safety issues, such as strengthening risk foresight, safety testing for models, and promoting model alignment.

Domestic Governance

Most of the key provincial-level jurisdictions for AI have released policies on AGI or large models.

69

• China’s 3 economic megaregions, Beijing-Tianjin-Hebei, Yangtze River Delta, and Greater Bay Area, are home to over 80% of China’s AI innovation and development and also feature recent local government policies on frontier AI.
• Beijing, Shanghai, and Guangdong (leading each of these respective regions) have all issued AGI or large model policies in the last year.
• Zhejiang, in the Yangtze River Delta region, issued an AI development policy in late 2023.
• Outside these regions, in the last year, Anhui has also issued an AGI policy, and Fujian has issued an AI development policy.
• These policies focus on development, but also contain AI safety-relevant measures that could foreshadow national policies, given China’s common practice of testing out policies first at the local level.

Domestic Governance

70

Testing of frontier AI safety measures in the provinces could inform and foreshadow future national actions.

Domestic Governance

71

Section 1: Introduction and scope

Section 2: Technical safety research

Section 3: International governance

Section 4: Domestic governance

Section 5: Lab and industry practices

Section 6: Expert views on AI risks

Section 7: Public opinion on AI

Section 8: Additional resources

Section 9: About us

Table of Contents

72

Overview of key developments since October 2023

Lab and industry practices

• Industry alliances, particularly the AI Industry Alliance of China (AIIA), have substantially increased interest in AI safety, including holding a seminar on AGI risks and pursuing concrete projects on evaluations and benchmarks.
• New models from SHLAB, Zhipu AI, and DeepSeek were accompanied by explanations of their respective safety practices, which include RLHF alignment that prioritizes human intentions and preventing damaging content without much attention to frontier safety risks.
• Corporate internal AI ethics and governance practices remain largely a black box. Nevertheless, reports by Tencent and Alibaba indicate growing understanding of frontier AI risks, and Ant Group claims to devote a substantial portion of AI R&D resources to ethics.

73

5.1 Industry alliance projects: At least 2 influential industry alliances are actively engaged in initiatives on AI safety, security, and governance.

​

5.2 Safety of published models

​

5.3 Corporate ethics and governance work

Lab and industry practices

74

AIIA and the Cyber Security Association of China (CSAC) are major government-backed players pursuing projects on AI safety, security, and governance.

Lab and industry practices

75

Thus far, AIIA’s Safety and Security Governance Committee has been the most active on frontier safety, though the Policy and Law working group has also shown interest.

Lab and industry practices

• AIIA is overseen by 4 central government departments and works closely with the government-affiliated CAICT think tank. The key committees or working groups for AI safety are:
• Safety and Security Governance Committee (安全治理委员会), announced in September 2023.
• The committee published an AI safety benchmark, which focuses more on issues of “content security” and “data security” and also tests for 2 aspects of AI “consciousness,” including AI “appealing for rights” and “anti-humanity tendencies.”
• The committee is also pursuing a standard on the safety of coding large models, working on an AI risk management framework, and exploring a project on alignment.
• Policy and Law working group (政策法规工作组), which dates from 2017.
• The Policy and Law working group held a meeting on AGI risks in January, showing new interest in frontier AI safety. It also seems to be participating in the CUPL-led AI Law expert draft, previously discussed here.
• Science and Technology Ethics working group (科技伦理工作组), announced in �December 2023.

76

Meanwhile, CSAC has been focused on corpus development, safety or security testing, and multimodal AI.

Lab and industry practices

• CSAC is overseen by the CAC, affording it a close relationship with regulators.
• CSAC established an AI Safety and Security Governance Expert Committee in October 2023, which is led by the deputy director of the National Computer Network Emergency Response Technical Team/Coordination Center of China (CNCERT/CC), also under CAC.
• The committee has released a Chinese basic corpus and conducted unspecified safety/security evaluation work.
• Given CSAC’s role under CAC and the participation of CNCERT/CC, this group seems poised to focus mainly on content security and cybersecurity issues, but many details have not yet been released.

77

5.1 Industry alliance projects

​

5.2 Safety of published models: Over the past 6 months, 3 additional labs released details about safety measures for models they published, but they appear to have taken little action on frontier AI safety.

​

5.3 Corporate ethics and governance work

Lab and industry practices

78

SHLAB, Zhipu AI, and DeepSeek’s disclosures reveal some efforts to align models to human intentions and prevent toxic content, but not testing for more frontier risks.

Lab and industry practices

• SHLAB claimed to apply a novel RLHF approach on InternLM2 to reduce reward hacking.
• However, the technical report does not indicate how SHLAB measures success in reducing reward hacking, and SHLAB’s benchmarks test primarily against performance, without testing against safety benchmarks.
• Zhipu AI released a paper on their use of RLHF methods in the ChatGLM family, but primarily focus on intent alignment.
• Their definition of safety focuses on harmful content, toxic content, and content that could provoke controversy, rather than frontier safety issues.
• Zhipu AI Chief Scientist TANG Jie (唐杰) stated that he is pursuing work on “superalignment” to ensure that AI will be aligned with human values and can conduct self-reflection, but Zhipu has not yet released any public research papers on superalignment.
• DeepSeek’s technical paper for the DeepSeek-V2 model claims that DeepSeek’s ultimately objective is alignment with human values. However, concrete safety measures in their model are lacking.
• The paper does not discuss testing alignment against any safety benchmarks.

79

5.1 Industry alliance projects

​

5.2 Safety of published models

​

5.3 Corporate ethics and governance work: Details about how companies implement AI ethics and governance measures are largely unknown, though Ant Group asserts it has made significant investments. Other companies have produced reports analyzing frontier AI risks.

Lab and industry practices

80

Ant Group claims that 20% of its large model technical personnel work on S&T ethics, but this is difficult to verify.

Lab and industry practices

• Ant Group Senior Vice President and Chairman of the Technology Strategy Committee NI Xingjun (倪行军) claimed in December 2023 that:
• Nearly 20% of the large model technical team works on ethics construction.
• Ant Group has invested human resources and compute into creating risk assessment and defense mechanisms for large models.
• While it is not currently possible to independently verify these claims, this may indicate that Chinese companies are incentivized to improve safety practices.
• Ant Group’s substantial participation in a WDTA standard on LLM security testing released in April – all of the “Lead Authors” listed were from Ant Group – does partially back up their claims.

81

Recent reports by commercial actors have also begun to discuss frontier AI risks in greater sophistication and lay out company efforts to combat such risks.

Lab and industry practices

• Alibaba and Tencent published reports in December and January that substantially explored frontier AI safety and governance issues. We were not aware of any major reports from labs discussing the issue previously.
• Alibaba’s paper discussed robustness, embedding human values, �and watermarking mechanisms.
• Tencent’s paper had a full chapter on large model value alignment, �noting OpenAI’s allocation of resources to superalignment and �scalable oversight research.
• Baidu’s security/safety team wrote an article on red-teaming in April.
• The article frames red-teaming primarily in terms of content�security, referencing China’s regulations and standards on �generative AI.
• It also discusses preventing jailbreak attacks and GPT-4’s red-�teaming against alignment, disinformation, and biological misuse.

82

Section 1: Introduction and scope

Section 2: Technical safety research

Section 3: International governance

Section 4: Domestic governance

Section 5: Lab and industry practices

Section 6: Expert views on AI risks

Section 7: Public opinion on AI

Section 8: Additional resources

Section 9: About us

Table of Contents

83

Overview of key developments since October 2023

Expert Views

• A small but influential group of Chinese experts in dialogue with foreign scholars came to a consensus on AI “red lines” that must not be crossed in order to avoid existential risks.
• The idea of devoting a minimum level of AI R&D funding to safety, governance, or ethics, which previously was essentially absent from Chinese discourse, has gained support, particularly within this small set of influential experts.
• Multiple experts have begun to write on the risks of AI for biological security for the first time.
• While many previous discussions of frontier AI risks occurred at leading AI conferences, now some experts are discussing these risks in venues directed towards government and party officials.

84

6.1 International coordination: Top Chinese and foreign experts have signed a consensus statement on key aspects of frontier AI risks, policy recommendations, and red lines in a recent dialogue.

​

6.2 R&D funding devoted to AI safety

​

6.3 AI and biological security

​

6.4 Discussion in party venues

Expert Views

85

The 2 IDAIS meetings show that a number of influential Chinese and Western experts agree on measures for ensuring safety of frontier AI models.

Expert Views

• For more on IDAIS, see the International Governance section.
• Joint policy recommendations include developing “red lines,” mandating registration of models above a certain capability, and increasing funding for AI safety and governance research.
• Key signatories included former Vice Minister of Foreign Affairs FU Ying (傅莹), Tsinghua Institute for AI International Governance (I-AIIG) Dean XUE Lan (薛澜), BAAI leadership, Zhipu AI CEO ZHANG Peng (张鹏), and ByteDance Head of Research LI Hang (李航), who signed in a personal capacity.
• The five red lines agreed upon in Beijing were:
• autonomous replication or improvement;
• power seeking;
• assisting weapon development;
• cyberattacks;
• deception.

86

6.1 International coordination

​

6.2 R&D funding devoted to AI safety: The idea of devoting a minimum level of national and corporate R&D funding to AI safety or governance research has received some attention and support in Chinese domestic discourse.

​

6.3 AI and biological security

​

6.4 Discussion in party venues

Expert Views

87

Kai-Fu Lee and several other leading Chinese AI experts expressed support for minimum funding or resourcing levels for AI safety.

Expert Views

• Both IDAIS readouts and the “Managing AI Risks” paper called for a minimum of one-third of corporate and government AI R&D funds to be spent on AI safety and governance.
• Investor and 01.AI founder Kai-Fu Lee (李开复), Tsinghua dean and former Baidu President ZHANG Ya-Qin (张亚勤), and Founding Chairman of BAAI ZHANG Hongjiang (张宏江) all support companies allocating a minimum level of staff or funding for AI safety issues, listing figures between 10% and 20%.
• The Senior Vice President of Ant Group NI Xingjun (倪行军) claims that nearly 20% of technical personnel in Ant’s large model team already �work on ethics issues.
• The CASS AI model law suggests providing�tax credits for safety and governance work �by AI developers and providers.

88

6.1 International coordination

​

6.2 R&D funding devoted to AI safety

​

6.3 AI and biological security: There is nascent discussion in policy advisory circles about the risks of AI combined with biological risks.

​

6.4 Discussion in party venues

Expert Views

89

While these discussions are nascent, all actors who have weighed in are influential policy advisors.

Expert Views

• Tianjin University Center for Biosafety Research and Strategy Director ZHANG Weiwen (张卫文) said in October 2023 that it is important to develop talent to manage risks from AI combined with synthetic biology.
• The Tianjin center is perhaps China’s foremost university center researching biosafety and security.
• A researcher at the Development Research Center of the State Council (DRC) discussed biosecurity risks from LLMs and biological design tools in a January 2024 article.
• DRC is a think tank subordinated to China’s cabinet, the State Council.
• A CAICT report cited the interim report of the UN High-Level Advisory Body on AI, including references to AI’s chemical and biological risks, as well as the possibility of replacement of human values and knowledge.
• CAICT is a public institution overseen by MIIT.

90

6.1 International coordination

​

6.2 R&D funding devoted to AI safety

​

6.3 AI and biological security

​

6.4 Discussion in party venues: Warnings regarding the potential risks of frontier AI have also begun to arise in venues directed more towards party elites than scientific audiences.

Expert Views

91

2 leading experts discussed frontier AI safety risks in notable party venues in recent months.

Expert Views

• Academician GAO Wen (高文) penned 2 articles for newspapers under the Central Party School and the Communist Party of China (CPC) Publicity Department in November and December.
• Gao had previously given a presentation on AI to President Xi and other top leaders in 2018.
• In the new articles, Gao noted the risk of AGI leading to “extinction of humanity” and calls for ensuring safety, controllability, and alignment of AI.
• Beijing Institute for General Artificial Intelligence (BIGAI) director ZHU Songchun (朱松纯) also mentioned AGI risks in a speech for delegates to the top national political advisory body, the China People’s Political Consultative Conference (CPPCC).
• Zhu discussed paths to realizing AGI, China’s �competitive advantages, and risks of loss of control.
• He also called for giving AGI a value system and �cognitive structure, creating a “heart” in the machine, �so that it is aligned with human values and norms.

92

Section 1: Introduction and scope

Section 2: Technical safety research

Section 3: International governance

Section 4: Domestic governance

Section 5: Lab and industry practices

Section 6: Expert views on AI risks

Section 7: Public opinion on AI

Section 8: Additional resources

Section 9: About us

Table of Contents

93

Overview of key developments since October 2023

Public Opinion

• Polls remain of limited quality. None have fully representative samples, and the ones that tackle frontier AI safety questions are not at all representative.
• The public still seem* to view benefits of AI as outweighing the risks.
• The public still seem* to think that frontier AI development could cause human extinction, but also seem to think that the risks are controllable.

* These conclusions should be treated with caution due to the lack of representative polls.

94

There was only one new relevant poll over the past six months, which did not yield any clarifying results.

Public Opinion

• The poll was conducted in early March 2024 by The Paper (澎湃新闻).
• The Paper is a state-backed Chinese media outlet, notable in the past for investigative work.
• The sample in this poll was not representative of the population, with over two-thirds of respondents under the age of 35.
• The poll found that 63% of respondents agree that AI’s continued development might lead to loss of control. However, it is not clear what was meant by “loss of control,” and the poll did not ask how respondents would balance the benefits and dangers of AI.

95

Section 1: Introduction and scope

Section 2: Technical safety research

Section 3: International governance

Section 4: Domestic governance

Section 5: Lab and industry practices

Section 6: Expert views on AI risks

Section 7: Public opinion on AI

Section 8: Additional resources

Section 9: About us

Table of Contents

96

Thank you for reading our report!

Additional Resources

• We appreciate your support and welcome questions, feedback, and other follow-up engagement. Feel free to reach out to us directly at info@concordia-ai.com.
• In addition to the report, we have compiled several running databases that may be helpful for researchers of China and AI safety.
• Appendix A: China’s AI Governance Documents, a running list including both domestic and international governance documents.
• Appendix B: Chinese Technical AI Safety Database, with a list of Chinese Frontier AI Safety Papers and Key Chinese AI Safety-relevant Research Groups.

97

Key acronyms (1)

Additional Resources

98

Key acronyms (2)

Additional Resources

99

Key acronyms (3)

Additional Resources

100

Key acronyms (4)

Additional Resources

101

Section 1: Introduction and scope

Section 2: Technical safety research

Section 3: International governance

Section 4: Domestic governance

Section 5: Lab and industry practices

Section 6: Expert views on AI risks

Section 7: Public opinion on AI

Section 8: Additional resources

Section 9: About us

Table of Contents

102

About Concordia AI (安远AI)

About Us

• Concordia AI is a certified social enterprise based in Beijing, the only social enterprise in China focused on AI safety and governance.
• Controlling and steering increasingly advanced AI systems is a critical challenge for our time.
• Concordia AI aims to ensure that AI is developed and deployed in a way that is safe and aligned with global interests.

103

• Attended the Global AI Safety Summit at Bletchley Park.
• Launched and published the bi-weekly AI Safety in China Newsletter, with over 700 subscribers from AI labs, governments, think tanks, and media publications.

We have 3 main areas of work. See our 2023 Annual Review for more details.

About Us

Focus 2:

Technical AI safety field-building in China

Focus 1:

Advising on Chinese AI safety and governance

Focus 3:

Promoting international cooperation

• Selected as deputy chief expert of AI Safety Governance Committee in China’s Artificial Intelligence Industry Alliance.
• Co-authored report “Responsible Open-Sourcing of Foundation Models.”

• Co-hosted a full-day forum on AI Safety and Alignment during the Beijing Academy of AI (BAAI) conference in June 2023.
• Organized the first AI Safety and Alignment Fellowship program in China.

104

Conflicts of interest

About us

• Concordia AI is an independent institution, not affiliated to or funded by any government or political group.
• Concordia AI actively participates in and advises on AI safety within China through various channels, including hosting forums, organizing lectures, and advising on policy.
• Our work in this field places us in a unique position to understand and analyze information regarding the state of AI safety in China.
• In the course of operations, we have received consulting fees from various companies in mainland China, Hong Kong, and Singapore.
• Nevertheless, we believe our findings are the result of objective analysis, and we disclose this potential conflict to readers for full transparency. No financial engagement with these companies was related to this report’s creation.

105

About Us

Email

info@concordia-ai.com

Substack

aisafetychina.substack.com

Follow our work through our Substack newsletter, translations of AI expert views, and more!

106

This report was authored by Jason Zhou, Kwan Yee Ng, and Brian Tse.

​

We would like to express our sincere gratitude to the entire Concordia AI team for their tireless contributions throughout the development of this report. Their constructive feedback and dedicated analytical support were instrumental in shaping the content and ensuring the quality of our work. We are also deeply indebted to our network of affiliates and collaborators for multiple rounds of meticulous review of various sections of the report. Their assistance in curating our database of frontier AI safety papers was an essential foundation for our research. We additionally thank our expert reviewers for their suggestions on improving the methodology for the technical AI papers database and other valuable feedback.

Acknowledgements

About Us

© 2024 Concordia AI, all rights reserved.