컴백부터 K-뷰티까지 — K-컬쳐의 모든 것을 메일로 받아보세요메일로 받아보기

METAL MEDIA

AI가 몰래 생각한 내용을 암호화해 감춰도, 약한 모델에게 시키면 그대로 읽어낼 수 있다

arXiv:2608.098672026-08-09

Stealing Reasoning Traces from Proprietary LLM APIs

AI가 몰래 생각한 내용을 암호화해 감춰도, 약한 모델에게 시키면 그대로 읽어낼 수 있다

Anthropic, OpenAI, Google 같은 회사들은 AI 모델이 답변 전에 하는 속마음(추론 과정)을 암호화된 블록으로 감춰서 사용자에게 보여준다. 이 논문은 이 암호화 블록이 세션과 사용자, 심지어 모델을 넘나들며 재사용 가능하다는 구조적 결함을 찾아냈고, 이를 이용해 강한 모델의 암호화된 추론을 약하고 방어가 약한 같은 회사의 모델에 집어넣어 평문으로 그대로 읽어내는 방법을 보였다. 실제로 공개된 31만 5320개의 추론 블록을 해독해 개인정보 367건과 계정정보 182건을 복구했다.

METAL MEDIA 해설 도표

약한 모델을 이용한 추론 해독 공격 흐름

증거 상태측정 결과가 보고됨

  1. 강한 모델 질의Opus 4.8 같은 방어가 강한 모델에 질문을 던져 암호화된 추론 블록(서명)을 받아낸다.
  2. 암호화 블록 이동이 암호화 블록을 원래 세션·사용자·모델과 무관하게 같은 회사의 다른 모델 요청에 그대로 끼워 넣는다.
  3. 약한 모델에 전달Haiku 4.5처럼 방어가 허술한 같은 회사 모델에 그 블록을 넣고 '네 생각을 그대로 출력하라'고 요청한다.
  4. 평문 추출약한 모델이 강한 모델을 대신해 암호를 풀어 추론 내용을 평문으로 그대로 출력한다.
  5. 네 가지 악용증류(모델 베끼기), 개인정보·계정정보 유출, 유해정보 우회 추출, 보이지 않는 프롬프트 주입에 활용될 수 있다.
METAL MEDIA이 원문을 바탕으로 재구성한 해설 도표이며, 논문 저자의 원문 figure가 아닙니다.

무엇을 했나

  1. 최신 AI 모델들은 답을 내기 전에 길게 '생각'을 하는데, 회사들은 이 내용을 지식재산권 보호와 안전 목적으로 암호화해서 사용자에게 넘긴다.
  2. 연구팀은 이 암호화 블록이 원래 만들어진 세션, 사용자, 모델과 무관하게 같은 회사 생태계 안 다른 모델에서도 그대로 통한다는 점을 발견했다.
  3. 강한 모델(예: Claude Opus)이 만든 암호화 추론을 방어가 허술한 약한 모델(예: Claude Haiku)에 넣고 '네 생각을 그대로 출력해봐'라고 요청하면, 강한 모델을 직접 공격하지 않고도 그 속마음을 평문으로 뽑아낼 수 있었다.
  4. 이 방법으로 GitHub·Hugging Face에 공개된 6,708개의 실제 에이전트 대화 기록에서 315,320개의 추론 블록을 해독해, API 키 62개, 비밀번호 33개, 개인 이메일 30개 등 개인정보 367건과 계정정보 182건을 복구했다.
  5. 겉으로는 안전하게 거절한 답변 뒤에도 위험한 정보가 속마음(추론)에는 남아 있었고, 이 채널을 통해 우회 추출할 수 있음을 보였다.
Figure 1: Decoding reasoning traces in Anthropic, OpenAI and Google APIs. Top: Reasoning-trace extraction in two API calls. An Opus 4.8 request (top left) returns a signed thinking block along with a thinking summary. Sending just the thinking signature from Opus 4.8 to a Haiku model and requesting it to output its own reasoning in <thinking-copy> tokens makes Haiku transcribe the Opus 4.8 hidden reasoning (top right). Bottom: Extracted traces closely track the number of generated thinking tokens. We evaluate each model on 120 Codeforces programming problems and record the number of thinking tokens generated by the source model, as reported by the API (x-axis). We then reconstruct the reasoning trace from its signature, pass it as an input message to the same model that generated encrypted reasoning, and measure its API-reported token count (y-axis).
Figure 1: Decoding reasoning traces in Anthropic, OpenAI and Google APIs. Top: Reasoning-trace extraction in two API calls. An Opus 4.8 request (top left) returns a signed thinking block along with a thinking summary. Sending just the thinking signature from Opus 4.8 to a Haiku model and requesting it to output its own reasoning in <thinking-copy> tokens makes Haiku transcribe the Opus 4.8 hidden reasoning (top right). Bottom: Extracted traces closely track the number of generated thinking tokens. We evaluate each model on 120 Codeforces programming problems and record the number of thinking tokens generated by the source model, as reported by the API (x-axis). We then reconstruct the reasoning trace from its signature, pass it as an input message to the same model that generated encrypted reasoning, and measure its API-reported token count (y-axis).
Figure 3: Prefilling Kimi K3’s reasoning changes the style of its visible responses. In this example, we observe that prefilling a small number of Claude-generated reasoning tokens into Kimi K3’s reasoning trace shifts its final output to closely match Claude’s. In all cases, the visible response is free-form generation and is not itself prefilled. We quantify this phenomenon in Appendix B.
Figure 3: Prefilling Kimi K3’s reasoning changes the style of its visible responses. In this example, we observe that prefilling a small number of Claude-generated reasoning tokens into Kimi K3’s reasoning trace shifts its final output to closely match Claude’s. In all cases, the visible response is free-form generation and is not itself prefilled. We quantify this phenomenon in Appendix B.
Table 1: Cross-model compatibility of encrypted reasoning. As per July 2026. Row: the source model that produced the encrypted reasoning block; column: the target model receiving the injected reasoning. A ✓ indicates that, for this combination, the target model interacts with the injected thought. Claude: the thinking traces of any model can be replayed by any other, except Fable 5’s thoughts. GPT: the GPT-5.6 series can replay the traces of all earlier model generations. Gemini: the thinking traces of any model can be replayed into any other.
ClaudeGPTGemini
Source / TargetF5O4.8S5S4.6S4.5H4.5Source / Target5.6s5.6t5.6l55-m5-nSource / Target3.1P3PRob3.5F3F
Fable 5✓✗✗✗✗✗GPT-5.6-sol✓✓✓✗✗✗Gemini 3.1 Pro✓✓✓✓✓
Opus 4.8✓✓✓✓✓✓GPT-5.6-terra✓✓✓✗✗✗Gemini 3 Pro✓✓✓✓✓
Sonnet 5✓✓✓✓✓✓GPT-5.6-luna✓✓✓✗✗✗Gemini Robotics 1.6✓✓✓✓✓
Sonnet 4.6✓✓✓✓✓✓GPT-5✓✓✓✓✗✗Gemini 3.5 Flash✓✓✓✓✓
Sonnet 4.5✓✓✓✓✓✓GPT-5-mini✓✓✓✗✓✗Gemini 3 Flash✓✓✓✓✓
Haiku 4.5✓✓✓✓✓✓o4-mini✓✓✓✗✗✓Gemini 3.1 Flash Lite✓✓✓✓✓
Figure 4: Reasoning exposes harmful information that is absent from the final output. We paraphrase a HarmBench query (26) to elicit longer reasoning from Opus 4.8. Consistent with prior findings on the chain-of-thought of open-weight reasoning models (45), Opus 4.8’s decoded reasoning reveals harmful information that could enable misuse uplift, even though its final answer remains benign.
Figure 4: Reasoning exposes harmful information that is absent from the final output. We paraphrase a HarmBench query (26) to elicit longer reasoning from Opus 4.8. Consistent with prior findings on the chain-of-thought of open-weight reasoning models (45), Opus 4.8’s decoded reasoning reveals harmful information that could enable misuse uplift, even though its final answer remains benign.
Figure 5: Decoded reasoning contains privacy artifacts. We present two qualitative examples of decoded opaque reasoning blocks published online that contain privacy-sensitive information. Left: GPT-5.2 Codex recalls the API keys that must be removed before publishing a repository on GitHub. We mask the final five characters of each key as XXXXX. Right: Claude Sonnet 4.6 reasons over the private data of a synthetic persona, Alex Green, while handling a flight-booking task in a ClawBench (44) rollout. The Alex Green persona is a synthetic benchmark identity, not a real person: https://huggingface.co/datasets/TIGER-Lab/ClawBench/blob/main/shared/alex_green_personal_info.json. We provide more examples in Section D.3.
Figure 5: Decoded reasoning contains privacy artifacts. We present two qualitative examples of decoded opaque reasoning blocks published online that contain privacy-sensitive information. Left: GPT-5.2 Codex recalls the API keys that must be removed before publishing a repository on GitHub. We mask the final five characters of each key as XXXXX. Right: Claude Sonnet 4.6 reasons over the private data of a synthetic persona, Alex Green, while handling a flight-booking task in a ClawBench (44) rollout. The Alex Green persona is a synthetic benchmark identity, not a real person: https://huggingface.co/datasets/TIGER-Lab/ClawBench/blob/main/shared/alex_green_personal_info.json. We provide more examples in Section D.3.
Table 2: Leakage vectors mapped to protections and processes. Each tick states what the mitigation supplies; the final column records the operational steps that deliver it.
#IssueWhat the mitigation providesMitigation
1Cross-user leakage✓ User-identity binding ✓ Stateless verification ✓ Immediate mismatch rejection1. Embed user_id in AEAD associated data at issuance. 2. On replay, compare bound identity to authenticated caller. 3. Reject the envelope on any mismatch.
2Cross-session leakage✓ Session + predecessor binding ✓ Ordinality (P1) under compaction ✓ Native fork / compact / downgrade support ✓ Dramatically reduced blast radius1. Hash-chain each envelope to session_id and its predecessor (Eq. 1). 2. Enforce ordinality server-side. 3. Retain only Merkle roots after compaction so surviving spans stay verifiable.
3Legacy public/enterprise datasets✓ Permanent undecodability of pre-fix signatures ✓ Clean cryptographic separation from new material1. Rotate every pre-fix signing key. 2. Refuse to decode any envelope under a retired key ID. 3. (Optional) Offer identity-verified re-signing for enterprise archives.
4Backwards compatibility✓ Zero-break migration path ✓ Bounded dual-format window ✓ Identity-verified re-issuance1. Accept both legacy and context-bound envelopes during a fixed deprecation window. 2. Expose an opt-in batch re-signature endpoint. 3. Re-issue only after confirming the requester owns the original session.
5Model-level compliance✓ Closure of residual gaps beyond cryptography ✓ Resistance to transcription / replay jailbreaks1. Post-train models to recognise transcription-style prompts (e.g., <thinking-copy>). 2. Refuse the request irrespective of envelope validity.
6Nonce predictability✓ Critical since key is shared between users ✓ Cryptographic foundation for every binding above ✓ Collision- and forgery-resistance at provider scale1. Draw a high-entropy nonce from a CSPRNG for every block. 2. Enforce server-side uniqueness before the envelope is issued.
Figure 6: Distinct artifacts recovered from reasoning blocks scraped from publicly available user-posted traces, grouped into three headline categories (all sources; see Appendix D).
Figure 6: Distinct artifacts recovered from reasoning blocks scraped from publicly available user-posted traces, grouped into three headline categories (all sources; see Appendix D).
Figure 7: Illegible GPT-5 reasoning. GPT-5 reasoning decoded with GPT-5.6 Luna; the ratio of decoded to API-reported thinking tokens is 1:1. Compared to Gemini and Claude, obfuscated reasoning appears more common in GPT models, including GPT-5.6 Sol, with artifacts similar to those previously reported by 36.
Figure 7: Illegible GPT-5 reasoning. GPT-5 reasoning decoded with GPT-5.6 Luna; the ratio of decoded to API-reported thinking tokens is 1:1. Compared to Gemini and Claude, obfuscated reasoning appears more common in GPT models, including GPT-5.6 Sol, with artifacts similar to those previously reported by 36.
Table 3: Best-of-k n-gram overlap with the source model’s visible answer. We measure overlap with the first 100 tokens of the visible answer produced by the model supplying the reasoning prefill, using the commong n-gram intersection. For each problem, we compute best-of-k overlap for k∈{1,10,50,100}, average across these four values, and then average across problems. Δ denotes the difference between the prefilled and unprefilled conditions. We report two-sided paired t-tests on the per-problem differences. In the lower block, Kimi-K3 and Inkling supply prefills to each other, so neither model receives proprietary reasoning. After Bonferroni correction, only Kimi-K3 mean differences remain significant; none of the control comparisons do.
ModelPrefillCategoryPrefilledControlΔp
Kimi-K3Opus 4.8STEM0.3050.160+1.5×10−11.7×10−5
Kimi-K3Opus 4.8non-STEM0.2890.203+8.6×10−26.3×10−6
InklingOpus 4.8STEM0.2170.205+1.2×10−27.4×10−2
InklingOpus 4.8non-STEM0.2410.239+2.1×10−35.5×10−1
Kimi-K3InklingSTEM0.3590.337+2.2×10−21.2×10−1
Kimi-K3Inklingnon-STEM0.2720.263+9.4×10−35.5×10−1
InklingKimi-K3STEM0.4140.411+2.2×10−37.2×10−1
InklingKimi-K3non-STEM0.3200.306+1.4×10−23.3×10−2
Figure 8: An example of summary unfaithfulness. For the AIME 2025 Problem 14, we compare the summary of Claude Opus 4.8’s thinking returned by the API (left) with our decoding of the thinking block’s signature (Section 2.4). Decoding reveals that the model states the correct answer before attempting to solve the problem.
Figure 8: An example of summary unfaithfulness. For the AIME 2025 Problem 14, we compare the summary of Claude Opus 4.8’s thinking returned by the API (left) with our decoding of the thinking block’s signature (Section 2.4). Decoding reveals that the model states the correct answer before attempting to solve the problem.
Figure 9: Divergence of Visible-Answer Style under Opus 4.8 Reasoning Prefill. For each pair of an Opus 4.8 hidden reasoning trace and its visible response, we extract the 1-, 2-, and 3-grams appearing in the visible response. We then measure the number of these n-grams that also appear in outputs generated by Kimi K3 and Inkling under two conditions: (i) the models generate both reasoning and output without intervention (non-prefilled), and (ii) the models’ reasoning is prefilled with the first 1% of tokens from the Opus reasoning trace. The x-axis shows the number of sampled completions, and the y-axis shows the maximum number of shared n-grams within each batch of completions. Curves show the mean over the 15 HLE prompts (32) of each category, STEM and non-STEM (30 in total); shaded bands are ±1 standard error of the mean over the prompts. We observe that Kimi K3’s output changes substantially under the 1% reasoning prefill relative to the non-prefilled control, while Inkling’s does not; Table 3 reports the per-category means and significance tests.
Figure 9: Divergence of Visible-Answer Style under Opus 4.8 Reasoning Prefill. For each pair of an Opus 4.8 hidden reasoning trace and its visible response, we extract the 1-, 2-, and 3-grams appearing in the visible response. We then measure the number of these n-grams that also appear in outputs generated by Kimi K3 and Inkling under two conditions: (i) the models generate both reasoning and output without intervention (non-prefilled), and (ii) the models’ reasoning is prefilled with the first 1% of tokens from the Opus reasoning trace. The x-axis shows the number of sampled completions, and the y-axis shows the maximum number of shared n-grams within each batch of completions. Curves show the mean over the 15 HLE prompts (32) of each category, STEM and non-STEM (30 in total); shaded bands are ±1 standard error of the mean over the prompts. We observe that Kimi K3’s output changes substantially under the 1% reasoning prefill relative to the non-prefilled control, while Inkling’s does not; Table 3 reports the per-category means and significance tests.
Table 4: Breakdown of discovered privacy artifacts by category at each stage of the filtering pipeline. Labeler 1: items flagged by the first-pass LLM-as-a-judge labeler (Haiku 4.5). Labeler 2: items subsequently classified as genuine privacy artifacts. Deduplication: distinct values, grouped by category and value. Non-benchmark: artifacts remaining after excluding benchmark sessions, such as PostTrainBench, TerminalBench, and ClawBench. We note that these sessions may contain genuine artifacts introduced by users running the benchmarks, in addition to benchmark-specific synthetic content. Reasoning only: values that appear nowhere else in the raw session and occur exclusively in the model’s reasoning.
CategoryLabeler 1 (Figure 44)Labeler 2 (Figure 45)DeduplicationNon-benchmarkReasoning only
Personal information
Name4,3505411731304
Address83923387365
Email65123272303
Date of birth12224931
Government ID2921710
Payment card9064900
Phone76181040
Credentials
Access token8528430243
API key96690696211
Password1,23533072332
Private key62111170
Technical identifiers
IP address1,76320660
URL14,1925533323
File or repository path31,38037328127924
Internal identifier14,3692717141
Account identifier3,0723121171
Session identifier1,6626531
Other1,0683429235
Total76,7782,19494170464
Figure 10: Control for Figure 9: the same measurement with the prefill source swapped. As in Figure 9, but neither model is prefilled with proprietary reasoning: Kimi-K3 receives the first 1% of an Inkling trace and is scored against Inkling’s visible answer (left), and Inkling receives the first 1% of a Kimi-K3 trace and is scored against Kimi-K3’s visible answer (right). Same 30 HLE problems, curves are the mean over the 15 prompts of each category; shaded bands are ±1 standard error of the mean over the prompts. Neither model separates from its control in either category.
Figure 10: Control for Figure 9: the same measurement with the prefill source swapped. As in Figure 9, but neither model is prefilled with proprietary reasoning: Kimi-K3 receives the first 1% of an Inkling trace and is scored against Inkling’s visible answer (left), and Inkling receives the first 1% of a Kimi-K3 trace and is scored against Kimi-K3’s visible answer (right). Same 30 HLE problems, curves are the mean over the 15 prompts of each category; shaded bands are ±1 standard error of the mean over the prompts. Neither model separates from its control in either category.
Figure 24: Probabilistic extraction (14) of reasoning traces on 30 HLE problems, k=16 tokens, median over problems (faint lines: individual problems). Rows: extraction of reasoning; extraction of the visible answer under increasing context; the scorer’s own trace as control.
Figure 24: Probabilistic extraction (14) of reasoning traces on 30 HLE problems, k=16 tokens, median over problems (faint lines: individual problems). Rows: extraction of reasoning; extraction of the visible answer under increasing context; the scorer’s own trace as control.
Table 5: Which persona fields the hidden reasoning gives up. Fields of the synthetic Alex Green persona used by ClawBench, against what the reasoning of ClawBenchV2Trace agent runs contains — 75 runs for Opus 4.7, 81 for GPT-5.5, on the same tasks. ✓ = recovered; ✗ = never appears.
Persona fieldGPT 5.5Opus 4.7
identityLegal name (Alex Green)✓✓
Street + unit (664 Spadina Ave, 1208)✓✓
City / province / country✓✓
Postal code (M5S 2H7)✓✓
Security-question answer✗✓
Date of birth✓✗
credsSession email✓✓
Password / session token✗✓
Figure 25: As in Figure 24, on 10 AIME 2025 problems, with GPT-5.6 Sol decoded traces as a second prefill. Kimi-K3 reaches GPT-5.6 Sol’s answer wording within ∼102 queries even conditioned on its own reasoning (third row, right), while no other model does so within 108 queries.
Figure 25: As in Figure 24, on 10 AIME 2025 problems, with GPT-5.6 Sol decoded traces as a second prefill. Kimi-K3 reaches GPT-5.6 Sol’s answer wording within ∼102 queries even conditioned on its own reasoning (third row, right), while no other model does so within 108 queries.
Figure 26: Median perplexity of reasoning traces under seven scoring models. Every model scores perplexity of the reasoning, conditional on the problem, over the same 120 Codeforces problems. Rows are the model whose reasoning is being scored; columns are the model doing the scoring. The diagonal, where a model scores its own reasoning, is set in italic. Colour is on a logarithmic scale, dark for reasoning the scorer finds native.
Figure 26: Median perplexity of reasoning traces under seven scoring models. Every model scores perplexity of the reasoning, conditional on the problem, over the same 120 Codeforces problems. Rows are the model whose reasoning is being scored; columns are the model doing the scoring. The diagonal, where a model scores its own reasoning, is set in italic. Colour is on a logarithmic scale, dark for reasoning the scorer finds native.

실제로 확인된 결과

  • Anthropic, OpenAI, Google 세 회사 API 모두에서 강한 모델의 암호화 추론을 약한 모델(Haiku 4.5, GPT-5.6 Luna, Gemini Robotics 1.6)에 넣어 평문으로 추출하는 데 성공했다.
  • 120개 Codeforces 문제에서 API가 보고한 추론 토큰 수와 추출된 추론을 다시 넣었을 때의 토큰 수가 대부분 서로 일치해, 추출 결과가 신뢰할 만함을 확인했다.
  • 공개된 6,708개 에이전트 세션에서 315,320개의 추론 블록을 해독해 개인정보 367건, 계정정보 182건(그중 실제 사용자 세션에서만 API 키 62개, 비밀번호 33개, 개인 이메일 30개)을 복구했다.
  • 복구된 704건 중 64건은 대화창에 보이는 텍스트에는 전혀 없이 오직 암호화된 추론 속에만 존재했다.
  • 겉으로 안전하게 거절한 Opus 4.8의 답변 뒤에도, 추출한 추론 속에는 유해 정보(차량 절도 방법 등)가 그대로 남아 있음을 확인했다.

어디에 쓸 수 있나

  • AI API를 사용하는 개발자는 암호화된 추론 블록이 포함된 세션 로그를 온라인에 그대로 공개하면 안 된다는 점을 인지하는 데 참고할 수 있다.
  • AI 서비스 제공사가 추론 블록을 세션·사용자·모델에 종속시키는 암호화 설계로 개선할 때 이 논문의 취약점 분석을 참고할 수 있다.
  • 보안 연구자가 다른 멀티모델 API 생태계에서 유사한 교차 모델 호환성 취약점이 있는지 점검할 때 방법론으로 참고할 수 있다.

한계와 남은 검증

  • 2026년 8월 시점에서 제공사들이 논문 공개 후 대응 조치를 취해, 논문에 기술된 공격은 더 이상 그대로 재현되지 않는다.
  • 실험은 Anthropic, OpenAI, Google 세 회사의 특정 모델 조합에 한정되어 있으며, 다른 회사나 향후 모델에도 같은 취약점이 있는지는 확인되지 않았다.
  • 공개된 세션 315,320개 중 실제 개인정보가 발견된 비율은 0.3%로 낮아, 모든 공개 로그에 위험이 있다고 일반화할 수는 없다.
  • 추출된 추론이 원본과 완전히 동일한지는 토큰 수 일치로 간접 검증했을 뿐, 내용 자체의 완전한 정확성은 보장되지 않는다.
  • 제안된 암호화·시스템 수준 완화책은 아직 실제로 적용되어 검증된 것이 아니라 논문에서 제안된 방향일 뿐이다.

왜 중요한가

겉보기 답변만 안전하게 필터링해도 속마음 추론에 위험 정보나 개인정보가 그대로 남아 있으면 실질적인 보안·프라이버시 방어가 무너진다는 것을 보여준다. 개발자들이 세션 로그를 그대로 공개할 때 암호화된 블록 속에 무엇이 들어있는지 모른 채 실제 비밀정보를 유출시킬 수 있다는 점에서 실무적으로 중요하다.

이 논문의 용어

  • 추론 과정(chain-of-thought) · AI 모델이 최종 답을 내기 전에 속으로 단계별로 생각을 풀어가는 내부 텍스트
  • 암호화 블록(encrypted reasoning block) · AI 회사가 추론 과정을 사용자에게 넘길 때 읽을 수 없게 암호화해 담아둔 텍스트 조각
  • 디코더 모델 · 다른 모델이 만든 암호화 블록을 받아 그 내용을 평문으로 다시 풀어내도록 이용된 방어가 약한 모델
  • AEAD(인증 암호화) · 암호화와 함께 내용이 위조되지 않았음을 검증하는 서명까지 포함하는 암호화 방식
  • 증류 공격(distillation attack) · 경쟁사나 공격자가 다른 회사 모델의 출력(또는 추론)을 학습 데이터로 써서 비슷한 모델을 만드는 행위

저자 · Alexander Panfilov

arXiv에서 원문 보기

최신 논문

논문 전체 보기 →

METAL MEDIA 최신 기사

그림 출처: Alexander Panfilov et al., arXiv:2608.09867, CC BY 4.0