Debugging a Python malware scanner with YARA and VirusTotal - a technical journey

One thing I have learned quickly while administering servers is that fantastic scripts found online rarely work on the first try in a production environment. The underlying concept might be excellent, the architecture might look sound, but real-world execution often stumbles upon specific Python versioning, missing dependencies, broken regular expressions, or structural bugs that only surface when you actually point the tool at a live server directory.

Recently, I needed to perform a deep forensic scan on a web directory:

/home/user/public_html

The objective was to hunt for potential webshells, obfuscated code blocks, malicious PHP files, suspicious use of eval, base64_decode chains, unexpected compression (gzip/zlib), and any other artifacts indicating a compromise in a shared hosting environment.

To achieve this, I chose a robust Python script called encfind.py. Its value proposition was strong: using regular expressions to detect known malware signatures, calculating Shannon Entropy to identify highly randomized or obfuscated blocks, integrating YARA for professional-grade rule matching, and querying file hashes against the VirusTotal API.

On paper, it looked like the perfect scenario. In practice, the very first execution reminded me that a security tool only becomes an operational asset after it has been debugged, tested, and validated in the field.

Scope and objectives#

My goal was not merely to "run a script." I wanted a reliable static analysis tool for incident response and operational auditing of web applications, specifically those built on PHP.

The scanner needed to deliver actionable intelligence on:

• Known Webshells: Detecting common backdoors like China Chopper or C99.
• Dangerous Functions: Highlighting the use of eval, passthru, or shell_exec.
• Obfuscation Detection: Flagging chains like base64_decode(gzinflate(...)).
• High Entropy: Identifying files that are likely packed or encrypted (high randomness).
• YARA Indicators: Using custom rules for fine-grained detection.
• VirusTotal Reputation: Cross-referencing hashes with global threat intelligence.
• Risk Prioritization: Scoring findings to reduce the time spent in manual triaging.

In a hosting environment, this is critical. A public_html directory can contain thousands of files - legitimate plugins, themes, backups, and caches - mixed with suspicious artifacts. I needed a tool to bridge the gap between "suspected compromise" and "technical evidence for action."

Challenge 1: The Python 3.6 "ghost"#

During the initial attempt, running as root, I was met with a standard but blocking error:

[root@server scripts]# python3 encfind.py
Traceback (most recent call last):
  File "encfind.py", line 25, in <module>
from dataclasses import dataclass
ModuleNotFoundError: No module named 'dataclasses'

The error was explicit, but the decision process required caution. The dataclasses module was introduced natively in Python 3.7. However, this specific server was running Python 3.6 as the default python3 binary.

I could have upgraded the entire system's Python runtime, but in a production environment, that is rarely my first choice. The system's Python is often a core dependency for OS-level tools, panel scripts (like cPanel/WHM), and native automation packages. Upgrading it could solve a small problem while creating five silent, catastrophic ones.

My solution was to install the official dataclasses backport via pip3:

pip3 install dataclasses

This fixed the dependency without altering the server's global runtime. It was the cleanest solution for this context, maintaining compatibility with the existing Python 3.6 environment while allowing the script to proceed.

Challenge 2: The missing parenthesis in regex#

After resolving the dependency, I ran the script again. It progressed further but halted at another traceback:

sre_constants.error: missing ), unterminated subpattern at position 28

This error occurred when the script attempted to compile its dictionary of webshell signatures using the re library.

I went straight to the WEBSHELL_SIGNATURES dictionary and located the broken entry:

'file_upload': r'move_uploaded_file\s*\(.*\$_(FILES',

The issue was obvious in the pattern:

$_(FILES

The parenthesis before FILES was interpreted by the regex engine as the start of a capture group, but it was never closed. Since the string had precisely that offset until the error, the position 28 message was accurate.

My fix involved closing the group properly and ensuring the pattern matched the intended PHP variable:

'file_upload': r'move_uploaded_file\s*\(.*\$_(FILES)',

This adjustment allowed the Python interpreter to successfully compile the entire signature set.

Operational lesson: signature integrity#

This type of failure is deceptively simple but catastrophic. Because the invalid regex was part of a global dictionary compiled at startup, the scanner could not operate at all. It failed before even analyzing the first file.

After fixing the specific line, I performed a full audit of the signature block to check for:

• Unclosed parentheses or brackets.
• Inconsistent escape characters.
• Mismatched capture groups.
• Ambiguous use of shell-style variables like $_.
• Raw strings that didn't align with regex execution intent.

In a security tool, a broken signature is a total failure of tool availability.

Challenge 3: Structural chaos and attributeerror#

Once the corrections were made, the script finally initialized. It displayed the banner, showed the worker count, and began scanning the directory:

[root@server public_html]# encfind.py
🔍 Scanning: /home/user/public_html
   Workers: 6 | Entropy Threshold: 4.5

It seemed the hurdles were over. However, upon completing the scan and attempting to print the report to the screen, a new error surfaced:

Traceback (most recent call last):
  File "/usr/local/bin/encfind.py", line 1010, in main
results.display()
AttributeError: 'MatchList' object has no attribute 'display'

This error was a clear indicator of structural rot. The results object, an instance of MatchList, lacked the display() method.

Looking at the source code, I discovered something worse than a missing function. The file was structurally a mess - likely due to repeated copy-pasting or fragmented edits over time. The entire code block had been duplicated within the same file. There were repeated class definitions, loose blocks of code, and most importantly, the display() function was sitting outside the scope of the class it was intended for.

The scanner was performing the heavy lifting of collection but failing at the final step: presentation.

Deep structural cleanup#

My solution was a brute-force cleanup of the source file. I removed all redundant class definitions, reorganized the hierarchy, and ensured that the display() method was correctly scoped as a member of the MatchList class.

The goal wasn't just to make it "work"; it was to restore proper Object-Oriented integrity so that the call:

results.display()

made sense within the object returned by the scan pipeline. I treated the tool as production-grade software: dependencies fixed, regexes validated, duplicates purged, and presentation methods restored to their rightful objects.

Forensic methodology: validating the detection#

With a functional script, I needed to prove it could actually find threats. I did not want to upload real malware to a live production server. Instead, I created a safe, controlled test environment by simulating malicious behavior.

I created a file named teste_shell.php in the web directory:

<?php
// Simulated China Chopper signature
$mock_chopper = "eval(base64_decode(\$_POST['cmd']));";

// Simulating obfuscation with str_rot13 for AST decoding tests
$ofuscado = str_rot13('echo "This is an obfuscation test";');
eval($ofuscado);
?>

This file contained critical test elements:

• A classic eval(base64_decode(...)) chain.
• A direct reference to $_POST['cmd'].
• Dynamic execution via eval.
• Obfuscated strings using str_rot13.

The objective was to validate that the regex, heuristics, and entropy scoring could correctly identify and flag these patterns without needing a real virus.

Why use controlled samples?#

In a production environment, placing real malware - even for testing purposes - is a major operational risk. It can trigger AV alerts, WAF blocks, backup quarantine, or account suspensions. A controlled sample is superior because it validates:

• File read capabilities.
• Regex pattern matching.
• Dynamic execution classification.
• Entropy scoring accuracy.
• Report generation logic.

VirusTotal integration and the EICAR test#

Since the script integrated with VirusTotal, I needed to validate external API communication and hash-lookup logic. For this, I used the EICAR Standard Antivirus Test File string:

X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*

I created a simple .txt file with this string. EICAR is not malware; it is a universal standard designed to test detection mechanisms safely.

Final scan execution#

After preparing the test environment, I ran the final command:

encfind.py -d /home/user/public_html/ --vt

The results were perfect. The terminal immediately flagged teste_shell.php, highlighting the obfuscation, execution risk, and specific regex matches.

For the EICAR file, the VirusTotal integration worked flawlessly. The API returned a high detection rate, and the script correctly interpreted this as a maximum-risk indicator. This proved:

1. External communication was healthy.
2. Hash calculation was accurate.
3. The reporting logic for third-party intelligence was functional.

Technical quality audit of results#

After the scan, I didn't just check for "red text." I audited the output for consistency:

• Simulated PHP File: Did it detect eval? Did it see the $_POST input? Is the score representative of the risk?
• EICAR File: was the hash sent correctly? Did the VT integration report detection? Was it cached properly?

This level of scrutiny prevents a false sense of security. A tool can "run" without errors and still fail to detect anything. My criterion was simple: Run, Detect, Explain, and Triage.

Operational checklist#

The finalized workflow for using the scanner in a technical environment:

1. Environment Prep: Check Python version and install backports if on < 3.7.
2. Binary Health: Ensure encfind.py is in the path and has executable permissions.
3. Execution: Run with -d for directory and --vt for VirusTotal enrichment.
4. Validation: Use a simulated teste_shell.php to verify the local signatures.
5. Enrichment: Check API quotas and communication for VT/YARA.

Practical takeaway#

What started as a simple five-minute execution evolved into a deep debugging session involving runtime dependencies, regex syntax, structural Refactoring, and forensic validation.

I moved from a script that crashed on import, then crashed on regex compilation, then crashed on reporting, to a production-ready tool that identifies suspicious patterns, uses Shannon Entropy for heuristics, and integrates global threat intelligence. This transition from "downloaded script" to "operational tool" is fundamental. In cybersecurity, the existence of a script is secondary to its reliability. By understanding its dependencies, fixing its flaws, and testing it with controlled samples, I created a reliable forensic asset for auditing web servers.

See also

Cybersecurity Forensic analysis of a CloudLinux and cPanel compromise: from the sshd_config trap to PAM and IPMI persistence

Cybersecurity Decoupled architecture of Imunify email: reverse engineering thresholds and quarantine monitoring

Infrastructure Anatomy of a SIGKILL in login failure daemon: debugging LFD service startup failure on Linux

This post is licensed under CC BY-NC.