Posted: May 27, 2021

Welcome to the third installment of my Journey of Learning.  This series is a look into my studying and preparation for the OSCP exam.  If you missed the previous story, you can catch up here.  The ContentOverload application I am building to explore some of the exam topics has some new functionality.  While users still cannot do anything useful yet, they can update their profiles.  In addition to updating their name and email address, the user can upload a PNG file as an avatar.

I spent more time figuring out how to auto-generate default avatars with RoboHash than I did on the rest of the process.  GitHub, TryHackMe, and HackTheBox all generate a randomized avatar when you create an account, and I though it was kind of fun.  If users do not like the randomized carton kitten avatar, they can use their own PNG file instead.  I don’t know why anyone would choose NOT to use a random kitten for an avatar, but some people are just weird like that.

The dangers of allowing users to upload files into an application was meant to be the focus of this post.  When I sketched out the outline for this story, I thought about all the CTF challenges I have completed that involved uploading files.  I wanted to tell a story about evading file type filters, disguising webshells as PNG or SVG files, and other tricks that I have learned.  I really wanted to try out something like this exploit where the PNG is encoded with some PHP code.  Along with strategies for evading filtering, and the exploits themselves,  I intended to showcase the improved visibility and functionality of the auditd logging to detect the attacks.  You may notice a lot of past tense being used here.

It wasn’t until I was trying to upload a PHP shell that I realized I had missed an important step in the process – you can’t exploit PHP if it’s not installed.  Most of the techniques that I am aware of for file upload attacks involve hijacking PHP to get something done.  PHP is common on web servers, but with the rise of JavaScript and Python frameworks for web applications, the dependency on PHP is being reduced.  This reduces the attack surface as well, or requires new and different kinds of attacks to be used.

This is a Python app; PHP is not installed on the server.  I could mangle the application logic a little to make an attack work, or I could switch gears and try to approach this topic differently.  During an assessment, a pen tester does not always get access to the box.  That does not mean that the assessment was not useful.  Looking at an application from an attacker’s perspective can still improve the application by improving things like monitoring and observability.  Since I was not able to make an exploit work, this post  about preventing malicious file uploads is off to a good start!  I’m not abandoning the idea of file upload vulnerabilities, just adjusting the approach a little bit.

Defensive security starts with visibility.

The server that is hosting the application is running auditd with a robust rule set, and those logs are being ingested to Splunk.  While fixing the issues I found during my last post, I learned that the Splunk App for *nix includes a scripted input that utilizes the ausearch functionality  to ingest the data in a more usable format.  Without this scripted input a lot of the data from auditd log files is ingested with hex encoded strings, which is not very helpful.

Even after the decoding, the auditd data is still complicated. I stumbled around executing commands on the server and trying to find the audit of those commands in Splunk.  What I didn’t know then, but would have known if I read the documentation, is that auditd breaks up the events into multiple record types.  A single command can result in many different audit records each with different kinds of information.  The nice thing is that the records share a timestamp and an ID number.  A single “git pull” command looks something like this:

The type=SYSCALL event shows a lot of information about who ran the command.  The type=EXECVE event shows the that a “pull” argument was passed to the command.  Two records with the type=PATH show files that were accessed, and finally there is a type=proctitle that shows the git pull command again.  While I can see the git pull command happened, I cannot see where it was run from … what directory did this happen in, what local repository was updated?

You know how everyone says you should read and understand everything you pull from the web before you use it?  There are good reasons for that.  It is time-consuming, but it can save some time and frustration later.  On the other hand, just giving something a try can be a good way to learn things too.  It’s a little riskier but working through problems does provide learning opportunities.

The config file I am using included a line “-a always,exclude -F msgtype=CWD”.  The comment above this line is “Ignore current working directory records”.  I think auditing without knowing where the commands ran from ran is kind of pointless. Maybe there is a good reason for why that configuration was built that way, but I don’t know what it is, so it’s out of here!  If this causes me a problem down the road, I will deal with it when I see it.

After the change, I can now see where commands are being run from.  So that is a win.  Auditd data is still complex, and its greatest use is likely to be backward looking forensics, but I know a lot more about this data than I used to.

In addition to the basic system information, the application logs to systemd, which is also ingested into Splunk.  I am not sure that I like this better than writing to log files directly, but since I haven’t done application logging this way before, I figured I would give it a try. This is a learning exercise, trying new things is kind of the point.  The combination of auditd, application, and Nginx access logs gives me a pretty good record of what is going on within the application and its server, it is time to look at the application itself.

For the review of the application, I’m going to do a little table-top exercise.  I’m going to walk through how I would attack the application, while thinking about how I would detect and mitigate the things that I find.  The results of the exercise get turned into new tasks for future development.  Think of it like playing Dungeons and Dragons by myself, without all the fuss of character sheets, dice, and costumes.  Costumes are part of a normal D&D game … aren’t’ they?

Login attacks are low-hanging fruit for an adversary.  I can analyze the login request volume using the Nginx logs, but I won’t see what is going on without additional information that isn’t in the standard access logs.  It is tempting to log a hashed version of the password, but that can lead to leaked information. Even though it would be nice to be able to definitively see the same passwords being used on multiple accounts, the risk of data leakage outweighs the visibility gain.  Logging the failed attempts, along with the successful logins including the reason for failure, and the remote address provides the basic building blocks for analysis.

The profile page, where the user can change their avatar image along with their name and email address has some logging in place already.  The file name for the image being uploaded uses a naming convention based on the datetime, and the name of the user.  No matter what the user calls the file when it is uploaded, it is going to be renamed when it is stored on the server.  Denying an attacker the ability to control the file name shifts the balance of power to the defender’s side.  There are ways to try to bypass the rename such as null-bytes and other escape sequences, but when you cannot control the file name things are a lot harder.

The problem here is that the name of the user can be changed.  What happens if the user changes their name to something like “usershell.php”?  Well for this application, nothing happens.   The file name becomes something like “20210511204320_usershell.php_avatar.png”, but ultimately everything works as expected.  The file name is generated using a utility function “secure_filename”  that will replace strange characters and unsafe patterns in a file name, rendering it “safe” to use with the file system.  I put safe in quotes because it still allows me to include non-alphanumeric symbols, which isn’t good for a file name.  There is logging in place to record file names for the uploaded file and the name that the file will be saved as. If someone figures out how to bypass the file name protections, the logs will show what file names slipped through so that it can be handled.  Even though it doesn’t result in an exploitable vulnerability, refining the logic to not use any data directly from the user would be wise.

While looking over the logic to handle the form data being sent to the server, I noticed something interesting.  The logic for updating profile attributes basically checks the fields that were sent in the form, against the fields names in the record returned from the database.  A Python list comprehension, a fancy way of doing a loop, updates the values that were sent in the webform.  Any field that exists in the web form, that also exists in the database record, except for the password field will be updated.

The issue here is that there is a field “avatar” on the database record.  The avatar isn’t part of the web form – the file can be uploaded but the field is called NewAvatar, and handled differently.  Could an attacker figure out that this field exists? Could I fuzz the application to figure this out? Maybe.  The word “avatar” appears in at least one word list used for fuzzing and enumerating applications.  Using Burp Suite, I can intercept the request from the browser, and make a tiny adjustment to the form before submitting it to the server to be processed.

Looking at the database, I can see that the change was successful.  This is a bug.

When I tested this out, the value was overwritten back to a system-defined value the next time the page was loaded. The bad file name that I injected didn’t exist on the server, so a new avatar image file was generated, and the avatar value was updated in the database.  This could be the start of a local-file-inclusion vulnerability, but it would need some work to get it working in any usable way.  The attacker didn’t see any errors or messages about this value, and they do not see the database fields or results. But low severity bugs can be combined into high-severity exploits, it looks benign now, but it’s an abnormal behavior that should be corrected. The logic should be adjusted to process the update explicitly; only update fields that are intended to be updated.  There is no need for the clever loop.

Thinking about the image file being uploaded, checking the file contents to make sure the file is the PNG file that we expect is another good step.  A simple “file” or “stats” command gives some information about the file that can be used as a starting point.  It might be tempting to just use the OS commands, but that may not work if the application needs to be deployed on different operating systems.  It looks like a Python library called python-magic might do the job though.

There are several libraries in Python for more complex operations on images, though sometimes that can cause problems too.  The ImageTragic exploits for ImageMagick are really old, but a stroll through the 600+ CVE’s for ImageMagick over the years show that automated processing of arbitrary data from an untrusted source is really tricky, and scary.  I don’t see many recent CVEs that lead directly to code execution, but there are a lot of overflow and crash errors reported here that can disrupt application availability.  The lesson here is that if you are going to be doing automated processing of files, don’t do it “in line”.  Move it to a protected asynchronous process, and watch those processes for failures very closely.

Putting files through some sort of sandbox analysis would be a good idea to explore.  This is the kind of thing that lambda functions and containers are good at.  If the file passes all the checks, it can be used, otherwise, flag the results for analysis and lock the file up, well away from the users and application code, preferably at the bottom of the nearest lake, in a lead-lined box, wrapped in chains.  That might be overkill, a little. At this very early stage, I am going to stick with some rudimentary checks of the file type and contents and reject anything that isn’t actually a PNG image with a file size less than 80 kilobytes.  Simple checks that will weed out the riffraff, more complex strategies can be introduced later.

Keep moving the needle:

Security teams are not all-mighty when pushing policy and objectives. The security team still must cooperate with project managers and application owners who have their own roadmaps and deliverables to manage.  Getting a few minor changes into the next release to improve the security posture, without significant impact to the deliverable timelines is better than a political fight over priorities which puts any improvement at risk .

In my experience, taking several small steps is a lot more effective than trying to overhaul a system.  Race teams do not try to change the tires while the car is racing.  No matter how many times I hear that analogy, it just does not happen.  Big changes will be met with resistance, small changes will usually be accommodated.  Stack up enough small targeted changes, and the big overhaul is no longer necessary.

Four low-complexity enhancements were identified during this exercise.  All of them improve the security posture of the application in some way.  While the development team (me) works on the requests, the security team (also me) can go to the SIEM and other tools that they use and prepare dashboards and alerts targeting the new data that is going to be coming in.  I might even build a test suite to trigger the alerts that can be run periodically to confirm that everything is working after future releases.

If someone decides to try to brute for the login page, I am going to be able to use the Nginx and application logs to detect the abnormal number of failed logins.  I will be able to see which user, or users is being attacked. I will be able to identify the source IP of the attacks, which could then be locked out at the firewall, or trigger a device quarantine, or something like that.

I am going to run into even more trouble getting anything, but a small PNG file uploaded to the system.  Even if I manage to get something up on the server that I can exploit, the auditing rules are going to be sniffing out any actions that are taken.

It seems kind of late in the article to be noticing this, but I didn’t talk a lot about malicious files in this post about the dangers of file uploads.  The number of attack types, and things to try to discover vulnerabilities is extensive.  When I was trying to figure out the attack for this post, I tried almost everything on that list before I realized that PHP wasn’t installed.  An experienced pentester probably would have realized the problem a lot sooner, probably from the very beginning.  Instead of wallowing in self-pity about failing to find a file upload vulnerability, I did a little pivot and found something else to improve in the same general area.

That was an important lesson for me.  Before trying to exploit something, make sure it is available to be exploited.  Before trying to initiate a reverse shell, try to catch a simple ping to verify connectivity.  Recon and enumeration are about patience and diligence far more than just being clever.  Like reading documentation before putting a configuration file or piece of code into use, a little patience up front will save hours of hassles later.

As I study and prepare for the OSCP exam, I have done a many CTF challenges that involve file uploads.  Few attacks start with an uploaded file.  Most of the time, the application has a vulnerability that must be exploited to get to the file upload functionality.  Keeping an attacker from getting authenticated into the application can prevent them from being able to try the fancy filter bypass techniques, null-byte escapes, and disguised payloads.  It does not matter if your file upload functionality is bullet-proof if an adversary can sidestep your login page and dump your database.  At the same time, if an adversary gets a hold of credentials for the application from a phishing attack, there had better be protection in place for malicious activity once they get logged in.

The next post is going to be about kernel exploits, those things with the funny names like Dirty Cow and Juicy Potato.  When I do a CTF challenge, the designers of the challenges have always been nice and included the various packages that are needed to compile things right on the server.  It has always seemed to me that would not work in the real world, so I’m going to dig into that and see what else I can learn.


This blog was written by Greg Porterfield, Senior Security Consultant at Set Solutions.