We all sometimes work with incomplete information and get by quite well. In fact, having the full picture is rarely required in day-to-day life. It only becomes important when dealing with problems that require extensive knowledge about the subject to get everything exactly right, especially in areas where you can’t immediately check if what you’re doing is effective. Application security is an excellent example of this—but before we get to dissecting XSS, let me start with a little anecdote about my past experiences.
Everything starts with incomplete information
I’m a father, and if you don’t have kids, you may be surprised to learn that the most important perk of parenthood is you have an excuse to eat bubblegum ice cream with rainbow sprinkles when your daughter doesn’t finish her helping. And all without a single judgmental look from the ice cream guy. “Yes, I am sure she wants six scoops. Of course she will eat all of it.”
A close second is getting to think deeply about topics you are so used to that you don’t question them anymore. To kids, everything is new and fascinating because they haven’t yet made all the mental connections you have. And so, they think about a whole bunch of things and fill in the blanks using their current limited understanding of the world.
In practice, this means that whenever my daughter thinks about something and can’t quite figure it out by herself, she comes to me and asks for the missing clue, expecting a well-thought-out, factual answer. To her mind, I’m probably the smartest man on the planet. In her defense, the poor child just doesn’t know any better.
The hard truth behind duck factoids
Her questions start with simple topics, such as “Why can’t I just pet any dog that walks by?” After all, our golden retriever loves nothing more than being petted all day. That one is easy: “Because some dogs don’t enjoy being petted by strangers and they may bark or even bite you.” But sometimes, we have more difficult questions, like “Are there any dinosaurs living today?” To which I confidently replied: “Well yes, of course. Birds are dinosaurs!” And I know that for a fact because I learned it from a show about dinosaurs that we once watched together, back when she was more interested in feeding ducks in the park than in determining their detailed taxonomy.
However, something did not sit right with me after I said it. At a surface level, my answer was correct. You can state “Birds are dinosaurs” and, on the basis of modern science, nobody can disagree. (I double-checked on Wikipedia, so it must be true.) But here’s the problem: based on my off-the-cuff answer, my daughter now thinks that when dinosaurs roamed the Earth, there were already ducks swimming in a pond somewhere. In reality, the first ducks didn’t appear until like 40 million years after the (obviously inferior) non-bird dinosaur varieties went extinct. So even though they are technically considered dinosaurs, they weren’t around back when all the T-Rex action happened.
Quite honestly, I could do well in life without knowing that particular distinction. Unless someone comes out of the woodwork demanding my money if I can’t tell him everything about the evolutionary history of birds (which has happened to me twice before), it’s not information I would ever really need. But in my parental haze of rejuvenated mental connections, I suddenly realized I know how it feels to work with information that superficially looks correct but is, in fact, fundamentally incomplete—and it’s how one young developer felt when faced with his first security vulnerabilities.
Facing my first “real” vulnerability
Back before my interest in application security was sparked, I was “developing” PHP applications, mostly for personal projects and, by today’s standards, nothing to write home about. After finishing one of them, I remember scanning it with a free open-source scanner because I didn’t know too much about security and was a bit freaked out about just having my project exposed to the whole web. What if a hacker came across my site, exploited a flaw, and defaced my very reputable… Well, actually, it was just a subdomain on a free hoster, but it mattered to me.
So I ran the tool and, surprisingly, only one security issue popped up: “Directory Listing enabled.” Oh no! And I had no clue what that meant, by the way. I was still just developing sites for fun and having that scary message in front of me was enough to convince me I should immediately do something about it. I quickly googled something along the lines of “how to disable directory listing” and just did what it said. Problem solved, crisis averted, everything is fine. Phew!
The key to security wisdom: It all depends
I was still quite young when that happened and, years later, I would become more and more interested in application security. All the different types of vulnerabilities fascinated me and I began researching them. I started with the most prevalent ones (back then and now), like XSS and SQL injections, until I became confident enough to try them on a real target: the German Telekom website.
They were inviting hackers to try their luck and would hand out a bounty if anyone found a critical vulnerability (XSS was not included but would still land you a spot in their hall of fame). So I went ahead, tried finding even a single XSS or SQL injection, and failed miserably. I probed every parameter and tried every trick in my still very limited arsenal, but there was nothing to be found. Or so I thought— because through sheer luck, I stumbled upon a link that led me to a very familiar vulnerability: directory listing! Jackpot! But… what’s next?
I actually had no idea what to do with it, or why it was so dangerous that a vulnerability scanner would go out of its way to flag it as a problem. Maybe there was some way to view folders outside of the structure it showed me? Because all there was were images. I tried reaching other folders, but pressing the little up arrow just led me back to the website. There was no interesting folder in that list either. I did some more online research and figured it out: the mysterious and frightening vulnerability that threatened my own website years ago was… just not a big deal. There was no inherent danger in a directory listing.
Today, I know that whether it’s a security risk strongly depends on what’s in the folder, and in the vast majority of cases, it’s a non-issue. Don’t get me wrong, there were certainly directory listings out there that have led to massive data breaches. Long, randomly generated file names containing personal information in a folder with directory listing enabled? That’s critical. Having a bunch of public PDF files visible in a download folder? Not so much.
How hard could it possibly be to filter out XSS?
And so, on the surface, I was correct in believing that directory listings can, in fact, be dangerous vulnerabilities. I just did not know enough to realize that whether they really are dangerous strongly depends on the context. And that sudden realization stuck with me because I’m reminded of it each time I audit some code or website and see security measures that may look great on the surface but fall flat when confronted with an attacker who has extensive knowledge about the vulnerabilities they are trying to prevent.
A great example of this are XSS filters. It’s a super interesting topic because it’s so complex that even browser developers struggled to keep up with updating their built-in filters and eventually gave up trying. To see why it’s all so complicated, imagine you are developing a web application, like I was back in the day, and want to ensure there are no vulnerabilities in it. You read about some common vulnerabilities, encounter cross-site scripting, or XSS, and wonder what all the fuss is about.
The first thing you realize is that letting users publish HTML code on your web page is A Bad Thing. Right after that, you realize this is exactly what the comment feature on your site does. You’re now faced with a choice: you can encode everything, which would also prevent your trusted user base from using the full range of HTML tags to express their many emotions, or you can simply filter out the bad stuff. After checking out some XSS examples, you notice they all use the
They can use event handlers? In that case, maybe let’s just block all event handlers. And also the word alert, oh and the word eval, now that you’ve read up on it and realized that it somehow allows you to run code. And also ban parentheses, that way they can’t call any function! Hah, there is no way anybody can get around that one!