Email Obfuscation Research

Click here to visit the GitHub repository.

The main logic extracts emails from websites by using requests to fetch content, parsing the HTML with BeautifulSoup, and removing scripts, styles, and comments to clean up the HTML. Then, the program normalizes URLs, scans for common obfuscated emails with regex patterns, and replaced encoded characters or phrases with the correct values. If no email is found, the program recursively searches for contact, about, or email sections that might contain an email address. URLs that yield an email address are labeled as "regular" or "obfuscated" accordingly, while unresponsive URLs are labeled "broken."

One of the imports is csvreader, a custom Python file that contains data about professors, with the fifth row containing the URL to their website. There is also a function that converts the URL data from each row into a list, which is then used in main.py to test the logic's performance. This library was not intended to test the program's large-scale capabilities, but rather debug and improve the current program.

My mentor and I decided to use Python for this program due to its available libraries and widespread use in the cybersecurity field. Because I had no prior knowledge of the language, I had to learn while doing; this meant searching for syntax conventions as I went, rather than sitting down and learning without making progress. Additionally, I familiarized myself with Visual Studio Code as a local python compiler for this project, later learning how to upload the code to a GitHub repository.

To properly scrape websites for emails, I needed to research different email obfuscation techniques so that my program could catch as many cases as possible. Because professors are often targeted by email scraping, I used their websites as a starting point. After searching through hundreds of professor websites, common obfuscation techniques I found included spelling out "." and "@", including emails in "mailto:", translating emails into URLs, breaking up emails into multiple strings, and using "firstname" or "lastname" as placeholders.

As an intern for UCSB's SecLab, I explored the question "How effective are email address obfuscation techniques against automated scraping?" This was done by writing a program in Python and comparing the results to those produced by AI. Some general skills learned during this project include:

- Scraping information from URLs with Beautiful Soup

- Using prompt engineering to extract data from AI

- Cleaning data with regular expressions

- Scaling project by using csv data

- Basics of ethical hacking

After dedicating approximately 72 hours of work to this project, I took a step back to focus on school. Despite not seeing it to completion, the skills I took away from this internship experience solidified my passion for programming and problem solving. I am confident that I will be able to work on similar projects and continue to contribute to computer science research.

Hard programming is one way to scrape websites, but another way is implementing the help of artificial intelligence. This is why the second phase of this research was comparing my program's performance with that of AI. To scale this process, I began writing a program that would send an AI model the proper information and receive an email in return. Sending the actual URL to the AI model was not successful, as it cannot scrap from websites directly. A workaround I discovered was using requests and BeautifulSoup to fetch websites' HTML and sending that to the AI model instead. I found that the HTML from a web page far exceeds the token limit, so I split the HTML from each URL into chunks. Those chunks are then individually sent to the AI model to process.

While my Python program often struggled to detect certain obfuscation techniques, such as emails contained in "mailto:", the AI model was able to correctly identify emails in a variety of formats and obfuscation patterns on a small scale.