Analyzing publicly exposed AWS S3 Honey Bucket Logs using MSTICPy

For this use case, we created a honey bucket named microsoft-devtest on Feb 07, 2020

Once you have registered the honey bucket with desired juicy name and provided your email address, the service will monitor any HTTP or API traffic and alert you via email with necessary details. Sample email notification look like below. In the email, you will also see a dashboard link, or you can click View alarm details to see all your alerts in a single view.

As mentioned earlier the Breach Insider service provides extended logging as compared to standard AWS S3 bucket access logs, which will be beneficial while analyzing activity. You can read more about the underlying of the how the service processes logs but here is a snippet from the blog which provides the general concept on logging.

All of the historical alerts can be accessed via a single dashboard link. Unfortunately, even though the logs are parsed , they do not have uniform structure hence cannot be consumed readily for analysis.

Here is an example alert:

As you can see log data is poorly formatted and not helpful to analyze hence we can use Python`s powerful data analysis libraries to clean and transform into structured JSON file.

After using the helper function, here is output JSON looks like.

Once we have all the enrichments completed, you can start to do data exploration to get interesting insights about the scanning behavior from Public IP space.

Below are the sample insights that we included in the notebook to understand the scanning and access patterns from different public IP address spaces.

Post data analysis, we have also visualized the patterns using python`s standard data visualization libraries such as matplotlib, seaborn and MSTICPy built-in Folium map plotting.

As an example, below shows the monthly distribution of alerts since the honey bucket was created in Feb-2020. As you can see, an unusual spike in alerts started in Jan-2022 related to active scanning. This was plotted as simple timeseries using matplotlib.

Since we have enriched data with Geolocation details, we can also understand access patterns from different countries. In the below visualization, we have plotted a Tree map with size and color indicating the number of alerts observed. As you can see, there were many countries with just a small number of alerts vs some countries such as France and Belgium which have highest number of alerts.

Further, we also enriched IP addresses with ASN, we can visualize the access pattern by ASN via heatmap. For effective visualization, we have filtered out the outlier ASN Online SAS, FRwhich has the highest number of alerts.

As you can see, ASNM247,GBis appearing from multiple countries as compared to other ASN which are with just 1/2 countries. Although it is not unusual to see ASN with multiple countries in your logs, based on open-source intelligence lookup, M247 is a high fraud risk IP hosting various anonymizing VPN , proxy servers often used for reconnaissance hence traffic attempts coming from this ISP are likely to be suspicious.

Lastly, we are using MSTICPy Folium map plotting to understand Geolocation access patterns. In the plot below, IP addresses flagged by TI as high, or warning are plotted on Geolocation map.

Below is a summary of interesting insight uncovered from analyzing S3 honey bucket logs.

In this blog, we investigated how to register for AWS S3 honey bucket for free using the Breach insider service and analyzed them using MSTICPy. The service offers central text files with all historical logging telemetry however it was not in structured format, so we converted text file into structured JSON file using Jupyter notebook and Python for further analysis. Clean dataset was also contributed to Security datasets project. This clean dataset was then analyzed further using MSTICPy features such as GeoIP, whois and ThreatIntel lookups. We performed exploratory data analysis on enriched data to get interesting insights about the logs and also visualized data using Python libraries and MSTICPy visualization features. This notebook along with real world dataset can be used as a demonstration to analyze reconnaissance activity on publicly exposed storage buckets. It also showcases how you can use various built-in data enrichment and visualization modules of MSTICPy to uncover interesting patterns from security logs easily. Hope you will find this blog useful and will be able to use MSTICPy in similar use cases. Happy Hunting !!

Related posts:

Dans cet article je vais vous présenter les différents types de gap qui existent. Ce que nous allons voir peut s’appliquer sur différents marchés : forex, bourse, crypto-monnaies etc… Dans un premier…

Websites are the essential “business card” in the online realm. Having the joy of creating them for 20 years (and counting) I felt the urge to create this article, and share with you some of the most…

If you want to move to a place that will make things easier for you and where you can still enjoy your freedom, moving to a senior independent living community is a perfect option for you…