Mordax

Software Developer

View My GitHub Profile

Thoughts on the Overscripted Challenge

Back in August 2018, I was approached by my real life friend biskit1 about a competition Mozilla was hosting called the Overscripted challenge.

Side note: She wrote a fantastic blog post on it, found here.

Using a trove of crawled data, 70GB in total, we had to make a Jupyter Notebook outlining observations found in the dataset. This crawl had gathered information of the types of Javascript calls being made by thousands of websites and Mozilla was (and still is) interested in what unseen things were happening behind the scenes of the Web. Repo can be found here.

We were lucky- a previous group had already made some good analyses that we could follow along with, such as eval calls, cryptojacking, and so forth. They’ve got a repo as well. Discussing our methodology is beyond the scope of this particular post - I will most likely discuss it at a later time.

This competition had been ongoing for roughly a month and we had joined when there was 2 weeks to spare (and later it was extended last minute which gave us an extra week). Neither of us were data scientists nor of any scientific background (except my partner who briefly dabbled in biology). We were merely programmers, of a +15 to our C++ ability score.

We had to learn a multitude of things in a short time, including Python, its libraries such as Pandas and Dask, Apache Spark and wrap it up neatly inside Jupyter. We never touched any of the tools before the challenge (we also did Javascript analysis, but we were familiar with that).

When we did a hackathon back in January 2018, we did a bunch of research and programming for a project. But in the end we were too disappointed with what we made to submit it. And for students anxious about having side projects, it was a heavy blow. So for this challenge I told my partner-in-crime that we had to put something up. No matter how ugly or how broken it was. It was going up no matter what.




Snapshot of the analysis at time of writing

I meticulously made sure we covered every minutiae mentioned in the challenge rules, including the fact that we must do a pull request, have our emails included, mention which topic to submit to, and so on.

In the end we were the only people who had the courage to properly submit their analysis. There were more than a dozen people who had forked the challenge repo - one even finished their analysis. The rules even said, one winner from 3 topics posted would win. That person could’ve won, but they didn’t follow the instructions all the way to the end.

The prize was an all expense paid trip to Mozfest 2018 in London, England.

I was certain that they would call off the contest due to the lack of participants. At least we had a unique personal project. The contest ended in September and I put it out of my mind. That is until October.

And we then got the email that we were being invited to Mozfest. Cue the jumping around.

The only catch was that Mozfest was happening on October 26 to 28 and the amount of last minute planning we had to do was crazy. We also extended the trip since neither of us had gone to London and 3 days wouldn’t leave enough time to go sight seeing.

We got to meet the people who were kind enough to review our analysis on Github - it was a surreal experience.

Mozfest was a very personal experience and an important professional experience. I had yet to be recognized for any software work I had done and it came at a time where I was struggling a lot.

The lesson I’d want others to take from this post is that both showing up and putting something up is vital.

You need to show up when it counts. You need to show what you have made in an easy-to-understand way. And of course, the most important lesson of all - always follow the instructions.

The great thing about code is you can always go back and improve it if you really need to. The important thing is having something that you can even improve in the first place.



Special thanks to the following folks for making this trip so very special to the both of us -

And everyone else we managed to encounter on this awesome trip!