by Chris Erin on June 11, 2019

On May 13th, til.hashrocket.com (TIL) experienced an attack. This attack had the effect of overstating the relative amount of affection that the users of TIL felt towards the posts on the front page of TIL. While most of our developers will bask in the glow of any and all affection, either automated or not, the unanticipated scale of affection combined with the notifications that our slack channel receives on relatively small increments of "likes" both chafed at our sense of truth and also served as a Denial of Chat (DOC) attack in our main chat channel. The root cause of this attack was a script making requests from a computer in Dearborn, Michigan written by an author who had discovered that there is no limit to the amount of liking that can be expressed on TIL through clicking and post requests.

• May 13th 5:47 AM America/Chicago First Evidence of Attack - The Hashrocket General channel received a notification that post Values clause in a select statement had 10 likes, subsequent to that notification, the channel received another notification that post Simulate componentDidMount with a useEffect also had 10 likes
• May 13th 6:02 AM America/Chicago TIL Slackbot Integration Removed - To stop the DOC attack, the only employee working at this time removed the slack integration.
• May 13th 6:30 PM America/Chicago Like Functionality Taken Down - Without anyone to address this issue until after billing hours, and with no way to block the requests, the decision was made to disable the like functionality and to restore a backup database from Sunday May 12th.
• May 22nd 9:00 AM America/Chicago Like Functionality Restored - After crowdsourcing a hivemind solution we were able to implement a reasonable limit to our like functionality without having to severely restrain users that are truly thankful for the posts that we write. Like functionality was down for a total of 207 hours.

Ever since the inception of TIL we've wanted to allow users to express appreciation without the usual internet barriers. We don't think a user should have to sign-in to our platform just to "Like" a post, nor do we want to build sign-in functionality. This no-sign-in-for-appreciation functionality is at odds with most of the internet, which views the desire to casually interact with a system as a chance to harvest an email address, the harvesting of such being just one of many ways to monetize your behavior.

As such, we placed no restrictions on how much you could like a post on the backend. In the past, we've had co-workers who demonstrated how this could be taking advantage of by writing a shell script utilizing curl to post to the "Like" endpoint until their post was the most liked and at the top of the "Most Liked Posts" list on the stats page. These co-workers always had the good manners to reverse the untruthful and automated likes.

On the front-end we restrict "likes" by storing the liked posts as cookies. When the user comes back to the site, we reference the cookie through JavaScript to determine if the user has liked a post before, and render the "like" in the "liked" state. When in the "liked" state the user is not able to re-"like" that post unless they "unlike" it first. When first implemented this feature, we stored each hash of a liked posts in one cookie. Cookies however have a character limit, and some power users of TIL (it was me, your author, a power user) ran into that limit while expressing enthusiasm for their co-workers' posts. Subsequently we switched to one cookie per "liked" post.

We've always known that without a hard server restriction someone could write a script to indiscriminately, callously and maliciously "like" posts. What we did not know until the morning of May 13th, was that someone would actually do this. The script was actively loading the page and clicking the button. We know this because our Google Analytics spiked during that time period and in particular showed a spike of traffic from an ip address in Dearborn, Michigan. Without loading an entire web page, the Google Analytics script would not be requested.

We are not ruling out that this could have been the indiscriminate work of a web crawler, but another curious and lazy feature of this attack was to only like posts on the first page. There are 50 posts on the front page, and only those posts were exposed to the excessive affection. We think that it is unlikely that a web crawler would not move on to the second page.

We take periodic database backups. This is a free feature of our Heroku supported database. We were able to restore the data to a state that was prior to the attack without losing any posts. This enabled us to get back to a state where all of our posts had the same number of likes that we had before the attack.

We also made the decision that we wouldn't restore the "Like" functionality if it would allow for the same disruptive behaviour. The "Like" feature is beloved in our organization. For a little bit, we considered removing it for good, and perhaps relying on Twitter or other social media for our endorphonic feedback. Eventually we determined that we wanted users to express appreciation on their terms and not on the terms of third-party gigantic negativity infested platforms.

Our solution was to place a soft rate limit on the "Like" functionality. TIL is an Elixir app which offers some nice in-memory data stores like GenServers and we took advantage of that functionality to create a custom rate limiter.

In the end, this was a slightly annoying, slightly amusing incident that exposed some purposely-left holes in our application that hadn't been exploited in four years. We were able to turn it into something positive by using it as an opportunity to write a simple rate limiter, increasing the scope of TIL as an example application for Elixir and Phoenix.

Thank you for bearing with us and please remember to like a TIL post if you think it deserves recognition!