• PRODUCTS
    • Radarr
    • Command Center
    • Sentiment API
  • PLANS
  • RESOURCES
    • Reports & Whitepapers
    • Webinars
    • Radarr Blog
  • COMPANY
    • About Us
    • Careers
    • Media Coverage
  • SUPPORT
    • System Status
    • Contact Us
  • ACADEMY
    • Radarr Academy
  • TOOLS & API
    • Sentiment API
  • BOOK A DEMO
  • LOGIN/SIGNUP

Growth of 20/Twenty Social Monitoring & Intelligence Platform – Under the Hood

  January 4, 2017
  rbhamidi
  Media Monitoring | Social Media Monitoring | Team | Technology
Check out how the 20/Twenty Social Monitoring platform has grown!

Growth of 20/Twenty Social Media Monitoring & Intelligence Platform – Under the Hood

We have often been asked about the challenges we faced in scaling up our technology stack to manage big data. I have attempted to address this in this post which is the first of a series of blog posts on this and similar topics.

Radarr was created ground-up as the most intuitive and easy to use cloud based (SaaS) Social Media Monitoring & Intelligence platform in the world.   Based on our deep understanding of what marketers needed and the awesome designs we created, we signed up our first client even before the product was officially launched. The pressure to quickly deliver the first version of the product was intense 🙂

From an engineering point of view, there’s a huge amount of data that we pull (Think Big Data!), process, augment and then visualize in the platform all on a near real-time basis. Imagine someone tweeting and it appears on our platform within a few seconds along with augmented information including Gender, Sentiment, Engagement, Spam score etc.

The evolution of Radarr has already seen a few stages of growth. The graph below shows how 20/Twenty data has grown over the last 2 years since our product launch. This is a really cool growth for a startup like Circus Social both from a business perspective as well as from an engineering standpoint. We used several tricks from the books as well as a few practical hacks to ensure our ability to fetch, process, augment and visualize high volumes of data continued to become better, though this journey was not without pain!

social-intelligence-Radarr-big-data-growth

Stage 1

We created over 200 custom marketing applications in our previous avatar at Circus Social working with some of the biggest brands in the world. We used the same open-source technologies (PHP / MySQL) to create the first version of 20/Twenty. This worked well and as our data grew in the first few months, we continued to grow vertically by adding more capacity (CPU/RAM).

Most of the queries from the application were read queries whereas a bulk of “write operations” were being performed by our data crawlers. We, therefore, created an efficient master-slave architecture where the application would read from the slaves and the crawler scripts would write into the master. This worked well in general but the exponential increase in the volume of data meant that certain queries were running extremely slow and impacting the user experience.

Stage 2

Since our data volume was growing exponentially and the relational aspects of the database were not the core of our application, we realized that sooner or later, we would have to move to a NoSQL database. However, the performance issues that were cropping up had to be sorted quickly and without downtime. We quickly realized that we needed a dedicated search engine and MySQL was not good enough for this purpose.

We explored several options and Elasticsearch came to our rescue here. Elasticsearch is a distributed, RESTful search and analytics engine that centrally stores your data in a manner which can be retrieved/read really fast by your applications. Our awesome tech team deployed this in a matter of days. The improvement in performance was remarkable. The plan worked and we cheered!

Stage 3

Word spread in Singapore and Asia about how good our platform was (and our sales team did a good job too!) and we continued to sign up new clients. The volume of data continued to grow for existing clients as well as new clients. The tech stack of MySQL and Elastic Search did not let us down but we wanted to create an architecture that would scale infinitely if there’s a thing like that.

In Stage 3, we moved the core of our database from MySQL to Cassandra (Elasticsearch was now interacting with Cassandra) and the backend code from PHP to Node.js. We also migrated most of our front-end code to Angular.js for better performance. This was a major architectural change on a live application being used by several clients so we created a parallel production-like environment and ran it parallelly for several weeks to ensure everything was working as desired before switching over.

20/Twenty Social Monitoring and Listening Platform Blog PostWhile we did the above, we continued to work on cool new features on the product and opened up our data apis to a few clients who wanted a deeper integration with their own applications. Other tools we used during this and other stages were Postman, Github and JIRA.

As we scale further from here, we will probably have newer and more exciting technology challenges and we will keep posting about them. If you are excited to work on some of these, do write to us at joinus@www.radarr.com

Recent Posts

View All Posts

CASE STUDY
Data is nothing but numbers if you are not able to weave a story and create meaningful insights.
Scroll down to view and download our free reports and whitepapers and learn how we utilize Radarr to give you insights into industry trends, social media happenings and much more.
Download Case Studies
how to find the right influencers
Find the Right Influencers: 10 Steps to Find the Right Influencer for Your Campaign
| Influencer Marketing

Struggling to increase your campaign reach? Find the right influencer. With social commerce gaining immense popularity over the past

social media engagement KPIs
Which of These KPIs Demonstrate Social Media Engagement?
| Social Media Marketing

Tracking the right KPIs helps you evaluate your content better and takes you closer to your social media goals.

influencer marketing campaigns
How to Run Influencer Marketing Campaigns: Best Practices and Examples
| Influencer Marketing

Learn more about the best influencer marketing campaigns and how to run one yourself. Influencer marketing is a method

tiktok growth strategy and tiktok algorithm
TikTok Growth Strategy – TikTok Algorithm And Strategies
| Social Media Marketing

Tiktok is one of the fastest-growing apps in history. It’s currently valued at $50 billion and has over 1

tiktok marketing statistics
23 TikTok Marketing Statistics You Need to Know in 2022
| Social Media Marketing

List of all the important TikTok marketing statistics you need to know. Whether you love it or hate it,

online reputation management
Online Reputation Management: What It Is, Why It Is Important and Best Practices to Follow
| ORM / Online Reputation Management

Whether you’re a small business or a Fortune 500 company, it’s important to have a good online reputation. Your

linkedin marketing statistics
37 LinkedIn Marketing Statistics You Need to Know in 2022
| Social Media Marketing

Planning to use LinkedIn for marketing and advertising? Here are the LinkedIn marketing statistics to know of. To market

guide to using youtube hashtags
Ultimate Guide to Using YouTube Hashtags to Grow Your Channel
| Social Media Marketing

Learn how to use YouTube hashtags to grow on the video streaming platform.  Did you know YouTube attracts 14.3

attribution model for social media marketing
What Is an Attribution Model for Social Media Marketing and Why Does It Matter?
| Social Media Analytics, Social Media Marketing

Struggling to understand if social media really works? Introducing attribution model. When using social media platforms for marketing, businesses

what is marketing analytics
Marketing Analytics: What It Is and Why It Matters for Fast-scaling Brands
| Market Analysis

Just like diamonds are said to be a girl’s best friend, we call marketing analytics a marketer’s best friend. 

Radarr Newsletter

Become part of our list for updates and get first dibs on free industry reports. Sign up today!

Radarr

Radarr Command Center

Radarr Sentiment API

Copyright Radarr 2021

Privacy Policy

Terms of Use