• PRODUCTS
    • Listen: Social and Digital Intelligence
    • Respond: Social Media Response
    • Analyze: Owned Page Analytics
    • Publish: Social Media Publishing
    • Social Studio Alternative
    • Command Center
    • Sentiment API
  • PRICING
  • RESOURCES
    • Reports & Whitepapers
    • Premium Reports
    • Webinars
    • Radarr Blog
    • Radarr Academy
  • COMPANY
    • About Us
    • Careers
    • Media Coverage
  • SUPPORT
    • System Status
    • Contact Us
  • FREE 7-DAY TRIAL
  • BOOK A DEMO

Growth of 20/Twenty Social Monitoring & Intelligence Platform – Under the Hood

  January 4, 2017
  rbhamidi
  Media Monitoring | Social Media Monitoring | Team | Technology
Check out how the 20/Twenty Social Monitoring platform has grown!

Growth of 20/Twenty Social Media Monitoring & Intelligence Platform – Under the Hood

We have often been asked about the challenges we faced in scaling up our technology stack to manage big data. I have attempted to address this in this post which is the first of a series of blog posts on this and similar topics.

Radarr was created ground-up as the most intuitive and easy to use cloud based (SaaS) Social Media Monitoring & Intelligence platform in the world.   Based on our deep understanding of what marketers needed and the awesome designs we created, we signed up our first client even before the product was officially launched. The pressure to quickly deliver the first version of the product was intense 🙂

From an engineering point of view, there’s a huge amount of data that we pull (Think Big Data!), process, augment and then visualize in the platform all on a near real-time basis. Imagine someone tweeting and it appears on our platform within a few seconds along with augmented information including Gender, Sentiment, Engagement, Spam score etc.

The evolution of Radarr has already seen a few stages of growth. The graph below shows how 20/Twenty data has grown over the last 2 years since our product launch. This is a really cool growth for a startup like Circus Social both from a business perspective as well as from an engineering standpoint. We used several tricks from the books as well as a few practical hacks to ensure our ability to fetch, process, augment and visualize high volumes of data continued to become better, though this journey was not without pain!

social-intelligence-Radarr-big-data-growth

Stage 1

We created over 200 custom marketing applications in our previous avatar at Circus Social working with some of the biggest brands in the world. We used the same open-source technologies (PHP / MySQL) to create the first version of 20/Twenty. This worked well and as our data grew in the first few months, we continued to grow vertically by adding more capacity (CPU/RAM).

Most of the queries from the application were read queries whereas a bulk of “write operations” were being performed by our data crawlers. We, therefore, created an efficient master-slave architecture where the application would read from the slaves and the crawler scripts would write into the master. This worked well in general but the exponential increase in the volume of data meant that certain queries were running extremely slow and impacting the user experience.

Stage 2

Since our data volume was growing exponentially and the relational aspects of the database were not the core of our application, we realized that sooner or later, we would have to move to a NoSQL database. However, the performance issues that were cropping up had to be sorted quickly and without downtime. We quickly realized that we needed a dedicated search engine and MySQL was not good enough for this purpose.

We explored several options and Elasticsearch came to our rescue here. Elasticsearch is a distributed, RESTful search and analytics engine that centrally stores your data in a manner which can be retrieved/read really fast by your applications. Our awesome tech team deployed this in a matter of days. The improvement in performance was remarkable. The plan worked and we cheered!

Stage 3

Word spread in Singapore and Asia about how good our platform was (and our sales team did a good job too!) and we continued to sign up new clients. The volume of data continued to grow for existing clients as well as new clients. The tech stack of MySQL and Elastic Search did not let us down but we wanted to create an architecture that would scale infinitely if there’s a thing like that.

In Stage 3, we moved the core of our database from MySQL to Cassandra (Elasticsearch was now interacting with Cassandra) and the backend code from PHP to Node.js. We also migrated most of our front-end code to Angular.js for better performance. This was a major architectural change on a live application being used by several clients so we created a parallel production-like environment and ran it parallelly for several weeks to ensure everything was working as desired before switching over.

20/Twenty Social Monitoring and Listening Platform Blog PostWhile we did the above, we continued to work on cool new features on the product and opened up our data apis to a few clients who wanted a deeper integration with their own applications. Other tools we used during this and other stages were Postman, Github and JIRA.

As we scale further from here, we will probably have newer and more exciting technology challenges and we will keep posting about them. If you are excited to work on some of these, do write to us at joinus@www.radarr.com

Recent Posts

View All Posts

Trusted by 1000+ brands

  • Book A Demo with Radarr

Book A Demo

Name

Company Name

Work Email

Job Title

Phone Number

Thank you for contacting us!
We will be in touch with you shortly.
storytelling marketing examples
16 Unforgettable Storytelling Marketing Examples
| Uncategorized

Learn from the best storytelling marketing campaigns. Every marketer, at some point, realizes the importance of a good story.

social media post ideas - radarr
100+ Social Media Post Ideas and Examples for 2023
| Social Media Marketing

100+ Social Media Post Ideas and Examples for 2023  “Content is fire and social media is gasoline,” said Jay

improve social media response time
Best Practices and Hacks to Follow to Improve Social Media Response Time
| Social Media Marketing

List of underrated tips to improve social media response time. Social media channels continue to become platforms for sharing

social media publishing frequency
How to Find the Right Social Media Publishing Frequency
| Social Media Marketing

Learn how to find the social media publishing frequency best suited for your brand. The power of social media

should you use a social media scheduler
Should You Be Publishing via a Social Media Scheduler? Here’s Everything You Need to Know
| Social Media Marketing

See why we think you should use a social media scheduler to manage accounts better. In today’s digital age,

gopro social media strategy for marketing
GoPro Social Media Strategy Breakdown and What You Can Learn From Them
| Social Media Marketing

Learn from GoPro social media strategy for better marketing. Social media has become an essential tool for businesses looking

guide to metaverse marketing by radarr
Metaverse Marketing – Everything You Need to Know!
| Social Media Marketing

Learn all about metaverse marketing and why you should get started now. The world of marketing is changing. Virtual

best year in review campaigns
Best Year in Review Campaigns: What Your Brand Can Learn and How to Create Yours
| Social Media Marketing

Learn from the best Year in Review campaigns. Do you know what’s better than having a great year? Having

Demo Post Testing
| Uncategorized

social insights from world cup 2022 finals
Social Media Insights from FIFA World Cup 2022: WHAT A FINAL!
| Market Analysis, Social Listening

Radarr Newsletter

Become part of our list for updates and get first dibs on free industry reports. Sign up today!

Radarr

Radarr Command Center

Radarr Sentiment API

Copyright Radarr 2021

Privacy Policy

Terms of Use