About the role
SamKnows is the global leader in internet performance measurement. We’ve been collecting data since 2008 on all manner of metrics, from how good Netflix performance is to figuring out why Fortnite isn’t working for gamers.
We're growing fast and, particularly over the past couple of years, have already undergone a period of exponential growth. This is especially true when it comes to the sheer volume of internet performance data we’re generating around the world. In anticipation of this sudden data increase, our engineers rebuilt our cloud-based data platform from scratch, now called SamKnows One. SamKnows One confidently handles and visualises the data gathered by over 30 million measurement agents every day, with this number continuing to rise. We're looking for a full-time Data Engineer to come and join our team in our London office, to help us make the most of all our data.
- Work closely with engineers and key business stakeholders to ensure we are getting the best out of our data.
- Monitor and ensure data integrity across our ‘big data’ platform.
- Identify and implement cluster tuning to improve big data cluster performance.
- Investigate and provide innovative solutions for our data ingestion pipelines.
- Forecast and plan for data scaling to perform in line with business growth.
- Maintain and integrate new data structures for our wide range of measurements.
- Ensure the continued accuracy of reports and analytics, that we provide to our clients, on a daily basis.
- Hadoop, Hive and Presto - Our big data platform
- Java - Data collection infrastructure
- PHP - Application API layer and current analytics layer
- Mysql - Business metadata stores
- RabbitMQ - Messaging platform
- Improving the internet excites you.
- You are a proactive, problem solver.
- You have a strong background in Data Engineering and experience of working with very large data sets.
- You’re keen to learn more about new technologies.
- You have experience managing Hadoop and Hive plus ideally some experience with Presto, PHP and Java.