An Insider Look: Data Engineering at Shopee

We love Data. It’s the fuel of the digital economy, and it is what powers Shopee’s hyper-growth as a regional player in the E-Commerce industry. 

So we met up with Andrew, a Data Engineer in our Shopee Data Engineering team, to get the insider look at the driving force behind our obsession with data. 

What’s the Data Engineering team at Shopee all about?

A: We, the Data Engineers, provide data solutions to all our markets and business teams. 

From maintaining our Data Warehouse for all our different Shopee Teams like Business Intelligence and Operations, to building Data Products to monitor high-visibility business operations and campaign performance, we do it all.

Describe the day in a life of a Shopee Data Engineer.

A: Tons of problem-solving, brainstorming and coding! 

In the world of data, there is never a one-size-fits-all solution. Everything we build is specially designed according to the unique business requirements and nature of the data required.  

The type of the data needed for each project is never the same, and this affects the decisions we make on the kind of data we use. Considerations that we’re grappling with on a day-to-day basis ranges from the data’s frequency to source location, modelling & structuring choices and even down to how we’d like to present it to our stakeholders. 

Can you give us your honest opinion on the challenges you face as a Data Engineer?

A: It’s definitely not a walk in the park, especially when we’re dealing with massive amounts of data. 

We’re constantly challenged to think outside the box for new methods of handling our increasing data volume, and better ways to process our data so that we can deliver higher quality products to our users.

I think it takes a certain amount of grit and love for data to be a Data Engineer at Shopee. Our role requires both technical expertise, and a good understanding of the business in order to design a complete data solution. 

What’s unique about being a Data Engineer in Shopee?

A: Based on my experience as a Data Engineer in the industry, it’s rare that we’re able to work with as much data as we do here at Shopee — nearly hundreds of petabytes in the cluster, and billions of records in the database! 

We also make use of open-sourced technologies such as Spark, Hive, Kafka, and Airflow, which is really exciting especially since all of us at Shopee share a passion for advancements in the tech space. 

What does the Shopee help you improve as a Data Engineer?

A: In the Data Engineering team, everyone is focused on improving and growing together. We place a huge emphasis on mentorship initiatives, by making sure that for every new engineer we bring into the team, we develop a custom training plan for them. 

We also frequently organise technical sharings to keep everyone updated of the latest technology developments or best practices in modelling techniques.

Do you have any final thoughts to share with us?

A: I like to think that the work we do in Shopee is like playing a game on ‘Hard’ Mode. However, that makes our success all that much sweeter. 

It’s one thing to be able to work on data projects I’m passionate about, but to see my work create such a direct impact on our users across the region, that’s an opportunity I’ll always be grateful for. 

If you love data as much as us, there’s no better place to show it than here at Shopee.

Apply here to join the Shopee Data Engineering team today. 

Or read more about #LifeatShopee here!