{"id":40726,"date":"2020-08-13T10:39:06","date_gmt":"2020-08-13T02:39:06","guid":{"rendered":"https:\/\/shopee.sg\/blog\/?p=40726"},"modified":"2020-08-13T10:39:06","modified_gmt":"2020-08-13T02:39:06","slug":"insider-data-engineering-shopee","status":"publish","type":"post","link":"https:\/\/shopee.sg\/blog\/insider-data-engineering-shopee\/","title":{"rendered":"An Insider Look: Data Engineering at Shopee"},"content":{"rendered":"

We love Data. It\u2019s the fuel of the digital economy, and it is what powers Shopee\u2019s hyper-growth as a regional player in the E-Commerce industry.\u00a0<\/span><\/p>\n

So we met up with Andrew, a Data Engineer in our Shopee Data Engineering team, to get the insider look at the driving force behind our obsession with data.\u00a0<\/span><\/p>\n

\n
\"Shopee
Andrew and the Shopee Data Engineering Team at Work<\/figcaption><\/figure>\n

What\u2019s the Data Engineering team at Shopee all about?<\/span><\/h2>\n

A:<\/i><\/b> We, the Data Engineers, provide data solutions to all our markets and business teams.\u00a0<\/span><\/p>\n

From maintaining our Data Warehouse for all our different Shopee Teams like Business Intelligence and Operations, to building Data Products to monitor high-visibility business operations and campaign performance, we do it all.<\/span><\/p>\n

\"Introduction
Introduction to Shopee Data Platform<\/figcaption><\/figure>\n

Describe the day in a life of a Shopee Data Engineer.<\/span><\/h2>\n

A:<\/i><\/b> Tons of problem-solving, brainstorming and coding!\u00a0<\/span><\/p>\n

In the world of data, there is never a one-size-fits-all solution. Everything we build is specially designed according to the unique business requirements and nature of the data required.\u00a0\u00a0<\/span><\/p>\n

The type of the data needed for each project is never the same, and this affects the decisions we make on the kind of data we use. Considerations that we\u2019re grappling with on a day-to-day basis ranges from the data\u2019s frequency to source location, modelling & structuring choices and even down to how we\u2019d like to present it to our stakeholders.\u00a0<\/span><\/p>\n

\"Andrew
Andrew and the Shopee Data Engineering Team in a Presentation<\/figcaption><\/figure>\n

Can you give us your honest opinion on the challenges you face as a Data Engineer?\u00a0<\/span><\/h2>\n

A:<\/i><\/b> It\u2019s definitely not a walk in the park, especially when we\u2019re dealing with massive amounts of data.\u00a0<\/span><\/p>\n

We\u2019re constantly challenged to think outside the box for new methods of handling our increasing data volume, and better ways to process our data so that we can deliver higher quality products to our users.<\/span><\/p>\n

I think it takes a certain amount of grit and love for data to be a Data Engineer at Shopee. Our role requires both technical expertise, and a good understanding of the business in order to design a complete data solution.\u00a0<\/span><\/p>\n

 <\/p>\n

What\u2019s unique about being a Data Engineer in Shopee?<\/span><\/h2>\n

A:<\/i><\/b> Based on my experience as a Data Engineer in the industry, it\u2019s rare that we\u2019re able to work with as much data as we do here at Shopee \u2014 nearly hundreds of petabytes in the cluster, and billions of records in the database!\u00a0<\/span><\/p>\n

We also make use of open-sourced technologies such as Spark, Hive, Kafka, and Airflow, which is really exciting especially since all of us at Shopee share a passion for advancements in the tech space.\u00a0<\/span><\/p>\n

\"Shopee
Shopee Techademy Training Workshop<\/figcaption><\/figure>\n

What does the Shopee help you improve as a Data Engineer?\u00a0<\/span><\/h2>\n

A: <\/i><\/b>In the Data Engineering team, everyone is focused on improving and growing together. We place a huge emphasis on mentorship initiatives, by making sure that for every new engineer we bring into the team, we develop a custom training plan for them.\u00a0<\/span><\/p>\n

We also frequently organise technical sharings to keep everyone updated of the latest technology developments or best practices in modelling techniques.<\/span><\/p>\n

 <\/p>\n

Do you have any final thoughts to share with us?<\/span><\/h2>\n

A: <\/i><\/b>I like to think that the work we do in Shopee is like playing a game on \u2018Hard\u2019 Mode. However, that makes our success all that much sweeter.\u00a0<\/span><\/p>\n

It\u2019s one thing to be able to work on data projects I\u2019m passionate about, but to see my work create such a direct impact on our users across the region, that\u2019s an opportunity I\u2019ll always be grateful for.\u00a0<\/span><\/p>\n

\"Shopee
The Shopee Data Engineering Team<\/figcaption><\/figure>\n
\n

If you love data as much as us, there\u2019s no better place to show it than here at Shopee.<\/span><\/p>\n

Apply <\/span>here<\/span><\/a> to join the Shopee Data Engineering team today.\u00a0<\/span><\/p>\n

Or read more about #LifeatShopee <\/span>here<\/span><\/a>!\u00a0<\/span> \t<\/p>\n","protected":false},"excerpt":{"rendered":"

We love Data. It\u2019s the fuel of the digital economy, and it is what powers Shopee\u2019s hyper-growth as a regional player in the E-Commerce industry.\u00a0 So we met up with Andrew, a Data Engineer in our Shopee Data Engineering team, to get the insider look at the driving force behind our obsession with data.\u00a0 What\u2019s […]<\/p>\n","protected":false},"author":7,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"jnews-multi-image_gallery":[],"jnews_single_post":[],"jnews_primary_category":[],"jnews_override_bookmark_settings":[],"jnews_social_meta":[],"jnews_review":[],"enable_review":"","type":"","name":"","summary":"","brand":"","sku":"","good":[],"bad":[],"score_override":"","override_value":"","rating":[],"price":[],"jnews_override_counter":[],"jnews_post_split":[],"footnotes":""},"categories":[],"tags":[],"class_list":["post-40726","post","type-post","status-publish","format-standard","hentry"],"yoast_head":"\nAn Insider Look: Data Engineering at Shopee - Shopee Singapore Official Blog<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/shopee.sg\/blog\/insider-data-engineering-shopee\/\" \/>\n<meta property=\"og:locale\" content=\"en_GB\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"An Insider Look: Data Engineering at Shopee - Shopee Singapore Official Blog\" \/>\n<meta property=\"og:description\" content=\"We love Data. It\u2019s the fuel of the digital economy, and it is what powers Shopee\u2019s hyper-growth as a regional player in the E-Commerce industry.\u00a0 So we met up with Andrew, a Data Engineer in our Shopee Data Engineering team, to get the insider look at the driving force behind our obsession with data.\u00a0 What\u2019s […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/shopee.sg\/blog\/insider-data-engineering-shopee\/\" \/>\n<meta property=\"og:site_name\" content=\"Shopee Singapore Official Blog\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/ShopeeSingapore\" \/>\n<meta property=\"article:published_time\" content=\"2020-08-13T02:39:06+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/shopee.sg\/blog\/wp-content\/uploads\/2020\/08\/DSC_2242-300x200.jpg\" \/>\n<meta name=\"author\" content=\"Shopee People Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Shopee People Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Estimated reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"An Insider Look: Data Engineering at Shopee - Shopee Singapore Official Blog","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/shopee.sg\/blog\/insider-data-engineering-shopee\/","og_locale":"en_GB","og_type":"article","og_title":"An Insider Look: Data Engineering at Shopee - Shopee Singapore Official Blog","og_description":"We love Data. It\u2019s the fuel of the digital economy, and it is what powers Shopee\u2019s hyper-growth as a regional player in the E-Commerce industry.\u00a0 So we met up with Andrew, a Data Engineer in our Shopee Data Engineering team, to get the insider look at the driving force behind our obsession with data.\u00a0 What\u2019s […]","og_url":"https:\/\/shopee.sg\/blog\/insider-data-engineering-shopee\/","og_site_name":"Shopee Singapore Official Blog","article_publisher":"https:\/\/www.facebook.com\/ShopeeSingapore","article_published_time":"2020-08-13T02:39:06+00:00","og_image":[{"url":"https:\/\/shopee.sg\/blog\/wp-content\/uploads\/2020\/08\/DSC_2242-300x200.jpg","type":"","width":"","height":""}],"author":"Shopee People Team","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Shopee People Team","Estimated reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/shopee.sg\/blog\/insider-data-engineering-shopee\/","url":"https:\/\/shopee.sg\/blog\/insider-data-engineering-shopee\/","name":"An Insider Look: Data Engineering at Shopee - Shopee Singapore Official Blog","isPartOf":{"@id":"https:\/\/shopee.sg\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/shopee.sg\/blog\/insider-data-engineering-shopee\/#primaryimage"},"image":{"@id":"https:\/\/shopee.sg\/blog\/insider-data-engineering-shopee\/#primaryimage"},"thumbnailUrl":"https:\/\/shopee.sg\/blog\/wp-content\/uploads\/2020\/08\/DSC_2242-300x200.jpg","datePublished":"2020-08-13T02:39:06+00:00","author":{"@id":"https:\/\/shopee.sg\/blog\/#\/schema\/person\/d4002fcc1d52f71f9ce6ff112f6881ff"},"breadcrumb":{"@id":"https:\/\/shopee.sg\/blog\/insider-data-engineering-shopee\/#breadcrumb"},"inLanguage":"en-GB","potentialAction":[{"@type":"ReadAction","target":["https:\/\/shopee.sg\/blog\/insider-data-engineering-shopee\/"]}]},{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/shopee.sg\/blog\/insider-data-engineering-shopee\/#primaryimage","url":"","contentUrl":""},{"@type":"BreadcrumbList","@id":"https:\/\/shopee.sg\/blog\/insider-data-engineering-shopee\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/shopee.sg\/blog\/"},{"@type":"ListItem","position":2,"name":"An Insider Look: Data Engineering at Shopee"}]},{"@type":"WebSite","@id":"https:\/\/shopee.sg\/blog\/#website","url":"https:\/\/shopee.sg\/blog\/","name":"Shopee Singapore Official Blog","description":"Lifestyle & Shopping Recommendations by locals, for locals","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/shopee.sg\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-GB"},{"@type":"Person","@id":"https:\/\/shopee.sg\/blog\/#\/schema\/person\/d4002fcc1d52f71f9ce6ff112f6881ff","name":"Shopee People Team","image":{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/shopee.sg\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/28c406737aabe98bc5695a48149147a31160a724eb7b089cefd6e563ed9b786b?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/28c406737aabe98bc5695a48149147a31160a724eb7b089cefd6e563ed9b786b?s=96&d=mm&r=g","caption":"Shopee People Team"},"description":"Bringing you our people and their stories about Life at Shopee! Get to know what our teams do, how we grow in a vibrant and collaborative environment, and how we make an impact across the globe.","sameAs":["http:\/\/careers.shopee.com","www.linkedin.com\/company\/shopee\/"],"url":"https:\/\/shopee.sg\/blog\/author\/shopee-people-team\/"}]}},"_links":{"self":[{"href":"https:\/\/shopee.sg\/blog\/wp-json\/wp\/v2\/posts\/40726","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/shopee.sg\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/shopee.sg\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/shopee.sg\/blog\/wp-json\/wp\/v2\/users\/7"}],"replies":[{"embeddable":true,"href":"https:\/\/shopee.sg\/blog\/wp-json\/wp\/v2\/comments?post=40726"}],"version-history":[{"count":0,"href":"https:\/\/shopee.sg\/blog\/wp-json\/wp\/v2\/posts\/40726\/revisions"}],"wp:attachment":[{"href":"https:\/\/shopee.sg\/blog\/wp-json\/wp\/v2\/media?parent=40726"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/shopee.sg\/blog\/wp-json\/wp\/v2\/categories?post=40726"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/shopee.sg\/blog\/wp-json\/wp\/v2\/tags?post=40726"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}