Grokking Engineering - Data Analytics Infrastructure at Viki - Huy Nguyen

Editor's Notes

• #2: Theme of the talk: We have all this data at viki, how do we collect it, make it usable and then derive biz value from it. Other cool technologies that we use.
• #5: What are the predominant countries of origin for our content (Korea)? How many Registered Users do we have and what is the growth trajectory? What videos are translated into what languages and what is the percent completion rate? What are our top performing shows (video starts) in the last 3 months? By geographic region? What are the sources of our Registered Users (how do they find us, embed referral traffic, Google search, etc.)? What videos are translated into what languages and what are the corresponding consumption patterns?
• #6: What are the predominant countries of origin for our content (Korea)? How many Registered Users do we have and what is the growth trajectory? What videos are translated into what languages and what is the percent completion rate? What are our top performing shows (video starts) in the last 3 months? By geographic region? What are the sources of our Registered Users (how do they find us, embed referral traffic, Google search, etc.)? What videos are translated into what languages and what are the corresponding consumption patterns?
• #8: Website database tables: a user registration web flow would persist information provided by the user in a PostgreSQL database table. Accounting data: highly transactional data integrity requirements are high, Enterprise Resource Planning (ERP) and General Ledger (GL) accounting systems. where the structure of the data is predefined and fixed and the integrity of which can be reasonably relied upon. s of which can be distributed in a Hadoop cluster
• #11: Add example of an event JSON
• #12: Simple? Client libraries We collect the events and put them in a queue Structured and unstructured data
• #13: Simple? Client libraries We collect the events and put them in a queue Structured and unstructured data
• #14: Simple? Client libraries We collect the events and put them in a queue Structured and unstructured data
• #16: Centralizing All Data Sources Data Cleanliness Data Transformation Managing Job Dependencies
• #17: Centralizing All Data Sources Data Cleanliness Data Transformation Managing Job Dependencies
• #18: To effectively run queries on our data, we need to bring all the data into the same database. In this case we choose Postgres since all our databases are already in Postgres. Anyone here knows Postgres? It’s like mysql, but it’s better. We’ve built command line tools to copy tables from database to database. So the following command copies all tables in public schema of gaia database to our analytics database, and give them a separate schema. In PG, what schema means is something like namespace for tables.
• #19: Take a look at 1 sample event being stored in Hadoop in semi-structured JSON form, you have a video play event for that video id running on an ipad device, coming from an autoplay feature, from Toronto, Ontario, Canada. That’s a hell lot of dimensions. We want to aggregate and select a subset of dimensions to port into PG. The Hadoop Provider we uses (Treasure Data) has a feature that allows you to specify a destination data storage (in this case Postgres), it’ll execute the Hadoop job and write the results into the selected database. It’s the equivalent of using Sqoop to bulk export data into Postgres.
• #21: As we develop, our data changes, we make mistake, we forgot to set a variable somewhere, we change our data structure. So the new data gets mixed up with the old data. And to make meaningful, and the simple query becomes not so simple.
• #24: Centralizing All Data Sources Data Cleanliness Data Transformation Managing Job Dependencies
• #25: Once all our data are in Postgres, we start to perform transformation/aggregation to them, depending on various different purposes.
• #26: For example, to reduce the size of the table to serve them on a web UI front-end, we aggregate the data further. In this example, we’re dropping the video_id dimension, thus grouping the 2 records together as a new record with the cnt field total to 20. And that reduces the table size.
• #27: For example, to reduce the size of the table to serve them on a web UI front-end, we aggregate the data further. In this example, we’re dropping the video_id dimension, thus grouping the 2 records together as a new record with the cnt field total to 20. And that reduces the table size.
• #28: For example, to reduce the size of the table to serve them on a web UI front-end, we aggregate the data further. In this example, we’re dropping the video_id dimension, thus grouping the 2 records together as a new record with the cnt field total to 20. And that reduces the table size.
• #29: We also chunk our data tables by month, so that when the new month comes, you don’t touch the old months’ data. This also reduce the index size and make it easier to archive your old data. When we first implemented this, we didn’t know how to query cross-month, so we have to write complicated query (like UNION), sometimes we even have to load the data into memory and process them. But then we found out out this awesome feature in Postgres called Table Inheritance. It lets you define a parent table with a bunch of children. And you just need to query the parent table, and depending on your query, it’ll find out the correct children tables to hit.
• #30: Centralizing All Data Sources Data Cleanliness Data Transformation Managing Job Dependencies
• #31: Can anyone tell me what this means? Ok no one can. That’s exactly my point. At some point, our daily job workflow grew so complicated that it’s becoming hard to use crontab to manage them.
• #32: Can anyone tell me what this means? Ok no one can. That’s exactly my point. At some point, our daily job workflow grew so complicated that it’s becoming hard to use crontab to manage them.
• #36: There is too many reports! I want to see the high level metrics all in one place
• #37: Enabling the product and business folks to “write” their own queries
• #38: How do you process 1k events a second? Scalable and distributed Guaranteed Message Passing Fault Tolerant
• #39: How do you process 1k events a second? Scalable and distributed Guaranteed Message Passing Fault Tolerant
• #41: We try to detect peaks and valleus in our real time data, and send out alerts every hour if any.
• #43: We are always exploring new technologies and finding the best tool for the job. The real reason is we get bored ;) I wasn’t hired as a rails developer, I was hired as a developer 