How Facebook, Flipkart & others decide what you see
Shelle
717 Times
454 People
The
man responsible to ensure that 25 million-plus registered users of
Snapdeal.com get to see an individually and intelligently customized
portfolio of offerings on the home page is Amitabh Misra, vice
president, engineering. He and his team of 50 engineers (total IT team
is about 500) write algorithms that soak up user data and present what
they believe are the right products for the right people.
Algorithms have played a crucial role in Snapdeal's sales growth -- 35-40% of its sales are driven by algorithm, compared to less than 15% a year back.
"In the real world, a camera buyer won't enter a shoe shop," says Misra. The simple goal of extremely complex algorithms is to understand buyer behaviour every nano-second and spew out products that he is most likely to buy.
From global giants like Facebook to Amazon, local e-tailers Snapdeal to Flipkart, startups Zomato to Socialcops, this piece of code -- the now ubiquitous algorithm -- is becoming one of the most important tools to manage scale of operations, speed of execution and complexity of human nature.
The more time millions of people use smart devices to search the internet for jobs, shop online, pay bills, search for information, look for directions, play virtual games or simply look, the more digital foot prints they leave behind. Algorithms are relentlessly picking up these prints, and synthesizing them to present you with intelligently customized choices. Eventually, algorithms may be able to predict and preempt your online behaviour even better than yourself.
"35-40% of sales on Snapdeal are driven by the algorithm," says Misra. "The code is better in predictive marketing than humans."
India's e-commerce market is tipped to grow from $11 billion in FY14 to $20 billion FY15. Algorithms will set the pace of growth.
Adds Ganesh Ramamoorthy, research vice president, Gartner: "Algorithms are moving from the back-end (helping companies manage inventory) to the front-end (helping companies decide what products and services users are likely to avail of)." Globally, app-based taxi service provider Uber uses algorithm to determine periods of surge pricing.
In an interview with ET, before Uber's services were banned in Delhi, Neeraj Singhal, country manager, India & subcontinent, said: "Using algorithms, Uber is able to optimize downtime, ensuring drivers get more trips and also get a better picture of peak demand, to help get more cabs on roads and ease supply constrain in real time." With Uber's ?staffing policy of only three managers per city, doing this manually would have been impossible.
Algorithms can process tons of more data compared to most humans to arrive at smarter decisions. To understand behaviour of 35 million odd online shoppers manually, an e-tailer would need a workforce larger than the Indian army and still not be sure whether they got it right.
"That complexity is best left to the code," says Rajesh Janey, president India & Saarc, EMC, a $23 billion storage solutions provider. "Algorithms have the potential to simplify our task. They have been used by brokers and traders in stock markets -- to take buy and sell calls in milliseconds. They are now jumping to consumer areas. It's basically Big Data analytics in real time, which individual humans won't be capable of doing," he adds.
Adds Rajeev Rastogi, director, machine learning, Amazon India: "Information aggregated by algorithm helps deliver a personalized experience. Retail business needs to order right amount of quantity and able to predict correctly. Algorithms help us do that."
At the world's largest social media site Facebook, algorithms are replacing humans in deciding what news in its News Feed feature do 1.35 billion global users (110 million in India) get to read. A user interested in golf might get updates on how Tiger Woods is faring in his comeback rather than the finer details of Virat Kohli's back-to-back centuries in the first test match that India lost against Australia.
Facebook engineer Greg Marra leads the team behind its News Feed algorithm. In an interaction with New York Times, Marra says that a human editor for each individual would be ideal, "but it's not realistic to do that at scale for every person on the planet." The goal of Facebook's News Feed algorithm called EdgeRank is to identify what users most enjoy, and its results vary around the world. For instance in India, broadly Facebook believes users like to share the ABCDs or astrology, Bollywood, cricket and divinity.
"What you do helps determine what you see," says a spokesperson from Facebook's News Feed team. "If you like something, you will see more of it and if you hide something you will see less of it."
Data enriches the code
Making that decision is the algorithm -- a pre-automated set of procedures to help achieve a task. But to achieve a goal pushing right ads to right users, automatically send promotional mails or delivering the right in formation -- algorithms feed on data.
"We look for signals that certain content might be interesting to someone -- such as what kind of stories they have commented on and what stories they have shared with friends. These signals are proxies for relevance," says the Facebook spokesperson.
This data collection is not limited to a single site that a user goes to -- but scales up into creating users' social graphs, by picking up data across the users browsing habits. For instance, you may search for Langkawi on Google for your next holiday, look at air fares to the destination on Makemytrip and end up getting more holiday options when you next visit Facebook or get binocular and beach wear sellers on the home page of Snapdeal.com, with the site's algorithm anticipating that's the right gear to take to Langkawi.
Says Misra of snapdeal.com: "We generate one terabyte of data daily on user behaviour, compared to 10GB per day a year back. This data includes what a user searched for, age, gender, location, what the person does on a site and so on. The number crunching is done by a proprietary code and that output makes the experience better each time you visit the site. Much like search engine works -- matching keywords to what you might be looking for."
Adds Roy Cherian, co-founder & CEO, Marketelligent, a data analytics startup: "Algorithms improve your chances versus random marketing."
At Snapdeal.com, an algorithm even decides which courier out of the 25-plus courier companies on its rolls is best suited to deliver a packet to a destination. "This is based on past behaviour and user rankings (on efficiency of delivering a packet on time, in proper condition) -- which are matched automatically. Package routing is via algorithms," says Misra. The feature viewed is done using algorithms."People with similar profiles (say oil & gas engineers, 35-40 year old, working on offshore rigs) will get the same stuff to see. It's an algorithm which matches people with similar profiles," he adds.
Companies use Hadoop technology to process Big Data -- developed by a Yahoo engineer in 2005, Hadoop is an open source software for processing large data-- of the order of terabyte and petabytes. On top of Hadoop, companies run specific, proprietary applications or algorithms to achieve specific output -- like professional networking site LinkedIn uses to match job openings with candidates who are likely to be interested in it. Or even credit card companies that use algorithms to detect frauds -- like tracking unusual behavior based on users' spending history.
Startups need scale to use algorithms
The algorithm works best if the scale is large -- that is data collected is large.
"We are at the beginning stage of use of algorithms. It's level 1 with accuracy of 45-50%. At level 5 we could get to over 98% accuracy. We will get to that that accuracy by crunching more and more data," says Ramamoorthy of Gartner.
Adds Sandeep Ladda, India technology lead, PricewaterhouseCoopers (PwC): "It is an evolving space that's helping companies like Facebook deliver a personalized newspaper to users. Accuracy is high compared to walk in customers at physical stores, where retailers don't have much clue unless the shopkeeper knows him personally."
But accuracy depends on the quantum of data. That's where start-ups find it difficult to use them in a meaningful way. "It's early days for us," says Bipin Preet Singh, founder, Mobikwik, a mobile wallet. "We are collecting data and learning about user behaviour, transactions and ticket size. But we are investing in creating those hooks to understand user patterns." Mobikwik founded in 2009 has 10 million users and collects 2-3GB of data daily. Data collected is small, compared to established e-commerce companies like Snapdeal.com.
Adds Varun Banka, co-founder, socialcops, a Delhi based startup, "We are getting base data ready in three areas -- education, healthcare and public information. But our scale is small -- about two million data points. The more data you have the better output will it give." Much like an election exit poll survey -- accuracy improves as scale of data becomes large -- running into terabytes rather than gigabytes.
Man-machine play
While algorithms manage scale very well -- sending News Feeds to 1.4 billion Facebook users or providing recommendations based on user profiles to shoppers online, they may not work in all situations. These include new product launches, running a flash sale and first time visitors to a site. Simply because a new product or a fist time visitor has no past data.
"Every event can't be predicted," says Rastogi of Amazon. "In stock markets, it's difficult to predict the future based solely on past data. Algorithms can find general trends and give personal recommendations. Machines can sift through billions of records in milliseconds, but can't understand natural language. Eventually, it's a combination of man and machine."
Adds Ladda: "Limitation is the human touch. Algorithms address scale issues, but could throw up false positives." For example a shopping site might send a catalogue of gym equipment, if you simply browsed on workout stuff, but may not want to buy the gear.
Says Singh of Mobikwik: "Algorithms won't replace human judgment -- lot of decisions will be left to manual verification. For example, what looks like a suspicious activity based on a credit card users past history, might be a genuine, fresh, first time transaction. Though, this in turn will be fed back into the machine to update users' histories."
At LinkedIn, the use of algorithms takes a blended man-machine approach. "We want to solve a problem at scale, so the machine part becomes important, but how do you get human judgment," asks Nishant Rao, managing director, LinkedIn India. "An algorithm will send content, say an article only to people it believes might be interested in it. But someone interested in healthcare might also be interested in cameras and cars.We can't leave everything to algorithms."
Rao, however, adds algorithms are self-learning, self-thinking and translate clicks to intelligence. "If I click a lot on leadership, the algorithm is seeing that I have a higher interest in leadership topics and pushes relevant content. But to eliminate or reduce false positive, you need human intervention. There are aspects of judgment and nuances that systems can't fully comprehend."
The big limitation of algorithm believes Janey of EMC is, "it cannot create but only learn."
In any case, algorithm will be better than a random solution. "The power we are seeing in new technologies will eventually make the limitations of algorithms reduce," believes Rao.
The lack of skilled IT workers is hurting the deployment of emerging technology, according to a new survey from Gartner. In areas from cloud to cybersecurity, this crisis is expected to last for years to come.
The
man responsible to ensure that 25 million-plus registered users of
Snapdeal.com get to see an individually and intelligently customized
portfolio of offerings on the home page is Amitabh Misra, vice
president, engineering. He and his team of 50 engineers (total IT team
is about 500) write algorithms that soak up user data and present what
they believe are the right products for the right people.Algorithms have played a crucial role in Snapdeal's sales growth -- 35-40% of its sales are driven by algorithm, compared to less than 15% a year back.
"In the real world, a camera buyer won't enter a shoe shop," says Misra. The simple goal of extremely complex algorithms is to understand buyer behaviour every nano-second and spew out products that he is most likely to buy.
From global giants like Facebook to Amazon, local e-tailers Snapdeal to Flipkart, startups Zomato to Socialcops, this piece of code -- the now ubiquitous algorithm -- is becoming one of the most important tools to manage scale of operations, speed of execution and complexity of human nature.
The more time millions of people use smart devices to search the internet for jobs, shop online, pay bills, search for information, look for directions, play virtual games or simply look, the more digital foot prints they leave behind. Algorithms are relentlessly picking up these prints, and synthesizing them to present you with intelligently customized choices. Eventually, algorithms may be able to predict and preempt your online behaviour even better than yourself.
"35-40% of sales on Snapdeal are driven by the algorithm," says Misra. "The code is better in predictive marketing than humans."
India's e-commerce market is tipped to grow from $11 billion in FY14 to $20 billion FY15. Algorithms will set the pace of growth.
Adds Ganesh Ramamoorthy, research vice president, Gartner: "Algorithms are moving from the back-end (helping companies manage inventory) to the front-end (helping companies decide what products and services users are likely to avail of)." Globally, app-based taxi service provider Uber uses algorithm to determine periods of surge pricing.
In an interview with ET, before Uber's services were banned in Delhi, Neeraj Singhal, country manager, India & subcontinent, said: "Using algorithms, Uber is able to optimize downtime, ensuring drivers get more trips and also get a better picture of peak demand, to help get more cabs on roads and ease supply constrain in real time." With Uber's ?staffing policy of only three managers per city, doing this manually would have been impossible.
Algorithms can process tons of more data compared to most humans to arrive at smarter decisions. To understand behaviour of 35 million odd online shoppers manually, an e-tailer would need a workforce larger than the Indian army and still not be sure whether they got it right.
"That complexity is best left to the code," says Rajesh Janey, president India & Saarc, EMC, a $23 billion storage solutions provider. "Algorithms have the potential to simplify our task. They have been used by brokers and traders in stock markets -- to take buy and sell calls in milliseconds. They are now jumping to consumer areas. It's basically Big Data analytics in real time, which individual humans won't be capable of doing," he adds.
Adds Rajeev Rastogi, director, machine learning, Amazon India: "Information aggregated by algorithm helps deliver a personalized experience. Retail business needs to order right amount of quantity and able to predict correctly. Algorithms help us do that."
At the world's largest social media site Facebook, algorithms are replacing humans in deciding what news in its News Feed feature do 1.35 billion global users (110 million in India) get to read. A user interested in golf might get updates on how Tiger Woods is faring in his comeback rather than the finer details of Virat Kohli's back-to-back centuries in the first test match that India lost against Australia.
Facebook engineer Greg Marra leads the team behind its News Feed algorithm. In an interaction with New York Times, Marra says that a human editor for each individual would be ideal, "but it's not realistic to do that at scale for every person on the planet." The goal of Facebook's News Feed algorithm called EdgeRank is to identify what users most enjoy, and its results vary around the world. For instance in India, broadly Facebook believes users like to share the ABCDs or astrology, Bollywood, cricket and divinity.
"What you do helps determine what you see," says a spokesperson from Facebook's News Feed team. "If you like something, you will see more of it and if you hide something you will see less of it."
Data enriches the code
Making that decision is the algorithm -- a pre-automated set of procedures to help achieve a task. But to achieve a goal pushing right ads to right users, automatically send promotional mails or delivering the right in formation -- algorithms feed on data.
"We look for signals that certain content might be interesting to someone -- such as what kind of stories they have commented on and what stories they have shared with friends. These signals are proxies for relevance," says the Facebook spokesperson.
This data collection is not limited to a single site that a user goes to -- but scales up into creating users' social graphs, by picking up data across the users browsing habits. For instance, you may search for Langkawi on Google for your next holiday, look at air fares to the destination on Makemytrip and end up getting more holiday options when you next visit Facebook or get binocular and beach wear sellers on the home page of Snapdeal.com, with the site's algorithm anticipating that's the right gear to take to Langkawi.
Says Misra of snapdeal.com: "We generate one terabyte of data daily on user behaviour, compared to 10GB per day a year back. This data includes what a user searched for, age, gender, location, what the person does on a site and so on. The number crunching is done by a proprietary code and that output makes the experience better each time you visit the site. Much like search engine works -- matching keywords to what you might be looking for."
Adds Roy Cherian, co-founder & CEO, Marketelligent, a data analytics startup: "Algorithms improve your chances versus random marketing."
At Snapdeal.com, an algorithm even decides which courier out of the 25-plus courier companies on its rolls is best suited to deliver a packet to a destination. "This is based on past behaviour and user rankings (on efficiency of delivering a packet on time, in proper condition) -- which are matched automatically. Package routing is via algorithms," says Misra. The feature viewed is done using algorithms."People with similar profiles (say oil & gas engineers, 35-40 year old, working on offshore rigs) will get the same stuff to see. It's an algorithm which matches people with similar profiles," he adds.
Companies use Hadoop technology to process Big Data -- developed by a Yahoo engineer in 2005, Hadoop is an open source software for processing large data-- of the order of terabyte and petabytes. On top of Hadoop, companies run specific, proprietary applications or algorithms to achieve specific output -- like professional networking site LinkedIn uses to match job openings with candidates who are likely to be interested in it. Or even credit card companies that use algorithms to detect frauds -- like tracking unusual behavior based on users' spending history.
Startups need scale to use algorithms
The algorithm works best if the scale is large -- that is data collected is large.
"We are at the beginning stage of use of algorithms. It's level 1 with accuracy of 45-50%. At level 5 we could get to over 98% accuracy. We will get to that that accuracy by crunching more and more data," says Ramamoorthy of Gartner.
Adds Sandeep Ladda, India technology lead, PricewaterhouseCoopers (PwC): "It is an evolving space that's helping companies like Facebook deliver a personalized newspaper to users. Accuracy is high compared to walk in customers at physical stores, where retailers don't have much clue unless the shopkeeper knows him personally."
But accuracy depends on the quantum of data. That's where start-ups find it difficult to use them in a meaningful way. "It's early days for us," says Bipin Preet Singh, founder, Mobikwik, a mobile wallet. "We are collecting data and learning about user behaviour, transactions and ticket size. But we are investing in creating those hooks to understand user patterns." Mobikwik founded in 2009 has 10 million users and collects 2-3GB of data daily. Data collected is small, compared to established e-commerce companies like Snapdeal.com.
Adds Varun Banka, co-founder, socialcops, a Delhi based startup, "We are getting base data ready in three areas -- education, healthcare and public information. But our scale is small -- about two million data points. The more data you have the better output will it give." Much like an election exit poll survey -- accuracy improves as scale of data becomes large -- running into terabytes rather than gigabytes.
Man-machine play
While algorithms manage scale very well -- sending News Feeds to 1.4 billion Facebook users or providing recommendations based on user profiles to shoppers online, they may not work in all situations. These include new product launches, running a flash sale and first time visitors to a site. Simply because a new product or a fist time visitor has no past data.
"Every event can't be predicted," says Rastogi of Amazon. "In stock markets, it's difficult to predict the future based solely on past data. Algorithms can find general trends and give personal recommendations. Machines can sift through billions of records in milliseconds, but can't understand natural language. Eventually, it's a combination of man and machine."
Adds Ladda: "Limitation is the human touch. Algorithms address scale issues, but could throw up false positives." For example a shopping site might send a catalogue of gym equipment, if you simply browsed on workout stuff, but may not want to buy the gear.
Says Singh of Mobikwik: "Algorithms won't replace human judgment -- lot of decisions will be left to manual verification. For example, what looks like a suspicious activity based on a credit card users past history, might be a genuine, fresh, first time transaction. Though, this in turn will be fed back into the machine to update users' histories."
At LinkedIn, the use of algorithms takes a blended man-machine approach. "We want to solve a problem at scale, so the machine part becomes important, but how do you get human judgment," asks Nishant Rao, managing director, LinkedIn India. "An algorithm will send content, say an article only to people it believes might be interested in it. But someone interested in healthcare might also be interested in cameras and cars.We can't leave everything to algorithms."
Rao, however, adds algorithms are self-learning, self-thinking and translate clicks to intelligence. "If I click a lot on leadership, the algorithm is seeing that I have a higher interest in leadership topics and pushes relevant content. But to eliminate or reduce false positive, you need human intervention. There are aspects of judgment and nuances that systems can't fully comprehend."
The big limitation of algorithm believes Janey of EMC is, "it cannot create but only learn."
In any case, algorithm will be better than a random solution. "The power we are seeing in new technologies will eventually make the limitations of algorithms reduce," believes Rao.
Comments:(0)
Leave a Reply
