Viewed 717 Times
Viewed by 454 People
The lack of skilled IT workers is hurting the deployment of emerging technology, according to a new survey from Gartner. In areas from cloud to cybersecurity, this crisis is expected to last for years to come.
man responsible to ensure that 25 million-plus registered users of
Snapdeal.com get to see an individually and intelligently customized
portfolio of offerings on the home page is Amitabh Misra, vice
president, engineering. He and his team of 50 engineers (total IT team
is about 500) write algorithms that soak up user data and present what
they believe are the right products for the right people.
Algorithms have played a crucial role in Snapdeal's sales growth --
35-40% of its sales are driven by algorithm, compared to less than 15% a
"In the real world, a camera buyer won't enter a
shoe shop," says Misra. The simple goal of extremely complex algorithms
is to understand buyer behaviour every nano-second and spew out products
that he is most likely to buy.
From global giants like
Facebook to Amazon, local e-tailers Snapdeal to Flipkart, startups
Zomato to Socialcops, this piece of code -- the now ubiquitous algorithm
-- is becoming one of the most important tools to manage scale of
operations, speed of execution and complexity of human nature.
The more time millions of people use smart devices to search the
internet for jobs, shop online, pay bills, search for information, look
for directions, play virtual games or simply look, the more digital foot
prints they leave behind. Algorithms are relentlessly picking up these
prints, and synthesizing them to present you with intelligently
customized choices. Eventually, algorithms may be able to predict and
preempt your online behaviour even better than yourself.
"35-40% of sales on Snapdeal are driven by the algorithm," says Misra.
"The code is better in predictive marketing than humans."
India's e-commerce market is tipped to grow from $11 billion in FY14 to
$20 billion FY15. Algorithms will set the pace of growth.
Ganesh Ramamoorthy, research vice president, Gartner: "Algorithms are
moving from the back-end (helping companies manage inventory) to the
front-end (helping companies decide what products and services users are
likely to avail of)." Globally, app-based taxi service provider Uber
uses algorithm to determine periods of surge pricing.
interview with ET, before Uber's services were banned in Delhi, Neeraj
Singhal, country manager, India & subcontinent, said: "Using
algorithms, Uber is able to optimize downtime, ensuring drivers get more
trips and also get a better picture of peak demand, to help get more
cabs on roads and ease supply constrain in real time." With Uber's
?staffing policy of only three managers per city, doing this manually
would have been impossible.
Algorithms can process tons of more
data compared to most humans to arrive at smarter decisions. To
understand behaviour of 35 million odd online shoppers manually, an
e-tailer would need a workforce larger than the Indian army and still
not be sure whether they got it right.
"That complexity is best
left to the code," says Rajesh Janey, president India & Saarc, EMC,
a $23 billion storage solutions provider. "Algorithms have the
potential to simplify our task. They have been used by brokers and
traders in stock markets -- to take buy and sell calls in milliseconds.
They are now jumping to consumer areas. It's basically Big Data
analytics in real time, which individual humans won't be capable of
doing," he adds.
Adds Rajeev Rastogi, director, machine
learning, Amazon India: "Information aggregated by algorithm helps
deliver a personalized experience. Retail business needs to order right
amount of quantity and able to predict correctly. Algorithms help us do
At the world's largest social media site Facebook,
algorithms are replacing humans in deciding what news in its News Feed
feature do 1.35 billion global users (110 million in India) get to read.
A user interested in golf might get updates on how Tiger Woods is
faring in his comeback rather than the finer details of Virat Kohli's
back-to-back centuries in the first test match that India lost against
Facebook engineer Greg Marra leads the team behind
its News Feed algorithm. In an interaction with New York Times, Marra
says that a human editor for each individual would be ideal, "but it's
not realistic to do that at scale for every person on the planet." The
goal of Facebook's News Feed algorithm called EdgeRank is to identify
what users most enjoy, and its results vary around the world. For
instance in India, broadly Facebook believes users like to share the
ABCDs or astrology, Bollywood, cricket and divinity.
do helps determine what you see," says a spokesperson from Facebook's
News Feed team. "If you like something, you will see more of it and if
you hide something you will see less of it."
Data enriches the code
Making that decision is the algorithm -- a pre-automated set of
procedures to help achieve a task. But to achieve a goal pushing right
ads to right users, automatically send promotional mails or delivering
the right in formation -- algorithms feed on data.
"We look for
signals that certain content might be interesting to someone -- such as
what kind of stories they have commented on and what stories they have
shared with friends. These signals are proxies for relevance," says the
This data collection is not limited to a
single site that a user goes to -- but scales up into creating users'
social graphs, by picking up data across the users browsing habits. For
instance, you may search for Langkawi on Google for your next holiday,
look at air fares to the destination on Makemytrip and end up getting
more holiday options when you next visit Facebook or get binocular and
beach wear sellers on the home page of Snapdeal.com, with the site's
algorithm anticipating that's the right gear to take to Langkawi.
Says Misra of snapdeal.com: "We generate one terabyte of data daily on
user behaviour, compared to 10GB per day a year back. This data includes
what a user searched for, age, gender, location, what the person does
on a site and so on. The number crunching is done by a proprietary code
and that output makes the experience better each time you visit the
site. Much like search engine works -- matching keywords to what you
might be looking for."
Adds Roy Cherian, co-founder & CEO,
Marketelligent, a data analytics startup: "Algorithms improve your
chances versus random marketing."
At Snapdeal.com, an algorithm
even decides which courier out of the 25-plus courier companies on its
rolls is best suited to deliver a packet to a destination. "This is
based on past behaviour and user rankings (on efficiency of delivering a
packet on time, in proper condition) -- which are matched
automatically. Package routing is via algorithms," says Misra. The
feature viewed is done using algorithms."People with similar profiles
(say oil & gas engineers, 35-40 year old, working on offshore rigs)
will get the same stuff to see. It's an algorithm which matches people
with similar profiles," he adds.
Companies use Hadoop
technology to process Big Data -- developed by a Yahoo engineer in 2005,
Hadoop is an open source software for processing large data-- of the
order of terabyte and petabytes. On top of Hadoop, companies run
specific, proprietary applications or algorithms to achieve specific
output -- like professional networking site LinkedIn uses to match job
openings with candidates who are likely to be interested in it. Or even
credit card companies that use algorithms to detect frauds -- like
tracking unusual behavior based on users' spending history.
Startups need scale to use algorithms
The algorithm works best if the scale is large -- that is data collected is large.
"We are at the beginning stage of use of algorithms. It's level 1 with
accuracy of 45-50%. At level 5 we could get to over 98% accuracy. We
will get to that that accuracy by crunching more and more data," says
Ramamoorthy of Gartner.
Adds Sandeep Ladda, India technology
lead, PricewaterhouseCoopers (PwC): "It is an evolving space that's
helping companies like Facebook deliver a personalized newspaper to
users. Accuracy is high compared to walk in customers at physical
stores, where retailers don't have much clue unless the shopkeeper knows
But accuracy depends on the quantum of data.
That's where start-ups find it difficult to use them in a meaningful
way. "It's early days for us," says Bipin Preet Singh, founder,
Mobikwik, a mobile wallet. "We are collecting data and learning about
user behaviour, transactions and ticket size. But we are investing in
creating those hooks to understand user patterns." Mobikwik founded in
2009 has 10 million users and collects 2-3GB of data daily. Data
collected is small, compared to established e-commerce companies like
Adds Varun Banka, co-founder, socialcops, a Delhi
based startup, "We are getting base data ready in three areas --
education, healthcare and public information. But our scale is small --
about two million data points. The more data you have the better output
will it give." Much like an election exit poll survey -- accuracy
improves as scale of data becomes large -- running into terabytes rather
algorithms manage scale very well -- sending News Feeds to 1.4 billion
Facebook users or providing recommendations based on user profiles to
shoppers online, they may not work in all situations. These include new
product launches, running a flash sale and first time visitors to a
site. Simply because a new product or a fist time visitor has no past
"Every event can't be predicted," says Rastogi of Amazon.
"In stock markets, it's difficult to predict the future based solely on
past data. Algorithms can find general trends and give personal
recommendations. Machines can sift through billions of records in
milliseconds, but can't understand natural language. Eventually, it's a
combination of man and machine."
Adds Ladda: "Limitation is the
human touch. Algorithms address scale issues, but could throw up false
positives." For example a shopping site might send a catalogue of gym
equipment, if you simply browsed on workout stuff, but may not want to
buy the gear.
Says Singh of Mobikwik: "Algorithms won't replace
human judgment -- lot of decisions will be left to manual verification.
For example, what looks like a suspicious activity based on a credit
card users past history, might be a genuine, fresh, first time
transaction. Though, this in turn will be fed back into the machine to
update users' histories."
At LinkedIn, the use of algorithms
takes a blended man-machine approach. "We want to solve a problem at
scale, so the machine part becomes important, but how do you get human
judgment," asks Nishant Rao, managing director, LinkedIn India. "An
algorithm will send content, say an article only to people it believes
might be interested in it. But someone interested in healthcare might
also be interested in cameras and cars.We can't leave everything to
Rao, however, adds algorithms are self-learning,
self-thinking and translate clicks to intelligence. "If I click a lot on
leadership, the algorithm is seeing that I have a higher interest in
leadership topics and pushes relevant content. But to eliminate or
reduce false positive, you need human intervention. There are aspects of
judgment and nuances that systems can't fully comprehend."
The big limitation of algorithm believes Janey of EMC is, "it cannot create but only learn."
In any case, algorithm will be better than a random solution. "The
power we are seeing in new technologies will eventually make the
limitations of algorithms reduce," believes Rao.