Semantic similarity for domain-specific chatbots: Evaluation and comparison with human-level accuracies

Abstract

The overall market for conversational platforms in fragmented and overcrowded. There are around 1000 to 1500 vendors available in the market as per Gartner [1]. Many of them are local vendors providing services to a specific demographic region. The high dependence on linguistics is the major constraint and local vendors might have better services than global players. With the focus shifting from manual intervention reduction and cost reduction to customer experience [2], the underlying technology and vendor landscape will have significant changes. The global conversational AI market size is expected to grow from USD 4.8 billion in 2020 to USD 13.9 billion by 2025, at a Compound Annual Growth Rate (CAGR) of 21.9% during the forecast period [3]. According to Gartner [4], the conversation AI (Virtual Customer Assistance/ VCA) market would grow between 25% to 45% over the next three years. As per the report [4], the VCA market exceeded $1 billion mark in 2018. From the deployment at regional perspective, the American region is biggest with 47% respondents. APAC region has lowest number of deployments (10%) [4], but is set to grow the fastest over the next 3 to 5 years [3]. According to Gartner’s report [4], 31% CIO said they have already deployed some form of conversational platforms in 2019. Majority (82%) of these deployments were in cloud. Gartner expects that by 2021, 1 in 6 customer service interactions would involve AI based platforms. Gartner predicts that 40% of chatbot application deployed in 2018 will be abandoned by 2020. This shows the low accuracy and oversimplification of currently existing chatbots. Many VCAs deployed are transaction based and the market has majority of low-end VCAs. Gartner puts VCA at the trough of disillusionment on the Gartner hype cycle and expects that the technology will have more informed buyers going into the future. From the applications perspective, 67% of respondents in Gartner’s research mentioned customer service as the most important application at present with 60% of them mentioning Sales and Marketing as high priority for future. Sectors like Banking, Communications, Insurance, Utilities and Hospitality have deployed the most VCAs. Healthcare and Retail are important from future’s perspective.