Key Points: Short on time? Skip to these points in the video:
We were keen to mention a lot with regard to the UK market this month. Following the UK market overview we have 3 key topics covered as indicated below.
Executive Summary
Market Overview: Tracking Key Players
In the UK, we track the performance of 20 banks and financial institutions by using the Engaged Customer Score, based on online app reviews from Google Play and the App Store. This metric combines ratings and comments to understand user satisfaction and pinpoint issues.
Identifying Budding Issues
Virgin Money's Decline
Virgin Money's Engaged Customer Score dropped from 3.7 in February to 3.0 in April. Analysis reveals that 34% of users rated the app low on Reliability and Stability in April, with feedback on these issues correlating with the decline.
Emerging Concerns
Initial analysis using Touchpoint Group’s Ipiphany platform highlighted key themes in user feedback, including "change bank," indicating potential customer attrition. Comments such as "used to be good, not been able to get into it for a week" and "going to change banks'' emphasise this trend. Early flagging of issues leading to retention risk and churn can enable banks to tackle issues fast and retain more customers.
Third-Party Data Sharing
An emerging issue for Virgin Money involves concerns about third-party data sharing. Users reported being required to agree to data sharing to use the app, causing alarm and dissatisfaction. Most feedback on this issue came in the last week of April. This was aligned with a version release that forced customers to accept the new terms and conditions, and some customers were alarmed to see it included the requirement to share some of their data with third parties.
The terms and conditions relating to third-party data sharing may not have been new at all, but rather, recently picked up and become an issue for app users with the release of the new version of the app and associated request/requirement to read and accept the T’s & C’s.
Having access to the Ipiphany platform, data and reporting would create alerts early and allow Virgin Money to both respond in the comments to address concerns and also develop a communication strategy to mitigate risk around this becoming a bigger issue than it needs to be.
Why Analyse Online App Reviews?
Sensitivity to User Experience Changes
Online app reviews provide sensitive and unsolicited feedback, making them critical for understanding user experience. For instance, a spike in reviews for Santander coincided with the launch of version 5.2.1, highlighting the immediate impact of app updates.
Granular Insights
Analysing reviews by versions and key themes helps identify specific issues. For Santander, the new version saw increased reports of Reliability, Design, and Authentication issues, leading to a drop in the Engaged Customer Score from 3.5 to 2.
Staying Proactive with New App Versions
Santander’s New App Version Elicits Customer Feedback
Santander’s new app version led to a surge in negative feedback, particularly regarding Reliability and Design. Design-related issues, often expected with major updates, were compounded by significant authentication problems, reducing user satisfaction.
Continuous Monitoring and Quick Response
Tracking app performance daily, enables quick identification and resolution of issues. For example, feedback on a recent version highlighted problems like large fonts and missing features, which were promptly addressed to improve user satisfaction.
Conclusion
Regular analysis of online app reviews is essential for maintaining and improving user experience in banking apps. By identifying issues early and responding quickly, banks can enhance customer satisfaction and reduce churn risk.
Video Transcript
00:00:05 - Glenn: Hello there and welcome back to another Touchpoint Group Banking App Insights Session. My name is Glenn Marvin and joining me as usual today is the Touchpoint Group Global head of AI Analytics and CX Strategy, Yazad Karkaria. Thanks for joining us as we delve into the market looking at the month of April 2024.
Now, we have often heard Senior Execs complain that there is a lot of data but not enough insights to act upon, and translating data into insights is a critical component of improving your customer experience and profitability, and that's exactly what we help our clients achieve.
So, today we are going to focus on the market overview followed by three key topics. Firstly, identifying budding issues. Then we're going to move into why analyse online app reviews data and the third subject is how can you stay on top of the game when you're launching new app versions? So, let's pass it over to Yazad and start with the market overview.
00:01:13 - Yazad Thanks, Glenn. So, in the UK market, we are tracking about 20 banks and financial institutes and let's start by taking a quick look at the performance of some of the key players in this market. So, what we are currently looking at, the number that we are tracking is the Engaged Customer Score, which is based on the online app reviews received on Google Play and App Store. What we do is we consider all those reviews where customers or users of the app have given a rating along with a comment. Because that comment then allows us to understand why certain customers are happy or unhappy with certain banking apps. So, let's jump into it.
Firstly, we have Lloyds over here in the market and as we can see, their performance has been quite strong throughout we looking at this since January 2023. There was a brief period starting December 23 to around February 24 when they dropped marginally. But then again, they have upped their game and they're around 4.5. They continue to be the market leaders here. Barclays is another very interesting player. Since November 23, Barclays have really upped their game and we can see they are right up there, very close to Lloyds, in fact, scoring better than them in a month or two in the last few months.
So, Barclays is a close second over here in the set that we are looking at. Another important bank that we have is Capital One. We've covered Capital One in the last few sessions because we saw that their performance has declined since January 2024, but the good news for them is that in April, we are seeing a partial recovery over here, and the score has now moved to about 4. The next bank that we will be looking at is Santander, and this is really an interesting one because, as you can see, in the month of April, there is a sharp, sharp drop in user experience compared to any other months that you have on the screen over here in the last 16 months, and then we have Virgin Money where again, performance has been declining in the last couple of months. But specifically in April, there is something very interesting that we would like to highlight to our viewers.
Now, in order to understand why customers have rated these banks the way they have, we have developed our own proprietary framework using all the experience that we have analysing over a million of millions of app reviews over the last couple of years across 100 banks globally, and what we found is in order for any banking app to perform successfully in a market, it needs to do well on these four foundational pillars. That's App Authentication, Design and UI, App Reliability and Stability and Features and Functionality. So, these are the four main pillars that we track and within each of these pillars, we also have a lot of details in the form of different subtopics. For example, over here, within Features and Functionality, you can see there are a lot of things like, checking account balance, fund transfer, making payments, card management alerts, notifications and so on. This is the framework that we would be using today to analyse the performance of some of our banks.
Firstly, let's go to Virgin Money and take a look at what's happened to Virgin Money in the last couple of months. So, on the screen over here on top, we're seeing how they've performed in terms of their Engaged Customer Score, and we can see that in February they are at 3.7 dropped to 3.4 in March, and in April they've dropped to 3. There has been a gradual decline in the last couple of months. And on the left-hand side over here, you would recollect that these are the same four foundational pillars that we saw a few moments back.
What we are essentially tracking over here are pain points. So any customers who form a part of this percentage over here has shared negative feedback regarding that app. In this case, 34% app users in the month of April have rated Virgin Money low on Reliability and Stability as they've highlighted these issues and the drop in their performance correlates to the increase in Reliability and Authentication related issues. As we can see over here, there has been an increase across these two pain points. Here, on the right-hand side we have a lot of subtopics, as I mentioned, under each of these topics, which give us a much more granular picture and help us understand what are the details going on here.
Now, back to Glenn's point of uncovering budding issues, there's something very interesting that we saw that's going on in the month of April amongst Virgin users. Let's take a quick look. Here, on the right hand side, what we have is a dynamic word cloud which picks up key themes based on user feedback and highlights specific insights. Here, apart from the regular insights that you would expect, like the app is not working, I have to try multiple times. I get certain messages, login problem, etc, there are a couple of topics which are quite, which stand out. The first one over here is change bank, which indicates to me that customers are unhappy and indicating that they would change banks. This is a sign of customer attrition, as we can see over here, and here are just a couple of comments highlighting that the app 'Used to be good. Not been able to get into it for a week... 'We're likely to change banks', same with this customer as well, 'Going to change banks'.
00:06:35 - Glenn: And those are verbatim specific comments from the clients, aren't they, the users of the app? They are word for word from the review stores
00:06:45 - Yazad: Absolutely, and as you saw, Glenn, within just a couple of clicks, we were able to straightaway look at indicators of customer attrition and there's a lot of deep dive that we can just do in that topic as well. But here, the idea is that very quickly we were able to see what's going on. Now, to answer your point of identifying budding issues, one key theme that catches my eye, and I'm curious to understand what's here is this 'third party', what is it that customers are talking about, the third party? We really sort of need to understand what's going on over here because this is something we've not heard of before, and when we take a closer look, we see customers talking about 'if you don't sign up to them being able to take your data from your phone so, they can sell it to third parties, you cannot use the app'.
What customers are saying is that they are being sort of asked to sign up to Virgin being able to share their information with third parties so, they can use the app. This is something that rings alarm bells amongst customers and it's not surprising. I don't think many customers would be happy with that, and that's strongly come true in the feedback that we've seen over here as well.
Now, the interesting thing that I would like to highlight over here, Glenn, is that we have a very well defined framework over here on the left hand side to capture customer feedback, but this third party related feedback which has come through, we don't have a specific topic that we've created because we can't just create a million topics by preempting what customers can say, but very easily and effectively, we have been able to, able to uncover this budding issue over here. Now, one more thing, if I look at carefully, is the date when the reviews have come through. Most of them have come through in the last week of April. This is something specifically that I want to understand a little more in detail.
Instead of looking at this result on a monthly basis, now I'm just going to change the view to look at it for a period of 24th to the 30th of April so, that's what I've highlighted, and now what I would do is I want to get a brief summary or explanation of what is it that customers are talking about the most in this short time period to get a sense of customer sentiment and this decline in user experience. What's happening over here is we've integrated our platform to OpenAI, and OpenAI is currently analysing customer comments to give us that final granular level of insights by classifying information into different themes that insights teams and products teams can straight away read and use. They don't need to go through a lot of comments. They will have direct access to this straight away.
00:09:27 - Glenn: And those comments around third party doesn't necessarily mean that the bank is doing anything underhand. It could actually just highlight a communication disconnect between some of the policy and what people understand. So, these are the sorts of things that if you're aware of, you can get ahead of early.
00:09:46 - Yazad: Absolutely, Glenn. Very rightly pointed out. Here, when I am analysing the results from the period of 24th to the 30th of April, I do see a number of things coming up which are related to app functionality, customer service related complaints, ease of use features, related feedback and the third party related feedback has also come up in this form of privacy concerns. As you can see, several users explain concerns about apps privacy policies, particularly the requirement to agree to data sharing with third parties in order to use the app.
So, Glenn, I think it's been around five minutes that we've looked at this data, and in just that much time, we've been able to quickly identify this budding issue. Once we've set up this analysis, our clients have real time access to this. It's just a few clicks here and a few clicks there, and of course, we help them with this to identify how your app is performing and if there is anything that you need to be mindful of while analysing your data.
00:10:43 - Glenn: Yeah, that's great. Thank you so much for that. Now, can you show us how banks can stay proactive while they're launching a new app?
00:10:52 - Yazad: Sure, definitely. Launching new app versions can be quite tricky because the idea of launching new versions is to improve customer experience and enhance the app itself, but there have been times, in fact quite often, where it does backfire in some way or the other and user experience declines. Now, let's take a look at the example of Santander in this case. So, what I have over here in front of my screen is I have split the number of reviews that Santander customers have posted on a daily basis. Every day we can see that there's a standard number of reviews between 25 and 50 that we have been receiving in the month of April until 23rd of April, when suddenly in a single day it skyrocketed to about 200, and then, it did remain at a higher level compared to the rest of the month subsequently as well.
So, Glenn, with this I would answer another question that we are going to explain to our viewers today, and that is why analyse online app data. The main reason for analysing online app data is because this source of customer feedback is very sensitive to changes in user experience and that's what makes it critical to add this to your insights ecosystem. This feedback is unsolicited, unlike other surveys where you are inviting customers and requesting them to share feedback with you. This is a classic example. Something has happened in the app for sure, and that's why we are seeing a sharp increase in the number of reviews coming through.
My hypothesis over here is that possibly a new version has been launched and when we investigate further, we do see that, yes, exactly at the same time where we saw the spike in the number of reviews. Version 5.2.1, this is the Android version we are looking at, has been launched over here. The number of reviews for other versions were quite consistent earlier, but with the launch of this version, we can see that there is a sudden spike in the number of reviews that we have received.
Now, in order to understand how is this version performing compared to the other versions, we will go back to our pain points analysis and compare version 5.2.1 to one of the earlier versions, one of the recent versions, which was the most prominent one recently, which is 4.30.0.(79). Again, to do this, we are analysing the results on the four foundational pillars, that's Reliability, Authentication, Design and Features, and what you see over here are pain points. So the incidence that gets reported over here is based on the pain points that users have highlighted.
When we look at the earlier version, we can see that about 26% customers reported Reliability related issues, which is still high, and about only 9% spoke about Authentication related issues and 6.4 about Design related issues and only 4% customers spoke about Features and Functionality related issues. Now, in comparison, when you look at this latest version which has been launched, your reliability related issues have increased to 41% Design and UI related issues have also increased three times, from 6% to 20%. Now, this is not very surprising because we have seen this in the past, that when a new version gets launched which has significant amount of Design and UI changes, you do get some negative feedback initially as customers are getting used to the new version and they are trying to find their way around and get to the areas in that they need to get to. So, this is not very surprising, nor is this a major cause of concern.
But the biggest cause of concern over here is that Design related issues, which anyways would bring down the user experience temporarily, have been accompanied also by Authentication related issues. So, Authentication is one of the key drivers. Like Authentication related issues are like the key drivers for your Engaged Customer Score to decline and that is what we can see over here. The score has dropped from 3.5 to just two for the new version, and we can see that Authentication related issues are playing a major factor over here. What we can also do is we can dive a lot deeper into Design and UI and see what is it that customers are talking about.
The first thing that we see that anyone who's given feedback on Design and UI is likely to give you only a score of 1.5 versus the overall average of 3.6 and what we can do is we can look at the key themes coming up over here in Design and UI. You have a lot of customers talking about the new app versus the old app. 'that's an awful update, New update, previous version' We can quite clearly pinpoint that all these comments are mainly regarding the recent update that's taken place. Here are just some customer comments. 'The new app is horrifically unintuitive and waste space with non information forcing you to scroll around to see stuff that's actually important...' This is just one example. There's a lot of detail that we have in this space and also we just don't stop at the four foundational pillars. What we can also do is we can dive much deeper into this and look at the numerous subtopics that we have as well. Here we have things like, Technical Issues, Login Errors, the App being Difficult to Use, Interface Related Feedback, Bugs and Crashes, and the app being confusing. So, there is a lot to go through, Glenn, But I think we've explained in brief what were the key changes that users have experienced in the Santander app, which has taken the user experience down in the month of April.
00:16:25 - Glenn: That's incredibly insightful, Yazad. Especially when you can drill down to those next layers and look at things like that app login area I'm sorry–error, has increased more than fourfold. So it can definitely help you drill down. How can our clients use this to really help them manage their apps and the client experience?
00:16:49 - Yazad: Yeah. So, Glenn, what we've done right now is we've retrospectively done this analysis for Virgin Money and for Santader, and then we've taken a look at the issues which popped up. But with our clients, we track this on a daily basis and work very closely with product and insights teams and we help them prioritise the next steps for the app development based on customer-driven insights. For example, one of our clients recently, the feedback for the new version came through that the app font is too big or they need to scroll around quite a bit to get to some basic information, and the quick balance widget was missing in the new version, etc. So, we get that information to product teams really quickly, it allows them to see what is the impact of these changes on user experience and helps them take quick actions. As a result, some of these changes were rolled back and then customers have in fact expressed their satisfaction that, yes, the bank is listening to their feedback and is making changes accordingly.
00:17:43 - Glenn: That's fantastic. I think the immediacy of information is something that is really unusual where you see it in so many other industries, but not necessarily in the banking industry, where there is quarterly or six monthly customer surveys, but if you wait that long when you're having issues, the churn risk is significant. So, yet again, some great insights. Appreciate you taking the time to walk us through this and if you would like us to host you for a one-on-one insights session on your own bank, feel free to sing out and we're more than happy to walk you through the platform.