Market Overview
Touchpoint Group’s Global Head of AI Analytics and CX Strategy, Yazad Karkaria provided an overview of the performance of several prominent US banks based on their Engaged Customer Scores. These scores are derived from ratings and comments on Google Play and the App Store.
M&T Bank: A Significant Decline in Performance
M&T Bank's performance drop was a primary focus of the session. To understand this decline the analysis leveraged Touchpoint Group’s Ipiphany platform - focussing on 4 key areas: Authentication, Design and UI, Reliability, and Features and Functionality.
Analysis of M&T Bank’s Decline
The number of app reviews for M&T Bank saw a sharp increase in April, rising from 132 in March to 631. This spike indicated underlying issues needing further investigation. The Engaged Customer Score for M&T Bank dropped from 2.5 in March to 1.5 in April, highlighting significant customer dissatisfaction.
Key Issues Identified:
1. Reliability and Stability: 77% of reviews mentioned issues in this area
2. Authentication: 36% of reviews highlighted problems, particularly with regard to login errors and session terminations
In-Depth Look at Authentication Issues
Using the Ipiphany platform, the session explored specific complaints about authentication:
Visualization and Insights
A unique feature of Ipiphany, the relationship view, helped visualise connections between different topics in customer feedback. This feature uncovered how login issues were linked to security features and inactivity messages, providing a clear picture of the problems customers faced.
Comparison with Other Banks
Benchmarking showed that M&T Bank had the highest percentage of authentication-related complaints among the banks analyzed. Huntington Bank followed with 25%, while others like Citizens and Chase had much lower numbers.
Signs of Improvement
Despite the low scores in April, there were indications that the situation might improve. The number of daily reviews began to return to normal levels by the end of the month. However, real-time tracking and close collaboration with Product and Insights teams are essential to ensure timely resolution of issues.
The analysis of M&T Bank's decline highlighted 3 key reasons continual monitoring and analysis of your banking app through reviews are important;
Online app reviews are a critical source of data for banks, offering valuable insights into customer experiences. Analyzing these reviews can significantly enhance service delivery and customer satisfaction.
The ability to swiftly identify and resolve issues is essential for maintaining a positive customer experience. Delays in addressing problems can lead to increased dissatisfaction and erode trust.
M&T Bank’s case highlighted how spikes in negative feedback could signal deeper issues that need addressing to prevent long term damage to customer experience.
Video Transcript
00:00:05 - Glenn: Welcome back to another Touchpoint Group Banking App Insights Session. My name is Glenn Marvin and joining me as usual today is the Touchpoint Group Global Head of AI Analytics and CX Strategy, Yazad Karkaria. Thanks for joining us as we delve into the US market looking at the month of April 2024. So, Yazad, talk us through what we're going to be looking at in the US market today.
00:00:30 - Yazad: Sure. Thanks Glenn. So, we are analysing the performance of about 40 banks of financial institutes operating in the US and today, apart from the regular market overview, we are going to be focusing on three main points. Why analyse online app reviews data? We will also demonstrate how important it is to quickly identify issues to ensure quick resolution, and also we will give you the promise of not missing any new insights in your data. So, with that in mind, let's start with the market overview. Here on my screen, I have a few prominent banks operating in the US market, and the numbers that we are looking at on the screen, it's the Engaged Customer Score. So, it's based on all the online app reviews received on Google Play and App Store.
What we do is we take into consideration all the customers who have given a rating along with a comment and analyse those results. It helps us understand why certain customers have rated certain banks the way they have. So, with that in mind, let's take a quick look at how some of these banks have performed.
Starting with Citibank, we can see that Citibank has been a consistently high performer over here with an Engaged Customer Score just about 4.5. Following Citi we have closely, the next one is Bank of America. Bank of America has also been relatively consistent throughout this time period with an Engaged Customer Score in the range of 4 to 4.5 and they're not very far away behind Citi.
Another key bank that we have over here is Capital One. Now with Capital One. We've covered this in the previous sessions as well. The performance had dropped gradually over the last few months, but there is some good news for them in the month of April, as we do see a partial recovery in their performance.
Then, of course, we do have PNC as well. We did cover PNC in the last month where we identified the key reasons for the decline in their performance and we did also highlight that we are seeing signs of improvement and we are happy to see that, yes, in April they have recovered very sharply and pretty much recovered completely to the same level at which they were in December 23 the last bank that we have on the list over here today is M&T Bank.
Now, M&T Bank's performance, as you can see over here, is in a much lower range compared to other banks that we have. But the important thing also is that in the month of April, their performance has dropped to the lowest point in the last 16 months. This is what we are going to try and answer, or uncover the main reasons why their performance has dropped. In order to understand why the performance has dropped, we are going to be making use of our proprietary framework that we have, which is called the Foundational Pillars.
So, with our extensive experience in terms of analysing app reviews data, we've analysed over a million customer comments across over 100 banks globally. We've arrived at this framework where we've identified four main pillars on which any banking app needs to be performing well in order for that app to be successful in that market. The first pillar is Authentication, Design and UI, Reliability and then we have Features and Functionality.
Though they are just four pillars, we have a lot of detail within each of these pillars that we share with our clients and help them get that granular level of insights and understanding that they are after. With this background, let's jump into analysing the performance of M&T Bank. One key thing before we take a look at why their performance dropped, we would like to answer one key question that we promised we'll answer today and that's why is it important to analyse app reviews data.
Here you can see the number of app reviews received for M&T Bank on a monthly basis. We do see that it's consistent somewhere in the range of about 80 to hundreds, some occasional spikes. But in the month of April, there has been a very, very sharp increase of about 5x. It was 132 in March, the number of reviews they received, and it jumped straight to 631 in the month of April, which is the first indicator telling us that there is something going on over here and we need to look at that in more detail.
Let's take a look at their performance. Here on my screen, this blue line is indicating how they've performed over the last six months. So, we can see that in the month of January, 2.7, dropped to 2.1 in February, improved to 2.5 in March, and then dropped sharply to 1.5 in the month of April.
On the left hand side over here, I do have my four foundational pillars that we very briefly spoke about a few moments back and the numbers that we are seeing over here are pain points, which means that any customers who fall under these categories here are highlighting negative experiences with their banking app. So, an example would be that in the month of April, about 77% customers of M&T have highlighted Reliability and Stability related issues, and 36% have highlighted Authentication related issues. Now, 36% is a really, really high number for a banking app to have under the Authentication related foundational pillar, because this is one of the main areas, or hygiene factor that customers expect the app to perform well on because unless they can get into the app, they can't do much.
Now, with that in mind, we have a lot of subtopics over here which give us some granular information as well. We can see that there are sudden spikes in App Login Error, App Session Termination Issue, which is an interesting one. It's popped up from almost nothing to 15%. What we'll now do is we just take a quick look and analyse what's going on here. Here I can see a really quick summary of the Authentication rigid feedback. And if I just want to get a quick sense of how M&T compares to other banks, I can do that by using the benchmark feature over here. Here we can see that, yes, 36% M&T users have highlighted Authentication related issues. This is the worst performing bank out of the entire lot that we are tracking over here. The next one is Huntington, which is about 25%, 1 in 4 Huntington users have faced Authentication related issues, followed by other banks like Citizens, and Chase, with much lower numbers.
Now, what I also want to do is I want to understand what's going on in the Authentication space. Why is it that customers are facing problems or what problems are customers facing here? So, to do that, what I've done is I have actually requested Ipiphany to take help of OpenAI and summarise all Authentication related issues for me.
Here, very clearly, we can see that customers have had generic login related issues. The app often crashes immediately, logs users out some more information about app crashes. The issue persists despite users trying to fix it by updating the app, deleting, downloading and reinstalling it. A very specific one is over here regarding inactivity logout. Users are often logged out due to inactivity immediately after logging in. The app seems to incorrectly detect that the session has been inactive for certain period of time, even though users have just logged in. Users are attributing this to some of the recent update related issues and inconsistent performance of the app. Now, what I want to do is I want to try and understand this in some greater detail because I'm keen to know what's going on over here.
What I will do is I am going to take you to a slightly different view. I will be zooming in so, don't worry. Here, this particular view is the relationship view that we have in Ipiphany, which is a very unique feature which helps us unearth hidden insights. Using this we are able to understand customer feedback holistically by seeing the relationship between the different topics that customers are talking about.
Let me just zoom in a little over here on a specific cluster of topics which customers are talking about. So here let's focus on this particular part. Customers are talking about issues at a broad level and then they are talking about receiving a message which is to do with inactivity and is linked to logging out or being kicked out of the app, and which in turn also is linked to App Security related features.
So using this particular visualisation, I can very clearly form the connection between these different topics that customers are talking about and identify a common problem that needs to be addressed. Now, the thing to keep in mind here is that these topics are not defined by us in the framework. This is, in fact, something that our platform is designed to pick up. Our platform is designed to uncover any new insights coming through to make sure that we don't miss out on any emerging issues over here and this is a very clear example of that happening.
Now, it just doesn't show me these topics and how they are related to one another, but I can actually click on these. So, if I click on message over here, I can get the entire summary, I can get the same explanation that we saw earlier in terms of what is it that customers are talking about and I can read the actual customer comments over here as well.
Another key feature is if I click on this line between Message and Inactivity, it's going to show the comments where customers have spoken about the Message and Inactivity together. So, here we can see the details. 'The app has been working fine since initial install years ago until last week. Now it keeps giving an error message when logging in regarding logout due to inactivity'. So, we get this really, really clear picture in terms of what is it that customers are experiencing and what should the next steps for the product team and the insights team be.
00:10:22 - Glenn: That's pretty insightful, especially bearing in mind that this is on a scale of 1 to 5 the lowest possible score that somebody can give or rating somebody can give is a 1. So, to get an average of 1.5, that must mean there is a very large majority of reviews coming in at that 1 out of 5, which nobody ever wants to get a one star review. Are we seeing any signs of recovery or improvement from M&T?
00:10:49 - Yazad: Yes, definitely. Let's take a quick look and see how things are looking for M&T. Here what I have on this screen is the number of reviews coming through daily for M&T and here in the beginning of the month, the number of reviews is quite consistent to what we had in the earlier time periods. But then we did see the spike which starts spiking around the 4th-5th April, and it continues to spike. The peak, of course, is around the 23rd of April when we received about -the 23rd, and in fact even around the 19th April when we received about 50 reviews in a single day for M&T, which has not happened in the past. But towards the end of the month, we can see this tapering down as well and returning to a similar level as it was in the past.
So, this does indicate that things are returning to normal. But in terms of actual performance, we really need to wait and see next month and track and see if it has improved from 1.5 to anything better than that. As you rightly pointed out, Glenn, it's 1.5 is really, really low and right now we are looking at this retrospectively. But while working with clients, our clients have real time access to this and we are the ones who also work closely with product and insights teams to update them about any budding issues. Real time to make sure that they take the right measures in time before customer experience drops to a really low level.
00:12:10 - Glenn: Thanks once again for such an insightful session, Yazad. And if you'd like us to take a look at your banking app feel free to reach out and we can organise a one on one and give you a no obligation walk through the platform and even benchmark you against some of your competition.