Introduction #
What is an App server? #
An App Server (short for Application Server) can have many different definitions and meaning according to its context. I will primarily be talking about an App Server in context of its use in RADAR-base platform as a mediation service.
In layman terms, an app server is a piece of software that supports the working of a mobile/web application (or any frontend application) by mediating interactions with other services. The app server is isolated from the frontend app and runs on the backend consuming, processing and providing important information from and to the frontend app.
An Application Server is an integral part of a multi-tier software application. The Frontend mobile/web application is used primarily for presenting the information to the user and for taking information from the user.
Then the application server processes the request received from the frontend app (either itself or proxying the request to other components. This is shown in the diagram below –
A single frontend app can have multiple app-servers supporting it or a single app-server fulfilling various different functions. This helps making the application as lightweight as possible for better user experience. This also allows complex computational scenarios which cannot be handled by limited resources on mobile hardware in particular (battery) , added computation resources required for certain tasks (like image classification) among many others which we will see later in this blog post.
If there are multiple frontend apps that rely on same type of information and processing then the app server can be used as a reusable entity lowering the overhead of developing, deploying and maintaining software. One such example is having a Frontend website and mobile app that fulfil the same client-side functionality.
Motivation #
In this section, I will discuss the motivation for moving towards using an Application Server for RADAR-Questionnaire/ Active Remote Monitoring App (aRMT) instead of a standalone mobile application.
At first, we started out by creating a standalone Cordova/Ionic cross-platform application for the active monitoring (see: Questionnaire app component of the RADAR-base platform using our Apache Kafka based backend to ingest data (using HTTP REST endpoints of Confluent Rest Proxy and a management portal for managing users and projects and authentication/authorisation across the platform.
The app worked fine on Android devices using the native plugins provided in Cordova. The problems started out when the functionality of the app was continuously extended and ended up being hard to maintain.
One of the major problems we saw with the app was with the Notifications due in part to the long term data collection periods (e.g. upto 2 years) of the projects in RADAR-CNS which meant scheduling hundreds of notification for the app in advance.
Background on the local notifications #
Why we chose local notifications? #
We chose Local Notifications as it was very easy to implement as Ionic supports them out of the box. These were very fast to get off the ground and start testing. Additionally, because some of the questionnaires had short notification delivery windows (10 minutes in the case of Experience Sampling Method or ESM questionnaires) local notifications, not depending on the network seemed like a good idea.
When the initial designs were first considered around 3 years ago network connectivity was worse back then and fewer participants had data plans. These days most people are connected to the internet at all times and it is only growing as shown in this graph.
Both of the above points were in favour of Local Notifications and against Push Notifications (which were not so mature back then too). Local notifications, unlike Push Notifications, have no network dependency and do not need any additional backend software, infrastructure and resources. Though we invested several months in this, the problems with the local notifications on Ionic/Cordova were complex and could not readily be fixed to a degree where they worked well enough.
Problems encountered with local Notifications? #
Due to the long term study and the amount of notifications to be scheduled, some vendors (like Samsung) put a hard limit (500 for Samsung) on the number of notifications (in the AlarmManager that can be scheduled from one app. So notifications from our app were blocked by the OS after a period of time. This can be solved if the application is a native android application (see below) but there is no way when using Cordova plugins (without a low level re-working of the plugin code, and even then, it may not be possible or may not work as expected).
Cordova as a cross-platform framework is essentially a web application and cannot run on the device in the background for long periods of time, hence even where notifications were being scheduled correctly, they did not display as expected (e.g. when you restart the phone all notifications used to pop-up at once). We tried various different tweaks and hacks to work-around this problem but albeit better, they were still not working as intended.
The main problems associated with the Cordova Local Notification plugin were experienced on devices running Android 8.0 Oreo or higher. The plugin had not been updated for more than 8-9 months and Android had made some drastic changes to their notification architecture.
Furthermore, Because of the variety of devices, hardware, vendors and custom softwares in the Android ecosystem, each of the device can behave differently even if the version of Android is same, especially when using a hybrid application instead of a natively built application.
Most of the problems associated with the plugin on Android 8.x have already been heavily discussed on the official Github repo issues, some of which are closed and some still open. Most of the early issues involved not getting notifications when the app was closed or running in the background, not receiving notifications when device was restarted or issues relating to new notification channels introduced in Android 8.0 Oreo.
Our (failed) attempts to fix the Problems #
We tried for a long time to get the notifications to work on the aRMT app. The app schedules a lot of notifications to fill out questionnaires and also generates schedules for the questionnaires based on a protocol which is specific to the study of which the participant is a part of. It also has added functionality of performing clinical tasks which changes the schedule (and hence the notifications) every time it is performed. This complicates the debugging since multiple factors influence the delivery of notifications and thus various break points are introduced. So after several attempts and hard work to get the app working, with little progress, we decided to test local notifications this in a much simpler environment.
The early versions of the aRMT app scheduled a lot of notifications and every time it was opened, it cancelled and rescheduled the notifications since the schedule could have changed. In total we had more than 600 notifications scheduled at the start because of the long duration of the studies. Considering the magnitude of these, we decided to create a very basic application called notify test to test the working of the Cordova local notification plugin on different devices with different android versions in a much simpler and less complex way. The test app has options to schedule a notification right now, to schedule a single notification sometime in the future and to schedule n number of notifications with a specified interval. The last case is most useful to us since it is similar to what we need for the aRMT.
This app was based on a similar architecture of the aRMT with ionic on top of Cordova with local notification plugin.
Surprisingly, most of the functionality around notifications worked straight away on the test app on Samsung S8 with android 8.0 thanks to the 0.9-beta-3 release of the Cordova local notification plugin. So we narrowed down the problem to the bulk notifications scheduled by the aRMT app.
We only had Samsung devices with android 8.0 for testing. All our other devices where android 7.x or earlier. So for a fair amount of time we assumed that the problem was with android 8.0, but when the plugin worked well with the test app above we started to realise that the problem could be with Samsung specific handsets, and we were right. Looking closely at the logs, when the aRMT app was opened, it cancelled and rescheduled some notifications, at this time the logs reported an error message from SamsungAlarmManager. The same message as reported on SO here and still persists as reported in this recent issue.
Assuming problem was with scheduling notifications in bulk we tried a work-around where we schedule only first 100 notifications and then schedule the next 100 on trigger of the 100th notification from the previous set. The problem was not solved with this because apparently the Pending Intents still stay around in the system even after triggering and delivery. Also the app needed to reschedule the notifications (cancelling and scheduling all again) in cases the protocol for the schedule changed. The cancelling using the Cordova local notification plugin does not actually cancel the pending intents but creates identical pending intent to cancel the notification, hence the cancelling adds pending intents instead of removing them. All of this causes huge amount of pain and debugging it gets harder at each step. This is easy to fix in Android native code (as evident from this SO answer) but not possible on Cordova unless we gain a deep understanding of the plugin and extend the plugin code ourselves which requires substantial resources and gets hard to maintain and support over time.
There were also several associated problems but suffice to say we were not making progress with local notifications and decided the time was ripe for a new approach.
Push notifications to the Rescue #
So finally without being able to fix the problems with local notifications we migrated to use the Push Notifications using Firebase Cloud Messaging (FCM) and I am so glad we did. It solves all the problems we faced with local notifications and provides so much more value as we will discuss below.
Push notifications were relatively easy to get started with using the already available Cordova plugin for FCM; however, there were a few caveats, the plugin mentioned above did not implement the upstream functionality present in the FCM SDK and we needed this to send messages to the server for scheduling notifications in an easy and reliable way. In hindsight, this was probably not the best decision but at the time this was a critical fix as the app was in production and needed to be working as soon as possible — after all an active remote monitoring app that needs to deliver questionnaires on a specified protocol must have a means to notify or it simply won’t work. With these thoughts in mind, we extended it and added the Upstream functionality for the android devices (all the devices were android in our case although more recently we have turned our attention to the iOS implementation). Another caveat was that there was no proper server side java library for FCM XMPP protocol (so it can consume upstream messages). Fortunately, we found an implementation of it using the open source Smack library on GitHub. We forked the Repo and added multiple extensions to it to cater to our needs. These improvements are listed on the README of our implementation of the FCM XMPP server. We also updated our simple test app notify test to add support for push notifications to ease testing.
Another caveat was that push notifications needed an internet connection on the device to be delivered. But the pros outweigh the cons by a fairly large margin and as we will see later in this post, the stats show that only a small amount of notifications were not received principally as a result of network connectivity, currently ~10% though we expect this to decrease as connectivity improves in the future (see Fig 3).
Centralising logic on the server means more control and insight into the notification system remotely (as you have access to the server) than you can on individual devices in the case of local notifications. This not only makes debugging easier but also provides metrics on the process down to the individual participant. This was very helpful when using the XMPP protocol for sending downstream messages, FCM provides delivery_receipt, so we can know if an individual notification was delivered along with aggregate stats in the Firebase dashboard (Fig 4) Another major advantage of this system is that it is independent of the hardware and the software of the device that’s running the application. It will keep working regardless of which platform,OS version,UI,etc the device is on and also ensures backwards and future compatibility. This is very important in case of web applications as the underlying frameworks and native plugins don’t evolve as quickly as the operating system does(as seen in the previous section).
Another dilemma we were faced with when developing the notification server for our app was which type of message to use. Firebase cloud messaging has support for 2 types of messages which can be sent downstream. As you can see in the docs that data messages are handed over to the app to be handled. If we sent a notification this way, FCM will hand over all the content in the data payload to the app and then the app will handle the display of the notification. Notification Messages on the other hand are automatically displayed as a notification on the device by FCM without any callback to the application. If the notification is clicked then the app is opened with a bundled intent of the optional data that was sent with the notification.
We chose the Notification Messages as a mode of sending notification as we wanted to minimise any dependance on Cordova for the new notification system. Also there may be complications in handling incoming messages if the app was not allowed to run in the background.
Below is shown the diagram of the FCM XMPP server that we deployed for RADAR-CNS.
The above architecture worked really well and stable in most of our tests so we decided to go into production as soon as possible to solve the notifications issue. Below is a plot of Firebase notification metrics for Sent, Received, Displayed and Opened notifications which is a marked improvement compared to the local notifications system). Also mentioned above is the collection of metrics in the app via Firebase, here we have more insight and debugging information. Further to this, through the use of Firebase Events we can collect granular data on each user interaction with the app such as opening a questionnaire, completing a questionnaire, successful data sends, notification dismissed etc. this subject matter will be covered in a future post.
RADAR App Server and other enhancements #
While the first iteration of remote notifications was a major step forward, we first noticed problems with the above system when we were running this for quite sometime in production and found out that this was not flexible. It was great at only one task, i.e. scheduling and sending notifications. But the database used in the above platform was hard to evolve and it did not integrate well with the management platform of RADAR-platform and was not able to perform or provide insight on anything else.
The major issue was, as you can see from the above plot, that user compliance (% of opened notifications) was somewhat low and the project admins needed an individual level insight on which notifications/users are delivered/active so they can intervene and improve the compliance.
We have 30 days worth of logs (which can be increased) on the FCM XMPP notification server which contain delivery information but it was hard to get, parse and interpret that data. Also it will be very hard to deliver this info in a secure, authorised manner to project admins without somehow integrating this with the Management platform of RADAR – The Management Portal. Lastly, this does not guarantee that the notification is displayed, it just denotes that the notification reached the device but individual device level settings may not allow the notification to be displayed (eg – Do not disturb is on, etc). So these information are hard to extract, interpret and use.
So we decided to solve these problems with two new solutions –
- Add FCM analytics into the app so as to get insight into the low compliance. The value of these insights cannot easily be understated (discussion for another blog post perhaps).
- Implement a new General Purpose Application server (RADAR-base App Server) which will serve as more than just a notification scheduler. We decide to use the most mature Java framework – Spring Framework and Spring boot for easy development and deployment of the application. We also decided to support the legacy protocol (XMPP) as the previous server (as delivery receipt is important) and also expose a RESTful service for the same tasks (and more). This had multiple benefits including a Response to the clients for each request, ability to flexibly add/update/delete and query the information, set a hierarchy of different entities, batch scheduling of notifications, sending notifications from other automated systems to user using RADAR specific user identifiers and security mechanisms, etc. The advantages were abundant and the disadvantages none.
For the App Server, we used Spring JPA and Liquibase for easy management of Database queries and its evolution. At first instance it is geared specifically towards notifications but can be easily extended and already has a user metrics data for different user related info (list the last time the app was opened, etc).
This app server currently under development as part of the RADAR-base platform project which spun out of the 22 million IMI RADAR-CNS project.
The most recent work on the app server is located on GitHub and its README.md file has some basic documentation about its features and usage.
But because of the complex nature of backend applications these days it is important that the architecture be explained properly for others to understand your code and the use-cases they can use or extend the app-server in. This also helps explain the abstraction and composition of different components within the application and helps developers collaborate and make suggestions for improvements.
Below is shown a very high level view of the new App Server similar to the one above –
We will discuss the detailed architecture for the app server in the next post. Stay tuned!
