What is an App server?
An App Server (short for Application Server) can have many different definitions and meaning according to its context. I will primarily be talking about an App Server in context of its use in RADAR-base platform.
In layman terms, an app server is a piece of software that supports the working of a mobile/web application (or any frontend application). The app server is isolated from the frontend app and runs on a backend consuming, processing and providing important information from and to the frontend app.
To explain this better consider the following example, consider an analogy to a large e-commerce organisation (like Amazon). There are sales and support people who are directly in contact with customers and support them in various ways (could be providing appropriate information about products, solving any issues, taking orders, receiving feedback, etc). But this is only what a customer perceives from his point of view. But in reality the sales/support staff need help from various other departments to complete these tasks properly. This is shown in the very nerdy (ASCII) diagram below. Imagine what would happen if the IT/Software staff was unavailable and a customer was unable to open the website. The customer contacts the support department but the support staff don’t have any idea of how to solve this issue as the IT/Software department is not present. They can only make assumptions and provide a temporary reassurance to the customers.
This is very similar to how a multi-tier software application works. The Frontend mobile/web application is used primarily for presenting the information to the user and for taking information from the user.
Then the application server processes the request received from the frontend app (either itself or proxying the request to other components — analogy to different departments in the above example). This is shown in the diagram below –
Similar to the sales and support staff requiring support form multiple different departments, a single frontend app can have multiple app-servers supporting it or a single app-server fulfilling various different functions. This helps making the application as lightweight as possible for better user interface. This also allows complex computational scenarios which cannot be handled by limited resources on mobile hardware. Other benefits includes low battery consumption for mobile devices and the list goes on.
If you have multiple frontend apps that rely on same type of information and processing then the app server can be used as a reusable entity lowering the overhead of developing, deploying and maintaining software. One such example is having a Frontend website and mobile app that fulfil the same functionality.
As discussed above, there are multiple advantages of using an app server and that list is not exhaustive. In this section, I will discuss the motivation for moving towards using an Application Server for RADAR-Questionnaire/ Active Remote Monitoring App (aRMT) instead of a standalone mobile application.
At first, we started out by creating a standalone Cordova/Ionic cross-platform application for the active component of the RADAR-base platform since we already had a Apache Kafka based backend to ingest data (using HTTP REST endpoints of Confluent Rest Proxy) and a management portal for managing users and projects and authentication/authorisation across the platform.
The app was working fine on Android devices using the native plugins provided in Cordova. The problems started out when the functionality of the app was continuously extended and ended up being hard to maintain and keep out the crawling bugs.
One of the major problems we saw with the app was with the Notifications. Because of the long term duration of the projects in RADAR-CNS that means scheduling hundreds of notification for the app.
Background on the local notifications
At first we implemented the aRMT app using local notifications (that don’t need internet access) but there were severe complications with it. Because of the long term study and the amount of questionnaires to be scheduled, some vendors (like Samsung) put a limit of 500 on the number of notifications (in the AlarmManager) that can be scheduled from one app. So our notifications were being stopped after a period of time. This can be solved if the application is a native android application but there is no way when using Cordova plugins (except for diving deep into the plugin code and updating it ourselves which is not feasible and it may not be possible/not work as expected even then).
We also faced problems using local notifications because of the cross-platform Cordova app as this is essentially a web application and cannot run on the device in the background for long periods of time hence not showing notifications. Then when you restart the phone all notifications used to pop-up at once. We tried various different tweaks and hacks to find a work-around for this problem but albeit better were still not working as intended.
The main problems associated with the Cordova Local Notification plugin were experienced on devices running Android 8.0 Oreo or higher. The plugin has not been updated since the last 8-9 months and Android has made some drastic changes to their notification architecture.
Furthermore, Because of the variety of devices, hardware, vendors and custom softwares in the Android ecosystem, each of the device can behave differently even if the version of Android is same, especially when using a hybrid application instead of a natively built application.
Most of the problems associated with the plugin on Android 8.x have already been heavily discussed on the official Github repo issues, some closed and some still open. Most of the early issues involved not getting notifications when the app was closed or running in the background, not receiving notifications when device was restarted or issues relating to new notification channels introduced in Android 8.0 Oreo.
We were trying for a long time to get the notifications to work on the aRMT app. The app schedules a lot of notifications to fill out questionnaires and also generates schedules for the questionnaires based on a protocol which is specific to the study of which the participant is a part of. It also has added functionality of performing clinical tasks which changes the schedule every time it is performed. This complicates the debugging since multiple factors and thus break points are introduced. So after several attempts and hard work to get the app working, with little progress, we decided to test this in a much simpler environment.
The early revisions of the aRMT app scheduled a lot of notifications and every time it was opened, it cancelled and rescheduled the notifications since the schedule could have changed. In total we had more than 600 notifications scheduled at the start because of the long duration of the studies. Because of the magnitude of these, we decided to create a very basic application called notify test to test the working of the Cordova local notification plugin on different devices with different android versions in a much simpler and less complex way. The test app has options to schedule a notification right now, to schedule a single notification sometime in the future and to schedule n number of notifications with a specified interval. The last case is most useful to us since it is similar to what we need for the aRMT.
This app was based on a similar architecture of the aRMT with ionic on top of Cordova with local notification plugin.
Surprisingly, most of the functionality around notifications worked straight away on the test app on Samsung S8 with android 8.0 thanks to the 0.9-beta-3 release of the Cordova local notification plugin. So we narrowed down the problem to the bulk notifications scheduled by the aRMT app.
We only had Samsung devices with android 8.0 for testing. All our other devices where android 7.x or earlier. So for a fair amount of time we assumed that the problem was with android 8.0, but when the plugin worked well with the test app above we started to realise that the problem could be with Samsung specific handsets, and we were right. Looking closely at the logs, when the aRMT app was opened, it cancelled and rescheduled some notifications, at this time the logs reported an error message from SamsungAlarmManager. The same message as reported on SO here.
Assuming problem was with scheduling notifications in bulk we tried a work-around where we schedule only first 100 notifications and then schedule the next 100 on trigger of the 100th notification from the previous set. The problem was not solved with this because apparently the pending intents still stay around in the system even after triggering and delivery. Also the app needed to re-schedule the notifications (cancelling and scheduling all again) in cases the protocol for the schedule changed. The cancelling using the Cordova local notification plugin does not actually cancel the pending intents but creates identical pending intent to cancel the notification, hence the cancelling adds pending intents instead of removing them. All of this causes huge amount of pain in the backside and debugging it gets harder at each step.
There were also several other problems with this but I think I have talked enough about the background of local notifications.
Push notifications to the Rescue
So finally without being able to fix the problems with local notifications we migrated to use the Push Notifications using Firebase Cloud Messaging (FCM) (FCM) and I am so glad we did. It provides so much more value than just solving the problems with local notifications as we will discuss below.
It was relatively easy to get started with and there was already a Cordova plugin for FCM. But there were a few caveats, the plugin mentioned above did not implement the upstream functionality present in the FCM SDK (we needed this so as to send messages to the server for scheduling notifications in an easy and quick way. In hindsight, this was probably not the best decision but at the time this was a critical fix as the app was in production and needed to be working as soon as possible — An Active Remote Monitoring app supposed to deliver questionnaires on a specified protocol without notifications – What a Joke!). So we extended it and added the Upstream functionality for the android devices (all the devices were android in our case and we did not have any IOS developers in our team). Also there was no proper server side java library for FCM XMPP protocol (it can consume upstream messages). But fortunately, we found an implementation of it using the open source Smack library on GitHub. We forked the Repo and added multiple extensions to it to cater to our needs. These improvements are listed on the README of our implementation of the FCM XMPP server.
Another caveat was that push notifications needed internet to be available to be delivered. But the pros outweigh the cons by a fairly large margin.
You also get more control and insight into the notification system remotely (as you have access to the server) than you can on individual devices in local notification. This also makes debugging easier and helps faster resolving of issues. Another great plus was that when using the XMPP protocol for sending downstream messages, FCM provides delivery_receipt, so we can know if a notification was delivered or not.
Another dilemma we were faced with when developing the notification server for our app was which type of message to use. Firebase cloud messaging has support for 2 types of messages which can be sent downstream. As you can see in the docs that
data messages are handed over to the app to be handled. If we sent a notification this way, FCM will hand over all the content in the data payload to the app and then the app will handle the display of the notification.
Notification Messages on the other hand are automatically displayed as a notification on the device by FCM without any callback to the application. If the notification is clicked then the app is opened with a bundled intent of the optional data that was sent with the notification.
We chose the Notification messages as a means of sending notification as we did not want to rely on Cordova at all for the new notification system. Also there may be complications in handling incoming messages if the app was not allowed to run in the background.
Below is shown the diagram of the FCM XMPP server that we deployed for RADAR-CNS.
The above architecture worked really well and stable in most of our tests so we decided to go into production as soon as possible to solve the notifications issue once and for all. Below is shown the plot of firebase notification metrics for Sent, Received, Displayed and Opened notifications which in my opinion is looking great (Compared to the crappy local notifications system). Here we have much more insight and debugging info than we can ever imagine on a local notification system.
RADAR App Server
The problems with the above system arose when we were running this for quite sometime in production and found out that this was not flexible. It was great at only one task of scheduling and sending notifications. But the data base used in the above platform was hard to evolve and it did not integrate well with the management platform of RADAR-platform and was not able to perform or provide insight on anything else.
The major issue was, as you can see from the above plot, that user compliance (% of opened notifications) was very low and the project admins needed an individual level insight on which notifications/users are delivered/active so they can intervene and improve the compliance.
We have 30 days worth of logs (which can be increased) on the FCM XMPP notification server which contain delivery information but it will be hard to get, parse and interpret that data. Also it will be very hard to deliver this info in a secure, authorised manner to project admins without somehow integrating this with the Management platform of RADAR – The Management Portal. Also, this does not guarantee that the notification is displayed, it just denotes that the notification reached the device but individual device level settings may not allow the notification to be displayed (eg – Do not disturb is on, etc). So these information are hard to extract, interpret and use.
So we decided to solve the problem with two new solutions –
- Add FCM analytics into the app so as to get insight into the low compliance. These have been one of the most useful addition (discussion for another blog post).
- Implement a new General Purpose Application server which will serve as more than just a notification scheduler. We decide to use the most mature Java framework – Spring Framework and Spring boot for easy development and deployment of the application. We also decided to support the legacy protocol (XMPP) as the previous server and also expose a RESTful service for the same tasks (and more). This had multiple benefits including a Response to the clients for each request, ability to flexibly add/update/delete and query the information, set a hierarchy of different entities, batch scheduling of notifications, sending notifications from other automated systems to user using RADAR specific user identifiers, etc. The advantages were endless and the disadvantages none.
For the App server, we used Spring JPA and Liquibase for easy management of Database queries and its evolution. At first instance it is geared specifically towards notifications but can be easily extended and already has a user metrics data for different user related info (list the last time the app was opened, etc).
The app server currently under development as part of the RADAR-base platform project which spun out of the 22 million IMI RADAR-CNS project.
The most recent work on the app server is located on GitHub and its
README.md file has some basic documentation about its features and usage.
But because of the complex nature of backend applications these days it is important that the architecture be explained properly for others to understand your code and the use-cases they can use or extend the app-server in. This also helps explain the abstraction and composition of different components within the application and helps developers collaborate and make suggestions for improvements.
Below is shown a very high level view of the new App Server similar to the one above –
We will discuss the detailed architecture for the app server in the next post. Keep tuned!