Planning, Implementing, Developing & Deploying in Days!

Discussing the lessions learnt going desktop analysis to deploying on Shiny application in the on Shinyapps.io using a simple example to highlight potential pitfalls to a novel user

Over 150 attendees at the PSI conference played the Exploristics Higher or Lower web app - and didn’t they all do well!? I was delighted to see the game generated so much interest with many people expressed an interest in how we deployed the app on the cloud - that’s the motivation for this blog. This app is a microcosm of developing and deploying a Statistical Shiny app on the cloud. This blog is aimed at people who are experienced in developing desktop R scripts and have a desire to take the next step to create a Shiny app and deploy on the cloud.

The premise of our app was simply, to act as a hook to encourage attendees to visit the Exploristics stand and provide a fun activity for participants to engage in whilst stimulating further discussions around simulations or issues in creating user interfaces for clients to interact and visualize complex statistical models. The challenge was to gather requirements, design, develop, implement, test and deploy an application in only a few days in time for the conference!

The Higher or Lower game is based on the classic card game: an initial card is drawn from a standard deck of cards. Another card is then drawn from the deck and the player must guess if this drawn card will be of higher or lower value than the initial card. If the player guesses correct this process is repeated until

The R code to achieve this code is quite a simple process – create a deck of cards as a data.frame, random sampling this data.frame without replacement a deck of cards using the sample function.

drawnCard <- sample(DeckOfCards, size = 1, replace = FALSE)

The outcome of the guess is also a simple logic-based approach using the > and < operators.

# In the Shiny app the guess is retried from the UI from the inputs
guess <- "<" # or can be ">" from UI

# Assess if the logic of the guess is correct - TRUE
do.call("guess", drawnCard$value,  handCard$value)

Although the methodology may seem trivial, they provide tangible examples of some of the common issues faced. When you attempt to deploy more complex statistical models it’ll be easier to identify the issues you’re facing, how to debug your code and overcome these hurdles!

At Exploristics we use Shiny apps to provide dynamic and interactive visualization tools to enable clients to interpret the complex multi-dimensional data or models generated by KerusCloud. At the Jumping Rivers Shiny in Production conference, I spoke about the experiences and technology to create robust enterprise grade Shiny applications. However, the remit for the Higher or Lower application was very different from how we usually develop software – this was to be a one-off application to be used at the PSI conference and we only had a couple of days for development. This mean that many of our usual processes needed to be accelerated and we had to accept some trade-offs.

Although we have experience in developing Shiny apps using a new technology such as Shinyapps.io usually leads to unexpected changes that take time to resolve. The time frame was extremely short to deal with any technology issues and  address any feedback from stakeholders. However, as our preliminary research into Shinyapps.io looked promising we decided to have a go!.

If I have seen further than others, it is by standing upon the shoulders of giants.

Isaac Newton

R and Shiny are open source and supported by a vibrant and passionate community of individuals. Without the years of effort and innovation from these individuals I would not have been able to deliver the app within our timescales. Any difficulty I faced on my part is not a criticism of these individuals but rather my experience under extremely tight time constraints that resulted in a limited amount of time to research appropriate solutions.  

How should I structure my code?

There are multiple ways to structure the files of a Shiny application and deciding which is the best often comes with experience or researching the best approach based upon your use case. I’d previously had the privilege of presenting along with Colin Fay at the Jumping Rivers Shiny in Production Conference and have read his book Engineering Production Grade Shiny apps but had never formally used the golem 📦. This provided me an opportunity to test out this framework and bring any lessons learnt into other Shiny apps.

Using the golem 📦 package for the first time was a delight, it created much of the boiler plate code for the application, provided functions to carry out common tasks and laid out the process in a systematic order to follow. This enforced good practice such as modularization of code and enabled me to rapidly write production quality code. I’ve previously developed production grade Shiny apps and was already very familiar concepts such as Shiny modules. I’d really recommend reading Colin’s book as it articulates and conceptualizes many of the issues I have faced. My biggest take home message from the book is – “Don’t jump straight into coding!”

How do I store and retrieve data?

Shiny apps are transient by design and sessions don’t persist. This means that all information is lost between using the app. Therefore, if you wish to store information that the app needs to access in the future or provide a central data store you need to provide your own persistent data storage. There’s a great article on Posit for doing providing persistent data storage. For someone used to working on a desktop computer this is an unusual concept as you always have access to your hard-drive and by in large don’t need to worry about access.

Usually, we use AWS S3 as our storage solution, but we didn’t want set up and manage access credentials for a one off project so decided to use Googlesheets as it was well documented and the package was actively developed. Whilst we got this to work locally, we could not get the authorization to work on Shinyapps.io and so decided to use rdrop2 📦 instead.

The rdrop2 📦 has no maintainer on CRAN so we had some initial reservations and we had to write some custom functionality to have refresh-able credentials. This cost us over a day’s development time which in the timescales of our project was huge!

Going forward we will assess the Pins 📦 or use AWS S3, though the AWS solutions is not be suitable for people wanting to avoid a paid for service, although it has a free tier with limited usage.

Failure to build….what do you mean!?

When working with an application, it’s a very iterative process and when you are inexperienced with the technology or rushing this can lead to lots of simple mistakes. Within a Shiny application this can mean the package fails to build, or a failure at the deployment stage or the application crashing when in use. This can be a frustrating process to those not used to it as it can take several minutes to go through these processes before realizing you’ve just nested a bracket in the wrong place!

This can lead to taking shortcuts in manual testing of an application which allows bugs to creep in and only manifest when in production. Structuring your application as a package allows you to leverage resources such as unit testing using testthat 📦 or shinytest2 📦 and automate the testing process, saving time and enforcing that these tests are carried out.

In this project, I often found that I would add a package on my computer but forget to add it to the DESCRIPTION file, this would result in the package not being accessible to the app on Shinyapps.io and failing to build and deploy. To the uninitiated this can be a frustrating experience as identifying the issue often takes much longer than implementing a solution.

But it works on my machine!

Another issue that is conceptually different for those used to developing on the desktop is that different environments can result in an app working on your computer but not in the cloud. This is a very common occurrence in software development which can arise from having different or missing environment variables or the file system being different.

One issue that I had was that the setting of options required for the “gargle” package were silently being dropped when deployed on Shinyapps.io . This mean that locally I could get security tokens to work but they were failing on Shinyapps.io. This was a difficult issue to identify especially as the interactive debugging methods on a desktop can’t be used on a cloud-based environment. This requires the use of the logs to identify the issue.

You’re not meant to use the app that way!

When you develop and subsequently test an application you have an inherent idea of how it should be used. Different people will have a range of IT abilities or understanding of terminology. This often leads to the application being used in a way you hadn’t anticipated, or they hit an unusual use case from a scenario that your testing hadn’t encountered.

When testing the game I’d only played from start to finish and then redeployed with subsequent changes. What I hadn’t realized was that my application had state, I’d reset the user details after each game but not the deck of cards. Effectively this meant a single deck of cards was used across games and eventually when there were no more cards to draw from the deck and the application crashed.

I really enjoyed making the Exploristics Higher or Lower app and I hope everyone enjoyed playing it too. Hopefully the lessons learnt by me will help you on your journey to creating interactive tools for your clients!

Follow Exploristics on Twitter or LinkedIn to get the latest updates clinical trail simulation, interactive data visualization and the