Software Development & Mapping

A tale of two fields by Bamwite Baron Stephen

We’ve all been there. The start. Where you begin with something and wonder if you’ll manage to get through with it. But you tighten your belt and proceed. Because the only other choice is to give up. A definite nono in in these times.

Those were my thoughts as I began my journey, working with Satellite Generated Data so that it could be served in charts and web maps. Looking back, there are a couple of things I probably got right, and others I definitely got wrong. To wit;

React vs Vanilla

There were at least three separate occasions in which we had the discussion of whether to go with plain old Javascript(ES6), or fancy new React(v16.8). The two had strong advantages going for them, namely;

• React streamlines the data movement through whichever application you are building. Couple that with Redux, as we did, and a team will do wonders.

• The relative rigidity that can be implemented for an applications design in React allows people new to it to adopt faster when on-boarding onto a new project.

For some of the projects we built, Javascript was at a disadvantage when it came to the above two points, but held its own in other areas;

• Leaflet seems to be the strongest library out there for rendering web maps. But the philosophies used to display information to a user are different between it and React (React uses a virtual DOM). With Javascript, this is not an issue and so it provides much easier access and manipulation to the Leaflet API.

• Libraries also suffer from the above mismatch. Mostly built in Javascript, it becomes an issue if one, in a quest to solve a problem, comes across a leaflet library.

We built several applications using both tools and everyone on the team took full advantage of the stated advantages. This was therefore a choice we changed on different occasions and then had to work around the accompanying challenges.

Databases

Satellite Generated data is the kind with an extra zing. And for that, special tools are developed database-wise to efficiently deal with it (PostGis). Going back to that initial moment, when one hasn’t implemented something before, it was decided that this would be a good way to go, in dealing with how we would store clients’ data. But then came deploying all this, in which GCP’s App Engine (AE) was used (abstracts away a lot of dev-ops for the lean team hence its pick). However, Gdal is a core library needed by GeoDjango (sits on top of the PostGis enabled database) and thus requires a flexible AE environment to implement a successful deployment. Cost-wise it was a poor choice, for we saw low traffic on days on which there were no updates. Added to that was the fact that no use was made of the powerful tools GeoDjango came with, which was simply because most transformations were actually made on the lower-dimension data (lists and dictionaries of various values). The more efficient choice was thus made of storing the geographical data as jsons, which can be seen in this commit. As we scaled, we did see the occasional need for the Django module, such as in the conversation below, but ways around the hurdles were nevertheless found.

Automation

This is a concept we got wrong by simply not implementing. To be fair, the numbers were thin and the schedule was tight. In any case, an automated Extract-Transform-Load pipeline would have been beneficial in scaling the application’s periodic data updates. Thankfully the team was stellar and able to generate and run scripts easily, making this a lower priority task.

Looking at the available options on Google-Cloud-Platform (GCP), App Engine’s Cron service was a tool we were looking at leveraging so that we could simultaneously generate the data from Sentinel-Hub’s API (Cloud Functions would likely have played a role here as well) and ingest it into the databases feeding the  various platforms.

Testing

Last but most definitely not least. Focusing on the frontend, testing Leaflet powered applications appears to not be done that often. But a few tries have been made. Coming from  the experience of testing asynchronous functions in python, I recognize the challenges likely to be faced in approaching the unknown. And let’s not forget the “React vs Vanilla” debate, for it is easy to test React-based applications, but not the latter. Solutions do exist, but as mentioned, Leaflet will have a significant part to play in them.

A possible solution is to use Selenium which does have a learning curve, but doesn’t everything? 

Photo by USGS.