Credits: http://gatelogsystems.com/wp-content/uploads/2018/04/database-graphic-hi-res.jpg
For many years, Apache CouchDB and Cloudant have encouraged developers of mobile apps to:
- Create a database per user
- Replicate that database to the user’s device
- Use the CouchDB replication protocol to sync data from device to cloud, cloud to device, or in both directions
This approach creates a clear boundary between individual users’ data, making the leaking of data between users impossible (bar the loss of security credentials, of course). In practice, when an application becomes popular and generates thousands or even hundreds of thousands of users, it becomes tricky to deal with many small pots of data. It becomes especially difficult if you want to deal with the data set as a whole for backup purposes or for reporting and analytics.
This is where Cloudant Envoy comes to the fore. It is designed to sit between the mobile device and the Cloudant service. The client-side app remains unchanged. It still thinks it is using a open-database-per-user approach, but Cloudant Envoy seamlessly modifies the data and stores it in a single Cloudant database. This approach gives the best of both worlds:
- the client-side code only has one user’s data stored locally where it can be used in an Offline-First manner
- the server-side database is all in one place allowing simple, incremental backup and querying across user data sets
Cloudant Envoy is an open-source project actively managed by IBM. It’s released under the Apache-2.0 license, so you can use it for free in your own application. Envoy makes it simple to create offline-first applications that continue to work even with no network connection, giving your uses 100% uptime and a great user experience.
How does Cloudant Envoy work?
Envoy is a web application that sits between your mobile Client and Cloudant:
+—————+ +—————+ +—————+
| Mobile device | <——–> | Envoy | <——–> | Cloudant |
+—————+ +—————+ +—————+
The mobile device thinks it’s talking to a Cloudant/CouchDB service, as Envoy emulates enough of the Cloudant API that it can accept pull or push replication API calls. Envoy’s main job is to keep each mobile users’ data separately. It does this by manipulating the document ids.
- Mobile device tries to write document id abc to Cloudant
- Envoy modifies the id to be <hash of user id>:abc – prefixing the id with the hash of the requesting user’s id. This keeps each user’s data sorted in the index and compartmentalised so that other users can’t access it.
- If the mobile device requests document abc, Envoy performs the same id manipulation and fetches <hash of user id>:abc instead.
The effect of this is to allow many thousands of users to replicate with a single Cloudant database – the mobile device is under the illusion that is using a one-database-per-user architecture, but the server-side database is one database for all users.
In addition to manipulating the document IDs as they pass through, Envoy also keeps an SQLite database of changes. This allows the changes feed to be indexed by the user, making it the speed of fetching the changes feed to remain constant as the size of the database grows.
Why Cloudant Envoy?
Database-per-user is a common pattern with CouchDB when there is a requirement for each application user to have their own set of documents which can be synced (e.g. to a mobile device or browser). On the surface, this is a good solution – Cloudant handles a large number of databases within a single installation very well. However, there are some problems:
- Querying data across many databases (e.g. for analytics) can be difficult. Cloudant does not have native support for a cross-database query and aggregating the data into a single database using e.g. replication can be resource intensive or require manual scheduling.
- There is a high management cost of many (thousands of) databases. Consider backup, replication in cross-region scenarios, design document management. These all require coordination/resource management outside of Cloudant.
Envoy aims to work around these problems by emulating database-per-user using a single backend database and a proxy.
How to Install Cloudant?
Via npm
Cloudant Envoy is published to npm. It can be installed and run if you have Node.js and npm installed:
| npm install -g cloudant-envoy export COUCH_HOST=’https://key:passwd@account.cloudant.com’ envoy |
Deploy to Bluemix
Deploy Cloudant Envoy to Bluemix by clicking the Deploy to Bluemix button below.
Manual installation
To install the code yourself, clone the repo and run npm install. The Envoy server needs admin credentials for the backing Cloudant database, and it expects a COUCH_HOST environment variable to be set:
| export COUCH_HOST=’https://key:passwd@account.cloudant.com’ |
After those variables are set, you can start the Envoy server with npm start. Note that the port is the port that Envoy will listen to, not the port of the Cloudant server.
That’s it. You have successfully created a Cloudant database. Enjoy using it and let us know comments if you have any doubts or suggestions. (give contact details?)
Feel Free to share your thoughts in the comment section below.
Don't forget to follow us on Twitter, like our Facebook Fan Page and Add us to your circles on Google+ to keep you updated with the latest technology news, gadget reviews, launches around the world and much more
