The Kloudless Events API provides a queryable event stream for developers to monitor user activity using WebHooks. The more popular upstream services like Google Drive provide an event stream natively, but other upstream services don’t support it. However, Kloudless aims to provide a unified developer experience. Kloudless periodically polls the Jive API to recreate an event stream since it isn’t natively supported. Kloudless will then notify developers via WebHooks whenever events have been updated.

In this blog post, we will first introduce the Kloudless Events API and then explain how we implement the Jive event stream.

Kloudless Events API

Developers primarily use the List Events endpoint to retrieve an event stream of the upstream service. The Kloudless Events API supports creating, modifying, or deleting resources. They are represented as a stream of event objects. We currently present 7 types of events: add, delete, update, rename, move, copy, restore.

Here is an example of the response after querying the List Events endpoint:

As mentioned previously, developers can subscribe to Kloudless’ WebHooks, which will notify them when new events become available for a connected account of an upstream service belonging to your application.

Poll Jive Contents

Because Jive doesn’t natively provide an event stream, Klouldless polls Jive periodically, every 5 minutes for example, to obtain modified resources. The Kloudless Unified Cloud Storage API focuses on Jive Content objects, and Jive provides a Get Contents API endpoint, which will return a paginated list of content objects. We can then set the query parameter modificationDate(beforeDate, afterDate) between the polling periods. Then we can retrieve and sort according to latestActivityAsc to retrieve the latest activities first.

Here is the request example:
https://example.jive.domain/api/core/v3/contents?filter=modificationDate(2019-02-28T15:05:00Z,2019-02-28T15:00:00Z)&sort=latestActivityAsc

Using this Get Contents endpoint, we can obtain newly created or modified Content resources within the last polling interval. We can create an event stream from the parsed information and then notify developers via WebHooks.

Poll Jive Deleted Contents

However, this event stream still lacks delete events. Jive provides another API endpoint called Get Deleted Contents, which can list Content objects that have been deleted. We can also set the query parameter since based on the last polling interval and set sort according to eventDateAsc to retrieve the latest activities first.

Here is the request example:
https://example.jive.domain/api/core/v3/contents?filter=since(2019-02-28T15:05:00Z)&sort=eventDateAsc

We utilize since instead of modificationDate(beforeDate, afterDate) for this API endpoint, so we need to filter out any events that are later than beforeDate. If we don’t sync the request time period of Get Contents and Get Deleted Contents, then the event stream will be in the wrong order and may skip event data.

For example:
t means timestamp
The first round responses are:
Get Contents: afterDate, t1, t2, beforeDate
Get Deleted Contents: since(afterDate), t1, t2, beforeDate, t4
And in second round responses are:
Get Contents: beforeDate, t3
Get Deleted Contents: since(beforeDate), t4
Assume t3 < t4
If we don’t filter out t4 in first round, we would return t4 first, and then t3 in the second round, causing events to be out of order.

Checkpoints

Checkpoints are points in time that we store when polling the different API endpoints. The simplest way to set checkpoints is to store the time of the latest event plus one millisecond (because we retrieve data less than or equal to beforeDate). However, this condition only exists when the response includes all events, and there is no pagination. If we need to process additional data, then we need to make additional queries. The simplest checkpoint may be incorrect and result in edge cases.

For example:
t means timestamp:
Page 1: t1 t2 t2 t3 t3
Page 2: t3 t4 t4

In this case, the same timestamp may appear across two pages. When we retrieve page 1, we will set the checkpoint as t2 + 1 and not store t3 t3.

Because Kloudless needs to poll both API endpoints, we maintain two internal checkpoints.

Conclusion

To avoid the complex logic of monitoring Jive events, developers can just use the Kloudless Events API instead.